text:
PEP: 492 Title: Coroutines with async and await syntax Author: Yury
Selivanov<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015,
27-Apr-2015, 29-Apr-2015, 05-May-2015

Abstract

The growth of Internet and general connectivity has triggered the
proportionate need for responsive and scalable code. This proposal aims
to answer that need by making writing explicitly asynchronous,
concurrent Python code easier and more Pythonic.

It is proposed to make coroutines a proper standalone concept in Python,
and introduce new supporting syntax. The ultimate goal is to help
establish a common, easily approachable, mental model of asynchronous
programming in Python and make it as close to synchronous programming as
possible.

This PEP assumes that the asynchronous tasks are scheduled and
coordinated by an Event Loop similar to that of stdlib module
asyncio.events.AbstractEventLoop. While the PEP is not tied to any
specific Event Loop implementation, it is relevant only to the kind of
coroutine that uses yield as a signal to the scheduler, indicating that
the coroutine will be waiting until an event (such as IO) is completed.

We believe that the changes proposed here will help keep Python relevant
and competitive in a quickly growing area of asynchronous programming,
as many other languages have adopted, or are planning to adopt, similar
features:[1],[2],[3],[4],[5],[6].

API Design and Implementation Revisions

1.  Feedback on the initial beta release of Python 3.5 resulted in a
    redesign of the object model supporting this PEP to more clearly
    separate native coroutines from generators - rather than being a new
    kind of generator, native coroutines are now their own completely
    distinct type (implemented in[7]).

    This change was implemented based primarily due to problems
    encountered attempting to integrate support for native coroutines
    into the Tornado web server (reported in[8]).

2.  In CPython 3.5.2, the __aiter__ protocol was updated.

    Before 3.5.2, __aiter__ was expected to return an awaitable
    resolving to an asynchronous iterator. Starting with 3.5.2,
    __aiter__ should return asynchronous iterators directly.

    If the old protocol is used in 3.5.2, Python will raise a
    PendingDeprecationWarning.

    In CPython 3.6, the old __aiter__ protocol will still be supported
    with a DeprecationWarning being raised.

    In CPython 3.7, the old __aiter__ protocol will no longer be
    supported: a RuntimeError will be raised if __aiter__ returns
    anything but an asynchronous iterator.

    See[9] and[10] for more details.

Rationale and Goals

Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the yield from syntax introduced in PEP 380.
This approach has a number of shortcomings:

-   It is easy to confuse coroutines with regular generators, since they
    share the same syntax; this is especially true for new developers.
-   Whether or not a function is a coroutine is determined by a presence
    of yield or yield from statements in its body, which can lead to
    unobvious errors when such statements appear in or disappear from
    function body during refactoring.
-   Support for asynchronous calls is limited to expressions where yield
    is allowed syntactically, limiting the usefulness of syntactic
    features, such as with and for statements.

This proposal makes coroutines a native Python language feature, and
clearly separates them from generators. This removes generator/coroutine
ambiguity, and makes it possible to reliably define coroutines without
reliance on a specific library. This also enables linters and IDEs to
improve static code analysis and refactoring.

Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new async with
statement lets Python programs perform asynchronous calls when entering
and exiting a runtime context, and the new async for statement makes it
possible to perform asynchronous calls in iterators.

Specification

This proposal introduces new syntax and semantics to enhance coroutine
support in Python.

This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax
changes proposed here comes from the asyncio framework (PEP 3156) and
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).

From this point in this document we use the word native coroutine to
refer to functions declared using the new syntax. generator-based
coroutine is used where necessary to refer to coroutines that are based
on generator syntax. coroutine is used in contexts where both
definitions are applicable.

New Coroutine Declaration Syntax

The following new syntax is used to declare a native coroutine:

    async def read_data(db):
        pass

Key properties of coroutines:

-   async def functions are always coroutines, even if they do not
    contain await expressions.
-   It is a SyntaxError to have yield or yield from expressions in an
    async function.
-   Internally, two new code object flags were introduced:
    -   CO_COROUTINE is used to mark native coroutines (defined with new
        syntax).
    -   CO_ITERABLE_COROUTINE is used to make generator-based coroutines
        compatible with native coroutines (set by types.coroutine()
        function).
-   Regular generators, when called, return a generator object;
    similarly, coroutines return a coroutine object.
-   StopIteration exceptions are not propagated out of coroutines, and
    are replaced with a RuntimeError. For regular generators such
    behavior requires a future import (see PEP 479).
-   When a native coroutine is garbage collected, a RuntimeWarning is
    raised if it was never awaited on (see also Debugging Features).
-   See also Coroutine objects section.

types.coroutine()

A new function coroutine(fn) is added to the types module. It allows
interoperability between existing generator-based coroutines in asyncio
and native coroutines introduced by this PEP:

    @types.coroutine
    def process_data(db):
        data = yield from read_data(db)
        ...

The function applies CO_ITERABLE_COROUTINE flag to generator-function's
code object, making it return a coroutine object.

If fn is not a generator function, it is wrapped. If it returns a
generator, it will be wrapped in an awaitable proxy object (see below
the definition of awaitable objects).

Note, that the CO_COROUTINE flag is not applied by types.coroutine() to
make it possible to separate native coroutines defined with new syntax,
from generator-based coroutines.

Await Expression

The following new await expression is used to obtain a result of
coroutine execution:

    async def read_data(db):
        data = await db.fetch('SELECT ...')
        ...

await, similarly to yield from, suspends execution of read_data
coroutine until db.fetch awaitable completes and returns the result
data.

It uses the yield from implementation with an extra step of validating
its argument. await only accepts an awaitable, which can be one of:

-   A native coroutine object returned from a native coroutine function.

-   A generator-based coroutine object returned from a function
    decorated with types.coroutine().

-   An object with an __await__ method returning an iterator.

    Any yield from chain of calls ends with a yield. This is a
    fundamental mechanism of how Futures are implemented. Since,
    internally, coroutines are a special kind of generators, every await
    is suspended by a yield somewhere down the chain of await calls
    (please refer to PEP 3156 for a detailed explanation).

    To enable this behavior for coroutines, a new magic method called
    __await__ is added. In asyncio, for instance, to enable Future
    objects in await statements, the only change is to add
    __await__ = __iter__ line to asyncio.Future class.

    Objects with __await__ method are called Future-like objects in the
    rest of this PEP.

    It is a TypeError if __await__ returns anything but an iterator.

-   Objects defined with CPython C API with a tp_as_async.am_await
    function, returning an iterator (similar to __await__ method).

It is a SyntaxError to use await outside of an async def function (like
it is a SyntaxError to use yield outside of def function).

It is a TypeError to pass anything other than an awaitable object to an
await expression.

Updated operator precedence table

await keyword is defined as follows:

    power ::=  await ["**" u_expr]
    await ::=  ["await"] primary

where "primary" represents the most tightly bound operations of the
language. Its syntax is:

    primary ::=  atom | attributeref | subscription | slicing | call

See Python Documentation[11] and Grammar Updates section of this
proposal for details.

The key await difference from yield and yield from operators is that
await expressions do not require parentheses around them most of the
times.

Also, yield from allows any expression as its argument, including
expressions like yield from a() + b(), that would be parsed as
yield from (a() + b()), which is almost always a bug. In general, the
result of any arithmetic operation is not an awaitable object. To avoid
this kind of mistakes, it was decided to make await precedence lower
than [], (), and ., but higher than ** operators.

  ------------------------------------------------------------------
  Operator                       Description
  ------------------------------ -----------------------------------
  yield x, yield from x          Yield expression

  lambda                         Lambda expression

  if -- else                     Conditional expression

  or                             Boolean OR

  and                            Boolean AND

  not x                          Boolean NOT

  in, not in, is, is not, <, <=, Comparisons, including membership
  >, >=, !=, ==                  tests and identity tests

  |                              Bitwise OR

  ^                              Bitwise XOR

  &                              Bitwise AND

  <<, >>                         Shifts

  +, -                           Addition and subtraction

  *, @, /, //, %                 Multiplication, matrix
                                 multiplication, division, remainder

  +x, -x, ~x                     Positive, negative, bitwise NOT

  **                             Exponentiation

  await x                        Await expression

  x[index], x[index:index],      Subscription, slicing, call,
  x(arguments...), x.attribute   attribute reference

  (expressions...),              Binding or tuple display, list
  [expressions...],              display, dictionary display, set
  {key: value...},               display
  {expressions...}
  ------------------------------------------------------------------

Examples of "await" expressions

Valid syntax examples:

  Expression                     Will be parsed as
  ------------------------------ ---------------------------------
  if await fut: pass             if (await fut): pass
  if await fut + 1: pass         if (await fut) + 1: pass
  pair = await fut, 'spam'       pair = (await fut), 'spam'
  with await fut, open(): pass   with (await fut), open(): pass
  await foo()['spam'].baz()()    await ( foo()['spam'].baz()() )
  return await coro()            return ( await coro() )
  res = await coro() ** 2        res = (await coro()) ** 2
  func(a1=await coro(), a2=0)    func(a1=(await coro()), a2=0)
  await foo() + await bar()      (await foo()) + (await bar())
  -await foo()                   -(await foo())

Invalid syntax examples:

  Expression           Should be written as
  -------------------- ----------------------
  await await coro()   await (await coro())
  await -coro()        await (-coro())

Asynchronous Context Managers and "async with"

An asynchronous context manager is a context manager that is able to
suspend execution in its enter and exit methods.

To make this possible, a new protocol for asynchronous context managers
is proposed. Two new magic methods are added: __aenter__ and __aexit__.
Both must return an awaitable.

An example of an asynchronous context manager:

    class AsyncContextManager:
        async def __aenter__(self):
            await log('entering context')

        async def __aexit__(self, exc_type, exc, tb):
            await log('exiting context')

New Syntax

A new statement for asynchronous context managers is proposed:

    async with EXPR as VAR:
        BLOCK

which is semantically equivalent to:

    mgr = (EXPR)
    aexit = type(mgr).__aexit__
    aenter = type(mgr).__aenter__

    VAR = await aenter(mgr)
    try:
        BLOCK
    except:
        if not await aexit(mgr, *sys.exc_info()):
            raise
    else:
        await aexit(mgr, None, None, None)

As with regular with statements, it is possible to specify multiple
context managers in a single async with statement.

It is an error to pass a regular context manager without __aenter__ and
__aexit__ methods to async with. It is a SyntaxError to use async with
outside of an async def function.

Example

With asynchronous context managers it is easy to implement proper
database transaction managers for coroutines:

    async def commit(session, data):
        ...

        async with session.transaction():
            ...
            await session.update(data)
            ...

Code that needs locking also looks lighter:

    async with lock:
        ...

instead of:

    with (yield from lock):
        ...

Asynchronous Iterators and "async for"

An asynchronous iterable is able to call asynchronous code in its iter
implementation, and asynchronous iterator can call asynchronous code in
its next method. To support asynchronous iteration:

1.  An object must implement an __aiter__ method (or, if defined with
    CPython C API, tp_as_async.am_aiter slot) returning an asynchronous
    iterator object.
2.  An asynchronous iterator object must implement an __anext__ method
    (or, if defined with CPython C API, tp_as_async.am_anext slot)
    returning an awaitable.
3.  To stop iteration __anext__ must raise a StopAsyncIteration
    exception.

An example of asynchronous iterable:

    class AsyncIterable:
        def __aiter__(self):
            return self

        async def __anext__(self):
            data = await self.fetch_data()
            if data:
                return data
            else:
                raise StopAsyncIteration

        async def fetch_data(self):
            ...

New Syntax

A new statement for iterating through asynchronous iterators is
proposed:

    async for TARGET in ITER:
        BLOCK
    else:
        BLOCK2

which is semantically equivalent to:

    iter = (ITER)
    iter = type(iter).__aiter__(iter)
    running = True
    while running:
        try:
            TARGET = await type(iter).__anext__(iter)
        except StopAsyncIteration:
            running = False
        else:
            BLOCK
    else:
        BLOCK2

It is a TypeError to pass a regular iterable without __aiter__ method to
async for. It is a SyntaxError to use async for outside of an async def
function.

As for with regular for statement, async for has an optional else
clause.

Example 1

With asynchronous iteration protocol it is possible to asynchronously
buffer data during iteration:

    async for data in cursor:
        ...

Where cursor is an asynchronous iterator that prefetches N rows of data
from a database after every N iterations.

The following code illustrates new asynchronous iteration protocol:

    class Cursor:
        def __init__(self):
            self.buffer = collections.deque()

        async def _prefetch(self):
            ...

        def __aiter__(self):
            return self

        async def __anext__(self):
            if not self.buffer:
                self.buffer = await self._prefetch()
                if not self.buffer:
                    raise StopAsyncIteration
            return self.buffer.popleft()

then the Cursor class can be used as follows:

    async for row in Cursor():
        print(row)

which would be equivalent to the following code:

    i = Cursor().__aiter__()
    while True:
        try:
            row = await i.__anext__()
        except StopAsyncIteration:
            break
        else:
            print(row)

Example 2

The following is a utility class that transforms a regular iterable to
an asynchronous one. While this is not a very useful thing to do, the
code illustrates the relationship between regular and asynchronous
iterators.

    class AsyncIteratorWrapper:
        def __init__(self, obj):
            self._it = iter(obj)

        def __aiter__(self):
            return self

        async def __anext__(self):
            try:
                value = next(self._it)
            except StopIteration:
                raise StopAsyncIteration
            return value

    async for letter in AsyncIteratorWrapper("abc"):
        print(letter)

Why StopAsyncIteration?

Coroutines are still based on generators internally. So, before PEP 479,
there was no fundamental difference between

    def g1():
        yield from fut
        return 'spam'

and

    def g2():
        yield from fut
        raise StopIteration('spam')

And since PEP 479 is accepted and enabled by default for coroutines, the
following example will have its StopIteration wrapped into a
RuntimeError

    async def a1():
        await fut
        raise StopIteration('spam')

The only way to tell the outside code that the iteration has ended is to
raise something other than StopIteration. Therefore, a new built-in
exception class StopAsyncIteration was added.

Moreover, with semantics from PEP 479, all StopIteration exceptions
raised in coroutines are wrapped in RuntimeError.

Coroutine objects

Differences from generators

This section applies only to native coroutines with CO_COROUTINE flag,
i.e. defined with the new async def syntax.

The behavior of existing *generator-based coroutines* in asyncio remains
unchanged.

Great effort has been made to make sure that coroutines and generators
are treated as distinct concepts:

1.  Native coroutine objects do not implement __iter__ and __next__
    methods. Therefore, they cannot be iterated over or passed to
    iter(), list(), tuple() and other built-ins. They also cannot be
    used in a for..in loop.

    An attempt to use __iter__ or __next__ on a native coroutine object
    will result in a TypeError.

2.  Plain generators cannot yield from native coroutines: doing so will
    result in a TypeError.

3.  generator-based coroutines (for asyncio code must be decorated with
    @asyncio.coroutine[12]) can yield from native coroutine objects.

4.  inspect.isgenerator() and inspect.isgeneratorfunction() return False
    for native coroutine objects and native coroutine functions.

Coroutine object methods

Coroutines are based on generators internally, thus they share the
implementation. Similarly to generator objects, coroutines have throw(),
send() and close() methods. StopIteration and GeneratorExit play the
same role for coroutines (although PEP 479 is enabled by default for
coroutines). See PEP 342, PEP 380, and Python Documentation[13] for
details.

throw(), send() methods for coroutines are used to push values and raise
errors into Future-like objects.

Debugging Features

A common beginner mistake is forgetting to use yield from on coroutines:

    @asyncio.coroutine
    def useful():
        asyncio.sleep(1) # this will do nothing without 'yield from'

For debugging this kind of mistakes there is a special debug mode in
asyncio, in which @coroutine decorator wraps all functions with a
special object with a destructor logging a warning. Whenever a wrapped
generator gets garbage collected, a detailed logging message is
generated with information about where exactly the decorator function
was defined, stack trace of where it was collected, etc. Wrapper object
also provides a convenient __repr__ function with detailed information
about the generator.

The only problem is how to enable these debug capabilities. Since debug
facilities should be a no-op in production mode, @coroutine decorator
makes the decision of whether to wrap or not to wrap based on an OS
environment variable PYTHONASYNCIODEBUG. This way it is possible to run
asyncio programs with asyncio's own functions instrumented.
EventLoop.set_debug, a different debug facility, has no impact on
@coroutine decorator's behavior.

With this proposal, coroutines is a native, distinct from generators,
concept. In addition to a RuntimeWarning being raised on coroutines that
were never awaited, it is proposed to add two new functions to the sys
module: set_coroutine_wrapper and get_coroutine_wrapper. This is to
enable advanced debugging facilities in asyncio and other frameworks
(such as displaying where exactly coroutine was created, and a more
detailed stack trace of where it was garbage collected).

New Standard Library Functions

-   types.coroutine(gen). See types.coroutine() section for details.
-   inspect.iscoroutine(obj) returns True if obj is a native coroutine
    object.
-   inspect.iscoroutinefunction(obj) returns True if obj is a native
    coroutine function.
-   inspect.isawaitable(obj) returns True if obj is an awaitable.
-   inspect.getcoroutinestate(coro) returns the current state of a
    native coroutine object (mirrors inspect.getfgeneratorstate(gen)).
-   inspect.getcoroutinelocals(coro) returns the mapping of a native
    coroutine object's local variables to their values (mirrors
    inspect.getgeneratorlocals(gen)).
-   sys.set_coroutine_wrapper(wrapper) allows to intercept creation of
    native coroutine objects. wrapper must be either a callable that
    accepts one argument (a coroutine object), or None. None resets the
    wrapper. If called twice, the new wrapper replaces the previous one.
    The function is thread-specific. See Debugging Features for more
    details.
-   sys.get_coroutine_wrapper() returns the current wrapper object.
    Returns None if no wrapper was set. The function is thread-specific.
    See Debugging Features for more details.

New Abstract Base Classes

In order to allow better integration with existing frameworks (such as
Tornado, see[14]) and compilers (such as Cython, see[15]), two new
Abstract Base Classes (ABC) are added:

-   collections.abc.Awaitable ABC for Future-like classes, that
    implement __await__ method.

-   collections.abc.Coroutine ABC for coroutine objects, that implement
    send(value), throw(type, exc, tb), close() and __await__() methods.

    Note that generator-based coroutines with CO_ITERABLE_COROUTINE flag
    do not implement __await__ method, and therefore are not instances
    of collections.abc.Coroutine and collections.abc.Awaitable ABCs:

        @types.coroutine
        def gencoro():
            yield

        assert not isinstance(gencoro(), collections.abc.Coroutine)

        # however:
        assert inspect.isawaitable(gencoro())

To allow easy testing if objects support asynchronous iteration, two
more ABCs are added:

-   collections.abc.AsyncIterable -- tests for __aiter__ method.
-   collections.abc.AsyncIterator -- tests for __aiter__ and __anext__
    methods.

Glossary

Native coroutine function

    A coroutine function is declared with async def. It uses await and
    return value; see New Coroutine Declaration Syntax for details.

Native coroutine

    Returned from a native coroutine function. See Await Expression for
    details.

Generator-based coroutine function

    Coroutines based on generator syntax. Most common example are
    functions decorated with @asyncio.coroutine.

Generator-based coroutine

    Returned from a generator-based coroutine function.

Coroutine

    Either native coroutine or generator-based coroutine.

Coroutine object

    Either native coroutine object or generator-based coroutine object.

Future-like object

    An object with an __await__ method, or a C object with
    tp_as_async->am_await function, returning an iterator. Can be
    consumed by an await expression in a coroutine. A coroutine waiting
    for a Future-like object is suspended until the Future-like object's
    __await__ completes, and returns the result. See Await Expression
    for details.

Awaitable

    A Future-like object or a coroutine object. See Await Expression for
    details.

Asynchronous context manager

    An asynchronous context manager has __aenter__ and __aexit__ methods
    and can be used with async with. See Asynchronous Context Managers
    and "async with" for details.

Asynchronous iterable

    An object with an __aiter__ method, which must return an
    asynchronous iterator object. Can be used with async for. See
    Asynchronous Iterators and "async for" for details.

Asynchronous iterator

    An asynchronous iterator has an __anext__ method. See Asynchronous
    Iterators and "async for" for details.

Transition Plan

To avoid backwards compatibility issues with async and await keywords,
it was decided to modify tokenizer.c in such a way, that it:

-   recognizes async def NAME tokens combination;
-   while tokenizing async def block, it replaces 'async' NAME token
    with ASYNC, and 'await' NAME token with AWAIT;
-   while tokenizing def block, it yields 'async' and 'await' NAME
    tokens as is.

This approach allows for seamless combination of new syntax features
(all of them available only in async functions) with any existing code.

An example of having "async def" and "async" attribute in one piece of
code:

    class Spam:
        async = 42

    async def ham():
        print(getattr(Spam, 'async'))

    # The coroutine can be executed and will print '42'

Backwards Compatibility

This proposal preserves 100% backwards compatibility.

asyncio

asyncio module was adapted and tested to work with coroutines and new
statements. Backwards compatibility is 100% preserved, i.e. all existing
code will work as-is.

The required changes are mainly:

1.  Modify @asyncio.coroutine decorator to use new types.coroutine()
    function.
2.  Add __await__ = __iter__ line to asyncio.Future class.
3.  Add ensure_future() as an alias for async() function. Deprecate
    async() function.

asyncio migration strategy

Because plain generators cannot yield from native coroutine objects (see
Differences from generators section for more details), it is advised to
make sure that all generator-based coroutines are decorated with
@asyncio.coroutine before starting to use the new syntax.

async/await in CPython code base

There is no use of await names in CPython.

async is mostly used by asyncio. We are addressing this by renaming
async() function to ensure_future() (see asyncio section for details).

Another use of async keyword is in Lib/xml/dom/xmlbuilder.py, to define
an async = False attribute for DocumentLS class. There is no
documentation or tests for it, it is not used anywhere else in CPython.
It is replaced with a getter, that raises a DeprecationWarning, advising
to use async_ attribute instead. 'async' attribute is not documented and
is not used in CPython code base.

Grammar Updates

Grammar changes are fairly minimal:

    decorated: decorators (classdef | funcdef | async_funcdef)
    async_funcdef: ASYNC funcdef

    compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt
                    | funcdef | classdef | decorated | async_stmt)

    async_stmt: ASYNC (funcdef | with_stmt | for_stmt)

    power: atom_expr ['**' factor]
    atom_expr: [AWAIT] atom trailer*

Deprecation Plans

async and await names will be softly deprecated in CPython 3.5 and 3.6.
In 3.7 we will transform them to proper keywords. Making async and await
proper keywords before 3.7 might make it harder for people to port their
code to Python 3.

Design Considerations

PEP 3152

PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines
(called "cofunctions"). Some key points:

1.  A new keyword codef to declare a cofunction. Cofunction is always a
    generator, even if there is no cocall expressions inside it. Maps to
    async def in this proposal.

2.  A new keyword cocall to call a cofunction. Can only be used inside a
    cofunction. Maps to await in this proposal (with some differences,
    see below).

3.  It is not possible to call a cofunction without a cocall keyword.

4.  cocall grammatically requires parentheses after it:

        atom: cocall | <existing alternatives for atom>
        cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
        cotrailer: '[' subscriptlist ']' | '.' NAME

5.  cocall f(*args, **kwds) is semantically equivalent to
    yield from f.__cocall__(*args, **kwds).

Differences from this proposal:

1.  There is no equivalent of __cocall__ in this PEP, which is called
    and its result is passed to yield from in the cocall expression.
    await keyword expects an awaitable object, validates the type, and
    executes yield from on it. Although, __await__ method is similar to
    __cocall__, but is only used to define Future-like objects.

2.  await is defined in almost the same way as yield from in the grammar
    (it is later enforced that await can only be inside async def). It
    is possible to simply write await future, whereas cocall always
    requires parentheses.

3.  To make asyncio work with PEP 3152 it would be required to modify
    @asyncio.coroutine decorator to wrap all functions in an object with
    a __cocall__ method, or to implement __cocall__ on generators. To
    call cofunctions from existing generator-based coroutines it would
    be required to use costart(cofunc, *args, **kwargs) built-in.

4.  Since it is impossible to call a cofunction without a cocall
    keyword, it automatically prevents the common mistake of forgetting
    to use yield from on generator-based coroutines. This proposal
    addresses this problem with a different approach, see Debugging
    Features.

5.  A shortcoming of requiring a cocall keyword to call a coroutine is
    that if is decided to implement coroutine-generators --coroutines
    with yield or async yield expressions -- we wouldn't need a cocall
    keyword to call them. So we'll end up having __cocall__ and no
    __call__ for regular coroutines, and having __call__ and no
    __cocall__ for coroutine-generators.

6.  Requiring parentheses grammatically also introduces a whole lot of
    new problems.

    The following code:

        await fut
        await function_returning_future()
        await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2))

    would look like:

        cocall fut()  # or cocall costart(fut)
        cocall (function_returning_future())()
        cocall asyncio.gather(costart(coro1, arg1, arg2),
                              costart(coro2, arg1, arg2))

7.  There are no equivalents of async for and async with in PEP 3152.

Coroutine-generators

With async for keyword it is desirable to have a concept of a
coroutine-generator -- a coroutine with yield and yield from
expressions. To avoid any ambiguity with regular generators, we would
likely require to have an async keyword before yield, and
async yield from would raise a StopAsyncIteration exception.

While it is possible to implement coroutine-generators, we believe that
they are out of scope of this proposal. It is an advanced concept that
should be carefully considered and balanced, with a non-trivial changes
in the implementation of current generator objects. This is a matter for
a separate PEP.

Why "async" and "await" keywords

async/await is not a new concept in programming languages:

-   C# has it since long time ago[16];
-   proposal to add async/await in ECMAScript 7[17]; see also Traceur
    project[18];
-   Facebook's Hack/HHVM[19];
-   Google's Dart language[20];
-   Scala[21];
-   proposal to add async/await to C++[22];
-   and many other less popular languages.

This is a huge benefit, as some users already have experience with
async/await, and because it makes working with many languages in one
project easier (Python with ECMAScript 7 for instance).

Why "__aiter__" does not return an awaitable

PEP 492 was accepted in CPython 3.5.0 with __aiter__ defined as a
method, that was expected to return an awaitable resolving to an
asynchronous iterator.

In 3.5.2 (as PEP 492 was accepted on a provisional basis) the __aiter__
protocol was updated to return asynchronous iterators directly.

The motivation behind this change is to make it possible to implement
asynchronous generators in Python. See[23] and[24] for more details.

Importance of "async" keyword

While it is possible to just implement await expression and treat all
functions with at least one await as coroutines, this approach makes
APIs design, code refactoring and its long time support harder.

Let's pretend that Python only has await keyword:

    def useful():
        ...
        await log(...)
        ...

    def important():
        await useful()

If useful() function is refactored and someone removes all await
expressions from it, it would become a regular python function, and all
code that depends on it, including important() would be broken. To
mitigate this issue a decorator similar to @asyncio.coroutine has to be
introduced.

Why "async def"

For some people bare async name(): pass syntax might look more appealing
than async def name(): pass. It is certainly easier to type. But on the
other hand, it breaks the symmetry between async def, async with and
async for, where async is a modifier, stating that the statement is
asynchronous. It is also more consistent with the existing grammar.

Why not "await for" and "await with"

async is an adjective, and hence it is a better choice for a statement
qualifier keyword. await for/with would imply that something is awaiting
for a completion of a for or with statement.

Why "async def" and not "def async"

async keyword is a statement qualifier. A good analogy to it are
"static", "public", "unsafe" keywords from other languages. "async for"
is an asynchronous "for" statement, "async with" is an asynchronous
"with" statement, "async def" is an asynchronous function.

Having "async" after the main statement keyword might introduce some
confusion, like "for async item in iterator" can be read as "for each
asynchronous item in iterator".

Having async keyword before def, with and for also makes the language
grammar simpler. And "async def" better separates coroutines from
regular functions visually.

Why not a __future__ import

Transition Plan section explains how tokenizer is modified to treat
async and await as keywords only in async def blocks. Hence async def
fills the role that a module level compiler declaration like
from __future__ import async_await would otherwise fill.

Why magic methods start with "a"

New asynchronous magic methods __aiter__, __anext__, __aenter__, and
__aexit__ all start with the same prefix "a". An alternative proposal is
to use "async" prefix, so that __anext__ becomes __async_next__.
However, to align new magic methods with the existing ones, such as
__radd__ and __iadd__ it was decided to use a shorter version.

Why not reuse existing magic names

An alternative idea about new asynchronous iterators and context
managers was to reuse existing magic methods, by adding an async keyword
to their declarations:

    class CM:
        async def __enter__(self): # instead of __aenter__
            ...

This approach has the following downsides:

-   it would not be possible to create an object that works in both with
    and async with statements;
-   it would break backwards compatibility, as nothing prohibits from
    returning a Future-like objects from __enter__ and/or __exit__ in
    Python <= 3.4;
-   one of the main points of this proposal is to make native coroutines
    as simple and foolproof as possible, hence the clear separation of
    the protocols.

Why not reuse existing "for" and "with" statements

The vision behind existing generator-based coroutines and this proposal
is to make it easy for users to see where the code might be suspended.
Making existing "for" and "with" statements to recognize asynchronous
iterators and context managers will inevitably create implicit suspend
points, making it harder to reason about the code.

Comprehensions

Syntax for asynchronous comprehensions could be provided, but this
construct is outside of the scope of this PEP.

Async lambda functions

Syntax for asynchronous lambda functions could be provided, but this
construct is outside of the scope of this PEP.

Performance

Overall Impact

This proposal introduces no observable performance impact. Here is an
output of python's official set of benchmarks[25]:

    python perf.py -r -b default ../cpython/python.exe ../cpython-aw/python.exe

    [skipped]

    Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:
    Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
    x86_64 i386

    Total CPU cores: 8

    ### etree_iterparse ###
    Min: 0.365359 -> 0.349168: 1.05x faster
    Avg: 0.396924 -> 0.379735: 1.05x faster
    Significant (t=9.71)
    Stddev: 0.01225 -> 0.01277: 1.0423x larger

    The following not significant results are hidden, use -v to show them:
    django_v2, 2to3, etree_generate, etree_parse, etree_process, fastpickle,
    fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.

Tokenizer modifications

There is no observable slowdown of parsing python files with the
modified tokenizer: parsing of one 12Mb file (Lib/test/test_binop.py
repeated 1000 times) takes the same amount of time.

async/await

The following micro-benchmark was used to determine performance
difference between "async" functions and generators:

    import sys
    import time

    def binary(n):
        if n <= 0:
            return 1
        l = yield from binary(n - 1)
        r = yield from binary(n - 1)
        return l + 1 + r

    async def abinary(n):
        if n <= 0:
            return 1
        l = await abinary(n - 1)
        r = await abinary(n - 1)
        return l + 1 + r

    def timeit(func, depth, repeat):
        t0 = time.time()
        for _ in range(repeat):
            o = func(depth)
            try:
                while True:
                    o.send(None)
            except StopIteration:
                pass
        t1 = time.time()
        print('{}({}) * {}: total {:.3f}s'.format(
            func.__name__, depth, repeat, t1-t0))

The result is that there is no observable performance difference:

    binary(19) * 30: total 53.321s
    abinary(19) * 30: total 55.073s

    binary(19) * 30: total 53.361s
    abinary(19) * 30: total 51.360s

    binary(19) * 30: total 49.438s
    abinary(19) * 30: total 51.047s

Note that depth of 19 means 1,048,575 calls.

Reference Implementation

The reference implementation can be found here:[26].

List of high-level changes and new protocols

1.  New syntax for defining coroutines: async def and new await keyword.
2.  New __await__ method for Future-like objects, and new
    tp_as_async.am_await slot in PyTypeObject.
3.  New syntax for asynchronous context managers: async with. And
    associated protocol with __aenter__ and __aexit__ methods.
4.  New syntax for asynchronous iteration: async for. And associated
    protocol with __aiter__, __aexit__ and new built-in exception
    StopAsyncIteration. New tp_as_async.am_aiter and
    tp_as_async.am_anext slots in PyTypeObject.
5.  New AST nodes: AsyncFunctionDef, AsyncFor, AsyncWith, Await.
6.  New functions: sys.set_coroutine_wrapper(callback),
    sys.get_coroutine_wrapper(), types.coroutine(gen),
    inspect.iscoroutinefunction(func), inspect.iscoroutine(obj),
    inspect.isawaitable(obj), inspect.getcoroutinestate(coro), and
    inspect.getcoroutinelocals(coro).
7.  New CO_COROUTINE and CO_ITERABLE_COROUTINE bit flags for code
    objects.
8.  New ABCs: collections.abc.Awaitable, collections.abc.Coroutine,
    collections.abc.AsyncIterable, and collections.abc.AsyncIterator.
9.  C API changes: new PyCoro_Type (exposed to Python as
    types.CoroutineType) and PyCoroObject. PyCoro_CheckExact(*o) to test
    if o is a native coroutine.

While the list of changes and new things is not short, it is important
to understand, that most users will not use these features directly. It
is intended to be used in frameworks and libraries to provide users with
convenient to use and unambiguous APIs with async def, await, async for
and async with syntax.

Working example

All concepts proposed in this PEP are implemented[27] and can be tested.

    import asyncio

    async def echo_server():
        print('Serving on localhost:8000')
        await asyncio.start_server(handle_connection,
                                   'localhost', 8000)

    async def handle_connection(reader, writer):
        print('New connection...')

        while True:
            data = await reader.read(8192)

            if not data:
                break

            print('Sending {:.10}... back'.format(repr(data)))
            writer.write(data)

    loop = asyncio.get_event_loop()
    loop.run_until_complete(echo_server())
    try:
        loop.run_forever()
    finally:
        loop.close()

Acceptance

PEP 492 was accepted by Guido, Tuesday, May 5, 2015[28].

Implementation

The implementation is tracked in issue 24017[29]. It was committed on
May 11, 2015.

References

Acknowledgments

I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew
Svetlov, Łukasz Langa, Greg Ewing, Stephen J. Turnbull, Jim J. Jewett,
Brett Cannon, Alyssa Coghlan, Steven D'Aprano, Paul Moore, Nathaniel
Smith, Ethan Furman, Stefan Behnel, Paul Sokolovsky, Victor Petrovykh,
and many others for their feedback, ideas, edits, criticism, code
reviews, and discussions around this PEP.

Copyright

This document has been placed in the public domain.

[1] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions

[2] https://msdn.microsoft.com/en-us/library/hh191443.aspx

[3] http://docs.hhvm.com/manual/en/hack.async.php

[4] https://www.dartlang.org/articles/await-async/

[5] http://docs.scala-lang.org/sips/pending/async.html

[6] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf
(PDF)

[7] https://hg.python.org/cpython/rev/7a0a1a4ac639

[8] http://bugs.python.org/issue24400

[9] http://bugs.python.org/issue27243

[10] https://docs.python.org/3/reference/datamodel.html#async-iterators

[11] https://docs.python.org/3/reference/expressions.html#primaries

[12] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine

[13] https://docs.python.org/3/reference/expressions.html#generator-iterator-methods

[14] https://mail.python.org/pipermail/python-dev/2015-May/139851.html

[15] https://github.com/python/asyncio/issues/233

[16] https://msdn.microsoft.com/en-us/library/hh191443.aspx

[17] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions

[18] https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental

[19] http://docs.hhvm.com/manual/en/hack.async.php

[20] https://www.dartlang.org/articles/await-async/

[21] http://docs.scala-lang.org/sips/pending/async.html

[22] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf
(PDF)

[23] http://bugs.python.org/issue27243

[24] https://docs.python.org/3/reference/datamodel.html#async-iterators

[25] https://hg.python.org/benchmarks

[26] https://github.com/1st1/cpython/tree/await

[27] https://github.com/1st1/cpython/tree/await

[28] https://mail.python.org/pipermail/python-dev/2015-May/139844.html

[29] http://bugs.python.org/issue24017


text:
PEP: 450 Title: Adding A Statistics Module To The Standard Library
Version: $Revision$ Last-Modified: $Date$ Author: Steven D'Aprano
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 01-Aug-2013 Python-Version: 3.4 Post-History:
13-Sep-2013

Abstract

This PEP proposes the addition of a module for common statistics
functions such as mean, median, variance and standard deviation to the
Python standard library. See also http://bugs.python.org/issue18606

Rationale

The proposed statistics module is motivated by the "batteries included"
philosophy towards the Python standard library. Raymond Hettinger and
other senior developers have requested a quality statistics library that
falls somewhere in between high-end statistics libraries and ad hoc
code.[1] Statistical functions such as mean, standard deviation and
others are obvious and useful batteries, familiar to any Secondary
School student. Even cheap scientific calculators typically include
multiple statistical functions such as:

-   mean
-   population and sample variance
-   population and sample standard deviation
-   linear regression
-   correlation coefficient

Graphing calculators aimed at Secondary School students typically
include all of the above, plus some or all of:

-   median
-   mode
-   functions for calculating the probability of random variables from
    the normal, t, chi-squared, and F distributions
-   inference on the mean

and others[2]. Likewise spreadsheet applications such as Microsoft
Excel, LibreOffice and Gnumeric include rich collections of statistical
functions[3].

In contrast, Python currently has no standard way to calculate even the
simplest and most obvious statistical functions such as mean. For those
who need statistical functions in Python, there are two obvious
solutions:

-   install numpy and/or scipy[4];
-   or use a Do It Yourself solution.

Numpy is perhaps the most full-featured solution, but it has a few
disadvantages:

-   It may be overkill for many purposes. The documentation for numpy
    even warns

      "It can be hard to know what functions are available in numpy.
      This is not a complete list, but it does cover most of them."[5]

    and then goes on to list over 270 functions, only a small number of
    which are related to statistics.

-   Numpy is aimed at those doing heavy numerical work, and may be
    intimidating to those who don't have a background in computational
    mathematics and computer science. For example, numpy.mean takes four
    arguments:

        mean(a, axis=None, dtype=None, out=None)

    although fortunately for the beginner or casual numpy user, three
    are optional and numpy.mean does the right thing in simple cases:

        >>>  numpy.mean([1, 2, 3, 4])
        2.5

-   For many people, installing numpy may be difficult or impossible.
    For example, people in corporate environments may have to go through
    a difficult, time-consuming process before being permitted to
    install third-party software. For the casual Python user, having to
    learn about installing third-party packages in order to average a
    list of numbers is unfortunate.

This leads to option number 2, DIY statistics functions. At first
glance, this appears to be an attractive option, due to the apparent
simplicity of common statistical functions. For example:

    def mean(data):
        return sum(data)/len(data)

    def variance(data):
        # Use the Computational Formula for Variance.
        n = len(data)
        ss = sum(x**2 for x in data) - (sum(data)**2)/n
        return ss/(n-1)

    def standard_deviation(data):
        return math.sqrt(variance(data))

The above appears to be correct with a casual test:

    >>> data = [1, 2, 4, 5, 8]
    >>> variance(data)
    7.5

But adding a constant to every data point should not change the
variance:

    >>> data = [x+1e12 for x in data]
    >>> variance(data)
    0.0

And variance should never be negative:

    >>> variance(data*100)
    -1239429440.1282566

By contrast, the proposed reference implementation gets the exactly
correct answer 7.5 for the first two examples, and a reasonably close
answer for the third: 6.012. numpy does no better[6].

Even simple statistical calculations contain traps for the unwary,
starting with the Computational Formula itself. Despite the name, it is
numerically unstable and can be extremely inaccurate, as can be seen
above. It is completely unsuitable for computation by computer[7]. This
problem plagues users of many programming language, not just Python[8],
as coders reinvent the same numerically inaccurate code over and over
again[9], or advise others to do so[10].

It isn't just the variance and standard deviation. Even the mean is not
quite as straightforward as it might appear. The above implementation
seems too simple to have problems, but it does:

-   The built-in sum can lose accuracy when dealing with floats of
    wildly differing magnitude. Consequently, the above naive mean fails
    this "torture test":

        assert mean([1e30, 1, 3, -1e30]) == 1

    returning 0 instead of 1, a purely computational error of 100%.

-   Using math.fsum inside mean will make it more accurate with float
    data, but it also has the side-effect of converting any arguments to
    float even when unnecessary. E.g. we should expect the mean of a
    list of Fractions to be a Fraction, not a float.

While the above mean implementation does not fail quite as
catastrophically as the naive variance does, a standard library function
can do much better than the DIY versions.

The example above involves an especially bad set of data, but even for
more realistic data sets accuracy is important. The first step in
interpreting variation in data (including dealing with ill-conditioned
data) is often to standardize it to a series with variance 1 (and often
mean 0). This standardization requires accurate computation of the mean
and variance of the raw series. Naive computation of mean and variance
can lose precision very quickly. Because precision bounds accuracy, it
is important to use the most precise algorithms for computing mean and
variance that are practical, or the results of standardization are
themselves useless.

Comparison To Other Languages/Packages

The proposed statistics library is not intended to be a competitor to
such third-party libraries as numpy/scipy, or of proprietary
full-featured statistics packages aimed at professional statisticians
such as Minitab, SAS and Matlab. It is aimed at the level of graphing
and scientific calculators.

Most programming languages have little or no built-in support for
statistics functions. Some exceptions:

R

R (and its proprietary cousin, S) is a programming language designed for
statistics work. It is extremely popular with statisticians and is
extremely feature-rich[11].

C#

The C# LINQ package includes extension methods to calculate the average
of enumerables[12].

Ruby

Ruby does not ship with a standard statistics module, despite some
apparent demand[13]. Statsample appears to be a feature-rich third-party
library, aiming to compete with R[14].

PHP

PHP has an extremely feature-rich (although mostly undocumented) set of
advanced statistical functions[15].

Delphi

Delphi includes standard statistical functions including Mean, Sum,
Variance, TotalVariance, MomentSkewKurtosis in its Math library[16].

GNU Scientific Library

The GNU Scientific Library includes standard statistical functions,
percentiles, median and others[17]. One innovation I have borrowed from
the GSL is to allow the caller to optionally specify the pre-calculated
mean of the sample (or an a priori known population mean) when
calculating the variance and standard deviation[18].

Design Decisions Of The Module

My intention is to start small and grow the library as needed, rather
than try to include everything from the start. Consequently, the current
reference implementation includes only a small number of functions:
mean, variance, standard deviation, median, mode. (See the reference
implementation for a full list.)

I have aimed for the following design features:

-   Correctness over speed. It is easier to speed up a correct but slow
    function than to correct a fast but buggy one.
-   Concentrate on data in sequences, allowing two-passes over the data,
    rather than potentially compromise on accuracy for the sake of a
    one-pass algorithm. Functions expect data will be passed as a list
    or other sequence; if given an iterator, they may internally convert
    to a list.
-   Functions should, as much as possible, honour any type of numeric
    data. E.g. the mean of a list of Decimals should be a Decimal, not a
    float. When this is not possible, treat float as the "lowest common
    data type".
-   Although functions support data sets of floats, Decimals or
    Fractions, there is no guarantee that mixed data sets will be
    supported. (But on the other hand, they aren't explicitly rejected
    either.)
-   Plenty of documentation, aimed at readers who understand the basic
    concepts but may not know (for example) which variance they should
    use (population or sample?). Mathematicians and statisticians have a
    terrible habit of being inconsistent with both notation and
    terminology[19], and having spent many hours making sense of the
    contradictory/confusing definitions in use, it is only fair that I
    do my best to clarify rather than obfuscate the topic.
-   But avoid going into tedious[20] mathematical detail.

API

The initial version of the library will provide univariate (single
variable) statistics functions. The general API will be based on a
functional model function(data, ...) -> result, where data is a
mandatory iterable of (usually) numeric data.

The author expects that lists will be the most common data type used,
but any iterable type should be acceptable. Where necessary, functions
may convert to lists internally. Where possible, functions are expected
to conserve the type of the data values, for example, the mean of a list
of Decimals should be a Decimal rather than float.

Calculating mean, median and mode

The mean, median* and mode functions take a single mandatory argument
and return the appropriate statistic, e.g.:

    >>> mean([1, 2, 3])
    2.0

Functions provided are:

-

    mean(data)

        arithmetic mean of data.

-

    median(data)

        median (middle value) of data, taking the average of the two
        middle values when there are an even number of values.

-

    median_high(data)

        high median of data, taking the larger of the two middle values
        when the number of items is even.

-

    median_low(data)

        low median of data, taking the smaller of the two middle values
        when the number of items is even.

-

    median_grouped(data, interval=1)

        50th percentile of grouped data, using interpolation.

-

    mode(data)

        most common data point.

mode is the sole exception to the rule that the data argument must be
numeric. It will also accept an iterable of nominal data, such as
strings.

Calculating variance and standard deviation

In order to be similar to scientific calculators, the statistics module
will include separate functions for population and sample variance and
standard deviation. All four functions have similar signatures, with a
single mandatory argument, an iterable of numeric data, e.g.:

    >>> variance([1, 2, 2, 2, 3])
    0.5

All four functions also accept a second, optional, argument, the mean of
the data. This is modelled on a similar API provided by the GNU
Scientific Library[21]. There are three use-cases for using this
argument, in no particular order:

1)  The value of the mean is known a priori.
2)  You have already calculated the mean, and wish to avoid calculating
    it again.
3)  You wish to (ab)use the variance functions to calculate the second
    moment about some given point other than the mean.

In each case, it is the caller's responsibility to ensure that given
argument is meaningful.

Functions provided are:

-

    variance(data, xbar=None)

        sample variance of data, optionally using xbar as the sample
        mean.

-

    stdev(data, xbar=None)

        sample standard deviation of data, optionally using xbar as the
        sample mean.

-

    pvariance(data, mu=None)

        population variance of data, optionally using mu as the
        population mean.

-

    pstdev(data, mu=None)

        population standard deviation of data, optionally using mu as
        the population mean.

Other functions

There is one other public function:

-

    sum(data, start=0)

        high-precision sum of numeric data.

Specification

As the proposed reference implementation is in pure Python, other Python
implementations can easily make use of the module unchanged, or adapt it
as they see fit.

What Should Be The Name Of The Module?

This will be a top-level module statistics.

There was some interest in turning math into a package, and making this
a sub-module of math, but the general consensus eventually agreed on a
top-level module. Other potential but rejected names included stats (too
much risk of confusion with existing stat module), and statslib
(described as "too C-like").

Discussion And Resolved Issues

This proposal has been previously discussed here[22].

A number of design issues were resolved during the discussion on
Python-Ideas and the initial code review. There was a lot of concern
about the addition of yet another sum function to the standard library,
see the FAQs below for more details. In addition, the initial
implementation of sum suffered from some rounding issues and other
design problems when dealing with Decimals. Oscar Benjamin's assistance
in resolving this was invaluable.

Another issue was the handling of data in the form of iterators. The
first implementation of variance silently swapped between a one- and
two-pass algorithm, depending on whether the data was in the form of an
iterator or sequence. This proved to be a design mistake, as the
calculated variance could differ slightly depending on the algorithm
used, and variance etc. were changed to internally generate a list and
always use the more accurate two-pass implementation.

One controversial design involved the functions to calculate median,
which were implemented as attributes on the median callable, e.g.
median, median.low, median.high etc. Although there is at least one
existing use of this style in the standard library, in unittest.mock,
the code reviewers felt that this was too unusual for the standard
library. Consequently, the design has been changed to a more traditional
design of separate functions with a pseudo-namespace naming convention,
median_low, median_high, etc.

Another issue that was of concern to code reviewers was the existence of
a function calculating the sample mode of continuous data, with some
people questioning the choice of algorithm, and whether it was a
sufficiently common need to be included. So it was dropped from the API,
and mode now implements only the basic schoolbook algorithm based on
counting unique values.

Another significant point of discussion was calculating statistics of
timedelta objects. Although the statistics module will not directly
support timedelta objects, it is possible to support this use-case by
converting them to numbers first using the timedelta.total_seconds
method.

Frequently Asked Questions

Shouldn't this module spend time on PyPI before being considered for the standard library?

Older versions of this module have been available on PyPI[23] since
2010. Being much simpler than numpy, it does not require many years of
external development.

Does the standard library really need yet another version of sum?

This proved to be the most controversial part of the reference
implementation. In one sense, clearly three sums is two too many. But in
another sense, yes. The reasons why the two existing versions are
unsuitable are described here[24] but the short summary is:

-   the built-in sum can lose precision with floats;
-   the built-in sum accepts any non-numeric data type that supports
    the + operator, apart from strings and bytes;
-   math.fsum is high-precision, but coerces all arguments to float.

There was some interest in "fixing" one or the other of the existing
sums. If this occurs before 3.4 feature-freeze, the decision to keep
statistics.sum can be re-considered.

Will this module be backported to older versions of Python?

The module currently targets 3.3, and I will make it available on PyPI
for 3.3 for the foreseeable future. Backporting to older versions of the
3.x series is likely (but not yet decided). Backporting to 2.7 is less
likely but not ruled out.

Is this supposed to replace numpy?

No. While it is likely to grow over the years (see open issues below) it
is not aimed to replace, or even compete directly with, numpy. Numpy is
a full-featured numeric library aimed at professionals, the nuclear
reactor of numeric libraries in the Python ecosystem. This is just a
battery, as in "batteries included", and is aimed at an intermediate
level somewhere between "use numpy" and "roll your own version".

Future Work

-   At this stage, I am unsure of the best API for multivariate
    statistical functions such as linear regression, correlation
    coefficient, and covariance. Possible APIs include:

    -   Separate arguments for x and y data:

            function([x0, x1, ...], [y0, y1, ...])

    -   A single argument for (x, y) data:

            function([(x0, y0), (x1, y1), ...])

        This API is preferred by GvR[25].

    -   Selecting arbitrary columns from a 2D array:

            function([[a0, x0, y0, z0], [a1, x1, y1, z1], ...], x=1, y=2)

    -   Some combination of the above.

    In the absence of a consensus of preferred API for multivariate
    stats, I will defer including such multivariate functions until
    Python 3.5.

-   Likewise, functions for calculating probability of random variables
    and inference testing (e.g. Student's t-test) will be deferred until
    3.5.

-   There is considerable interest in including one-pass functions that
    can calculate multiple statistics from data in iterator form,
    without having to convert to a list. The experimental stats package
    on PyPI includes co-routine versions of statistics functions.
    Including these will be deferred to 3.5.

References

Copyright

This document has been placed in the public domain.

[1] https://mail.python.org/pipermail/python-dev/2010-October/104721.html

[2] http://support.casio.com/pdf/004/CP330PLUSver310_Soft_E.pdf

[3] Gnumeric:: https://projects.gnome.org/gnumeric/functions.shtml

LibreOffice:
https://help.libreoffice.org/Calc/Statistical_Functions_Part_One
https://help.libreoffice.org/Calc/Statistical_Functions_Part_Two
https://help.libreoffice.org/Calc/Statistical_Functions_Part_Three
https://help.libreoffice.org/Calc/Statistical_Functions_Part_Four
https://help.libreoffice.org/Calc/Statistical_Functions_Part_Five

[4] Scipy: http://scipy-central.org/ Numpy: http://www.numpy.org/

[5] http://wiki.scipy.org/Numpy_Functions_by_Category

[6] Tested with numpy 1.6.1 and Python 2.7.

[7] http://www.johndcook.com/blog/2008/09/26/comparing-three-methods-of-computing-standard-deviation/

[8] http://rosettacode.org/wiki/Standard_deviation

[9] https://bitbucket.org/larsyencken/simplestats/src/c42e048a6625/src/basic.py

[10] http://stackoverflow.com/questions/2341340/calculate-mean-and-variance-with-one-iteration

[11] http://www.r-project.org/

[12] http://msdn.microsoft.com/en-us/library/system.linq.enumerable.average.aspx

[13] https://www.bcg.wisc.edu/webteam/support/ruby/standard_deviation

[14] http://ruby-statsample.rubyforge.org/

[15] http://www.php.net/manual/en/ref.stats.php

[16] http://www.ayton.id.au/gary/it/Delphi/D_maths.htm#Delphi%20Statistical%20functions.

[17] http://www.gnu.org/software/gsl/manual/html_node/Statistics.html

[18] http://www.gnu.org/software/gsl/manual/html_node/Mean-and-standard-deviation-and-variance.html

[19] http://mathworld.wolfram.com/Skewness.html

[20] At least, tedious to those who don't like this sort of thing.

[21] http://www.gnu.org/software/gsl/manual/html_node/Mean-and-standard-deviation-and-variance.html

[22] https://mail.python.org/pipermail/python-ideas/2011-September/011524.html

[23] https://pypi.python.org/pypi/stats/

[24] https://mail.python.org/pipermail/python-ideas/2013-August/022630.html

[25] https://mail.python.org/pipermail/python-dev/2013-September/128429.html

license: Public Domain


text:
PEP: 741 Title: Python Configuration C API Author: Victor Stinner
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403
Status: Final Type: Standards Track Created: 18-Jan-2024 Python-Version:
3.14 Post-History: 19-Jan-2024, 08-Feb-2024, Resolution:
https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/88

Abstract

Add a C API to configure the Python initialization without relying on C
structures and the ability to make ABI-compatible changes in the future.

Complete PEP 587 API by adding PyInitConfig_AddModule() which can be
used to add a built-in extension module; feature previously referred to
as the "inittab".

Add PyConfig_Get() and PyConfig_Set() functions to get and set the
current runtime configuration.

PEP 587 "Python Initialization Configuration" unified all the ways to
configure the Python initialization. This PEP unifies also the
configuration of the Python preinitialization and the Python
initialization in a single API. Moreover, this PEP only provides a
single choice to embed Python, instead of having two "Python" and
"Isolated" choices (PEP 587), to simplify the API further.

The lower level PEP 587 PyConfig API remains available for use cases
with an intentionally higher level of coupling to CPython implementation
details (such as emulating the full functionality of CPython's CLI,
including its configuration mechanisms).

Rationale

Get the runtime configuration

PEP 587 has no API to get the current runtime configuration, only to
configure the Python initialization.

For example, the global configuration variable Py_UnbufferedStdioFlag
was deprecated in Python 3.12 and using PyConfig.buffered_stdio is
recommended instead. It only works to configure Python, there is no
public API to get PyConfig.buffered_stdio.

Users of the limited C API are asking for a public API to get the
current runtime configuration.

Cython needs to get the optimization_level configuration option: issue.

When global configuration variables were deprecated in 2022, Marc-André
Lemburg requested a C API to access these configuration variables at
runtime (not only during Python initialization).

Security fix

To fix CVE-2020-10735, a denial-of-service when converting a very large
string to an integer (in base 10), it was discussed to add a new
PyConfig member to stable branches which affects the ABI.

Gregory P. Smith proposed a different API using text based configuration
file to not be limited by PyConfig members: FR: Allow private runtime
config to enable extending without breaking the PyConfig ABI (August
2022).

In the end, it was decided to not add a new PyConfig member to stable
branches, but only add a new PyConfig.int_max_str_digits member to the
development branch (which became Python 3.12). A dedicated private
global variable (unrelated to PyConfig) is used in stable branches.

Redundancy between PyPreConfig and PyConfig

The Python preinitialization uses the PyPreConfig structure and the
Python initialization uses the PyConfig structure. Both structures have
four duplicated members: dev_mode, parse_argv, isolated and
use_environment.

The redundancy is caused by the fact that the two structures are
separated, whereas some PyConfig members are needed by the
preinitialization.

Embedding Python

Applications embedding Python

Examples:

-   Blender 3D graphics.
-   fontforge font editor.
-   Gimp.
-   LibreOffice.
-   OBS Studio.
-   Tiled.
-   vim text editor.

On Linux, FreeBSD and macOS, applications are usually either statically
linked to a libpython, or load dynamically a libpython . The libpython
shared library is versioned, example: libpython3.12.so for Python 3.12
on Linux.

The vim project can target the stable ABI. Usually, the "system Python"
version is used. It's not currently possible to select which Python
version to use. Users would like the ability to select a newer Python on
demand.

On Linux, another approach to deploy an application embedding Python,
such as GIMP, is to include Python in Flatpack, AppImage or Snap
"container". In this case, the application brings its own copy of Python
version with the container.

Libraries embedding Python

Examples:

-   Apache mod_wsgi (source).
-   nimpy: Nim - Python bridge.
-   PyO3: Rust bindings for the Python interpreter.

Utilities creating standalone applications

-   py2app for macOS.
-   py2exe for Windows.
-   pyinstaller.
-   PyOxidizer: it uses the PEP 587 PyConfig API.

These utilities create standalone applications, they are not linked to
libpython.

Set the runtime configuration

Marc-André Lemburg requested a C API to set the value of some
configuration options at runtime:

-   optimization_level
-   verbose
-   parser_debug
-   inspect
-   write_bytecode

Previously, it was possible to set directly global configuration
variables:

-   Py_OptimizeFlag
-   Py_VerboseFlag
-   Py_DebugFlag
-   Py_InspectFlag
-   Py_DontWriteBytecodeFlag

But these configuration flags were deprecated in Python 3.12 and are
scheduled for removal in Python 3.14.

Specification

Add C API functions and structure to configure the Python
initialization:

-   Create config:
    -   PyInitConfig opaque structure.
    -   PyInitConfig_Create().
    -   PyInitConfig_Free(config).
-   Get options:
    -   PyInitConfig_HasOption(config, name).
    -   PyInitConfig_GetInt(config, name, &value).
    -   PyInitConfig_GetStr(config, name, &value).
    -   PyInitConfig_GetStrList(config, name, &length, &items).
    -   PyInitConfig_FreeStrList().
-   Set options:
    -   PyInitConfig_SetInt(config, name, value).
    -   PyInitConfig_SetStr(config, name, value).
    -   PyInitConfig_SetStrList(config, name, length, items).
    -   PyInitConfig_AddModule(config, name, initfunc)
-   Initialize:
    -   Py_InitializeFromInitConfig(config).
-   Error handling:
    -   PyInitConfig_GetError(config, &err_msg).
    -   PyInitConfig_GetExitcode(config, &exitcode).

Add C API functions to get and set the current runtime configuration:

-   PyConfig_Get(name).
-   PyConfig_GetInt(name, &value).
-   PyConfig_Set(name).
-   PyConfig_Names().

The C API uses null-terminated UTF-8 encoded strings to refer to a
configuration option name.

These C API functions are excluded from the limited C API.

PyInitConfig structure

The PyInitConfig structure is implemented by combining the three
structures of the PyConfig API and has an inittab member as well:

-   PyPreConfig preconfig
-   PyConfig config
-   PyStatus status
-   struct _inittab *inittab for PyInitConfig_AddModule()

The PyStatus status is no longer separated, but part of the unified
PyInitConfig structure, which makes the API easier to use.

Configuration Options

Configuration options are named after PyPreConfig and PyConfig structure
members. See the PyPreConfig documentation and the PyConfig
documentation.

Deprecating and removing configuration options is out of the scope of
the PEP and should be discussed on a case by case basis.

Public configuration options

Following options can be get by PyConfig_Get() and set and
PyConfig_Set().

  ------------------------------------------------------------------------------
  Option                Type             Comment
  --------------------- ---------------- ---------------------------------------
  argv                  list[str]        API: sys.argv.

  base_exec_prefix      str              API: sys.base_exec_prefix.

  base_executable       str              API: sys._base_executable.

  base_prefix           str              API: sys.base_prefix.

  bytes_warning         int              API: sys.flags.bytes_warning.

  exec_prefix           str              API: sys.exec_prefix.

  executable            str              API: sys.executable.

  inspect               bool             API: sys.flags.inspect (int).

  int_max_str_digits    int              API: sys.flags.int_max_str_digits,
                                         sys.get_int_max_str_digits() and
                                         sys.set_int_max_str_digits().

  interactive           bool             API: sys.flags.interactive.

  module_search_paths   list[str]        API: sys.path.

  optimization_level    int              API: sys.flags.optimize.

  parser_debug          bool             API: sys.flags.debug (int).

  platlibdir            str              API: sys.platlibdir.

  prefix                str              API: sys.base_prefix.

  pycache_prefix        str              API: sys.pycache_prefix.

  quiet                 bool             API: sys.flags.quiet (int).

  stdlib_dir            str              API: sys._stdlib_dir.

  use_environment       bool             API: sys.flags.ignore_environment
                                         (int).

  verbose               int              API: sys.flags.verbose.

  warnoptions           list[str]        API: sys.warnoptions.

  write_bytecode        bool             API: sys.flags.dont_write_bytecode
                                         (int) and sys.dont_write_bytecode
                                         (bool).

  xoptions              dict[str, str]   API: sys._xoptions.
  ------------------------------------------------------------------------------

Some option names are different than sys attributes, such as
optimization_level option and sys.flags.optimize attribute.
PyConfig_Set() sets the corresponding sys attribute.

The xoptions is a list of strings in PyInitConfig where each string has
the format key (value is True implicitly) or key=value. In the current
runtime configuration, it becomes a dictionary (key: str →
value: str | True).

Read-only configuration options

Following options can be get by PyConfig_Get(), but cannot be set by
PyConfig_Set().

  ------------------------------------------------------------------------------------
  Option                       Type            Comment
  ---------------------------- --------------- ---------------------------------------
  allocator                    int

  buffered_stdio               bool

  check_hash_pycs_mode         str

  code_debug_ranges            bool

  coerce_c_locale              bool

  coerce_c_locale_warn         bool

  configure_c_stdio            bool

  configure_locale             bool

  cpu_count                    int             API: os.cpu_count() (int | None).

  dev_mode                     bool            API: sys.flags.dev_mode.

  dump_refs                    bool

  dump_refs_file               str

  faulthandler                 bool            API: faulthandler.is_enabled().

  filesystem_encoding          str             API: sys.getfilesystemencoding().

  filesystem_errors            str             API: sys.getfilesystemencodeerrors().

  hash_seed                    int

  home                         str

  import_time                  bool

  install_signal_handlers      bool

  isolated                     bool            API: sys.flags.isolated (int).

  legacy_windows_fs_encoding   bool            Windows only.

  legacy_windows_stdio         bool            Windows only.

  malloc_stats                 bool

  orig_argv                    list[str]       API: sys.orig_argv.

  parse_argv                   bool

  pathconfig_warnings          bool

  perf_profiling               bool            API: sys.is_stack_trampoline_active().

  program_name                 str

  run_command                  str

  run_filename                 str

  run_module                   str

  run_presite                  str             need a debug build.

  safe_path                    bool

  show_ref_count               bool

  site_import                  bool            API: sys.flags.no_site (int).

  skip_source_first_line       bool

  stdio_encoding               str             API: sys.stdin.encoding,
                                               sys.stdout.encoding and
                                               sys.stderr.encoding.

  stdio_errors                 str             API: sys.stdin.errors,
                                               sys.stdout.errors and
                                               sys.stderr.errors.

  tracemalloc                  int             API: tracemalloc.is_tracing() (bool).

  use_frozen_modules           bool

  use_hash_seed                bool

  user_site_directory          bool            API: sys.flags.no_user_site (int).

  utf8_mode                    bool

  warn_default_encoding        bool

  _pystats                     bool            API: sys._stats_on(), sys._stats_off().
                                               Need a Py_STATS build.
  ------------------------------------------------------------------------------------

Create Config

PyInitConfig structure:

    Opaque structure to configure the Python preinitialization and the
    Python initialization.

PyInitConfig* PyInitConfig_Create(void):

    Create a new initialization configuration using default values of
    the Isolated Configuration.

    It must be freed with PyInitConfig_Free().

    Return NULL on memory allocation failure.

void PyInitConfig_Free(PyInitConfig *config):

    Free memory of an initialization configuration.

Get Options

The configuration option name parameter must be a non-NULL
null-terminated UTF-8 encoded string.

int PyInitConfig_HasOption(PyInitConfig *config, const char *name):

    Test if the configuration has an option called name.

    Return 1 if the option exists, or return 0 otherwise.

int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value):

    Get an integer configuration option.

    -   Set *value, and return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_GetStr(PyInitConfig *config, const char *name, char **value):

    Get a string configuration option as a null-terminated UTF-8 encoded
    string.

    -   Set *value, and return 0 on success.
    -   Set an error in config and return -1 on error.

    On success, the string must be released with free(value).

int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items):

    Get a string list configuration option as an array of
    null-terminated UTF-8 encoded strings.

    -   Set *length and *value, and return 0 on success.
    -   Set an error in config and return -1 on error.

    On success, the string list must be released with
    PyInitConfig_FreeStrList(length, items).

void PyInitConfig_FreeStrList(size_t length, char **items):

    Free memory of a string list created by PyInitConfig_GetStrList().

Set Options

The configuration option name parameter must be a non-NULL
null-terminated UTF-8 encoded string.

Some configuration options have side effects on other options. This
logic is only implemented when Py_InitializeFromInitConfig() is called,
not by the "Set" functions below. For example, setting dev_mode to 1
does not set faulthandler to 1.

int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value):

    Set an integer configuration option.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_SetStr(PyInitConfig *config, const char *name, const char *value):

    Set a string configuration option from a null-terminated UTF-8
    encoded string. The string is copied.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_SetStrList(PyInitConfig *config, const char *name, size_t length, char * const *items):

    Set a string list configuration option from an array of
    null-terminated UTF-8 encoded strings. The string list is copied.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_AddModule(PyInitConfig *config, const char *name, PyObject* (*initfunc)(void)):

    Add a built-in extension module to the table of built-in modules.

    The new module can be imported by the name name, and uses the
    function initfunc as the initialization function called on the first
    attempted import.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

    If Python is initialized multiple times, PyInitConfig_AddModule()
    must be called at each Python initialization.

    Similar to the PyImport_AppendInittab() function.

Initialize Python

int Py_InitializeFromInitConfig(PyInitConfig *config):

    Initialize Python from the initialization configuration.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.
    -   Set an exit code in config and return -1 if Python wants to
        exit.

    See PyInitConfig_GetExitcode() for the exitcode case.

Error Handling

int PyInitConfig_GetError(PyInitConfig* config, const char **err_msg):

    Get the config error message.

    -   Set *err_msg and return 1 if an error is set.
    -   Set *err_msg to NULL and return 0 otherwise.

    An error message is an UTF-8 encoded string.

    If config has an exit code, format the exit code as an error
    message.

    The error message remains valid until another PyInitConfig function
    is called with config. The caller doesn't have to free the error
    message.

int PyInitConfig_GetExitcode(PyInitConfig* config, int *exitcode):

    Get the config exit code.

    -   Set *exitcode and return 1 if Python wants to exit.
    -   Return 0 if config has no exit code set.

    Only the Py_InitializeFromInitConfig() function can set an exit code
    if the parse_argv option is non-zero.

    An exit code can be set when parsing the command line failed (exit
    code 2) or when a command line option asks to display the command
    line help (exit code 0).

Get and Set the Runtime Configuration

The configuration option name parameter must be a non-NULL
null-terminated UTF-8 encoded string.

PyObject* PyConfig_Get(const char *name):

    Get the current runtime value of a configuration option as a Python
    object.

    -   Return a new reference on success.
    -   Set an exception and return NULL on error.

    The object type depends on the option: see Configuration Options
    tables.

    Other options are get from internal PyPreConfig and PyConfig
    structures.

    The caller must hold the GIL. The function cannot be called before
    Python initialization nor after Python finalization.

int PyConfig_GetInt(const char *name, int *value):

    Similar to PyConfig_Get(), but get the value as an integer.

    -   Set *value and return 0 success.
    -   Set an exception and return -1 on error.

PyObject* PyConfig_Names(void):

    Get all configuration option names as a frozenset.

    Set an exception and return NULL on error.

    The caller must hold the GIL.

PyObject* PyConfig_Set(const char *name, PyObject *value):

    Set the current runtime value of a configuration option.

    -   Raise a ValueError if there is no option name.
    -   Raise a ValueError if value is an invalid value.
    -   Raise a ValueError if the option is read-only: cannot be set.
    -   Raise a TypeError if value has not the proper type.

    Read-only configuration options cannot be set.

    The caller must hold the GIL. The function cannot be called before
    Python initialization nor after Python finalization.

Stability

The behavior of options, the default option values, and the Python
behavior can change at each Python version: they are not "stable".

Moreover, configuration options can be added, deprecated and removed
following the usual PEP 387 deprecation process.

Interaction with the PyPreConfig and PyConfig APIs

The lower level PEP 587 PyPreConfig and PyConfig APIs remain available
and fully supported. As noted in the Abstract, they remain the preferred
approach for embedding use cases that are aiming to closely emulate the
behaviour of the full CPython CLI, rather than just making a Python
runtime available as part of a larger application.

The PyPreConfig APIs may be used in combination with the initialization
API in this PEP. In such cases, the read-only vs read/write restrictions
for preconfiguration settings apply to PyInitConfig_SetInt in addition
to PyConfig_Set once the interpreter has been preconfigured
(specifically, only use_environment may be updated, attempting to update
any of the other preconfiguration variables will report an error).

Examples

Initialize Python

Example initializing Python, set configuration options of various types,
return -1 on error:

    int init_python(void)
    {
        PyInitConfig *config = PyInitConfig_Create();
        if (config == NULL) {
            printf("PYTHON INIT ERROR: memory allocation failed\n");
            return -1;
        }

        // Set an integer (dev mode)
        if (PyInitConfig_SetInt(config, "dev_mode", 1) < 0) {
            goto error;
        }

        // Set a list of UTF-8 strings (argv)
        char *argv[] = {"my_program", "-c", "pass"};
        if (PyInitConfig_SetStrList(config, "argv",
                                     Py_ARRAY_LENGTH(argv), argv) < 0) {
            goto error;
        }

        // Set a UTF-8 string (program name)
        if (PyInitConfig_SetStr(config, "program_name", L"my_program") < 0) {
            goto error;
        }

        // Initialize Python with the configuration
        if (Py_InitializeFromInitConfig(config) < 0) {
            goto error;
        }
        PyInitConfig_Free(config);
        return 0;

    error:
        // Display the error message
        const char *err_msg;
        (void)PyInitConfig_GetError(config, &err_msg);
        printf("PYTHON INIT ERROR: %s\n", err_msg);
        PyInitConfig_Free(config);

        return -1;
    }

Increase initialization bytes_warning option

Example increasing the bytes_warning option of an initialization
configuration:

    int config_bytes_warning(PyInitConfig *config)
    {
        int64_t bytes_warning;
        if (PyInitConfig_GetInt(config, "bytes_warning", &bytes_warning)) {
            return -1;
        }
        bytes_warning += 1;
        if (PyInitConfig_SetInt(config, "bytes_warning", bytes_warning)) {
            return -1;
        }
        return 0;
    }

Get the runtime verbose option

Example getting the current runtime value of the configuration option
verbose:

    int get_verbose(void)
    {
        int verbose;
        if (PyConfig_GetInt("verbose", &verbose) < 0) {
            // Silently ignore the error
            PyErr_Clear();
            return -1;
        }
        return verbose;
    }

On error, the function silently ignores the error and returns -1. In
practice, getting the verbose option cannot fail, unless a future Python
version removes the option.

Implementation

-   Issue: [C API] PEP 741: Add PyInitConfig C API to customize the
    Python initialization
-   PR: Add PyConfig_Get() function
-   PR: Add PyInitConfig C API

Backwards Compatibility

Changes are fully backward compatible. Only new APIs are added.

Existing API such as the PyConfig C API (PEP 587) are left unchanged.

Rejected Ideas

Configuration as text

It was proposed to provide the configuration as text to make the API
compatible with the stable ABI and to allow custom options.

Example:

    # integer
    bytes_warning = 2

    # string
    filesystem_encoding = "utf8"   # comment

    # list of strings
    argv = ['python', '-c', 'code']

The API would take the configuration as a string, not as a file. Example
with a hypothetical PyInit_SetConfig() function:

    void stable_abi_init_demo(int set_path)
    {
        PyInit_SetConfig(
            "isolated = 1\n"
            "argv = ['python', '-c', 'code']\n"
            "filesystem_encoding = 'utf-8'\n"
        );
        if (set_path) {
            PyInit_SetConfig("pythonpath = '/my/path'");
        }
    }

The example ignores error handling to make it easier to read.

The problem is that generating such configuration text requires adding
quotes to strings and to escape quotes in strings. Formatting an array
of strings becomes non-trivial.

Providing an API to format a string or an array of strings is not really
worth it, whereas Python can provide directly an API to set a
configuration option where the value is passed directly as a string or
an array of strings. It avoids giving special meaning to some
characters, such as newline characters, which would have to be escaped.

Refer to an option with an integer

Using strings to refer to a configuration option requires comparing
strings which can be slower than comparing integers.

Use integers, similar to type "slots" such as Py_tp_doc, to refer to a
configuration option. The const char *name parameter is replaced with
int option.

Accepting custom options is more likely to cause conflicts when using
integers, since it's harder to maintain "namespaces" (ranges) for
integer options. Using strings, a simple prefix with a colon separator
can be used.

Integers also requires maintaining a list of integer constants and so
make the C API and the Python API larger.

Python 3.13 only has around 62 configuration options, and so performance
is not really a blocker issue. If better performance is needed later, a
hash table can be used to get an option by its name.

If getting a configuration option is used in hot code, the value can be
read once and cached. By the way, most configuration options cannot be
changed at runtime.

Multi-phase initialization (similar to PEP 432)

Eric Snow expressed concerns that this proposal might reinforce with
embedders the idea that initialization is a single monolithic step. He
argued that initialization involves 5 distinct phases and even suggested
that the API should reflect this explicitly. Eric proposed that, at the
very least, the implementation of initialization should reflect the
phases, in part for improved code health. Overall, his explanation has
some similarities with PEP 432 and PEP 587.

Another of Eric's key points relevant to this PEP was that, ideally, the
config passed to Py_InitializeFromConfig() should be complete before
that function is called, whereas currently initialization actually
modifies the config.

While Eric wasn't necessarily suggesting an alternative to PEP 741, any
proposal to add a granular initialization API around phases is
effectively the opposite of what this PEP is trying to accomplish. Such
API is more complicated, it requires adding new public structures and
new public functions. It makes the Python initialization more
complicated, rather than this PEP tries to unify existing APIs and make
them simpler (the opposite). Having multiple structures for similar
purpose can lead to duplicate members, similar issue than duplicated
members between existing PyPreConfig and PyConfig structures.

Locale encoding and wide strings

Accepting strings encoded to the locale encoding and accepting wide
strings (wchar_t*) in the PyInitConfig API was deferred to keep the
PyInitConfig API simple and avoid the complexity of the Python
preinitialization. These features are also mostly needed when emulating
the full CPython CLI behaviour, and hence better served by the lower
level PEP 587 API.

Discussions

-   PEP 741: Python Configuration C API (second version) (February
    2024).
-   PEP 741: Python Configuration C API (January 2024).
-   FR: Allow private runtime config to enable extending without
    breaking the PyConfig ABI (August 2022).

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 8103 Title: 2022 Term Steering Council election Author: Ewa
Jodlowska<EMAIL_ADDRESS>Ee Durbin<EMAIL_ADDRESS>Joe Carey
<EMAIL_ADDRESS>Sponsor: Barry Warsaw<EMAIL_ADDRESS>Status: Final
Type: Informational Topic: Governance Content-Type: text/x-rst Created:
04-Oct-2021

Abstract

This document describes the schedule and other details of the December
2021 election for the Python steering council, as specified in PEP 13.
This is the steering council election for the 2022 term (i.e. Python
3.11).

Election Administration

The steering council appointed the Python Software Foundation Director
of Infrastructure, Ee Durbin and Accounting Manager, Joe Carey, to
coadminister the election.

Schedule

There will be a two-week nomination period, followed by a two-week vote.

The nomination period was: November 1, 2021 through November 16, 2021
12:00 UTC (The end of November 15, 2021 Anywhere on Earth). As announced
on python-committers, and took place on discuss.python.org.

The voting period is: December 1, 2021 12:00 UTC through December 16,
2021 12:00 UTC (The end of December 15, 2021 Anywhere on Earth).

Candidates

Candidates must be nominated by a core team member. If the candidate is
a core team member, they may nominate themselves.

Nominees (in alphabetical order):

-   Brett Cannon
-   Mariatta
-   David Mertz
-   Dong-hee Na
-   Pablo Galindo Salgado
-   Gregory P. Smith
-   Victor Stinner
-   Petr Viktorin
-   Barry Warsaw
-   Thomas Wouters

Withdrawn nominations:

-   None

Voter Roll

All active Python core team members are eligible to vote. Active status
is determined as described in PEP 13 <13#membership> and implemented via
the software at python/voters[1].

Ballots will be distributed based on the The Python Voter Roll for this
election [2].

While this file is not public as it contains private email addresses,
the Complete Voter Roll by name will be made available when the roll is
created.

Election Implementation

The election will be conducted using the Helios Voting Service.

Configuration

Short name: 2022-python-steering-council

Name: 2022 Python Steering Council Election

Description:
Election for the Python steering council, as specified in PEP 13. This is steering council election for the 2022 term.

type: Election

Use voter aliases: [X]

Randomize answer order: [X]

Private: [X]

Help Email Address<EMAIL_ADDRESS>
Voting starts at: December 1, 2021 00:00 UTC

Voting ends at: December 16, 2021 12:00 UTC

This will create an election in which:

-   Voting is not open to the public, only those on the Voter Roll may
    participate. Ballots will be emailed when voting starts.
-   Candidates are presented in random order, to help avoid bias.
-   Voter identities and ballots are protected against cryptographic
    advances.

Questions

Question 1

Select between 0 and - (approval) answers. Result Type: absolute

Question: Select candidates for the Python Steering Council

Answer #1 - #N: Candidates from Candidates_ Section

Results

Of 85 eligible voters, 67 cast ballots.

The top five vote-getters are:

-   Pablo Galindo Salgado
-   Petr Viktorin
-   Thomas Wouters
-   Gregory P. Smith
-   Brett Cannon

No conflict of interest as defined in PEP 13 were observed.

The full vote counts are as follows:

  ----------------------------------------
  Candidate               Votes Received
  ----------------------- ----------------
  Pablo Galindo Salgado   61

  Petr Viktorin           48

  Thomas Wouters          48

  Gregory P. Smith        44

  Brett Cannon            42

  Barry Warsaw            39

  Victor Stinner          35

  Mariatta                34

  Dong-hee Na             26

  David Mertz             24
  ----------------------------------------

Copyright

This document has been placed in the public domain.

Complete Voter Roll

Active Python core developers

    Abhilash Raj
    Alex Gaynor
    Ammar Askar
    Andrew Kuchling
    Andrew Svetlov
    Antoine Pitrou
    Barry Warsaw
    Batuhan Taskaya
    Benjamin Peterson
    Berker Peksağ
    Brandt Bucher
    Brett Cannon
    Brian Curtin
    Brian Quinlan
    Carol Willing
    Cheryl Sabella
    Chris Jerdonek
    Chris Withers
    Christian Heimes
    Dino Viehland
    Dong-hee Na
    Éric Araujo
    Eric Snow
    Eric V. Smith
    Ethan Furman
    Facundo Batista
    Fred Drake
    Giampaolo Rodolà
    Gregory P. Smith
    Guido van Rossum
    Hynek Schlawack
    Inada Naoki
    Irit Katriel
    Ivan Levkivskyi
    Jason R. Coombs
    Jeremy Kloth
    Jesús Cea
    Joannah Nanjekye
    Julien Palard
    Karthikeyan Singaravelan
    Ken Jin
    Kushal Das
    Kyle Stanley
    Larry Hastings
    Lisa Roach
    Łukasz Langa
    Lysandros Nikolaou
    Marc-André Lemburg
    Mariatta
    Mark Dickinson
    Mark Shannon
    Nathaniel J. Smith
    Ned Deily
    Neil Schemenauer
    Alyssa Coghlan
    Pablo Galindo
    Paul Ganssle
    Paul Moore
    Petr Viktorin
    Raymond Hettinger
    Ronald Oussoren
    Senthil Kumaran
    Serhiy Storchaka
    Stefan Behnel
    Stéphane Wirtel
    Steve Dower
    Tal Einat
    Terry Jan Reedy
    Thomas Wouters
    Tim Golden
    Tim Peters
    Victor Stinner
    Vinay Sajip
    Xiang Zhang
    Yury Selivanov
    Zachary Ware

[1] This repository is private and accessible only to Python Core
Developers, administrators, and Python Software Foundation Staff as it
contains personal email addresses.

[2] This repository is private and accessible only to Python Core
Developers, administrators, and Python Software Foundation Staff as it
contains personal email addresses.

authors:
Ee Durbin
Ewa Jodlowska
Joe Carey