text:
PEP: 525 Title: Asynchronous Generators Version: $Revision$
Last-Modified: $Date$ Author: Yury Selivanov<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards
Track Content-Type: text/x-rst Created: 28-Jul-2016 Python-Version: 3.6
Post-History: 02-Aug-2016, 23-Aug-2016, 01-Sep-2016, 06-Sep-2016

Abstract

PEP 492 introduced support for native coroutines and async/await syntax
to Python 3.5. It is proposed here to extend Python's asynchronous
capabilities by adding support for asynchronous generators.

Rationale and Goals

Regular generators (introduced in PEP 255) enabled an elegant way of
writing complex data producers and have them behave like an iterator.

However, currently there is no equivalent concept for the asynchronous
iteration protocol (async for). This makes writing asynchronous data
producers unnecessarily complex, as one must define a class that
implements __aiter__ and __anext__ to be able to use it in an async for
statement.

Essentially, the goals and rationale for PEP 255, applied to the
asynchronous execution case, hold true for this proposal as well.

Performance is an additional point for this proposal: in our testing of
the reference implementation, asynchronous generators are 2x faster than
an equivalent implemented as an asynchronous iterator.

As an illustration of the code quality improvement, consider the
following class that prints numbers with a given delay once iterated:

    class Ticker:
        """Yield numbers from 0 to `to` every `delay` seconds."""

        def __init__(self, delay, to):
            self.delay = delay
            self.i = 0
            self.to = to

        def __aiter__(self):
            return self

        async def __anext__(self):
            i = self.i
            if i >= self.to:
                raise StopAsyncIteration
            self.i += 1
            if i:
                await asyncio.sleep(self.delay)
            return i

The same can be implemented as a much simpler asynchronous generator:

    async def ticker(delay, to):
        """Yield numbers from 0 to `to` every `delay` seconds."""
        for i in range(to):
            yield i
            await asyncio.sleep(delay)

Specification

This proposal introduces the concept of asynchronous generators to
Python.

This specification presumes knowledge of the implementation of
generators and coroutines in Python (PEP 342, PEP 380 and PEP 492).

Asynchronous Generators

A Python generator is any function containing one or more yield
expressions:

    def func():            # a function
        return

    def genfunc():         # a generator function
        yield

We propose to use the same approach to define asynchronous generators:

    async def coro():      # a coroutine function
        await smth()

    async def asyncgen():  # an asynchronous generator function
        await smth()
        yield 42

The result of calling an asynchronous generator function is an
asynchronous generator object, which implements the asynchronous
iteration protocol defined in PEP 492.

It is a SyntaxError to have a non-empty return statement in an
asynchronous generator.

Support for Asynchronous Iteration Protocol

The protocol requires two special methods to be implemented:

1.  An __aiter__ method returning an asynchronous iterator.
2.  An __anext__ method returning an awaitable object, which uses
    StopIteration exception to "yield" values, and StopAsyncIteration
    exception to signal the end of the iteration.

Asynchronous generators define both of these methods. Let's manually
iterate over a simple asynchronous generator:

    async def genfunc():
        yield 1
        yield 2

    gen = genfunc()

    assert gen.__aiter__() is gen

    assert await gen.__anext__() == 1
    assert await gen.__anext__() == 2

    await gen.__anext__()  # This line will raise StopAsyncIteration.

Finalization

PEP 492 requires an event loop or a scheduler to run coroutines. Because
asynchronous generators are meant to be used from coroutines, they also
require an event loop to run and finalize them.

Asynchronous generators can have try..finally blocks, as well as
async with. It is important to provide a guarantee that, even when
partially iterated, and then garbage collected, generators can be safely
finalized. For example:

    async def square_series(con, to):
        async with con.transaction():
            cursor = con.cursor(
                'SELECT generate_series(0, $1) AS i', to)
            async for row in cursor:
                yield row['i'] ** 2

    async for i in square_series(con, 1000):
        if i == 100:
            break

The above code defines an asynchronous generator that uses async with to
iterate over a database cursor in a transaction. The generator is then
iterated over with async for, which interrupts the iteration at some
point.

The square_series() generator will then be garbage collected, and
without a mechanism to asynchronously close the generator, Python
interpreter would not be able to do anything.

To solve this problem we propose to do the following:

1.  Implement an aclose method on asynchronous generators returning a
    special awaitable. When awaited it throws a GeneratorExit into the
    suspended generator and iterates over it until either a
    GeneratorExit or a StopAsyncIteration occur.

    This is very similar to what the close() method does to regular
    Python generators, except that an event loop is required to execute
    aclose().

2.  Raise a RuntimeError, when an asynchronous generator executes a
    yield expression in its finally block (using await is fine, though):

        async def gen():
            try:
                yield
            finally:
                await asyncio.sleep(1)   # Can use 'await'.

                yield                    # Cannot use 'yield',
                                         # this line will trigger a
                                         # RuntimeError.

3.  Add two new methods to the sys module: set_asyncgen_hooks() and
    get_asyncgen_hooks().

The idea behind sys.set_asyncgen_hooks() is to allow event loops to
intercept asynchronous generators iteration and finalization, so that
the end user does not need to care about the finalization problem, and
everything just works.

sys.set_asyncgen_hooks() accepts two arguments:

-   firstiter: a callable which will be called when an asynchronous
    generator is iterated for the first time.
-   finalizer: a callable which will be called when an asynchronous
    generator is about to be GCed.

When an asynchronous generator is iterated for the first time, it stores
a reference to the current finalizer.

When an asynchronous generator is about to be garbage collected, it
calls its cached finalizer. The assumption is that the finalizer will
schedule an aclose() call with the loop that was active when the
iteration started.

For instance, here is how asyncio is modified to allow safe finalization
of asynchronous generators:

    # asyncio/base_events.py

    class BaseEventLoop:

        def run_forever(self):
            ...
            old_hooks = sys.get_asyncgen_hooks()
            sys.set_asyncgen_hooks(finalizer=self._finalize_asyncgen)
            try:
                ...
            finally:
                sys.set_asyncgen_hooks(*old_hooks)
                ...

        def _finalize_asyncgen(self, gen):
            self.create_task(gen.aclose())

The second argument, firstiter, allows event loops to maintain a weak
set of asynchronous generators instantiated under their control. This
makes it possible to implement "shutdown" mechanisms to safely finalize
all open generators and close the event loop.

sys.set_asyncgen_hooks() is thread-specific, so several event loops
running in parallel threads can use it safely.

sys.get_asyncgen_hooks() returns a namedtuple-like structure with
firstiter and finalizer fields.

asyncio

The asyncio event loop will use sys.set_asyncgen_hooks() API to maintain
a weak set of all scheduled asynchronous generators, and to schedule
their aclose() coroutine methods when it is time for generators to be
GCed.

To make sure that asyncio programs can finalize all scheduled
asynchronous generators reliably, we propose to add a new event loop
coroutine method loop.shutdown_asyncgens(). The method will schedule all
currently open asynchronous generators to close with an aclose() call.

After calling the loop.shutdown_asyncgens() method, the event loop will
issue a warning whenever a new asynchronous generator is iterated for
the first time. The idea is that after requesting all asynchronous
generators to be shutdown, the program should not execute code that
iterates over new asynchronous generators.

An example of how shutdown_asyncgens coroutine should be used:

    try:
        loop.run_forever()
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

Asynchronous Generator Object

The object is modeled after the standard Python generator object.
Essentially, the behaviour of asynchronous generators is designed to
replicate the behaviour of synchronous generators, with the only
difference in that the API is asynchronous.

The following methods and properties are defined:

1.  agen.__aiter__(): Returns agen.

2.  agen.__anext__(): Returns an awaitable, that performs one
    asynchronous generator iteration when awaited.

3.  agen.asend(val): Returns an awaitable, that pushes the val object in
    the agen generator. When the agen has not yet been iterated, val
    must be None.

    Example:

        async def gen():
            await asyncio.sleep(0.1)
            v = yield 42
            print(v)
            await asyncio.sleep(0.2)

        g = gen()

        await g.asend(None)      # Will return 42 after sleeping
                                 # for 0.1 seconds.

        await g.asend('hello')   # Will print 'hello' and
                                 # raise StopAsyncIteration
                                 # (after sleeping for 0.2 seconds.)

4.  agen.athrow(typ, [val, [tb]]): Returns an awaitable, that throws an
    exception into the agen generator.

    Example:

        async def gen():
            try:
                await asyncio.sleep(0.1)
                yield 'hello'
            except ZeroDivisionError:
                await asyncio.sleep(0.2)
                yield 'world'

        g = gen()
        v = await g.asend(None)
        print(v)                # Will print 'hello' after
                                # sleeping for 0.1 seconds.

        v = await g.athrow(ZeroDivisionError)
        print(v)                # Will print 'world' after
                                $ sleeping 0.2 seconds.

5.  agen.aclose(): Returns an awaitable, that throws a GeneratorExit
    exception into the generator. The awaitable can either return a
    yielded value, if agen handled the exception, or agen will be closed
    and the exception will propagate back to the caller.

6.  agen.__name__ and agen.__qualname__: readable and writable name and
    qualified name attributes.

7.  agen.ag_await: The object that agen is currently awaiting on, or
    None. This is similar to the currently available gi_yieldfrom for
    generators and cr_await for coroutines.

8.  agen.ag_frame, agen.ag_running, and agen.ag_code: defined in the
    same way as similar attributes of standard generators.

StopIteration and StopAsyncIteration are not propagated out of
asynchronous generators, and are replaced with a RuntimeError.

Implementation Details

Asynchronous generator object (PyAsyncGenObject) shares the struct
layout with PyGenObject. In addition to that, the reference
implementation introduces three new objects:

1.  PyAsyncGenASend: the awaitable object that implements __anext__ and
    asend() methods.
2.  PyAsyncGenAThrow: the awaitable object that implements athrow() and
    aclose() methods.
3.  _PyAsyncGenWrappedValue: every directly yielded object from an
    asynchronous generator is implicitly boxed into this structure. This
    is how the generator implementation can separate objects that are
    yielded using regular iteration protocol from objects that are
    yielded using asynchronous iteration protocol.

PyAsyncGenASend and PyAsyncGenAThrow are awaitables (they have __await__
methods returning self) and are coroutine-like objects (implementing
__iter__, __next__, send() and throw() methods). Essentially, they
control how asynchronous generators are iterated:

[image]

PyAsyncGenASend and PyAsyncGenAThrow

PyAsyncGenASend is a coroutine-like object that drives __anext__ and
asend() methods and implements the asynchronous iteration protocol.

agen.asend(val) and agen.__anext__() return instances of PyAsyncGenASend
(which hold references back to the parent agen object.)

The data flow is defined as follows:

1.  When PyAsyncGenASend.send(val) is called for the first time, val is
    pushed to the parent agen object (using existing facilities of
    PyGenObject.)

    Subsequent iterations over the PyAsyncGenASend objects, push None to
    agen.

    When a _PyAsyncGenWrappedValue object is yielded, it is unboxed, and
    a StopIteration exception is raised with the unwrapped value as an
    argument.

2.  When PyAsyncGenASend.throw(*exc) is called for the first time, *exc
    is thrown into the parent agen object.

    Subsequent iterations over the PyAsyncGenASend objects, push None to
    agen.

    When a _PyAsyncGenWrappedValue object is yielded, it is unboxed, and
    a StopIteration exception is raised with the unwrapped value as an
    argument.

3.  return statements in asynchronous generators raise
    StopAsyncIteration exception, which is propagated through
    PyAsyncGenASend.send() and PyAsyncGenASend.throw() methods.

PyAsyncGenAThrow is very similar to PyAsyncGenASend. The only difference
is that PyAsyncGenAThrow.send(), when called first time, throws an
exception into the parent agen object (instead of pushing a value into
it.)

New Standard Library Functions and Types

1.  types.AsyncGeneratorType -- type of asynchronous generator object.
2.  sys.set_asyncgen_hooks() and sys.get_asyncgen_hooks() methods to set
    up asynchronous generators finalizers and iteration interceptors in
    event loops.
3.  inspect.isasyncgen() and inspect.isasyncgenfunction() introspection
    functions.
4.  New method for asyncio event loop: loop.shutdown_asyncgens().
5.  New collections.abc.AsyncGenerator abstract base class.

Backwards Compatibility

The proposal is fully backwards compatible.

In Python 3.5 it is a SyntaxError to define an async def function with a
yield expression inside, therefore it's safe to introduce asynchronous
generators in 3.6.

Performance

Regular Generators

There is no performance degradation for regular generators. The
following micro benchmark runs at the same speed on CPython with and
without asynchronous generators:

    def gen():
        i = 0
        while i < 100000000:
            yield i
            i += 1

    list(gen())

Improvements over asynchronous iterators

The following micro-benchmark shows that asynchronous generators are
about 2.3x faster than asynchronous iterators implemented in pure
Python:

    N = 10 ** 7

    async def agen():
        for i in range(N):
            yield i

    class AIter:
        def __init__(self):
            self.i = 0

        def __aiter__(self):
            return self

        async def __anext__(self):
            i = self.i
            if i >= N:
                raise StopAsyncIteration
            self.i += 1
            return i

Design Considerations

aiter() and anext() builtins

Originally, PEP 492 defined __aiter__ as a method that should return an
awaitable object, resulting in an asynchronous iterator.

However, in CPython 3.5.2, __aiter__ was redefined to return
asynchronous iterators directly. To avoid breaking backwards
compatibility, it was decided that Python 3.6 will support both ways:
__aiter__ can still return an awaitable with a DeprecationWarning being
issued.

Because of this dual nature of __aiter__ in Python 3.6, we cannot add a
synchronous implementation of aiter() built-in. Therefore, it is
proposed to wait until Python 3.7.

Asynchronous list/dict/set comprehensions

Syntax for asynchronous comprehensions is unrelated to the asynchronous
generators machinery, and should be considered in a separate PEP.

Asynchronous yield from

While it is theoretically possible to implement yield from support for
asynchronous generators, it would require a serious redesign of the
generators implementation.

yield from is also less critical for asynchronous generators, since
there is no need provide a mechanism of implementing another coroutines
protocol on top of coroutines. And to compose asynchronous generators a
simple async for loop can be used:

    async def g1():
        yield 1
        yield 2

    async def g2():
        async for v in g1():
            yield v

Why the asend() and athrow() methods are necessary

They make it possible to implement concepts similar to
contextlib.contextmanager using asynchronous generators. For instance,
with the proposed design, it is possible to implement the following
pattern:

    @async_context_manager
    async def ctx():
        await open()
        try:
            yield
        finally:
            await close()

    async with ctx():
        await ...

Another reason is that it is possible to push data and throw exceptions
into asynchronous generators using the object returned from __anext__
object, but it is hard to do that correctly. Adding explicit asend() and
athrow() will pave a safe way to accomplish that.

In terms of implementation, asend() is a slightly more generic version
of __anext__, and athrow() is very similar to aclose(). Therefore,
having these methods defined for asynchronous generators does not add
any extra complexity.

Example

A working example with the current reference implementation (will print
numbers from 0 to 9 with one second delay):

    async def ticker(delay, to):
        for i in range(to):
            yield i
            await asyncio.sleep(delay)

    async def run():
        async for i in ticker(1, 10):
            print(i)

    import asyncio
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(run())
    finally:
        loop.close()

Acceptance

PEP 525 was accepted by Guido, September 6, 2016[1].

Implementation

The implementation is tracked in issue 28003[2]. The reference
implementation git repository is available at[3].

References

Acknowledgments

I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Nathaniel
Smith, Łukasz Langa, Andrew Svetlov and many others for their feedback,
code reviews, and discussions around this PEP.

Copyright

This document has been placed in the public domain.

[1] https://mail.python.org/pipermail/python-dev/2016-September/146267.html

[2] http://bugs.python.org/issue28003

[3] https://github.com/1st1/cpython/tree/async_gen


text:
PEP: 383 Title: Non-decodable Bytes in System Character Interfaces
Author: Martin von Löwis<EMAIL_ADDRESS>Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 22-Apr-2009
Python-Version: 3.1 Post-History:

Abstract

File names, environment variables, and command line arguments are
defined as being character data in POSIX; the C APIs however allow
passing arbitrary bytes - whether these conform to a certain encoding or
not. This PEP proposes a means of dealing with such irregularities by
embedding the bytes in character strings in such a way that allows
recreation of the original byte string.

Rationale

The C char type is a data type that is commonly used to represent both
character data and bytes. Certain POSIX interfaces are specified and
widely understood as operating on character data, however, the system
call interfaces make no assumption on the encoding of these data, and
pass them on as-is. With Python 3, character strings use a Unicode-based
internal representation, making it difficult to ignore the encoding of
byte strings in the same way that the C interfaces can ignore the
encoding.

On the other hand, Microsoft Windows NT has corrected the original
design limitation of Unix, and made it explicit in its system interfaces
that these data (file names, environment variables, command line
arguments) are indeed character data, by providing a Unicode-based API
(keeping a C-char-based one for backwards compatibility).

For Python 3, one proposed solution is to provide two sets of APIs: a
byte-oriented one, and a character-oriented one, where the
character-oriented one would be limited to not being able to represent
all data accurately. Unfortunately, for Windows, the situation would be
exactly the opposite: the byte-oriented interface cannot represent all
data; only the character-oriented API can. As a consequence, libraries
and applications that want to support all user data in a cross-platform
manner have to accept mish-mash of bytes and characters exactly in the
way that caused endless troubles for Python 2.x.

With this PEP, a uniform treatment of these data as characters becomes
possible. The uniformity is achieved by using specific encoding
algorithms, meaning that the data can be converted back to bytes on
POSIX systems only if the same encoding is used.

Being able to treat such strings uniformly will allow application
writers to abstract from details specific to the operating system, and
reduces the risk of one API failing when the other API would have
worked.

Specification

On Windows, Python uses the wide character APIs to access
character-oriented APIs, allowing direct conversion of the environmental
data to Python str objects (PEP 277).

On POSIX systems, Python currently applies the locale's encoding to
convert the byte data to Unicode, failing for characters that cannot be
decoded. With this PEP, non-decodable bytes >= 128 will be represented
as lone surrogate codes U+DC80..U+DCFF. Bytes below 128 will produce
exceptions; see the discussion below.

To convert non-decodable bytes, a new error handler (PEP 293)
"surrogateescape" is introduced, which produces these surrogates. On
encoding, the error handler converts the surrogate back to the
corresponding byte. This error handler will be used in any API that
receives or produces file names, command line arguments, or environment
variables.

The error handler interface is extended to allow the encode error
handler to return byte strings immediately, in addition to returning
Unicode strings which then get encoded again (also see the discussion
below).

Byte-oriented interfaces that already exist in Python 3.0 are not
affected by this specification. They are neither enhanced nor
deprecated.

External libraries that operate on file names (such as GUI file
choosers) should also encode them according to the PEP.

Discussion

This surrogateescape encoding is based on Markus Kuhn's idea that he
called UTF-8b[1].

While providing a uniform API to non-decodable bytes, this interface has
the limitation that chosen representation only "works" if the data get
converted back to bytes with the surrogateescape error handler also.
Encoding the data with the locale's encoding and the (default) strict
error handler will raise an exception, encoding them with UTF-8 will
produce nonsensical data.

Data obtained from other sources may conflict with data produced by this
PEP. Dealing with such conflicts is out of scope of the PEP.

This PEP allows the possibility of "smuggling" bytes in character
strings. This would be a security risk if the bytes are
security-critical when interpreted as characters on a target system,
such as path name separators. For this reason, the PEP rejects smuggling
bytes below 128. If the target system uses EBCDIC, such smuggled bytes
may still be a security risk, allowing smuggling of e.g. square brackets
or the backslash. Python currently does not support EBCDIC, so this
should not be a problem in practice. Anybody porting Python to an EBCDIC
system might want to adjust the error handlers, or come up with other
approaches to address the security risks.

Encodings that are not compatible with ASCII are not supported by this
specification; bytes in the ASCII range that fail to decode will cause
an exception. It is widely agreed that such encodings should not be used
as locale charsets.

For most applications, we assume that they eventually pass data received
from a system interface back into the same system interfaces. For
example, an application invoking os.listdir() will likely pass the
result strings back into APIs like os.stat() or open(), which then
encodes them back into their original byte representation. Applications
that need to process the original byte strings can obtain them by
encoding the character strings with the file system encoding, passing
"surrogateescape" as the error handler name. For example, a function
that works like os.listdir, except for accepting and returning bytes,
would be written as:

    def listdir_b(dirname):
        fse = sys.getfilesystemencoding()
        dirname = dirname.decode(fse, "surrogateescape")
        for fn in os.listdir(dirname):
            # fn is now a str object
            yield fn.encode(fse, "surrogateescape")

The extension to the encode error handler interface proposed by this PEP
is necessary to implement the 'surrogateescape' error handler, because
there are required byte sequences which cannot be generated from
replacement Unicode. However, the encode error handler interface
presently requires replacement Unicode to be provided in lieu of the
non-encodable Unicode from the source string. Then it promptly encodes
that replacement Unicode. In some error handlers, such as the
'surrogateescape' proposed here, it is also simpler and more efficient
for the error handler to provide a pre-encoded replacement byte string,
rather than forcing it to calculating Unicode from which the encoder
would create the desired bytes.

A few alternative approaches have been proposed:

-   create a new string subclass that supports embedded bytes
-   use different escape schemes, such as escaping with a NUL character,
    or mapping to infrequent characters.

Of these proposals, the approach of escaping each byte XX with the
sequence U+0000 U+00XX has the disadvantage that encoding to UTF-8 will
introduce a NUL byte in the UTF-8 sequence. As a consequence, C
libraries may interpret this as a string termination, even though the
string continues. In particular, the gtk libraries will truncate text in
this case; other libraries may show similar problems.

References

Copyright

This document has been placed in the public domain.

[1] UTF-8b
https://web.archive.org/web/20090830064219/http://mail.nl.linux.org/linux-utf8/2000-07/msg00040.html

license: Public Domain


text:
PEP: 625 Title: Filename of a Source Distribution Author: Tzu-ping Chung
<EMAIL_ADDRESS>Paul Moore<EMAIL_ADDRESS>PEP-Delegate:
Pradyun Gedam<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/draft-pep-file-name-of-a-source-distribution/4686
Status: Accepted Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 08-Jul-2020 Post-History: 08-Jul-2020 Resolution:
https://discuss.python.org/t/pep-625-file-name-of-a-source-distribution/4686/159

Abstract

This PEP describes a standard naming scheme for a Source Distribution,
also known as an sdist. An sdist is distinct from an arbitrary archive
file containing source code of Python packages, and can be used to
communicate information about the distribution to packaging tools.

A standard sdist specified here is a gzipped tar file with a specially
formatted filename and the usual .tar.gz suffix. This PEP does not
specify the contents of the tarball, as that is covered in other
specifications.

Motivation

An sdist is a Python package distribution that contains "source code" of
the Python package, and requires a build step to be turned into a wheel
on installation. This format is often considered as an unbuilt
counterpart of a PEP 427 wheel, and given special treatments in various
parts of the packaging ecosystem.

The content of an sdist is specified in PEP 517 and PEP 643, but
currently the filename of the sdist is incompletely specified, meaning
that consumers of the format must download and process the sdist to
confirm the name and version of the distribution included within.

Installers currently rely on heuristics to infer the name and/or version
from the filename, to help the installation process. pip, for example,
parses the filename of an sdist from a PEP 503 index, to obtain the
distribution's project name and version for dependency resolution
purposes. But due to the lack of specification, the installer does not
have any guarantee as to the correctness of the inferred data, and must
verify it at some point by locally building the distribution metadata.

This build step is awkward for a certain class of operations, when the
user does not expect the build process to occur. pypa/pip#8387 describes
an example. The command pip download --no-deps --no-binary=numpy numpy
is expected to only download an sdist for numpy, since we do not need to
check for dependencies, and both the name and version are available by
introspecting the downloaded filename. pip, however, cannot assume the
downloaded archive follows the convention, and must build and check the
metadata. For a PEP 518 project, this means running the
prepare_metadata_for_build_wheel hook specified in PEP 517, which incurs
significant overhead.

Rationale

By creating a special filename scheme for the sdist format, this PEP
frees up tools from the time-consuming metadata verification step when
they only need the metadata available in the filename.

This PEP also serves as the formal specification to the long-standing
filename convention used by the current sdist implementations. The
filename contains the distribution name and version, to aid tools
identifying a distribution without needing to download, unarchive the
file, and perform costly metadata generation for introspection, if all
the information they need is available in the filename.

Specification

The name of an sdist should be {distribution}-{version}.tar.gz.

-   distribution is the name of the distribution as defined in PEP 345,
    and normalised as described in the wheel spec e.g. 'pip',
    'flit_core'.
-   version is the version of the distribution as defined in PEP 440,
    e.g. 20.2, and normalised according to the rules in that PEP.

An sdist must be a gzipped tar archive in pax format, that is able to be
extracted by the standard library tarfile module with the open flag
'r:gz'.

Code that produces an sdist file MUST give the file a name that matches
this specification. The specification of the build_sdist hook from PEP
517 is extended to require this naming convention.

Code that processes sdist files MAY determine the distribution name and
version by simply parsing the filename, and is not required to verify
that information by generating or reading the metadata from the sdist
contents.

Conforming sdist files can be recognised by the presence of the .tar.gz
suffix and a single hyphen in the filename. Note that some legacy files
may also match these criteria, but this is not expected to be an issue
in practice. See the "Backwards Compatibility" section of this document
for more details.

Backwards Compatibility

The new filename scheme is a subset of the current informal naming
convention for sdist files, so tools that create or publish files
conforming to this standard will be readable by older tools that only
understand the previous naming conventions.

Tools that consume sdist filenames would technically not be able to
determine whether a file is using the new standard or a legacy form.
However, a review of the filenames on PyPI determined that 37% of files
are obviously legacy (because they contain multiple or no hyphens) and
of the remainder, parsing according to this PEP gives the correct answer
in all but 0.004% of cases.

Currently, tools that consume sdists should, if they are to be fully
correct, treat the name and version parsed from the filename as
provisional, and verify them by downloading the file and generating the
actual metadata (or reading it, if the sdist conforms to PEP 643). Tools
supporting this specification can treat the name and version from the
filename as definitive. In theory, this could risk mistakes if a legacy
filename is assumed to conform to this PEP, but in practice the chance
of this appears to be vanishingly small.

Rejected Ideas

Rely on the specification for sdist metadata

Since this PEP was first written, PEP 643 has been accepted, defining a
trustworthy, standard sdist metadata format. This allows distribution
metadata (and in particular name and version) to be determined
statically.

This is not considered sufficient, however, as in a number of
significant cases (for example, reading filenames from a package index)
the application only has access to the filename, and reading metadata
would involve a potentially costly download.

Use a dedicated file extension

The original version of this PEP proposed a filename of
{distribution}-{version}.sdist. This has the advantage of being
explicit, as well as allowing a future change to the storage format
without needing a further change of the file naming convention.

However, there are significant compatibility issues with a new
extension. Index servers may currently disallow unknown extensions, and
if we introduced a new one, it is not clear how to handle cases like a
legacy index trying to mirror an index that hosts new-style sdists. Is
it acceptable to only partially mirror, omitting sdists for newer
versions of projects? Also, build backends that produce the new format
would be incompaible with index servers that only accept the old format,
and as there is often no way for a user to request an older version of a
backend when doing a build, this could make it impossible to build and
upload sdists.

Augment a currently common sdist naming scheme

A scheme {distribution}-{version}.sdist.tar.gz was raised during the
initial discussion. This was abandoned due to backwards compatibility
issues with currently available installation tools. pip 20.1, for
example, would parse distribution-1.0.sdist.tar.gz as project
distribution with version 1.0.sdist. This would cause the sdist to be
downloaded, but fail to install due to inconsistent metadata.

The main advantage of this proposal was that it is easier for tools to
recognise the new-style naming. But this is not a particularly
significant benefit, given that all sdists with a single hyphen in the
name are parsed the same way under the old and new rules.

Open Issues

The contents of an sdist are required to contain a single top-level
directory named {name}-{version}. Currently no normalisation rules are
required for the components of this name. Should this PEP require that
the same normalisation rules are applied here as for the filename? Note
that in practice, it is likely that tools will create the two names
using the same code, so normalisation is likely to happen naturally,
even if it is not explicitly required.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 245 Title: Python Interface Syntax Version: $Revision$
Last-Modified: $Date$ Author: Michel Pelletier
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 11-Jan-2001 Python-Version: 2.2
Post-History: 21-Mar-2001

Note

The no-longer-available Zope interfaces wiki page
(https://www.zope.org/Wikis/Interfaces) originally linked here,
containing links to further resources for this PEP, can be found on the
Wayback Machine archive. Also, the Interface-Dev Zope mailing list on
which this PEP was discussed was shut down, but its archives remain
available.

Rejection Notice

I'm rejecting this PEP. It's been five years now. While at some point I
expect that Python will have interfaces, it would be naive to expect it
to resemble the syntax in this PEP. Also, PEP 246 is being rejected in
favor of something completely different; interfaces won't play a role in
adaptation or whatever will replace it. GvR.

Introduction

This PEP describes a proposed syntax for creating interface objects in
Python.

Overview

In addition to thinking about adding a static type system to Python, the
Types-SIG was also charged to devise an interface system for Python. In
December of 1998, Jim Fulton released a prototype interfaces system
based on discussions from the SIG. Many of the issues and background
information on this discussion and prototype can be found in the SIG
archives[1].

Around the end of 2000, Digital Creations began thinking about better
component model designs for Zope[2]. Zope's future component model
relies heavily on interface objects. This led to further development of
Jim's "Scarecrow" interfaces prototype. Starting with version 2.3, Zope
comes with an Interface package as standard software. Zope's Interface
package is used as the reference implementation for this PEP.

The syntax proposed by this PEP relies on syntax enhancements describe
in PEP 232 and describes an underlying framework which PEP 233 could be
based upon. There is some work being done with regard to interface
objects and Proxy objects, so for those optional parts of this PEP you
may want to see[3].

The Problem

Interfaces are important because they solve a number of problems that
arise while developing software:

-   There are many implied interfaces in Python, commonly referred to as
    "protocols". Currently determining those protocols is based on
    implementation introspection, but often that also fails. For
    example, defining __getitem__ implies both a sequence and a mapping
    (the former with sequential, integer keys). There is no way for the
    developer to be explicit about which protocols the object intends to
    implement.
-   Python is limited, from the developer's point of view, by the split
    between types and classes. When types are expected, the consumer
    uses code like 'type(foo) == type("")' to determine if 'foo' is a
    string. When instances of classes are expected, the consumer uses
    'isinstance(foo, MyString)' to determine if 'foo' is an instance of
    the 'MyString' class. There is no unified model for determining if
    an object can be used in a certain, valid way.
-   Python's dynamic typing is very flexible and powerful, but it does
    not have the advantage of static typed languages that provide type
    checking. Static typed languages provide you with much more type
    safety, but are often overly verbose because objects can only be
    generalized by common subclassing and used specifically with casting
    (for example, in Java).

There are also a number of documentation problems that interfaces try to
solve.

-   Developers waste a lot of time looking at the source code of your
    system to figure out how objects work.
-   Developers who are new to your system may misunderstand how your
    objects work, causing, and possibly propagating, usage errors.
-   Because a lack of interfaces means usage is inferred from the
    source, developers may end up using methods and attributes that are
    meant for "internal use only".
-   Code inspection can be hard, and very discouraging to novice
    programmers trying to properly understand code written by gurus.
-   A lot of time is wasted when many people try very hard to understand
    obscurity (like undocumented software). Effort spend up front
    documenting interfaces will save much of this time in the end.

Interfaces try to solve these problems by providing a way for you to
specify a contractual obligation for your object, documentation on how
to use an object, and a built-in mechanism for discovering the contract
and the documentation.

Python has very useful introspection features. It is well known that
this makes exploring concepts in the interactive interpreter easier,
because Python gives you the ability to look at all kinds of information
about the objects: the type, doc strings, instance dictionaries, base
classes, unbound methods and more.

Many of these features are oriented toward introspecting, using and
changing the implementation of software, and one of them ("doc strings")
is oriented toward providing documentation. This proposal describes an
extension to this natural introspection framework that describes an
object's interface.

Overview of the Interface Syntax

For the most part, the syntax of interfaces is very much like the syntax
of classes, but future needs, or needs brought up in discussion, may
define new possibilities for interface syntax.

A formal BNF description of the syntax is givena later in the PEP, for
the purposes of illustration, here is an example of two different
interfaces created with the proposed syntax:

    interface CountFishInterface:
        "Fish counting interface"

        def oneFish():
            "Increments the fish count by one"

        def twoFish():
            "Increments the fish count by two"

        def getFishCount():
            "Returns the fish count"

    interface ColorFishInterface:
        "Fish coloring interface"

        def redFish():
            "Sets the current fish color to red"

        def blueFish():
            "Sets the current fish color to blue"

        def getFishColor():
            "This returns the current fish color"

This code, when evaluated, will create two interfaces called
CountFishInterface and ColorFishInterface. These interfaces are defined
by the interface statement.

The prose documentation for the interfaces and their methods come from
doc strings. The method signature information comes from the signatures
of the def statements. Notice how there is no body for the def
statements. The interface does not implement a service to anything; it
merely describes one. Documentation strings on interfaces and interface
methods are mandatory, a 'pass' statement cannot be provided. The
interface equivalent of a pass statement is an empty doc string.

You can also create interfaces that "extend" other interfaces. Here, you
can see a new type of Interface that extends the CountFishInterface and
ColorFishInterface:

    interface FishMarketInterface(CountFishInterface, ColorFishInterface):
        "This is the documentation for the FishMarketInterface"

        def getFishMonger():
            "Returns the fish monger you can interact with"

        def hireNewFishMonger(name):
            "Hire a new fish monger"

        def buySomeFish(quantity=1):
            "Buy some fish at the market"

The FishMarketInterface extends upon the CountFishInterface and
ColorfishInterface.

Interface Assertion

The next step is to put classes and interfaces together by creating a
concrete Python class that asserts that it implements an interface. Here
is an example FishMarket component that might do this:

    class FishError(Error):
        pass

    class FishMarket implements FishMarketInterface:
        number = 0
        color = None
        monger_name = 'Crusty Barnacles'

        def __init__(self, number, color):
            self.number = number
            self.color = color

        def oneFish(self):
            self.number += 1

        def twoFish(self):
            self.number += 2

        def redFish(self):
            self.color = 'red'

        def blueFish(self):
            self.color = 'blue'

        def getFishCount(self):
            return self.number

        def getFishColor(self):
            return self.color

        def getFishMonger(self):
            return self.monger_name

        def hireNewFishMonger(self, name):
            self.monger_name = name

        def buySomeFish(self, quantity=1):
            if quantity > self.count:
                raise FishError("There's not enough fish")
            self.count -= quantity
            return quantity

This new class, FishMarket defines a concrete class which implements the
FishMarketInterface. The object following the implements statement is
called an "interface assertion". An interface assertion can be either an
interface object, or tuple of interface assertions.

The interface assertion provided in a class statement like this is
stored in the class's __implements__ class attribute. After interpreting
the above example, you would have a class statement that can be examined
like this with an 'implements' built-in function:

    >>> FishMarket
    <class FishMarket at 8140f50>
    >>> FishMarket.__implements__
    (<Interface FishMarketInterface at 81006f0>,)
    >>> f = FishMarket(6, 'red')
    >>> implements(f, FishMarketInterface)
    1
    >>>

A class can realize more than one interface. For example, say you had an
interface called ItemInterface that described how an object worked as an
item in a container object. If you wanted to assert that FishMarket
instances realized the ItemInterface interface as well as the
FishMarketInterface, you can provide an interface assertion that
contained a tuple of interface objects to the FishMarket class:

    class FishMarket implements FishMarketInterface, ItemInterface:
        # ...

Interface assertions can also be used if you want to assert that one
class implements an interface, and all of the interfaces that another
class implements:

    class MyFishMarket implements FishMarketInterface, ItemInterface:
        # ...

    class YourFishMarket implements FooInterface, MyFishMarket.__implements__:
        # ...

This new class YourFishMarket, asserts that it implements the
FooInterface, as well as the interfaces implemented by the MyFishMarket
class.

It's worth going into a little bit more detail about interface
assertions. An interface assertion is either an interface object, or a
tuple of interface assertions. For example:

    FooInterface

    FooInterface, (BarInterface, BobInterface)

    FooInterface, (BarInterface, (BobInterface, MyClass.__implements__))

Are all valid interface assertions. When two interfaces define the same
attributes, the order in which information is preferred in the assertion
is from top-to-bottom, left-to-right.

There are other interface proposals that, in the need for simplicity,
have combined the notion of class and interface to provide simple
interface enforcement. Interface objects have a deferred method that
returns a deferred class that implements this behavior:

    >>> FM = FishMarketInterface.deferred()
    >>> class MyFM(FM): pass

    >>> f = MyFM()
    >>> f.getFishMonger()
    Traceback (innermost last):
      File "<stdin>", line 1, in ?
    Interface.Exceptions.BrokenImplementation:
    An object has failed to implement interface FishMarketInterface

            The getFishMonger attribute was not provided.
    >>>

This provides for a bit of passive interface enforcement by telling you
what you forgot to do to implement that interface.

Formal Interface Syntax

Python syntax is defined in a modified BNF grammar notation described in
the Python Reference Manual[4]. This section describes the proposed
interface syntax using this grammar:

    interfacedef:   "interface" interfacename [extends] ":" suite
    extends:        "(" [expression_list] ")"
    interfacename:  identifier

An interface definition is an executable statement. It first evaluates
the extends list, if present. Each item in the extends list should
evaluate to an interface object.

The interface's suite is then executed in a new execution frame (see the
Python Reference Manual, section 4.1), using a newly created local
namespace and the original global namespace. When the interface's suite
finishes execution, its execution frame is discarded but its local
namespace is saved as interface elements. An interface object is then
created using the extends list for the base interfaces and the saved
interface elements. The interface name is bound to this interface object
in the original local namespace.

This PEP also proposes an extension to Python's 'class' statement:

    classdef:    "class" classname [inheritance] [implements] ":" suite
    implements:  "implements" implist
    implist:     expression-list

    classname,
    inheritance,
    suite,
    expression-list:  see the Python Reference Manual

Before a class' suite is executed, the 'inheritance' and 'implements'
statements are evaluated, if present. The 'inheritance' behavior is
unchanged as defined in Section 7.6 of the Language Reference.

The 'implements', if present, is evaluated after inheritance. This must
evaluate to an interface specification, which is either an interface, or
a tuple of interface specifications. If a valid interface specification
is present, the assertion is assigned to the class object's
'__implements__' attribute, as a tuple.

This PEP does not propose any changes to the syntax of function
definitions or assignments.

Classes and Interfaces

The example interfaces above do not describe any kind of behavior for
their methods, they just describe an interface that a typical FishMarket
object would realize.

You may notice a similarity between interfaces extending from other
interfaces and classes sub-classing from other classes. This is a
similar concept. However it is important to note that interfaces extend
interfaces and classes subclass classes. You cannot extend a class or
subclass an interface. Classes and interfaces are separate.

The purpose of a class is to share the implementation of how an object
works. The purpose of an interface is to document how to work with an
object, not how the object is implemented. It is possible to have
several different classes with very different implementations realize
the same interface.

It's also possible to implement one interface with many classes that mix
in pieces the functionality of the interface or, conversely, it's
possible to have one class implement many interfaces. Because of this,
interfaces and classes should not be confused or intermingled.

Interface-aware built-ins

A useful extension to Python's list of built-in functions in the light
of interface objects would be implements(). This builtin would expect
two arguments, an object and an interface, and return a true value if
the object implements the interface, false otherwise. For example:

    >>> interface FooInterface: pass
    >>> class Foo implements FooInterface: pass
    >>> f = Foo()
    >>> implements(f, FooInterface)
    1

Currently, this functionality exists in the reference implementation as
functions in the Interface package, requiring an "import Interface" to
use it. Its existence as a built-in would be purely for a convenience,
and not necessary for using interfaces, and analogous to isinstance()
for classes.

Backward Compatibility

The proposed interface model does not introduce any backward
compatibility issues in Python. The proposed syntax, however, does.

Any existing code that uses interface as an identifier will break. There
may be other kinds of backwards incompatibility that defining interface
as a new keyword will introduce. This extension to Python's syntax does
not change any existing syntax in any backward incompatible way.

The new from __future__ Python syntax (PEP 236), and the new warning
framework (PEP 230) is ideal for resolving this backward
incompatibility. To use interface syntax now, a developer could use the
statement:

    from __future__ import interfaces

In addition, any code that uses the keyword interface as an identifier
will be issued a warning from Python. After the appropriate period of
time, the interface syntax would become standard, the above import
statement would do nothing, and any identifiers named interface would
raise an exception. This period of time is proposed to be 24 months.

Summary of Proposed Changes to Python

Adding new interface keyword and extending class syntax with implements.

Extending class interface to include __implements__.

Add 'implements(obj, interface)' built-in.

Risks

This PEP proposes adding one new keyword to the Python language,
interface. This will break code.

Open Issues

Goals

Syntax

Architecture

Dissenting Opinion

This PEP has not yet been discussed on python-dev.

References

Copyright

This document has been placed in the public domain.

[1] https://mail.python.org/pipermail/types-sig/1998-December/date.html

[2] http://www.zope.org

[3] http://www.lemburg.com/files/python/mxProxy.html

[4] Python Reference Manual http://docs.python.org/reference/


text:
PEP: 530 Title: Asynchronous Comprehensions Version: $Revision$
Last-Modified: $Date$ Author: Yury Selivanov<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards
Track Content-Type: text/x-rst Created: 03-Sep-2016 Python-Version: 3.6
Post-History: 03-Sep-2016

Abstract

PEP 492 and PEP 525 introduce support for native coroutines and
asynchronous generators using async / await syntax. This PEP proposes to
add asynchronous versions of list, set, dict comprehensions and
generator expressions.

Rationale and Goals

Python has extensive support for synchronous comprehensions, allowing to
produce lists, dicts, and sets with a simple and concise syntax. We
propose implementing similar syntactic constructions for the
asynchronous code.

To illustrate the readability improvement, consider the following
example:

    result = []
    async for i in aiter():
        if i % 2:
            result.append(i)

With the proposed asynchronous comprehensions syntax, the above code
becomes as short as:

    result = [i async for i in aiter() if i % 2]

The PEP also makes it possible to use the await expressions in all kinds
of comprehensions:

    result = [await fun() for fun in funcs]

Specification

Asynchronous Comprehensions

We propose to allow using async for inside list, set and dict
comprehensions. Pending PEP 525 approval, we can also allow creation of
asynchronous generator expressions.

Examples:

-   set comprehension: {i async for i in agen()};
-   list comprehension: [i async for i in agen()];
-   dict comprehension: {i: i ** 2 async for i in agen()};
-   generator expression: (i ** 2 async for i in agen()).

It is allowed to use async for along with if and for clauses in
asynchronous comprehensions and generator expressions:

    dataset = {data for line in aiter()
                    async for data in line
                    if check(data)}

Asynchronous comprehensions are only allowed inside an async def
function.

In principle, asynchronous generator expressions are allowed in any
context. However, in Python 3.6, due to async and await soft-keyword
status, asynchronous generator expressions are only allowed in an
async def function. Once async and await become reserved keywords in
Python 3.7, this restriction will be removed.

await in Comprehensions

We propose to allow the use of await expressions in both asynchronous
and synchronous comprehensions:

    result = [await fun() for fun in funcs]
    result = {await fun() for fun in funcs}
    result = {fun: await fun() for fun in funcs}

    result = [await fun() for fun in funcs if await smth]
    result = {await fun() for fun in funcs if await smth}
    result = {fun: await fun() for fun in funcs if await smth}

    result = [await fun() async for fun in funcs]
    result = {await fun() async for fun in funcs}
    result = {fun: await fun() async for fun in funcs}

    result = [await fun() async for fun in funcs if await smth]
    result = {await fun() async for fun in funcs if await smth}
    result = {fun: await fun() async for fun in funcs if await smth}

This is only valid in async def function body.

Grammar Updates

The proposal requires one change on the grammar level: adding the
optional "async" keyword to comp_for:

    comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter]

The comprehension AST node will have the new is_async argument.

Backwards Compatibility

The proposal is fully backwards compatible.

Acceptance

PEP 530 was accepted by Guido, September 6, 2016[1].

Implementation

The implementation is tracked in issue 28008[2]. The reference
implementation git repository is available at[3].

References

Acknowledgments

I thank Guido van Rossum, Victor Stinner and Elvis Pranskevichus for
their feedback, code reviews, and discussions around this PEP.

Copyright

This document has been placed in the public domain.

[1] https://mail.python.org/pipermail/python-ideas/2016-September/042141.html

[2] http://bugs.python.org/issue28008

[3] https://github.com/1st1/cpython/tree/asyncomp


text:
PEP: 425 Title: Compatibility Tags for Built Distributions Version:
$Revision$ Last-Modified: 07-Aug-2012 Author: Daniel Holth
<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 27-Jul-2012 Python-Version: 3.4 Post-History:
08-Aug-2012, 18-Oct-2012, 15-Feb-2013 Resolution:
https://mail.python.org/pipermail/python-dev/2013-February/124116.html

packaging:platform-compatibility-tags

Abstract

This PEP specifies a tagging system to indicate with which versions of
Python a built or binary distribution is compatible. A set of three tags
indicate which Python implementation and language version, ABI, and
platform a built distribution requires. The tags are terse because they
will be included in filenames.

PEP Acceptance

This PEP was accepted by Alyssa Coghlan on 17th February, 2013.

Rationale

Today "python setup.py bdist" generates the same filename on PyPy and
CPython, but an incompatible archive, making it inconvenient to share
built distributions in the same folder or index. Instead, built
distributions should have a file naming convention that includes enough
information to decide whether or not a particular archive is compatible
with a particular implementation.

Previous efforts come from a time where CPython was the only important
implementation and the ABI was the same as the Python language release.
This specification improves upon the older schemes by including the
Python implementation, language version, ABI, and platform as a set of
tags.

By comparing the tags it supports with the tags listed by the
distribution, an installer can make an educated decision about whether
to download a particular built distribution without having to read its
full metadata.

Overview

The tag format is {python tag}-{abi tag}-{platform tag}

python tag

    ‘py27’, ‘cp33’

abi tag

    ‘cp32dmu’, ‘none’

platform tag

    ‘linux_x86_64’, ‘any’

For example, the tag py27-none-any indicates compatible with Python 2.7
(any Python 2.7 implementation) with no abi requirement, on any
platform.

Use

The wheel built package format includes these tags in its filenames, of
the form
{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.
Other package formats may have their own conventions.

Details

Python Tag

The Python tag indicates the implementation and version required by a
distribution. Major implementations have abbreviated codes, initially:

-   py: Generic Python (does not require implementation-specific
    features)
-   cp: CPython
-   ip: IronPython
-   pp: PyPy
-   jy: Jython

Other Python implementations should use sys.implementation.name.

The version is py_version_nodot. CPython gets away with no dot, but if
one is needed the underscore _ is used instead. PyPy should probably use
its own versions here pp18, pp19.

The version can be just the major version 2 or 3 py2, py3 for many
pure-Python distributions.

Importantly, major-version-only tags like py2 and py3 are not shorthand
for py20 and py30. Instead, these tags mean the packager intentionally
released a cross-version-compatible distribution.

A single-source Python 2/3 compatible distribution can use the compound
tag py2.py3. See Compressed Tag Sets, below.

ABI Tag

The ABI tag indicates which Python ABI is required by any included
extension modules. For implementation-specific ABIs, the implementation
is abbreviated in the same way as the Python Tag, e.g. cp33d would be
the CPython 3.3 ABI with debugging.

The CPython stable ABI is abi3 as in the shared library suffix.

Implementations with a very unstable ABI may use the first 6 bytes (as 8
base64-encoded characters) of the SHA-256 hash of their source code
revision and compiler flags, etc, but will probably not have a great
need to distribute binary distributions. Each implementation's community
may decide how to best use the ABI tag.

Platform Tag

The platform tag is simply distutils.util.get_platform() with all
hyphens - and periods . replaced with underscore _.

-   win32
-   linux_i386
-   linux_x86_64

Use

The tags are used by installers to decide which built distribution (if
any) to download from a list of potential built distributions. The
installer maintains a list of (pyver, abi, arch) tuples that it will
support. If the built distribution's tag is in the list, then it can be
installed.

It is recommended that installers try to choose the most feature
complete built distribution available (the one most specific to the
installation environment) by default before falling back to pure Python
versions published for older Python releases. Installers are also
recommended to provide a way to configure and re-order the list of
allowed compatibility tags; for example, a user might accept only the
*-none-any tags to only download built packages that advertise
themselves as being pure Python.

Another desirable installer feature might be to include "re-compile from
source if possible" as more preferable than some of the compatible but
legacy pre-built options.

This example list is for an installer running under CPython 3.3 on a
linux_x86_64 system. It is in order from most-preferred (a distribution
with a compiled extension module, built for the current version of
Python) to least-preferred (a pure-Python distribution built with an
older version of Python):

1.  cp33-cp33m-linux_x86_64
2.  cp33-abi3-linux_x86_64
3.  cp3-abi3-linux_x86_64
4.  cp33-none-linux_x86_64*
5.  cp3-none-linux_x86_64*
6.  py33-none-linux_x86_64*
7.  py3-none-linux_x86_64*
8.  cp33-none-any
9.  cp3-none-any
10. py33-none-any
11. py3-none-any
12. py32-none-any
13. py31-none-any
14. py30-none-any

-   Built distributions may be platform specific for reasons other than
    C extensions, such as by including a native executable invoked as a
    subprocess.

Sometimes there will be more than one supported built distribution for a
particular version of a package. For example, a packager could release a
package tagged cp33-abi3-linux_x86_64 that contains an optional C
extension and the same distribution tagged py3-none-any that does not.
The index of the tag in the supported tags list breaks the tie, and the
package with the C extension is installed in preference to the package
without because that tag appears first in the list.

Compressed Tag Sets

To allow for compact filenames of bdists that work with more than one
compatibility tag triple, each tag in a filename can instead be a
'.'-separated, sorted, set of tags. For example, pip, a pure-Python
package that is written to run under Python 2 and 3 with the same source
code, could distribute a bdist with the tag py2.py3-none-any. The full
list of simple tags is:

    for x in pytag.split('.'):
        for y in abitag.split('.'):
            for z in archtag.split('.'):
                yield '-'.join((x, y, z))

A bdist format that implements this scheme should include the expanded
tags in bdist-specific metadata. This compression scheme can generate
large numbers of unsupported tags and "impossible" tags that are
supported by no Python implementation e.g. "cp33-cp31u-win64", so use it
sparingly.

FAQ

What tags are used by default?

    Tools should use the most-preferred architecture dependent tag e.g.
    cp33-cp33m-win32 or the most-preferred pure python tag e.g.
    py33-none-any by default. If the packager overrides the default it
    indicates that they intended to provide cross-Python compatibility.

What tag do I use if my distribution uses a feature exclusive to the newest version of Python?

    Compatibility tags aid installers in selecting the most compatible
    build of a single version of a distribution. For example, when there
    is no Python 3.3 compatible build of beaglevote-1.2.0 (it uses a
    Python 3.4 exclusive feature) it may still use the py3-none-any tag
    instead of the py34-none-any tag. A Python 3.3 user must combine
    other qualifiers, such as a requirement for the older release
    beaglevote-1.1.0 that does not use the new feature, to get a
    compatible build.

Why isn't there a . in the Python version number?

    CPython has lasted 20+ years without a 3-digit major release. This
    should continue for some time. Other implementations may use _ as a
    delimiter, since both - and . delimit the surrounding filename.

Why normalise hyphens and other non-alphanumeric characters to underscores?

    To avoid conflicting with the "." and "-" characters that separate
    components of the filename, and for better compatibility with the
    widest range of filesystem limitations for filenames (including
    being usable in URL paths without quoting).

Why not use special character <X> rather than "." or "-"?

    Either because that character is inconvenient or potentially
    confusing in some contexts (for example, "+" must be quoted in URLs,
    "~" is used to denote the user's home directory in POSIX), or
    because the advantages weren't sufficiently compelling to justify
    changing the existing reference implementation for the wheel format
    defined in PEP 427 (for example, using "," rather than "." to
    separate components in a compressed tag).

Who will maintain the registry of abbreviated implementations?

    New two-letter abbreviations can be requested on the python-dev
    mailing list. As a rule of thumb, abbreviations are reserved for the
    current 4 most prominent implementations.

Does the compatibility tag go into METADATA or PKG-INFO?

    No. The compatibility tag is part of the built distribution's
    metadata. METADATA / PKG-INFO should be valid for an entire
    distribution, not a single build of that distribution.

Why didn't you mention my favorite Python implementation?

    The abbreviated tags facilitate sharing compiled Python code in a
    public index. Your Python implementation can use this specification
    too, but with longer tags. Recall that all "pure Python" built
    distributions just use 'py'.

Why is the ABI tag (the second tag) sometimes "none" in the reference implementation?

    Since Python 2 does not have an easy way to get to the SOABI (the
    concept comes from newer versions of Python 3) the reference
    implementation at the time of writing guesses "none". Ideally it
    would detect "py27(du)" analogous to newer versions of Python, but
    in the meantime "none" is a good enough way to say "don't know".

References

[1] Egg Filename-Embedded Metadata
(http://peak.telecommunity.com/DevCenter/EggFormats#filename-embedded-metadata)

[2] Creating Built Distributions
(https://docs.python.org/3.4/distutils/builtdist.html)

Acknowledgements

The author thanks Paul Moore, Alyssa Coghlan, Marc Abramowitz, and Mr.
Michele Lacchia for their valuable help and advice.

Copyright

This document has been placed in the public domain.