text:
PEP: 533 Title: Deterministic cleanup for iterators Version: $Revision$
Last-Modified: $Date$ Author: Nathaniel J. Smith BDFL-Delegate: Yury
Selivanov<EMAIL_ADDRESS>Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 18-Oct-2016 Post-History: 18-Oct-2016

Abstract

We propose to extend the iterator protocol with a new __(a)iterclose__
slot, which is called automatically on exit from (async) for loops,
regardless of how they exit. This allows for convenient, deterministic
cleanup of resources held by iterators without reliance on the garbage
collector. This is especially valuable for asynchronous generators.

Note on timing

In practical terms, the proposal here is divided into two separate
parts: the handling of async iterators, which should ideally be
implemented ASAP, and the handling of regular iterators, which is a
larger but more relaxed project that can't start until 3.7 at the
earliest. But since the changes are closely related, and we probably
don't want to end up with async iterators and regular iterators
diverging in the long run, it seems useful to look at them together.

Background and motivation

Python iterables often hold resources which require cleanup. For
example: file objects need to be closed; the WSGI spec
<333> adds a close method on top of the regular iterator protocol and
demands that consumers call it at the appropriate time (though
forgetting to do so is a frequent source of bugs); and PEP 342 (based on
PEP 325) extended generator objects to add a close method to allow
generators to clean up after themselves.

Generally, objects that need to clean up after themselves also define a
__del__ method to ensure that this cleanup will happen eventually, when
the object is garbage collected. However, relying on the garbage
collector for cleanup like this causes serious problems in several
cases:

-   In Python implementations that do not use reference counting (e.g.
    PyPy, Jython), calls to __del__ may be arbitrarily delayed -- yet
    many situations require prompt cleanup of resources. Delayed cleanup
    produces problems like crashes due to file descriptor exhaustion, or
    WSGI timing middleware that collects bogus times.
-   Async generators (PEP 525) can only perform cleanup under the
    supervision of the appropriate coroutine runner. __del__ doesn't
    have access to the coroutine runner; indeed, the coroutine runner
    might be garbage collected before the generator object. So relying
    on the garbage collector is effectively impossible without some kind
    of language extension. (PEP 525 does provide such an extension, but
    it has a number of limitations that this proposal fixes; see the
    "alternatives" section below for discussion.)

  -   Causality preservation, context preservation
  -   Exception swallowing

Fortunately, Python provides a standard tool for doing resource cleanup
in a more structured way: with blocks. For example, this code opens a
file but relies on the garbage collector to close it:

    def read_newline_separated_json(path):
        for line in open(path):
            yield json.loads(line)

    for document in read_newline_separated_json(path):
        ...

and recent versions of CPython will point this out by issuing a
ResourceWarning, nudging us to fix it by adding a with block:

    def read_newline_separated_json(path):
        with open(path) as file_handle:      # <-- with block
            for line in file_handle:
                yield json.loads(line)

    for document in read_newline_separated_json(path):  # <-- outer for loop
        ...

But there's a subtlety here, caused by the interaction of with blocks
and generators. with blocks are Python's main tool for managing cleanup,
and they're a powerful one, because they pin the lifetime of a resource
to the lifetime of a stack frame. But this assumes that someone will
take care of cleaning up the stack frame... and for generators, this
requires that someone close them.

In this case, adding the with block is enough to shut up the
ResourceWarning, but this is misleading -- the file object cleanup here
is still dependent on the garbage collector. The with block will only be
unwound when the read_newline_separated_json generator is closed. If the
outer for loop runs to completion then the cleanup will happen
immediately; but if this loop is terminated early by a break or an
exception, then the with block won't fire until the generator object is
garbage collected.

The correct solution requires that all users of this API wrap every for
loop in its own with block:

    with closing(read_newline_separated_json(path)) as genobj:
        for document in genobj:
            ...

This gets even worse if we consider the idiom of decomposing a complex
pipeline into multiple nested generators:

    def read_users(path):
        with closing(read_newline_separated_json(path)) as gen:
            for document in gen:
                yield User.from_json(document)

    def users_in_group(path, group):
        with closing(read_users(path)) as gen:
            for user in gen:
                if user.group == group:
                    yield user

In general if you have N nested generators then you need N+1 with blocks
to clean up 1 file. And good defensive programming would suggest that
any time we use a generator, we should assume the possibility that there
could be at least one with block somewhere in its (potentially
transitive) call stack, either now or in the future, and thus always
wrap it in a with. But in practice, basically nobody does this, because
programmers would rather write buggy code than tiresome repetitive code.
In simple cases like this there are some workarounds that good Python
developers know (e.g. in this simple case it would be idiomatic to pass
in a file handle instead of a path and move the resource management to
the top level), but in general we cannot avoid the use of with/finally
inside of generators, and thus dealing with this problem one way or
another. When beauty and correctness fight then beauty tends to win, so
it's important to make correct code beautiful.

Still, is this worth fixing? Until async generators came along I would
have argued yes, but that it was a low priority, since everyone seems to
be muddling along okay -- but async generators make it much more urgent.
Async generators cannot do cleanup at all without some mechanism for
deterministic cleanup that people will actually use, and async
generators are particularly likely to hold resources like file
descriptors. (After all, if they weren't doing I/O, they'd be
generators, not async generators.) So we have to do something, and it
might as well be a comprehensive fix to the underlying problem. And it's
much easier to fix this now when async generators are first rolling out,
than it will be to fix it later.

The proposal itself is simple in concept: add a __(a)iterclose__ method
to the iterator protocol, and have (async) for loops call it when the
loop is exited, even if this occurs via break or exception unwinding.
Effectively, we're taking the current cumbersome idiom (with block + for
loop) and merging them together into a fancier for. This may seem
non-orthogonal, but makes sense when you consider that the existence of
generators means that with blocks actually depend on iterator cleanup to
work reliably, plus experience showing that iterator cleanup is often a
desirable feature in its own right.

Alternatives

PEP 525 asyncgen hooks

PEP 525 proposes a set of global thread-local hooks
<525#finalization> managed by new sys.{get/set}_asyncgen_hooks()
functions, which allow event loops to integrate with the garbage
collector to run cleanup for async generators. In principle, this
proposal and PEP 525 are complementary, in the same way that with blocks
and __del__ are complementary: this proposal takes care of ensuring
deterministic cleanup in most cases, while PEP 525's GC hooks clean up
anything that gets missed. But __aiterclose__ provides a number of
advantages over GC hooks alone:

-   The GC hook semantics aren't part of the abstract async iterator
    protocol, but are instead restricted specifically to the async
    generator concrete type. If you have an async iterator implemented
    using a class, like:

        class MyAsyncIterator:
            async def __anext__():
                ...

    then you can't refactor this into an async generator without
    changing its semantics, and vice-versa. This seems very unpythonic.
    (It also leaves open the question of what exactly class-based async
    iterators are supposed to do, given that they face exactly the same
    cleanup problems as async generators.) __aiterclose__, on the other
    hand, is defined at the protocol level, so it's duck-type friendly
    and works for all iterators, not just generators.

-   Code that wants to work on non-CPython implementations like PyPy
    cannot in general rely on GC for cleanup. Without __aiterclose__,
    it's more or less guaranteed that developers who develop and test on
    CPython will produce libraries that leak resources when used on
    PyPy. Developers who do want to target alternative implementations
    will either have to take the defensive approach of wrapping every
    for loop in a with block, or else carefully audit their code to
    figure out which generators might possibly contain cleanup code and
    add with blocks around those only. With __aiterclose__, writing
    portable code becomes easy and natural.

-   An important part of building robust software is making sure that
    exceptions always propagate correctly without being lost. One of the
    most exciting things about async/await compared to traditional
    callback-based systems is that instead of requiring manual chaining,
    the runtime can now do the heavy lifting of propagating errors,
    making it much easier to write robust code. But, this beautiful new
    picture has one major gap: if we rely on the GC for generator
    cleanup, then exceptions raised during cleanup are lost. So, again,
    with __aiterclose__, developers who care about this kind of
    robustness will either have to take the defensive approach of
    wrapping every for loop in a with block, or else carefully audit
    their code to figure out which generators might possibly contain
    cleanup code. __aiterclose__ plugs this hole by performing cleanup
    in the caller's context, so writing more robust code becomes the
    path of least resistance.

-   The WSGI experience suggests that there exist important
    iterator-based APIs that need prompt cleanup and cannot rely on the
    GC, even in CPython. For example, consider a hypothetical WSGI-like
    API based around async/await and async iterators, where a response
    handler is an async generator that takes request headers + an async
    iterator over the request body, and yields response headers + the
    response body. (This is actually the use case that got me interested
    in async generators in the first place, i.e. this isn't
    hypothetical.) If we follow WSGI in requiring that child iterators
    must be closed properly, then without __aiterclose__ the absolute
    most minimalistic middleware in our system looks something like:

        async def noop_middleware(handler, request_header, request_body):
            async with aclosing(handler(request_body, request_body)) as aiter:
                async for response_item in aiter:
                    yield response_item

    Arguably in regular code one can get away with skipping the with
    block around for loops, depending on how confident one is that one
    understands the internal implementation of the generator. But here
    we have to cope with arbitrary response handlers, so without
    __aiterclose__, this with construction is a mandatory part of every
    middleware.

    __aiterclose__ allows us to eliminate the mandatory boilerplate and
    an extra level of indentation from every middleware:

        async def noop_middleware(handler, request_header, request_body):
            async for response_item in handler(request_header, request_body):
                yield response_item

So the __aiterclose__ approach provides substantial advantages over GC
hooks.

This leaves open the question of whether we want a combination of GC
hooks + __aiterclose__, or just __aiterclose__ alone. Since the vast
majority of generators are iterated over using a for loop or equivalent,
__aiterclose__ handles most situations before the GC has a chance to get
involved. The case where GC hooks provide additional value is in code
that does manual iteration, e.g.:

    agen = fetch_newline_separated_json_from_url(...)
    while True:
        document = await type(agen).__anext__(agen)
        if document["id"] == needle:
            break
    # doesn't do 'await agen.aclose()'

If we go with the GC-hooks + __aiterclose__ approach, this generator
will eventually be cleaned up by GC calling the generator __del__
method, which then will use the hooks to call back into the event loop
to run the cleanup code.

If we go with the no-GC-hooks approach, this generator will eventually
be garbage collected, with the following effects:

-   its __del__ method will issue a warning that the generator was not
    closed (similar to the existing "coroutine never awaited" warning).
-   The underlying resources involved will still be cleaned up, because
    the generator frame will still be garbage collected, causing it to
    drop references to any file handles or sockets it holds, and then
    those objects's __del__ methods will release the actual operating
    system resources.
-   But, any cleanup code inside the generator itself (e.g. logging,
    buffer flushing) will not get a chance to run.

The solution here -- as the warning would indicate -- is to fix the code
so that it calls __aiterclose__, e.g. by using a with block:

    async with aclosing(fetch_newline_separated_json_from_url(...)) as agen:
        while True:
            document = await type(agen).__anext__(agen)
            if document["id"] == needle:
                break

Basically in this approach, the rule would be that if you want to
manually implement the iterator protocol, then it's your responsibility
to implement all of it, and that now includes __(a)iterclose__.

GC hooks add non-trivial complexity in the form of (a) new global
interpreter state, (b) a somewhat complicated control flow (e.g., async
generator GC always involves resurrection, so the details of PEP 442 are
important), and (c) a new public API in asyncio
(await loop.shutdown_asyncgens()) that users have to remember to call at
the appropriate time. (This last point in particular somewhat undermines
the argument that GC hooks provide a safe backup to guarantee cleanup,
since if shutdown_asyncgens() isn't called correctly then I think it's
possible for generators to be silently discarded without their cleanup
code being called; compare this to the __aiterclose__-only approach
where in the worst case we still at least get a warning printed. This
might be fixable.) All this considered, GC hooks arguably aren't worth
it, given that the only people they help are those who want to manually
call __anext__ yet don't want to manually call __aiterclose__. But Yury
disagrees with me on this :-). And both options are viable.

Always inject resources, and do all cleanup at the top level

Several commentators on python-dev and python-ideas have suggested that
a pattern to avoid these problems is to always pass resources in from
above, e.g. read_newline_separated_json should take a file object rather
than a path, with cleanup handled at the top level:

    def read_newline_separated_json(file_handle):
        for line in file_handle:
            yield json.loads(line)

    def read_users(file_handle):
        for document in read_newline_separated_json(file_handle):
            yield User.from_json(document)

    with open(path) as file_handle:
        for user in read_users(file_handle):
            ...

This works well in simple cases; here it lets us avoid the "N+1 with
blocks problem". But unfortunately, it breaks down quickly when things
get more complex. Consider if instead of reading from a file, our
generator was reading from a streaming HTTP GET request --while handling
redirects and authentication via OAUTH. Then we'd really want the
sockets to be managed down inside our HTTP client library, not at the
top level. Plus there are other cases where finally blocks embedded
inside generators are important in their own right: db transaction
management, emitting logging information during cleanup (one of the
major motivating use cases for WSGI close), and so forth. So this is
really a workaround for simple cases, not a general solution.

More complex variants of __(a)iterclose

The semantics of __(a)iterclose__ are somewhat inspired by with blocks,
but context managers are more powerful: __(a)exit__ can distinguish
between a normal exit versus exception unwinding, and in the case of an
exception it can examine the exception details and optionally suppress
propagation. __(a)iterclose__ as proposed here does not have these
powers, but one can imagine an alternative design where it did.

However, this seems like unwarranted complexity: experience suggests
that it's common for iterables to have close methods, and even to have
__exit__ methods that call self.close(), but I'm not aware of any common
cases that make use of __exit__'s full power. I also can't think of any
examples where this would be useful. And it seems unnecessarily
confusing to allow iterators to affect flow control by swallowing
exceptions -- if you're in a situation where you really want that, then
you should probably use a real with block anyway.

Specification

This section describes where we want to eventually end up, though there
are some backwards compatibility issues that mean we can't jump directly
here. A later section describes the transition plan.

Guiding principles

Generally, __(a)iterclose__ implementations should:

-   be idempotent,
-   perform any cleanup that is appropriate on the assumption that the
    iterator will not be used again after __(a)iterclose__ is called. In
    particular, once __(a)iterclose__ has been called then calling
    __(a)next__ produces undefined behavior.

And generally, any code which starts iterating through an iterable with
the intention of exhausting it, should arrange to make sure that
__(a)iterclose__ is eventually called, whether or not the iterator is
actually exhausted.

Changes to iteration

The core proposal is the change in behavior of for loops. Given this
Python code:

    for VAR in ITERABLE:
        LOOP-BODY
    else:
        ELSE-BODY

we desugar to the equivalent of:

    _iter = iter(ITERABLE)
    _iterclose = getattr(type(_iter), "__iterclose__", lambda: None)
    try:
        traditional-for VAR in _iter:
            LOOP-BODY
        else:
            ELSE-BODY
    finally:
        _iterclose(_iter)

where the "traditional-for statement" here is meant as a shorthand for
the classic 3.5-and-earlier for loop semantics.

Besides the top-level for statement, Python also contains several other
places where iterators are consumed. For consistency, these should call
__iterclose__ as well using semantics equivalent to the above. This
includes:

-   for loops inside comprehensions
-   * unpacking
-   functions which accept and fully consume iterables, like list(it),
    tuple(it), itertools.product(it1, it2, ...), and others.

In addition, a yield from that successfully exhausts the called
generator should as a last step call its __iterclose__ method.
(Rationale: yield from already links the lifetime of the calling
generator to the called generator; if the calling generator is closed
when half-way through a yield from, then this will already automatically
close the called generator.)

Changes to async iteration

We also make the analogous changes to async iteration constructs, except
that the new slot is called __aiterclose__, and it's an async method
that gets awaited.

Modifications to basic iterator types

Generator objects (including those created by generator comprehensions):

-   __iterclose__ calls self.close()
-   __del__ calls self.close() (same as now), and additionally issues a
    ResourceWarning if the generator wasn't exhausted. This warning is
    hidden by default, but can be enabled for those who want to make
    sure they aren't inadvertently relying on CPython-specific GC
    semantics.

Async generator objects (including those created by async generator
comprehensions):

-   __aiterclose__ calls self.aclose()
-   __del__ issues a RuntimeWarning if aclose has not been called, since
    this probably indicates a latent bug, similar to the "coroutine
    never awaited" warning.

QUESTION: should file objects implement __iterclose__ to close the file?
On the one hand this would make this change more disruptive; on the
other hand people really like writing for line in open(...): ..., and if
we get used to iterators taking care of their own cleanup then it might
become very weird if files don't.

New convenience functions

The operator module gains two new functions, with semantics equivalent
to the following:

    def iterclose(it):
        if not isinstance(it, collections.abc.Iterator):
            raise TypeError("not an iterator")
        if hasattr(type(it), "__iterclose__"):
            type(it).__iterclose__(it)

    async def aiterclose(ait):
        if not isinstance(it, collections.abc.AsyncIterator):
            raise TypeError("not an iterator")
        if hasattr(type(ait), "__aiterclose__"):
            await type(ait).__aiterclose__(ait)

The itertools module gains a new iterator wrapper that can be used to
selectively disable the new __iterclose__ behavior:

    # QUESTION: I feel like there might be a better name for this one?
    class preserve(iterable):
        def __init__(self, iterable):
            self._it = iter(iterable)

        def __iter__(self):
            return self

        def __next__(self):
            return next(self._it)

        def __iterclose__(self):
            # Swallow __iterclose__ without passing it on
            pass

Example usage (assuming that file objects implements __iterclose__):

    with open(...) as handle:
        # Iterate through the same file twice:
        for line in itertools.preserve(handle):
            ...
        handle.seek(0)
        for line in itertools.preserve(handle):
            ...

    @contextlib.contextmanager
    def iterclosing(iterable):
        it = iter(iterable)
        try:
            yield preserve(it)
        finally:
            iterclose(it)

------------------------------------------------------------------------

Python ships a number of iterator types that act as wrappers around
other iterators: map, zip, itertools.accumulate, csv.reader, and others.
These iterators should define a __iterclose__ method which calls
__iterclose__ in turn on their underlying iterators. For example, map
could be implemented as:

    # Helper function
    map_chaining_exceptions(fn, items, last_exc=None):
        for item in items:
            try:
                fn(item)
            except BaseException as new_exc:
                if new_exc.__context__ is None:
                    new_exc.__context__ = last_exc
                last_exc = new_exc
        if last_exc is not None:
            raise last_exc

    class map:
        def __init__(self, fn, *iterables):
            self._fn = fn
            self._iters = [iter(iterable) for iterable in iterables]

        def __iter__(self):
            return self

        def __next__(self):
            return self._fn(*[next(it) for it in self._iters])

        def __iterclose__(self):
            map_chaining_exceptions(operator.iterclose, self._iters)

    def chain(*iterables):
        try:
            while iterables:
                for element in iterables.pop(0):
                    yield element
        except BaseException as e:
            def iterclose_iterable(iterable):
                operations.iterclose(iter(iterable))
            map_chaining_exceptions(iterclose_iterable, iterables, last_exc=e)

In some cases this requires some subtlety; for example, itertools.tee
should not call __iterclose__ on the underlying iterator until it has
been called on all of the clone iterators.

Example / Rationale

The payoff for all this is that we can now write straightforward code
like:

    def read_newline_separated_json(path):
        for line in open(path):
            yield json.loads(line)

and be confident that the file will receive deterministic cleanup
without the end-user having to take any special effort, even in complex
cases. For example, consider this silly pipeline:

    list(map(lambda key: key.upper(),
             doc["key"] for doc in read_newline_separated_json(path)))

If our file contains a document where doc["key"] turns out to be an
integer, then the following sequence of events will happen:

1.  key.upper() raises an AttributeError, which propagates out of the
    map and triggers the implicit finally block inside list.
2.  The finally block in list calls __iterclose__() on the map object.
3.  map.__iterclose__() calls __iterclose__() on the generator
    comprehension object.
4.  This injects a GeneratorExit exception into the generator
    comprehension body, which is currently suspended inside the
    comprehension's for loop body.
5.  The exception propagates out of the for loop, triggering the for
    loop's implicit finally block, which calls __iterclose__ on the
    generator object representing the call to
    read_newline_separated_json.
6.  This injects an inner GeneratorExit exception into the body of
    read_newline_separated_json, currently suspended at the yield.
7.  The inner GeneratorExit propagates out of the for loop, triggering
    the for loop's implicit finally block, which calls __iterclose__()
    on the file object.
8.  The file object is closed.
9.  The inner GeneratorExit resumes propagating, hits the boundary of
    the generator function, and causes read_newline_separated_json's
    __iterclose__() method to return successfully.
10. Control returns to the generator comprehension body, and the outer
    GeneratorExit continues propagating, allowing the comprehension's
    __iterclose__() to return successfully.
11. The rest of the __iterclose__() calls unwind without incident, back
    into the body of list.
12. The original AttributeError resumes propagating.

(The details above assume that we implement file.__iterclose__; if not
then add a with block to read_newline_separated_json and essentially the
same logic goes through.)

Of course, from the user's point of view, this can be simplified down to
just:

1.  int.upper() raises an AttributeError
2.  The file object is closed.
3.  The AttributeError propagates out of list

So we've accomplished our goal of making this "just work" without the
user having to think about it.

Transition plan

While the majority of existing for loops will continue to produce
identical results, the proposed changes will produce
backwards-incompatible behavior in some cases. Example:

    def read_csv_with_header(lines_iterable):
        lines_iterator = iter(lines_iterable)
        for line in lines_iterator:
            column_names = line.strip().split("\t")
            break
        for line in lines_iterator:
            values = line.strip().split("\t")
            record = dict(zip(column_names, values))
            yield record

This code used to be correct, but after this proposal is implemented
will require an itertools.preserve call added to the first for loop.

[QUESTION: currently, if you close a generator and then try to iterate
over it then it just raises Stop(Async)Iteration, so code the passes the
same generator object to multiple for loops but forgets to use
itertools.preserve won't see an obvious error -- the second for loop
will just exit immediately. Perhaps it would be better if iterating a
closed generator raised a RuntimeError? Note that files don't have this
problem -- attempting to iterate a closed file object already raises
ValueError.]

Specifically, the incompatibility happens when all of these factors come
together:

-   The automatic calling of __(a)iterclose__ is enabled
-   The iterable did not previously define __(a)iterclose__
-   The iterable does now define __(a)iterclose__
-   The iterable is re-used after the for loop exits

So the problem is how to manage this transition, and those are the
levers we have to work with.

First, observe that the only async iterables where we propose to add
__aiterclose__ are async generators, and there is currently no existing
code using async generators (though this will start changing very soon),
so the async changes do not produce any backwards incompatibilities.
(There is existing code using async iterators, but using the new async
for loop on an old async iterator is harmless, because old async
iterators don't have __aiterclose__.) In addition, PEP 525 was accepted
on a provisional basis, and async generators are by far the biggest
beneficiary of this PEP's proposed changes. Therefore, I think we should
strongly consider enabling __aiterclose__ for async for loops and async
generators ASAP, ideally for 3.6.0 or 3.6.1.

For the non-async world, things are harder, but here's a potential
transition path:

In 3.7:

Our goal is that existing unsafe code will start emitting warnings,
while those who want to opt-in to the future can do that immediately:

-   We immediately add all the __iterclose__ methods described above.
-   If from __future__ import iterclose is in effect, then for loops
    and * unpacking call __iterclose__ as specified above.
-   If the future is not enabled, then for loops and * unpacking do not
    call __iterclose__. But they do call some other method instead, e.g.
    __iterclose_warning__.
-   Similarly, functions like list use stack introspection (!!) to check
    whether their direct caller has __future__.iterclose enabled, and
    use this to decide whether to call __iterclose__ or
    __iterclose_warning__.
-   For all the wrapper iterators, we also add __iterclose_warning__
    methods that forward to the __iterclose_warning__ method of the
    underlying iterator or iterators.
-   For generators (and files, if we decide to do that),
    __iterclose_warning__ is defined to set an internal flag, and other
    methods on the object are modified to check for this flag. If they
    find the flag set, they issue a PendingDeprecationWarning to inform
    the user that in the future this sequence would have led to a
    use-after-close situation and the user should use preserve().

In 3.8:

-   Switch from PendingDeprecationWarning to DeprecationWarning

In 3.9:

-   Enable the __future__ unconditionally and remove all the
    __iterclose_warning__ stuff.

I believe that this satisfies the normal requirements for this kind of
transition -- opt-in initially, with warnings targeted precisely to the
cases that will be effected, and a long deprecation cycle.

Probably the most controversial / risky part of this is the use of stack
introspection to make the iterable-consuming functions sensitive to a
__future__ setting, though I haven't thought of any situation where it
would actually go wrong yet...

Acknowledgements

Thanks to Yury Selivanov, Armin Rigo, and Carl Friedrich Bolz for
helpful discussion on earlier versions of this idea.

Copyright

This document has been placed in the public domain.


text:
PEP: 650 Title: Specifying Installer Requirements for Python Projects
Author: Vikram Jayanthi<EMAIL_ADDRESS>Dustin Ingram
<EMAIL_ADDRESS>Brett Cannon<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657
Status: Withdrawn Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 16-Jul-2020 Post-History: 14-Jan-2021

Abstract

Python package installers are not completely interoperable with each
other. While pip is the most widely used installer and a de facto
standard, other installers such as Poetry or Pipenv are popular as well
due to offering unique features which are optimal for certain workflows
and not directly in line with how pip operates.

While the abundance of installer options is good for end-users with
specific needs, the lack of interoperability between them makes it hard
to support all potential installers. Specifically, the lack of a
standard requirements file for declaring dependencies means that each
tool must be explicitly used in order to install dependencies specified
with their respective format. Otherwise tools must emit a requirements
file which leads to potential information loss for the installer as well
as an added export step as part of a developer's workflow.

By providing a standardized API that can be used to invoke a compatible
installer, we can solve this problem without needing to resolve
individual concerns, unique requirements, and incompatibilities between
different installers and their lock files.

Installers that implement the specification can be invoked in a uniform
way, allowing users to use their installer of choice as if they were
invoking it directly.

Terminology

Installer interface

    The interface by which an installer backend and a universal
    installer interact.

Universal installer

    An installer that can invoke an installer backend by calling the
    optional invocation methods of the installer interface. This can
    also be thought of as the installer frontend, à la the build project
    for PEP 517.

Installer backend

    An installer that implements the installer interface, allowing it to
    be invoked by a universal installer. An installer backend may also
    be a universal installer as well, but it is not required. In
    comparison to PEP 517, this would be Flit. Installer backends may be
    wrapper packages around a backing installer, e.g. Poetry could
    choose to not support this API, but a package could act as a wrapper
    to invoke Poetry as appropriate to use Poetry to perform an
    installation.

Dependency group

    A set of dependencies that are related and required to be installed
    simultaneously for some purpose. For example, a "test" dependency
    group could include the dependencies required to run the test suite.
    How dependency groups are specified is up to the installer backend.

Motivation

This specification allows anyone to invoke and interact with installer
backends that implement the specified interface, allowing for a
universally supported layer on top of existing tool-specific
installation processes.

This in turn would enable the use of all installers that implement the
specified interface to be used in environments that support a single
universal installer, as long as that installer implements this
specification as well.

Below, we identify various use-cases applicable to stakeholders in the
Python community and anyone who interacts with Python package
installers. For developers or companies, this PEP would allow for
increased functionality and flexibility with Python package installers.

Providers

Providers are the parties (organization, person, community, etc.) that
supply a service or software tool which interacts with Python packaging
and consequently Python package installers. Two different types of
providers are considered:

Platform/Infrastructure Providers

Platform providers (cloud environments, application hosting, etc.) and
infrastructure service providers need to support package installers for
their users to install Python dependencies. Most only support pip,
however there is user demand for other Python installers. Most providers
do not want to maintain support for more than one installer because of
the complexity it adds to their software or service and the resources it
takes to do so.

Via this specification, we can enable a provider-supported universal
installer to invoke the user-desired installer backend without the
provider’s platform needing to have specific knowledge of said backend.
What this means is if Poetry implemented the installer backend API
proposed by this PEP (or some other package wrapped Poetry to provide
the API), then platform providers would support Poetry implicitly.

IDE Providers

Integrated development environments may interact with Python package
installation and management. Most only support pip as a Python package
installer, and users are required to find work arounds to install their
dependencies using other package installers. Similar to the situation
with PaaS & IaaS providers, IDE providers do not want to maintain
support for N different Python installers. Instead, implementers of the
installer interface (installer backends) could be invoked by the IDE by
it acting as a universal installer.

Developers

Developers are teams, people, or communities that code and use Python
package installers and Python packages. Three different types of
developers are considered:

Developers using PaaS & IaaS providers

Most PaaS and IaaS providers only support one Python package installer:
pip. (Some exceptions include Heroku's Python buildpack, which supports
pip and Pipenv). This dictates the installers that developers can use
while working with these providers, which might not be optimal for their
application or workflow.

Installers adopting this PEP to become installer backends would allow
users to use third party platforms/infrastructure without having to
worry about which Python package installer they are required to use as
long as the provider uses a universal installer.

Developers using IDEs

Most IDEs only support pip or a few Python package installers.
Consequently, developers must use workarounds or hacky methods to
install their dependencies if they use an unsupported package installer.

If the IDE uses/provides a universal installer it would allow for any
installer backend that the developer wanted to be used to install
dependencies, freeing them of any extra work to install their
dependencies in order to integrate into the IDE's workflow more closely.

Developers working with other developers

Developers want to be able to use the installer of their choice while
working with other developers, but currently have to synchronize their
installer choice for compatibility of dependency installation. If all
preferred installers instead implemented the specified interface, it
would allow for cross use of installers, allowing developers to choose
an installer regardless of their collaborator’s preference.

Upgraders & Package Infrastructure Providers

Package upgraders and package infrastructure in CI/CD such as
Dependabot, PyUP, etc. currently support a few installers. They work by
parsing and editing the installer-specific dependency files directly
(such as requirements.txt or poetry.lock) with relevant package
information such as upgrades, downgrades, or new hashes. Similar to
Platform and IDE providers, most of these providers do not want to
support N different Python package installers as that would require
supporting N different file types.

Currently, these services/bots have to implement support for each
package installer individually. Inevitably, the most popular installers
are supported first, and less popular tools are often never supported.
By implementing this specification, these services/bots can support any
(compliant) installer, allowing users to select the tool of their
choice. This will allow for more innovation in the space, as platforms
and IDEs are no longer forced to prematurely select a "winner".

Open Source Community

Specifying installer requirements and adopting this PEP will reduce the
friction between Python package installers and people's workflows.
Consequently, it will reduce the friction between Python package
installers and 3rd party infrastructure/technologies such as PaaS or
IDEs. Overall, it will allow for easier development, deployment and
maintenance of Python projects as Python package installation becomes
simpler and more interoperable.

Specifying requirements and creating an interface for installers can
also increase the pace of innovation around installers. This would allow
for installers to experiment and add unique functionality without
requiring the rest of the ecosystem to do the same. Support becomes
easier and more likely for a new installer regardless of the
functionality it adds and the format in which it writes dependencies,
while reducing the developer time and resources needed to do so.

Specification

Similar to how PEP 517 specifies build systems, the install system
information will live in the pyproject.toml file under the
install-system table.

[install-system]

The install-system table is used to store install-system relevant data
and information. There are multiple required keys for this table:
requires and install-backend. The requires key holds the minimum
requirements for the installer backend to execute and which will be
installed by the universal installer. The install-backend key holds the
name of the install backend’s entry point. This will allow the universal
installer to install the requirements for the installer backend itself
to execute (not the requirements that the installer backend itself will
install) as well as invoke the installer backend.

If either of the required keys are missing or empty then the universal
installer SHOULD raise an error.

All package names interacting with this interface are assumed to follow
PEP 508's "Dependency specification for Python Software Packages"
format.

An example install-system table:

    #pyproject.toml
    [install-system]
    #Eg : pipenv
    requires = ["pipenv"]
    install-backend = "pipenv.api:main"

Installer Requirements:

The requirements specified by the requires key must be within the
constraints specified by PEP 517. Specifically, that dependency cycles
are not permitted and the universal installer SHOULD refuse to install
the dependencies if a cycle is detected.

Additional parameters or tool specific data

Additional parameters or tool (installer backend) data may also be
stored in the pyproject.toml file. This would be in the “tool.*” table
as specified by PEP 518. For example, if the installer backend is Poetry
and you wanted to specify multiple dependency groups, the tool.poetry
tables could look like this:

    [tool.poetry.dev-dependencies]
    dependencies = "dev"

    [tool.poetry.deploy]
    dependencies = "deploy"

Data may also be stored in other ways as the installer backend sees fit
(e.g. separate configuration file).

Installer interface:

The installer interface contains mandatory and optional hooks. Compliant
installer backends MUST implement the mandatory hooks and MAY implement
the optional hooks. A universal installer MAY implement any of the
installer backend hooks itself, to act as both a universal installer and
installer backend, but this is not required.

All hooks take **kwargs arbitrary parameters that a installer backend
may require that are not already specified, allowing for backwards
compatibility. If unexpected parameters are passed to the installer
backend, it should ignore them.

The following information is akin to the corresponding section in PEP
517. The hooks may be called with keyword arguments, so installer
backends implementing them should be careful to make sure that their
signatures match both the order and the names of the arguments above.

All hooks MAY print arbitrary informational text to stdout and stderr.
They MUST NOT read from stdin, and the universal installer MAY close
stdin before invoking the hooks.

The universal installer may capture stdout and/or stderr from the
backend. If the backend detects that an output stream is not a
terminal/console (e.g. not sys.stdout.isatty()), it SHOULD ensure that
any output it writes to that stream is UTF-8 encoded. The universal
installer MUST NOT fail if captured output is not valid UTF-8, but it
MAY not preserve all the information in that case (e.g. it may decode
using the replace error handler in Python). If the output stream is a
terminal, the installer backend is responsible for presenting its output
accurately, as for any program running in a terminal.

If a hook raises an exception, or causes the process to terminate, then
this indicates an error.

Mandatory hooks:

invoke_install

Installs the dependencies:

    def invoke_install(
        path: Union[str, bytes, PathLike[str]],
        *,
        dependency_group: str = None,
        **kwargs
    ) -> int:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_group : An optional flag specifying a dependency group
    that the installer backend should install. The install will error if
    the dependency group doesn't exist. A user can find all dependency
    groups by calling get_dependency_groups() if dependency groups are
    supported by the installer backend.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns : An exit code (int). 0 if successful, any positive integer
    if unsuccessful.

The universal installer will use the exit code to determine if the
installation is successful and SHOULD return the exit code itself.

Optional hooks:

invoke_uninstall

Uninstall the specified dependencies:

    def invoke_uninstall(
        path: Union[str, bytes, PathLike[str]],
        *,
        dependency_group: str = None,
        **kwargs
    ) -> int:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_group : An optional flag specifying a dependency group
    that the installer backend should uninstall.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns : An exit code (int). 0 if successful, any positive integer
    if unsuccessful.

The universal installer MUST invoke the installer backend at the same
path that the universal installer itself was invoked.

The universal installer will use the exit code to determine if the
uninstall is successful and SHOULD return the exit code itself.

get_dependencies_to_install

Returns the dependencies that would be installed by invoke_install(...).
This allows package upgraders (e.g., Dependabot) to retrieve the
dependencies attempting to be installed without parsing the dependency
file:

    def get_dependencies_to_install(
        path: Union[str, bytes, PathLike[str]],
        *,
        dependency_group: str = None,
        **kwargs
    ) -> Sequence[str]:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_group : Specify a dependency group to get the
    dependencies invoke_install(...) would install for that dependency
    group.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns: A list of dependencies (PEP 508 strings) to install.

If the group is specified, the installer backend MUST return the
dependencies corresponding to the provided dependency group. If the
specified group doesn't exist, or dependency groups are not supported by
the installer backend, the installer backend MUST raise an error.

If the group is not specified, and the installer backend provides the
concept of a default/unspecified group, the installer backend MAY return
the dependencies for the default/unspecified group, but otherwise MUST
raise an error.

get_dependency_groups

Returns the dependency groups available to be installed. This allows
universal installers to enumerate all dependency groups the installer
backend is aware of:

    def get_dependency_groups(
        path: Union[str, bytes, PathLike[str]],
        **kwargs
    ) -> AbstractSet[str]:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns: A set of known dependency groups, as strings The empty set
    represents no dependency groups.

update_dependencies

Outputs a dependency file based on inputted package list:

    def update_dependencies(
        path: Union[str, bytes, PathLike[str]],
        dependency_specifiers: Iterable[str],
        *,
        dependency_group=None,
        **kwargs
    ) -> int:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_specifiers : An iterable of dependencies as PEP 508
    strings that are being updated, for example :
    ["requests==2.8.1", ...]. Optionally for a specific dependency
    group.
-   dependency_group : The dependency group that the list of packages is
    for.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns : An exit code (int). 0 if successful, any positive integer
    if unsuccessful.

Example

Let's consider implementing an installer backend that uses pip and its
requirements files for dependency groups. An implementation may (very
roughly) look like the following:

    import subprocess
    import sys

    def invoke_install(path, *, dependency_group=None, **kwargs):
        try:
            return subprocess.run(
                [
                    sys.executable,
                    "-m",
                    "pip",
                    "install",
                    "-r",
                    dependency_group or "requirements.txt",
                ],
                cwd=path,
            ).returncode
        except subprocess.CalledProcessError as e:
            return e.returncode

If we named this package pep650pip, then we could specify in
pyproject.toml:

    [install-system]
      #Eg : pipenv
      requires = ["pep650pip", "pip"]
      install-backend = "pep650pip:main"

Rationale

All hooks take **kwargs to allow for backwards compatibility and allow
for tool specific installer backend functionality which requires a user
to provide additional information not required by the hook.

While installer backends must be Python packages, what they do when
invoked is an implementation detail of that tool. For example, an
installer backend could act as a wrapper for a platform package manager
(e.g., apt).

The interface does not in any way try to specify how installer backends
should function. This is on purpose so that installer backends can be
allowed to innovate and solve problem in their own way. This also means
this PEP takes no stance on OS packaging as that would be an installer
backend's domain.

Defining the API in Python does mean that some Python code will
eventually need to be executed. That does not preclude non-Python
installer backends from being used, though (e.g. mamba), as they could
be executed as a subprocess from Python code.

Backwards Compatibility

This PEP would have no impact on pre-existing code and functionality as
it only adds new functionality to a universal installer. Any existing
installer should maintain its existing functionality and use cases,
therefore having no backwards compatibility issues. Only code aiming to
take advantage of this new functionality will have motivation to make
changes to their pre existing code.

Security Implications

A malicious user has no increased ability or easier access to anything
with the addition of standardized installer specifications. The
installer that could be invoked by a universal installer via the
interface specified in this PEP would be explicitly declared by the
user. If the user has chosen a malicious installer, then invoking it
with a universal installer is no different than the user invoking the
installer directly. A malicious installer being an installer backend
doesn't give it additional permissions or abilities.

Rejected Ideas

A standardized lock file

A standardized lock file would solve a lot of the same problems that
specifying installer requirements would. For example, it would allow for
PaaS/IaaS to just support one installer that could read the standardized
lock file regardless of the installer that created it. The problem with
a standardized lock file is the difference in needs between Python
package installers as well as a fundamental issue with creating
reproducible environments via the lockfile (one of the main benefits).

Needs and information stored in dependency files between installers
differ significantly and are dependent on installer functionality. For
example, a Python package installer such as Poetry requires information
for all Python versions and platforms and calculates appropriate hashes
while pip doesn't. Additionally, pip would not be able to guarantee
recreating the same environment (install the exact same dependencies) as
it is outside the scope of its functionality. This makes a standardized
lock file harder to implement and makes it seem more appropriate to make
lock files tool specific.

Have installer backends support creating virtual environments

Because installer backends will very likely have a concept of virtual
environments and how to install into them, it was briefly considered to
have them also support creating virtual environments. In the end,
though, it was considered an orthogonal idea.

Open Issues

Should the dependency_group argument take an iterable?

This would allow for specifying non-overlapping dependency groups in a
single call, e.g. "docs" and "test" groups which have independent
dependencies but which a developer may want to install simultaneously
while doing development.

Is the installer backend executed in-process?

If the installer backend is executed in-process then it greatly
simplifies knowing what environment to install for/into, as the live
Python environment can be queried for appropriate information.

Executing out-of-process allows for minimizing potential issues of
clashes between the environment being installed into and the installer
backend (and potentially universal installer).

Enforce that results from the proposed interface feed into other parts?

E.g. the results from get_dependencies_to_install() and
get_dependency_groups() can be passed into invoke_install(). This would
prevent drift between the results of various parts of the proposed
interface, but it makes more of the interface required instead of
optional.

Raising exceptions instead of exit codes for failure conditions

It has been suggested that instead of returning an exit code the API
should raise exceptions. If you view this PEP as helping to translate
current installers into installer backends, then relying on exit codes
makes sense. There's is also the point that the APIs have no specific
return value, so passing along an exit code does not interfere with what
the functions return.

Compare that to raising exceptions in case of an error. That could
potentially provide a more structured approach to error raising,
although to be able to capture errors it would require specifying
exception types as part of the interface.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

license: Public Domain


text:
PEP: 241 Title: Metadata for Python Software Packages Author: A.M.
Kuchling<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status:
Superseded Type: Standards Track Topic: Packaging Created: 12-Mar-2001
Post-History: 19-Mar-2001 Superseded-By: 314

packaging:core-metadata

Introduction

This PEP describes a mechanism for adding metadata to Python packages.
It includes specifics of the field names, and their semantics and usage.

Including Metadata in Packages

The Distutils 'sdist' command will be modified to extract the metadata
fields from the arguments and write them to a file in the generated
zipfile or tarball. This file will be named PKG-INFO and will be placed
in the top directory of the source distribution (where the README,
INSTALL, and other files usually go).

Developers may not provide their own PKG-INFO file. The "sdist" command
will, if it detects an existing PKG-INFO file, terminate with an
appropriate error message. This should prevent confusion caused by the
PKG-INFO and setup.py files being out of sync.

The PKG-INFO file format is a single set of 822 headers parseable by the
rfc822.py module. The field names listed in the following section are
used as the header names. There's no extension mechanism in this simple
format; the Catalog and Distutils SIGs will aim at getting a more
flexible format ready for Python 2.2.

Fields

This section specifies the names and semantics of each of the supported
metadata fields.

Fields marked with "(Multiple use)" may be specified multiple times in a
single PKG-INFO file. Other fields may only occur once in a PKG-INFO
file. Fields marked with "(optional)" are not required to appear in a
valid PKG-INFO file, all other fields must be present.

Metadata-Version

Version of the file format; currently "1.0" is the only legal value
here.

Example:

    Metadata-Version: 1.0

Name

The name of the package.

Example:

    Name: BeagleVote

Version

A string containing the package's version number. This field should be
parseable by one of the Version classes (StrictVersion or LooseVersion)
in the distutils.version module.

Example:

    Version: 1.0a2

Platform (multiple use)

A comma-separated list of platform specifications, summarizing the
operating systems supported by the package. The major supported
platforms are listed below, but this list is necessarily incomplete.

    POSIX, MacOS, Windows, BeOS, Palm OS.

Binary distributions will use the Supported-Platform field in their
metadata to specify the OS and CPU for which the binary package was
compiled. The semantics of the Supported-Platform are not specified in
this PEP.

Example:

    Platform: POSIX, Windows

Summary

A one-line summary of what the package does.

Example:

    Summary: A module for collecting votes from beagles.

Description (optional)

A longer description of the package that can run to several paragraphs.
(Software that deals with metadata should not assume any maximum size
for this field, though one hopes that people won't include their
instruction manual as the long-description.)

Example:

    Description: This module collects votes from beagles
                 in order to determine their electoral wishes.
                 Do NOT try to use this module with basset hounds;
                 it makes them grumpy.

Keywords (optional)

A list of additional keywords to be used to assist searching for the
package in a larger catalog.

Example:

    Keywords: dog puppy voting election

Home-page (optional)

A string containing the URL for the package's home page.

Example:

    Home-page: http://www.example.com/~cschultz/bvote/

Author (optional)

A string containing at a minimum the author's name. Contact information
can also be added, separating each line with newlines.

Example:

    Author: C. Schultz
            Universal Features Syndicate
            Los Angeles, CA

Author-email

A string containing the author's e-mail address. It can contain a name
and e-mail address in the legal forms for a 822 'From:' header. It's not
optional because cataloging systems can use the e-mail portion of this
field as a unique key representing the author. A catalog might provide
authors the ability to store their GPG key, personal home page, and
other additional metadata about the author, and optionally the ability
to associate several e-mail addresses with the same person.
Author-related metadata fields are not covered by this PEP.

Example:

    Author-email: "C. Schultz"<EMAIL_ADDRESS>
License

A string selected from a short list of choices, specifying the license
covering the package. Some licenses result in the software being freely
redistributable, so packagers and resellers can automatically know that
they're free to redistribute the software. Other licenses will require a
careful reading by a human to determine how the software can be
repackaged and resold.

The choices are:

    Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT",
    Mozilla PL, "public domain", Python, Qt PL, Zope PL, unknown,
    nocommercial, nosell, nosource, shareware, other

Definitions of some of the licenses are:

  --------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  DFSG            The license conforms to the Debian Free Software Guidelines, but does not use one of the other DFSG conforming licenses listed here. More information is available at: http://www.debian.org/social_contract#guidelines
  Python          Python 1.6 or higher license. Version 1.5.2 and earlier are under the MIT license.
  public domain   Software is public domain, not copyrighted.
  unknown         Status is not known
  nocommercial    Free private use but commercial use not permitted
  nosell          Free use but distribution for profit by arrangement
  nosource        Freely distributable but no source code
  shareware       Payment is requested if software is used
  other           General category for other non-DFSG licenses
  --------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Some of these licenses can be interpreted to mean the software is freely
redistributable. The list of redistributable licenses is:

    Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT",
    Mozilla PL, "public domain", Python, Qt PL, Zope PL,
    nosource, shareware

Note that being redistributable does not mean a package qualifies as
free software, 'nosource' and 'shareware' being examples.

Example:

    License: MIT

Acknowledgements

Many changes and rewrites to this document were suggested by the readers
of the Distutils SIG. In particular, Sean Reifschneider often
contributed actual text for inclusion in this PEP.

The list of licenses was compiled using the SourceForge license list and
the CTAN license list compiled by Graham Williams; Carey Evans also
offered several useful suggestions on this list.

Copyright

This document has been placed in the public domain.


text:
PEP: 290 Title: Code Migration and Modernization Version: $Revision$
Last-Modified: $Date$ Author: Raymond Hettinger<EMAIL_ADDRESS>Status:
Active Type: Informational Content-Type: text/x-rst Created: 06-Jun-2002
Post-History:

Abstract

This PEP is a collection of procedures and ideas for updating Python
applications when newer versions of Python are installed.

The migration tips highlight possible areas of incompatibility and make
suggestions on how to find and resolve those differences. The
modernization procedures show how older code can be updated to take
advantage of new language features.

Rationale

This repository of procedures serves as a catalog or checklist of known
migration issues and procedures for addressing those issues.

Migration issues can arise for several reasons. Some obsolete features
are slowly deprecated according to the guidelines in PEP 4. Also, some
code relies on undocumented behaviors which are subject to change
between versions. Some code may rely on behavior which was subsequently
shown to be a bug and that behavior changes when the bug is fixed.

Modernization options arise when new versions of Python add features
that allow improved clarity or higher performance than previously
available.

Guidelines for New Entries

Developers with commit access may update this PEP directly. Others can
send their ideas to a developer for possible inclusion.

While a consistent format makes the repository easier to use, feel free
to add or subtract sections to improve clarity.

Grep patterns may be supplied as tool to help maintainers locate code
for possible updates. However, fully automated search/replace style
regular expressions are not recommended. Instead, each code fragment
should be evaluated individually.

The contra-indications section is the most important part of a new
entry. It lists known situations where the update SHOULD NOT be applied.

Migration Issues

Comparison Operators Not a Shortcut for Producing 0 or 1

Prior to Python 2.3, comparison operations returned 0 or 1 rather than
True or False. Some code may have used this as a shortcut for producing
zero or one in places where their boolean counterparts are not
appropriate. For example:

    def identity(m=1):
        """Create and m-by-m identity matrix"""
        return [[i==j for i in range(m)] for j in range(m)]

In Python 2.2, a call to identity(2) would produce:

    [[1, 0], [0, 1]]

In Python 2.3, the same call would produce:

    [[True, False], [False, True]]

Since booleans are a subclass of integers, the matrix would continue to
calculate normally, but it will not print as expected. The list
comprehension should be changed to read:

    return [[int(i==j) for i in range(m)] for j in range(m)]

There are similar concerns when storing data to be used by other
applications which may expect a number instead of True or False.

Modernization Procedures

Procedures are grouped by the Python version required to be able to take
advantage of the modernization.

Python 2.4 or Later

Inserting and Popping at the Beginning of Lists

Python's lists are implemented to perform best with appends and pops on
the right. Use of pop(0) or insert(0, x) triggers O(n) data movement for
the entire list. To help address this need, Python 2.4 introduces a new
container, collections.deque() which has efficient append and pop
operations on the both the left and right (the trade-off is much slower
getitem/setitem access). The new container is especially helpful for
implementing data queues:

Pattern:

    c = list(data)   -->   c = collections.deque(data)
    c.pop(0)         -->   c.popleft()
    c.insert(0, x)   -->   c.appendleft()

Locating:

    grep pop(0 or
    grep insert(0

Simplifying Custom Sorts

In Python 2.4, the sort method for lists and the new sorted built-in
function both accept a key function for computing sort keys. Unlike the
cmp function which gets applied to every comparison, the key function
gets applied only once to each record. It is much faster than cmp and
typically more readable while using less code. The key function also
maintains the stability of the sort (records with the same key are left
in their original order.

Original code using a comparison function:

    names.sort(lambda x,y: cmp(x.lower(), y.lower()))

Alternative original code with explicit decoration:

    tempnames = [(n.lower(), n) for n in names]
    tempnames.sort()
    names = [original for decorated, original in tempnames]

Revised code using a key function:

    names.sort(key=str.lower)       # case-insensitive sort

Locating: grep sort *.py

Replacing Common Uses of Lambda

In Python 2.4, the operator module gained two new functions,
itemgetter() and attrgetter() that can replace common uses of the lambda
keyword. The new functions run faster and are considered by some to
improve readability.

Pattern:

    lambda r: r[2]      -->  itemgetter(2)
    lambda r: r.myattr  -->  attrgetter('myattr')

Typical contexts:

    sort(studentrecords, key=attrgetter('gpa'))   # set a sort field
    map(attrgetter('lastname'), studentrecords)   # extract a field

Locating: grep lambda *.py

Simplified Reverse Iteration

Python 2.4 introduced the reversed builtin function for reverse
iteration. The existing approaches to reverse iteration suffered from
wordiness, performance issues (speed and memory consumption), and/or
lack of clarity. A preferred style is to express the sequence in a
forwards direction, apply reversed to the result, and then loop over the
resulting fast, memory friendly iterator.

Original code expressed with half-open intervals:

    for i in range(n-1, -1, -1):
        print seqn[i]

Alternative original code reversed in multiple steps:

    rseqn = list(seqn)
    rseqn.reverse()
    for value in rseqn:
        print value

Alternative original code expressed with extending slicing:

    for value in seqn[::-1]:
        print value

Revised code using the reversed function:

    for value in reversed(seqn):
        print value

Python 2.3 or Later

Testing String Membership

In Python 2.3, for string2 in string1, the length restriction on string2
is lifted; it can now be a string of any length. When searching for a
substring, where you don't care about the position of the substring in
the original string, using the in operator makes the meaning clear.

Pattern:

    string1.find(string2) >= 0   -->  string2 in string1
    string1.find(string2) != -1  -->  string2 in string1

Replace apply() with a Direct Function Call

In Python 2.3, apply() was marked for Pending Deprecation because it was
made obsolete by Python 1.6's introduction of * and ** in function
calls. Using a direct function call was always a little faster than
apply() because it saved the lookup for the builtin. Now, apply() is
even slower due to its use of the warnings module.

Pattern:

    apply(f, args, kwds)  -->  f(*args, **kwds)

Note: The Pending Deprecation was removed from apply() in Python 2.3.3
since it creates pain for people who need to maintain code that works
with Python versions as far back as 1.5.2, where there was no
alternative to apply(). The function remains deprecated, however.

Python 2.2 or Later

Testing Dictionary Membership

For testing dictionary membership, use the 'in' keyword instead of the
'has_key()' method. The result is shorter and more readable. The style
becomes consistent with tests for membership in lists. The result is
slightly faster because has_key requires an attribute search and uses a
relatively expensive function call.

Pattern:

    if d.has_key(k):  -->  if k in d:

Contra-indications:

1.  Some dictionary-like objects may not define a __contains__() method:

        if dictlike.has_key(k)

Locating: grep has_key

Looping Over Dictionaries

Use the new iter methods for looping over dictionaries. The iter methods
are faster because they do not have to create a new list object with a
complete copy of all of the keys, values, or items. Selecting only keys,
values, or items (key/value pairs) as needed saves the time for creating
throwaway object references and, in the case of items, saves a second
hash look-up of the key.

Pattern:

    for key in d.keys():      -->  for key in d:
    for value in d.values():  -->  for value in d.itervalues():
    for key, value in d.items():
                              -->  for key, value in d.iteritems():

Contra-indications:

1.  If you need a list, do not change the return type:

        def getids():  return d.keys()

2.  Some dictionary-like objects may not define iter methods:

        for k in dictlike.keys():

3.  Iterators do not support slicing, sorting or other operations:

        k = d.keys(); j = k[:]

4.  Dictionary iterators prohibit modifying the dictionary:

        for k in d.keys(): del[k]

stat Methods

Replace stat constants or indices with new os.stat attributes and
methods. The os.stat attributes and methods are not order-dependent and
do not require an import of the stat module.

Pattern:

    os.stat("foo")[stat.ST_MTIME]  -->  os.stat("foo").st_mtime
    os.stat("foo")[stat.ST_MTIME]  -->  os.path.getmtime("foo")

Locating: grep os.stat or grep stat.S

Reduce Dependency on types Module

The types module is likely to be deprecated in the future. Use built-in
constructor functions instead. They may be slightly faster.

Pattern:

    isinstance(v, types.IntType)      -->  isinstance(v, int)
    isinstance(s, types.StringTypes)  -->  isinstance(s, basestring)

Full use of this technique requires Python 2.3 or later (basestring was
introduced in Python 2.3), but Python 2.2 is sufficient for most uses.

Locating: grep types *.py | grep import

Avoid Variable Names that Clash with the __builtins__ Module

In Python 2.2, new built-in types were added for dict and file. Scripts
should avoid assigning variable names that mask those types. The same
advice also applies to existing builtins like list.

Pattern:

    file = open('myfile.txt') --> f = open('myfile.txt')
    dict = obj.__dict__ --> d = obj.__dict__

Locating: grep 'file ' *.py

Python 2.1 or Later

whrandom Module Deprecated

All random-related methods have been collected in one place, the random
module.

Pattern:

    import whrandom --> import random

Locating: grep whrandom

Python 2.0 or Later

String Methods

The string module is likely to be deprecated in the future. Use string
methods instead. They're faster too.

Pattern:

    import string ; string.method(s, ...)  -->  s.method(...)
    c in string.whitespace                 -->  c.isspace()

Locating: grep string *.py | grep import

startswith and endswith String Methods

Use these string methods instead of slicing. No slice has to be created
and there's no risk of miscounting.

Pattern:

    "foobar"[:3] == "foo"   -->  "foobar".startswith("foo")
    "foobar"[-3:] == "bar"  -->  "foobar".endswith("bar")

The atexit Module

The atexit module supports multiple functions to be executed upon
program termination. Also, it supports parameterized functions.
Unfortunately, its implementation conflicts with the sys.exitfunc
attribute which only supports a single exit function. Code relying on
sys.exitfunc may interfere with other modules (including library
modules) that elect to use the newer and more versatile atexit module.

Pattern:

    sys.exitfunc = myfunc  -->  atexit.register(myfunc)

Python 1.5 or Later

Class-Based Exceptions

String exceptions are deprecated, so derive from the Exception base
class. Unlike the obsolete string exceptions, class exceptions all
derive from another exception or the Exception base class. This allows
meaningful groupings of exceptions. It also allows an "except Exception"
clause to catch all exceptions.

Pattern:

    NewError = 'NewError'  -->  class NewError(Exception): pass

Locating: Use PyChecker.

All Python Versions

Testing for None

Since there is only one None object, equality can be tested with
identity. Identity tests are slightly faster than equality tests. Also,
some object types may overload comparison, so equality testing may be
much slower.

Pattern:

    if v == None  -->  if v is None:
    if v != None  -->  if v is not None:

Locating: grep '== None' or grep '!= None'

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:


text:
PEP: 263 Title: Defining Python Source Code Encodings Author: Marc-André
Lemburg<EMAIL_ADDRESS>Martin von Löwis<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
06-Jun-2001 Python-Version: 2.3 Post-History:

Abstract

This PEP proposes to introduce a syntax to declare the encoding of a
Python source file. The encoding information is then used by the Python
parser to interpret the file using the given encoding. Most notably this
enhances the interpretation of Unicode literals in the source code and
makes it possible to write Unicode literals using e.g. UTF-8 directly in
an Unicode aware editor.

Problem

In Python 2.1, Unicode literals can only be written using the Latin-1
based encoding "unicode-escape". This makes the programming environment
rather unfriendly to Python users who live and work in non-Latin-1
locales such as many of the Asian countries. Programmers can write their
8-bit strings using the favorite encoding, but are bound to the
"unicode-escape" encoding for Unicode literals.

Proposed Solution

I propose to make the Python source code encoding both visible and
changeable on a per-source file basis by using a special comment at the
top of the file to declare the encoding.

To make Python aware of this encoding declaration a number of concept
changes are necessary with respect to the handling of Python source code
data.

Defining the Encoding

Python will default to ASCII as standard encoding if no other encoding
hints are given.

To define a source code encoding, a magic comment must be placed into
the source files either as first or second line in the file, such as:

    # coding=<encoding name>

or (using formats recognized by popular editors):

    #!/usr/bin/python
    # -*- coding: <encoding name> -*-

or:

    #!/usr/bin/python
    # vim: set fileencoding=<encoding name> :

More precisely, the first or second line must match the following
regular expression:

    ^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)

The first group of this expression is then interpreted as encoding name.
If the encoding is unknown to Python, an error is raised during
compilation. There must not be any Python statement on the line that
contains the encoding declaration. If the first line matches the second
line is ignored.

To aid with platforms such as Windows, which add Unicode BOM marks to
the beginning of Unicode files, the UTF-8 signature \xef\xbb\xbf will be
interpreted as 'utf-8' encoding as well (even if no magic encoding
comment is given).

If a source file uses both the UTF-8 BOM mark signature and a magic
encoding comment, the only allowed encoding for the comment is 'utf-8'.
Any other encoding will cause an error.

Examples

These are some examples to clarify the different styles for defining the
source code encoding at the top of a Python source file:

1.  With interpreter binary and using Emacs style file encoding comment:

        #!/usr/bin/python
        # -*- coding: latin-1 -*-
        import os, sys
        ...

        #!/usr/bin/python
        # -*- coding: iso-8859-15 -*-
        import os, sys
        ...

        #!/usr/bin/python
        # -*- coding: ascii -*-
        import os, sys
        ...

2.  Without interpreter line, using plain text:

        # This Python file uses the following encoding: utf-8
        import os, sys
        ...

3.  Text editors might have different ways of defining the file's
    encoding, e.g.:

        #!/usr/local/bin/python
        # coding: latin-1
        import os, sys
        ...

4.  Without encoding comment, Python's parser will assume ASCII text:

        #!/usr/local/bin/python
        import os, sys
        ...

5.  Encoding comments which don't work:

    1.  Missing "coding:" prefix:

            #!/usr/local/bin/python
            # latin-1
            import os, sys
            ...

    2.  Encoding comment not on line 1 or 2:

            #!/usr/local/bin/python
            #
            # -*- coding: latin-1 -*-
            import os, sys
            ...

    3.  Unsupported encoding:

            #!/usr/local/bin/python
            # -*- coding: utf-42 -*-
            import os, sys
            ...

Concepts

The PEP is based on the following concepts which would have to be
implemented to enable usage of such a magic comment:

1.  The complete Python source file should use a single encoding.
    Embedding of differently encoded data is not allowed and will result
    in a decoding error during compilation of the Python source code.

    Any encoding which allows processing the first two lines in the way
    indicated above is allowed as source code encoding, this includes
    ASCII compatible encodings as well as certain multi-byte encodings
    such as Shift_JIS. It does not include encodings which use two or
    more bytes for all characters like e.g. UTF-16. The reason for this
    is to keep the encoding detection algorithm in the tokenizer simple.

2.  Handling of escape sequences should continue to work as it does now,
    but with all possible source code encodings, that is standard string
    literals (both 8-bit and Unicode) are subject to escape sequence
    expansion while raw string literals only expand a very small subset
    of escape sequences.

3.  Python's tokenizer/compiler combo will need to be updated to work as
    follows:

    1.  read the file
    2.  decode it into Unicode assuming a fixed per-file encoding
    3.  convert it into a UTF-8 byte string
    4.  tokenize the UTF-8 content
    5.  compile it, creating Unicode objects from the given Unicode data
        and creating string objects from the Unicode literal data by
        first reencoding the UTF-8 data into 8-bit string data using the
        given file encoding

Note that Python identifiers are restricted to the ASCII subset of the
encoding, and thus need no further conversion after step 4.

Implementation

For backwards-compatibility with existing code which currently uses
non-ASCII in string literals without declaring an encoding, the
implementation will be introduced in two phases:

1.  Allow non-ASCII in string literals and comments, by internally
    treating a missing encoding declaration as a declaration of
    "iso-8859-1". This will cause arbitrary byte strings to correctly
    round-trip between step 2 and step 5 of the processing, and provide
    compatibility with Python 2.2 for Unicode literals that contain
    non-ASCII bytes.

    A warning will be issued if non-ASCII bytes are found in the input,
    once per improperly encoded input file.

2.  Remove the warning, and change the default encoding to "ascii".

The builtin compile() API will be enhanced to accept Unicode as input.
8-bit string input is subject to the standard procedure for encoding
detection as described above.

If a Unicode string with a coding declaration is passed to compile(), a
SyntaxError will be raised.

SUZUKI Hisao is working on a patch; see[1] for details. A patch
implementing only phase 1 is available at[2].

Phases

Implementation of steps 1 and 2 above were completed in 2.3, except for
changing the default encoding to "ascii".

The default encoding was set to "ascii" in version 2.5.

Scope

This PEP intends to provide an upgrade path from the current
(more-or-less) undefined source code encoding situation to a more robust
and portable definition.

References

History

-   1.10 and above: see CVS history
-   1.8: Added '.' to the coding RE.
-   1.7: Added warnings to phase 1 implementation. Replaced the Latin-1
    default encoding with the interpreter's default encoding. Added
    tweaks to compile().
-   1.4 - 1.6: Minor tweaks
-   1.3: Worked in comments by Martin v. Loewis: UTF-8 BOM mark
    detection, Emacs style magic comment, two phase approach to the
    implementation

Copyright

This document has been placed in the public domain.

[1] Phase 2 implementation: https://bugs.python.org/issue534304

[2] Phase 1 implementation: https://bugs.python.org/issue526840