text:
PEP: 385 Title: Migrating from Subversion to Mercurial Version:
$Revision$ Last-Modified: $Date$ Author: Dirkjan Ochtman
<EMAIL_ADDRESS>Antoine Pitrou<EMAIL_ADDRESS>Georg Brandl
<EMAIL_ADDRESS>Status: Final Type: Process Content-Type: text/x-rst
Created: 25-May-2009

Motivation

After having decided to switch to the Mercurial DVCS, the actual
migration still has to be performed. In the case of an important piece
of infrastructure like the version control system for a large,
distributed project like Python, this is a significant effort. This PEP
is an attempt to describe the steps that must be taken for further
discussion. It's somewhat similar to PEP 347, which discussed the
migration to SVN.

To make the most of hg, we would like to make a high-fidelity
conversion, such that (a) as much of the svn metadata as possible is
retained, and (b) all metadata is converted to formats that are common
in Mercurial. This way, tools written for Mercurial can be optimally
used. In order to do this, we want to use the hgsubversion software to
do an initial conversion. This hg extension is focused on providing
high-quality conversion from Subversion to Mercurial for use in two-way
correspondence, meaning it doesn't throw away as much available metadata
as other solutions.

Such a conversion also seems like a good time to reconsider the contents
of the repository and determine if some things are still valuable. In
this spirit, the following sections also propose discarding some of the
older metadata.

Timeline

The current schedule for conversion milestones:

-   2011-02-24: availability of a test repo at hg.python.org

    Test commits will be allowed (and encouraged) from all committers to
    the Subversion repository. The test repository and all test commits
    will be removed once the final conversion is done. The server-side
    hooks will be installed for the test repository, in order to test
    buildbot, diff-email and whitespace checking integration.

-   2011-03-05: final conversion (tentative)

    Commits to the Subversion branches now maintained in Mercurial will
    be blocked. Developers should refrain from pushing to the Mercurial
    repositories until all infrastructure is ensured to work after their
    switch over to the new repository.

Transition plan

Branch strategy

Mercurial has two basic ways of using branches: cloned branches, where
each branch is kept in a separate repository, and named branches, where
each revision keeps metadata to note on which branch it belongs. The
former makes it easier to distinguish branches, at the expense of
requiring more disk space on the client. The latter makes it a little
easier to switch between branches, but all branch names are a persistent
part of history.[1]

Differences between named branches and cloned branches:

-   Tags in a different (maintenance) clone aren't available in the
    local clone
-   Clones with named branches will be larger, since they contain more
    data

We propose to use named branches for release branches and adopt cloned
branches for feature branches.

History management

In order to minimize the loss of information due to the conversion, we
propose to provide several repositories as a conversion result:

-   A repository trimmed to the mainline trunk (and py3k), as well as
    past and present maintenance branches -- this is called the
    "working" repo and is where development continues. This repository
    has all the history needed for development work, including
    annotating source files with changes back up to 1990 and other
    common history-digging operations.

    The default branch in that repo is what is known as py3k in
    Subversion, while the Subversion trunk lives on with the branch name
    legacy-trunk; however in Mercurial this branch will be closed.
    Release branches are named after their major.minor version, e.g.
    3.2.

-   A repository with the full, unedited conversion of the Subversion
    repository (actually, its /python subdirectory) -- this is called
    the "historic" or "archive" repo and will be offered as a read-only
    resource.[2]

-   One more repository per active feature branch; "active" means that
    at least one core developer asks for the branch to be provided. Each
    such repository will contain both the feature branch and all
    ancestor changesets from mainline (coming from trunk and/or py3k in
    SVN).

Since all branches are present in the historic repo, they can later be
extracted as separate repositories at any time should it prove to be
necessary.

The final revision map between SVN revision numbers, Mercurial
changesets and SVN branch names will be made available in a file stored
in the Misc directory. Its format is as following:

    [...]
    88483 e65daae6cf4499a0863cb7645109a4798c28d83e issue10276-snowleopard
    88484 835cb57abffeceaff0d85c2a3aa0625458dd3e31 py3k
    88485 d880f9d8492f597a030772c7485a34aadb6c4ece release32-maint
    88486 0c431b8c22f5dbeb591414c154acb7890c1809df py3k
    88487 82cda1f21396bbd10db8083ea20146d296cb630b release32-maint
    88488 8174d00d07972d6f109ed57efca8273a4d59302c release27-maint
    [...]

Converting tags

The SVN tags directory contains a lot of old stuff. Some of these are
not, in fact, full tags, but contain only a smaller subset of the
repository. All release tags will be kept; other tags will be included
based on requests from the developer community. We propose to make the
tag naming scheme consistent, in this style: v3.2.1a2.

Author map

In order to provide user names the way they are common in hg (in the
'First Last<EMAIL_ADDRESS>format), we need an author map to map
cvs and svn user names to real names and their email addresses. We have
a complete version of such a map in the migration tools repository (not
publicly accessible to avoid leaking addresses to harvesters). The email
addresses in it might be out of date; that's bound to happen, although
it would be nice to try and have as many people as possible review it
for addresses that are out of date. The current version also still seems
to contain some encoding problems.

Generating .hgignore

The .hgignore file can be used in Mercurial repositories to help ignore
files that are not eligible for version control. It does this by
employing several possible forms of pattern matching. The current Python
repository already includes a rudimentary .hgignore file to help with
using the hg mirrors.

Since the current Python repository already includes a .hgignore file
(for use with hg mirrors), we'll just use that. Generating full history
of the file was debated but deemed impractical (because it's relatively
hard with fairly little gain, since ignoring is less important for older
revisions).

Repository size

A bare conversion result of the current Python repository weighs 1.9 GB;
although this is smaller than the Subversion repository (2.7 GB) it is
not feasible.

The size becomes more manageable by the trimming applied to the working
repository, and by a process called "revlog reordering" that optimizes
the layout of internal Mercurial storage very efficiently.

After all optimizations done, the size of the working repository is
around 180 MB on disk. The amount of data transferred over the network
when cloning is estimated to be around 80 MB.

Other repositories

There are a number of other projects hosted in svn.python.org's
"projects" repository. The "peps" directory will be converted along with
the main Python one. Richard Tew has indicated that he'd like the
Stackless repository to also be converted. What other projects in the
svn.python.org repository should be converted?

There's now an initial stab at converting the Jython repository. The
current tip of hgsubversion unfortunately fails at some point. Pending
investigation.

Other repositories that would like to converted to Mercurial can
announce themselves to me after the main Python migration is done, and
I'll take care of their needs.

Infrastructure

hg-ssh

Developers should access the repositories through ssh, similar to the
current setup. Public keys can be used to grant people access to a
shared hg@ account. A hgwebdir instance also has been set up at
hg.python.org for easy browsing and read-only access. It is configured
so that developers can trivially start new clones (for longer-term
features that profit from development in a separate repository).

Also, direct creation of public repositories is allowed for core
developers, although it is not yet decided which naming scheme will be
enforced:

    $ hg init<EMAIL_ADDRESS>    repo created, public URL is http://hg.python.org/sandbox/mywork

Hooks

A number of hooks is currently in use. The hg equivalents for these
should be developed and deployed. The following hooks are being used:

-   check whitespace: a hook to reject commits in case the whitespace
    doesn't match the rules for the Python codebase. In a changegroup,
    only the tip is checked (this allows cleanup commits for changes
    pulled from third-party repos). We can also offer a whitespace hook
    for use with client-side repositories that people can use; it could
    either warn about whitespace issues and/or truncate trailing
    whitespace from changed lines.
-   push mails: Emails will include diffs for each changeset pushed to
    the public repository, including the username which pushed the
    changesets (this is not necessarily the same as the author recorded
    in the changesets).
-   buildbots: the python.org build master will be notified of each
    changeset pushed to the cpython repository, and will trigger an
    appropriate build on every build slave for the branch in which the
    changeset occurs.

The hooks repository contains ports of these server-side hooks to
Mercurial, as well as a couple additional ones:

-   check branch heads: a hook to reject pushes which create a new head
    on an existing branch. The pusher then has to merge the excess heads
    and try pushing again.
-   check branches: a hook to reject all changesets not on an allowed
    named branch. This hook's whitelist will have to be updated when we
    want to create new maintenance branches.
-   check line endings: a hook, based on the eol extension, to reject
    all changesets committing files with the wrong line endings. The
    commits then have to be stripped and redone, possibly with the eol
    extension enabled on the comitter's computer.

One additional hook could be beneficial:

-   check contributors: in the current setup, all changesets bear the
    username of committers, who must have signed the contributor
    agreement. We might want to use a hook to check if the committer is
    a contributor if we keep a list of registered contributors. Then,
    the hook might warn users that push a group of revisions containing
    changesets from unknown contributors.

End-of-line conversions

Discussion about the lack of end-of-line conversion support in
Mercurial, which was provided initially by the win32text extension, led
to the development of the new eol extension that supports a versioned
management of line-ending conventions on a file-by-file basis, akin to
Subversion's svn:eol-style properties. This information is kept in a
versioned file called .hgeol, and such a file has already been checked
into the Subversion repository.

A hook also exists on the server side to reject any changeset
introducing inconsistent newline data (see above).

hgwebdir

A more or less stock hgwebdir installation should be set up. We might
want to come up with a style to match the Python website.

A small WSGI application has been written that can look up Subversion
revisions and redirect to the appropriate hgweb page for the given
changeset, regardless in which repository the converted revision ended
up (since one big Subversion repository is converted into several
Mercurial repositories). It can also look up Mercurial changesets by
their hexadecimal ID.

roundup

By pointing Roundup to the URL of the lookup script mentioned above,
links to SVN revisions will continue to work, and links to Mercurial
changesets can be created as well, without having to give repository and
changeset ID.

After migration

Where to get code

After migration, the hgwebdir will live at hg.python.org. This is an
accepted standard for many organizations, and an easy parallel to
svn.python.org. The working repo might live at
http://hg.python.org/cpython/, for example, with the archive repo at
http://hg.python.org/cpython-archive/. For write access, developers will
have to use ssh, which could be<EMAIL_ADDRESS>
code.python.org was also proposed as the hostname. We think that using
the VCS name in the hostname is good because it prevents confusion: it
should be clear that you can't use svn or bzr for hg.python.org.

hgwebdir can already provide tarballs for every changeset. This obviates
the need for daily snapshots; we can just point users to tip.tar.gz
instead, meaning they will get the latest. If desired, we could even use
buildbot results to point to the last good changeset.

Python-specific documentation

hg comes with good built-in documentation (available through hg help)
and a wiki that's full of useful information and recipes, not to mention
a popular book (readable online).

In addition to that, the recently overhauled Python Developer's Guide
already has a branch with instructions for Mercurial instead of
Subversion; an online build of this branch is also available.

Proposed workflow

We propose two workflows for the migration of patches between several
branches.

For migration within 2.x or 3.x branches, we propose a patch always gets
committed to the oldest branch where it applies first. Then, the
resulting changeset can be merged using hg merge to all newer branches
within that series (2.x or 3.x). If it does not apply as-is to the newer
branch, hg revert can be used to easily revert to the new-branch-native
head, patch in some alternative version of the patch (or none, if it's
not applicable), then commit the merge. The premise here is that all
changesets from an older branch within the series are eventually merged
to all newer branches within the series.

The upshot is that this provides for the most painless merging
procedure. This means that in the general case, people have to think
about the oldest branch to which the patch should be applied before
actually applying it. Usually, that is one of only two branches: the
latest maintenance branch and the trunk, except for security fixes
applicable to older branches in security-fix-only mode.

For merging bug fixes from the 3.x to the 2.7 maintenance branch (2.6
and 2.5 are in security-fix-only mode and their maintenance will
continue in the Subversion repository), changesets should be
transplanted (not merged) in some other way. The transplant extension,
import/export and bundle/unbundle work equally well here.

Choosing this approach allows 3.x not to carry all of the 2.x
history-since-it-was-branched, meaning the clone is not as big and the
merges not as complicated.

The future of Subversion

What happens to the Subversion repositories after the migration? Since
the svn server contains a bunch of repositories, not just the CPython
one, it will probably live on for a bit as not every project may want to
migrate or it takes longer for other projects to migrate. To prevent
people from staying behind, we may want to move migrated projects from
the repository to a new, read-only repository with a new name.

Build identification

Python currently provides the sys.subversion tuple to allow Python code
to find out exactly what version of Python it's running against. The
current version looks something like this:

-   ('CPython', 'tags/r262', '71600')
-   ('CPython', 'trunk', '73128M')

Another value is returned from Py_GetBuildInfo() in the C API, and
available to Python code as part of sys.version:

-   'r262:71600, Jun 2 2009, 09:58:33'
-   'trunk:73128M, Jun 2 2009, 01:24:14'

I propose that the revision identifier will be the short version of hg's
revision hash, for example 'dd3ebf81af43', augmented with '+' (instead
of 'M') if the working directory from which it was built was modified.
This mirrors the output of the hg id command, which is intended for this
kind of usage. The sys.subversion value will also be renamed to
sys.mercurial to reflect the change in VCS.

For the tag/branch identifier, I propose that hg will check for tags on
the currently checked out revision, use the tag if there is one ('tip'
doesn't count), and uses the branch name otherwise. sys.subversion
becomes

-   ('CPython', 'v2.6.2', 'dd3ebf81af43')
-   ('CPython', 'default', 'af694c6a888c+')

and the build info string becomes

-   'v2.6.2:dd3ebf81af43, Jun 2 2009, 09:58:33'
-   'default:af694c6a888c+, Jun 2 2009, 01:24:14'

This reflects that the default branch in hg is called 'default' instead
of Subversion's 'trunk', and reflects the proposed new tag format.

Mercurial also allows to find out the latest tag and the number of
changesets separating the current changeset from that tag, allowing for
a descriptive version string:

    $ hg parent --template "{latesttag}+{latesttagdistance}-{node|short}\n"
    v3.2+37-4b5d0d260e72
    $ hg up 2.7
    3316 files updated, 0 files merged, 379 files removed, 0 files unresolved
    $ hg parent --template "{latesttag}+{latesttagdistance}-{node|short}\n"
    v2.7.1+216-9619d21d8198

Footnotes

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] The Mercurial book discourages the use of named branches, but it is,
in this respect, somewhat outdated. Named branches have gotten much
easier to use since that comment was written, due to improvements in hg.

[2] Since the initial working repo is a subset of the archive repo, it
would also be feasible to pull changes from the working repo into the
archive repo periodically.


text:
PEP: 335 Title: Overloadable Boolean Operators Version: $Revision$
Last-Modified: $Date$ Author: Gregory Ewing
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 29-Aug-2004 Python-Version: 3.3
Post-History: 05-Sep-2004, 30-Sep-2011, 25-Oct-2011

Rejection Notice

This PEP was rejected. See
https://mail.python.org/pipermail/python-dev/2012-March/117510.html

Abstract

This PEP proposes an extension to permit objects to define their own
meanings for the boolean operators 'and', 'or' and 'not', and suggests
an efficient strategy for implementation. A prototype of this
implementation is available for download.

Background

Python does not currently provide any '__xxx__' special methods
corresponding to the 'and', 'or' and 'not' boolean operators. In the
case of 'and' and 'or', the most likely reason is that these operators
have short-circuiting semantics, i.e. the second operand is not
evaluated if the result can be determined from the first operand. The
usual technique of providing special methods for these operators
therefore would not work.

There is no such difficulty in the case of 'not', however, and it would
be straightforward to provide a special method for this operator. The
rest of this proposal will therefore concentrate mainly on providing a
way to overload 'and' and 'or'.

Motivation

There are many applications in which it is natural to provide custom
meanings for Python operators, and in some of these, having boolean
operators excluded from those able to be customised can be inconvenient.
Examples include:

1.  NumPy, in which almost all the operators are defined on arrays so as
    to perform the appropriate operation between corresponding elements,
    and return an array of the results. For consistency, one would
    expect a boolean operation between two arrays to return an array of
    booleans, but this is not currently possible.

    There is a precedent for an extension of this kind: comparison
    operators were originally restricted to returning boolean results,
    and rich comparisons were added so that comparisons of NumPy arrays
    could return arrays of booleans.

2.  A symbolic algebra system, in which a Python expression is evaluated
    in an environment which results in it constructing a tree of objects
    corresponding to the structure of the expression.

3.  A relational database interface, in which a Python expression is
    used to construct an SQL query.

A workaround often suggested is to use the bitwise operators '&', '|'
and '~' in place of 'and', 'or' and 'not', but this has some drawbacks:

-   The precedence of these is different in relation to the other
    operators, and they may already be in use for other purposes (as in
    example 1).
-   It is aesthetically displeasing to force users to use something
    other than the most obvious syntax for what they are trying to
    express. This would be particularly acute in the case of example 3,
    considering that boolean operations are a staple of SQL queries.
-   Bitwise operators do not provide a solution to the problem of
    chained comparisons such as 'a < b < c' which involve an implicit
    'and' operation. Such expressions currently cannot be used at all on
    data types such as NumPy arrays where the result of a comparison
    cannot be treated as having normal boolean semantics; they must be
    expanded into something like (a < b) & (b < c), losing a
    considerable amount of clarity.

Rationale

The requirements for a successful solution to the problem of allowing
boolean operators to be customised are:

1.  In the default case (where there is no customisation), the existing
    short-circuiting semantics must be preserved.
2.  There must not be any appreciable loss of speed in the default case.
3.  Ideally, the customisation mechanism should allow the object to
    provide either short-circuiting or non-short-circuiting semantics,
    at its discretion.

One obvious strategy, that has been previously suggested, is to pass
into the special method the first argument and a function for evaluating
the second argument. This would satisfy requirements 1 and 3, but not
requirement 2, since it would incur the overhead of constructing a
function object and possibly a Python function call on every boolean
operation. Therefore, it will not be considered further here.

The following section proposes a strategy that addresses all three
requirements. A prototype implementation of this strategy is available
for download.

Specification

Special Methods

At the Python level, objects may define the following special methods.

+-------------------+--------------------+----------------------------+
| Unary             | Binary, phase 1    | Binary, phase 2            |
+===================+====================+============================+
| -   __not__(self) | -   __and1__(self) | -   __and2__(self, other)  |
|                   | -   __or1__(self)  | -   __or2__(self, other)   |
|                   |                    | -   __rand2__(self, other) |
|                   |                    | -   __ror2__(self, other)  |
+-------------------+--------------------+----------------------------+

The __not__ method, if defined, implements the 'not' operator. If it is
not defined, or it returns NotImplemented, existing semantics are used.

To permit short-circuiting, processing of the 'and' and 'or' operators
is split into two phases. Phase 1 occurs after evaluation of the first
operand but before the second. If the first operand defines the relevant
phase 1 method, it is called with the first operand as argument. If that
method can determine the result without needing the second operand, it
returns the result, and further processing is skipped.

If the phase 1 method determines that the second operand is needed, it
returns the special value NeedOtherOperand. This triggers the evaluation
of the second operand, and the calling of a relevant phase 2 method.
During phase 2, the __and2__/__rand2__ and __or2__/__ror2__ method pairs
work as for other binary operators.

Processing falls back to existing semantics if at any stage a relevant
special method is not found or returns NotImplemented.

As a special case, if the first operand defines a phase 2 method but no
corresponding phase 1 method, the second operand is always evaluated and
the phase 2 method called. This allows an object which does not want
short-circuiting semantics to simply implement the phase 2 methods and
ignore phase 1.

Bytecodes

The patch adds four new bytecodes, LOGICAL_AND_1, LOGICAL_AND_2,
LOGICAL_OR_1 and LOGICAL_OR_2. As an example of their use, the bytecode
generated for an 'and' expression looks like this:

    .
    .
    .
    evaluate first operand
    LOGICAL_AND_1  L
    evaluate second operand
    LOGICAL_AND_2
    L:   .
    .
    .

The LOGICAL_AND_1 bytecode performs phase 1 processing. If it determines
that the second operand is needed, it leaves the first operand on the
stack and continues with the following code. Otherwise it pops the first
operand, pushes the result and branches to L.

The LOGICAL_AND_2 bytecode performs phase 2 processing, popping both
operands and pushing the result.

Type Slots

At the C level, the new special methods are manifested as five new slots
in the type object. In the patch, they are added to the tp_as_number
substructure, since this allows making use of some existing code for
dealing with unary and binary operators. Their existence is signalled by
a new type flag, Py_TPFLAGS_HAVE_BOOLEAN_OVERLOAD.

The new type slots are:

    unaryfunc nb_logical_not;
    unaryfunc nb_logical_and_1;
    unaryfunc nb_logical_or_1;
    binaryfunc nb_logical_and_2;
    binaryfunc nb_logical_or_2;

Python/C API Functions

There are also five new Python/C API functions corresponding to the new
operations:

    PyObject *PyObject_LogicalNot(PyObject *);
    PyObject *PyObject_LogicalAnd1(PyObject *);
    PyObject *PyObject_LogicalOr1(PyObject *);
    PyObject *PyObject_LogicalAnd2(PyObject *, PyObject *);
    PyObject *PyObject_LogicalOr2(PyObject *, PyObject *);

Alternatives and Optimisations

This section discusses some possible variations on the proposal, and
ways in which the bytecode sequences generated for boolean expressions
could be optimised.

Reduced special method set

For completeness, the full version of this proposal includes a mechanism
for types to define their own customised short-circuiting behaviour.
However, the full mechanism is not needed to address the main use cases
put forward here, and it would be possible to define a simplified
version that only includes the phase 2 methods. There would then only be
5 new special methods (__and2__, __rand2__, __or2__, __ror2__, __not__)
with 3 associated type slots and 3 API functions.

This simplified version could be expanded to the full version later if
desired.

Additional bytecodes

As defined here, the bytecode sequence for code that branches on the
result of a boolean expression would be slightly longer than it
currently is. For example, in Python 2.7,

    if a and b:
        statement1
    else:
        statement2

generates

    LOAD_GLOBAL         a
    POP_JUMP_IF_FALSE   false_branch
    LOAD_GLOBAL         b
    POP_JUMP_IF_FALSE   false_branch
    <code for statement1>
    JUMP_FORWARD        end_branch
    false_branch:
    <code for statement2>
    end_branch:

Under this proposal as described so far, it would become something like

    LOAD_GLOBAL         a
    LOGICAL_AND_1       test
    LOAD_GLOBAL         b
    LOGICAL_AND_2
    test:
    POP_JUMP_IF_FALSE   false_branch
    <code for statement1>
    JUMP_FORWARD        end_branch
    false_branch:
    <code for statement2>
    end_branch:

This involves executing one extra bytecode in the short-circuiting case
and two extra bytecodes in the non-short-circuiting case.

However, by introducing extra bytecodes that combine the logical
operations with testing and branching on the result, it can be reduced
to the same number of bytecodes as the original:

    LOAD_GLOBAL         a
    AND1_JUMP           true_branch, false_branch
    LOAD_GLOBAL         b
    AND2_JUMP_IF_FALSE  false_branch
    true_branch:
    <code for statement1>
    JUMP_FORWARD        end_branch
    false_branch:
    <code for statement2>
    end_branch:

Here, AND1_JUMP performs phase 1 processing as above, and then examines
the result. If there is a result, it is popped from the stack, its truth
value is tested and a branch taken to one of two locations.

Otherwise, the first operand is left on the stack and execution
continues to the next bytecode. The AND2_JUMP_IF_FALSE bytecode performs
phase 2 processing, pops the result and branches if it tests false

For the 'or' operator, there would be corresponding OR1_JUMP and
OR2_JUMP_IF_TRUE bytecodes.

If the simplified version without phase 1 methods is used, then early
exiting can only occur if the first operand is false for 'and' and true
for 'or'. Consequently, the two-target AND1_JUMP and OR1_JUMP bytecodes
can be replaced with AND1_JUMP_IF_FALSE and OR1_JUMP_IF_TRUE, these
being ordinary branch instructions with only one target.

Optimisation of 'not'

Recent versions of Python implement a simple optimisation in which
branching on a negated boolean expression is implemented by reversing
the sense of the branch, saving a UNARY_NOT opcode.

Taking a strict view, this optimisation should no longer be performed,
because the 'not' operator may be overridden to produce quite different
results from usual. However, in typical use cases, it is not envisaged
that expressions involving customised boolean operations will be used
for branching -- it is much more likely that the result will be used in
some other way.

Therefore, it would probably do little harm to specify that the compiler
is allowed to use the laws of boolean algebra to simplify any expression
that appears directly in a boolean context. If this is inconvenient, the
result can always be assigned to a temporary name first.

This would allow the existing 'not' optimisation to remain, and would
permit future extensions of it such as using De Morgan's laws to extend
it deeper into the expression.

Usage Examples

Example 1: NumPy Arrays

    #-----------------------------------------------------------------
    #
    #   This example creates a subclass of numpy array to which
    #   'and', 'or' and 'not' can be applied, producing an array
    #   of booleans.
    #
    #-----------------------------------------------------------------

    from numpy import array, ndarray

    class BArray(ndarray):

        def __str__(self):
            return "barray(%s)" % ndarray.__str__(self)

        def __and2__(self, other):
            return (self & other)

        def __or2__(self, other):
            return (self & other)

        def __not__(self):
            return (self == 0)

    def barray(*args, **kwds):
        return array(*args, **kwds).view(type = BArray)

    a0 = barray([0, 1, 2, 4])
    a1 = barray([1, 2, 3, 4])
    a2 = barray([5, 6, 3, 4])
    a3 = barray([5, 1, 2, 4])

    print "a0:", a0
    print "a1:", a1
    print "a2:", a2
    print "a3:", a3
    print "not a0:", not a0
    print "a0 == a1 and a2 == a3:", a0 == a1 and a2 == a3
    print "a0 == a1 or a2 == a3:", a0 == a1 or a2 == a3

Example 1 Output

    a0: barray([0 1 2 4])
    a1: barray([1 2 3 4])
    a2: barray([5 6 3 4])
    a3: barray([5 1 2 4])
    not a0: barray([ True False False False])
    a0 == a1 and a2 == a3: barray([False False False  True])
    a0 == a1 or a2 == a3: barray([False False False  True])

Example 2: Database Queries

    #-----------------------------------------------------------------
    #
    #   This example demonstrates the creation of a DSL for database
    #   queries allowing 'and' and 'or' operators to be used to
    #   formulate the query.
    #
    #-----------------------------------------------------------------

    class SQLNode(object):

        def __and2__(self, other):
            return SQLBinop("and", self, other)

        def __rand2__(self, other):
            return SQLBinop("and", other, self)

        def __eq__(self, other):
            return SQLBinop("=", self, other)

    class Table(SQLNode):

        def __init__(self, name):
            self.__tablename__ = name

        def __getattr__(self, name):
            return SQLAttr(self, name)

        def __sql__(self):
            return self.__tablename__

    class SQLBinop(SQLNode):

        def __init__(self, op, opnd1, opnd2):
            self.op = op.upper()
            self.opnd1 = opnd1
            self.opnd2 = opnd2

        def __sql__(self):
            return "(%s %s %s)" % (sql(self.opnd1), self.op, sql(self.opnd2))

    class SQLAttr(SQLNode):

        def __init__(self, table, name):
            self.table = table
            self.name = name

        def __sql__(self):
            return "%s.%s" % (sql(self.table), self.name)

    class SQLSelect(SQLNode):

        def __init__(self, targets):
            self.targets = targets
            self.where_clause = None

        def where(self, expr):
            self.where_clause = expr
            return self

        def __sql__(self):
            result = "SELECT %s" % ", ".join([sql(target) for target in self.targets])
            if self.where_clause:
                result = "%s WHERE %s" % (result, sql(self.where_clause))
            return result

    def sql(expr):
        if isinstance(expr, SQLNode):
            return expr.__sql__()
        elif isinstance(expr, str):
            return "'%s'" % expr.replace("'", "''")
        else:
            return str(expr)

    def select(*targets):
        return SQLSelect(targets)

    #-----------------------------------------------------------------

    dishes = Table("dishes")
    customers = Table("customers")
    orders = Table("orders")

    query = select(customers.name, dishes.price, orders.amount).where(
        customers.cust_id == orders.cust_id and orders.dish_id == dishes.dish_id
        and dishes.name == "Spam, Eggs, Sausages and Spam")

    print repr(query)
    print sql(query)

Example 2 Output

    <__main__.SQLSelect object at 0x1cc830>
    SELECT customers.name, dishes.price, orders.amount WHERE
    (((customers.cust_id = orders.cust_id) AND (orders.dish_id =
    dishes.dish_id)) AND (dishes.name = 'Spam, Eggs, Sausages and Spam'))

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

license: Public Domain


text:
PEP: 591 Title: Adding a final qualifier to typing Author: Michael J.
Sullivan<EMAIL_ADDRESS>Ivan Levkivskyi<EMAIL_ADDRESS>BDFL-Delegate: Guido van Rossum<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Typing
Created: 15-Mar-2019 Python-Version: 3.8 Post-History: Resolution:
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><typing.final> and
typing:uppercase-final/typing.Final

Abstract

This PEP proposes a "final" qualifier to be added to the typing
module---in the form of a final decorator and a Final type
annotation---to serve three related purposes:

-   Declaring that a method should not be overridden
-   Declaring that a class should not be subclassed
-   Declaring that a variable or attribute should not be reassigned

Motivation

The final decorator

The current typing module lacks a way to restrict the use of inheritance
or overriding at a typechecker level. This is a common feature in other
object-oriented languages (such as Java), and is useful for reducing the
potential space of behaviors of a class, easing reasoning.

Some situations where a final class or method may be useful include:

-   A class wasn’t designed to be subclassed or a method wasn't designed
    to be overridden. Perhaps it would not work as expected, or be
    error-prone.
-   Subclassing or overriding would make code harder to understand or
    maintain. For example, you may want to prevent unnecessarily tight
    coupling between base classes and subclasses.
-   You want to retain the freedom to arbitrarily change the class
    implementation in the future, and these changes might break
    subclasses.

The Final annotation

The current typing module lacks a way to indicate that a variable will
not be assigned to. This is a useful feature in several situations:

-   Preventing unintended modification of module and class level
    constants and documenting them as constants in a checkable way.
-   Creating a read-only attribute that may not be overridden by
    subclasses. (@property can make an attribute read-only but does not
    prevent overriding)
-   Allowing a name to be used in situations where ordinarily a literal
    is expected (for example as a field name for NamedTuple, a tuple of
    types passed to isinstance, or an argument to a function with
    arguments of Literal type (PEP 586)).

Specification

The final decorator

The typing.final decorator is used to restrict the use of inheritance
and overriding.

A type checker should prohibit any class decorated with @final from
being subclassed and any method decorated with @final from being
overridden in a subclass. The method decorator version may be used with
all of instance methods, class methods, static methods, and properties.

For example:

    from typing import final

    @final
    class Base:
        ...

    class Derived(Base):  # Error: Cannot inherit from final class "Base"
        ...

and:

    from typing import final

    class Base:
        @final
        def foo(self) -> None:
            ...

    class Derived(Base):
        def foo(self) -> None:  # Error: Cannot override final attribute "foo"
                                # (previously declared in base class "Base")
            ...

For overloaded methods, @final should be placed on the implementation
(or on the first overload, for stubs):

    from typing import Any, overload

    class Base:
        @overload
        def method(self) -> None: ...
        @overload
        def method(self, arg: int) -> int: ...
        @final
        def method(self, x=None):
            ...

It is an error to use @final on a non-method function.

The Final annotation

The typing.Final type qualifier is used to indicate that a variable or
attribute should not be reassigned, redefined, or overridden.

Syntax

Final may be used in one of several forms:

-   With an explicit type, using the syntax Final[<type>]. Example:

        ID: Final[float] = 1

-   With no type annotation. Example:

        ID: Final = 1

    The typechecker should apply its usual type inference mechanisms to
    determine the type of ID (here, likely, int). Note that unlike for
    generic classes this is not the same as Final[Any].

-   In class bodies and stub files you can omit the right hand side and
    just write ID: Final[float]. If the right hand side is omitted,
    there must be an explicit type argument to Final.

-   Finally, as self.id: Final = 1 (also optionally with a type in
    square brackets). This is allowed only in __init__ methods, so that
    the final instance attribute is assigned only once when an instance
    is created.

Semantics and examples

The two main rules for defining a final name are:

-   There can be at most one final declaration per module or class for a
    given attribute. There can't be separate class-level and
    instance-level constants with the same name.
-   There must be exactly one assignment to a final name.

This means a type checker should prevent further assignments to final
names in type-checked code:

    from typing import Final

    RATE: Final = 3000

    class Base:
        DEFAULT_ID: Final = 0

    RATE = 300  # Error: can't assign to final attribute
    Base.DEFAULT_ID = 1  # Error: can't override a final attribute

Note that a type checker need not allow Final declarations inside loops
since the runtime will see multiple assignments to the same variable in
subsequent iterations.

Additionally, a type checker should prevent final attributes from being
overridden in a subclass:

    from typing import Final

    class Window:
        BORDER_WIDTH: Final = 2.5
        ...

    class ListView(Window):
        BORDER_WIDTH = 3  # Error: can't override a final attribute

A final attribute declared in a class body without an initializer must
be initialized in the __init__ method (except in stub files):

    class ImmutablePoint:
        x: Final[int]
        y: Final[int]  # Error: final attribute without an initializer

        def __init__(self) -> None:
            self.x = 1  # Good

Type checkers should infer a final attribute that is initialized in a
class body as being a class variable. Variables should not be annotated
with both ClassVar and Final.

Final may only be used as the outermost type in assignments or variable
annotations. Using it in any other position is an error. In particular,
Final can't be used in annotations for function arguments:

    x: List[Final[int]] = []  # Error!

    def fun(x: Final[List[int]]) ->  None:  # Error!
        ...

Note that declaring a name as final only guarantees that the name will
not be re-bound to another value, but does not make the value immutable.
Immutable ABCs and containers may be used in combination with Final to
prevent mutating such values:

    x: Final = ['a', 'b']
    x.append('c')  # OK

    y: Final[Sequence[str]] = ['a', 'b']
    y.append('x')  # Error: "Sequence[str]" has no attribute "append"
    z: Final = ('a', 'b')  # Also works

Type checkers should treat uses of a final name that was initialized
with a literal as if it was replaced by the literal. For example, the
following should be allowed:

    from typing import NamedTuple, Final

    X: Final = "x"
    Y: Final = "y"
    N = NamedTuple("N", [(X, int), (Y, int)])

Reference Implementation

The mypy[1] type checker supports Final and final. A reference
implementation of the runtime component is provided in the
typing_extensions[2] module.

Rejected/deferred Ideas

The name Const was also considered as the name for the Final type
annotation. The name Final was chosen instead because the concepts are
related and it seemed best to be consistent between them.

We considered using a single name Final instead of introducing final as
well, but @Final just looked too weird to us.

A related feature to final classes would be Scala-style sealed classes,
where a class is allowed to be inherited only by classes defined in the
same module. Sealed classes seem most useful in combination with pattern
matching, so it does not seem to justify the complexity in our case.
This could be revisited in the future.

It would be possible to have the @final decorator on classes dynamically
prevent subclassing at runtime. Nothing else in typing does any runtime
enforcement, though, so final will not either. A workaround for when
both runtime enforcement and static checking is desired is to use this
idiom (possibly in a support module):

    if typing.TYPE_CHECKING:
        from typing import final
    else:
        from runtime_final import final

References

Copyright

This document has been placed in the public domain.

[1] http://www.mypy-lang.org/

[2] https://github.com/python/typing/tree/master/typing_extensions

authors:
Ivan Levkivskyi
Michael J. Sullivan


text:
PEP: 247 Title: API for Cryptographic Hash Functions Version: $Revision$
Last-Modified: $Date$ Author: A.M. Kuchling<EMAIL_ADDRESS>Status: Final
Type: Informational Content-Type: text/x-rst Created: 23-Mar-2001
Post-History: 20-Sep-2001

Abstract

There are several different modules available that implement
cryptographic hashing algorithms such as MD5 or SHA. This document
specifies a standard API for such algorithms, to make it easier to
switch between different implementations.

Specification

All hashing modules should present the same interface. Additional
methods or variables can be added, but those described in this document
should always be present.

Hash function modules define one function:

new([string])            (unkeyed hashes)
new([key] , [string])    (keyed hashes)

  Create a new hashing object and return it. The first form is for
  hashes that are unkeyed, such as MD5 or SHA. For keyed hashes such as
  HMAC, key is a required parameter containing a string giving the key
  to use. In both cases, the optional string parameter, if supplied,
  will be immediately hashed into the object's starting state, as if
  obj.update(string) was called.

  After creating a hashing object, arbitrary strings can be fed into the
  object using its update() method, and the hash value can be obtained
  at any time by calling the object's digest() method.

  Arbitrary additional keyword arguments can be added to this function,
  but if they're not supplied, sensible default values should be used.
  For example, rounds and digest_size keywords could be added for a hash
  function which supports a variable number of rounds and several
  different output sizes, and they should default to values believed to
  be secure.

Hash function modules define one variable:

digest_size

  An integer value; the size of the digest produced by the hashing
  objects created by this module, measured in bytes. You could also
  obtain this value by creating a sample object and accessing its
  digest_size attribute, but it can be convenient to have this value
  available from the module. Hashes with a variable output size will set
  this variable to None.

Hashing objects require a single attribute:

digest_size

  This attribute is identical to the module-level digest_size variable,
  measuring the size of the digest produced by the hashing object,
  measured in bytes. If the hash has a variable output size, this output
  size must be chosen when the hashing object is created, and this
  attribute must contain the selected size. Therefore, None is not a
  legal value for this attribute.

Hashing objects require the following methods:

copy()

  Return a separate copy of this hashing object. An update to this copy
  won't affect the original object.

digest()

  Return the hash value of this hashing object as a string containing
  8-bit data. The object is not altered in any way by this function; you
  can continue updating the object after calling this function.

hexdigest()

  Return the hash value of this hashing object as a string containing
  hexadecimal digits. Lowercase letters should be used for the digits a
  through f. Like the .digest() method, this method mustn't alter the
  object.

update(string)

  Hash string into the current state of the hashing object. update() can
  be called any number of times during a hashing object's lifetime.

Hashing modules can define additional module-level functions or object
methods and still be compliant with this specification.

Here's an example, using a module named MD5:

    >>> from Crypto.Hash import MD5
    >>> m = MD5.new()
    >>> m.digest_size
    16
    >>> m.update('abc')
    >>> m.digest()
    '\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr'
    >>> m.hexdigest()
    '900150983cd24fb0d6963f7d28e17f72'
    >>> MD5.new('abc').digest()
    '\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr'

Rationale

The digest size is measured in bytes, not bits, even though hash
algorithm sizes are usually quoted in bits; MD5 is a 128-bit algorithm
and not a 16-byte one, for example. This is because, in the sample code
I looked at, the length in bytes is often needed (to seek ahead or
behind in a file; to compute the length of an output string) while the
length in bits is rarely used. Therefore, the burden will fall on the
few people actually needing the size in bits, who will have to multiply
digest_size by 8.

It's been suggested that the update() method would be better named
append(). However, that method is really causing the current state of
the hashing object to be updated, and update() is already used by the
md5 and sha modules included with Python, so it seems simplest to leave
the name update() alone.

The order of the constructor's arguments for keyed hashes was a sticky
issue. It wasn't clear whether the key should come first or second. It's
a required parameter, and the usual convention is to place required
parameters first, but that also means that the string parameter moves
from the first position to the second. It would be possible to get
confused and pass a single argument to a keyed hash, thinking that
you're passing an initial string to an unkeyed hash, but it doesn't seem
worth making the interface for keyed hashes more obscure to avoid this
potential error.

Changes

2001-09-17: Renamed clear() to reset(); added digest_size attribute to
objects; added .hexdigest() method.

2001-09-20: Removed reset() method completely.

2001-09-28: Set digest_size to None for variable-size hashes.

Acknowledgements

Thanks to Aahz, Andrew Archibald, Rich Salz, Itamar Shtull-Trauring, and
the readers of the python-crypto list for their comments on this PEP.

Copyright

This document has been placed in the public domain.


text:
PEP: 273 Title: Import Modules from Zip Archives Author: James C.
Ahlstrom<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 11-Oct-2001 Python-Version: 2.3
Post-History: 26-Oct-2001

Abstract

This PEP adds the ability to import Python modules *.py, *.py[co] and
packages from zip archives. The same code is used to speed up normal
directory imports provided os.listdir is available.

Note

Zip imports were added to Python 2.3, but the final implementation uses
an approach different from the one described in this PEP. The 2.3
implementation is SourceForge patch #652586[1], which adds new import
hooks described in PEP 302.

The rest of this PEP is therefore only of historical interest.

Specification

Currently, sys.path is a list of directory names as strings. If this PEP
is implemented, an item of sys.path can be a string naming a zip file
archive. The zip archive can contain a subdirectory structure to support
package imports. The zip archive satisfies imports exactly as a
subdirectory would.

The implementation is in C code in the Python core and works on all
supported Python platforms.

Any files may be present in the zip archive, but only files *.py and
*.py[co] are available for import. Zip import of dynamic modules (*.pyd,
*.so) is disallowed.

Just as sys.path currently has default directory names, a default zip
archive name is added too. Otherwise there is no way to import all
Python library files from an archive.

Subdirectory Equivalence

The zip archive must be treated exactly as a subdirectory tree so we can
support package imports based on current and future rules. All zip data
is taken from the Central Directory, the data must be correct, and brain
dead zip files are not accommodated.

Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip", and we
are trying to import modfoo from the Q package. Then import.c will
generate a list of paths and extensions and will look for the file. The
list of generated paths does not change for zip imports. Suppose
import.c generates the path "/A/B/SubDir/Q/R/modfoo.pyc". Then it will
also generate the path "/C/D/E/Archive.zip/Q/R/modfoo.pyc". Finding the
SubDir path is exactly equivalent to finding "Q/R/modfoo.pyc" in the
archive.

Suppose you zip up /A/B/SubDir/* and all its subdirectories. Then your
zip file will satisfy imports just as your subdirectory did.

Well, not quite. You can't satisfy dynamic modules from a zip file.
Dynamic modules have extensions like .dll, .pyd, and .so. They are
operating system dependent, and probably can't be loaded except from a
file. It might be possible to extract the dynamic module from the zip
file, write it to a plain file and load it. But that would mean creating
temporary files, and dealing with all the dynload_*.c, and that's
probably not a good idea.

When trying to import *.pyc, if it is not available then *.pyo will be
used instead. And vice versa when looking for *.pyo. If neither *.pyc
nor *.pyo is available, or if the magic numbers are invalid, then *.py
will be compiled and used to satisfy the import, but the compiled file
will not be saved. Python would normally write it to the same directory
as *.py, but surely we don't want to write to the zip file. We could
write to the directory of the zip archive, but that would clutter it up,
not good if it is /usr/bin for example.

Failing to write the compiled files will make zip imports very slow, and
the user will probably not figure out what is wrong. So it is best to
put *.pyc and *.pyo in the archive with the *.py.

Efficiency

The only way to find files in a zip archive is linear search. So for
each zip file in sys.path, we search for its names once, and put the
names plus other relevant data into a static Python dictionary. The key
is the archive name from sys.path joined with the file name (including
any subdirectories) within the archive. This is exactly the name
generated by import.c, and makes lookup easy.

This same mechanism is used to speed up directory (non-zip) imports. See
below.

zlib

Compressed zip archives require zlib for decompression. Prior to any
other imports, we attempt an import of zlib. Import of compressed files
will fail with a message "missing zlib" unless zlib is available.

Booting

Python imports site.py itself, and this imports os, nt, ntpath, stat,
and UserDict. It also imports sitecustomize.py which may import more
modules. Zip imports must be available before site.py is imported.

Just as there are default directories in sys.path, there must be one or
more default zip archives too.

The problem is what the name should be. The name should be linked with
the Python version, so the Python executable can correctly find its
corresponding libraries even when there are multiple Python versions on
the same machine.

We add one name to sys.path. On Unix, the directory is
sys.prefix + "/lib", and the file name is
"python%s%s.zip" % (sys.version[0], sys.version[2]). So for Python 2.2
and prefix /usr/local, the path /usr/local/lib/python2.2/ is already on
sys.path, and /usr/local/lib/python22.zip would be added. On Windows,
the file is the full path to python22.dll, with "dll" replaced by "zip".
The zip archive name is always inserted as the second item in sys.path.
The first is the directory of the main.py (thanks Tim).

Directory Imports

The static Python dictionary used to speed up zip imports can be used to
speed up normal directory imports too. For each item in sys.path that is
not a zip archive, we call os.listdir, and add the directory contents to
the dictionary. Then instead of calling fopen() in a double loop, we
just check the dictionary. This greatly speeds up imports. If os.listdir
doesn't exist, the dictionary is not used.

Benchmarks

  Case   Original 2.2a3      Using os.listdir    Zip Uncomp   Zip Compr
  ------ ------------------- ------------------- ------------ ------------
  1      3.2 2.5 3.2->1.02   2.3 2.5 2.3->0.87   1.66->0.93   1.5->1.07
  2      2.8 3.9 3.0->1.32   Same as Case 1.
  3      5.7 5.7 5.7->5.7    2.1 2.1 2.1->1.8    1.25->0.99   1.19->1.13
  4      9.4 9.4 9.3->9.35   Same as Case 3.

Case 1: Local drive C:, sys.path has its default value. Case 2: Local
drive C:, directory with files is at the end of sys.path. Case 3:
Network drive, sys.path has its default value. Case 4: Network drive,
directory with files is at the end of sys.path.

Benchmarks were performed on a Pentium 4 clone, 1.4 GHz, 256 Meg. The
machine was running Windows 2000 with a Linux/Samba network server.
Times are in seconds, and are the time to import about 100 Lib modules.
Case 2 and 4 have the "correct" directory moved to the end of sys.path.
"Uncomp" means uncompressed zip archive, "Compr" means compressed.

Initial times are after a re-boot of the system; the time after "->" is
the time after repeated runs. Times to import from C: after a re-boot
are rather highly variable for the "Original" case, but are more
realistic.

Custom Imports

The logic demonstrates the ability to import using default searching
until a needed Python module (in this case, os) becomes available. This
can be used to bootstrap custom importers. For example, if "importer()"
in __init__.py exists, then it could be used for imports. The
"importer()" can freely import os and other modules, and these will be
satisfied from the default mechanism. This PEP does not define any
custom importers, and this note is for information only.

Implementation

A C implementation is available as SourceForge patch 492105. Superseded
by patch 652586 and current CVS.[2]

A newer version (updated for recent CVS by Paul Moore) is 645650.
Superseded by patch 652586 and current CVS.[3]

A competing implementation by Just van Rossum is 652586, which is the
basis for the final implementation of PEP 302. PEP 273 has been
implemented using PEP 302's import hooks.[4]

References

Copyright

This document has been placed in the public domain.

[1] Just van Rossum, New import hooks + Import from Zip files
https://bugs.python.org/issue652586

[2] Import from Zip archive, James C. Ahlstrom
https://bugs.python.org/issue492105

[3] Import from Zip Archive, Paul Moore
https://bugs.python.org/issue645650

[4] Just van Rossum, New import hooks + Import from Zip files
https://bugs.python.org/issue652586


text:
PEP: 789 Title: Preventing task-cancellation bugs by limiting yield in
async generators Author: Zac Hatfield-Dodds<EMAIL_ADDRESS>Nathaniel J.
Smith<EMAIL_ADDRESS>PEP-Delegate: Discussions-To:
https://discuss.python.org/t/preventing-yield-inside-certain-context-managers/1091
Status: Draft Type: Standards Track Created: 14-May-2024 Python-Version:
3.14

Abstract

Structured concurrency is increasingly popular in Python. Interfaces
such as the asyncio.TaskGroup and asyncio.timeout context managers
support compositional reasoning, and allow developers to clearly scope
the lifetimes of concurrent tasks. However, using yield to suspend a
frame inside such a context leads to situations where the wrong task is
canceled, timeouts are ignored, and exceptions are mishandled. More
fundamentally, suspending a frame inside a TaskGroup violates the
structured concurrency design principle that child tasks are
encapsulated within their parent frame.

To address these issues, this PEP proposes a new sys.prevent_yields()
context manager. When syntactically inside this context, attempting to
yield will raise a RuntimeError, preventing the task from yielding.
Additionally, a mechanism will be provided for decorators such as
@contextmanager to allow yields inside the decorated function.
sys.prevent_yields() will be used by asyncio and downstream libraries to
implement task groups, timeouts, and cancellation; and a related
mechanism by contextlib etc. to convert generators into context managers
which allow safe yields.

Background

Structured concurrency is increasingly popular in Python, in the form of
newer asyncio interfaces and third-party libraries such as Trio and
anyio. These interfaces support compositional reasoning, so long as
users never write a yield which suspends a frame while inside a cancel
scope.

A cancel scope is a context manager which can... cancel... whatever work
occurs within that context (...scope). In asyncio, this is implicit in
the design of with asyncio.timeout(): or
async with asyncio.TaskGroup() as tg:, which respectively cancel the
contained work after the specified duration, or cancel sibling tasks
when one of them raises an exception. The core functionality of a cancel
scope is synchronous, but the user-facing context managers may be either
sync or async.[1][2]

This structured approach works beautifully, unless you hit one specific
sharp edge: breaking the nesting structure by yielding inside a cancel
scope. This has much the same effect on structured control flow as
adding just a few cross-function gotos, and the effects are truly dire:

-   The wrong task can be canceled, whether due to a timeout, an error
    in a sibling task, or an explicit request to cancel some other task
-   Exceptions, including CancelledError, can be delivered to the wrong
    task
-   Exceptions can go missing entirely, being dropped instead of added
    to an ExceptionGroup

Problem statement

Here's the fundamental issue: yield suspends a call frame. It only makes
sense to yield in a leaf frame -- i.e., if your call stack goes like A
-> B -> C, then you can suspend C, but you can't suspend B while leaving
C running.

But, TaskGroup is a kind of "concurrent call" primitive, where a single
frame can have multiple child frames that run concurrently. This means
that if we allow people to mix yield and TaskGroup, then we can end up
in exactly this situation, where B gets suspended but C is actively
running. This is nonsensical, and causes serious practical problems
(e.g., if C raises an exception and A has returned, we have no way to
propagate it).

This is a fundamental incompatibility between generator control flow and
structured concurrency control flow, not something we can fix by
tweaking our APIs. The only solution seems to be to forbid yield inside
a TaskGroup.

Although timeouts don't leave a child task running, the close analogy
and related problems lead us to conclude that yield should be forbidden
inside all cancel scopes, not only TaskGroups. See just-deliver for
discussion.

Motivating examples

Let's consider three examples, to see what this might look like in
practice.

Leaking a timeout to the outer scope

Suppose that we want to iterate over an async iterator, but wait for at
most max_time seconds for each element. We might naturally encapsulate
the logic for doing so in an async generator, so that the call site can
continue to use a straightforward async for loop:

    async def iter_with_timeout(ait, max_time):
        try:
            while True:
                with timeout(max_time):
                    yield await anext(ait)
        except StopAsyncIteration:
            return

    async def fn():
        async for elem in iter_with_timeout(ait, max_time=1.0):
            await do_something_with(elem)

Unfortunately, there's a bug in this version: the timeout might expire
after the generator yields but before it is resumed! In this case, we'll
see a CancelledError raised in the outer task, where it cannot be caught
by the with timeout(max_time): statement.

The fix is fairly simple: get the next element inside the timeout
context, and then yield outside that context.

    async def correct_iter_with_timeout(ait, max_time):
        try:
            while True:
                with timeout(max_time):
                    tmp = await anext(ait)
                yield tmp
        except StopAsyncIteration:
            return

Leaking background tasks (breaks cancellation and exception handling)

Timeouts are not the only interface which wrap a cancel scope - and if
you need some background worker tasks, you can't simply close the
TaskGroup before yielding.

As an example, let's look at a fan-in generator, which we'll use to
merge the feeds from several "sensors". We'll also set up our mock
sensors with a small buffer, so that we'll raise an error in the
background task while control flow is outside the combined_iterators
generator.

    import asyncio, itertools

    async def mock_sensor(name):
        for n in itertools.count():
            await asyncio.sleep(0.1)
            if n == 1 and name == "b":  # 'presence detection'
                yield "PRESENT"
            elif n == 3 and name == "a":  # inject a simple bug
                print("oops, raising RuntimeError")
                raise RuntimeError
            else:
                yield f"{name}-{n}"  # non-presence sensor data

    async def move_elements_to_queue(ait, queue):
        async for obj in ait:
            await queue.put(obj)

    async def combined_iterators(*aits):
        """Combine async iterators by starting N tasks, each of
        which move elements from one iterable to a shared queue."""
        q = asyncio.Queue(maxsize=2)
        async with asyncio.TaskGroup() as tg:
            for ait in aits:
                tg.create_task(move_elements_to_queue(ait, q))
            while True:
                yield await q.get()

    async def turn_on_lights_when_someone_gets_home():
        combined = combined_iterators(mock_sensor("a"), mock_sensor("b"))
        async for event in combined:
            print(event)
            if event == "PRESENT":
                break
        print("main task sleeping for a bit")
        await asyncio.sleep(1)  # do some other operation

    asyncio.run(turn_on_lights_when_someone_gets_home())

When we run this code, we see the expected sequence of observations,
then a 'detection', and then while the main task is sleeping we trigger
that RuntimeError in the background. But... we don't actually observe
the RuntimeError, not even as the __context__ of another exception!

    >> python3.11 demo.py
    a-0
    b-0
    a-1
    PRESENT
    main task sleeping for a bit
    oops, raising RuntimeError

    Traceback (most recent call last):
      File "demo.py", line 39, in <module>
        asyncio.run(turn_on_lights_when_someone_gets_home())
      ...
      File "demo.py", line 37, in turn_on_lights_when_someone_gets_home
        await asyncio.sleep(1)  # do some other operation
      File ".../python3.11/asyncio/tasks.py", line 649, in sleep
        return await future
    asyncio.exceptions.CancelledError

Here, again, the problem is that we've yielded inside a cancel scope;
this time the scope which a TaskGroup uses to cancel sibling tasks when
one of the child tasks raises an exception. However, the CancelledError
which was intended for the sibling task was instead injected into the
outer task, and so we never got a chance to create and raise an
ExceptionGroup(..., [RuntimeError()]).

To fix this, we need to turn our async generator into an async context
manager, which yields an async iterable - in this case a generator
wrapping the queue; in future perhaps the queue itself:

    async def queue_as_aiterable(queue):
        # async generators that don't `yield` inside a cancel scope are fine!
        while True:
            try:
                yield await queue.get()
            except asyncio.QueueShutDown:
                return

    @asynccontextmanager  #  yield-in-cancel-scope is OK in a context manager
    async def combined_iterators(*aits):
        q = asyncio.Queue(maxsize=2)
        async with asyncio.TaskGroup() as tg:
            for ait in aits:
                tg.create_task(move_elements_to_queue(ait, q))
            yield queue_as_aiterable(q)

    async def turn_on_lights_when_someone_gets_home():
        ...
        async with combined_iterators(...) as ait:
            async for event in ait:
                ...

In a user-defined context manager

Yielding inside a cancel scope can be safe, if and only if you're using
the generator to implement a context manager[3] - in this case any
propagating exceptions will be redirected to the expected task.

We've also implemented the ASYNC101 linter rule in flake8-async, which
warns against yielding in known cancel scopes. Could user education be
sufficient to avoid these problems? Unfortunately not: user-defined
context managers can also wrap a cancel scope, and it's infeasible to
recognize or lint for all such cases.

This regularly arises in practice, because 'run some background tasks
for the duration of this context' is a very common pattern in structured
concurrency. We saw that in combined_iterators() above; and have seen
this bug in multiple implementations of the websocket protocol:

    async def get_messages(websocket_url):
        # The websocket protocol requires background tasks to manage the socket heartbeat
        async with open_websocket(websocket_url) as ws:  # contains a TaskGroup!
            while True:
                yield await ws.get_message()

    async with open_websocket(websocket_url) as ws:
        async for message in get_messages(ws):
            ...

Specification

To prevent these problems, we propose:

1.  a new context manager, with sys.prevent_yields(reason): ... which
    will raise a RuntimeError if you attempt to yield while inside
    it.[4] Cancel-scope-like context managers in asyncio and downstream
    code can then wrap this to prevent yielding inside their with-block.
2.  a mechanism by which generator-to-context-manager decorators can
    allow yields across one call. We're not yet sure what this should
    look like; the leading candidates are:
    a.  a code-object attribute, fn.__code__.co_allow_yields = True, or
    b.  some sort of invocation flag, e.g. fn.__invoke_with_yields__, to
        avoid mutating a code object that might be shared between
        decorated and undecorated functions

Implementation - tracking frames

The new sys.prevent_yields context manager will require interpreter
support. For each frame, we track the entries and exits of this context
manager.

We're not particularly attached to the exact representation; we'll
discuss it as a stack (which would support clear error messages), but
more compact representations such as pair-of-integers would also work.

-   When entering a newly-created or resumed frame, initialize empty
    stacks of entries and exits.
-   When returning from a frame, merge these stacks into that of the
    parent frame.
-   When yielding:
    -   if entries != [] and not frame.allow_yield_flag, raise a
        RuntimeError instead of yielding (the new behavior this PEP
        proposes)
    -   otherwise, merge stacks into the parent frame as for a return.

Because this is about yielding frames within a task, not switching
between tasks, syntactic yield and yield from should be affected, but
await expressions should not.

We can reduce the overhead by storing this metadata in a single stack
per thread for all stack frames which are not generators.

Worked examples

No-yield example

In this example, we see multiple rounds of the stack merging as we
unwind from sys.prevent_yields, through the user-defined ContextManager,
back to the original Frame. For brevity, the reason for preventing
yields is not shown; it is part of the "1 enter" state.

[image]

With no yield we don't raise any errors, and because the number of
enters and exits balance the frame returns as usual with no further
tracking.

Attempts-to-yield example

In this example, the Frame attempts to yield while inside the
sys.prevent_yields context. This is detected by the interpreter, which
raises a RuntimeError instead of suspending the frame.

[image]

Allowed-to-yield example

In this example, a decorator has marked the Frame as allowing yields.
This could be @contextlib.contextmanager or a related decorator.

[image]

When the Frame is allowed to yield, the entry/exit stack is merged into
the parent frame's stack before suspending. When the Frame resumes, its
stack is empty. Finally, when the Frame exits, the exit is merged into
the parent frame's stack, rebalancing it.

This ensures that the parent frame correctly inherits any remaining
sys.prevent_yields state, while allowing the Frame to safely suspend and
resume.

Allowing yield for context managers

TODO: this section is a placeholder, pending a decision on the mechanism
for ``@contextmanager`` to re-enable yields in the wrapped function.

-   Explain and show a code sample of how @asynccontextmanager sets the
    flag

Note that third-party decorators such as @pytest.fixture demonstrate
that we can't just have the interpreter special-case contextlib.

Behavior if sys.prevent_yields is misused

While unwise, it's possible to call sys.prevent_yields.__enter__ and
.__exit__ in an order that does not correspond to any valid nesting, or
get an invalid frame state in some other way.

There are two ways sys.prevent_yields.__exit__ could detect an invalid
state. First, if yields are not prevented, we can simply raise an
exception without changing the state. Second, if an unexpected entry is
at the top of the stack, we suggest popping that entry and raising an
exception -- this ensures that out-of-order calls will still clear the
stack, while still making it clear that something is wrong.

(and if we choose e.g. an integer- rather than stack-based
representation, such states may not be distinguishable from correct
nesting at all, in which case the question will not arise)

Anticipated uses

In the standard library, sys.prevent_yields could be used by
asyncio.TaskGroup, asyncio.timeout, and asyncio.timeout_at. Downstream,
we expect to use it in trio.CancelScope, async fixtures (in pytest-trio,
anyio, etc.), and perhaps other places.

We consider use-cases unrelated to async correctness, such as preventing
decimal.localcontext from leaking out of a generator, out of scope for
this PEP.

The generator-to-context-manager support would be used by
@contextlib.(async)contextmanager, and if necessary in (Async)ExitStack.

Backwards Compatibility

The addition of the sys.prevent_yields context manager, changes to
@contextlib.(async)contextmanager, and corresponding interpreter support
are all fully backwards-compatible.

Preventing yields inside asyncio.TaskGroup, asycio.timeout, and
asyncio.timeout_at would be a breaking change to at least some code in
the wild, which (however unsafe and prone to the motivating problems
above) may work often enough to make it into production.

We will seek community feedback on appropriate deprecation pathways for
standard-library code, including the suggested length of any deprecation
period. As an initial suggestion, we could make suspending stdlib
contexts emit a DeprecationWarning only under asyncio debug mode in
3.14; then transition to warn-by-default and error under debug mode in
3.15; and finally a hard error in 3.16.

Irrespective of stdlib usage, downstream frameworks would adopt this
functionality immediately.

How widespread is this bug?

We don't have solid numbers here, but believe that many projects are
affected in the wild. Since hitting a moderate and a critical bug
attributed to suspending a cancel scope in the same week at work, we've
used static analysis with some success. Three people Zac spoke to at
PyCon recognized the symptoms and concluded that they had likely been
affected.

TODO: run the ASYNC101 lint rule across ecosystem projects, e.g. the
aio-libs packages, and get some sense of frequency in widely-used PyPI
packages? This would help inform the break/deprecation pathways for
stdlib code.

How to Teach This

Async generators are very rarely taught to novice programmers.

Most intermediate and advanced Python programmers will only interact
with this PEP as users of TaskGroup, timeout, and @contextmanager. For
this group, we expect a clear exception message and documentation to be
sufficient.

-   A new section will be added to the developing with asyncio page,
    which briefly states that async generators are not permitted to
    yield when inside a "cancel scope" context, i.e. TaskGroup or
    timeout context manager. We anticipate that the problem-restatement
    and some parts of the motivation section will provide a basis for
    these docs.
    -   When working in codebases which avoid async generators
        entirely[5], we've found that an async context manager yielding
        an async iterable is a safe and ergonomic replacement for async
        generators -- and avoids the delayed-cleanup problems described
        in PEP 533, which this proposal does not address.
-   In the docs for each context manager which wraps a cancel scope, and
    thus now sys.prevent_yields, include a standard sentence such as "If
    used within an async generator, [it is an error to yield inside this
    context manager]." with a hyperlink to the explanation above.

For asyncio, Trio, curio, or other-framework maintainers who implement
cancel scope semantics, we will ensure that the documentation of
sys.prevent_yields gives a full explanation distilled from the solution
and implementation sections of this PEP. We anticipate consulting most
such maintainers for their feedback on the draft PEP.

Rejected alternatives

PEP 533, deterministic cleanup for iterators

PEP 533 proposes adding __[a]iterclose__ to the iterator protocol,
essentially wrapping a with [a]closing(ait) around each (async) for
loop. While this would be useful for ensuring timely and deterministic
cleanup of resources held by iterators, the problem it aims to solve, it
does not fully address the issues that motivate this PEP.

Even with PEP 533, misfired cancellations would still be delivered to
the wrong task and could wreak havoc before the iterator is closed.
Moreover, it does not address the fundamental structured concurrency
problem with TaskGroup, where suspending a frame that owns a TaskGroup
is incompatible with the model of child tasks being fully encapsulated
within their parent frame.

Deprecate async generators entirely

At the 2024 language summit, several attendees suggested instead
deprecating async generators in toto. Unfortunately, while the
common-in-practice cases all use async generators, Trio code can trigger
the same problem with standard generators:

    # We use Trio for this example, because while `asyncio.timeout()` is async,
    # Trio's CancelScope type and timeout context managers are synchronous.
    import trio

    def abandon_each_iteration_after(max_seconds):
        # This is of course broken, but I can imagine someone trying it...
        while True:
            with trio.move_on_after(max_seconds):
                yield

    @trio.run
    async def main():
        for _ in abandon_each_iteration_after(max_seconds=1):
            await trio.sleep(3)

If it wasn't for the bug in question, this code would look pretty
idiomatic -but after about a second, instead of moving on to the next
iteration it raises:

    Traceback (most recent call last):
      File "demo.py", line 10, in <module>
        async def main():
      File "trio/_core/_run.py", line 2297, in run
        raise runner.main_task_outcome.error
      File "demo.py", line 12, in main
        await trio.sleep(3)
      File "trio/_timeouts.py", line 87, in sleep
        await sleep_until(trio.current_time() + seconds)
      ...
      File "trio/_core/_run.py", line 1450, in raise_cancel
        raise Cancelled._create()
    trio.Cancelled: Cancelled

Furthermore, there are some non-cancel-scope synchronous context
managers which exhibit related problems, such as the abovementioned
decimal.localcontext. While fixing the example below is not a goal of
this PEP, it demonstrates that yield-within-with problems are not
exclusive to async generators:

    import decimal

    def why_would_you_do_this():
        with decimal.localcontext(decimal.Context(prec=1)):
            yield

    one = decimal.Decimal(1)
    print(one / 3)  # 0.3333333333333333333333333333
    next(gen := why_would_you_do_this())
    print(one / 3)  # 0.3

While I've had good experiences in async Python without async generators
[6], I'd prefer to fix the problem than remove them from the language.

Can't we just deliver exceptions to the right place?

If we implemented PEP 568 (Generator-sensitivity for Context Variables;
see also PEP 550), it would be possible to handle exceptions from
timeouts: the event loop could avoid firing a CancelledError until the
generator frame which contains the context manager is on the stack -
either when the generator is resumed, or when it is finalized.

This can take arbitrarily long; even if we implemented PEP 533 to ensure
timely cleanup on exiting (async) for-loops it's still possible to drive
a generator manually with next/send.

However, this doesn't address the other problem with TaskGroup. The
model for generators is that you put a stack frame in suspended
animation and can then treat it as an inert value which can be stored,
moved around, and maybe discarded or revived in some arbitrary place.
The model for structured concurrency is that your stack becomes a tree,
with child tasks encapsulated within some parent frame. They're
extending the basic structured programming model in different, and
unfortunately incompatible, directions.

Suppose for example that suspending a frame containing an open TaskGroup
also suspended all child tasks. This would preserve the 'downward'
structured concurrency, in that children remain encapsulated - albeit at
the cost of deadlocking both of our motivating examples, and much
real-world code. However, it would still be possible to resume the
generator in a different task, violating the 'upwards' invariant of
structured concurrency.

We don't think it's worth adding this much machinery to handle cancel
scopes, while still leaving task groups broken.

Alternative implementation - inspecting bytecode

Jelle Zijlstra has sketched an alternative, where sys.prevent_yields
inspects the bytecode of callers until satisfied that there is no yield
between the calling instruction pointer and the next context exit. We
expect that support for syntatically-nested context managers could be
added fairly easily.

However, it's not yet clear how this would work when user-defined
context managers wrap sys.prevent_yields. Worse, this approach ignores
explicit calls to __enter__() and __exit__(), meaning that the context
management protocol would vary depending on whether the with statement
was used.

The 'only pay if you use it' performance cost is very attractive.
However, inspecting frame objects is prohibitively expensive for core
control-flow constructs, and causes whole-program slowdowns via
de-optimization. On the other hand, adding interpreter support for
better performance leads back to the same pay-regardless semantics as
our preferred solution above.

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] While cancel scopes are implicit in asyncio, the analogous
trio:trio.fail_after (sync) and trio:trio.open_nursery (async) context
managers literally wrap an instance of trio:trio.CancelScope. We'll
stick with asyncio for examples here, but say "cancel scope" when
referring to the framework-independent concept.

[2] A TaskGroup is not _only a cancel scope, but preventing yields would
resolve their further problem too. See just-deliver.

[3] via e.g. contextlib.[async]contextmanager, or moral equivalents such
as @pytest.fixture

[4] Note that this prevents yields in both sync and async generators, so
that downstream frameworks can safely define sync cancel scope countexts
such as trio:trio.fail_after.

[5] see Zac's experience report here

[6] see Zac's experience report here