text:
PEP: 605 Title: A rolling feature release stream for CPython Version:
$Revision$ Last-Modified: $Date$ Author: Steve Dower
<EMAIL_ADDRESS>Alyssa Coghlan<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-605-a-rolling-feature-release-stream-for-cpython/2418
Status: Rejected Type: Informational Content-Type: text/x-rst Created:
20-Sep-2019 Python-Version: 3.9 Post-History: 01-Oct-2019, 06-Oct-2019,
20-Oct-2019

Rejection Notice

This PEP was rejected in favour of PEP 602. The potential alpha/beta
alternation was deemed too confusing and the two year cadence between
releases deemed too long.

Abstract

For a long time, CPython's nominal feature release cadence has been
"every 18-24 months", and in recent years, has been pretty consistently
on the "18 month" end of that window. PEP 607 provides some common
background on the problems that arise due to that choice of cadence, as
well as some of the risks that need to be accounted for when proposing
to change it.

The proposal in this PEP aims to allow CPython's user base to
self-select into two distinct but overlapping groups:

-   users of stable feature releases (and their associated maintenance
    release streams) that are published every 24 months; and
-   early adopters of a new rolling release stream that takes the place
    of the previous CPython pre-release process

As part of this proposal, the usage guidance given for beta releases
would become "suitable for production use only in environments with
sufficiently robust compatibility testing and operational monitoring
capabilities", rather than the currently unqualified "not for production
use".

Similarly, the guidance given for alpha releases would be amended to
state "intended for library compatibility testing and the creation of
ABI compatible binary artifacts", rather than simply saying "not for
production use".

The PEP authors believe these outcomes can be achieved by amending
CPython's pre-release management process as described in the Proposal
section below.

This PEP also proposes that the frequency of X.Y.0 releases be adjusted
to begin each new release series in August every two years (starting in
2021, around two years after the release of Python 3.8.0).

Example Future Release Schedules

Under this proposal, Python 3.9.0a1 would be released in December 2019,
two months after the Python 3.8.0 baseline feature release in October
2019.

Assuming no further breaking changes were made to the full CPython ABI,
the 3.9.0b2 release would then follow 2 months later in February 2020,
continuing through to 3.9.0b9 in April 2021.

Any time a breaking change to the full CPython ABI was introduced, the
first pre-release that included it would be marked as an alpha release.

3.9.0rc1 would be published in June 2021, 3.9.0rc2 in July 2021, and
then the full release published as 3.9.0 in August 2021.

The cycle would start over again in October 2021, with the publication
of 3.10.0a1 (4 months after the creation of the 3.9.x maintenance
branch).

The exact schedule of maintenance releases would be up to the release
team, but assuming maintenance releases of 3.9.x were also to occur
every other month (offset from the 3.10.0 beta releases), the overall
release timeline would look like:

-   2019-12: 3.9.0a1
-   2020-02: 3.9.0b2
-   ... beta (or alpha) releases every other month
-   2021-04: 3.9.0b9
-   2021-06: 3.9.0rc1 (feature freeze, ABI freeze, pyc format freeze)
-   2021-07: 3.9.0rc2
-   2021-08: 3.9.0
-   2021-09: 3.9.1, 3.8.x (final 3.8.x binary maintenance release)
-   2021-10: 3.10.0a1
-   2021-11: 3.9.2
-   2021-12: 3.10.0b2
-   ... beta (or alpha) and maintenance releases continue in alternate
    months
-   2023-04: 3.10.0b10
-   2023-05: 3.9.11
-   2023-06: 3.10.0rc1 (feature freeze, ABI freeze, pyc format freeze)
-   2023-07: 3.10.0rc2, 3.9.12
-   2023-08: 3.10.0
-   2023-09: 3.10.1, 3.9.13 (final 3.9.x binary maintenance release)
-   2023-10: 3.11.0a1
-   2023-12: 3.11.0b2
-   ... etc

If we assume two additional pre-releases were made that introduced
breaking changes to the full CPython ABI in the 3.9.0a5 and 3.9.0a7
releases, then the overall calendar would look like:

[]

There are always two or three active maintenance branches in this model,
which preserves the status quo in that respect. The major difference is
that we would start encouraging publishers to provide pre-built binaries
for the pre-freeze rolling releases in addition to providing them for
the stable maintenance branches.

[]

Package publishers targeting the full CPython ABI that choose to provide
pre-built binaries for the rolling pre-freeze releases would at least
need to build new wheel archives following the 3.9.0a1 release. Whether
they needed to publish updated binaries after subsequent alpha releases
(e.g. 3.9.0a5 or 3.9.0a7 releases in the example timeline) would depend
on whether or not they were actually affected by the ABI breaks in those
later releases.

As with the status quo, all package publishers wishing to provide
pre-built binaries for the final release will need to build new wheel
archives following the ABI freeze date. Unlike the status quo, this date
will be clearly marked by the publication of the first release
candidate, and it will occur early enough to give publishers a couple of
months to get ready for the final release.

Example Future Release Announcements

If this PEP is accepted, the primary channels used to communicate the
updated pre-release management process to end users would be the Python
3.9 What's New document, and the release announcements themselves.

This section provides initial drafts of text that could be used for
those purposes.

Suggested "What's New in Python 3.9" Entry

The following subsection would be added to the Python 3.9 What's New
document, and then linked from each of the Python 3.9 alpha and beta
announcements.

PEP 605: Changes to the pre-release management process

As detailed in PEP 605, the pre-release management process has been
updated to produce a rolling series of beta releases that are considered
suitable for production use in environments with sufficiently robust
integration testing and operational monitoring capabilities.

Under this new rolling model, the alpha and beta releases are
intermingled as part of a combined "pre-freeze" period, with alpha
releases indicating breaks in the full CPython ABI that may require
recompilation of extension modules or embedding applications, and beta
releases indicating full binary compatibility with the immediately
preceding pre-release.

Unlike previous releases, publishing pre-built binaries for 3.9.0 alpha
and beta releases is actively encouraged, as a new pre-release ABI flag
("p") is now set when building and loading extension modules prior to
the full CPython ABI freeze, ensuring that all such pre-freeze extension
module builds will be ignored by post-freeze interpreter builds.

The full CPython ABI will be frozen, and the pre-release flag dropped
from the ABI flags, in 3.9.0rc1, which is expected to occur 2 months
prior to the final 3.9.0 release (refer to the release schedule in PEP
596 for exact target dates).

For application developers, migrating to the rolling release stream
provides the opportunity to be actively involved in the design and
development of enhancements to the standard library and reference
interpreter prior to the next stable release. It also provides the
opportunity to benefit from interpreter performance enhancements up to a
year or more before they become available in a stable release.

For library developers that publish pre-built wheel archives, opting in
to supporting the 3.9.x rolling release stream in addition to the 3.8
stable release series requires no specific action if the project is
already publishing either pure Python wheels (tagged as py3-none-any),
or builds against the stable C ABI (tagged as cp38-abi3-<platform>, or
the equivalent from an earlier CPython 3.x release). These same wheel
archives will also be usable on the subsequent 3.9 stable release
series.

For library developers that publish pre-built wheel archives that are
built against the full CPython ABI, the binaries for the 3.9 stable
release series will need to be built after the full CPython ABI freeze
(i.e. using 3.9.0rc1 or later).

Developers of these libraries may also opt in to supporting the rolling
release stream by building against the 3.9.0a1 release (or a subsequent
beta release) and publishing the result as normal.

In the ideal case, binaries built this way will continue working all the
way through to the last pre-freeze release. However, if the project is
affected by a change in the full CPython C ABI during the pre-freeze
period, then it will be necessary to publish a maintenance update that
rebuilds the affected binaries against the alpha release that changed
the relevant interface. In these cases, a corresponding Python-Requires
entry should be added to the project metadata. For example, if a project
is affected by an ABI change introduced in 3.9.0a5, then the
Python-Requires entry to add would be:

    Python-Requires: >= "3.9.0b6"; python_version == "3.9" and full_python_version != "3.9.0a5"

(This additional metadata ensures that the updated version won't be
installed on earlier pre-releases in the 3.9 series that offer an older
variant of the ABI)

As for application developers, library developers that choose to support
the rolling release stream will have the opportunity to provide feedback
on new and updated API designs before they're locked down for multiple
years in a stable release (or before they're included as a provisional
API in a stable release series).

Example announcement text for the 3.9.0a1 release

This is the first preview release of Python 3.9. As an alpha release, it
is intended for library and application compatibility testing and the
creation of ABI compatible binary artifacts. It is not recommended for
use in production environments.

Changes to the pre-release management process

CPython has switched to a new pre-release management process that is
designed to produce a rolling series of beta releases that are
considered suitable for production use in environments with sufficiently
robust integration testing and operational monitoring capabilities.
Refer to the Python 3.9 What's New document (hyperlinked to relevant
section) for details.

Major new features of the 3.9 series, compared to 3.8

Many new features for Python 3.9 are still being planned and written.
Among the major new features and changes already implemented:

-   ...
-   (Hey, fellow core developer or user of the rolling release stream,
    if a feature you find important is missing from this list, let <the
    release manager> know.)

The next pre-release of Python 3.9 is expected to be 3.8.0b2, currently
scheduled for 2020-02-02.

Example announcement text for the 3.9.0b2 release

This is the second preview release of Python 3.9. As a beta release, it
is fully binary compatible with the preceding 3.9.0a1 release. It is
recommended for production use only in environments with sufficiently
robust integration testing and operational monitoring capabilities.

(Remainder as per 3.9.0a1 announcement, with updates for implemented
changes and the next expected release being 3.9.0b3)

Example announcement text for 3.9.0a5 (a mid-stream alpha release)

This is the fifth preview release of Python 3.9. As an alpha release, it
is NOT fully binary compatible with the preceding 3.9.0b4 release. This
release is intended for library and application compatibility testing
and the creation of ABI compatible binary artifacts. It is not
recommended for use in production environments.

Breaking changes in the full CPython ABI between 3.9.0b4 and 3.9.0a5

-   new field ob_example added to the PyObject struct
-   provisional field tp_example removed from the PyTypeObject struct

Projects that are supporting the rolling release stream and require a
rebuild to restore binary compatibility should add the following
metadata to their updated release:

    Python-Requires: >= "3.9.0b6"; python_version == "3.9" and full_python_version != "3.9.0a5"

(Remainder as per 3.9.0a1 announcement, with updates for implemented
changes and the next expected release being 3.9.0b6)

Example announcement text for 3.9.0rc1

This is the first release candidate for Python 3.9. As a release
candidate, this release is now feature complete, the full CPython ABI is
now frozen, and the pre-release marker has been removed from the ABI
compatibility flags. It is recommended for production use only in
environments with sufficiently robust integration testing and
operational monitoring capabilities.

Preparation for the final 3.9.0 release

With the full CPython ABI now frozen, library developers targeting that
ABI are encouraged to build and publish binaries for the stable 3.9.x
series.

Application developers that have not been testing against the rolling
release stream are encouraged to test their applications against the
release candidate and report any compatibility regressions not already
mentioned in the Porting Guide (hyperlinked to relevant What's New
section).

A second release candidate is planned for 2021-07-02, and then the final
3.9.0 release is planned for 2021-08-02.

Major new features of the 3.9 series, compared to 3.8

Some of the major new features and changes in this release:

-   ...
-   (Hey, fellow core developer or user of the rolling release stream,
    if a feature you find important is missing from this list, let <the
    release manager> know.)

Motivation

The current CPython pre-release and release management processes were
developed in an era where automated continuous integration and
operational monitoring systems were still relatively immature. Since
that time, many organisations have adopted deployment models that allow
them to incorporate new CPython feature releases without adding
substantially more risk than they incur for any other code change. Newer
deployment models, such as lightweight task specific application
containers, also make it easier to combine an application with a
language runtime in a CI pipeline, and then keep them together until the
entire container image is later replaced by an updated one.

In light of those changes in the wider environment, PEP 602 has proposed
reducing the feature delivery latency for the Python standard library
and CPython reference interpreter by increasing the frequency of CPython
feature releases from every 18-24 months to instead occur every 12
months.

Unfortunately, for many organisations, the cost of adopting a new Python
release doesn't automatically scale down with a reduced number of
changes in the release, as the primary costs aren't associated with
resolving any discovered issues; the primary costs are associated with
the search for issues. This search may involve manual testing of
software systems, human review of written materials, and other
activities where the time required scales with the size of the existing
system, rather than with the number of changes between the versions of
Python.

For third party library developers, costs are primarily associated with
the number of distinct Python versions in widespread usage. This
currently tends to be influenced by a combination of which releases are
still actively maintained by python-dev, and which releases are the
latest versions offered by particular redistributors (with the Debian,
Ubuntu LTS, and RHEL/CentOS system Python versions being particularly
popular development targets). In addition to the basic CI cost of
testing against more Python versions, having more variants in widespread
use can make it more difficult to determine when a fault report is an
actual error in the project, or an issue in the reporting user's
environment.

PEP 602 proposes that affected organisations and projects simply switch
to adopting every second or third CPython release, rather than
attempting to adopt every release, but that creates its own set of new
problems to be resolved, both practical (e.g. deprecations would need to
cover more than one release if we're expecting users to routinely skip
releases) and cultural (e.g. with a larger number of versions in active
use, there is a much higher chance that open source library maintainers
will receive bug reports that only occur on Python versions that they're
not using themselves).

PEP 598 was an initial attempt by one of the authors of this PEP to
propose an alternative scheme to reduce feature delivery latency by
adopting a semantic versioning style policy that allowed for the
incremental delivery of backwards compatible features within a release
series, until that series reached feature complete status. That variant
still had the undesirable consequence of imposing visible changes on end
users that are happy enough with the current release management model.

This PEP takes the view that both PEP 598 and PEP 602 share a common
flaw: they are attempting to satisfy the needs of two quite distinct
audiences within the constraints of a single release model, which
results in conflicting design requirements, and the need for awkward
trade-offs between those conflicting requirements. The proposal in this
PEP aims to avoid that flaw by proposing the creation of two distinct
production-ready release streams, with the existing release stream being
largely left alone, while the new release stream is tailored towards the
audience that would most benefit from a reduction in feature delivery
latency.

Aims of this Proposal

The proposal in this PEP arises from making the following key
assumptions:

-   the vast majority of Python's users aren't actively clamouring for
    new language and runtime level features, and instead only upgrade
    when either the version they were previously using is no longer
    supported, when their Python provider switches to offering them a
    newer version by default, or when enough changes that are of
    interest to them have accumulated to make a compelling case for
    upgrading
-   for many users of new releases, much of the work that arises when
    adopting a new release doesn't arise from compatibility issues at
    the language level, but from compatibility issues at the component
    installation level (i.e. filename and installation path changes)
-   that there's a subset of Python's user base that would be willing to
    run production-ready pre-releases (akin to Windows Insider or Debian
    testing builds) for at least some of their use cases

The core of the proposal in this PEP is changing the CPython pre-release
process to produce a rolling stream of incremental feature releases at a
regular cadence, and to ensure that most of those builds offer a
sufficient level of stability as to be suitable for use in appropriately
managed production systems.

By adopting this approach, the proposal aims to provide an improved
outcome for almost all Python users and contributors:

-   for users of the new incremental feature release stream, targeting
    the pre-release phase allows for even lower feature delivery latency
    than the annual cadence proposed in PEP 602;
-   for core developers working on new features, increased frequency and
    adoption of pre-releases should improve pre-release feedback cycles;
-   for users of the established release stream, the increased adoption
    and improved feedback cycles during the pre-release period should
    result in increased feature maturity at the time of its first X.Y.0
    release, as well as higher levels of ecosystem readiness;
-   for Python library maintainers, the rolling stream of pre-releases
    will hopefully provide more opportunities to identify and resolve
    design issues before they make it into a full stable release than is
    offered by the current pre-release management process; and
-   for developers of alternative Python implementations, the rolling
    stream of pre-releases may provide an additional incentive for
    extension module authors to migrate from the full CPython ABI to the
    Python stable ABI, which would also serve to make more of the
    ecosystem compatible with implementations that don't emulate the
    full CPython C API.

That said, it is acknowledged that not all the outcomes of this proposal
will be beneficial for all members of the wider Python ecosystem:

-   for Python library maintainers, both this PEP and PEP 602 would
    likely result in user pressure to support the faster release
    cadence. While this PEP attempts to mitigate that by clearly marking
    which pre-releases include potentially breaking changes to the full
    CPython C ABI, and PEP 602 attempts to mitigate it by keeping the
    minimum time between full releases at 12 months, it isn't possible
    to eliminate this downside completely;
-   for third party extension module maintainers, both this PEP and PEP
    602 would likely result in user pressure to start supporting the
    stable ABI in order to provide wheel archives that work on the new
    version as soon as it is available. Whether that's a net negative or
    not will depend on how the request is presented to them (it could be
    a positive if the request comes in the form of a courteous
    contribution to their project from a developer interested in
    supporting the rolling pre-freeze releases);
-   for some users of the established release stream that rely on the
    availability of pre-built wheel archives, switching to adopting a
    new release every 12 months may be an acceptable rate increase,
    while moving consistently to the 24 month end of the historical
    18-24 month cadence would be an undesirable rate reduction relative
    to the 18-month cycle used for recent releases. Whether this
    proposal would be a net negative for these users will depend on
    whether or not we're able to persuade library maintainers that it's
    worth their while to support the upcoming stable release throughout
    its pre-freeze period, rather than waiting until its API and ABI
    have been frozen.

Proposal

The majority of the proposed changes in this PEP only affect the
handling of pre-release versions. The one change affecting full release
versions is a suggested change to their cadence.

Two year cadence for stable releases

With the rolling pre-freeze releases available to users that are looking
to use leading edge versions of the reference interpreter and standard
library, this PEP proposes that the frequency of X.Y.0 releases be
adjusted to publish a new stable release in August every two years
(starting in 2021, around two years after the release of Python 3.8.0).

This change is arguably orthogonal to the proposed changes to the
handling of the pre-freeze release period, but the connection is that
without those pre-release management changes, the downsides of a
two-year full release cadence would probably outweigh the upsides,
whereas the opposite is true for a 12-month release cadence (i.e. with
the pre-release management changes proposed in this PEP in place, the
downsides of a 12-month full release cadence would outweigh the
upsides).

Merging of the alpha and beta phases into a "pre-freeze" phase

Rather than continuing the status quo where the pre-release alpha and
beta phases are distinct and sequential, this PEP proposes that they
instead be combined into a single "pre-freeze" phase with a
monotonically increasing serial number on the releases.

Rather than denoting distinct phases, the "alpha" and "beta" names would
instead indicate whether or not the release contains breaking changes to
the full CPython C ABI:

-   "alpha" releases would be "ABI breaking" releases where extension
    modules built against the full CPython ABI in the preceding
    pre-release are not necessarily going to load correctly
-   "beta" releases would be "binary compatible" releases, where
    extension modules built against the full CPython ABI in the
    preceding pre-release are expected to load correctly, as long as
    those modules abide by the following additional criteria:
    -   the module must not be using any provisional or private C APIs
        (either from the previous stable release series, or the in
        development pre-release series) that were removed in this beta
        release, or were changed in an ABI incompatible way
    -   the module must not be using any C APIs that were deprecated in
        the previous stable release series, and removed in this beta
        release

Pre-freeze phase duration and cadence

Rather than being released monthly for a period of a few months while
preparing a new X.Y.0 release, pre-freeze releases would instead be
consistently published every two months.

The only time this would not be the case is during the two month release
candidate period for an upcoming X.Y.0 release (see the release
candidate section below for more details). This means two otherwise
scheduled releases would be skipped (one corresponding with the first
release candidate date, one with the final release date).

The pre-freeze phase would typically be expected to start 2 months after
the preceding stable X.Y.0 release.

The first pre-freeze release for any new release series will always be
X.Y.0a1 (as there is no preceding release with the same ABI version
markers to judge binary compatibility against).

Pre-freeze releases would gain an additional flag in their C ABI
compatibility markers to avoid binary compatibility issues with the
eventual stable release.

Release policy for beta releases

This PEP proposes that the policy for beta releases be set as follows:

-   as with current beta releases, the stable BuildBot fleet is expected
    to be green prior to preparation and publication of the beta release
-   as with current beta releases, the release manager is expected to
    review open release blocker issues prior to preparation and
    publication of the beta release
-   as with current beta releases, any additions to the abi3 stable C
    ABI would be expected to become a permanent part of that ABI unless
    and until that stable ABI version is retired completely (Note: there
    are no current plans to increment the stable ABI version)
-   unlike current beta releases, beta releases under this PEP would not
    be considered feature complete for the next X.Y.0 release
-   unlike current beta releases, all APIs added since the last CPython
    feature release (other than additions to the stable C ABI) would be
    considered provisional
-   unlike current beta releases, beta releases under this PEP would be
    prepared and published from the master development branch
-   unlike current alpha or beta releases, beta releases under this PEP
    would be required to be fully ABI compatible with the immediately
    preceding pre-release in the series (excluding any changes to
    provisional APIs, or the removal of APIs that were deprecated in the
    previous release series)

Release policy for alpha releases

This PEP proposes that the policy for alpha releases be set as follows:

-   as with current alpha releases, the stable BuildBot fleet is
    expected to be green prior to preparation and publication of the
    alpha release
-   as with current alpha releases, the release manager is expected to
    review open release blocker issues prior to preparation and
    publication of the beta release
-   unlike current alpha release, the release manager would be expected
    to target a similar level of stability to the current beta releases,
    even for the alpha releases

Under this PEP, an alpha release would be published whenever it isn't
possible to publish a release that satisfies the criteria for a beta
release, and allowing some additional time before making the release
won't resolve the issue.

It is expected that the full CPython API changing in a way that breaks
ABI compatibility (for example, a field may have been added to or
removed from a public struct definition) will be the most likely reason
for publishing additional alpha releases beyond the initial
compatibility tag defining X.Y.0a1 release, but the decision for any
particular release rests with the release manager.

Release candidate policy, phase duration, and cadence

Given the proposed changes to the alpha and beta release phases, the
release candidate phase would see the following related adjustments:

-   Feature freeze, ABI freeze, pyc file format freeze, and maintenance
    branch creation would all correspond with the creation of X.Y.0rc1
    (currently these occur across a mixture of X.Y.0b1, the last beta
    release, and X.Y.0rc1)
-   The X.Y.0 release candidate period would be extended from 3 weeks to
    2 months
-   There would normally be two release candidates issued a month apart,
    but additional candidates may be published at the release manager's
    discretion
-   The final X.Y.0 release would occur between 1 and 4 weeks after the
    final release candidate (depending if additional release candidates
    were needed after the second)
-   If the final X.Y.0 release is delayed beyond the August target date,
    the subsequent release series is not affected, and will still be
    scheduled for August (now slightly less than two years later).

In addition to allowing more time for end user feedback on the release
candidate, this adjusted policy also provides additional time for
maintainers of Python projects to build and publish pre-built wheel
archives for the new stable release series, significantly improving the
initial user experience of the X.Y.0 release.

Changes to management of the CPython stable C ABI

The CPython stable ABI[1] makes the commitment that binary extension
modules built against any particular CPython release will continue to
work on future CPython releases that support the same stable ABI version
(this version is currently abi3).

Under the proposed rolling pre-freeze release model, this commitment
would be extended to also apply to the beta releases: once an
intentional addition to the abi3 stable ABI for the upcoming Python
version has been shipped in a beta release, then it will not be removed
from future releases for as long as the abi3 stable ABI remains
supported.

Two main mechanisms will be available for obtaining community feedback
on additions to the stable ABI:

-   the preferred mechanism will be to add new APIs to the full CPython
    API first, and only promote them to the stable ABI after they have
    been included in at least one published beta release and received
    relevant user feedback
-   for APIs where that approach is unavailable for some reason (e.g.
    some API additions may serve no useful purpose when the full CPython
    API is available), then developers may request that the release
    manager mark the next release as an alpha release (even in the
    absence of an ABI break in the full CPython API), and attempt to
    obtain further feedback that way

As a slight readability and usability improvement, this PEP also
proposes the introduction of aliases for each major stable ABI version:

    #define Py_LIMITED_API_3_3 0x03030000
    #define Py_LIMITED_API_3_4 0x03040000
    #define Py_LIMITED_API_3_5 0x03050000
    #define Py_LIMITED_API_3_6 0x03060000
    #define Py_LIMITED_API_3_7 0x03070000
    #define Py_LIMITED_API_3_8 0x03080000
    #define Py_LIMITED_API_3_9 0x03090000
    // etc...

These would be used both in extension module code to set the target ABI
version:

    #define Py_LIMITED_API Py_LIMITED_API_3_8

And also in the CPython interpreter implementation to check which
symbols should be made available:

    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= Py_LIMITED_API_3_9
    // A Python 3.9+ addition to the stable ABI would appear here
    #endif

The documentation for the rolling pre-freeze releases and the stable C
ABI would make it clear that extension modules built against the stable
ABI in a later pre-freeze release may not load correctly on an earlier
pre-freeze release.

The documentation for alpha releases and the stable C ABI would make it
clear that even extension modules built against the stable ABI in an
alpha release may not load correctly on the next release if two alpha
releases are published in a row (this situation would ideally be rare).

Changes to management of the full CPython ABI

This PEP proposes two changes to the management of the full CPython ABI.

An explicit commit and NEWS file convention to mark ABI breaking changes

The proposal in this PEP requires that release managers be able to
appropriately mark a pre-freeze release as either an alpha or a beta
release based on whether or not it includes an ABI breaking change.

To assist in that process, core developers would be requested to include
a "(CPython ABI break)" marker at the beginning of all NEWS file
snippets for changes that introduce a breaking change in the full
CPython C ABI.

The "CPython" marker is included to make it clear that these annotations
relate to the full CPython ABI, not the stable ABI.

For commit messages, the shorter marker "(ABI break)" would be placed at
the start of the summary line for the commit.

The pre-merge bots will be updated to ensure that if the ABI break
marker appears in one of the two locations, it appears in both of them.

If the marker is inadvertently omitted from the initial commit message
and NEWS entry, then the commit message marker should be included in the
subsequent commit that adds the marker to the NEWS entry.

In addition to being useful for release managers, these markers should
also be useful for developers investigating unexpected segfaults when
testing against the affected release.

Explicitly marking builds against the pre-freeze ABI

The full CPython ABI has long operated under a policy where binary
compatibility only applies within a release series after the ABI has
been declared frozen, and only source compatibility applies between
different release series.

This policy means that extension modules built against CPython
pre-releases prior to the ABI freeze for that release series may not
actually load correctly on the final release.

This is due to the fact that the extension module may be relying on
provisional or previously deprecated interfaces that were changed or
removed in a later alpha or beta release, or it may be due to public
structures used by the extension module changing size due to the
addition of new fields.

Historically, adoption of alpha and beta releases has been low enough
that this hasn't really been a problem in practice. However, this PEP
proposes to actively encourage widespread operational use of beta
releases, which makes it desirable to ensure that users of those
releases won't inadvertently publish binary extension modules that cause
segfaults for users running the release candidates and final releases.

To that end, this PEP proposes amending the extension module SOABI
marker on non-Windows systems to include a new "p" flag for CPython
pre-releases, and only switch back to omitting that flag once the ABI
for that particular X.Y.0 version has been frozen on entry to the
release candidate stage.

With this change, alpha and beta releases of 3.9.0 would get an SOABI
tag of cpython-39p, while all release candidates and final builds (for
both 3.9.0 and later 3.9.x releases) would get an unqualified SOABI tag
of cpython-39

Debug builds would still add the "d" to the end of the tag, giving
cpython-39pd for debug builds of pre-releases.

On Windows systems, the suffix for tagged pyd files in pre-release
builds would include "p" as a pre-release marker immediately after the
version number, giving markers like "cp39p-win_amd64".

A proposed reference implementation for this change is available at[2]
(Note: at time of writing, that implementation had not yet been tested
on Windows).

Updating Python-Requires for projects affected by full C ABI changes

When a project first opts in to providing pre-built binary wheels for
the rolling pre-freeze release series, they don't need to do anything
special: they would add the rolling release series to their build and
test matrices and publish binary archives that are flagged as being
compatible with that release series, just as they would if providing
pre-built binary wheels after the full CPython ABI freeze for that
release series.

However, if the project is affected by a CPython ABI compatibility break
in the rolling release stream, then they will need to issue a version
update that includes both the new binary build, and a new environment
constrained Python-Requires marker.

For example, if a project supporting the rolling release stream was
affected by a CPython ABI compatibility break in the 3.9.0a5 release,
then they would add the following metadata entry on the version that
published the updated binary build:

    Python-Requires: >= "3.9.0b6"; python_version == "3.9" and full_python_version != "3.9.0a5"

What this does is add an additional compatibility constraint as part of
the published packages, so Python 3.9.0 beta versions prior to 3.9.0b6
won't consider the updated package as a candidate for installation, and
the only alpha release that will consider the package is 3.9.0a5 itself.

Caveats and Limitations

Actual release dates may be scheduled up to a month earlier or later at
the discretion of the release manager, based on release team
availability, and the timing of other events (e.g. PyCon US, or the
annual core developer sprints). However, as one goal of the proposal is
to provide a consistent release cadence, adjustments should ideally be
rare.

Within a release series, the exact frequency of maintenance releases
would still be up to the release manager and the binary release team;
this PEP only proposes an expected cadence for pre-releases and X.Y.0
releases.

However, for the sake of the example timelines, the PEP assumes
maintenance releases every other month, allowing them to alternate
months with the rolling pre-freeze releases.

The release manager and Steering Council would also retain the power to
amend various details of the proposal in this PEP. Possible amendments
include (but are not limited to):

-   changing the timing for creation of the maintenance branch. If a
    major change that would require a new alpha release is landed
    relatively late in the pre-release process, the release manager
    could potentially choose to branch off from a point prior to that
    major change. For example, it may make sense to do this if the next
    scheduled release was intended to be the final beta release or the
    first release candidate.
-   the criteria for declaring an alpha release could potentially be
    expanded to include all changes that require a "Porting" entry in
    the What's New document
-   rather than declaring alpha releases on an as-needed basis, the
    release manager could declare some dates as alpha releases in
    advance, and ask core developers to time their higher risk changes
    accordingly.

The intent of the concrete proposal in the PEP is to provide a clear
illustrative example for reviewers to consider, not to constrain our
ability to adjust specific details based on practical experience with
the process.

Design Discussion

Why rolling pre-freeze releases over simply doing more frequent X.Y.0 releases?

For large parts of Python's user base, availability of new CPython
feature releases isn't the limiting factor on their adoption of those
new releases (this effect is visible in such metrics as PyPI download
metadata).

As such, any proposal based on speeding up full feature releases needs
to strike a balance between meeting the needs of users who would be
adopting each release as it became available, and those that would now
be in a position of adopting every 2nd, 3rd, or 4th release, rather than
being able to migrate to almost every release at some point within its
lifecycle.

This proposal aims to approach the problem from a different angle by
defining a new production-ready release stream that is more specifically
tailored to the interests of operating environments that are able to
consume new releases as fast as the CPython core team is prepared to
produce them.

Is it necessary to keep the "alpha" and "beta" naming scheme?

Using the "a" and "b" initials for the proposed rolling releases is a
design constraint imposed by some of the pragmatic aspects of the way
CPython version numbers are published.

Specifically, alpha releases, beta releases, and release candidates are
reported in some places using the strings "a", "b", and "c"
respectively, while in others they're reported using the hex digits 0xA,
0xB, and 0xC. We want to preserve that, while also ensuring that any
Python-Requires constraints are expressed against the beta releases
rather than the alpha releases (since the latter may not enforce the
abi3 stability requirements if two alpha releases occur in succession).

However, there isn't anything forcing us to say that the "a" stands for
"alpha" or the "b" stands for "beta".

That means that if we wanted to increase adoption amongst folks that
were only being put off by the "beta" label, then it may make sense to
emphasise the "*A*BI breaking" and "*B*inary compatible" names over the
"alpha" and "beta" names, giving:

-   3.9.0a1: ABI breaking pre-freeze release
-   3.9.0b2: binary compatible pre-freeze release
-   3.9.0rc1: release candidate
-   3.9.0: final release

This iteration of the PEP doesn't go that far, as limiting initial
adoption of the rolling pre-freeze releases to folks that are
comfortable with the "beta" label is likely to be a good thing, as it is
the early adopters of these releases that are going to encounter any
unexpected consequences that occur at the level of the wider Python
ecosystem, and we're going to need them to be willing to take an active
part in getting those issues resolved.

Moving away from the "beta" naming would then become an option to keep
in mind for the future, assuming the resulting experience is
sufficiently positive that we decide the approach is worth continuing.

Why rolling pre-freeze releases rather than alternating between stable and unstable release series?

Rather than using the beta period for rolling releases, another option
would be to alternate between traditional stable releases (for 3.8.x,
3.10.x, etc), and release series that used the new rolling release
cadence (for 3.9.x, 3.11.x, etc).

This idea suffers from the same core problem as PEP 598 and PEP 602: it
imposes changes on end users that are happy with the status quo without
offering them any clear compensating benefit.

It's also affected by one of the main concerns raised against PEP 598:
at least some core developers and end users strongly prefer that no
particular semantics be assigned to the value of any of the numbers in a
release version. These community members instead prefer that all the
semantic significance be associated with the position within the release
number that is changing.

By contrast, the rolling pre-freeze release proposal aims to address
that concern by ensuring that the proposed changes in policy all revolve
around whether a particular release is an alpha release, beta release,
release candidate, or final release.

Why not use Calendar Versioning for the rolling release stream?

Steve Dower's initial write-up of this proposal[3] suggested the use of
calendar versioning for the rolling release stream (so the first rolling
pre-release after Python 3.8.0 would have been Python 2019.12 rather
than 3.9.0b1).

Paul Moore pointed out[4] two major practical problems with that
proposal:

-   it isn't going to be clear to users of the calendar-based versions
    where they stand in relation to the traditionally numbered versions
-   it breaks Python-Requires metadata processing in packaging tools
    with no clear way of fixing it reliably (since all calendar versions
    would appear as newer than any standard version)

This PEP aims to address both of those problems by using the established
beta version numbers for the rolling releases.

As an example, consider the following question: "Does Python 2021.12
include all the new features released in Python 3.9.0?". With calendar
versioning on the rolling releases, that's impossible to answer without
consulting a release calendar to see when 3.9.0rc1 was branched off from
the rolling release series.

By contrast, the equivalent question for rolling pre-freeze releases is
straightforward to answer: "Does Python 3.10.0b2 include all the new
features released in Python 3.9.0?". Just from formulating the question,
the answer is clearly "Yes, unless they were provisional features that
got removed".

The beta numbering approach also avoids other questions raised by the
calendar versioning concept, such as how sys.version_info,
PY_VERSION_HEX, site-packages directory naming, and installed Python
binary and extension module naming would work.

How would users of the rolling pre-freeze releases detect API changes?

When adding new features, core developers would be strongly encouraged
to support feature detection and graceful fallback to alternative
approaches via mechanisms that don't rely on either sys.version_info or
runtime code object introspection.

In most cases, a simple hasattr check on the affected module will serve
this purpose, but when it doesn't, alternative approaches would be
considered as part of the feature addition. Prior art in this area
includes the pickle.HIGHEST_PROTOCOL attribute, the
hashlib.algorithms_available set, and the various os.supports_* sets
that the os module already offers for platform dependent capability
detection.

It would also be possible to add features that need to be explicitly
enabled via a __future__ import when first included in the rolling
pre-freeze releases, even if that feature flag was subsequently enabled
by default before its first appearance in an X.Y.0 release candidate.

The rationale behind these approaches is that explicit
detection/enabling like this would make it straightforward for users of
the rolling pre-freeze release stream to notice when we remove or change
provisional features (e.g. from __future__ imports break on compile if
the feature flag no longer exists), or to safely fall back on previous
functionality.

The interpreter's rich attribute lookup machinery means we can also
choose to add warnings for provisional or deprecated imports and
attributes that we don't have any practical way to add for checks
against the value of sys.version_info.

Why add a new pre-freeze ABI flag to force rebuilds after X.Y.0rc1?

The core development team currently actively discourage the creation of
public pre-built binaries for an X.Y series prior to the ABI freeze
date.

The reason we do that is to avoid the risk of painful debugging sessions
on the stable X.Y.0 release that get traced back to "Oh, our dependency
'superfast-binary-operation' was affected by a CPython ABI break in
X.Y.0a3, but the project hasn't published a new build since then".

With the proposed pre-freeze ABI flag in place, this aspect of the
release adoption process continues on essentially unchanged from the
status quo: a new CPython X.Y release series hits ABI freeze -> package
maintainers publish new binary extension modules for that release series
-> end users only get segfaults due to actual bugs, not just builds
against an incompatible ABI.

The primary goal of the new pre-freeze ABI flag is then to improve the
user experience of the rolling pre-freeze releases themselves, by
allowing pre-built binary archives to be published for those releases
without risking the problems that currently cause us to actively
discourage the publication of binary artifacts prior to ABI freeze.

In the ideal case, package maintainers will only need to publish one
pre-freeze binary build at X.Y.0a1, and then a post-freeze build after
X.Y.0rc1. The only situations that should require a rebuild in the
meantime are those where the project was actually affected by a CPython
ABI break in an intervening alpha release.

As a concrete example, consider the scenario where we end up having
three releases that include ABI breaks: X.Y.0a1, X.Y.0a5, X.Y.0a7. The
X.Y.0a7 ABI is then the ABI that carries through all the subsequent beta
releases and into X.Y.0rc1. (This is the scenario illustrated in figure
1)

Forcing everyone to rebuild the world every time there’s an alpha
release in the rolling release stream would almost certainly lead to
publishers deciding supporting the rolling releases was more trouble
than it was worth, so we want to allow modules built against X.Y.0a1 to
be loaded against X.Y.0a7, as they’re probably going to be compatible
(there are very few projects that use every C API that CPython
publishes, and most ABI breaks affect a single specific API).

Once we publish X.Y.0rc1 though, we want to ensure that any binaries
that were built against X.Y.0a1 and X.Y.0a4 are completely removed from
the end user experience. It would be nice to be able to keep the builds
against X.Y.0a7 and any subsequent beta releases (since it turned out
those actually were built against the post-freeze ABI, even if we didn’t
know that at the time), but losing them isn’t any worse than the status
quo.

This means that the pre-freeze flag is “the simplest thing that could
possibly work” to solve this problem - it’s just a new ABI flag, and we
already have the tools available to deal with ABI flags (both in the
interpreter and in package publication and installation tools).

Since the ABI flags have changed relative to the pre-releases, projects
don't even need to publish a new release: they can upload new wheel
archives to their existing releases, just as they can today.

A cleverer scheme that was able to retroactively accept everything built
against the last alpha or subsequent beta releases would likely be
possible, but it isn't considered necessary for adoption of this PEP, as
even if we initially start out with a simple pre-release ABI flag, it
would still be possible to devise a more sophisticated approach in the
future.

Why allow additional alpha releases after X.Y.0a1?

In an ideal world, all breaking changes to the full CPython ABI would
land in X.Y.0a1 alongside the filesystem layout changes, and the ABI for
the release series would remain stable after that.

However, recent history doesn't suggest that we'd be able to actually
make that commitment and stick to it, so the PEP assumes that ABI
changes will be made progressively throughout the pre-freeze period, and
the full lockdown will occur only with the creation of the X.Y.z
maintenance branch when preparing X.Y.0rc1.

Implications for CPython core development

The major change for CPython core development is the need to keep the
master branch more consistently release ready.

While the main requirement for that would be to keep the stable BuildBot
fleet green, there would also be encouragement to keep the development
version of the documentation up to date for the benefit of users of the
rolling pre-freeze releases. This will include providing draft What's
New entries for changes as they are implemented, although the initial
versions may be relatively sparse, and then expanded based on feedback
from beta release users.

For core developers working on the CPython C API, there would also be a
new requirement to consistently mark ABI breaking changes in their NEWS
file snippets.

On the specific topic of the stable ABI, most API designs will be able
to go through a process where they're first introduced as a provisional
part of the full CPython API (allowing changes between pre-freeze
releases), and only promoted to the stable ABI once developers are
confident that the interface is genuinely stable.

It's only in rare cases where an API serves no useful purpose outside
the stable ABI that it may make sense to publish an alpha release
containing a provisional stable ABI addition rather than iterating on
the design in the provisional CPython API instead.

Implications for Python library development

If this PEP is successful in its aims, then supporting the rolling
pre-freeze release stream shouldn't be subtantially more painful for
library authors than supporting the stable releases.

For publishers of pure Python packages, this would be a matter of
publishing "py3" tagged wheel archives, and potentially adding the
rolling pre-freeze release stream to their test matrix if that option is
available to them.

For publishers of binary extension modules, the preferred option would
be to target the stable C ABI (if feasible), and thus enjoy an
experience similar to that of pure Python packages, where a single
pre-built wheel archive is able to cover multiple versions of Python,
including the rolling pre-freeze release stream.

This option isn't going to be viable for all libraries, and the desired
outcome for those authors is that they be able to support the rolling
releases by building and publishing one additional wheel archive, built
against the initial X.Y.0a1 release. The subsequent build against
X.Y.0rc1 or later is then the same build that would have been needed if
only supporting the final stable release.

Additional wheel builds beyond those two should then only be needed if
that particular library is directly affected by an ABI break in any
other alpha release that occurs between those two points.

Having a rolling pre-freeze release stream available may also make it
more feasible for more CI providers to offer a "CPython beta release"
testing option. At the moment, this feature is only available from CI
providers that are willing and able to put the necessary time and effort
into creating, testing, and publishing their own builds from the CPython
master branch (e.g.[5]).

Implications for the proposed Scientific Python ecosystem support period

Based on discussions at SciPy 2019, NEP (NumPy Enhancement Proposal) 29
has been drafted[6] to propose a common convention across the Scientific
Python ecosystem for dropping support for older Python versions.

While the exact formulation of that policy is still being discussed, the
draft proposal (as of October 20, 2019) recommends that projects support
any Python feature release published within the last 42 months, with a
minimum of supporting the latest 2 Python feature releases.

For an 18-month feature release cadence, that works out to always
supporting at least the two most recent feature releases, and then
dropping support for all X.Y.Z releases around 6 months after X.(Y+2).0
is released. This means there is a 6-month period roughly every other
year where the three most recent feature releases are supported.

For a 12-month release cadence, it would work out to always supporting
at least the three most recent feature releases, and then dropping
support for all X.Y.Z releases around 6 months after X.(Y+3).0 is
released. This means that for half of each year, the four most recent
feature releases would be supported, with the other half of each year
hopefully being used to get ready for that year's feature release.

For a 24-month release cadence, the second clause takes priority over
the first, and the recommended Python version support period increases
to 48 months from the initial X.Y.0 release in order to consistently
support the two most recent CPython feature releases. For projects that
also support the rolling release stream, the number of supported feature
releases would increase to three.

Release cycle alignment for core development sprints

With the proposal in this PEP, it is expected that the focus of core
development sprints would shift slightly based on the current location
in the two-year cycle.

In release years, the timing of PyCon US is suitable for new
contributors to work on bug fixes and smaller features before the first
release candidate goes out, while the Language Summit and core developer
discussions can focus on plans for the next release series.

The pre-alpha core development sprint in release years will provide an
opportunity to incorporate feedback received on the previous release,
either as part of the next maintenance release (for bug fixes and
feedback on provisional APIs), or as part of the first alpha release of
the next release series (for feedback received on stable APIs).

Those initial alpha releases would also be the preferred target for ABI
breaking changes to the full CPython ABI (while changes later in the
release cycle would still be permitted as described in this PEP, landing
them in the X.Y.0a1 release means that they won't trigger any additional
work for publishers of pre-built binary packages).

The Steering Council elections for the next release cycle are also
likely to occur around the same time as the pre-alpha development
sprints.

In non-release years, the focus for both events would just be on the
upcoming maintenance and pre-freeze releases. These less intense years
would hopefully provide an opportunity to tackle various process changes
and infrastructure upgrades without impacting the release candidate
preparation process.

Release cycle alignment for prominent Linux distributions

Some rolling release Linux distributions (e.g. Arch, Gentoo) may be in a
position to consume the new rolling pre-freeze releases proposed in this
PEP, but it is expected that most distributions would continue to use
the established releases.

The specific dates for final releases proposed in this PEP are chosen to
align with the feature freeze schedules for the annual October releases
of the Ubuntu and Fedora Linux distributions.

For both Fedora and Ubuntu, it means that the release candidate phase
aligns with the development period for a distro release, which is the
ideal time for them to test a new version and provide feedback on
potential regressions and compatibility concerns.

For Ubuntu, this also means that their April LTS releases will have
benefited from a full short-term release cycle using the new system
Python version, while still having that CPython release be open to
upstream bug fixes for most of the time until the next Ubuntu LTS
release.

The one Linux release cycle alignment that is likely to be consistently
poor with the specific proposal in this PEP is with Debian, as that has
been released in the first half of odd-numbered years since 2005
(roughly 12 months offset from Ubuntu LTS releases).

With the annual release proposal in PEP 602, both Debian and Ubuntu LTS
would consistently get a system Python version that is around 6 months
old, but would also consistently select different Python versions from
each other.

With a two-year cadence, and CPython releases in the latter half of the
year, they're likely to select the same version as each other, but one
of them will be choosing a CPython release that is more than 18 months
behind the latest beta releases by the time the Linux distribution
ships.

If that situation does occur, and is deemed undesirable (but not
sufficiently undesirable for Debian to choose to adjust their release
timing), then that's where the additional complexity of the "incremental
feature release" proposal in PEP 598 may prove worthwhile.

(Moving CPython releases to the same half of the year as the Debian and
Ubuntu LTS releases would potentially help mitigate the problem, but
also creates new problems where a slip in the CPython release schedule
could directly affect the release schedule for a Linux distribution, or
else result in a distribution shipping a Python version that is more
than 18 months old)

Implications for simple deployment environments

For the purposes of this PEP, a "simple" deployment environment is any
use case where it is straightforward to ensure that all target
environments are updated to a new Python release at the same time (or at
least in advance of the rollout of new higher level application
versions), and any pre-release testing that occurs need only target a
single Python micro version.

The simplest such case would be scripting for personal use, where the
testing and target environments are the exact same environment.

Similarly simple environments would be containerised web services, where
the same Python container is used in the CI pipeline as is used on
deployment, and any application that bundles its own Python runtime,
rather than relying on a pre-existing Python deployment on the target
system.

For these use cases, there is a straightforward mechanism to minimise
the impact of this PEP: continue using the stable releases, and ignore
the rolling pre-freeze releases.

To actually adopt the rolling pre-freeze releases in these environments,
the main challenge will be handling the potential for extension module
segfaults when the next pre-freeze release is an alpha release rather
than a beta release, indicating that the CPython ABI may have changed in
an incompatible way.

If all extension modules in use target the stable ABI, then there's no
problem, and everything will work just as smoothly as it does on the
stable releases.

Alternatively, "rebuild and recache all extension modules" could become
a standard activity undertaken as part of updating to an alpha release.

Finally, it would also be reasonable to just not worry about it until
something actually breaks, and then handle it like any other library
compatibility issue found in a new alpha or beta release.

Aside from extension module ABI compatibility, the other main point of
additional complexity when using the rolling pre-freeze releases would
be "roll-back" compatibility for independently versioned features, such
as pickle and SQLite, where use of new or provisional features in the
beta stream may create files that are not readable by the stable
release. Applications that use these kinds of features and also require
the ability to reliably roll-back to a previous stable CPython release
would, as today, be advised to avoid adopting pre-release versions.

Implications for complex deployment environments

For the purposes of this PEP, "complex" deployment environments are use
cases which don't meet the "simple deployment" criteria above. They may
involve multiple distinct versions of Python, use of a personalised
build of Python, or "gatekeepers" who are required to approve use of a
new version prior to deployment.

For example, organisations that install Python on their users' machines
as part of a standard operating environment fall into this category, as
do those that provide a standard build environment. Distributions such
as conda-forge or WinPython that provide collections of consistently
built and verified packages are impacted in similar ways.

These organisations tend to either prefer high stability (for example,
all of those who are happily using the system Python in a stable Linux
distribution like Debian, RHEL/CentOS, or Ubuntu LTS as their preferred
Python environment) or fast turnaround (for example, those who regularly
contribute toward the latest CPython pre-releases).

In some cases, both usage models may exist within the same organisation
for different purposes, such as:

-   using a stable Python environment for mission critical systems, but
    allowing data scientists to use the latest available version for ad
    hoc data analysis
-   a hardware manufacturer deploying a stable Python version as part of
    their production firmware, but using the latest available version in
    the development and execution of their automated integration tests

Under any release model, each new release of Python generates work for
these organisations. This work may involve legal, security or technical
reviews of Python itself, assessment and verification of impactful
changes, reapplication of patches, recompilation and testing of
third-party dependencies, and only then deployment.

Organisations that can take updates quickly should be able to make use
of the more frequent beta releases. While each update will still require
similar investigative work to what they require today, the volume of
work required per release should be reduced as each release will be more
similar to the previous than it is under the present model. One
advantage of the proposed release-every-2-months model is that
organisations can choose their own adoption cadence from adopting every
beta release, to adopting one per quarter, or one every 6 months, or one
every year. Beyond that, it would likely make more sense to continue
using the stable releases instead.

For organisations with stricter evaluations or a preference for
stability, the longer release cycle for stable releases will reduce the
annual effort required to update, the longer release candidate period
will allow more time to do internal testing before the X.Y.0 release,
and the greater use by others during the beta period will provide more
confidence in the initial releases. Meanwhile, the organisation can
confidently upgrade through maintenance releases for a longer time
without fear of breaking changes.

Acknowledgements

Thanks to Łukasz Langa for creating PEP 602 and prompting this
discussion of possible improvements to the CPython release cadence, and
to Kyle Stanley and h-vetinari for constructive feedback on the initial
draft of this PEP.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] CPython stable ABI documentation
(https://docs.python.org/3/c-api/stable.html)

[2] Example implementation for a pre-release SOABI flag
(https://github.com/ncoghlan/cpython/pull/3)

[3] Steve Dower's initial "Fast and Stable releases" proposal
(https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/20)

[4] Paul Moore's initial comments on Steve's proposal
(https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/37)

[5] Travis CI nightly CPython builds
(https://docs.travis-ci.com/user/languages/python/#nightly-build-support)

[6] NEP 29 proposes a common policy for dropping support of old Python
versions (https://numpy.org/neps/nep-0029-deprecation_policy.html)

authors:
Alyssa Coghlan
Steve Dower


text:
PEP: 463 Title: Exception-catching expressions Version: $Revision$
Last-Modified: $Date$ Author: Chris Angelico<EMAIL_ADDRESS>Status:
Rejected Type: Standards Track Content-Type: text/x-rst Created:
15-Feb-2014 Python-Version: 3.5 Post-History: 20-Feb-2014, 16-Feb-2014
Resolution:
https://mail.python.org/pipermail/python-dev/2014-March/133118.html

Rejection Notice

From
https://mail.python.org/pipermail/python-dev/2014-March/133118.html:

""" I want to reject this PEP. I think the proposed syntax is acceptable
given the desired semantics, although it's still a bit jarring. It's
probably no worse than the colon used with lambda (which echoes the
colon used in a def just like the colon here echoes the one in a
try/except) and definitely better than the alternatives listed.

But the thing I can't get behind are the motivation and rationale. I
don't think that e.g. dict.get() would be unnecessary once we have
except expressions, and I disagree with the position that EAFP is better
than LBYL, or "generally recommended" by Python. (Where do you get that?
From the same sources that are so obsessed with DRY they'd rather
introduce a higher-order-function than repeat one line of code? :-)

This is probably the most you can get out of me as far as a
pronouncement. Given that the language summit is coming up I'd be happy
to dive deeper in my reasons for rejecting it there (if there's demand).

I do think that (apart from never explaining those dreadful acronyms :-)
this was a well-written and well-researched PEP, and I think you've done
a great job moderating the discussion, collecting objections, reviewing
alternatives, and everything else that is required to turn a heated
debate into a PEP. Well done Chris (and everyone who helped), and good
luck with your next PEP! """

Abstract

Just as PEP 308 introduced a means of value-based conditions in an
expression, this system allows exception-based conditions to be used as
part of an expression.

Motivation

A number of functions and methods have parameters which will cause them
to return a specified value instead of raising an exception. The current
system is ad-hoc and inconsistent, and requires that each function be
individually written to have this functionality; not all support this.

-   dict.get(key, default) - second positional argument in place of
    KeyError
-   next(iter, default) - second positional argument in place of
    StopIteration
-   list.pop() - no way to return a default
-   seq[index] - no way to handle a bounds error
-   min(sequence, default=default) - keyword argument in place of
    ValueError
-   statistics.mean(data) - no way to handle an empty iterator

Had this facility existed early in Python's history, there would have
been no need to create dict.get() and related methods; the one obvious
way to handle an absent key would be to respond to the exception. One
method is written which signals the absence in one way, and one
consistent technique is used to respond to the absence. Instead, we have
dict.get(), and as of Python 3.4, we also have min(... default=default),
and myriad others. We have a LBYL syntax for testing inside an
expression, but there is currently no EAFP notation; compare the
following:

    # LBYL:
    if key in dic:
        process(dic[key])
    else:
        process(None)
    # As an expression:
    process(dic[key] if key in dic else None)

    # EAFP:
    try:
        process(dic[key])
    except KeyError:
        process(None)
    # As an expression:
    process(dic[key] except KeyError: None)

Python generally recommends the EAFP policy, but must then proliferate
utility functions like dic.get(key,None) to enable this.

Rationale

The current system requires that a function author predict the need for
a default, and implement support for it. If this is not done, a full
try/except block is needed.

Since try/except is a statement, it is impossible to catch exceptions in
the middle of an expression. Just as if/else does for conditionals and
lambda does for function definitions, so does this allow exception
catching in an expression context.

This provides a clean and consistent way for a function to provide a
default: it simply raises an appropriate exception, and the caller
catches it.

With some situations, an LBYL technique can be used (checking if some
sequence has enough length before indexing into it, for instance). This
is not safe in all cases, but as it is often convenient, programmers
will be tempted to sacrifice the safety of EAFP in favour of the
notational brevity of LBYL. Additionally, some LBYL techniques (eg
involving getattr with three arguments) warp the code into looking like
literal strings rather than attribute lookup, which can impact
readability. A convenient EAFP notation solves all of this.

There's no convenient way to write a helper function to do this; the
nearest is something ugly using either lambda:

    def except_(expression, exception_list, default):
        try:
            return expression()
        except exception_list:
            return default()
    value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan"))

which is clunky, and unable to handle multiple exception clauses; or
eval:

    def except_(expression, exception_list, default):
        try:
            return eval(expression, globals_of_caller(), locals_of_caller())
        except exception_list as exc:
            l = locals_of_caller().copy()
            l['exc'] = exc
            return eval(default, globals_of_caller(), l)

    def globals_of_caller():
        return sys._getframe(2).f_globals

    def locals_of_caller():
        return sys._getframe(2).f_locals

    value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """)

which is even clunkier, and relies on implementation-dependent hacks.
(Writing globals_of_caller() and locals_of_caller() for interpreters
other than CPython is left as an exercise for the reader.)

Raymond Hettinger expresses a desire for such a consistent API.
Something similar has been requested multiple times in the past.

Proposal

Just as the 'or' operator and the three part 'if-else' expression give
short circuiting methods of catching a falsy value and replacing it,
this syntax gives a short-circuiting method of catching an exception and
replacing it.

This currently works:

    lst = [1, 2, None, 3]
    value = lst[2] or "No value"

The proposal adds this:

    lst = [1, 2]
    value = (lst[2] except IndexError: "No value")

Specifically, the syntax proposed is:

    (expr except exception_list: default)

where expr, exception_list, and default are all expressions. First, expr
is evaluated. If no exception is raised, its value is the value of the
overall expression. If any exception is raised, exception_list is
evaluated, and should result in either a type or a tuple, just as with
the statement form of try/except. Any matching exception will result in
the corresponding default expression being evaluated and becoming the
value of the expression. As with the statement form of try/except,
non-matching exceptions will propagate upward.

Parentheses are required around the entire expression, unless they would
be completely redundant, according to the same rules as generator
expressions follow. This guarantees correct interpretation of nested
except-expressions, and allows for future expansion of the syntax -see
below on multiple except clauses.

Note that the current proposal does not allow the exception object to be
captured. Where this is needed, the statement form must be used. (See
below for discussion and elaboration on this.)

This ternary operator would be between lambda and if/else in precedence.

Consider this example of a two-level cache:

    for key in sequence:
        x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key)))
        # do something with x

This cannot be rewritten as:

    x = lvl1.get(key, lvl2.get(key, f(key)))

which, despite being shorter, defeats the purpose of the cache, as it
must calculate a default value to pass to get(). The .get() version
calculates backwards; the exception-testing version calculates forwards,
as would be expected. The nearest useful equivalent would be:

    x = lvl1.get(key) or lvl2.get(key) or f(key)

which depends on the values being nonzero, as well as depending on the
cache object supporting this functionality.

Alternative Proposals

Discussion on python-ideas brought up the following syntax suggestions:

    value = expr except default if Exception [as e]
    value = expr except default for Exception [as e]
    value = expr except default from Exception [as e]
    value = expr except Exception [as e] return default
    value = expr except (Exception [as e]: default)
    value = expr except Exception [as e] try default
    value = expr except Exception [as e] continue with default
    value = default except Exception [as e] else expr
    value = try expr except Exception [as e]: default
    value = expr except default # Catches anything
    value = expr except(Exception) default # Catches only the named type(s)
    value = default if expr raise Exception
    value = expr or else default if Exception
    value = expr except Exception [as e] -> default
    value = expr except Exception [as e] pass default

It has also been suggested that a new keyword be created, rather than
reusing an existing one. Such proposals fall into the same structure as
the last form, but with a different keyword in place of 'pass'.
Suggestions include 'then', 'when', and 'use'. Also, in the context of
the "default if expr raise Exception" proposal, it was suggested that a
new keyword "raises" be used.

All forms involving the 'as' capturing clause have been deferred from
this proposal in the interests of simplicity, but are preserved in the
table above as an accurate record of suggestions.

The four forms most supported by this proposal are, in order:

    value = (expr except Exception: default)
    value = (expr except Exception -> default)
    value = (expr except Exception pass default)
    value = (expr except Exception then default)

All four maintain left-to-right evaluation order: first the base
expression, then the exception list, and lastly the default. This is
important, as the expressions are evaluated lazily. By comparison,
several of the ad-hoc alternatives listed above must (by the nature of
functions) evaluate their default values eagerly. The preferred form,
using the colon, parallels try/except by using "except exception_list:",
and parallels lambda by having "keyword name_list: subexpression"; it
also can be read as mapping Exception to the default value, dict-style.
Using the arrow introduces a token many programmers will not be familiar
with, and which currently has no similar meaning, but is otherwise quite
readable. The English word "pass" has a vaguely similar meaning
(consider the common usage "pass by value/reference" for function
arguments), and "pass" is already a keyword, but as its meaning is
distinctly unrelated, this may cause confusion. Using "then" makes sense
in English, but this introduces a new keyword to the language - albeit
one not in common use, but a new keyword all the same.

Left to right evaluation order is extremely important to readability, as
it parallels the order most expressions are evaluated. Alternatives such
as:

    value = (expr except default if Exception)

break this, by first evaluating the two ends, and then coming to the
middle; while this may not seem terrible (as the exception list will
usually be a constant), it does add to the confusion when multiple
clauses meet, either with multiple except/if or with the existing
if/else, or a combination. Using the preferred order, subexpressions
will always be evaluated from left to right, no matter how the syntax is
nested.

Keeping the existing notation, but shifting the mandatory parentheses,
we have the following suggestion:

    value = expr except (Exception: default)
    value = expr except(Exception: default)

This is reminiscent of a function call, or a dict initializer. The colon
cannot be confused with introducing a suite, but on the other hand, the
new syntax guarantees lazy evaluation, which a dict does not. The
potential to reduce confusion is considered unjustified by the
corresponding potential to increase it.

Example usage

For each example, an approximately-equivalent statement form is given,
to show how the expression will be parsed. These are not always strictly
equivalent, but will accomplish the same purpose. It is NOT safe for the
interpreter to translate one into the other.

A number of these examples are taken directly from the Python standard
library, with file names and line numbers correct as of early Feb 2014.
Many of these patterns are extremely common.

Retrieve an argument, defaulting to None:

    cond = (args[1] except IndexError: None)

    # Lib/pdb.py:803:
    try:
        cond = args[1]
    except IndexError:
        cond = None

Fetch information from the system if available:

    pwd = (os.getcwd() except OSError: None)

    # Lib/tkinter/filedialog.py:210:
    try:
        pwd = os.getcwd()
    except OSError:
        pwd = None

Attempt a translation, falling back on the original:

    e.widget = (self._nametowidget(W) except KeyError: W)

    # Lib/tkinter/__init__.py:1222:
    try:
        e.widget = self._nametowidget(W)
    except KeyError:
        e.widget = W

Read from an iterator, continuing with blank lines once it's exhausted:

    line = (readline() except StopIteration: '')

    # Lib/lib2to3/pgen2/tokenize.py:370:
    try:
        line = readline()
    except StopIteration:
        line = ''

Retrieve platform-specific information (note the DRY improvement); this
particular example could be taken further, turning a series of separate
assignments into a single large dict initialization:

    # sys.abiflags may not be defined on all platforms.
    _CONFIG_VARS['abiflags'] = (sys.abiflags except AttributeError: '')

    # Lib/sysconfig.py:529:
    try:
        _CONFIG_VARS['abiflags'] = sys.abiflags
    except AttributeError:
        # sys.abiflags may not be defined on all platforms.
        _CONFIG_VARS['abiflags'] = ''

Retrieve an indexed item, defaulting to None (similar to dict.get):

    def getNamedItem(self, name):
        return (self._attrs[name] except KeyError: None)

    # Lib/xml/dom/minidom.py:573:
    def getNamedItem(self, name):
        try:
            return self._attrs[name]
        except KeyError:
            return None

Translate numbers to names, falling back on the numbers:

    g = (grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid)
    u = (pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid)

    # Lib/tarfile.py:2198:
    try:
        g = grp.getgrnam(tarinfo.gname)[2]
    except KeyError:
        g = tarinfo.gid
    try:
        u = pwd.getpwnam(tarinfo.uname)[2]
    except KeyError:
        u = tarinfo.uid

Look up an attribute, falling back on a default:

    mode = (f.mode except AttributeError: 'rb')

    # Lib/aifc.py:882:
    if hasattr(f, 'mode'):
        mode = f.mode
    else:
        mode = 'rb'

    return (sys._getframe(1) except AttributeError: None)
    # Lib/inspect.py:1350:
    return sys._getframe(1) if hasattr(sys, "_getframe") else None

Perform some lengthy calculations in EAFP mode, handling division by
zero as a sort of sticky NaN:

    value = (calculate(x) except ZeroDivisionError: float("nan"))

    try:
        value = calculate(x)
    except ZeroDivisionError:
        value = float("nan")

Calculate the mean of a series of numbers, falling back on zero:

    value = (statistics.mean(lst) except statistics.StatisticsError: 0)

    try:
        value = statistics.mean(lst)
    except statistics.StatisticsError:
        value = 0

Looking up objects in a sparse list of overrides:

    (overrides[x] or default except IndexError: default).ping()

    try:
        (overrides[x] or default).ping()
    except IndexError:
        default.ping()

Narrowing of exception-catching scope

The following examples, taken directly from Python's standard library,
demonstrate how the scope of the try/except can be conveniently
narrowed. To do this with the statement form of try/except would require
a temporary variable, but it's far cleaner as an expression.

Lib/ipaddress.py:343:

    try:
        ips.append(ip.ip)
    except AttributeError:
        ips.append(ip.network_address)

Becomes:

    ips.append(ip.ip except AttributeError: ip.network_address)

The expression form is nearly equivalent to this:

    try:
        _ = ip.ip
    except AttributeError:
        _ = ip.network_address
    ips.append(_)

Lib/tempfile.py:130:

    try:
        dirlist.append(_os.getcwd())
    except (AttributeError, OSError):
        dirlist.append(_os.curdir)

Becomes:

    dirlist.append(_os.getcwd() except (AttributeError, OSError): _os.curdir)

Lib/asyncore.py:264:

    try:
        status.append('%s:%d' % self.addr)
    except TypeError:
        status.append(repr(self.addr))

Becomes:

    status.append('%s:%d' % self.addr except TypeError: repr(self.addr))

In each case, the narrowed scope of the try/except ensures that an
unexpected exception (for instance, AttributeError if "append" were
misspelled) does not get caught by the same handler. This is
sufficiently unlikely to be reason to break the call out into a separate
line (as per the five line example above), but it is a small benefit
gained as a side-effect of the conversion.

Comparisons with other languages

(With thanks to Andrew Barnert for compiling this section. Note that the
examples given here do not reflect the current version of the proposal,
and need to be edited.)

Ruby's "begin…rescue…rescue…else…ensure…end" is an expression
(potentially with statements inside it). It has the equivalent of an
"as" clause, and the equivalent of bare except. And it uses no
punctuation or keyword between the bare except/exception class/exception
class with as clause and the value. (And yes, it's ambiguous unless you
understand Ruby's statement/expression rules.)

    x = begin computation() rescue MyException => e default(e) end;
    x = begin computation() rescue MyException default() end;
    x = begin computation() rescue default() end;
    x = begin computation() rescue MyException default() rescue OtherException other() end;

In terms of this PEP:

    x = computation() except MyException as e default(e)
    x = computation() except MyException default(e)
    x = computation() except default(e)
    x = computation() except MyException default() except OtherException other()

Erlang has a try expression that looks like this

    x = try computation() catch MyException:e -> default(e) end;
    x = try computation() catch MyException:e -> default(e); OtherException:e -> other(e) end;

The class and "as" name are mandatory, but you can use "_" for either.
There's also an optional "when" guard on each, and a "throw" clause that
you can catch, which I won't get into. To handle multiple exceptions,
you just separate the clauses with semicolons, which I guess would map
to commas in Python. So:

    x = try computation() except MyException as e -> default(e)
    x = try computation() except MyException as e -> default(e), OtherException as e->other_default(e)

Erlang also has a "catch" expression, which, despite using the same
keyword, is completely different, and you don't want to know about it.

The ML family has two different ways of dealing with this, "handle" and
"try"; the difference between the two is that "try" pattern-matches the
exception, which gives you the effect of multiple except clauses and as
clauses. In either form, the handler clause is punctuated by "=>" in
some dialects, "->" in others.

To avoid confusion, I'll write the function calls in Python style.

Here's SML's "handle"

    let x = computation() handle MyException => default();;

Here's OCaml's "try"

    let x = try computation() with MyException explanation -> default(explanation);;

    let x = try computation() with

        MyException(e) -> default(e)
      | MyOtherException() -> other_default()
      | (e) -> fallback(e);;

In terms of this PEP, these would be something like:

    x = computation() except MyException => default()
    x = try computation() except MyException e -> default()
    x = (try computation()
         except MyException as e -> default(e)
         except MyOtherException -> other_default()
         except BaseException as e -> fallback(e))

Many ML-inspired but not-directly-related languages from academia mix
things up, usually using more keywords and fewer symbols. So, the Oz
would map to Python as

    x = try computation() catch MyException as e then default(e)

Many Lisp-derived languages, like Clojure, implement try/catch as
special forms (if you don't know what that means, think function-like
macros), so you write, effectively

    try(computation(), catch(MyException, explanation, default(explanation)))

    try(computation(),
        catch(MyException, explanation, default(explanation)),
        catch(MyOtherException, explanation, other_default(explanation)))

In Common Lisp, this is done with a slightly clunkier "handler-case"
macro, but the basic idea is the same.

The Lisp style is, surprisingly, used by some languages that don't have
macros, like Lua, where xpcall takes functions. Writing lambdas
Python-style instead of Lua-style

    x = xpcall(lambda: expression(), lambda e: default(e))

This actually returns (true, expression()) or (false, default(e)), but I
think we can ignore that part.

Haskell is actually similar to Lua here (except that it's all done with
monads, of course):

    x = do catch(lambda: expression(), lambda e: default(e))

You can write a pattern matching expression within the function to
decide what to do with it; catching and re-raising exceptions you don't
want is cheap enough to be idiomatic.

But Haskell infixing makes this nicer:

    x = do expression() `catch` lambda: default()
    x = do expression() `catch` lambda e: default(e)

And that makes the parallel between the lambda colon and the except
colon in the proposal much more obvious:

    x = expression() except Exception: default()
    x = expression() except Exception as e: default(e)

Tcl has the other half of Lua's xpcall; catch is a function which
returns true if an exception was caught, false otherwise, and you get
the value out in other ways. And it's all built around the implicit
quote-and-exec that everything in Tcl is based on, making it even harder
to describe in Python terms than Lisp macros, but something like

    if {[ catch("computation()") "explanation"]} { default(explanation) }

Smalltalk is also somewhat hard to map to Python. The basic version
would be

    x := computation() on:MyException do:default()

... but that's basically Smalltalk's passing-arguments-with-colons
syntax, not its exception-handling syntax.

Deferred sub-proposals

Multiple except clauses

An examination of use-cases shows that this is not needed as often as it
would be with the statement form, and as its syntax is a point on which
consensus has not been reached, the entire feature is deferred.

Multiple 'except' keywords could be used, and they will all catch
exceptions raised in the original expression (only):

    # Will catch any of the listed exceptions thrown by expr;
    # any exception thrown by a default expression will propagate.
    value = (expr
        except Exception1: default1
        except Exception2: default2
        # ... except ExceptionN: defaultN
    )

Currently, one of the following forms must be used:

    # Will catch an Exception2 thrown by either expr or default1
    value = (
        (expr except Exception1: default1)
        except Exception2: default2
    )
    # Will catch an Exception2 thrown by default1 only
    value = (expr except Exception1:
        (default1 except Exception2: default2)
    )

Listing multiple exception clauses without parentheses is a syntax error
(see above), and so a future version of Python is free to add this
feature without breaking any existing code.

Capturing the exception object

In a try/except block, the use of 'as' to capture the exception object
creates a local name binding, and implicitly deletes that binding (to
avoid creating a reference loop) in a finally clause. In an expression
context, this makes little sense, and a proper sub-scope would be
required to safely capture the exception object - something akin to the
way a list comprehension is handled. However, CPython currently
implements a comprehension's subscope with a nested function call, which
has consequences in some contexts such as class definitions, and is
therefore unsuitable for this proposal. Should there be, in future, a
way to create a true subscope (which could simplify comprehensions,
except expressions, with blocks, and possibly more), then this proposal
could be revived; until then, its loss is not a great one, as the simple
exception handling that is well suited to the expression notation used
here is generally concerned only with the type of the exception, and not
its value - further analysis below.

This syntax would, admittedly, allow a convenient way to capture
exceptions in interactive Python; returned values are captured by "_",
but exceptions currently are not. This could be spelled:

    >>> (expr except Exception as e: e)

An examination of the Python standard library shows that, while the use
of 'as' is fairly common (occurring in roughly one except clause in
five), it is extremely uncommon in the cases which could logically be
converted into the expression form. Its few uses can simply be left
unchanged. Consequently, in the interests of simplicity, the 'as' clause
is not included in this proposal. A subsequent Python version can add
this without breaking any existing code, as 'as' is already a keyword.

One example where this could possibly be useful is Lib/imaplib.py:568:

    try: typ, dat = self._simple_command('LOGOUT')
    except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]]

This could become:

    typ, dat = (self._simple_command('LOGOUT')
        except BaseException as e: ('NO', '%s: %s' % (type(e), e)))

Or perhaps some other variation. This is hardly the most compelling
use-case, but an intelligent look at this code could tidy it up
significantly. In the absence of further examples showing any need of
the exception object, I have opted to defer indefinitely the
recommendation.

Rejected sub-proposals

finally clause

The statement form try... finally or try... except... finally has no
logical corresponding expression form. Therefore, the finally keyword is
not a part of this proposal, in any way.

Bare except having different meaning

With several of the proposed syntaxes, omitting the exception type name
would be easy and concise, and would be tempting. For convenience's
sake, it might be advantageous to have a bare 'except' clause mean
something more useful than "except BaseException". Proposals included
having it catch Exception, or some specific set of "common exceptions"
(subclasses of a new type called ExpressionError), or have it look for a
tuple named ExpressionError in the current scope, with a built-in
default such as (ValueError, UnicodeError, AttributeError, EOFError,
IOError, OSError, LookupError, NameError, ZeroDivisionError). All of
these were rejected, for several reasons.

-   First and foremost, consistency with the statement form of
    try/except would be broken. Just as a list comprehension or ternary
    if expression can be explained by "breaking it out" into its
    vertical statement form, an expression-except should be able to be
    explained by a relatively mechanical translation into a
    near-equivalent statement. Any form of syntax common to both should
    therefore have the same semantics in each, and above all should not
    have the subtle difference of catching more in one than the other,
    as it will tend to attract unnoticed bugs.
-   Secondly, the set of appropriate exceptions to catch would itself be
    a huge point of contention. It would be impossible to predict
    exactly which exceptions would "make sense" to be caught; why bless
    some of them with convenient syntax and not others?
-   And finally (this partly because the recommendation was that a bare
    except should be actively encouraged, once it was reduced to a
    "reasonable" set of exceptions), any situation where you catch an
    exception you don't expect to catch is an unnecessary bug magnet.

Consequently, the use of a bare 'except' is down to two possibilities:
either it is syntactically forbidden in the expression form, or it is
permitted with the exact same semantics as in the statement form
(namely, that it catch BaseException and be unable to capture it with
'as').

Bare except clauses

PEP 8 rightly advises against the use of a bare 'except'. While it is
syntactically legal in a statement, and for backward compatibility must
remain so, there is little value in encouraging its use. In an
expression except clause, "except:" is a SyntaxError; use the equivalent
long-hand form "except BaseException:" instead. A future version of
Python MAY choose to reinstate this, which can be done without breaking
compatibility.

Parentheses around the except clauses

Should it be legal to parenthesize the except clauses, separately from
the expression that could raise? Example:

    value = expr (
        except Exception1 [as e]: default1
        except Exception2 [as e]: default2
        # ... except ExceptionN [as e]: defaultN
    )

This is more compelling when one or both of the deferred sub-proposals
of multiple except clauses and/or exception capturing is included. In
their absence, the parentheses would be thus:

    value = expr except ExceptionType: default
    value = expr (except ExceptionType: default)

The advantage is minimal, and the potential to confuse a reader into
thinking the except clause is separate from the expression, or into
thinking this is a function call, makes this non-compelling. The
expression can, of course, be parenthesized if desired, as can the
default:

    value = (expr) except ExceptionType: (default)

As the entire expression is now required to be in parentheses (which had
not been decided at the time when this was debated), there is less need
to delineate this section, and in many cases it would be redundant.

Short-hand for "except: pass"

The following was been suggested as a similar short-hand, though not
technically an expression:

    statement except Exception: pass

    try:
        statement
    except Exception:
        pass

For instance, a common use-case is attempting the removal of a file:

    os.unlink(some_file) except OSError: pass

There is an equivalent already in Python 3.4, however, in contextlib:

    from contextlib import suppress
    with suppress(OSError): os.unlink(some_file)

As this is already a single line (or two with a break after the colon),
there is little need of new syntax and a confusion of statement vs
expression to achieve this.

Common objections

Colons always introduce suites

While it is true that many of Python's syntactic elements use the colon
to introduce a statement suite (if, while, with, for, etcetera), this is
not by any means the sole use of the colon. Currently, Python syntax
includes four cases where a colon introduces a subexpression:

-   dict display - { ... key:value ... }
-   slice notation - [start:stop:step]
-   function definition - parameter : annotation
-   lambda - arg list: return value

This proposal simply adds a fifth:

-   except-expression - exception list: result

Style guides and PEP 8 should recommend not having the colon at the end
of a wrapped line, which could potentially look like the introduction of
a suite, but instead advocate wrapping before the exception list,
keeping the colon clearly between two expressions.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

license: Public Domain