text:
PEP: 598 Title: Introducing incremental feature releases Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828
Status: Withdrawn Type: Informational Content-Type: text/x-rst Created:
15-Jun-2019 Python-Version: 3.9

Abstract

PEP 602 proposes reducing the feature delivery latency for the Python
standard library and CPython reference interpreter by increasing the
frequency of CPython feature releases from every 18-24 months to instead
occur every 9-12 months.

This PEP proposes to instead reduce the frequency of new baseline
feature releases (with the associated filesystem layout changes,
bytecode format changes, and C ABI compatibility breaks) to occur only
every other year (2020, 2022, 2024, etc), but to combine that change
with a new policy and approach that allows the introduction of backwards
compatible features in the initial set of point releases within a given
release series.

PEP Withdrawal

This PEP has been withdrawn in favour of the rolling beta release stream
proposal in PEP 605.

However, the concerns raised in this PEP are likely to apply to any
other "Long Term Support branch" proposals that allow feature backports
to improve the developer experience of supporting such releases (such as
the EL Python draft at[1]), so the ideas presented here may provide
useful design suggestions for such proposals.

Summary

The proposal to keep the current CPython release compatibility
management process, but go through it more often has significant
practical downsides, as a CPython feature release carries certain
expectations (most notably, a 5-year maintenance lifecycle, support for
parallel installation with the previous feature release, and the
possibility of breaking changes to the CPython-specific ABI, requiring
recompilation of all extension modules) that mean faster feature
releases in their current form have the potential to significantly
increase the burden of maintaining 3rd party Python libraries and
applications across all actively supported CPython releases.

It's also arguable whether such an approach would noticeably reduce the
typical feature delivery latency in practice for most end users, as the
adoption cycle for new feature releases is typically measured in months
or years, so more frequent releases may just lead to end users updating
to every 3rd or 4th feature release, rather than every 2nd or 3rd
feature release (as often happens today).

This PEP presents a competing proposal to instead slow down the
frequency of parallel installable feature releases that change the
filesystem layout and CPython ABI to a consistent 24-month cycle, but to
compensate for this by introducing the notion of build compatible
incremental feature releases, and then deferring the full feature freeze
of a given feature release series from the initial baseline X.Y.0
release to a subsequent X.Y.Z feature complete release that occurs ~12
months after the initial baseline feature release.

A new feature_complete attribute on the sys.version_info structure will
provide a programmatic indicator as to whether or not a release series
remains open to further incremental feature releases. Alternate
implementations of Python would also be free to clear this flag to
indicate that their support for their nominal Python version may still
be a work in progress.

For compatibility testing purposes, and to maintain pickle compatibility
in mixed version environments, a new sys.feature_limit attribute (and
corresponding CPython CLI parameter, --feature-limit X.Y.Z, and
environment variable, PYTHONFEATURELIMIT) will provide a way to limit
the runtime availability of features added in incremental feature
releases.

The existing cycle and the new cycle would be synchronised on their
feature freeze release dates, so the full Python 3.9.x feature freeze
would occur in October 2021, 24 months after the Python 3.8.0 feature
release, with the initial Python 3.9.0 release taking place in October
2020.

Example Future Release Schedules

Under this proposal, Python 3.9.0a1 would be released in November 2019,
shortly after the Python 3.8.0 feature complete release in October 2019.

The 3.9.0b1 release would then follow 6 months later in May 2020, with
3.9.0 itself being released in October 2020.

Assuming micro releases of 3.9.x were to occur quarterly, then the
overall release timeline would look like:

-   2019-11: 3.9.0a1
-   ... additional alpha releases as determined by the release manager
-   2020-05: 3.9.0b1
-   ... additional beta releases as determined by the release manager
-   2020-08: 3.9.0bX (final beta release that locks ABI compatibility)
-   2020-09: 3.9.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2020-10: 3.9.0 (BFR - baseline feature release)
-   2021-01: 3.9.1 (IFR - incremental feature release)
-   2021-04: 3.9.2 (IFR)
-   2021-07: 3.9.3 (IFR)
-   2021-10: 3.9.4 (feature complete release)
-   2022-01: 3.9.5
-   2022-04: 3.9.6
-   2022-07: 3.9.7
-   2022-10: 3.9.8 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2025-10: 3.9.x branch closed

Feature complete release numbers would typically be written without any
qualifier (as they are today), while the baseline and incremental
feature releases would be expected to have a qualifier appended
indicating that they aren't a traditional CPython release (3.9.0 (BFR),
3.9.1 (IFR), etc).

The Python 3.10 release series would start being published the month
after the first Python 3.9 feature complete release, in parallel with
the final 12 months of routine Python 3.9 maintenance releases:

-   2021-11: 3.10.0a1
-   ... additional alpha releases as determined by the release manager
-   2022-05: 3.10.0b1
-   ... additional beta releases as determined by the release manager
-   2022-08: 3.10.0bX (final beta release that locks ABI compatibility)
-   2022-09: 3.10.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2022-10: 3.10.0 (BFR)
-   2023-01: 3.10.1 (IFR)
-   2023-04: 3.10.2 (IFR)
-   2023-07: 3.10.3 (IFR)
-   2023-10: 3.10.4
-   2024-01: 3.10.5
-   2024-04: 3.10.6
-   2024-07: 3.10.7
-   2024-10: 3.10.8 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2027-10: 3.10.x branch closed

In this model, there are always two or three active branches:

-   2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x
    maintenance
-   2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
-   2020-05 -> 2020-10: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x
    maintenance
-   2020-10 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x
    maintenance
-   2021-10 -> 2022-05: 3.10.0 pre-beta, 3.9.x maintenance
-   2022-05 -> 2022-10: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x
    maintenance
-   2022-10 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x
    maintenance
-   2023-10 -> 2024-05: 3.11.0 pre-beta, 3.10.x maintenance
-   2024-05 -> 2024-10: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x
    maintenance
-   ... etc

(Pre-alpha and pre-beta development occurs on the main git branch, all
other development occurs on a release specific branch with changes
typically backported from the main git branch)

TODO: this really needs a diagram to help explain it, so I'll add a
picture once I have one to add.

This is quite similar to the status quo, but with a more consistent
cadence, alternating between baseline feature release years (2020, 2022,
etc) that focus on the alpha and beta cycle for a new baseline feature
release (while continuing to publish maintenance releases for the
previous feature release series), and feature complete release years
(2021, 2023, etc), that focus on making smaller improvements to the
current feature release series (while making plans for the next feature
release series the following year).

Proposal

Excluding alpha and beta releases, CPython currently has 3 different
kinds of release increment:

-   Feature release (i.e. X.Y.0 releases)
-   Maintenance release (X.Y.Z releases within ~2 years of X.Y.0)
-   Source-only security release (subsequent X.Y.Z releases)

Feature freeze takes place at the time of the X.Y.0b1 release. Build
compatibility freeze now takes place at the time of the last beta
release (providing time for projects to upload wheel archives to PyPI
prior to the first release candidate).

This then creates the following periods in the lifecycle of a release
series:

-   Pre-beta (release series is the CPython development branch)
-   Beta (release enters maintenance mode, ABI compatibility mostly
    locked)
-   Maintenance (ABI locked, only bug fixes & docs enhancements
    accepted)
-   Security fix only (no further binary releases, only security fixes
    accepted)
-   End of life (no further releases of any kind)

The proposal in this PEP is that the "Feature release" category be split
up into three different kinds of feature release:

-   Baseline feature release (X.Y.0 releases)
-   Incremental feature release (any X.Y.Z releases published between a
    baseline feature release and the corresponding feature complete
    release)
-   Feature complete release (a specific X.Y.Z release ~1 year after
    X.Y.0)
-   Maintenance release (X.Y.Z releases within ~1 years of the feature
    complete release)
-   Source-only security release (subsequent X.Y.Z releases)

This would then introduce a new "Feature releases" phase in the release
series lifecycle:

-   Pre-beta (release series is the CPython development branch)
-   Beta (release enters feature additions mode, ABI compatibility not
    yet locked)
-   Feature releases (ABI locked, backwards compatible API additions
    accepted)
-   Maintenance (ABI locked, only bug fixes & docs enhancements
    accepted)
-   Security fix only (no further binary releases, only security fixes
    accepted)
-   End of life (no further releases of any kind)

The pre-release beta period would be relaxed to use the incremental
feature release policy for changes, rather than the stricter maintenance
release policy.

For governance purposes, baseline feature releases are the only releases
that would qualify as a "feature release" in the PEP 13 sense
(incremental feature releases wouldn't count).

Baseline feature releases and feature release series

Baseline feature releases are essentially just the existing feature
releases, given a new name to help distinguish them from the new
incremental feature releases, and also to help indicate that unlike
their predecessors, they are no longer considered feature complete at
release.

Baseline feature releases would continue to define a new feature release
series, locking in the following language, build, and installation
compatibility constraints for the remainder of that series:

-   Python language grammar
-   ast module AST format
-   CPython interpreter opcode format
-   pyc file magic number and filename compatibility tags
-   extension module filename compatibility tags
-   wheel archive compatibility tags
-   default package and module import directories
-   default installation filename and directories

Baseline feature releases would also continue to be the only releases
where:

-   new deprecations, pending deprecations, and other warnings can be
    introduced
-   existing pending deprecations can be converted to full deprecations
-   existing warnings can be converted to errors
-   other changes requiring "Porting to Python X.Y" entries in the
    What's New document can be introduced

Key characteristics of a feature release series:

-   an installation within one feature release series does not conflict
    with installations of other feature release series (i.e. they can be
    installed in parallel)
-   an installation within a feature release series can be updated to a
    later micro release within the same series without requiring
    reinstallation or any other changes to previously installed
    components

Key characteristics of a baseline feature release:

-   in a baseline feature release,
    sys.version_info.feature_complete == False
-   in a baseline feature release, sys.version_info.micro == 0
-   baseline feature releases may contain higher risk changes to the
    language and interpreter, such as grammar modifications, major
    refactoring of interpreter and standard library internals, or
    potentially invasive feature additions that carry a risk of
    unintended side effects on other existing functionality
-   features introduced in a baseline feature release are the only
    features permitted to rely on sys.version_info as their sole runtime
    indicator of the feature's availability

Key expectations around feature release series and baseline feature
releases:

-   most public projects will only actively test against the most recent
    micro release within a release series
-   many (most?) public projects will only add a new release series to
    their test matrix after the initial baseline feature release has
    already been published, which can make it difficult to resolve
    issues that require providing new flags or APIs to explicitly opt-in
    to old behaviour after a default behaviour changed
-   private projects with known target environments will test against
    whichever micro release version they're actually using
-   most private projects will also only consider migrating to a new
    release series after the initial baseline feature release has
    already been published, again posing a problem if the resolution of
    their problems requires an API addition

The key motivation of the proposal in this PEP is that the public and
private project behaviours described above aren't new expectations:
they're descriptions of the way CPython release series are already
handled by the wider community today. As such, the PEP represents an
attempt to adjust our release policies and processes to better match the
way the wider community already handles them, rather than changing our
processes in a way that then means the wider community needs to adjust
to us rather than the other way around.

Incremental feature releases

Incremental feature releases are the key new process addition being
proposed by this PEP. They are subject to the same strict runtime
compatibility requirements as the existing maintenance releases, but
would have the following more relaxed policies around API additions and
enhancements:

-   new public APIs can be added to any standard library module
    (including builtins)
-   subject to the feature detection requirement below, new optional
    arguments can be added to existing APIs (including builtins)
-   new public APIs can be added to the stable C ABI (with appropriate
    version guards)
-   new public APIs can be added to the CPython C API
-   with the approval of the release manager, backwards compatible
    reliability improvements can be made to existing APIs and syntactic
    constructs
-   with the approval of the release manager, performance improvements
    can be incorporated for existing APIs and syntactic constructs

The intent of this change in policy is to allow usability improvements
for new (and existing!) language features to be delivered in a more
timely fashion, rather than requiring users to incur the inherent delay
and costs of waiting for and then upgrading to the next feature release
series.

It is also designed such that the approval to add a feature to the next
baseline feature release can be considered separately from the question
of whether or not to make it available in the next incremental feature
release for the current release series, potentially allowing the first
task to be completed by volunteer contributors, while the latter
activity could be handled by paid contributors (e.g. customers of
commercial Python redistributors could potentially request that their
vendor backport a feature, or core developers could offer to undertake
specific backports on a contract basis). (There would be potential
ethical concerns with gating bug fixes this way, but those concerns
don't apply for backports of new features)

Key characteristics of an incremental feature release:

-   in an incremental feature release,
    sys.version_info.feature_complete == False
-   in an incremental feature release, sys.version_info.micro != 0
-   all API additions made in an incremental feature release must
    support efficient runtime feature detection that doesn't rely on
    either sys.version_info or runtime code object introspection. In
    most cases, a simple hasattr check on the affected module will serve
    this purpose, but when it doesn't, an alternative approach will need
    to be implemented as part of the feature addition. Prior art in this
    area includes the pickle.HIGHEST_PROTOCOL attribute, the
    hashlib.algorithms_available set, and the various os.supports_* sets
    that the os module already offers for platform dependent capability
    detection
-   to maintain pickle compatibility in mixed version environments, and
    to enable easier compatibility testing across multiple API versions
    within the same release series, all API additions made in an
    incremental feature release must support the new sys.feature_limit
    setting as described in the next section

Key expectations around incremental feature releases:

-   "don't break existing installations on upgrade" remains a key
    requirement for all micro releases, even with the more permissive
    change inclusion policy
-   more intrusive changes should still be deferred to the next baseline
    feature release
-   public Python projects that start relying on features added in an
    incremental feature release should set their Python-Requires
    metadata appropriately (projects already do this when necessary -
    e.g. aiohttp specifically requires 3.5.3 or later due to an issue
    with asyncio.get_event_loop() in earlier versions)

Some standard library modules may also impose their own restrictions on
acceptable changes in incremental feature releases (for example, only a
baseline feature release should ever add new hash algorithms to
hashlib.algorithms_guaranteed - incremental feature releases would only
be permitted to add algorithms to hashlib.algorithms_available)

Maintaining interoperability across incremental feature releases

It is a common practice to use Python's pickle module to exchange
information between Python processes running on different versions of
Python. Between release series, this compatibility is expected to only
run one way (i.e. excluding deprecated APIs, Python "X.Y+1" processes
should be able to read pickle archives produced by Python "X.Y"
processes, but the reverse does not hold, as the newer archives may
reference attributes and parameters that don't exist in the older
version).

Within a release series, however, it is expected to hold in both
directions, as the "No new features" policy means that almost all pickle
archives created on Python "X.Y.Z+1" will be readable by Python "X.Y.Z"
processes.

Similarly, Python libraries and applications are often only tested
against the latest version in a release series, and this is usually
sufficient to keep code working on earlier releases in that same series.

Allowing feature additions in later "X.Y.Z" releases with no way to turn
them off would pose a problem for these common practices, as a library
or application that works fine when tested on CPython version "X.Y.Z"
would fail on earlier versions if it used a feature newly introduced in
"X.Y.Z", and any pickle archives it creates that rely on those new
interfaces may also not be readable on the older versions.

To help address these problems, a new sys.feature_limit attribute would
be added, as a structured sequence corresponding to the first 3 fields
in sys.version_info (major, minor, micro).

A new CLI option (--feature-limit X.Y.Z) and environment variable
(PYTHONFEATURELIMIT=X.Y.Z) would be used to set this attribute. The
PyCoreConfig struct would also gain a new field:

    wchar_t *feature_limit;

If the limit is not set explicitly, it would default to the first 3
fields in sys.version_info. If the limit is set to a value outside the
lower bound of sys.version_info[:2] and the upper bound of
sys.version_info[:3], it will be clamped to those bounds, padding with
zeroes if necessary.

For example, given a current version of "3.9.3", nominal limits would be
converted to runtime sys.feature_limit values as follows:

    3 => (3, 9, 0)
    3.8.1 => (3, 9, 0)
    3.9 => (3, 9, 0)
    3.9.2 => (3, 9, 2)
    <unset> => (3, 9, 3)
    3.9.3 => (3, 9, 3)
    3.9.4 => (3, 9, 3)
    4 => (3, 9, 3)

New APIs backported to an incremental feature release would be expected
to include a guard that deletes the API from the module if the feature
limit is too low:

    def feature_api():
        ...

    _version_feature_api_added = (3, 9, 1)
    if _version_feature_api_added > sys.feature_limit:
        del feature_api

Similarly, new parameters would be expected to include a guard that
adjusts the function signature to match the old one:

    def feature_api(old_param1, old_param2, new_param=default):
        """Updated API docstring"""
        ...

    _version_feature_api_changed = (3, 9, 1)
    if _version_feature_api_changed > sys.feature_limit:
        _new_feature_api = feature_api
        def feature_api(old_param1, old_param2):
            """Legacy API docstring"""
            return _new_feature_api(old_param1, old_param2)

Structuring the guards this way would keep the code structure as similar
as possible between the main development branch and the backport
branches, so future bug fixes can still be backported automatically.

It is expected that convenience functions and/or additional automated
tests would eventually be added to help ensure these backported APIs are
guarded appropriately, but it seems reasonable to wait until specific
concrete examples are available to drive the design of those APIs and
automated tests, rather than designing them solely on the basis of
hypothetical examples.

Feature complete release and subsequent maintenance releases

The feature complete release for a given feature release series would be
developed under the normal policy for an incremental feature release,
but would have one distinguishing feature:

-   in a feature complete release,
    sys.version_info.feature_complete == True

Any subsequent maintenance and security fix only releases would also
have that flag set, and may informally be referred to as "feature
complete releases". For release series definition purposes though, the
feature complete release is the first one that sets that flag to "True".

Proposed policy adjustment for provisional APIs

To help improve consistency in management of provisional APIs, this PEP
proposes that provisional APIs be subject to regular backwards
compatibility requirements following the feature complete release for a
given release series.

Other aspects of managing provisional APIs would remain as they are
today, so as long as an API remains in the provisional state, regular
backwards compatibility requirements would not apply to that API in
baseline and incremental feature releases.

This policy is expected to provide increased clarity to end users (as
even provisional APIs will become stable for that release series in the
feature complete release), with minimal practical downsides for standard
library maintainers, based on the following analysis of documented API
additions and changes in micro releases of CPython since 3.0.0:

-   21 3.x.1 version added/changed notes
-   30 3.x.2 version added/changed notes
-   18 3.x.3 version added/changed notes
-   11 3.x.4 version added/changed notes
-   1 3.x.5 version added/changed notes
-   0 3.x.6+ version added/changed notes

When post-baseline-release changes need to be made, the majority of them
occur within the first two maintenance releases, which have always
occurred within 12 months of the baseline release.

(Note: these counts are not solely for provisional APIs - they cover all
APIs where semantic changes were made after the baseline release that
were considered necessary to cover in the documentation. To avoid double
counting changes, the numbers exclude any change markers from the What's
New section)

Motivation

The motivation for change in this PEP is essentially the same as the
motivation for change in PEP 596: the current 18-24 month gap between
feature releases has a lot of undesirable consequences, especially for
the standard library (see PEP 596 for further articulation of the
details).

This PEP's concern with the specific proposal in PEP 596 is that it
doubles the number of actively supported Python branches, increasing the
complexity of compatibility testing matrices for the entire Python
community, increasing the number of binary Python wheels to be uploaded
to PyPI when not using the stable ABI, and just generally having a high
chance of inflicting a relatively high level of additional cost across
the entire Python ecosystem.

The view taken in this PEP is that there's an alternative approach that
provides most of the benefits of a faster feature release without
actually incurring the associated costs: we can split the current X.Y.0
"feature freeze" into two parts, such that the baseline X.Y.0 release
only imposes a "runtime compatibility freeze", and the full standard
library feature freeze is deferred until later in the release series
lifecycle.

Caveats and Limitations

This proposal does NOT retroactively apply to Python 3.8 - it is being
proposed for Python 3.9 and later releases only.

Actual release dates may be adjusted up to a month earlier or later at
the discretion of the release manager, based on release team
availability, and the timing of other events (e.g. PyCon US, or the
annual core development sprints). However, part of the goal of this
proposal is to provide a consistent annual cadence for both contributors
and end users, so adjustments ideally would be rare.

This PEP does not dictate a specific cadence for micro releases within a
release series - it just specifies the rough timelines for transitions
between the release series lifecycle phases (pre-alpha, alpha, beta,
feature releases, bug fixes, security fixes). The number of micro
releases within each phase is determined by the release manager for that
series based on how frequently they and the rest of the release team for
that series are prepared to undertake the associated work.

However, for the sake of the example timelines, the PEP assumes
quarterly micro releases (the cadence used for Python 3.6 and 3.7,
splitting the difference between the twice yearly cadence used for some
historical release series, and the monthly cadence planned for Python
3.8 and 3.9).

Design Discussion

Why this proposal over simply doing more frequent baseline feature releases?

The filesystem layout changes and other inherently incompatible changes
involved in a baseline feature release create additional work for large
sections of the wider Python community.

Decoupling those layout changes from the Python version numbering scheme
is also something that would in and of itself involve making backwards
incompatible changes, as well as adjusting community expectations around
which versions will install over the top of each other, and which can be
installed in parallel on a single system.

We also don't have a straightforward means to communicate to the
community variations in support periods like "Only support Python
version X.Y until X.Y+1 is out, but support X.Z until X.Z+2 is out".

So this PEP takes as its starting assumption that the vast majority of
Python users simply shouldn't need to care that we're changing our
release policy, and the only folks that should be affected are those
that are eagerly waiting for standard library improvements (and other
backwards compatible interpreter enhancements), and those that need to
manage mission critical applications in complex deployment environments.

Implications for Python library development

Many Python libraries (both open source and proprietary) currently adopt
the practice of testing solely against the latest micro release within
each feature release series that the project still supports.

The design assumption in this PEP is that this practice will continue to
be followed during the feature release phase of a release series, with
the expectation being that anyone choosing to adopt a new release series
before it is feature complete will closely track the incremental feature
releases.

Libraries that support a previous feature release series are unlikely to
adopt features added in an incremental feature release, and if they do
adopt such a feature, then any associated fallback compatibility
strategies should be implemented in such a way that they're also
effective on the earlier releases in that release series.

Implications for the proposed Scientific Python ecosystem support period

Based on discussions at SciPy 2019, a NEP is currently being drafted[2]
to define a common convention across the Scientific Python ecosystem for
dropping support for older Python versions.

While the exact formulation of that policy is still being discussed, the
initial proposal was very simple: support any Python feature release
published within the last 42 months.

For an 18-month feature release cadence, that works out to always
supporting at least the two most recent feature releases, and then
dropping support for all X.Y.z releases around 6 months after X.(Y+2).0
is released. This means there is a 6-month period roughly every other
year where the three most recent feature releases are supported.

For a 12-month release cadence, it would work out to always supporting
at least the three most recent feature releases, and then dropping
support for all X.Y.z releases around 6 months after X.(Y+3).0 is
released. This means that for half of each year, the four most recent
feature releases would be supported.

For a 24-month release cadence, a 42-month support cycle works out to
always supporting at least the most recent feature release, and then
dropping support for all X.Y.z feature releases around 18 months after
X.(Y+1).0 is released. This means there is a 6-month period every other
year where only one feature release is supported (and that period
overlaps with the pre-release testing period for the X.(Y+2).0 baseline
feature release).

Importantly for the proposal in this PEP, that support period would
abide by the recommendation that library developers maintain support for
the previous release series until the latest release series has attained
feature complete status: dropping support 18 months after the baseline
feature release will be roughly equivalent to dropping support 6 months
after the feature complete release, without needing to track exactly
which release marked the series as feature complete.

Implications for simple deployment environments

For the purposes of this PEP, a "simple" deployment environment is any
use case where it is straightforward to ensure that all target
environments are updated to a new Python micro version at the same time
(or at least in advance of the rollout of new higher level application
versions), and there isn't any requirement for older Python versions to
be able to reliably read pickle streams generated with the newer Python
version, such that any pre-release testing that occurs need only target
a single Python micro version.

The simplest such case would be scripting for personal use, where the
testing and target environments are the exact same environment.

Similarly simple environments would be containerised web services, where
the same Python container is used in the CI pipeline as is used on
deployment, and any application that bundles its own Python runtime,
rather than relying on a pre-existing Python deployment on the target
system.

For these use cases, this PEP shouldn't have any significant
implications - only a single micro version needs to be tested,
independently of whether that version is feature complete or not.

Implications for complex deployment environments

For the purposes of this PEP, "complex" deployment environments are use
cases which don't meet the "simple deployment" criterion above: new
application versions are combined with two or more distinct micro
versions within the same release series as part of the deployment
process, rather than always targeting exactly one micro version at a
time.

If the proposal in this PEP has the desired effect of reducing feature
delivery latency, then it can be expected that developers using a
release series that is not yet feature complete will actually make use
of the new features as they're made available. This then means that
testing against a newer incremental feature release becomes an even less
valid test of compatibility with the baseline feature release and older
incremental feature releases than testing against a newer maintenance
release is for older maintenance releases.

One option for handling such cases is to simply prohibit the use of new
Python versions until the series has reached "feature complete" status.
Such a policy is effectively already adopted by many organisations when
it comes to new feature release series, with acceptance into operational
environments occurring months or years after the original release. If
this policy is adopted, then such organisations could potentially still
adopt a new Python version every other year - it would just be based on
the availability of the feature complete releases, rather than the
baseline feature releases.

A less strict alternative to outright prohibition would be to make use
of the proposed PYTHONFEATURELIMIT setting to enable phased migrations
to new incremental feature releases:

-   initially roll out Python X.Y.0 with PYTHONFEATURELIMIT=X.Y.0 set in
    CI and on deployment
-   roll out Python X.Y.1 to CI, keeping the PYTHONFEATURELIMIT=X.Y.0
    setting
-   deploy Python X.Y.1 to production based on successful CI results
-   update deployment environments to set PYTHONFEATURELIMIT=X.Y.1
-   set PYTHONFEATURELIMIT=X.Y.1 in CI only after all deployment
    environments have been updated
-   repeat this process for each new release up to and including the
    feature complete release for the release series
-   once the series is feature complete, either continue with this same
    process for consistency's sake, or else stop updating
    PYTHONFEATURELIMIT and leave it at the feature complete version
    number

Duration of the feature additions period

This PEP proposes that feature additions be limited to 12 months after
the initial baseline feature release.

The primary motivation for that is specifically to sync up with the
Ubuntu LTS timing, such that the feature complete release for the Python
3.9.x series gets published in October 2021, ready for inclusion in the
Ubuntu 22.04 release. (other LTS Linux distributions like RHEL, SLES,
and Debian don't have a fixed publishing cadence, so they can more
easily tweak their LTS timing a bit to align with stable versions of
their inputs. Canonical deliberately haven't given themselves that
flexibility with their own release cycle).

The 12 month feature addition period then arises from splitting the time
from the 2019-10 release of Python 3.8.0 and a final Python 3.9.x
incremental feature release in 2021-10 evenly between pre-release
development and subsequent incremental feature releases.

This is an area where this PEP could adopt part of the proposal in PEP
596, by instead making that split ~9 months of pre-release development,
and ~15 months of incremental feature releases:

-   2019-11: 3.9.0a1
-   ... additional alpha releases as determined by the release manager
-   2020-03: 3.9.0b1
-   2020-04: 3.9.0b2
-   2020-05: 3.9.0b3 (final beta release that locks ABI compatibility)
-   2020-06: 3.9.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2020-07: 3.9.0 (BFR)
-   2020-10: 3.9.1 (IFR)
-   2021-01: 3.9.2 (IFR)
-   2021-04: 3.9.3 (IFR)
-   2021-07: 3.9.4 (IFR)
-   2021-10: 3.9.5
-   2022-01: 3.9.6
-   2022-04: 3.9.7
-   2022-07: 3.9.8
-   2022-10: 3.9.9 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2025-10: 3.9.x branch closed

This approach would mean there were still always two or three active
branches, it's just that proportionally more time would be spent with a
branch in the "feature releases" phase, as compared to the "pre-alpha",
"pre-beta", and "pre-release" phases:

-   2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x
    maintenance
-   2019-10 -> 2020-03: 3.9.0 pre-beta, 3.8.x maintenance
-   2020-03 -> 2020-07: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x
    maintenance
-   2020-07 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x
    maintenance
-   2021-10 -> 2022-03: 3.10.0 pre-beta, 3.9.x maintenance
-   2022-03 -> 2022-07: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x
    maintenance
-   2022-07 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x
    maintenance
-   2023-10 -> 2024-03: 3.11.0 pre-beta, 3.10.x maintenance
-   2024-03 -> 2024-07: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x
    maintenance
-   ... etc

Duration of the unreleased pre-alpha period

In the baseline proposal in this PEP, the proposed timelines still
include periods where we go for 18 months without making a release from
the main git branch (e.g. 3.9.0b1 would branch off in 2020-05, and
3.10.0a1 wouldn't be published until 2021-11). They just allow for a
wider variety of changes to be backported to the most recent maintenance
branch for 12 of those months.

The variant of the proposal that moves the beta branch point earlier in
the release series lifecycle would increase that period of no direct
releases to 21 months - the only period where releases were made
directly from the main branch would be during the relatively short
window between the last incremental feature release of the previous
release series, and the beta branch point a few months later.

While alternating the annual cadence between "big foundational
enhancements" and "targeted low risk API usability improvements" is a
deliberate feature of this proposal, it still seems strange to wait that
long for feedback in the event that changes are made shortly after the
previous release series is branched.

An alternative way of handling this would be to start publishing alpha
releases for the next baseline feature release during the feature
addition period (similar to the way that PEP 596 proposes to starting
publishing Python 3.9.0 alpha releases during the Python 3.8.0 release
candidate period).

However, rather than setting specific timelines for that at a policy
level, it may make sense to leave that decision to individual release
managers, based on the specific changes that are being proposed for the
release they're managing.

Why not switch directly to full semantic versioning?

If this were a versioning design document for a new language, it would
use semantic versioning: the policies described above for baseline
feature releases would be applied to X.0.0 releases, the policies for
incremental feature releases would be applied to X.Y.0 releases, and the
policies for maintenance releases would be applied to X.Y.Z releases.

The problem for Python specifically is that all the policies and
properties for parallel installation support and ABI compatibility
definitions are currently associated with the first two fields of the
version number, and it has been that way for the better part of thirty
years.

As a result, it makes sense to split out the policy question of
introducing incremental feature releases in the first place from the
technical question of making the version numbering scheme better match
the semantics of the different release types.

If the proposal in this PEP were to be accepted by the Steering Council
for Python 3.9, then a better time to tackle that technical question
would be for the subsequent October 2022 baseline feature release, as
there are already inherent compatibility risks associated with the
choice of either "Python 4.0" (erroneous checks for the major version
being exactly 3 rather than 3 or greater), or "Python 3.10" (code
incorrectly assuming that the minor version will always contain exactly
one decimal digit)[3].

While the text of this PEP assumes that the release published in 2022
will be 3.10 (as the PEP author personally considers that the more
reasonable and most likely choice), there are complex pros and cons on
both sides of that decision, and this PEP does arguably add a potential
pro in favour of choosing the "Python 4.0" option (with the caveat that
we would also need to amend the affected installation layout and
compatibility markers to only consider the major version number, rather
than both the major and minor version).

If such a version numbering change were to be proposed and accepted,
then the example 3.10.x timeline given above would instead become the
following 4.x series timeline:

-   2021-11: 4.0.0a1
-   ... additional alpha releases as determined by the release manager
-   2022-05: 4.0.0b1
-   ... additional beta releases as determined by the release manager
-   2022-08: 4.0.0bX (final beta release that locks ABI compatibility)
-   2022-09: 4.0.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2022-10: 4.0.0 (BFR)
-   2023-01: 4.1.0 (IFR)
-   2023-04: 4.2.0 (IFR)
-   2023-07: 4.3.0 (IFR)
-   2023-10: 4.4.0 (IFR)
-   2024-01: 4.4.1
-   2024-04: 4.4.2
-   2024-07: 4.4.3
-   2024-10: 4.4.4 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2027-10: 4.x branch closed

And the 5 year schedule forecast would look like:

-   2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x
    maintenance
-   2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
-   2020-05 -> 2020-10: 4.0.0 pre-alpha, 3.9.0 pre-release, 3.8.x
    maintenance
-   2020-10 -> 2021-10: 4.0.0 pre-alpha, 3.9.x feature releases, 3.8.x
    maintenance
-   2021-10 -> 2022-05: 4.0.0 pre-beta, 3.9.x maintenance
-   2022-05 -> 2022-10: 5.0.0 pre-alpha, 4.0.0 pre-release, 3.9.x
    maintenance
-   2022-10 -> 2023-10: 5.0.0 pre-alpha, 4.x.0 feature releases, 3.9.x
    maintenance
-   2023-10 -> 2024-05: 5.0.0 pre-beta, 4.x.y maintenance
-   2024-05 -> 2024-10: 6.0.0 pre-alpha, 5.0.0 pre-release, 4.x.y
    maintenance
-   ... etc

References

Copyright

This document has been placed in the public domain.

[1] Draft Extended Lifecycle for Python (ELPython) design concept
(https://github.com/elpython/elpython-meta/blob/master/README.md)

[2] NEP proposing a standard policy for dropping support of old Python
versions (https://github.com/numpy/numpy/pull/14086)

[3] Anthony Sottile created a pseudo "Python 3.10" to find and fix such
issues (https://github.com/asottile-archive/python3.10)


text:
PEP: 3106 Title: Revamping dict.keys(), .values() and .items() Version:
$Revision$ Last-Modified: $Date$ Author: Guido van Rossum Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 19-Dec-2006
Python-Version: 3.0 Post-History:

Abstract

This PEP proposes to change the .keys(), .values() and .items() methods
of the built-in dict type to return a set-like or unordered container
object whose contents are derived from the underlying dictionary rather
than a list which is a copy of the keys, etc.; and to remove the
.iterkeys(), .itervalues() and .iteritems() methods.

The approach is inspired by that taken in the Java Collections
Framework[1].

Introduction

It has long been the plan to change the .keys(), .values() and .items()
methods of the built-in dict type to return a more lightweight object
than a list, and to get rid of .iterkeys(), .itervalues() and
.iteritems(). The idea is that code that currently (in 2.x) reads:

    for k, v in d.iteritems(): ...

should be rewritten as:

    for k, v in d.items(): ...

(and similar for .itervalues() and .iterkeys(), except the latter is
redundant since we can write that loop as for k in d.)

Code that currently reads:

    a = d.keys()    # assume we really want a list here

(etc.) should be rewritten as

  a = list(d.keys())

There are (at least) two ways to accomplish this. The original plan was
to simply let .keys(), .values() and .items() return an iterator, i.e.
exactly what iterkeys(), itervalues() and iteritems() return in Python
2.x. However, the Java Collections Framework[2] suggests that a better
solution is possible: the methods return objects with set behavior (for
.keys() and .items()) or multiset (== bag) behavior (for .values()) that
do not contain copies of the keys, values or items, but rather reference
the underlying dict and pull their values out of the dict as needed.

The advantage of this approach is that one can still write code like
this:

    a = d.items()
    for k, v in a: ...
    # And later, again:
    for k, v in a: ...

Effectively, iter(d.keys()) (etc.) in Python 3.0 will do what
d.iterkeys() (etc.) does in Python 2.x; but in most contexts we don't
have to write the iter() call because it is implied by a for-loop.

The objects returned by the .keys() and .items() methods behave like
sets. The object returned by the values() method behaves like a much
simpler unordered collection -- it cannot be a set because duplicate
values are possible.

Because of the set behavior, it will be possible to check whether two
dicts have the same keys by simply testing:

    if a.keys() == b.keys(): ...

and similarly for .items().

These operations are thread-safe only to the extent that using them in a
thread-unsafe way may cause an exception but will not cause corruption
of the internal representation.

As in Python 2.x, mutating a dict while iterating over it using an
iterator has an undefined effect and will in most cases raise a
RuntimeError exception. (This is similar to the guarantees made by the
Java Collections Framework.)

The objects returned by .keys() and .items() are fully interoperable
with instances of the built-in set and frozenset types; for example:

    set(d.keys()) == d.keys()

is guaranteed to be True (except when d is being modified simultaneously
by another thread).

Specification

I'm using pseudo-code to specify the semantics:

    class dict:

        # Omitting all other dict methods for brevity.
        # The .iterkeys(), .itervalues() and .iteritems() methods
        # will be removed.

        def keys(self):
            return d_keys(self)

        def items(self):
            return d_items(self)

        def values(self):
            return d_values(self)

    class d_keys:

        def __init__(self, d):
            self.__d = d

        def __len__(self):
            return len(self.__d)

        def __contains__(self, key):
            return key in self.__d

        def __iter__(self):
            for key in self.__d:
                yield key

        # The following operations should be implemented to be
        # compatible with sets; this can be done by exploiting
        # the above primitive operations:
        #
        #   <, <=, ==, !=, >=, > (returning a bool)
        #   &, |, ^, - (returning a new, real set object)
        #
        # as well as their method counterparts (.union(), etc.).
        #
        # To specify the semantics, we can specify x == y as:
        #
        #   set(x) == set(y)   if both x and y are d_keys instances
        #   set(x) == y        if x is a d_keys instance
        #   x == set(y)        if y is a d_keys instance
        #
        # and so on for all other operations.

    class d_items:

        def __init__(self, d):
            self.__d = d

        def __len__(self):
            return len(self.__d)

        def __contains__(self, (key, value)):
            return key in self.__d and self.__d[key] == value

        def __iter__(self):
            for key in self.__d:
                yield key, self.__d[key]

        # As well as the set operations mentioned for d_keys above.
        # However the specifications suggested there will not work if
        # the values aren't hashable.  Fortunately, the operations can
        # still be implemented efficiently.  For example, this is how
        # intersection can be specified:

        def __and__(self, other):
            if isinstance(other, (set, frozenset, d_keys)):
                result = set()
                for item in other:
                    if item in self:
                        result.add(item)
                return result
            if not isinstance(other, d_items):
                return NotImplemented
            d = {}
            if len(other) < len(self):
                self, other = other, self
            for item in self:
                if item in other:
                    key, value = item
                    d[key] = value
            return d.items()

        # And here is equality:

        def __eq__(self, other):
            if isinstance(other, (set, frozenset, d_keys)):
                if len(self) != len(other):
                    return False
                for item in other:
                    if item not in self:
                        return False
                return True
            if not isinstance(other, d_items):
                return NotImplemented
            # XXX We could also just compare the underlying dicts...
            if len(self) != len(other):
                return False
            for item in self:
                if item not in other:
                    return False
            return True

        def __ne__(self, other):
            # XXX Perhaps object.__ne__() should be defined this way.
            result = self.__eq__(other)
            if result is not NotImplemented:
                result = not result
            return result

    class d_values:

        def __init__(self, d):
            self.__d = d

        def __len__(self):
            return len(self.__d)

        def __contains__(self, value):
            # This is slow, and it's what "x in y" uses as a fallback
            # if __contains__ is not defined; but I'd rather make it
            # explicit that it is supported.
            for v in self:
                 if v == value:
                     return True
            return False

        def __iter__(self):
            for key in self.__d:
                yield self.__d[key]

        def __eq__(self, other):
            if not isinstance(other, d_values):
                return NotImplemented
            if len(self) != len(other):
                return False
            # XXX Sometimes this could be optimized, but these are the
            # semantics: we can't depend on the values to be hashable
            # or comparable.
            olist = list(other)
            for x in self:
                try:
                    olist.remove(x)
                except ValueError:
                    return False
            assert olist == []
            return True

        def __ne__(self, other):
            result = self.__eq__(other)
            if result is not NotImplemented:
                result = not result
            return result

Notes:

The view objects are not directly mutable, but don't implement
__hash__(); their value can change if the underlying dict is mutated.

The only requirements on the underlying dict are that it implements
__getitem__(), __contains__(), __iter__(), and __len__().

We don't implement .copy() -- the presence of a .copy() method suggests
that the copy has the same type as the original, but that's not feasible
without copying the underlying dict. If you want a copy of a specific
type, like list or set, you can just pass one of the above to the list()
or set() constructor.

The specification implies that the order in which items are returned by
.keys(), .values() and .items() is the same (just as it was in Python
2.x), because the order is all derived from the dict iterator (which is
presumably arbitrary but stable as long as a dict isn't modified). This
can be expressed by the following invariant:

    list(d.items()) == list(zip(d.keys(), d.values()))

Open Issues

Do we need more of a motivation? I would think that being able to do set
operations on keys and items without having to copy them should speak
for itself.

I've left out the implementation of various set operations. These could
still present small surprises.

It would be okay if multiple calls to d.keys() (etc.) returned the same
object, since the object's only state is the dict to which it refers. Is
this worth having extra slots in the dict object for? Should that be a
weak reference or should the d_keys (etc.) object live forever once
created? Strawman: probably not worth the extra slots in every dict.

Should d_keys, d_values and d_items have a public instance variable or
method through which one can retrieve the underlying dict? Strawman: yes
(but what should it be called?).

I'm soliciting better names than d_keys, d_values and d_items. These
classes could be public so that their implementations could be reused by
the .keys(), .values() and .items() methods of other mappings. Or should
they?

Should the d_keys, d_values and d_items classes be reusable? Strawman:
yes.

Should they be subclassable? Strawman: yes (but see below).

A particularly nasty issue is whether operations that are specified in
terms of other operations (e.g. .discard()) must really be implemented
in terms of those other operations; this may appear irrelevant but it
becomes relevant if these classes are ever subclassed. Historically,
Python has a really poor track record of specifying the semantics of
highly optimized built-in types clearly in such cases; my strawman is to
continue that trend. Subclassing may still be useful to add new methods,
for example.

I'll leave the decisions (especially about naming) up to whoever submits
a working implementation.

References

[1] Java Collections Framework
http://java.sun.com/docs/books/tutorial/collections/index.html

[2] Java Collections Framework
http://java.sun.com/docs/books/tutorial/collections/index.html

license: Public Domain


text:
PEP: 696 Title: Type Defaults for Type Parameters Author: James
Hilton-Balfe<EMAIL_ADDRESS>Sponsor: Jelle Zijlstra
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-696-type-defaults-for-typevarlikes/22569
Status: Final Type: Standards Track Topic: Typing Created: 14-Jul-2022
Python-Version: 3.13 Post-History: 22-Mar-2022, 08-Jan-2023, Resolution:
https://discuss.python.org/t/pep-696-type-defaults-for-typevarlikes/22569/34

typing:type_parameter_defaults and type-params

Abstract

This PEP introduces the concept of type defaults for type parameters,
including TypeVar, ParamSpec, and TypeVarTuple, which act as defaults
for type parameters for which no type is specified.

Default type argument support is available in some popular languages
such as C++, TypeScript, and Rust. A survey of type parameter syntax in
some common languages has been conducted by the author of PEP 695 and
can be found in its
Appendix A <695#appendix-a-survey-of-type-parameter-syntax>.

Motivation

    T = TypeVar("T", default=int)  # This means that if no type is specified T = int

    @dataclass
    class Box(Generic[T]):
        value: T | None = None

    reveal_type(Box())                      # type is Box[int]
    reveal_type(Box(value="Hello World!"))  # type is Box[str]

One place this regularly comes up is Generator. I propose changing the
stub definition to something like:

    YieldT = TypeVar("YieldT")
    SendT = TypeVar("SendT", default=None)
    ReturnT = TypeVar("ReturnT", default=None)

    class Generator(Generic[YieldT, SendT, ReturnT]): ...

    Generator[int] == Generator[int, None] == Generator[int, None, None]

This is also useful for a Generic that is commonly over one type.

    class Bot: ...

    BotT = TypeVar("BotT", bound=Bot, default=Bot)

    class Context(Generic[BotT]):
        bot: BotT

    class MyBot(Bot): ...

    reveal_type(Context().bot)         # type is Bot  # notice this is not Any which is what it would be currently
    reveal_type(Context[MyBot]().bot)  # type is MyBot

Not only does this improve typing for those who explicitly use it, it
also helps non-typing users who rely on auto-complete to speed up their
development.

This design pattern is common in projects like:

-   discord.py — where the example above was taken from.
-   NumPy — the default for types like ndarray's dtype would be float64.
    Currently it's Unknown or Any.
-   TensorFlow — this could be used for Tensor similarly to
    numpy.ndarray and would be useful to simplify the definition of
    Layer.

Specification

Default Ordering and Subscription Rules

The order for defaults should follow the standard function parameter
rules, so a type parameter with no default cannot follow one with a
default value. Doing so should ideally raise a TypeError in
typing._GenericAlias/types.GenericAlias, and a type checker should flag
this as an error.

    DefaultStrT = TypeVar("DefaultStrT", default=str)
    DefaultIntT = TypeVar("DefaultIntT", default=int)
    DefaultBoolT = TypeVar("DefaultBoolT", default=bool)
    T = TypeVar("T")
    T2 = TypeVar("T2")

    class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ...  # Invalid: non-default TypeVars cannot follow ones with defaults

    class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ...

    (
        NoNoneDefaults ==
        NoNoneDefaults[str] ==
        NoNoneDefaults[str, int]
    )  # All valid

    class OneDefault(Generic[T, DefaultBoolT]): ...

    OneDefault[float] == OneDefault[float, bool]  # Valid
    reveal_type(OneDefault)          # type is type[OneDefault[T, DefaultBoolT = bool]]
    reveal_type(OneDefault[float]()) # type is OneDefault[float, bool]

    class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ...

    reveal_type(AllTheDefaults)                  # type is type[AllTheDefaults[T1, T2, DefaultStrT = str, DefaultIntT = int, DefaultBoolT = bool]]
    reveal_type(AllTheDefaults[int, complex]())  # type is AllTheDefaults[int, complex, str, int, bool]
    AllTheDefaults[int]  # Invalid: expected 2 arguments to AllTheDefaults
    (
        AllTheDefaults[int, complex] ==
        AllTheDefaults[int, complex, str] ==
        AllTheDefaults[int, complex, str, int] ==
        AllTheDefaults[int, complex, str, int, bool]
    )  # All valid

With the new Python 3.12 syntax for generics (introduced by PEP 695),
this can be enforced at compile time:

    type Alias[DefaultT = int, T] = tuple[DefaultT, T]  # SyntaxError: non-default TypeVars cannot follow ones with defaults

    def generic_func[DefaultT = int, T](x: DefaultT, y: T) -> None: ...  # SyntaxError: non-default TypeVars cannot follow ones with defaults

    class GenericClass[DefaultT = int, T]: ...  # SyntaxError: non-default TypeVars cannot follow ones with defaults

ParamSpec Defaults

ParamSpec defaults are defined using the same syntax as TypeVar s but
use a list of types or an ellipsis literal "..." or another in-scope
ParamSpec (see Scoping Rules).

    DefaultP = ParamSpec("DefaultP", default=[str, int])

    class Foo(Generic[DefaultP]): ...

    reveal_type(Foo)                  # type is type[Foo[DefaultP = [str, int]]]
    reveal_type(Foo())                # type is Foo[[str, int]]
    reveal_type(Foo[[bool, bool]]())  # type is Foo[[bool, bool]]

TypeVarTuple Defaults

TypeVarTuple defaults are defined using the same syntax as TypeVar s but
use an unpacked tuple of types instead of a single type or another
in-scope TypeVarTuple (see Scoping Rules).

    DefaultTs = TypeVarTuple("DefaultTs", default=Unpack[tuple[str, int]])

    class Foo(Generic[*DefaultTs]): ...

    reveal_type(Foo)               # type is type[Foo[DefaultTs = *tuple[str, int]]]
    reveal_type(Foo())             # type is Foo[str, int]
    reveal_type(Foo[int, bool]())  # type is Foo[int, bool]

Using Another Type Parameter as default

This allows for a value to be used again when the type parameter to a
generic is missing but another type parameter is specified.

To use another type parameter as a default the default and the type
parameter must be the same type (a TypeVar's default must be a TypeVar,
etc.).

This could be used on builtins.slice where the start parameter should
default to int, stop default to the type of start and step default to
int | None.

    StartT = TypeVar("StartT", default=int)
    StopT = TypeVar("StopT", default=StartT)
    StepT = TypeVar("StepT", default=int | None)

    class slice(Generic[StartT, StopT, StepT]): ...

    reveal_type(slice)  # type is type[slice[StartT = int, StopT = StartT, StepT = int | None]]
    reveal_type(slice())                        # type is slice[int, int, int | None]
    reveal_type(slice[str]())                   # type is slice[str, str, int | None]
    reveal_type(slice[str, bool, timedelta]())  # type is slice[str, bool, timedelta]

    T2 = TypeVar("T2", default=DefaultStrT)

    class Foo(Generic[DefaultStrT, T2]):
        def __init__(self, a: DefaultStrT, b: T2) -> None: ...

    reveal_type(Foo(1, ""))  # type is Foo[int, str]
    Foo[int](1, "")          # Invalid: Foo[int, str] cannot be assigned to self: Foo[int, int] in Foo.__init__
    Foo[int]("", 1)          # Invalid: Foo[str, int] cannot be assigned to self: Foo[int, int] in Foo.__init__

When using a type parameter as the default to another type parameter,
the following rules apply, where T1 is the default for T2.

Scoping Rules

T1 must be used before T2 in the parameter list of the generic.

    T2 = TypeVar("T2", default=T1)

    class Foo(Generic[T1, T2]): ...   # Valid
    class Foo(Generic[T1]):
        class Bar(Generic[T2]): ...   # Valid

    StartT = TypeVar("StartT", default="StopT")  # Swapped defaults around from previous example
    StopT = TypeVar("StopT", default=int)
    class slice(Generic[StartT, StopT, StepT]): ...
                      # ^^^^^^ Invalid: ordering does not allow StopT to be bound

Using a type parameter from an outer scope as a default is not
supported.

Bound Rules

T1's bound must be a subtype of T2's bound.

    T1 = TypeVar("T1", bound=int)
    TypeVar("Ok", default=T1, bound=float)     # Valid
    TypeVar("AlsoOk", default=T1, bound=int)   # Valid
    TypeVar("Invalid", default=T1, bound=str)  # Invalid: int is not a subtype of str

Constraint Rules

The constraints of T2 must be a superset of the constraints of T1.

    T1 = TypeVar("T1", bound=int)
    TypeVar("Invalid", float, str, default=T1)         # Invalid: upper bound int is incompatible with constraints float or str

    T1 = TypeVar("T1", int, str)
    TypeVar("AlsoOk", int, str, bool, default=T1)      # Valid
    TypeVar("AlsoInvalid", bool, complex, default=T1)  # Invalid: {bool, complex} is not a superset of {int, str}

Type Parameters as Parameters to Generics

Type parameters are valid as parameters to generics inside of a default
when the first parameter is in scope as determined by the previous
section.

    T = TypeVar("T")
    ListDefaultT = TypeVar("ListDefaultT", default=list[T])

    class Bar(Generic[T, ListDefaultT]):
        def __init__(self, x: T, y: ListDefaultT): ...

    reveal_type(Bar)                    # type is type[Bar[T, ListDefaultT = list[T]]]
    reveal_type(Bar[int])               # type is type[Bar[int, list[int]]]
    reveal_type(Bar[int]())             # type is Bar[int, list[int]]
    reveal_type(Bar[int, list[str]]())  # type is Bar[int, list[str]]
    reveal_type(Bar[int, str]())        # type is Bar[int, str]

Specialisation Rules

Type parameters currently cannot be further subscripted. This might
change if Higher Kinded TypeVars are implemented.

Generic TypeAliases

Generic TypeAliases should be able to be further subscripted following
normal subscription rules. If a type parameter has a default that hasn't
been overridden it should be treated like it was substituted into the
TypeAlias. However, it can be specialised further down the line.

    class SomethingWithNoDefaults(Generic[T, T2]): ...

    MyAlias: TypeAlias = SomethingWithNoDefaults[int, DefaultStrT]  # Valid
    reveal_type(MyAlias)          # type is type[SomethingWithNoDefaults[int, DefaultStrT]]
    reveal_type(MyAlias[bool]())  # type is SomethingWithNoDefaults[int, bool]

    MyAlias[bool, int]  # Invalid: too many arguments passed to MyAlias

Subclassing

Subclasses of Generics with type parameters that have defaults behave
similarly to Generic TypeAliases. That is, subclasses can be further
subscripted following normal subscription rules, non-overridden defaults
should be substituted in, and type parameters with such defaults can be
further specialised down the line.

    class SubclassMe(Generic[T, DefaultStrT]):
        x: DefaultStrT

    class Bar(SubclassMe[int, DefaultStrT]): ...
    reveal_type(Bar)          # type is type[Bar[DefaultStrT = str]]
    reveal_type(Bar())        # type is Bar[str]
    reveal_type(Bar[bool]())  # type is Bar[bool]

    class Foo(SubclassMe[float]): ...

    reveal_type(Foo().x)  # type is str

    Foo[str]  # Invalid: Foo cannot be further subscripted

    class Baz(Generic[DefaultIntT, DefaultStrT]): ...

    class Spam(Baz): ...
    reveal_type(Spam())  # type is <subclass of Baz[int, str]>

Using bound and default

If both bound and default are passed default must be a subtype of bound.
Otherwise the type checker should generate an error.

    TypeVar("Ok", bound=float, default=int)     # Valid
    TypeVar("Invalid", bound=str, default=int)  # Invalid: the bound and default are incompatible

Constraints

For constrained TypeVars, the default needs to be one of the
constraints. A type checker should generate an error even if it is a
subtype of one of the constraints.

    TypeVar("Ok", float, str, default=float)     # Valid
    TypeVar("Invalid", float, str, default=int)  # Invalid: expected one of float or str got int

Function Defaults

In generic functions, type checkers may use a type parameter's default
when the type parameter cannot be solved to anything. We leave the
semantics of this usage unspecified, as ensuring the default is returned
in every code path where the type parameter can go unsolved may be too
hard to implement. Type checkers are free to either disallow this case
or experiment with implementing support.

    T = TypeVar('T', default=int)
    def func(x: int | set[T]) -> T: ...
    reveal_type(func(0))  # a type checker may reveal T's default of int here

Defaults following TypeVarTuple

A TypeVar that immediately follows a TypeVarTuple is not allowed to have
a default, because it would be ambiguous whether a type argument should
be bound to the TypeVarTuple or the defaulted TypeVar.

    Ts = TypeVarTuple("Ts")
    T = TypeVar("T", default=bool)

    class Foo(Generic[Ts, T]): ...  # Type checker error

    # Could be reasonably interpreted as either Ts = (int, str, float), T = bool
    # or Ts = (int, str), T = float
    Foo[int, str, float]

With the Python 3.12 built-in generic syntax, this case should raise a
SyntaxError.

However, it is allowed to have a ParamSpec with a default following a
TypeVarTuple with a default, as there can be no ambiguity between a type
argument for the ParamSpec and one for the TypeVarTuple.

    Ts = TypeVarTuple("Ts")
    P = ParamSpec("P", default=[float, bool])

    class Foo(Generic[Ts, P]): ...  # Valid

    Foo[int, str]  # Ts = (int, str), P = [float, bool]
    Foo[int, str, [bytes]]  # Ts = (int, str), P = [bytes]

Subtyping

Type parameter defaults do not affect the subtyping rules for generic
classes. In particular, defaults can be ignored when considering whether
a class is compatible with a generic protocol.

TypeVarTuples as Defaults

Using a TypeVarTuple as a default is not supported because:

-   Scoping Rules does not allow usage of type parameters from outer
    scopes.
-   Multiple TypeVarTuples cannot appear in the type parameter list for
    a single object, as specified in
    646#multiple-type-variable-tuples-not-allowed.

These reasons leave no current valid location where a TypeVarTuple could
be used as the default of another TypeVarTuple.

Binding rules

Type parameter defaults should be bound by attribute access (including
call and subscript).

    class Foo[T = int]:
        def meth(self) -> Self:
            return self

    reveal_type(Foo.meth)  # type is (self: Foo[int]) -> Foo[int]

Implementation

At runtime, this would involve the following changes to the typing
module.

-   The classes TypeVar, ParamSpec, and TypeVarTuple should expose the
    type passed to default. This would be available as a __default__
    attribute, which would be None if no argument is passed and NoneType
    if default=None.

The following changes would be required to both GenericAliases:

-   logic to determine the defaults required for a subscription.
-   ideally, logic to determine if subscription (like
    Generic[T, DefaultT]) would be valid.

The grammar for type parameter lists would need to be updated to allow
defaults; see below.

A reference implementation of the runtime changes can be found at
https://github.com/Gobot1234/cpython/tree/pep-696

A reference implementation of the type checker can be found at
https://github.com/Gobot1234/mypy/tree/TypeVar-defaults

Pyright currently supports this functionality.

Grammar changes

The syntax added in PEP 695 will be extended to introduce a way to
specify defaults for type parameters using the "=" operator inside of
the square brackets like so:

    # TypeVars
    class Foo[T = str]: ...

    # ParamSpecs
    class Baz[**P = [int, str]]: ...

    # TypeVarTuples
    class Qux[*Ts = *tuple[int, bool]]: ...

    # TypeAliases
    type Foo[T, U = str] = Bar[T, U]
    type Baz[**P = [int, str]] = Spam[**P]
    type Qux[*Ts = *tuple[str]] = Ham[*Ts]
    type Rab[U, T = str] = Bar[T, U]

Similarly to the bound for a type parameter <695-scoping-behavior>,
defaults should be lazily evaluated, with the same scoping rules to
avoid the unnecessary usage of quotes around them.

This functionality was included in the initial draft of PEP 695 but was
removed due to scope creep.

The following changes would be made to the grammar:

    type_param:
        | a=NAME b=[type_param_bound] d=[type_param_default]
        | a=NAME c=[type_param_constraint] d=[type_param_default]
        | '*' a=NAME d=[type_param_default]
        | '**' a=NAME d=[type_param_default]

    type_param_default:
        | '=' e=expression
        | '=' e=starred_expression

The compiler would enforce that type parameters without defaults cannot
follow type parameters with defaults and that TypeVars with defaults
cannot immediately follow TypeVarTuples.

Rejected Alternatives

Allowing the Type Parameters Defaults to Be Passed to type.__new__'s **kwargs

    T = TypeVar("T")

    @dataclass
    class Box(Generic[T], T=int):
        value: T | None = None

While this is much easier to read and follows a similar rationale to the
TypeVar unary syntax, it would not be backwards compatible as T might
already be passed to a metaclass/superclass or support classes that
don't subclass Generic at runtime.

Ideally, if PEP 637 wasn't rejected, the following would be acceptable:

    T = TypeVar("T")

    @dataclass
    class Box(Generic[T = int]):
        value: T | None = None

Allowing Non-defaults to Follow Defaults

    YieldT = TypeVar("YieldT", default=Any)
    SendT = TypeVar("SendT", default=Any)
    ReturnT = TypeVar("ReturnT")

    class Coroutine(Generic[YieldT, SendT, ReturnT]): ...

    Coroutine[int] == Coroutine[Any, Any, int]

Allowing non-defaults to follow defaults would alleviate the issues with
returning types like Coroutine from functions where the most used type
argument is the last (the return). Allowing non-defaults to follow
defaults is too confusing and potentially ambiguous, even if only the
above two forms were valid. Changing the argument order now would also
break a lot of codebases. This is also solvable in most cases using a
TypeAlias.

    Coro: TypeAlias = Coroutine[Any, Any, T]
    Coro[int] == Coroutine[Any, Any, int]

Having default Implicitly Be bound

In an earlier version of this PEP, the default was implicitly set to
bound if no value was passed for default. This while convenient, could
have a type parameter with no default follow a type parameter with a
default. Consider:

    T = TypeVar("T", bound=int)  # default is implicitly int
    U = TypeVar("U")

    class Foo(Generic[T, U]):
        ...

    # would expand to

    T = TypeVar("T", bound=int, default=int)
    U = TypeVar("U")

    class Foo(Generic[T, U]):
        ...

This would have also been a breaking change for a small number of cases
where the code relied on Any being the implicit default.

Allowing Type Parameters With Defaults To Be Used in Function Signatures

A previous version of this PEP allowed TypeVarLikes with defaults to be
used in function signatures. This was removed for the reasons described
in Function Defaults. Hopefully, this can be added in the future if a
way to get the runtime value of a type parameter is added.

Allowing Type Parameters from Outer Scopes in default

This was deemed too niche a feature to be worth the added complexity. If
any cases arise where this is needed, it can be added in a future PEP.

Acknowledgements

Thanks to the following people for their feedback on the PEP:

Eric Traut, Jelle Zijlstra, Joshua Butt, Danny Yamamoto, Kaylynn Morgan
and Jakub Kuczys

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.