text:
PEP: 413 Title: Faster evolution of the Python Standard Library Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Status: Withdrawn Type: Process Content-Type:
text/x-rst Created: 24-Feb-2012 Post-History: 24-Feb-2012, 25-Feb-2012

PEP Withdrawal

With the acceptance of PEP 453 meaning that pip will be available to
most new Python users by default, this will hopefully reduce the
pressure to add new modules to the standard library before they are
sufficiently mature.

The last couple of years have also seen increased usage of the model
where a standard library package also has an equivalent available from
the Python Package Index that also supports older versions of Python.

Given these two developments and the level of engagement throughout the
Python 3.4 release cycle, the PEP author no longer feels it would be
appropriate to make such a fundamental change to the standard library
development process.

Abstract

This PEP proposes the adoption of a separate versioning scheme for the
standard library (distinct from, but coupled to, the existing language
versioning scheme) that allows accelerated releases of the Python
standard library, while maintaining (or even slowing down) the current
rate of change in the core language definition.

Like PEP 407, it aims to adjust the current balance between measured
change that allows the broader community time to adapt and being able to
keep pace with external influences that evolve more rapidly than the
current release cycle can handle (this problem is particularly notable
for standard library elements that relate to web technologies).

However, it's more conservative in its aims than PEP 407, seeking to
restrict the increased pace of development to builtin and standard
library interfaces, without affecting the rate of change for other
elements such as the language syntax and version numbering as well as
the CPython binary API and bytecode format.

Rationale

To quote the PEP 407 abstract:

  Finding a release cycle for an open-source project is a delicate
  exercise in managing mutually contradicting constraints: developer
  manpower, availability of release management volunteers, ease of
  maintenance for users and third-party packagers, quick availability of
  new features (and behavioural changes), availability of bug fixes
  without pulling in new features or behavioural changes.

  The current release cycle errs on the conservative side. It is
  adequate for people who value stability over reactivity. This PEP is
  an attempt to keep the stability that has become a Python trademark,
  while offering a more fluid release of features, by introducing the
  notion of long-term support versions.

I agree with the PEP 407 authors that the current release cycle of the
standard library is too slow to effectively cope with the pace of change
in some key programming areas (specifically, web protocols and related
technologies, including databases, templating and serialisation
formats).

However, I have written this competing PEP because I believe that the
approach proposed in PEP 407 of offering full, potentially binary
incompatible releases of CPython every 6 months places too great a
burden on the wider Python ecosystem.

Under the current CPython release cycle, distributors of key binary
extensions will often support Python releases even after the CPython
branches enter "security fix only" mode (for example, Twisted currently
ships binaries for 2.5, 2.6 and 2.7, NumPy and SciPy support those 3
along with 3.1 and 3.2, PyGame adds a 2.4 binary release, wxPython
provides both 32-bit and 64-bit binaries for 2.6 and 2.7, etc).

If CPython were to triple (or more) its rate of releases, the developers
of those libraries (many of which are even more resource starved than
CPython) would face an unpalatable choice: either adopt the faster
release cycle themselves (up to 18 simultaneous binary releases for
PyGame!), drop older Python versions more quickly, or else tell their
users to stick to the CPython LTS releases (thus defeating the entire
point of speeding up the CPython release cycle in the first place).

Similarly, many support tools for Python (e.g. syntax highlighters) can
take quite some time to catch up with language level changes.

At a cultural level, the Python community is also accustomed to a
certain meaning for Python version numbers - they're linked to
deprecation periods, support periods, all sorts of things. PEP 407
proposes that collective knowledge all be swept aside, without offering
a compelling rationale for why such a course of action is actually
necessary (aside from, perhaps, making the lives of the CPython core
developers a little easier at the expense of everyone else).

However, if we go back to the primary rationale for increasing the pace
of change (i.e. more timely support for web protocols and related
technologies), we can note that those only require standard library
changes. That means many (perhaps even most) of the negative effects on
the wider community can be avoided by explicitly limiting which parts of
CPython are affected by the new release cycle, and allowing other parts
to evolve at their current, more sedate, pace.

Proposal

This PEP proposes the introduction of a new kind of CPython release:
"standard library releases". As with PEP 407, this will give CPython 3
kinds of release:

-   Language release: "x.y.0"
-   Maintenance release: "x.y.z" (where z > 0)
-   Standard library release: "x.y (xy.z)" (where z > 0)

Under this scheme, an unqualified version reference (such as "3.3")
would always refer to the most recent corresponding language or
maintenance release. It will never be used without qualification to
refer to a standard library release (at least, not by python-dev -
obviously, we can only set an example, not force the rest of the Python
ecosystem to go along with it).

Language releases will continue as they are now, as new versions of the
Python language definition, along with a new version of the CPython
interpreter and the Python standard library. Accordingly, a language
release may contain any and all of the following changes:

-   new language syntax
-   new standard library changes (see below)
-   new deprecation warnings
-   removal of previously deprecated features
-   changes to the emitted bytecode
-   changes to the AST
-   any other significant changes to the compilation toolchain
-   changes to the core interpreter eval loop
-   binary incompatible changes to the C ABI (although the PEP 384
    stable ABI must still be preserved)
-   bug fixes

Maintenance releases will also continue as they do today, being strictly
limited to bug fixes for the corresponding language release. No new
features or radical internal changes are permitted.

The new standard library releases will occur in parallel with each
maintenance release and will be qualified with a new version identifier
documenting the standard library version. Standard library releases may
include the following changes:

-   new features in pure Python modules
-   new features in C extension modules (subject to PEP 399
    compatibility requirements)
-   new features in language builtins (provided the C ABI remains
    unaffected)
-   bug fixes from the corresponding maintenance release

Standard library version identifiers are constructed by combining the
major and minor version numbers for the Python language release into a
single two digit number and then appending a sequential standard library
version identifier.

Release Cycle

When maintenance releases are created, two new versions of Python would
actually be published on python.org (using the first 3.3 maintenance
release, planned for February 2013 as an example):

    3.3.1       # Maintenance release
    3.3 (33.1)  # Standard library release

A further 6 months later, the next 3.3 maintenance release would again
be accompanied by a new standard library release:

    3.3.2       # Maintenance release
    3.3 (33.2)  # Standard library release

Again, the standard library release would be binary compatible with the
previous language release, merely offering additional features at the
Python level.

Finally, 18 months after the release of 3.3, a new language release
would be made around the same time as the final 3.3 maintenance and
standard library releases:

    3.3.3       # Maintenance release
    3.3 (33.3)  # Standard library release
    3.4.0       # Language release

The 3.4 release cycle would then follow a similar pattern to that for
3.3:

    3.4.1       # Maintenance release
    3.4 (34.1)  # Standard library release

    3.4.2       # Maintenance release
    3.4 (34.2)  # Standard library release

    3.4.3       # Maintenance release
    3.4 (34.3)  # Standard library release
    3.5.0       # Language release

Programmatic Version Identification

To expose the new version details programmatically, this PEP proposes
the addition of a new sys.stdlib_info attribute that records the new
standard library version above and beyond the underlying interpreter
version. Using the initial Python 3.3 release as an example:

    sys.stdlib_info(python=33, version=0, releaselevel='final', serial=0)

This information would also be included in the sys.version string:

    Python 3.3.0 (33.0, default, Feb 17 2012, 23:03:41)
    [GCC 4.6.1]

Security Fixes and Other "Out of Cycle" Releases

For maintenance releases the process of handling out-of-cycle releases
(for example, to fix a security issue or resolve a critical bug in a new
release), remains the same as it is now: the minor version number is
incremented and a new release is made incorporating the required bug
fixes, as well as any other bug fixes that have been committed since the
previous release.

For standard library releases, the process is essentially the same, but
the corresponding "What's New?" document may require some tidying up for
the release (as the standard library release may incorporate new
features, not just bug fixes).

User Scenarios

The versioning scheme proposed above is based on a number of user
scenarios that are likely to be encountered if this scheme is adopted.
In each case, the scenario is described for both the status quo (i.e.
slow release cycle) the versioning scheme in this PEP and the free
wheeling minor version number scheme proposed in PEP 407.

To give away the ending, the point of using a separate version number is
that for almost all scenarios, the important number is the language
version, not the standard library version. Most users won't even need to
care that the standard library version number exists. In the two
identified cases where it matters, providing it as a separate number is
actually clearer and more explicit than embedding the two different
kinds of number into a single sequence and then tagging some of the
numbers in the unified sequence as special.

Novice user, downloading Python from python.org in March 2013

Status quo: must choose between 3.3 and 2.7

This PEP: must choose between 3.3 (33.1), 3.3 and 2.7.

PEP 407: must choose between 3.4, 3.3 (LTS) and 2.7.

Verdict: explaining the meaning of a Long Term Support release is about
as complicated as explaining the meaning of the proposed standard
library release version numbers. I call this a tie.

Novice user, attempting to judge currency of third party documentation

Status quo: minor version differences indicate 18-24 months of language
evolution

This PEP: same as status quo for language core, standard library version
numbers indicate 6 months of standard library evolution.

PEP 407: minor version differences indicate 18-24 months of language
evolution up to 3.3, then 6 months of language evolution thereafter.

Verdict: Since language changes and deprecations can have a much bigger
effect on the accuracy of third party documentation than the addition of
new features to the standard library, I'm calling this a win for the
scheme in this PEP.

Novice user, looking for an extension module binary release

Status quo: look for the binary corresponding to the Python version you
are running.

This PEP: same as status quo.

PEP 407 (full releases): same as status quo, but corresponding binary
version is more likely to be missing (or, if it does exist, has to be
found amongst a much larger list of alternatives).

PEP 407 (ABI updates limited to LTS releases): all binary release pages
will need to tell users that Python 3.3, 3.4 and 3.5 all need the 3.3
binary.

Verdict: I call this a clear win for the scheme in this PEP. Absolutely
nothing changes from the current situation, since the standard library
version is actually irrelevant in this case (only binary extension
compatibility is important).

Extension module author, deciding whether or not to make a binary release

Status quo: unless using the PEP 384 stable ABI, a new binary release is
needed every time the minor version number changes.

This PEP: same as status quo.

PEP 407 (full releases): same as status quo, but becomes a far more
frequent occurrence.

PEP 407 (ABI updates limited to LTS releases): before deciding, must
first look up whether the new release is an LTS release or an interim
release. If it is an LTS release, then a new build is necessary.

Verdict: I call this another clear win for the scheme in this PEP. As
with the end user facing side of this problem, the standard library
version is actually irrelevant in this case. Moving that information out
to a separate number avoids creating unnecessary confusion.

Python developer, deciding priority of eliminating a Deprecation Warning

Status quo: code that triggers deprecation warnings is not guaranteed to
run on a version of Python with a higher minor version number.

This PEP: same as status quo

PEP 407: unclear, as the PEP doesn't currently spell this out. Assuming
the deprecation cycle is linked to LTS releases, then upgrading to a
non-LTS release is safe but upgrading to the next LTS release may
require avoiding the deprecated construct.

Verdict: another clear win for the scheme in this PEP since, once again,
the standard library version is irrelevant in this scenario.

Alternative interpreter implementor, updating with new features

Status quo: new Python versions arrive infrequently, but are a mish-mash
of standard library updates and core language definition and interpreter
changes.

This PEP: standard library updates, which are easier to integrate, are
made available more frequently in a form that is clearly and explicitly
compatible with the previous version of the language definition. This
means that, once an alternative implementation catches up to Python 3.3,
they should have a much easier time incorporating standard library
features as they happen (especially pure Python changes), leaving minor
version number updates as the only task that requires updates to their
core compilation and execution components.

PEP 407 (full releases): same as status quo, but becomes a far more
frequent occurrence.

PEP 407 (language updates limited to LTS releases): unclear, as the PEP
doesn't currently spell out a specific development strategy. Assuming a
3.3 compatibility branch is adopted (as proposed in this PEP), then the
outcome would be much the same, but the version number signalling would
be slightly less clear (since you would have to check to see if a
particular release was an LTS release or not).

Verdict: while not as clear cut as some previous scenarios, I'm still
calling this one in favour of the scheme in this PEP. Explicit is better
than implicit, and the scheme in this PEP makes a clear split between
the two different kinds of update rather than adding a separate "LTS"
tag to an otherwise ordinary release number. Tagging a particular
version as being special is great for communicating with version control
systems and associated automated tools, but it's a lousy way to
communicate information to other humans.

Python developer, deciding their minimum version dependency

Status quo: look for "version added" or "version changed" markers in the
documentation, check against sys.version_info

This PEP: look for "version added" or "version changed" markers in the
documentation. If written as a bare Python version, such as "3.3", check
against sys.version_info. If qualified with a standard library version,
such as "3.3 (33.1)", check against sys.stdlib_info.

PEP 407: same as status quo

Verdict: the scheme in this PEP actually allows third party libraries to
be more explicit about their rate of adoption of standard library
features. More conservative projects will likely pin their dependency to
the language version and avoid features added in the standard library
releases. Faster moving projects could instead declare their dependency
on a particular standard library version. However, since PEP 407 does
have the advantage of preserving the status quo, I'm calling this one
for PEP 407 (albeit with a slim margin).

Python developers, attempting to reproduce a tracker issue

Status quo: if not already provided, ask the reporter which version of
Python they're using. This is often done by asking for the first two
lines displayed by the interactive prompt or the value of sys.version.

This PEP: same as the status quo (as sys.version will be updated to also
include the standard library version), but may be needed on additional
occasions (where the user knew enough to state their Python version, but
that proved to be insufficient to reproduce the fault).

PEP 407: same as the status quo

Verdict: another marginal win for PEP 407. The new standard library
version is an extra piece of information that users may need to pass
back to developers when reporting issues with Python libraries (or
Python itself, on our own tracker). However, by including it in
sys.version, many fault reports will already include it, and it is easy
to request if needed.

CPython release managers, handling a security fix

Status quo: create a new maintenance release incorporating the security
fix and any other bug fixes under source control. Also create source
releases for any branches open solely for security fixes.

This PEP: same as the status quo for maintenance branches. Also create a
new standard library release (potentially incorporating new features
along with the security fix). For security branches, create source
releases for both the former maintenance branch and the standard library
update branch.

PEP 407: same as the status quo for maintenance and security branches,
but handling security fixes for non-LTS releases is currently an open
question.

Verdict: until PEP 407 is updated to actually address this scenario, a
clear win for this PEP.

Effects

Effect on development cycle

Similar to PEP 407, this PEP will break up the delivery of new features
into more discrete chunks. Instead of a whole raft of changes landing
all at once in a language release, each language release will be limited
to 6 months worth of standard library changes, as well as any changes
associated with new syntax.

Effect on workflow

This PEP proposes the creation of a single additional branch for use in
the normal workflow. After the release of 3.3, the following branches
would be in use:

    2.7         # Maintenance branch, no change
    3.3         # Maintenance branch, as for 3.2
    3.3-compat  # New branch, backwards compatible changes
    default     # Language changes, standard library updates that depend on them

When working on a new feature, developers will need to decide whether or
not it is an acceptable change for a standard library release. If so,
then it should be checked in on 3.3-compat and then merged to default.
Otherwise it should be checked in directly to default.

The "version added" and "version changed" markers for any changes made
on the 3.3-compat branch would need to be flagged with both the language
version and the standard library version. For example: "3.3 (33.1)".

Any changes made directly on the default branch would just be flagged
with "3.4" as usual.

The 3.3-compat branch would be closed to normal development at the same
time as the 3.3 maintenance branch. The 3.3-compat branch would remain
open for security fixes for the same period of time as the 3.3
maintenance branch.

Effect on bugfix cycle

The effect on the bug fix workflow is essentially the same as that on
the workflow for new features - there is one additional branch to pass
through before the change reaches the default branch.

If critical bugs are found in a maintenance release, then new
maintenance and standard library releases will be created to resolve the
problem. The final part of the version number will be incremented for
both the language version and the standard library version.

If critical bugs are found in a standard library release that do not
affect the associated maintenance release, then only a new standard
library release will be created and only the standard library's version
number will be incremented.

Note that in these circumstances, the standard library release may
include additional features, rather than just containing the bug fix. It
is assumed that anyone that cares about receiving only bug fixes without
any new features mixed in will already be relying strictly on the
maintenance releases rather than using the new standard library
releases.

Effect on the community

PEP 407 has this to say about the effects on the community:

  People who value stability can just synchronize on the LTS releases
  which, with the proposed figures, would give a similar support cycle
  (both in duration and in stability).

I believe this statement is just plain wrong. Life isn't that simple.
Instead, developers of third party modules and frameworks will come
under pressure to support the full pace of the new release cycle with
binary updates, teachers and book authors will receive complaints that
they're only covering an "old" version of Python ("You're only using
3.3, the latest is 3.5!"), etc.

As the minor version number starts climbing 3 times faster than it has
in the past, I believe perceptions of language stability would also fall
(whether such opinions were justified or not).

I believe isolating the increased pace of change to the standard
library, and clearly delineating it with a separate version number will
greatly reassure the rest of the community that no, we're not suddenly
asking them to triple their own rate of development. Instead, we're
merely going to ship standard library updates for the next language
release in 6-monthly installments rather than delaying them all until
the next language definition update, even those changes that are
backwards compatible with the previously released version of Python.

The community benefits listed in PEP 407 are equally applicable to this
PEP, at least as far as the standard library is concerned:

  People who value reactivity and access to new features (without taking
  the risk to install alpha versions or Mercurial snapshots) would get
  much more value from the new release cycle than currently.

  People who want to contribute new features or improvements would be
  more motivated to do so, knowing that their contributions will be more
  quickly available to normal users.

If the faster release cycle encourages more people to focus on
contributing to the standard library rather than proposing changes to
the language definition, I don't see that as a bad thing.

Handling News Updates

What's New?

The "What's New" documents would be split out into separate documents
for standard library releases and language releases. So, during the 3.3
release cycle, we would see:

-   What's New in Python 3.3?
-   What's New in the Python Standard Library 33.1?
-   What's New in the Python Standard Library 33.2?
-   What's New in the Python Standard Library 33.3?

And then finally, we would see the next language release:

-   What's New in Python 3.4?

For the benefit of users that ignore standard library releases, the 3.4
What's New would link back to the What's New documents for each of the
standard library releases in the 3.3 series.

NEWS

Merge conflicts on the NEWS file are already a hassle. Since this PEP
proposes introduction of an additional branch into the normal workflow,
resolving this becomes even more critical. While Mercurial phases may
help to some degree, it would be good to eliminate the problem entirely.

One suggestion from Barry Warsaw is to adopt a non-conflicting
separate-files-per-change approach, similar to that used by Twisted[1].

Given that the current manually updated NEWS file will be used for the
3.3.0 release, one possible layout for such an approach might look like:

    Misc/
      NEWS  # Now autogenerated from news_entries
      news_entries/
        3.3/
          NEWS # Original 3.3 NEWS file
          maint.1/ # Maintenance branch changes
            core/
              <news entries>
            builtins/
              <news entries>
            extensions/
              <news entries>
            library/
              <news entries>
            documentation/
              <news entries>
            tests/
              <news entries>
          compat.1/ # Compatibility branch changes
            builtins/
              <news entries>
            extensions/
              <news entries>
            library/
              <news entries>
            documentation/
              <news entries>
            tests/
              <news entries>
          # Add maint.2, compat.2 etc as releases are made
        3.4/
          core/
            <news entries>
          builtins/
            <news entries>
          extensions/
            <news entries>
          library/
            <news entries>
          documentation/
            <news entries>
          tests/
            <news entries>
          # Add maint.1, compat.1 etc as releases are made

Putting the version information in the directory hierarchy isn't
strictly necessary (since the NEWS file generator could figure out from
the version history), but does make it easier for humans to keep the
different versions in order.

Other benefits of reduced version coupling

Slowing down the language release cycle

The current release cycle is a compromise between the desire for
stability in the core language definition and C extension ABI, and the
desire to get new features (most notably standard library updates) into
user's hands more quickly.

With the standard library release cycle decoupled (to some degree) from
that of the core language definition, it provides an opportunity to
actually slow down the rate of change in the language definition. The
language moratorium for Python 3.2 effectively slowed that cycle down to
more than 3 years (3.1: June 2009, 3.3: August 2012) without causing any
major problems or complaints.

The NEWS file management scheme described above is actually designed to
allow us the flexibility to slow down language releases at the same time
as standard library releases become more frequent.

As a simple example, if a full two years was allowed between 3.3 and
3.4, the 3.3 release cycle would end up looking like:

    3.2.4       # Maintenance release
    3.3.0       # Language release

    3.3.1       # Maintenance release
    3.3 (33.1)  # Standard library release

    3.3.2       # Maintenance release
    3.3 (33.2)  # Standard library release

    3.3.3       # Maintenance release
    3.3 (33.3)  # Standard library release

    3.3.4       # Maintenance release
    3.3 (33.4)  # Standard library release
    3.4.0       # Language release

The elegance of the proposed branch structure and NEWS entry layout is
that this decision wouldn't really need to be made until shortly before
the planned 3.4 release date. At that point, the decision could be made
to postpone the 3.4 release and keep the 3.3 and 3.3-compat branches
open after the 3.3.3 maintenance release and the 3.3 (33.3) standard
library release, thus adding another standard library release to the
cycle. The choice between another standard library release or a full
language release would then be available every 6 months after that.

Further increasing the pace of standard library development

As noted in the previous section, one benefit of the scheme proposed in
this PEP is that it largely decouples the language release cycle from
the standard library release cycle. The standard library could be
updated every 3 months, or even once a month, without having any flow on
effects on the language version numbering or the perceived stability of
the core language.

While that pace of development isn't practical as long as the binary
installer creation for Windows and Mac OS X involves several manual
steps (including manual testing) and for as long as we don't have
separate "<branch>-release" trees that only receive versions that have
been marked as good by the stable buildbots, it's still a useful
criterion to keep in mind when considering proposed new versioning
schemes: what if we eventually want to make standard library releases
even faster than every 6 months?

If the practical issues were ever resolved, then the separate standard
library versioning scheme in this PEP could handle it. The tagged
version number approach proposed in PEP 407 could not (at least, not
without a lot of user confusion and uncertainty).

Other Questions

Why not use the major version number?

The simplest and most logical solution would actually be to map the
major.minor.micro version numbers to the language version, stdlib
version and maintenance release version respectively.

Instead of releasing Python 3.3.0, we would instead release Python 4.0.0
and the release cycle would look like:

    4.0.0  # Language release

    4.0.1  # Maintenance release
    4.1.0  # Standard library release

    4.0.2  # Maintenance release
    4.2.0  # Standard library release

    4.0.3  # Maintenance release
    4.3.0  # Standard library release
    5.0.0  # Language release

However, the ongoing pain of the Python 2 -> Python 3 transition (and
associated workarounds like the python3 and python2 symlinks to refer
directly to the desired release series) means that this simple option
isn't viable for historical reasons.

One way that this simple approach could be made to work is to merge the
current major and minor version numbers directly into a 2-digit major
version number:

    33.0.0  # Language release

    33.0.1  # Maintenance release
    33.1.0  # Standard library release

    33.0.2  # Maintenance release
    33.2.0  # Standard library release

    33.0.3  # Maintenance release
    33.3.0  # Standard library release
    34.0.0  # Language release

Why not use a four part version number?

Another simple versioning scheme would just add a "standard library"
version into the existing versioning scheme:

    <IP_ADDRESS>  # Language release

    <IP_ADDRESS>  # Maintenance release
    <IP_ADDRESS>  # Standard library release

    <IP_ADDRESS>  # Maintenance release
    <IP_ADDRESS>  # Standard library release

    <IP_ADDRESS>  # Maintenance release
    <IP_ADDRESS>  # Standard library release
    <IP_ADDRESS>  # Language release

However, this scheme isn't viable due to backwards compatibility
constraints on the sys.version_info structure.

Why not use a date-based versioning scheme?

Earlier versions of this PEP proposed a date-based versioning scheme for
the standard library. However, such a scheme made it very difficult to
handle out-of-cycle releases to fix security issues and other critical
bugs in standard library releases, as it required the following steps:

1.  Change the release version number to the date of the current month.
2.  Update the What's New, NEWS and documentation to refer to the new
    release number.
3.  Make the new release.

With the sequential scheme now proposed, such releases should at most
require a little tidying up of the What's New document before making the
release.

Why isn't PEP 384 enough?

PEP 384 introduced the notion of a "Stable ABI" for CPython, a limited
subset of the full C ABI that is guaranteed to remain stable. Extensions
built against the stable ABI should be able to support all subsequent
Python versions with the same binary.

This will help new projects to avoid coupling their C extension modules
too closely to a specific version of CPython. For existing modules,
however, migrating to the stable ABI can involve quite a lot of work
(especially for extension modules that define a lot of classes). With
limited development resources available, any time spent on such a change
is time that could otherwise have been spent working on features that
offer more direct benefits to end users.

There are also other benefits to separate versioning (as described
above) that are not directly related to the question of binary
compatibility with third party C extensions.

Why no binary compatible additions to the C ABI in standard library releases?

There's a case to be made that additions to the CPython C ABI could
reasonably be permitted in standard library releases. This would give C
extension authors the same freedom as any other package or module author
to depend either on a particular language version or on a standard
library version.

The PEP currently associates the interpreter version with the language
version, and therefore limits major interpreter changes (including C ABI
additions) to the language releases.

An alternative, internally consistent, approach would be to link the
interpreter version with the standard library version, with only changes
that may affect backwards compatibility limited to language releases.

Under such a scheme, the following changes would be acceptable in
standard library releases:

-   Standard library updates
    -   new features in pure Python modules
    -   new features in C extension modules (subject to PEP 399
        compatibility requirements)
    -   new features in language builtins
-   Interpreter implementation updates
    -   binary compatible additions to the C ABI
    -   changes to the compilation toolchain that do not affect the AST
        or alter the bytecode magic number
    -   changes to the core interpreter eval loop
-   bug fixes from the corresponding maintenance release

And the following changes would be acceptable in language releases:

-   new language syntax
-   any updates acceptable in a standard library release
-   new deprecation warnings
-   removal of previously deprecated features
-   changes to the AST
-   changes to the emitted bytecode that require altering the magic
    number
-   binary incompatible changes to the C ABI (although the PEP 384
    stable ABI must still be preserved)

While such an approach could probably be made to work, there does not
appear to be a compelling justification for it, and the approach
currently described in the PEP is simpler and easier to explain.

Why not separate out the standard library entirely?

A concept that is occasionally discussed is the idea of making the
standard library truly independent from the CPython reference
implementation.

My personal opinion is that actually making such a change would involve
a lot of work for next to no pay-off. CPython without the standard
library is useless (the build chain won't even run, let alone the test
suite). You also can't create a standalone pure Python standard library
either, because too many "standard library modules" are actually tightly
linked in to the internal details of their respective interpreters (for
example, the builtins, weakref, gc, sys, inspect, ast).

Creating a separate CPython development branch that is kept compatible
with the previous language release, and making releases from that branch
that are identified with a separate standard library version number
should provide most of the benefits of a separate standard library
repository with only a fraction of the pain.

Acknowledgements

Thanks go to the PEP 407 authors for starting this discussion, as well
as to those authors and Larry Hastings for initial discussions of the
proposal made in this PEP.

References

Copyright

This document has been placed in the public domain.

[1] Twisted's "topfiles" approach to NEWS generation
https://web.archive.org/web/20120305142914/http://twistedmatrix.com/trac/wiki/ReviewProcess#Newsfiles


text:
PEP: 322 Title: Reverse Iteration Version: $Revision$ Last-Modified:
$Date$ Author: Raymond Hettinger<EMAIL_ADDRESS>Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 24-Sep-2003
Python-Version: 2.4 Post-History: 24-Sep-2003

Abstract

This proposal is to add a builtin function to support reverse iteration
over sequences.

Motivation

For indexable objects, current approaches for reverse iteration are
error prone, unnatural, and not especially readable:

    for i in xrange(n-1, -1, -1):
        print seqn[i]

One other current approach involves reversing a list before iterating
over it. That technique wastes computer cycles, memory, and lines of
code:

    rseqn = list(seqn)
    rseqn.reverse()
    for value in rseqn:
        print value

Extended slicing is a third approach that minimizes the code overhead
but does nothing for memory efficiency, beauty, or clarity.

Reverse iteration is much less common than forward iteration, but it
does arise regularly in practice. See Real World Use Cases below.

Proposal

Add a builtin function called reversed() that makes a reverse iterator
over sequence objects that support __getitem__() and __len__().

The above examples then simplify to:

    for i in reversed(xrange(n)):
        print seqn[i]

    for elem in reversed(seqn):
        print elem

The core idea is that the clearest, least error-prone way of specifying
reverse iteration is to specify it in a forward direction and then say
reversed.

The implementation could be as simple as:

    def reversed(x):
        if hasattr(x, 'keys'):
            raise ValueError("mappings do not support reverse iteration")
        i = len(x)
        while i > 0:
            i -= 1
            yield x[i]

No language syntax changes are needed. The proposal is fully backwards
compatible.

A C implementation and unit tests are at:
https://bugs.python.org/issue834422

BDFL Pronouncement

This PEP has been conditionally accepted for Py2.4. The condition means
that if the function is found to be useless, it can be removed before
Py2.4b1.

Alternative Method Names

-   reviter -- Jeremy Fincher's suggestion matches use of iter()
-   ireverse -- uses the itertools naming convention
-   inreverse -- no one seems to like this one except me

The name reverse is not a candidate because it duplicates the name of
the list.reverse() which mutates the underlying list.

Discussion

The case against adoption of the PEP is a desire to keep the number of
builtin functions small. This needs to weighed against the simplicity
and convenience of having it as builtin instead of being tucked away in
some other namespace.

Real World Use Cases

Here are some instances of reverse iteration taken from the standard
library and comments on why reverse iteration was necessary:

-   atexit.exit_handlers() uses:

        while _exithandlers:
            func, targs, kargs = _exithandlers.pop()
                . . .

    In this application popping is required, so the new function would
    not help.

-   heapq.heapify() uses for i in xrange(n//2 - 1, -1, -1) because
    higher-level orderings are more easily formed from pairs of
    lower-level orderings. A forward version of this algorithm is
    possible; however, that would complicate the rest of the heap code
    which iterates over the underlying list in the opposite direction.
    The replacement code for i in reversed(xrange(n//2)) makes clear the
    range covered and how many iterations it takes.

-   mhlib.test() uses:

        testfolders.reverse();
        for t in testfolders:
            do('mh.deletefolder(%s)' % `t`)

    The need for reverse iteration arises because the tail of the
    underlying list is altered during iteration.

-   platform._dist_try_harder() uses
    for n in range(len(verfiles)-1,-1,-1) because the loop deletes
    selected elements from verfiles but needs to leave the rest of the
    list intact for further iteration.

-   random.shuffle() uses for i in xrange(len(x)-1, 0, -1) because the
    algorithm is most easily understood as randomly selecting elements
    from an ever diminishing pool. In fact, the algorithm can be run in
    a forward direction but is less intuitive and rarely presented that
    way in literature. The replacement code
    for i in reversed(xrange(1, len(x))) is much easier to verify
    visually.

-   rfc822.Message.__delitem__() uses:

        list.reverse()
        for i in list:
            del self.headers[i]

    The need for reverse iteration arises because the tail of the
    underlying list is altered during iteration.

Rejected Alternatives

Several variants were submitted that attempted to apply reversed() to
all iterables by running the iterable to completion, saving the results,
and then returning a reverse iterator over the results. While satisfying
some notions of full generality, running the input to the end is
contrary to the purpose of using iterators in the first place. Also, a
small disaster ensues if the underlying iterator is infinite.

Putting the function in another module or attaching it to a type object
is not being considered. Like its cousins, zip() and enumerate(), the
function needs to be directly accessible in daily programming. Each
solves a basic looping problem: lock-step iteration, loop counting, and
reverse iteration. Requiring some form of dotted access would interfere
with their simplicity, daily utility, and accessibility. They are core
looping constructs, independent of any one application domain.

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 759 Title: External Wheel Hosting Author: Barry Warsaw
<EMAIL_ADDRESS>Ethan Smith<EMAIL_ADDRESS>PEP-Delegate: Donald
Stufft<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-759-external-wheel-hosting/66458
Status: Draft Type: Standards Track Topic: Packaging Created:
01-Oct-2024 Post-History:

Abstract

This PEP proposes a mechanism by which projects hosted on pypi.org can
safely host wheel artifacts on external sites other than PyPI. This PEP
explicitly does not propose external hosting of projects, packages, or
their metadata. That functionality is already available by externally
hosting independent package indexes. Because this PEP only provides a
mechanism for projects to customize the download URL for specific
released wheel artifacts, dependency resolution as already implemented
by common installer tools such as pip and uv does not need to change.

This PEP defines what it means to be "safe" in this context, along with
a new package upload file format called a .rim file. It defines how .rim
files affect the metadata returned for a package's Simple Repository API
<packaging:simple-repository-api> in both HTML and JSON formats, and how
traditional wheels can easily be turned into .rim files.

Rationale

The Python Package Index, hosted at https://pypi.org, imposes default
limits on upload artifact file size (100 MiB) and total project size (10
GiB). Most projects can comfortably fit within these limits during the
lifetime of the project, through years of uploads. A few projects have
encountered these limits, and have been granted both file size and
project size exceptions, allowing them to continue uploading new
releases without having to take more drastic measures, such as removing
files which may potentially still be in use by consumers (e.g. through
version pins).

A related workaround is the "wheel stub" approach, which provides an
indirect link between PyPI and an external third party package index,
where such limitations can be avoided. Wheel stubs are
source distributions
<packaging:source-distribution-format> (a.k.a. "sdists") which utilize a
PEP 517 build backend that, instead of turning source code into a binary
wheel, performs some logic to calculate the URL for an existing,
externally hosted wheel to download and install. This approach works,
but it obscures the connection between PyPI, the sdist, and the
externally hosted wheel, since there is no way to present this
information to pip or other such tools.

Historical context

In 2013, PEP 438 proposed a "backward-compatible two-phase transition
process" to modify several aspects of release file hosting on PyPI. As
this PEP describes, PyPI originally supported only project and release
registration without also allowing for artifact file hosting. As such,
most projects hosted release file artifacts elsewhere. Artifact hosting
was later added, but the mix of externally and PyPI-hosted files led to
a wide range of usability and potential security-related problems. PEP
438 was an attempt to provide several facilities to allow external
hosting while promoting a PyPI-first hosting preference.

PEP 438 was complex, with three different "hosting modes", rel metadata
in the simple HTML index pages to signify hosting locations, and a
two-phase transition plan affecting PyPI and installer tools. PEP 438
was ultimately retracted in 2015 by PEP 470, which acknowledges that PEP
438 did succeed in...

  bringing about more people to utilize PyPI's repository features, an
  altogether good thing given the global CDN powering PyPI providing
  speed ups for a lot of people[...]

Instead of external hosting, PEP 470 promoted the use of explicit
multiple repositories, providing full package indexing and artifact
hosting, and enabled through installer tool support, such as
pip install --extra-index-url allowing pip to essentially treat multiple
repositories as one single global repository for package installation
resolution. Because this has been the blessed norm for so many years,
all Python package installation tools support querying multiple indexes
for dependency resolution.

The problem with multiple indexes

Why then does this PEP propose to allow a more limited form of external
hosting, and how does this proposal avoid the problems documented in PEP
470?

One well-known problem that consolidating multiple indexes enables is
dependency confusion attacks, to which Python can be particularly
vulnerable, due to the algorithm that pip install uses for resolving
package dependencies and preferred versions. The uv tool addresses this
by supporting an additional index strategy option, whereby users can
select between, e.g. a pip-compatible strategy, and a more limited
strategy that prevents such dependency confusion attacks.

PEP 708 provides additional background about dependency confusion
attacks, and takes a different approach to preventing them. At its core,
PEP 708 allows repository owners to indicate that projects track across
different repositories, which allows installers to determine how to
treat the global package namespace when combined across multiple
repositories. PEP 708 has been provisionally accepted, pending several
required conditions as outlined in PEP 708, some of which may have an
indeterminate future. As PEP 708 itself says, this won't by itself solve
dependency confusion attacks, but is one way to provide enough
information to installers to help minimize these attacks.

While there can still be valid use cases for standing up a totally
independent package index (such as providing richer platform support for
GPUs until a fully formed variant proposal is accepted), this PEP takes
a different, simpler approach and doesn't replace any of the existing,
proposed, or approved package index cooperation specifications.

This PEP also preserves the core purpose of PyPI, and allows it to
remain the traditional, canonical, centralized index of all Python
packages.

Addressing PyPI limits

This proposal also addresses the problem of size limits imposed by PyPI,
where there is a default artifact size limit of 100 MiB and a default
overall project size limit of 10 GiB. Most packages and artifacts can
easily fit in these limits, even for packages containing binary
extension modules for a variety of platforms. A small, but important
class of packages routinely exceed these limits, requiring them to
submit PyPI exception request support tickets. It's not necessarily
difficult to get resolution on such exceptions, but it is a special
process that can take some time to resolve, and the criteria for
granting such exceptions aren't well documented.

Reducing operational complexity

Setting up and maintaining an entire package index can be a complex
operational solution, both time and resource intensive. This is
especially true if the main purpose of such an index is just to avoid
file size limitations. The external index approach also imposes a tricky
UX on consumers of projects on the external index, requiring them to
understand how CLI options such as --external-index-url work, along with
the security implications of such flags. It would be much easier for
both producers and consumers of large wheel packages to just set up and
maintain a simple web server, capable of serving individual files with
no more complex API than HTTP GET. Such an interface is also easily
cacheable or placed behind a CDN. Simple HTTP servers are also much
easier to audit for security purposes, easier to proxy, and usually take
much less resources to run, support, and maintain. Even something like
Amazon S3 could be used to host external wheels.

This PEP proposes an approach that favors such operational simplicity.

Specification

A new type of uploadable file is defined, called a "RIM" (i.e. .rim), or
"Remote Installable Metadata" file. The name evokes the image of a wheel
with the tire removed, and emphasizes that .rim files are easily derived
from .whl files. The process of turning a .whl into a .rim is
outlined below <dismounting>. The file name format exactly matches the
wheel file naming format
<packaging:wheel-file-name-spec> specification, except that RIM files
use the suffix .rim. This means that all the tags used to discriminate
.whl files also distinguish between different .rim files, and thus can
be used during dependency resolution steps, exactly as .whl files are
today. In this respect, .whl and .rim files are interchangeable.

The content of a .rim file is nearly identical to .whl files, however
.rim files MUST contain only the .dist-info directory from a wheel. No
other top-level file or directory is allowed in the .rim zip file. The
.dist-info directory MUST contain a single additional file in addition
to those allowed in a .whl file's .dist-info directory: a file called
EXTERNAL-HOSTING.json.

This is a JSON file contains containing the following keys:

version

    This is the file format version, which for this PEP MUST be 1.0.

owner

    This MUST name the PyPI organization owner of this externally hosted
    file, for reasons which will be described in
    detail below <resiliency>.

uri

    This is a single URL naming the location of the physical .whl file
    hosted on an external site. This URL MUST use the https scheme.

size

    This is an integer value describing the size in bytes of the
    physical .whl file on the remote host.

hashes

    This is a dictionary of the format described in PEP 694, used to
    capture both the 694#upload-each-file of the physical .whl file,
    with the same constraints as proposed in that PEP. Since these
    hashes are immutable once uploaded to PyPI, they serve as a critical
    validation that the externally hosted wheel hasn't been corrupted or
    compromised.

Effects of the RIM file

The only effect of a .rim file is to change the download URL for the
wheel artifact in both the HTML and JSON interfaces in the simple
repository API. In the HTML page for a package release, the href
attribute MUST be the value of the uri key, including a
#<hashname>=<hashvalue> fragment. this hash fragment MUST be in exactly
the same format as described the PEP 376 originated signed wheel file
format in the .dist-info/RECORD file. The exact same rules for selection
of hash algorithm and encoding is used here.

Similarly in the JSON response the url key pointing to the download file
must be the value of the uri <file-format> key, and the hashes
dictionary MUST be included with values populated from the hashes
dictionary provided above.

In all other respects, a compliant package index should treat .rim files
the same as .whl files, with some other minor exceptions as outlined
below. For example, .rim files can be deleted and yanked (PEP 592) just
like any .whl file, with the exact same semantics (i.e. deletions are
permanent). When a .rim is deleted, an index MUST NOT allow a matching
.whl or .rim file to be (re-)uploaded.

Availability order

Externally hosted wheels MUST be available before the corresponding .rim
file is uploaded to PyPI, otherwise a publishing race condition is
introduced, although this requirement MAY be relaxed for .rim files
uploaded to a PEP 694 staged release.

Wheels can override RIMs

Indexes MUST reject .rim files if a matching .whl file already exists
with the exact same file name tags. However, indexes MAY accept a .whl
file if a matching .rim file exists, as long as that .rim file hasn't
been deleted or yanked. This allows uploaders to replace an externally
hosted wheel file with an index hosted wheel file, but the converse is
prohibited. Since the default is to host wheels on the same package
index that contains the package metadata, it is not allowed to
"downgrade" an existing wheel file once uploaded. When a .whl replaces a
.rim, the index MUST provide download URLs for the package using its own
hosted file service. When uploading the overriding .whl file, the
package index MUST validate the hash from the existing .rim file, and
these hashes must match or the overriding upload MUST be rejected.

PyPI API bump unnecessary

It's likely that the changes are backward compatible enough that a bump
in the PyPI repository version is not necessary. Since .rim files are
essentially changes only to the upload API, package resolvers and
package installers can continue to function with the APIs they've always
supported.

External hosting resiliency

One of the key concerns leading to PEP 438's revocation in PEP 470 was
potential user confusion when an external index disappeared. From PEP
470:

  This confusion comes down to end users of projects not realizing if a
  project is hosted on PyPI or if it relies on an external service. This
  often manifests itself when the external service is down but PyPI is
  not. People will see that PyPI works, and other projects works, but
  this one specific one does not. They oftentimes do not realize who
  they need to contact in order to get this fixed or what their
  remediation steps are.

While the problem of external wheel hosting service going down is not
directly solved by this PEP, several safeguards are in place to greatly
reduce the potential burden on PyPI administrators.

This PEP thus proposes that:

-   External wheel hosting is only allowed for packages which are owned
    by organization accounts. External hosting is an organization-wide
    setting.
-   Organization accounts do not automatically gain the ability to
    externally host wheels; this feature MUST be explicitly enabled by
    PyPI admins at their discretion. Since this will not be a common
    request, we don't expect the overhead to be nearly as burdensome as
    PEP 541 resolutions, account recovery requests, or even file/project
    size increase requests. External hosting requests would be handled
    in the same manner as those requests, i.e. via the PyPI GitHub
    support tracker.
-   Organization accounts requesting external wheel hosting MUST
    register their own support contact URI, be it a mailto URI for a
    contact email address, or the URL to the organization's support
    tracker. Such a contact URI is optional for organizations which do
    not avail themselves of external wheel file hosting.

Combined with the EXTERNAL-HOSTING.json file's owner key, this allows
for installer tools to unambiguously redirect any download errors away
from the PyPI support admins and squarely to the organization's support
admins.

While the exact mechanics of storing and retrieving this organization
support URL will be defined separately, for the sake of example, let's
say a package foo externally hosts wheel files on
`https://foo.example.com <https://foo.example.com>`__ and that host
becomes unreachable. When an installer tool tries to download and
install the package foo wheel, the download step will fail. The
installer would then be able to query PyPI to provide a useful error
message to the end user:

-   The installer downloads the .rim file and reads the owner key from
    the EXTERNAL-HOSTING.json file inside the .rim zip file.

-   The installer queries PyPI for the support URI for the organization
    owner of the externally hosted wheel.

-   An informative error message would then be displayed, e.g.:

      The externally hosted wheel file foo-....whl could not be
      downloaded. Please contact<EMAIL_ADDRESS>for help. Do
      not report this to the PyPI administrators.

Dismounting wheels

It is generally very easy to produce a .rim file from an existing .whl
file. This could be done efficiently by a PEP 518 build backend with an
additional command line option, or a separate tool which takes a .whl
file as input and creates the associated .rim file. To complete the
analogy, the act of turning a .whl into a .rim is called "dismounting".
The steps such a tool would take are:

-   Accept as input the source .whl file, the organization owner of the
    package, and URL at which the .whl will be hosted, and the support
    URI to report download problems from. These could in fact be
    captured in the pyproject.toml file, but that specification is out
    of scope for this PEP.
-   Unzip the .whl and create the .rim zip archive.
-   Omit from the .rim file any path in the .whl that isn't rooted at
    the .dist-info directory.
-   Calculate the hash of the source .whl file.
-   Add the EXTERNAL-HOSTING.json file containing the JSON keys and
    values as described above, to the .rim archive.

Changes to tools

Theoretically, installer tools shouldn't need any changes, since when
they have identified the wheel to download and install, they simply
consult the download URLs returned by PyPI's Simple API. In practice
though, tools such as pip and uv may have constrained lists of hosts
they will allow downloads from, such as PyPI's own pythonhosted.org
domain.

In this case, such tools will need to relax those constraints, but the
exact policy for this is left to the installer tools themselves. Any
number of approaches could be implemented, such as downloading the .rim
file and verifying the EXTERNAL-HOSTING.json metadata, or simply
trusting the external downloads for any wheel with a matching checksum.
They could also query PyPI for the project's organization owner and
support URI before trusting the download. They could warn the user when
externally hosted wheel files are encountered, and/or require the use of
a command line option to enable additional download hosts. Any of these
verification policies could be chosen in configuration files.

Installer tools should also probably provide better error messages when
externally hosted wheels cannot be downloaded, e.g. because a host is
unreachable. As described above, such tools could query enough metadata
from PyPI to provide clear and distinct error messages pointing users to
the package's external hosting support email or issue tracker.

Constraints for external hosting services

The following constraints lead to reliable and compatible external wheel
hosting services:

-   External wheels MUST be served over HTTPS, with a certificate signed
    by Mozilla's root certificate store. This ensures compatibility with
    pip and uv. At the time of this writing, pip 24.2 on Python 3.10 or
    newer uses the system certificate store in addition to the Mozilla
    store provided by the third party certifi Python package. uv uses
    the Mozilla store provided by the webpki-roots crate, but not the
    system store unless the --native-tls flag is given[1]. The PyPI
    administrators may modify this requirement in the future, but
    compatibility with popular installers will not be compromised.
-   External wheel hosts SHOULD use a content delivery network (CDN),
    just as PyPI does.
-   External wheel hosts MUST commit to a stable URL for all wheels they
    host.
-   Externally hosted wheels MUST NOT be removed from an external wheel
    host unless the corresponding .rim file is deleted from PyPI first,
    and MUST NOT remove external wheels for yanked releases.
-   External wheel hosts MUST support HTTP range requests.
-   External wheel hosts SHOULD support the HTTP/2 protocol.

Security

Several factors as described in this proposal should mitigate security
concerns with externally hosted wheels, such as:

-   Wheel file checksums MUST be included in .rim files, and once
    uploaded cannot be changed. Since the checksum stored on PyPI is
    immutable and required, it is not possible to spoof an external
    wheel file, even if the owning organization lost control of their
    hosting domain.
-   Externally hosted wheels MUST be served over HTTPS.
-   In order to serve externally hosted wheels, organizations MUST be
    approved by the PyPI admins.

When users identify malware or vulnerabilities in PyPI-hosted projects,
they can now report this using the malware reporting facilities on PyPI,
as also described in this blog post. The same process can be used to
report security issues in externally hosted wheels, and the same
remediation process should be used. In addition, since organizations
with external hosting enabled MUST provide a support contact URI, that
URI can be used in some cases to report the security issue to the
hosting organization. Such organization reporting won't make sense for
malware, but could indeed be a very useful way to report security
vulnerabilities in externally hosted wheels.

Rejected ideas

Several ideas were considered and rejected.

-   Requiring digital signatures on externally hosted wheel files,
    either in addition to or other than hashes. We deem this unnecessary
    since the checksum requirement should be enough to validate that the
    metadata on PyPI for a wheel exactly matches the downloaded wheel.
    The added complexity of key management outweighs any additional
    benefit such digital signatures might convey.
-   Hash verification on .rim file uploads. PyPI could verify that the
    hash in the uploaded .rim file matches the externally hosted wheel
    before it accepts the upload, but this requires downloading the
    external wheel and performing the checksum, which also implies that
    the upload of the .rim file cannot be accepted until this external
    .whl file is downloaded and verified. This increases PyPI bandwidth
    and slows down the upload query, although PEP 694 draft uploads
    could potentially mitigate these concerns. Still, the benefit is not
    likely worth the additional complexity.
-   Periodic verification of the download URLs by the index. PyPI could
    try to periodically ensure that the external wheel host or the
    external .whl file itself is still available, e.g. via an
    HTTP HEAD <9110#section-9.3.2> request. This is likely overkill and
    without also providing the file's checksum in the response[2], may
    not provide much additional benefit.
-   This PEP could allow for an organization to provide fallback
    download hosts, such that a secondary is available if the primary
    goes down. We believe that DNS-based replication is a much better,
    well-known technique, and probably much more resilient anyway.
-   .rim file replacement. While it is allowed for .whl files to replace
    existing .rim files, as long as a) the .rim file hasn't been deleted
    or yanked, b) the checksums match, we do not allow replacing .whl
    files with .rim files, nor do we allow a .rim file to overwrite an
    existing .rim file. This latter could be a technique to change the
    hosting URL for an externally hosted .whl; however, we do not think
    this is a good idea. There are other ways to "fix" an external host
    URL as described above, and we do not want to encourage mass
    re-uploads of existing .rim files.

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] The uv --native-tls flag replaces the webpki-roots store.

[2] There being no standard way to return the file's checksum in
response to an HTTP HEAD <9110#section-9.3.2> request.


text:
PEP: 438 Title: Transitioning to release-file hosting on PyPI Version:
$Revision$ Last-Modified: $Date$ Author: Holger Krekel
<EMAIL_ADDRESS>Carl Meyer<EMAIL_ADDRESS>BDFL-Delegate:
Richard Jones<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Superseded Type: Process Topic:
Packaging Content-Type: text/x-rst Created: 15-Mar-2013 Post-History:
19-May-2013 Superseded-By: 470 Resolution:
https://mail.python.org/pipermail/distutils-sig/2013-May/020773.html

Abstract

This PEP proposes a backward-compatible two-phase transition process to
speed up, simplify and robustify installing from the pypi.python.org
(PyPI) package index. To ease the transition and minimize client-side
friction, no changes to distutils or existing installation tools are
required in order to benefit from the first transition phase, which will
result in faster, more reliable installs for most existing packages.

The first transition phase implements easy and explicit means for a
package maintainer to control which release file links are served to
present-day installation tools. The first phase also includes the
implementation of analysis tools for present-day packages, to support
communication with package maintainers and the automated setting of
default modes for controlling release file links. The first phase also
will default newly-registered projects on PyPI to only serve links to
release files which were uploaded to PyPI.

The second transition phase concerns end-user installation tools, which
shall default to only install release files that are hosted on PyPI and
tell the user if external release files exist, offering a choice to
automatically use those external files. External release files shall in
the future be registered together with a checksum hash so that
installation tools can verify the integrity of the eventual download
(PyPI-hosted release files always carry such a checksum).

Alternative PyPI server implementations should implement the new simple
index serving behaviour of transition phase 1 to avoid installation
tools treating their release links as external ones in phase 2.

Rationale

History and motivations for external hosting

When PyPI went online, it offered release registration but had no
facility to host release files itself. When hosting was added, no
automated downloading tool existed yet. When Phillip Eby implemented
automated downloading (through setuptools), he made the choice to allow
people to use download hosts of their choice. The finding of
externally-hosted packages was implemented as follows:

1.  The PyPI simple/ index for a package contains all links found by
    scraping them from that package's long_description metadata for any
    release. Links in the "Download-URL" and "Home-page" metadata fields
    are given rel=download and rel=homepage attributes, respectively.
2.  Any of these links whose target is a file whose name appears to be
    in the form of an installable source or binary distribution, with
    name in the form "packagename-version.ARCHIVEEXT", is considered a
    potential installation candidate by installation tools.
3.  Similarly, any links suffixed with an "#egg=packagename-version"
    fragment are considered an installation candidate.
4.  Additionally, the rel=homepage and rel=download links are crawled by
    installation tools and, if HTML, are themselves scraped for
    release-file links in the above formats.

See the easy_install documentation for a complete description of this
behavior.[1]

Today, most packages indexed on PyPI host their release files on PyPI.
Out of 29,117 total projects on PyPI, only 2,581 (less than 10%) include
any links to installable files that are available only off-PyPI.[2]

There are many reasons[3] why people have chosen external hosting. To
cite just a few:

-   release processes and scripts have been developed already and upload
    to external sites
-   it takes too long to upload large files from some places in the
    world
-   export restrictions e.g. for crypto-related software
-   company policies which require offering open source packages through
    own sites
-   problems with integrating uploading to PyPI into one's release
    process (because of release policies)
-   desiring download statistics different from those maintained by PyPI
-   perceived bad reliability of PyPI
-   not aware that PyPI offers file-hosting

Irrespective of the present-day validity of these reasons, there clearly
is a history why people choose to host files externally and it even was
for some time the only way you could do things. This PEP takes the
position that there remain some valid reasons for external hosting even
today.

Problem

Today, python package installers (pip, easy_install, buildout, and
others) often need to query many non-PyPI URLs even if there are no
externally hosted files. Apart from querying pypi.python.org's simple
index pages, also all homepages and download pages ever specified with
any release of a package are crawled by an installer. The need for
installers to crawl external sites slows down installation and makes for
a brittle and unreliable installation process. Those sites and packages
also don't take part in the PEP 381 mirroring infrastructure, further
decreasing reliability and speed of automated installation processes
around the world.

Most packages are hosted directly on pypi.python.org[4]. Even for these
packages, installers still crawl their homepage and download-url, if
specified. Many package uploaders are not aware that specifying the
"homepage" or "download-url" in their package metadata will needlessly
slow down the installation process for all users.

Relying on third party sites also opens up more attack vectors for
injecting malicious packages into sites using automated installs. A
simple attack might just involve getting hold of an old now-unused
homepage domain and placing malicious packages there. Moreover,
performing a Man-in-The-Middle (MITM) attack between an installation
site and any of the download sites can inject malicious packages on the
installation site. As many homepages and download locations are using
HTTP and not HTTPS, such attacks are not hard to launch. Such MITM
attacks can easily happen even for packages which never intended to host
files externally as their homepages are contacted by installers anyway.

There is currently no way for package maintainers to avoid external-link
crawling, other than removing all homepage/download url metadata for all
historic releases. While a script[5] has been written to perform this
action, it is not a good general solution because it removes useful
metadata from PyPI releases.

Even if the sites referenced by "Homepage" and "Download-URL" links were
not scraped for further links, there is no obvious way under the current
system for a package owner to link to an installable file from a
long_description metadata field (which is shown as package documentation
on /pypi/PKG) without installation tools automatically considering that
file a candidate for installation. Conversely, there is no way to
explicitly register multiple external release files without putting them
in metadata fields.

Goals

These are the goals to be achieved by implementation of this PEP:

-   Package owners should be able to explicitly control which files are
    presented by PyPI to installer tools as installation candidates.
    Installation should not be slowed and made less reliable by
    extensive and unnecessary crawling of links that package owners did
    not explicitly nominate as installation files.
-   It should remain possible for package owners to choose to host their
    release files on their own hosting, external to PyPI. It should be
    easy for a user to request the installation of such releases using
    automated installer tools, especially if the external release files
    were registered together with a checksum hash.
-   Automated installer tools should not install externally-hosted
    packages by default, but require explicit authorization to do so by
    the user. When tools refuse to install such a package by default,
    they should tell the user exactly which external link(s) the
    installer needs to follow, and what option(s) the user can provide
    to authorize the tool to follow those links. PyPI should provide all
    necessary metadata for installer tools to implement this easily and
    within a single request/reply interaction.
-   Migration from the status quo to the above points should be gradual
    and minimize breakage. This includes tooling that makes it easy for
    package owners with an existing release process that uploads to
    non-PyPI hosting to also upload those release files to PyPI.

Solution / two transition phases

The first transition phase introduces a "hosting-mode" field for each
project on PyPI, allowing package owners explicit control of which
release file links are served to present-day installation tools in the
machine-readable simple/ index. The first transition will, after
successful hosting-mode manipulations by individual early-adopters, set
a default hosting mode for existing packages, based on automated
analysis. Maintainers will be notified one month ahead of any such
automated change. At completion of the first transition phase, all
present-day existing release and installation processes and tools are
expected to continue working. Any remaining errors or problems are
expected to only relate to installation of individual packages and can
be easily corrected by package maintainers or PyPI admins if maintainers
are not reachable.

Also in the first phase, each link served in the simple/ index will be
explicitly marked as rel="internal" if it is hosted by the index itself
(even if on a separate domain, which may be the case if the index uses a
CDN for file-serving). Any link not so marked will be considered an
external link.

In the second transition phase, PyPI client installation tools shall be
updated to default to only install rel="internal" packages unless a user
specifies option(s) to permit installing from external links. See second
transition phase for details on how installers should behave.

Maintainers of packages which currently host release files on non-PyPI
sites shall receive instructions and tools to ease "re-hosting" of their
historic and future package release files. This re-hosting tool MUST be
available before automated hosting-mode changes are announced to package
maintainers.

Implementation

Hosting modes

The foundation of the first transition phase is the introduction of
three "modes" of PyPI hosting for a package, affecting which links are
generated for the simple/ index. These modes are implemented without
requiring changes to installation tools via changes to the algorithm for
generating the machine-readable simple/ index.

The modes are:

-   pypi-scrape-crawl: no change from the current situation of
    generating machine-readable links for installation tools, as
    outlined in the history.
-   pypi-scrape: for a package in this mode, links to be added to the
    simple/ index are still scraped from package metadata. However, the
    "Home-page" and "Download-url" links are given rel=ext-homepage and
    rel=ext-download attributes instead of rel=homepage and
    rel=download. The effect of this (with no change in installation
    tools necessary) is that these links will not be followed and
    scraped for further candidate links by present-day installation
    tools: only installable files directly hosted from PyPI or linked
    directly from PyPI metadata will be considered for installation.
    Installation tools MAY evolve to offer an option to use the new
    rel-attribution to crawl external pages but MUST NOT default to it.
-   pypi-explicit: for a package in this mode, only links to release
    files uploaded to PyPI, and external links to release files
    explicitly nominated by the package owner, will be added to the
    simple/ index. PyPI will provide a new interface for package owners
    to supply external release-file URLs. These URLs MUST include a URL
    fragment in the form "#hashtype=hashvalue" specifying a hash of the
    externally-linked file which installer tools MUST use to validate
    that they have downloaded the intended file.

Thus the hope is that eventually all projects on PyPI can be migrated to
the pypi-explicit mode, while preserving the ability to install release
files hosted externally via installer tools. Deprecation of hosting
modes to eventually only allow the pypi-explicit mode is NOT REGULATED
by this PEP but is expected to become feasible some time after
successful implementation of the transition phases described in this
PEP. It is expected that deprecation requires a new process to deal with
abandoned packages because of unreachable maintainers for still popular
packages.

First transition phase (PyPI)

The proposed solution consists of multiple implementation and
communication steps:

1.  Implement in PyPI the three modes described above, with an interface
    for package owners to select the mode for each package and register
    explicit external file URLs.
2.  For packages in all modes, label links in the simple/ index to
    index-hosted files with rel="internal", to make it easier for client
    tools to distinguish these links in the second phase.
3.  Add an HTML tag <meta name="api-version" value="2"> to all simple/
    index pages, to allow clients to distinguish between indexes
    providing the rel="internal" metadata and older ones that do not.
4.  Default all newly-registered packages to pypi-explicit mode (package
    owners can still switch to the other modes as desired).
5.  Determine (via automated analysis[6]) which packages have all
    installable files available on PyPI itself (group A), which have all
    installable files on PyPI or linked directly from PyPI metadata
    (group B), and which have installable versions available that are
    linked only from external homepage/download HTML pages (group C).
6.  Send mail to maintainers of projects in group A that their project
    will be automatically configured to pypi-explicit mode in one month,
    and similarly to maintainers of projects in group B that their
    project will be automatically configured to pypi-scrape mode. Inform
    them that this change is not expected to affect installability of
    their project at all, but will result in faster and safer installs
    for their users. Encourage them to set this mode themselves sooner
    to benefit their users.
7.  Send mail to maintainers of packages in group C that their package
    hosting mode is pypi-scrape-crawl, list the URLs which currently are
    crawled, and suggest that they either re-host their packages
    directly on PyPI and switch to pypi-explicit, or at least provide
    direct links to release files in PyPI metadata and switch to
    pypi-scrape. Provide instructions and tools to help with these
    transitions.

Second transition phase (installer tools)

For the second transition phase, maintainers of installation tools are
asked to release two updates.

The first update shall provide clear warnings if externally-hosted
release files (that is, files whose link does not include
rel="internal") are selected for download, for which projects and URLs
exactly this happens, and warn that in future versions externally-hosted
downloads will be disabled by default.

The second update should change the default mode to allow only
installation of rel="internal" package files, and allow installation of
externally-hosted packages only when the user supplies an option.

The installer should distinguish between verifiable and non-verifiable
external links. A verifiable external link is a direct link to an
installable file from the PyPI simple/ index that includes a hash in the
URL fragment ("#hashtype=hashvalue") which can be used to verify the
integrity of the downloaded file. A non-verifiable external link is any
link (other than those explicitly supplied by the user of an installer
tool) without a hash, scraped from external HTML, or injected into the
search via some other non-PyPI source (e.g. setuptools' dependency_links
feature).

Installers should provide a blanket option to allow installing any
verifiable external link. Non-verifiable external links should only be
installed if the user-provided option specifies exactly which external
domains can be used or for which specific package names external links
can be used.

When download of an externally-hosted package is disallowed by the
default configuration, the user should be notified, with instructions
for how to make the install succeed and warnings about the implication
(that a file will be downloaded from a site that is not part of the
package index). The warning given for non-verifiable links should
clearly state that the installer cannot verify the integrity of the
downloaded file. The warning given for verifiable external links should
simply note that the file will be downloaded from an external URL, but
that the file integrity can be verified by checksum.

Alternative PyPI-compatible index implementations should upgrade to
begin providing the rel="internal" metadata and the
<meta name="api-version" value="2"> tag as soon as possible. For
alternative indexes which do not yet provide the meta tag in their
simple/ pages, installation tools should provide backwards-compatible
fallback behavior (treat links as internal as in pre-PEP times and
provide a warning).

API For Submitting External Distribution URLs

New distribution URLs may be submitted by performing a HTTP POST to the
URL:

  https://pypi.python.org/pypi

With the following form-encoded data:

  Name             Value
  ---------------- ---------------------------------
  :action          The string "urls"
  name             The package name as a string
  version          The release version as a string
  new-url          The new URL to store
  submit_new_url   The string "yes"

The POST must be accompanied by an HTTP Basic Auth header encoding the
username and password of the user authorized to maintain the package on
PyPI.

The HTTP response to this request will be one of:

  Code   Meaning        URL submission implications
  ------ -------------- -------------------------------------------------------------------------------------------
  200    OK             Everything worked just fine
  400    Bad request    Data provided for submission was malformed
  401    Unauthorised   The username or password supplied were incorrect
  403    Forbidden      User does not have permission to update the package information (not Owner or Maintainer)

References

Acknowledgments

Phillip Eby for precise information and the basic ideas to implement the
transition via server-side changes only.

Donald Stufft for pushing away from external hosting and offering to
implement both a Pull Request for the necessary PyPI changes and the
analysis tool to drive the transition phase 1.

Marc-Andre Lemburg, Alyssa Coghlan and catalog-sig in general for
thinking through issues regarding getting rid of "external hosting".

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Phillip Eby, easy_install 'Package Index "API"' documentation,
http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api

[2] Donald Stufft, automated analysis of PyPI project links,
https://github.com/dstufft/pypi.linkcheck

[3] Marc-Andre Lemburg, reasons for external hosting,
https://mail.python.org/pipermail/catalog-sig/2013-March/005626.html

[4] Donald Stufft, automated analysis of PyPI project links,
https://github.com/dstufft/pypi.linkcheck

[5] Holger Krekel, script to remove homepage/download metadata for all
releases
https://mail.python.org/pipermail/catalog-sig/2013-February/005423.html

[6] Donald Stufft, automated analysis of PyPI project links,
https://github.com/dstufft/pypi.linkcheck

authors:
Carl Meyer
Holger Krekel