text:
PEP: 227 Title: Statically Nested Scopes Author: Jeremy Hylton
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 01-Nov-2000 Python-Version: 2.1 Post-History:

Abstract

This PEP describes the addition of statically nested scoping (lexical
scoping) for Python 2.2, and as a source level option for python 2.1. In
addition, Python 2.1 will issue warnings about constructs whose meaning
may change when this feature is enabled.

The old language definition (2.0 and before) defines exactly three
namespaces that are used to resolve names -- the local, global, and
built-in namespaces. The addition of nested scopes allows resolution of
unbound local names in enclosing functions' namespaces.

The most visible consequence of this change is that lambdas (and other
nested functions) can reference variables defined in the surrounding
namespace. Currently, lambdas must often use default arguments to
explicitly creating bindings in the lambda's namespace.

Introduction

This proposal changes the rules for resolving free variables in Python
functions. The new name resolution semantics will take effect with
Python 2.2. These semantics will also be available in Python 2.1 by
adding "from __future__ import nested_scopes" to the top of a module.
(See PEP 236.)

The Python 2.0 definition specifies exactly three namespaces to check
for each name -- the local namespace, the global namespace, and the
builtin namespace. According to this definition, if a function A is
defined within a function B, the names bound in B are not visible in A.
The proposal changes the rules so that names bound in B are visible in A
(unless A contains a name binding that hides the binding in B).

This specification introduces rules for lexical scoping that are common
in Algol-like languages. The combination of lexical scoping and existing
support for first-class functions is reminiscent of Scheme.

The changed scoping rules address two problems -- the limited utility of
lambda expressions (and nested functions in general), and the frequent
confusion of new users familiar with other languages that support nested
lexical scopes, e.g. the inability to define recursive functions except
at the module level.

The lambda expression yields an unnamed function that evaluates a single
expression. It is often used for callback functions. In the example
below (written using the Python 2.0 rules), any name used in the body of
the lambda must be explicitly passed as a default argument to the
lambda.

    from Tkinter import *
    root = Tk()
    Button(root, text="Click here",
           command=lambda root=root: root.test.configure(text="..."))

This approach is cumbersome, particularly when there are several names
used in the body of the lambda. The long list of default arguments
obscures the purpose of the code. The proposed solution, in crude terms,
implements the default argument approach automatically. The "root=root"
argument can be omitted.

The new name resolution semantics will cause some programs to behave
differently than they did under Python 2.0. In some cases, programs will
fail to compile. In other cases, names that were previously resolved
using the global namespace will be resolved using the local namespace of
an enclosing function. In Python 2.1, warnings will be issued for all
statements that will behave differently.

Specification

Python is a statically scoped language with block structure, in the
traditional of Algol. A code block or region, such as a module, class
definition, or function body, is the basic unit of a program.

Names refer to objects. Names are introduced by name binding operations.
Each occurrence of a name in the program text refers to the binding of
that name established in the innermost function block containing the
use.

The name binding operations are argument declaration, assignment, class
and function definition, import statements, for statements, and except
clauses. Each name binding occurs within a block defined by a class or
function definition or at the module level (the top-level code block).

If a name is bound anywhere within a code block, all uses of the name
within the block are treated as references to the current block. (Note:
This can lead to errors when a name is used within a block before it is
bound.)

If the global statement occurs within a block, all uses of the name
specified in the statement refer to the binding of that name in the
top-level namespace. Names are resolved in the top-level namespace by
searching the global namespace, i.e. the namespace of the module
containing the code block, and in the builtin namespace, i.e. the
namespace of the __builtin__ module. The global namespace is searched
first. If the name is not found there, the builtin namespace is
searched. The global statement must precede all uses of the name.

If a name is used within a code block, but it is not bound there and is
not declared global, the use is treated as a reference to the nearest
enclosing function region. (Note: If a region is contained within a
class definition, the name bindings that occur in the class block are
not visible to enclosed functions.)

A class definition is an executable statement that may contain uses and
definitions of names. These references follow the normal rules for name
resolution. The namespace of the class definition becomes the attribute
dictionary of the class.

The following operations are name binding operations. If they occur
within a block, they introduce new local names in the current block
unless there is also a global declaration.

    Function definition: def name ...
    Argument declaration: def f(...name...), lambda ...name...
    Class definition: class name ...
    Assignment statement: name = ...
    Import statement: import name, import module as name,
        from module import name
    Implicit assignment: names are bound by for statements and except
        clauses

There are several cases where Python statements are illegal when used in
conjunction with nested scopes that contain free variables.

If a variable is referenced in an enclosed scope, it is an error to
delete the name. The compiler will raise a SyntaxError for 'del name'.

If the wild card form of import (import *) is used in a function and the
function contains a nested block with free variables, the compiler will
raise a SyntaxError.

If exec is used in a function and the function contains a nested block
with free variables, the compiler will raise a SyntaxError unless the
exec explicitly specifies the local namespace for the exec. (In other
words, "exec obj" would be illegal, but "exec obj in ns" would be
legal.)

If a name bound in a function scope is also the name of a module global
name or a standard builtin name, and the function contains a nested
function scope that references the name, the compiler will issue a
warning. The name resolution rules will result in different bindings
under Python 2.0 than under Python 2.2. The warning indicates that the
program may not run correctly with all versions of Python.

Discussion

The specified rules allow names defined in a function to be referenced
in any nested function defined with that function. The name resolution
rules are typical for statically scoped languages, with three primary
exceptions:

-   Names in class scope are not accessible.
-   The global statement short-circuits the normal rules.
-   Variables are not declared.

Names in class scope are not accessible. Names are resolved in the
innermost enclosing function scope. If a class definition occurs in a
chain of nested scopes, the resolution process skips class definitions.
This rule prevents odd interactions between class attributes and local
variable access. If a name binding operation occurs in a class
definition, it creates an attribute on the resulting class object. To
access this variable in a method, or in a function nested within a
method, an attribute reference must be used, either via self or via the
class name.

An alternative would have been to allow name binding in class scope to
behave exactly like name binding in function scope. This rule would
allow class attributes to be referenced either via attribute reference
or simple name. This option was ruled out because it would have been
inconsistent with all other forms of class and instance attribute
access, which always use attribute references. Code that used simple
names would have been obscure.

The global statement short-circuits the normal rules. Under the
proposal, the global statement has exactly the same effect that it does
for Python 2.0. It is also noteworthy because it allows name binding
operations performed in one block to change bindings in another block
(the module).

Variables are not declared. If a name binding operation occurs anywhere
in a function, then that name is treated as local to the function and
all references refer to the local binding. If a reference occurs before
the name is bound, a NameError is raised. The only kind of declaration
is the global statement, which allows programs to be written using
mutable global variables. As a consequence, it is not possible to rebind
a name defined in an enclosing scope. An assignment operation can only
bind a name in the current scope or in the global scope. The lack of
declarations and the inability to rebind names in enclosing scopes are
unusual for lexically scoped languages; there is typically a mechanism
to create name bindings (e.g. lambda and let in Scheme) and a mechanism
to change the bindings (set! in Scheme).

Examples

A few examples are included to illustrate the way the rules work.

    >>> def make_adder(base):
    ...     def adder(x):
    ...         return base + x
    ...     return adder
    >>> add5 = make_adder(5)
    >>> add5(6)
    11

    >>> def make_fact():
    ...     def fact(n):
    ...         if n == 1:
    ...             return 1L
    ...         else:
    ...             return n * fact(n - 1)
    ...     return fact
    >>> fact = make_fact()
    >>> fact(7)
    5040L

    >>> def make_wrapper(obj):
    ...     class Wrapper:
    ...         def __getattr__(self, attr):
    ...             if attr[0] != '_':
    ...                 return getattr(obj, attr)
    ...             else:
    ...                 raise AttributeError, attr
    ...     return Wrapper()
    >>> class Test:
    ...     public = 2
    ...     _private = 3
    >>> w = make_wrapper(Test())
    >>> w.public
    2
    >>> w._private
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    AttributeError: _private

An example from Tim Peters demonstrates the potential pitfalls of nested
scopes in the absence of declarations:

    i = 6
    def f(x):
        def g():
            print i
        # ...
        # skip to the next page
        # ...
        for i in x:  # ah, i *is* local to f, so this is what g sees
            pass
        g()

The call to g() will refer to the variable i bound in f() by the for
loop. If g() is called before the loop is executed, a NameError will be
raised.

Backwards compatibility

There are two kinds of compatibility problems caused by nested scopes.
In one case, code that behaved one way in earlier versions behaves
differently because of nested scopes. In the other cases, certain
constructs interact badly with nested scopes and will trigger
SyntaxErrors at compile time.

The following example from Skip Montanaro illustrates the first kind of
problem:

    x = 1
    def f1():
        x = 2
        def inner():
            print x
        inner()

Under the Python 2.0 rules, the print statement inside inner() refers to
the global variable x and will print 1 if f1() is called. Under the new
rules, it refers to the f1()'s namespace, the nearest enclosing scope
with a binding.

The problem occurs only when a global variable and a local variable
share the same name and a nested function uses that name to refer to the
global variable. This is poor programming practice, because readers will
easily confuse the two different variables. One example of this problem
was found in the Python standard library during the implementation of
nested scopes.

To address this problem, which is unlikely to occur often, the Python
2.1 compiler (when nested scopes are not enabled) issues a warning.

The other compatibility problem is caused by the use of import * and
'exec' in a function body, when that function contains a nested scope
and the contained scope has free variables. For example:

    y = 1
    def f():
        exec "y = 'gotcha'" # or from module import *
        def g():
            return y
        ...

At compile-time, the compiler cannot tell whether an exec that operates
on the local namespace or an import * will introduce name bindings that
shadow the global y. Thus, it is not possible to tell whether the
reference to y in g() should refer to the global or to a local name in
f().

In discussion of the python-list, people argued for both possible
interpretations. On the one hand, some thought that the reference in g()
should be bound to a local y if one exists. One problem with this
interpretation is that it is impossible for a human reader of the code
to determine the binding of y by local inspection. It seems likely to
introduce subtle bugs. The other interpretation is to treat exec and
import * as dynamic features that do not effect static scoping. Under
this interpretation, the exec and import * would introduce local names,
but those names would never be visible to nested scopes. In the specific
example above, the code would behave exactly as it did in earlier
versions of Python.

Since each interpretation is problematic and the exact meaning
ambiguous, the compiler raises an exception. The Python 2.1 compiler
issues a warning when nested scopes are not enabled.

A brief review of three Python projects (the standard library, Zope, and
a beta version of PyXPCOM) found four backwards compatibility issues in
approximately 200,000 lines of code. There was one example of case #1
(subtle behavior change) and two examples of import * problems in the
standard library.

(The interpretation of the import * and exec restriction that was
implemented in Python 2.1a2 was much more restrictive, based on language
that in the reference manual that had never been enforced. These
restrictions were relaxed following the release.)

Compatibility of C API

The implementation causes several Python C API functions to change,
including PyCode_New(). As a result, C extensions may need to be updated
to work correctly with Python 2.1.

locals() / vars()

These functions return a dictionary containing the current scope's local
variables. Modifications to the dictionary do not affect the values of
variables. Under the current rules, the use of locals() and globals()
allows the program to gain access to all the namespaces in which names
are resolved.

An analogous function will not be provided for nested scopes. Under this
proposal, it will not be possible to gain dictionary-style access to all
visible scopes.

Warnings and Errors

The compiler will issue warnings in Python 2.1 to help identify programs
that may not compile or run correctly under future versions of Python.
Under Python 2.2 or Python 2.1 if the nested_scopes future statement is
used, which are collectively referred to as "future semantics" in this
section, the compiler will issue SyntaxErrors in some cases.

The warnings typically apply when a function that contains a nested
function that has free variables. For example, if function F contains a
function G and G uses the builtin len(), then F is a function that
contains a nested function (G) with a free variable (len). The label
"free-in-nested" will be used to describe these functions.

import * used in function scope

The language reference specifies that import * may only occur in a
module scope. (Sec. 6.11) The implementation of C Python has supported
import * at the function scope.

If import * is used in the body of a free-in-nested function, the
compiler will issue a warning. Under future semantics, the compiler will
raise a SyntaxError.

bare exec in function scope

The exec statement allows two optional expressions following the keyword
"in" that specify the namespaces used for locals and globals. An exec
statement that omits both of these namespaces is a bare exec.

If a bare exec is used in the body of a free-in-nested function, the
compiler will issue a warning. Under future semantics, the compiler will
raise a SyntaxError.

local shadows global

If a free-in-nested function has a binding for a local variable that (1)
is used in a nested function and (2) is the same as a global variable,
the compiler will issue a warning.

Rebinding names in enclosing scopes

There are technical issues that make it difficult to support rebinding
of names in enclosing scopes, but the primary reason that it is not
allowed in the current proposal is that Guido is opposed to it. His
motivation: it is difficult to support, because it would require a new
mechanism that would allow the programmer to specify that an assignment
in a block is supposed to rebind the name in an enclosing block;
presumably a keyword or special syntax (x := 3) would make this
possible. Given that this would encourage the use of local variables to
hold state that is better stored in a class instance, it's not worth
adding new syntax to make this possible (in Guido's opinion).

The proposed rules allow programmers to achieve the effect of rebinding,
albeit awkwardly. The name that will be effectively rebound by enclosed
functions is bound to a container object. In place of assignment, the
program uses modification of the container to achieve the desired
effect:

    def bank_account(initial_balance):
        balance = [initial_balance]
        def deposit(amount):
            balance[0] = balance[0] + amount
            return balance
        def withdraw(amount):
            balance[0] = balance[0] - amount
            return balance
        return deposit, withdraw

Support for rebinding in nested scopes would make this code clearer. A
class that defines deposit() and withdraw() methods and the balance as
an instance variable would be clearer still. Since classes seem to
achieve the same effect in a more straightforward manner, they are
preferred.

Implementation

The implementation for C Python uses flat closures[1]. Each def or
lambda expression that is executed will create a closure if the body of
the function or any contained function has free variables. Using flat
closures, the creation of closures is somewhat expensive but lookup is
cheap.

The implementation adds several new opcodes and two new kinds of names
in code objects. A variable can be either a cell variable or a free
variable for a particular code object. A cell variable is referenced by
containing scopes; as a result, the function where it is defined must
allocate separate storage for it on each invocation. A free variable is
referenced via a function's closure.

The choice of free closures was made based on three factors. First,
nested functions are presumed to be used infrequently, deeply nested
(several levels of nesting) still less frequently. Second, lookup of
names in a nested scope should be fast. Third, the use of nested scopes,
particularly where a function that access an enclosing scope is
returned, should not prevent unreferenced objects from being reclaimed
by the garbage collector.

References

Copyright

[1] Luca Cardelli. Compiling a functional language. In Proc. of the 1984
ACM Conference on Lisp and Functional Programming, pp. 208-217, Aug.
1984 https://dl.acm.org/doi/10.1145/800055.802037


text:
PEP: 101 Title: Doing Python Releases 101 Author: Barry Warsaw
<EMAIL_ADDRESS>Guido van Rossum<EMAIL_ADDRESS>Status: Active
Type: Informational Created: 22-Aug-2001 Post-History: Replaces: 102

Abstract

Making a Python release is a thrilling and crazy process. You've heard
the expression "herding cats"? Imagine trying to also saddle those
purring little creatures up, and ride them into town, with some of their
buddies firmly attached to your bare back, anchored by newly sharpened
claws. At least they're cute, you remind yourself.

Actually, no, that's a slight exaggeration 😉 The Python release process
has steadily improved over the years and now, with the help of our
amazing community, is really not too difficult. This PEP attempts to
collect, in one place, all the steps needed to make a Python release.
Most of the steps are now automated or guided by automation, so manually
following this list is no longer necessary.

Things You'll Need

As a release manager there are a lot of resources you'll need to access.
Here's a hopefully-complete list.

-   A GPG key.

    Python releases are digitally signed with GPG; you'll need a key,
    which hopefully will be on the "web of trust" with at least one of
    the other release managers.

-   A bunch of software:

    -   A checkout of the python/release-tools repo. It contains a
        requirements.txt file that you need to install dependencies from
        first. Afterwards, you can fire up scripts in the repo, covered
        later in this PEP.
    -   blurb, the Misc/NEWS management tool. You can pip install it.

-   Access to servers where you will upload files:

    -   downloads.nyc1.psf.io, the server that hosts download files; and
    -   docs.nyc1.psf.io, the server that hosts the documentation.

-   Administrator access to python/cpython.

-   An administrator account on www.python.org, including an "API key".

-   Write access to the python/peps repository.

    If you're reading this, you probably already have this--the first
    task of any release manager is to draft the release schedule. But in
    case you just signed up... sucker! I mean, uh, congratulations!

-   Posting access to blog.python.org, a Blogger-hosted weblog. The RSS
    feed from this blog is used for the 'Python News' section on
    www.python.org.

-   A subscription to the super secret release manager mailing list,
    which may or may not be called python-cabal. Bug Barry about this.

-   A @python.org email address that you will use to sign your releases
    with. Ask postmaster@ for an address; you can either get a full
    account, or a redirecting alias + SMTP credentials to send email
    from this address that looks legit to major email providers.

-   Be added to the Python Security Response Team.

Types of Releases

There are several types of releases you will need to make. These
include:

-   alpha
-   begin beta, also known as beta 1, also known as new branch
-   beta 2+
-   release candidate 1
-   release candidate 2+
-   final
-   new branch
-   begin bugfix mode
-   begin security-only mode
-   end-of-life

Some of these release types actually involve more than one release
branch. In particular, a new branch is that point in the release cycle
when a new feature release cycle begins. Under the current organization
of the CPython Git repository, the main branch is always the target for
new features. At some point in the release cycle of the next feature
release, a new branch release is made which creates a new separate
branch for stabilization and later maintenance of the current
in-progress feature release (3.n.0) and the main branch is modified to
build a new version (which will eventually be released as 3.n+1.0).
While the new branch release step could occur at one of several points
in the release cycle, current practice is for it to occur at feature
code cutoff for the release which is scheduled for the first beta
release.

In the descriptions that follow, steps specific to release types are
labeled accordingly, for now, new branch and final.

How To Make A Release

Here are the steps taken to make a Python release. Some steps are more
fuzzy than others because there's little that can be automated (e.g.
writing the NEWS entries). Where a step is usually performed by An
Expert, the role of that expert is given. Otherwise, assume the step is
done by the Release Manager (RM), the designated person performing the
release. The roles and their current experts are:

-   RM = Release Manager
    -   Hugo van Kemenade<EMAIL_ADDRESS>(FI)
    -   Thomas Wouters<EMAIL_ADDRESS>(NL)
    -   Pablo Galindo Salgado<EMAIL_ADDRESS>(UK)
    -   Łukasz Langa<EMAIL_ADDRESS>(PL)
-   WE = Windows - Steve Dower<EMAIL_ADDRESS>-   ME = Mac - Ned Deily<EMAIL_ADDRESS>(US)
-   DE = Docs - Julien Palard<EMAIL_ADDRESS>(Central Europe)

Note

It is highly recommended that the RM contact the Experts the day before
the release. Because the world is round and everyone lives in different
timezones, the RM must ensure that the release tag is created in enough
time for the Experts to cut binary releases.

You should not make the release public (by updating the website and
sending announcements) before all experts have updated their bits. In
rare cases where the expert for Windows or Mac is MIA, you may add a
message "(Platform) binaries will be provided shortly" and proceed.

As much as possible, the release steps are automated and guided by the
release script, which is available in a separate repository:
python/release-tools.

We use the following conventions in the examples below. Where a release
number is given, it is of the form 3.X.YaN, e.g. 3.13.0a3 for Python
3.13.0 alpha 3, where "a" == alpha, "b" == beta, "rc" == release
candidate.

Release tags are named v3.X.YaN. The branch name for minor release
maintenance branches is 3.X.

This helps by performing several automatic editing steps, and guides you
to perform some manual editing steps.

-   Log into Discord and join the Python Core Devs server. Ask Thomas or
    Łukasz for an invite.

    You probably need to coordinate with other people around the world.
    This communication channel is where we've arranged to meet.

-   Check to see if there are any showstopper bugs.

    Go to https://github.com/python/cpython/issues and look for any open
    bugs that can block this release. You're looking at two relevant
    labels:

    release-blocker

        Stops the release dead in its tracks. You may not make any
        release with any open release blocker bugs.

    deferred-blocker

        Doesn't block this release, but it will block a future release.
        You may not make a final or candidate release with any open
        deferred blocker bugs.

    Review the release blockers and either resolve them, bump them down
    to deferred, or stop the release and ask for community assistance.
    If you're making a final or candidate release, do the same with any
    open deferred.

-   Check the stable buildbots.

    Go to https://buildbot.python.org/all/#/release_status

    Look at the buildbots for the release you're making. Ignore any that
    are offline (or inform the community so they can be restarted). If
    what remains are (mostly) green buildbots, you're good to go. If you
    have non-offline red buildbots, you may want to hold up the release
    until they are fixed. Review the problems and use your judgement,
    taking into account whether you are making an alpha, beta, or final
    release.

-   Make a release clone.

    On a fork of the CPython repository on GitHub, create a release
    branch within it (called the "release clone" from now on). You can
    use the same GitHub fork you use for CPython development. Using the
    standard setup recommended in the Python Developer's Guide, your
    fork would be referred to as origin and the standard CPython repo as
    upstream. You will use the branch on your fork to do the release
    engineering work, including tagging the release, and you will use it
    to share with the other experts for making the binaries.

    For a final or release candidate 2+ release, if you are going to
    cherry-pick a subset of changes for the next rc or final from all
    those merged since the last rc, you should create a release
    engineering branch starting from the most recent release candidate
    tag, i.e. v3.8.0rc1. You will then cherry-pick changes from the
    standard release branch as necessary into the release engineering
    branch and then proceed as usual. If you are going to take all of
    the changes since the previous rc, you can proceed as normal.

-   Make sure the current branch of your release clone is the branch you
    want to release from (git status).

-   Run blurb release <version> specifying the version number (e.g.
    blurb release 3.4.7rc1). This merges all the recent news blurbs into
    a single file marked with this release's version number.

-   Regenerate Lib/pydoc-topics.py.

    While still in the Doc directory, run:

        make pydoc-topics
        cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py

-   Commit your changes to pydoc_topics.py (and any fixes you made in
    the docs).

-   Consider running autoconf using the currently accepted standard
    version in case configure or other Autoconf-generated files were
    last committed with a newer or older version and may contain
    spurious or harmful differences. Currently, Autoconf 2.71 is our de
    facto standard. if there are differences, commit them.

-   Make sure the SOURCE_URI in Doc/tools/extensions/pyspecific.py
    points to the right branch in the Git repository (main or 3.X). For
    a new branch release, change the branch in the file from main to the
    new release branch you are about to create (3.X).

-   Bump version numbers via the release script:

        .../release-tools/release.py --bump 3.X.YaN

    Reminder: X, Y, and N should be integers. a should be one of a, b,
    or rc (e.g. 3.4.3rc1). For final releases omit the aN (3.4.3). For
    the first release of a new version Y should be 0 (3.6.0).

    This automates updating various release numbers, but you will have
    to modify a few files manually. If your $EDITOR environment variable
    is set up correctly, release.py will pop up editor windows with the
    files you need to edit.

    Review the blurb-generated Misc/NEWS file and edit as necessary.

-   Make sure all changes have been committed. (release.py --bump
    doesn't check in its changes for you.)

-   Check the years on the copyright notice. If the last release was
    some time last year, add the current year to the copyright notice in
    several places:

    -   README
    -   LICENSE (make sure to change on main and the branch)
    -   Python/getcopyright.c
    -   Doc/copyright.rst
    -   Doc/license.rst
    -   PC/python_ver_rc.h sets up the DLL version resource for Windows
        (displayed when you right-click on the DLL and select
        Properties). This isn't a C include file, it's a Windows
        "resource file" include file.

-   For a final major release, edit the first paragraph of
    Doc/whatsnew/3.X.rst to include the actual release date; e.g.
    "Python 2.5 was released on August 1, 2003." There's no need to edit
    this for alpha or beta releases.

-   Do a git status in this directory.

    You should not see any files, i.e., you better not have any
    uncommitted changes in your working directory.

-   Tag the release for 3.X.YaN:

        .../release-tools/release.py --tag 3.X.YaN

    This executes a git tag command with the -s option so that the
    release tag in the repo is signed with your GPG key. When prompted
    choose the private key you use for signing release tarballs etc.

-   For begin security-only mode and end-of-life releases, review the
    two files and update the versions accordingly in all active
    branches.

-   Time to build the source tarball. Use the release script to create
    the source gzip and xz tarballs, documentation tar and zip files,
    and GPG signature files:

        .../release-tools/release.py --export 3.X.YaN

    This can take a while for final releases, and it will leave all the
    tarballs and signatures in a subdirectory called 3.X.YaN/src, and
    the built docs in 3.X.YaN/docs (for final releases).

    Note that the script will sign your release with Sigstore. Use your
    @python.org email address for this. See here for more information:
    https://www.python.org/download/sigstore/.

-   Now you want to perform the very important step of checking the
    tarball you just created, to make sure a completely clean, virgin
    build passes the regression test. Here are the best steps to take:

        cd /tmp
        tar xvf /path/to/your/release/clone/<version>//Python-3.2rc2.tgz
        cd Python-3.2rc2
        ls
        # (Do things look reasonable?)
        ls Lib
        # (Are there stray .pyc files?)
        ./configure
        # (Loads of configure output)
        make test
        # (Do all the expected tests pass?)

    If you're feeling lucky and have some time to kill, or if you are
    making a release candidate or final release, run the full test
    suite:

        make buildbottest

    If the tests pass, then you can feel good that the tarball is fine.
    If some of the tests fail, or anything else about the freshly
    unpacked directory looks weird, you better stop now and figure out
    what the problem is.

-   Push your commits to the remote release branch in your GitHub fork:

        # Do a dry run first.
        git push --dry-run --tags origin
        # Make sure you are pushing to your GitHub fork,
        # *not* to the main python/cpython repo!
        git push --tags origin

-   Notify the experts that they can start building binaries.

Warning

STOP: at this point you must receive the "green light" from other
experts in order to create the release. There are things you can do
while you wait though, so keep reading until you hit the next STOP.

-   The WE generates and publishes the Windows files using the Azure
    Pipelines build scripts in .azure-pipelines/windows-release/,
    currently set up at
    https://dev.azure.com/Python/cpython/_build?definitionId=21.

    The build process runs in multiple stages, with each stage's output
    being available as a downloadable artifact. The stages are:

    -   Compile all variants of binaries (32-bit, 64-bit,
        debug/release), including running profile-guided optimization.
    -   Compile the HTML Help file containing the Python documentation.
    -   Codesign all the binaries with the PSF's certificate.
    -   Create packages for python.org, nuget.org, the embeddable distro
        and the Windows Store.
    -   Perform basic verification of the installers.
    -   Upload packages to python.org and nuget.org, purge download
        caches and run a test download.

    After the uploads are complete, the WE copies the generated hashes
    from the build logs and emails them to the RM. The Windows Store
    packages are uploaded manually to
    https://partner.microsoft.com/dashboard/home by the WE.

-   The ME builds Mac installer packages and uploads them to
    downloads.nyc1.psf.io together with GPG signature files.

-   scp or rsync all the files built by release.py --export to your home
    directory on downloads.nyc1.psf.io.

    While you're waiting for the files to finish uploading, you can
    continue on with the remaining tasks. You can also ask folks on
    Discord and/or discuss.python.org to download the files as they
    finish uploading so that they can test them on their platforms as
    well.

-   Now you need to go to downloads.nyc1.psf.io and move all the files
    in place over there. Our policy is that every Python version gets
    its own directory, but each directory contains all releases of that
    version.

    -   On downloads.nyc1.psf.io,
        cd /srv/www.python.org/ftp/python/3.X.Y creating it if
        necessary. Make sure it is owned by group downloads and
        group-writable.

    -   Move the release .tgz, and .tar.xz files into place, as well as
        the .asc GPG signature files. The Win/Mac binaries are usually
        put there by the experts themselves.

        Make sure they are world readable. They should also be group
        writable, and group-owned by downloads.

    -   Use gpg --verify to make sure they got uploaded intact.

    -   If this is a final or rc release: Move the doc zips and tarballs
        to /srv/www.python.org/ftp/python/doc/3.X.Y[rcA], creating the
        directory if necessary, and adapt the "current" symlink in
        .../doc to point to that directory. Note though that if you're
        releasing a maintenance release for an older version, don't
        change the current link.

    -   If this is a final or rc release (even a maintenance release),
        also unpack the HTML docs to
        /srv/docs.python.org/release/3.X.Y[rcA] on docs.nyc1.psf.io.
        Make sure the files are in group docs and are group-writeable.

    -   Let the DE check if the docs are built and work all right.

    -   Note both the documentation and downloads are behind a caching
        CDN. If you change archives after downloading them through the
        website, you'll need to purge the stale data in the CDN like
        this:

            curl -X PURGE https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz

        You should always purge the cache of the directory listing as
        people use that to browse the release files:

            curl -X PURGE https://www.python.org/ftp/python/3.12.0/

-   For the extra paranoid, do a completely clean test of the release.
    This includes downloading the tarball from www.python.org.

    Make sure the md5 checksums match. Then unpack the tarball, and do a
    clean make test:

        make distclean
        ./configure
        make test

    To ensure that the regression test suite passes. If not, you screwed
    up somewhere!

Warning

STOP and confirm:

-   Have you gotten the green light from the WE?
-   Have you gotten the green light from the ME?
-   Have you gotten the green light from the DE?

If green, it's time to merge the release engineering branch back into
the main repo.

-   In order to push your changes to GitHub, you'll have to temporarily
    disable branch protection for administrators. Go to the
    Settings | Branches page:

    https://github.com/python/cpython/settings/branches

    "Edit" the settings for the branch you're releasing on. This will
    load the settings page for that branch. Uncheck the "Include
    administrators" box and press the "Save changes" button at the
    bottom.

-   Merge your release clone into the main development repo:

        # Pristine copy of the upstream repo branch
        git clone git@github.com:python/cpython.git merge
        cd merge

        # Checkout the correct branch:
        # 1. For feature pre-releases up to and including a
        #    **new branch** release, i.e. alphas and first beta
        #   do a checkout of the main branch
        git checkout main

        # 2. Else, for all other releases, checkout the
        #       appropriate release branch.
        git checkout 3.X

        # Fetch the newly created and signed tag from your clone repo
        git fetch --tags git@github.com:your-github-id/cpython.git v3.X.YaN
        # Merge the temporary release engineering branch back into
        git merge --no-squash v3.X.YaN
        git commit -m 'Merge release engineering branch'

-   If this is a new branch release, i.e. first beta, now create the new
    release branch:

        git checkout -b 3.X

    Do any steps needed to setup the new release branch, including:

    -   In README.rst, change all references from main to the new
        branch, in particular, GitHub repo URLs.

-   For all releases, do the guided post-release steps with the release
    script:

        .../release-tools/release.py --done 3.X.YaN

-   For a final or release candidate 2+ release, you may need to do some
    post-merge cleanup. Check the top-level README.rst and
    include/patchlevel.h files to ensure they now reflect the desired
    post-release values for on-going development. The patchlevel should
    be the release tag with a +. Also, if you cherry-picked changes from
    the standard release branch into the release engineering branch for
    this release, you will now need to manually remove each blurb entry
    from the Misc/NEWS.d/next directory that was cherry-picked into the
    release you are working on since that blurb entry is now captured in
    the merged x.y.z.rst file for the new release. Otherwise, the blurb
    entry will appear twice in the changelog.html file, once under
    Python next and again under x.y.z.

-   Review and commit these changes:

        git commit -m 'Post release updates'

-   If this is a new branch release (e.g. the first beta), update the
    main branch to start development for the following feature release.
    When finished, the main branch will now build Python X.Y+1.

    -   First, set main up to be the next release, i.e. X.Y+1.a0:

            git checkout main
            .../release-tools/release.py --bump 3.9.0a0

    -   Edit all version references in README.rst

    -   Edit Doc/tutorial/interpreter.rst (two references to
        '[Pp]ython3x', one to 'Python 3.x', also make the date in the
        banner consistent).

    -   Edit Doc/tutorial/stdlib.rst and Doc/tutorial/stdlib2.rst, which
        have each one reference to '[Pp]ython3x'.

    -   Add a new whatsnew/3.x.rst file (with the comment near the top
        and the toplevel sections copied from the previous file) and add
        it to the toctree in whatsnew/index.rst. But beware that the
        initial whatsnew/3.x.rst checkin from previous releases may be
        incorrect due to the initial midstream change to blurb that
        propagates from release to release! Help break the cycle: if
        necessary make the following change:

            -For full details, see the :source:`Misc/NEWS` file.
            +For full details, see the :ref:`changelog <changelog>`.

    -   Update the version number in configure.ac and re-run autoconf.

    -   Make sure the SOURCE_URI in Doc/tools/extensions/pyspecific.py
        points to main.

    -   Update the version numbers for the Windows builds which have
        references to python38:

            ls PC/pyconfig.h.in PCbuild/rt.bat | xargs sed -i 's/python3\(\.\?\)[0-9]\+/python3\19/g'

    -   Edit the bug.yml and crash.yml issue templates in
        .github/ISSUE_TEMPLATE/ to add the new branch to the "versions"
        dropdown.

    -   Commit these changes to the main branch:

            git status
            git add ...
            git commit -m 'Bump to 3.9.0a0'

-   Do another git status in this directory.

    You should not see any files, i.e., you better not have any
    uncommitted changes in your working directory.

-   Commit and push to the main repo:

        # Do a dry run first.

        # For feature pre-releases prior to a **new branch** release,
        #   i.e. a feature alpha release:
        git push --dry-run --tags  git@github.com:python/cpython.git main
        # If it looks OK, take the plunge.  There's no going back!
        git push --tags  git@github.com:python/cpython.git main

        # For a **new branch** release, i.e. first beta:
        git push --dry-run --tags  git@github.com:python/cpython.git 3.X
        git push --dry-run --tags  git@github.com:python/cpython.git main
        # If it looks OK, take the plunge.  There's no going back!
        git push --tags  git@github.com:python/cpython.git 3.X
        git push --tags  git@github.com:python/cpython.git main

        # For all other releases:
        git push --dry-run --tags  git@github.com:python/cpython.git 3.X
        # If it looks OK, take the plunge.  There's no going back!
        git push --tags  git@github.com:python/cpython.git 3.X

-   If this is a new branch release, add a Branch protection rule for
    the newly created branch (3.X). Look at the values for the previous
    release branch (3.X-1) and use them as a template.
    https://github.com/python/cpython/settings/branches

    Also, add a needs backport to 3.X label to the GitHub repo.
    https://github.com/python/cpython/labels

-   You can now re-enable enforcement of branch settings against
    administrators on GitHub. Go back to the Settings | Branch page:

    https://github.com/python/cpython/settings/branches

    "Edit" the settings for the branch you're releasing on. Re-check the
    "Include administrators" box and press the "Save changes" button at
    the bottom.

Now it's time to twiddle the website. Almost none of this is automated,
sorry.

To do these steps, you must have the permission to edit the website. If
you don't have that, ask someone on<EMAIL_ADDRESS>for the proper
permissions.

-   Log in to https://www.python.org/admin

-   Create a new "release" for the release. Currently "Releases" are
    sorted under "Downloads".

    The easiest thing is probably to copy fields from an existing Python
    release "page", editing as you go.

    You can use Markdown or reStructured Text to describe your release.
    The former is less verbose, while the latter has nifty integration
    for things like referencing PEPs.

    Leave the "Release page" field on the form empty.

-   "Save" the release.

-   Populate the release with the downloadable files.

    Your friend and mine, Georg Brandl, made a lovely tool called
    add_to_pydotorg.py. You can find it in the python/release-tools repo
    (next to release.py). You run the tool on downloads.nyc1.psf.io,
    like this:

        AUTH_INFO=<username>:<python.org-api-key> python add_to_pydotorg.py <version>

    This walks the correct download directory for <version>, looks for
    files marked with <version>, and populates the "Release Files" for
    the correct "release" on the web site with these files. Note that
    clears the "Release Files" for the relevant version each time it's
    run. You may run it from any directory you like, and you can run it
    as many times as you like if the files happen to change. Keep a copy
    in your home directory on dl-files and keep it fresh.

    If new types of files are added to the release, someone will need to
    update add_to_pydotorg.py so it recognizes these new files. (It's
    best to update add_to_pydotorg.py when file types are removed, too.)

    The script will also sign any remaining files that were not signed
    with Sigstore until this point. Again, if this happens, do use your
    @python.org address for this process. More info:
    https://www.python.org/download/sigstore/

-   In case the CDN already cached a version of the Downloads page
    without the files present, you can invalidate the cache using:

        curl -X PURGE https://www.python.org/downloads/release/python-XXX/

-   If this is a final release:

    -   Add the new version to the Python Documentation by Version page
        https://www.python.org/doc/versions/ and remove the current
        version from any 'in development' section.

    -   For 3.X.Y, edit all the previous X.Y releases' page(s) to point
        to the new release. This includes the content field of the
        Downloads -> Releases entry for the release:

            Note: Python 3.x.(y-1) has been superseded by
            `Python 3.x.y </downloads/release/python-3xy/>`_.

        And, for those releases having separate release page entries
        (phasing these out?), update those pages as well, e.g.
        download/releases/3.x.y:

            Note: Python 3.x.(y-1) has been superseded by
            `Python 3.x.y </download/releases/3.x.y/>`_.

    -   Update the "Current Pre-release Testing Versions web page".

        There's a page that lists all the currently-in-testing versions
        of Python:

        -   https://www.python.org/download/pre-releases/

        Every time you make a release, one way or another you'll have to
        update this page:

        -   If you're releasing a version before 3.x.0, you should add
            it to this page, removing the previous pre-release of
            version 3.x as needed.
        -   If you're releasing 3.x.0 final, you need to remove the
            pre-release version from this page.

        This is in the "Pages" category on the Django-based website, and
        finding it through that UI is kind of a chore. However! If
        you're already logged in to the admin interface (which, at this
        point, you should be), Django will helpfully add a convenient
        "Edit this page" link to the top of the page itself. So you can
        simply follow the link above, click on the "Edit this page"
        link, and make your changes as needed. How convenient!

    -   If appropriate, update the "Python Documentation by Version"
        page:

        -   https://www.python.org/doc/versions/

        This lists all releases of Python by version number and links to
        their static (not built daily) online documentation. There's a
        list at the bottom of in-development versions, which is where
        all alphas/betas/RCs should go. And yes you should be able to
        click on the link above then press the shiny, exciting "Edit
        this page" button.

-   Write the announcement on discuss.python.org. This is the fuzzy bit
    because not much can be automated. You can use an earlier
    announcement as a template, but edit it for content!

-   Once the announcement is up on Discourse, send an equivalent to the
    following mailing lists:

    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>    -<EMAIL_ADDRESS>
-   Also post the announcement to the Python Insider blog. To add a new
    entry, go to your Blogger home page, here.

-   Update any release PEPs (e.g. 719) with the release dates.

-   Update the labels on https://github.com/python/cpython/issues:

    -   Flip all the deferred-blocker issues back to release-blocker for
        the next release.
    -   Add version 3.X+1 as when version 3.X enters alpha.
    -   Change non-doc feature requests to version 3.X+1 when version
        3.X enters beta.
    -   Update issues from versions that your release makes unsupported
        to the next supported version.
    -   Review open issues, as this might find lurking showstopper bugs,
        besides reminding people to fix the easy ones they forgot about.

-   You can delete the remote release clone branch from your repo clone.

-   If this is a new branch release, you will need to ensure various
    pieces of the development infrastructure are updated for the new
    branch. These include:

    -   Update the issue tracker for the new branch: add the new version
        to the versions list.
    -   Update the devguide to reflect the new branches and versions.
    -   Create a PR to update the supported releases table on the
        downloads page (see python/pythondotorg#1302).
    -   Ensure buildbots are defined for the new branch (contact Łukasz
        or Zach Ware).
    -   Ensure the various GitHub bots are updated, as needed, for the
        new branch, in particular, make sure backporting to the new
        branch works (contact the core-workflow team).
    -   Review the most recent commit history for the main and new
        release branches to identify and backport any merges that might
        have been made to the main branch during the release engineering
        phase and that should be in the release branch.
    -   Verify that CI is working for new PRs for the main and new
        release branches and that the release branch is properly
        protected (no direct pushes, etc).
    -   Verify that the on-line docs are building properly (this may
        take up to 24 hours for a complete build on the website).

What Next?

-   Verify! Pretend you're a user: download the files from
    www.python.org, and make Python from it. This step is too easy to
    overlook, and on several occasions we've had useless release files.
    Once a general server problem caused mysterious corruption of all
    files; once the source tarball got built incorrectly; more than once
    the file upload process on SF truncated files; and so on.
-   Rejoice. Drink. Be Merry. Write a PEP like this one. Or be like unto
    Guido and take A Vacation.

You've just made a Python release!

Moving to End-of-life

Under current policy, a release branch normally reaches end-of-life
status five years after its initial release. The policy is discussed in
more detail in the Python Developer's Guide. When end-of-life is
reached, there are a number of tasks that need to be performed either
directly by you as release manager or by ensuring someone else does
them. Some of those tasks include:

-   Optionally making a final release to publish any remaining
    unreleased changes.

-   Freeze the state of the release branch by creating a tag of its
    current HEAD and then deleting the branch from the CPython repo. The
    current HEAD should be at or beyond the final security release for
    the branch:

        git fetch upstream
        git tag --sign -m 'Final head of the former 3.3 branch' 3.3 upstream/3.3
        git push upstream refs/tags/3.3

-   If all looks good, delete the branch. This may require the
    assistance of someone with repo administrator privileges:

        git push upstream --delete 3.3  # or perform from GitHub Settings page

-   Remove the release from the list of "Active Python Releases" on the
    Downloads page. To do this, log in to the admin page for python.org,
    navigate to Boxes, and edit the downloads-active-releases entry.
    Strip out the relevant paragraph of HTML for your release. (You'll
    probably have to do the curl -X PURGE trick to purge the cache if
    you want to confirm you made the change correctly.)

-   Add a retired notice to each release page on python.org for the
    retired branch. For example:

    -   https://www.python.org/downloads/release/python-337/
    -   https://www.python.org/downloads/release/python-336/

-   In the developer's guide, set the branch status to end-of-life and
    update or remove references to the branch elsewhere in the devguide.

-   Retire the release from the issue tracker. Tasks include:

    -   remove version label from list of versions
    -   remove the needs backport to label for the retired version
    -   review and dispose of open issues marked for this branch

-   Announce the branch retirement in the usual places:

    -   discuss.python.org
    -   mailing lists (python-dev, python-list, python-announcements)
    -   Python Dev blog

-   Enjoy your retirement and bask in the glow of a job well done!

Windows Notes

Windows has a MSI installer, various flavors of Windows have "special
limitations", and the Windows installer also packs precompiled "foreign"
binaries (Tcl/Tk, expat, etc).

The installer is tested as part of the Azure Pipeline. In the past,
those steps were performed manually. We're keeping this for posterity.

Concurrent with uploading the installer, the WE installs Python from it
twice: once into the default directory suggested by the installer, and
later into a directory with embedded spaces in its name. For each
installation, the WE runs the full regression suite from a DOS box, and
both with and without -0. For maintenance release, the WE also tests
whether upgrade installations succeed.

The WE also tries every shortcut created under Start -> Menu -> the
Python group. When trying IDLE this way, you need to verify that Help ->
Python Documentation works. When trying pydoc this way (the "Module
Docs" Start menu entry), make sure the "Start Browser" button works, and
make sure you can search for a random module (like "random" <wink>) and
then that the "go to selected" button works.

It's amazing how much can go wrong here -- and even more amazing how
often last-second checkins break one of these things. If you're "the
Windows geek", keep in mind that you're likely the only person routinely
testing on Windows, and that Windows is simply a mess.

Repeat the testing for each target architecture. Try both an Admin and a
plain User (not Power User) account.

Copyright

This document has been placed in the public domain.

license: Public Domain

authors:
Barry Warsaw
Guido van Rossum


text:
PEP: 738 Title: Adding Android as a supported platform Author: Malcolm
Smith<EMAIL_ADDRESS>Sponsor: Petr Viktorin<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-738-adding-android-as-a-supported-platform/40975
Status: Final Type: Standards Track Created: 12-Dec-2023 Python-Version:
3.13 Resolution:
https://discuss.python.org/t/pep-738-adding-android-as-a-supported-platform/40975/23

python:using-android

Abstract

This PEP proposes adding Android as a supported platform in CPython. The
initial goal is for Android to achieve Tier 3 support in Python 3.13.

This PEP is based on PEP 730 – "Adding iOS as a supported platform" by
Russell Keith-Magee, and covers many of the same issues. Notable
differences between the two platforms can be found by searching for the
word "iOS".

Motivation

Over the last 15 years, mobile platforms have become increasingly
important parts of the computing landscape. Android is the operating
system that runs on about 70% of these devices. However, there is no
official support for Android in CPython.

The Chaquopy, BeeWare and Kivy projects have all supported Android for
many years, and they have all been used to generate applications that
have been accepted for publication in the Google Play Store. This
demonstrates the technical feasibility of Android support.

It is important for the future of Python as a language that it is able
to be used on any platform that has widespread adoption. Otherwise,
potential users will choose other languages that do provide support for
these platforms. This is especially true in education, where the next
generation of developers is in many cases already spending more time
using mobile platforms than desktop ones.

Rationale

General

Android is broadly a POSIX platform, based on a Linux kernel and the ELF
binary format. It does not use glibc, instead providing its own C
library implementation called Bionic. As a result, it is generally not
binary-compatible with any other Linux distribution, even if the
architecture matches. It also has its own filesystem layout which
doesn't resemble any other Unix.

However, Android's source-compatibility with Linux is quite good. In its
early years, the C library was very incomplete, but most of the gaps
were filled by around 2014. Since then, any C code that compiles for
Linux can usually be compiled for Android, unless it involves direct
access to hardware devices or operating system services.

This is also true of CPython. Although it has never officially supported
Android, recent versions (since 3.6) can already be compiled for Android
with minimal patching.

OS versions

Each Android version can be identified in three ways:

-   A conventional dotted version number (though recent versions have
    all used whole numbers)
-   A sequential integer "API level" (the most common form in developer
    documentation)
-   An alphabetic confectionery-themed code name (no longer used for
    marketing, but still appears in developer documentation)

There is no consistent pattern to link one of these to another; they
must be looked up in a table.

A new major Android version is released each year, but the updates
available to each device are entirely under the control of its
manufacturer. Unfortunately many manufacturers stop sending updates to
devices long before their users are ready to dispose of them. For
example, as of October 2023, the oldest Android version still receiving
security updates was API level 30, but according to Google's own
statistics, only 60% of devices were on that version or newer.

For Python 3.13 we therefore propose the minimum Android version to be
5.0 (API level 21), which was released in 2014. According to the
statistics above, this would cover 99% of active devices.

Development tools

The Android development tools are equally supported on Linux (x86_64),
Windows (x86_64) and macOS (x86_64 and ARM64). For CPython, the most
important tools are:

-   The NDK (native development kit) contains a C and C++ compiler
    (clang), linker (lld), and headers for all the system libraries.

    Binary compatibility between libraries compiled with different
    versions of the NDK is generally very good, but for reproducibility
    it would be best for each Python version to stick with one NDK
    version throughout its life. For Python 3.13, this would be the
    current NDK long-term support version, r26.

    Each NDK version can be set to target any of a wide range of Android
    versions. For example, NDK r26 supports API levels <738-os-versions>
    21 to 34. However, binaries compiled for an older Android version
    will usually keep on working indefinitely on newer versions;
    exceptions to this rule are only made for security reasons.

-   Gradle is the tool used to build complete, deployable apps.

-   The emulator, based on QEMU, is a simulated Android device running
    on a development machine. Unlike on iOS, an emulator uses the same
    ABI as a real device of the same architecture, and can run the same
    binaries.

These tools may all be used either from the command line, or through the
Android Studio IDE, which is based on IntelliJ IDEA.

Architectures

Android currently supports 4 architectures. Their names as used by the
Android tools are:

-   armeabi-v7a
-   arm64-v8a
-   x86
-   x86_64

Virtually all current physical devices use one of the ARM architectures.
x86 and x86_64 are supported for use in the emulator.

For Python 3.13 we propose that Tier 3 support will only cover the
64-bit platforms (arm64-v8a and x86_64):

-   x86 has not been supported as a development platform since 2020, and
    no new emulator images have been released since then.

-   armeabi-v7a's proportion of active devices is now less than 10% and
    steadily falling.

    It would also be more difficult to cover with a reliable buildbot,
    since there are no native hosts available for the emulator (ARM64
    Macs don't have hardware support for ARM32 code). Although
    cross-architecture emulation is possible, it has much worse
    performance and stability, which is why the armeabi-v7a emulator
    images have not been updated since 2016.

    However, it continues to be used for watches and ultra-low-cost
    phones. If this persists, we may need to consider adding it in a
    future Python version.

Even if 32-bit architectures are not officially supported, no changes
should be made which would impede any downstream projects which still
wish to build them.

App lifecycle

The primary programming language in Android apps is Java, or its modern
descendant Kotlin. As such, an app does not provide its own executable
file. Instead, all apps start off as a Java virtual machine running an
executable provided by the operating system. The app's Java code can
then add native code to the process by loading dynamic libraries and
calling them through JNI.

Unlike iOS, creating subprocesses is supported on Android. However apps
may only run executables in certain locations, none of which are
writable at runtime. Long-running subprocesses are officially
discouraged, and are not guaranteed to be supported in future Android
versions.

Android does provide a command-line shell, but this is intended only for
use by developers, and is not available to the typical end user.

For these reasons, the recommended way of running Python on Android will
be by loading libpython3.x.so into the main app process. A python3.x
executable will not be officially supported on this platform.

Specification

Scope of work

The focus of this work will be to produce an Android equivalent to the
existing Windows embeddable package, i.e. a set of compiled libraries
which developers can add to their apps. No installer will be required.

Adding Android as a Tier 3 platform only requires adding support for
compiling an Android-compatible build from the unpatched CPython source
code. It does not necessarily require there to be any officially
distributed Android artifacts on python.org, although these could be
added in the future.

Android will be built using the same configure and Makefile system as
other POSIX platforms, and must therefore be built on a POSIX platform.
Both Linux and macOS will be supported.

A Gradle project will be provided for the purpose of running the CPython
test suite. Tooling will be provided to automate the process of building
the test suite app, starting the emulator, installing the test suite,
and executing it.

Linkage

For the reasons discussed in App lifecycle, Python will be included in
the app as a dynamic libpython3.x.so library which can be loaded into an
app using dlopen.

Unlike Linux, Android does not implicitly use a dlopened library to
resolve relocations in subsequently-loaded libraries, even if
RTLD_GLOBAL is used. All Python extension modules must therefore be
explicitly linked against libpython3.x.so when building for Android.

An extension module linked against libpython3.x.so cannot be loaded by
an executable that has been statically linked against libpython3.x.a.
Therefore, a static libpython3.x.a library will not be supported on
Android. This is the same pattern used by CPython on Windows.

This approach also allows using the -Wl,--no-undefined option to detect
missing symbols at build time, which can be a significant time-saver.

Unlike iOS, Android allows dynamic libraries to be loaded from any
location, so a directory tree containing co-located .py, .pyc and .so
files can be handled by Python's standard importer.

Standard library

Unsupported modules

A number of standard library modules will not be supported on Android
because the underlying C APIs are not available:

-   curses and readline
-   dbm.gnu and dbm.ndbm
-   grp
-   multiprocessing – although subprocesses in general are allowed (see
    App lifecycle), Android does not support any part of the System V
    IPC API.
-   tkinter and turtle – these would require an Android build of Tk
    itself, which is not officially supported.

sys

sys.platform will return "android". Although Android is based on Linux,
it differs in enough significant ways that a separate name is justified.

When embedded in an Android app, the C-level stdio streams are not
connected to anything. So in this mode, sys.stdout and sys.stderr will
be redirected to the system Logcat, which can be viewed with the Android
development tools. sys.stdin will always return EOF.

platform

Most of the values returned by the platform module will match those
returned by os.uname(), with the exception of:

-   platform.system() - "Android", instead of the default "Linux"
-   platform.release() - Android version number, as a string (e.g.
    "14"), instead of the Linux kernel version

In addition, a platform.android_ver() method will be added, which
returns a namedtuple containing the following:

-   release - Android version of the device, as a string (e.g. "14")
-   api_level - API level <738-os-versions> of the device, as an integer
    (e.g. 34)
-   manufacturer - manufacturer of the device, as a string (e.g.
    "Google")
-   model - model name of the device, as a string (e.g. "Pixel 7")
-   device - device name of the device, as a string (e.g. "panther")
-   is_emulator - True if the device is an emulator; False if it’s a
    physical device.

Which one of model and device is more likely to be unique, and which one
is more likely to resemble the marketing name, varies between different
manufacturers.

os

os.uname() will return the raw result of a POSIX uname() call. This will
result in the following values:

-   sysname - "Linux"
-   release - The Linux kernel version (e.g.
    "5.10.157-android13-4-00003-gdfb1120f912b-ab10994928")

This approach treats the os module as a "raw" interface to system APIs,
and platform as a higher-level API providing more generally useful
values.

CI resources

Since Android emulators and physical devices use the same ABI, and come
with identical or very similar operating system binaries, testing on
emulators will be adequate. x86_64 emulators can be run on Linux, macOS
or Windows, but ARM64 emulators are only supported on ARM64 Macs.

Anaconda has offered to provide physical hardware to run Android
buildbots. These will include both Linux x86_64 and macOS ARM64
machines, which would cover both supported runtime architectures and
both supported build platforms.

CPython does not currently test Tier 3 platforms on GitHub Actions, but
if this ever changes, their Linux and macOS runners are also able to
host Android emulators. macOS ARM64 runners have been free to all public
repositories since January 2024.

Packaging

Android wheels will use tags in the format android_<api-level>_<abi>.
For example:

-   android_21_arm64_v8a
-   android_21_x86_64

For the meaning of <api-level>, see OS versions. In the context of the
wheel tag, it indicates the minimum Android version that was selected
when the wheel was compiled. Installation tools such as pip should
interpret this in a similar way to the existing macOS tags, i.e. an app
with a minimum API level of N can incorporate wheels tagged with API
level N or older.

This format originates from the Chaquopy project, which currently
maintains a wheel repository with tags varying between API levels 16 and
21.

However, relying on a small group of Android enthusiasts to build the
whole Python ecosystem is not a scalable solution. Until prominent
libraries routinely release their own Android wheels, the ability of the
community to adopt Python on Android will be limited.

Therefore, it will be necessary to clearly document how projects can add
Android builds to their CI and release tooling. Adding Android support
to tools like crossenv and cibuildwheel may be one way to achieve this.

The Android wheel tag format should also be added to the list of tags
accepted by PyPI.

PEP 11 Update

PEP 11 will be updated to include the two supported Android ABIs.
Autoconf already identifies them with the following triplets:

-   aarch64-linux-android
-   x86_64-linux-android

Petr Viktorin will serve as the initial core team contact for these
ABIs.

Backwards Compatibility

Adding a new platform does not introduce any backwards compatibility
concerns to CPython itself. However, there may be some backwards
compatibility implications on the projects that have historically
provided CPython support (i.e., BeeWare and Kivy) if the final form of
any CPython patches don't align with the patches they have historically
used.

Security Implications

Adding a new platform does not add any new security implications.

How to Teach This

The education needs related to this PEP relate to two groups of
developers.

First, developers of apps need to know how to build Python into an
Android app, along with their own Python code and any supporting
packages, and how to use them all at runtime. The documentation will
cover this in a similar form to the existing Windows embeddable package.
However, it will recommend most developers to use higher-level tools
such as Briefcase, Chaquopy and Buildozer, all of which already have
comprehensive documentation.

Second, developers of packages with binary components need to know how
to build and release them for Android (see Packaging).

Reference Implementation

The Chaquopy repository contains a reference patch and build scripts.
These will have to be decoupled from the other components of Chaquopy
before they can be upstreamed.

Briefcase provides a reference implementation of code to execute test
suites on Android devices and emulators. The Toga Testbed is an example
of a test suite that is executed on the Android emulator using GitHub
Actions.

Rejected Ideas

The following changes were made to the original specification of
platform.android_ver():

-   The min_api_level field was removed, because unlike all the other
    fields, it isn't a property of the current device. This information
    is still available from the pre-existing function
    sys.getandroidapilevel().
-   The is_emulator field was added, since experience during testing
    showed that some issues were emulator-specific.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 3128 Title: BList: A Faster List-like Type Version: $Revision$
Last-Modified: $Date$ Author: Daniel Stutzbach
<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
30-Apr-2007 Python-Version: 2.6, 3.0 Post-History: 30-Apr-2007

Rejection Notice

Rejected based on Raymond Hettinger's sage advice[1]:

  After looking at the source, I think this has almost zero chance for
  replacing list(). There is too much value in a simple C API, low space
  overhead for small lists, good performance is common use cases, and
  having performance that is easily understood. The BList implementation
  lacks these virtues and it trades-off a little performance in common
  cases in for much better performance in uncommon cases. As a Py3.0
  PEP, I think it can be rejected.

  Depending on its success as a third-party module, it still has a
  chance for inclusion in the collections module. The essential criteria
  for that is whether it is a superior choice for some real-world use
  cases. I've scanned my own code and found no instances where BList
  would have been preferable to a regular list. However, that scan has a
  selection bias because it doesn't reflect what I would have written
  had BList been available. So, after a few months, I intend to poll
  comp.lang.python for BList success stories. If they exist, then I have
  no problem with inclusion in the collections module. After all, its
  learning curve is near zero -- the only cost is the clutter factor
  stemming from indecision about the most appropriate data structure for
  a given task.

Abstract

The common case for list operations is on small lists. The current
array-based list implementation excels at small lists due to the strong
locality of reference and infrequency of memory allocation operations.
However, an array takes O(n) time to insert and delete elements, which
can become problematic as the list gets large.

This PEP introduces a new data type, the BList, that has array-like and
tree-like aspects. It enjoys the same good performance on small lists as
the existing array-based implementation, but offers superior asymptotic
performance for most operations. This PEP proposes replacing the makes
two mutually exclusive proposals for including the BList type in Python:

1.  Add it to the collections module, or
2.  Replace the existing list type

Motivation

The BList grew out of the frustration of needing to rewrite intuitive
algorithms that worked fine for small inputs but took O(n**2) time for
large inputs due to the underlying O(n) behavior of array-based lists.
The deque type, introduced in Python 2.4, solved the most common problem
of needing a fast FIFO queue. However, the deque type doesn't help if we
need to repeatedly insert or delete elements from the middle of a long
list.

A wide variety of data structure provide good asymptotic performance for
insertions and deletions, but they either have O(n) performance for
other operations (e.g., linked lists) or have inferior performance for
small lists (e.g., binary trees and skip lists).

The BList type proposed in this PEP is based on the principles of
B+Trees, which have array-like and tree-like aspects. The BList offers
array-like performance on small lists, while offering O(log n)
asymptotic performance for all insert and delete operations.
Additionally, the BList implements copy-on-write under-the-hood, so even
operations like getslice take O(log n) time. The table below compares
the asymptotic performance of the current array-based list
implementation with the asymptotic performance of the BList.

  Operation   Array-based list   BList
  ----------- ------------------ ------------------
  Copy        O(n)               O(1)
  Append      O(1)               O(log n)
  Insert      O(n)               O(log n)
  Get Item    O(1)               O(log n)
  Set Item    O(1)               O(log n)
  Del Item    O(n)               O(log n)
  Iteration   O(n)               O(n)
  Get Slice   O(k)               O(log n)
  Del Slice   O(n)               O(log n)
  Set Slice   O(n+k)             O(log k + log n)
  Extend      O(k)               O(log k + log n)
  Sort        O(n log n)         O(n log n)
  Multiply    O(nk)              O(log k)

An extensive empirical comparison of Python's array-based list and the
BList are available at[2].

Use Case Trade-offs

The BList offers superior performance for many, but not all, operations.
Choosing the correct data type for a particular use case depends on
which operations are used. Choosing the correct data type as a built-in
depends on balancing the importance of different use cases and the
magnitude of the performance differences.

For the common uses cases of small lists, the array-based list and the
BList have similar performance characteristics.

For the slightly less common case of large lists, there are two common
uses cases where the existing array-based list outperforms the existing
BList reference implementation. These are:

1.  A large LIFO stack, where there are many .append() and .pop(-1)
    operations. Each operation is O(1) for an array-based list, but
    O(log n) for the BList.
2.  A large list that does not change size. The getitem and setitem
    calls are O(1) for an array-based list, but O(log n) for the BList.

In performance tests on a 10,000 element list, BLists exhibited a 50%
and 5% increase in execution time for these two uses cases,
respectively.

The performance for the LIFO use case could be improved to O(n) time, by
caching a pointer to the right-most leaf within the root node. For lists
that do not change size, the common case of sequential access could also
be improved to O(n) time via caching in the root node. However, the
performance of these approaches has not been empirically tested.

Many operations exhibit a tremendous speed-up (O(n) to O(log n)) when
switching from the array-based list to BLists. In performance tests on a
10,000 element list, operations such as getslice, setslice, and
FIFO-style insert and deletes on a BList take only 1% of the time needed
on array-based lists.

In light of the large performance speed-ups for many operations, the
small performance costs for some operations will be worthwhile for many
(but not all) applications.

Implementation

The BList is based on the B+Tree data structure. The BList is a wide,
bushy tree where each node contains an array of up to 128 pointers to
its children. If the node is a leaf, its children are the user-visible
objects that the user has placed in the list. If node is not a leaf, its
children are other BList nodes that are not user-visible. If the list
contains only a few elements, they will all be a children of single node
that is both the root and a leaf. Since a node is little more than array
of pointers, small lists operate in effectively the same way as an
array-based data type and share the same good performance
characteristics.

The BList maintains a few invariants to ensure good (O(log n))
asymptotic performance regardless of the sequence of insert and delete
operations. The principle invariants are as follows:

1.  Each node has at most 128 children.
2.  Each non-root node has at least 64 children.
3.  The root node has at least 2 children, unless the list contains
    fewer than 2 elements.
4.  The tree is of uniform depth.

If an insert would cause a node to exceed 128 children, the node spawns
a sibling and transfers half of its children to the sibling. The sibling
is inserted into the node's parent. If the node is the root node (and
thus has no parent), a new parent is created and the depth of the tree
increases by one.

If a deletion would cause a node to have fewer than 64 children, the
node moves elements from one of its siblings if possible. If both of its
siblings also only have 64 children, then two of the nodes merge and the
empty one is removed from its parent. If the root node is reduced to
only one child, its single child becomes the new root (i.e., the depth
of the tree is reduced by one).

In addition to tree-like asymptotic performance and array-like
performance on small-lists, BLists support transparent copy-on-write. If
a non-root node needs to be copied (as part of a getslice, copy,
setslice, etc.), the node is shared between multiple parents instead of
being copied. If it needs to be modified later, it will be copied at
that time. This is completely behind-the-scenes; from the user's point
of view, the BList works just like a regular Python list.

Memory Usage

In the worst case, the leaf nodes of a BList have only 64 children each,
rather than a full 128, meaning that memory usage is around twice that
of a best-case array implementation. Non-leaf nodes use up a negligible
amount of additional memory, since there are at least 63 times as many
leaf nodes as non-leaf nodes.

The existing array-based list implementation must grow and shrink as
items are added and removed. To be efficient, it grows and shrinks only
when the list has grow or shrunk exponentially. In the worst case, it,
too, uses twice as much memory as the best case.

In summary, the BList's memory footprint is not significantly different
from the existing array-based implementation.

Backwards Compatibility

If the BList is added to the collections module, backwards compatibility
is not an issue. This section focuses on the option of replacing the
existing array-based list with the BList. For users of the Python
interpreter, a BList has an identical interface to the current
list-implementation. For virtually all operations, the behavior is
identical, aside from execution speed.

For the C API, BList has a different interface than the existing
list-implementation. Due to its more complex structure, the BList does
not lend itself well to poking and prodding by external sources.
Thankfully, the existing list-implementation defines an API of functions
and macros for accessing data from list objects. Google Code Search
suggests that the majority of third-party modules uses the well-defined
API rather than relying on the list's structure directly. The table
below summarizes the search queries and results:

+--------------------------+-------------------+
| Search String            | Number of Results |
+==========================+===================+
| PyList_GetItem           | 2,000             |
+--------------------------+-------------------+
| PySequence_GetItem       |   800             |
+--------------------------+-------------------+
| PySequence_Fast_GET_ITEM |   100             |
+--------------------------+-------------------+
| PyList_GET_ITEM          |   400             |
+--------------------------+-------------------+
| [^a-zA-Z_]ob_item        |   100             |
+--------------------------+-------------------+

This can be achieved in one of two ways:

1.  Redefine the various accessor functions and macros in listobject.h
    to access a BList instead. The interface would be unchanged. The
    functions can easily be redefined. The macros need a bit more care
    and would have to resort to function calls for large lists.

    The macros would need to evaluate their arguments more than once,
    which could be a problem if the arguments have side effects. A
    Google Code Search for "PyList_GET_ITEM([^)]+(" found only a handful
    of cases where this occurs, so the impact appears to be low.

    The few extension modules that use list's undocumented structure
    directly, instead of using the API, would break. The core code
    itself uses the accessor macros fairly consistently and should be
    easy to port.

2.  Deprecate the existing list type, but continue to include it.
    Extension modules wishing to use the new BList type must do so
    explicitly. The BList C interface can be changed to match the
    existing PyList interface so that a simple search-replace will be
    sufficient for 99% of module writers.

    Existing modules would continue to compile and work without change,
    but they would need to make a deliberate (but small) effort to
    migrate to the BList.

    The downside of this approach is that mixing modules that use BLists
    and array-based lists might lead to slow down if conversions are
    frequently necessary.

Reference Implementation

A reference implementations of the BList is available for CPython at[3].

The source package also includes a pure Python implementation,
originally developed as a prototype for the CPython version. Naturally,
the pure Python version is rather slow and the asymptotic improvements
don't win out until the list is quite large.

When compiled with Py_DEBUG, the C implementation checks the BList
invariants when entering and exiting most functions.

An extensive set of test cases is also included in the source package.
The test cases include the existing Python sequence and list test cases
as a subset. When the interpreter is built with Py_DEBUG, the test cases
also check for reference leaks.

Porting to Other Python Variants

If the BList is added to the collections module, other Python variants
can support it in one of three ways:

1.  Make blist an alias for list. The asymptotic performance won't be as
    good, but it'll work.
2.  Use the pure Python reference implementation. The performance for
    small lists won't be as good, but it'll work.
3.  Port the reference implementation.

Discussion

This proposal has been discussed briefly on the Python-3000 mailing
list[4]. Although a number of people favored the proposal, there were
also some objections. Below summarizes the pros and cons as observed by
posters to the thread.

General comments:

-   Pro: Will outperform the array-based list in most cases
-   Pro: "I've implemented variants of this ... a few different times"
-   Con: Desirability and performance in actual applications is unproven

Comments on adding BList to the collections module:

-   Pro: Matching the list-API reduces the learning curve to near-zero
-   Pro: Useful for intermediate-level users; won't get in the way of
    beginners
-   Con: Proliferation of data types makes the choices for developers
    harder.

Comments on replacing the array-based list with the BList:

-   Con: Impact on extension modules (addressed in Backwards
    Compatibility)
-   Con: The use cases where BLists are slower are important (see Use
    Case Trade-Offs for how these might be addressed).
-   Con: The array-based list code is simple and easy to maintain

To assess the desirability and performance in actual applications,
Raymond Hettinger suggested releasing the BList as an extension module
(now available at[5]). If it proves useful, he felt it would be a strong
candidate for inclusion in 2.6 as part of the collections module. If
widely popular, then it could be considered for replacing the
array-based list, but not otherwise.

Guido van Rossum commented that he opposed the proliferation of data
types, but favored replacing the array-based list if backwards
compatibility could be addressed and the BList's performance was
uniformly better.

On-going Tasks

-   Reduce the memory footprint of small lists
-   Implement TimSort for BLists, so that best-case sorting is O(n)
    instead of O(log n).
-   Implement __reversed__
-   Cache a pointer in the root to the rightmost leaf, to make LIFO
    operation O(n) time.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Raymond Hettinger's feedback on python-3000:
https://mail.python.org/pipermail/python-3000/2007-May/007491.html

[2] Empirical performance comparison between Python's array-based list
and the blist: http://stutzbachenterprises.com/blist/

[3] Reference Implementations for C and Python:
http://www.python.org/pypi/blist/

[4] Discussion on python-3000 starting at post:
https://mail.python.org/pipermail/python-3000/2007-April/006757.html

[5] Reference Implementations for C and Python:
http://www.python.org/pypi/blist/


text:
PEP: 226 Title: Python 2.1 Release Schedule Version: $Revision$
Last-Modified: $Date$ Author: Jeremy Hylton<EMAIL_ADDRESS>Status: Final Type: Informational Topic: Release Content-Type:
text/x-rst Created: 16-Oct-2000 Python-Version: 2.1 Post-History:

Abstract

This document describes the post Python 2.0 development and release
schedule. According to this schedule, Python 2.1 will be released in
April of 2001. The schedule primarily concerns itself with PEP-size
items. Small bug fixes and changes will occur up until the first beta
release.

Release Schedule

Tentative future release dates

[bugfix release dates go here]

Past release dates:

-   17-Apr-2001: 2.1 final release
-   15-Apr-2001: 2.1 release candidate 2
-   13-Apr-2001: 2.1 release candidate 1
-   23-Mar-2001: Python 2.1 beta 2 release
-   02-Mar-2001: First 2.1 beta release
-   02-Feb-2001: Python 2.1 alpha 2 release
-   22-Jan-2001: Python 2.1 alpha 1 release
-   16-Oct-2000: Python 2.0 final release

Open issues for Python 2.0 beta 2

Add a default unit testing framework to the standard library.

Guidelines for making changes for Python 2.1

The guidelines and schedule will be revised based on discussion in the
<EMAIL_ADDRESS>mailing list.

The PEP system was instituted late in the Python 2.0 development cycle
and many changes did not follow the process described in PEP 1. The
development process for 2.1, however, will follow the PEP process as
documented.

The first eight weeks following 2.0 final will be the design and review
phase. By the end of this period, any PEP that is proposed for 2.1
should be ready for review. This means that the PEP is written and
discussion has occurred on the<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>mailing lists.

The next six weeks will be spent reviewing the PEPs and implementing and
testing the accepted PEPs. When this period stops, we will end
consideration of any incomplete PEPs. Near the end of this period, there
will be a feature freeze where any small features not worthy of a PEP
will not be accepted.

Before the final release, we will have six weeks of beta testing and a
release candidate or two.

General guidelines for submitting patches and making changes

Use good sense when committing changes. You should know what we mean by
good sense or we wouldn't have given you commit privileges <0.5 wink>.
Some specific examples of good sense include:

-   Do whatever the dictator tells you.
-   Discuss any controversial changes on python-dev first. If you get a
    lot of +1 votes and no -1 votes, make the change. If you get a some
    -1 votes, think twice; consider asking Guido what he thinks.
-   If the change is to code you contributed, it probably makes sense
    for you to fix it.
-   If the change affects code someone else wrote, it probably makes
    sense to ask him or her first.
-   You can use the SourceForge (SF) Patch Manager to submit a patch and
    assign it to someone for review.

Any significant new feature must be described in a PEP and approved
before it is checked in.

Any significant code addition, such as a new module or large patch, must
include test cases for the regression test and documentation. A patch
should not be checked in until the tests and documentation are ready.

If you fix a bug, you should write a test case that would have caught
the bug.

If you commit a patch from the SF Patch Manager or fix a bug from the
Jitterbug database, be sure to reference the patch/bug number in the CVS
log message. Also be sure to change the status in the patch manager or
bug database (if you have access to the bug database).

It is not acceptable for any checked in code to cause the regression
test to fail. If a checkin causes a failure, it must be fixed within 24
hours or it will be backed out.

All contributed C code must be ANSI C. If possible check it with two
different compilers, e.g. gcc and MSVC.

All contributed Python code must follow Guido's Python style guide.
http://www.python.org/doc/essays/styleguide.html

It is understood that any code contributed will be released under an
Open Source license. Do not contribute code if it can't be released this
way.


text:
PEP: 503 Title: Simple Repository API Version: $Revision$ Last-Modified:
$Date$ Author: Donald Stufft<EMAIL_ADDRESS>BDFL-Delegate: Donald
Stufft<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 04-Sep-2015 Post-History: 04-Sep-2015 Resolution:
https://mail.python.org/pipermail/distutils-sig/2015-September/026899.html

Abstract

There are many implementations of a Python package repository and many
tools that consume them. Of these, the canonical implementation that
defines what the "simple" repository API looks like is the
implementation that powers PyPI. This document will specify that API,
documenting what the correct behavior for any implementation of the
simple repository API.

Specification

A repository that implements the simple API is defined by its base URL,
this is the top level URL that all additional URLs are below. The API is
named the "simple" repository due to the fact that PyPI's base URL is
https://pypi.org/simple/.

Note

All subsequent URLs in this document will be relative to this base URL
(so given PyPI's URL, a URL of /foo/ would be
https://pypi.org/simple/foo/.

Within a repository, the root URL (/ for this PEP which represents the
base URL) MUST be a valid HTML5 page with a single anchor element per
project in the repository. The text of the anchor tag MUST be the name
of the project and the href attribute MUST link to the URL for that
particular project. As an example:

    <!DOCTYPE html>

        frob
        spamspamspam

Below the root URL is another URL for each individual project contained
within a repository. The format of this URL is /<project>/ where the
<project> is replaced by the normalized name for that project, so a
project named "HolyGrail" would have a URL like /holygrail/. This URL
must respond with a valid HTML5 page with a single anchor element per
file for the project. The href attribute MUST be a URL that links to the
location of the file for download, and the text of the anchor tag MUST
match the final path component (the filename) of the URL. The URL SHOULD
include a hash in the form of a URL fragment with the following syntax:
#<hashname>=<hashvalue>, where <hashname> is the lowercase name of the
hash function (such as sha256) and <hashvalue> is the hex encoded
digest.

In addition to the above, the following constraints are placed on the
API:

-   All URLs which respond with an HTML5 page MUST end with a / and the
    repository SHOULD redirect the URLs without a / to add a / to the
    end.

-   URLs may be either absolute or relative as long as they point to the
    correct location.

-   There are no constraints on where the files must be hosted relative
    to the repository.

-   There may be any other HTML elements on the API pages as long as the
    required anchor elements exist.

-   Repositories MAY redirect unnormalized URLs to the canonical
    normalized URL (e.g. /Foobar/ may redirect to /foobar/), however
    clients MUST NOT rely on this redirection and MUST request the
    normalized URL.

-   Repositories SHOULD choose a hash function from one of the ones
    guaranteed to be available via the hashlib module in the Python
    standard library (currently md5, sha1, sha224, sha256, sha384,
    sha512). The current recommendation is to use sha256.

-   If there is a GPG signature for a particular distribution file it
    MUST live alongside that file with the same name with a .asc
    appended to it. So if the file /packages/HolyGrail-1.0.tar.gz
    existed and had an associated signature, the signature would be
    located at /packages/HolyGrail-1.0.tar.gz.asc.

-   A repository MAY include a data-gpg-sig attribute on a file link
    with a value of either true or false to indicate whether or not
    there is a GPG signature. Repositories that do this SHOULD include
    it on every link.

-   A repository MAY include a data-requires-python attribute on a file
    link. This exposes the Requires-Python metadata field, specified in
    PEP 345, for the corresponding release. Where this is present,
    installer tools SHOULD ignore the download when installing to a
    Python version that doesn't satisfy the requirement. For example:

        ...

    In the attribute value, < and > have to be HTML encoded as < and
    >, respectively.

Normalized Names

This PEP references the concept of a "normalized" project name. As per
PEP 426 the only valid characters in a name are the ASCII alphabet,
ASCII numbers, ., -, and _. The name should be lowercased with all runs
of the characters ., -, or _ replaced with a single - character. This
can be implemented in Python with the re module:

    import re

    def normalize(name):
        return re.sub(r"[-_.]+", "-", name).lower()

Changes

-   The optional data-requires-python attribute was added in July 2016.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: