text:
PEP: 727 Title: Documentation in Annotated Metadata Author: Sebastián
Ramírez<EMAIL_ADDRESS>Sponsor: Jelle Zijlstra
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/32566 Status: Draft Type: Standards Track
Topic: Typing Content-Type: text/x-rst Created: 28-Aug-2023
Python-Version: 3.13 Post-History: 30-Aug-2023

Abstract

This PEP proposes a standardized way to provide documentation strings
for Python symbols defined with ~typing.Annotated using a new class
typing.Doc.

Motivation

There's already a well-defined way to provide documentation for classes,
functions, class methods, and modules: using docstrings.

Currently there is no formalized standard to provide documentation
strings for other types of symbols: parameters, return values,
class-scoped variables (class variables and instance variables), local
variables, and type aliases.

Nevertheless, to allow documenting most of these additional symbols,
several conventions have been created as microsyntaxes inside of
docstrings, and are currently commonly used: Sphinx, numpydoc, Google,
Keras, etc.

There are two scenarios in which these conventions would be supported by
tools: for authors, while editing the contents of the documentation
strings and for users, while rendering that content in some way (in
documentation sites, tooltips in editors, etc).

Because each of these conventions uses a microsyntax inside a string,
when editing those docstrings, editors can't easily provide support for
autocompletion, inline errors for broken syntax, etc. Any type of
editing support for these conventions would be on top of the support for
editing standard Python syntax.

When documenting parameters with current conventions, because the
docstring is in a different place in the code than the actual parameters
and it requires duplication of information (the parameter name) the
information about a parameter is easily in a place in the code quite far
away from the declaration of the actual parameter and it is disconnected
from it. This means it's easy to refactor a function, remove a
parameter, and forget to remove its docs. The same happens when adding a
new parameter: it's easy to forget to add the docstring for it.

And because of this same duplication of information (the parameter name)
editors and other tools need complex custom logic to check or ensure the
consistency of the parameters in the signature and in their docstring,
or they simply don't fully support that.

As these existing conventions are different types of microsyntaxes
inside of strings, robustly parsing them for rendering requires complex
logic that needs to be implemented by the tools supporting them.
Additionally, libraries and tools don't have a straightforward way to
obtain the documentation for each individual parameter or variable at
runtime, without depending on a specific docstring convention parser.
Accessing the parameter documentation strings at runtime would be
useful, for example, for testing the contents of each parameter's
documentation, to ensure consistency across several similar functions,
or to extract and expose that same parameter documentation in some other
way (e.g. an API with FastAPI, a CLI with Typer, etc).

Some of these previous formats tried to account for the lack of type
annotations in older Python versions by including typing information in
the docstrings (e.g. Sphinx, numpydoc) but now that information doesn't
need to be in docstrings as there is now an official
syntax for type annotations <484>.

Rationale

This proposal intends to address these shortcomings by extending and
complementing the information in docstrings, keeping backwards
compatibility with existing docstrings (it doesn't deprecate them), and
doing it in a way that leverages the Python language and structure, via
type annotations with ~typing.Annotated, and a new class Doc in typing.

The reason why this would belong in the standard Python library instead
of an external package is because although the implementation would be
quite trivial, the actual power and benefit from it would come from
being a standard, to facilitate its usage from library authors and to
provide a default way to document Python symbols using
~typing.Annotated. Some tool providers (at least VS Code and PyCharm)
have shown they would consider implementing support for this only if it
was a standard.

This doesn't deprecate current usage of docstrings, docstrings should be
considered the preferred documentation method when available (not
available in type aliases, parameters, etc). And docstrings would be
complemented by this proposal for documentation specific to the symbols
that can be declared with ~typing.Annotated (currently only covered by
the several available microsyntax conventions).

This should be relatively transparent to common developers (library
users) unless they manually open the source files from libraries
adopting it.

It should be considered opt-in for library authors who would like to
adopt it and they should be free to decide to use it or not.

It would be only useful for libraries that are willing to use optional
type hints.

Summary

Here's a short summary of the features of this proposal in contrast to
current conventions:

-   Editing would be already fully supported by default by any editor
    (current or future) supporting Python syntax, including syntax
    errors, syntax highlighting, etc.
-   Rendering would be relatively straightforward to implement by static
    tools (tools that don't need runtime execution), as the information
    can be extracted from the AST they normally already create.
-   Deduplication of information: the name of a parameter would be
    defined in a single place, not duplicated inside of a docstring.
-   Elimination of the possibility of having inconsistencies when
    removing a parameter or class variable and forgetting to remove its
    documentation.
-   Minimization of the probability of adding a new parameter or class
    variable and forgetting to add its documentation.
-   Elimination of the possibility of having inconsistencies between the
    name of a parameter in the signature and the name in the docstring
    when it is renamed.
-   Access to the documentation string for each symbol at runtime,
    including existing (older) Python versions.
-   A more formalized way to document other symbols, like type aliases,
    that could use ~typing.Annotated.
-   No microsyntax to learn for newcomers, it's just Python syntax.
-   Parameter documentation inheritance for functions captured by
    ~typing.ParamSpec.

Specification

The main proposal is to introduce a new class, typing.Doc. This class
should only be used within ~typing.Annotated annotations. It takes a
single positional-only string argument. It should be used to document
the intended meaning and use of the symbol declared using
~typing.Annotated.

For example:

    from typing import Annotated, Doc

    class User:
        name: Annotated[str, Doc("The user's name")]
        age: Annotated[int, Doc("The user's age")]

        ...

~typing.Annotated is normally used as a type annotation, in those cases,
any typing.Doc inside of it would document the symbol being annotated.

When ~typing.Annotated is used to declare a type alias, typing.Doc would
then document the type alias symbol.

For example:

    from typing import Annotated, Doc, TypeAlias

    from external_library import UserResolver

    CurrentUser: TypeAlias = Annotated[str, Doc("The current system user"), UserResolver()]

    def create_user(name: Annotated[str, Doc("The user's name")]): ...

    def delete_user(name: Annotated[str, Doc("The user to delete")]): ...

In this case, if a user imported CurrentUser, tools like editors could
provide a tooltip with the documentation string when a user hovers over
that symbol, or documentation tools could include the type alias with
its documentation in their generated output.

For tools extracting the information at runtime, they would normally use
~typing.get_type_hints with the parameter include_extras=True, and as
~typing.Annotated is normalized (even with type aliases), this would
mean they should use the last typing.Doc available, if more than one is
used, as that is the last one used.

At runtime, typing.Doc instances have an attribute documentation with
the string passed to it.

When a function's signature is captured by a ~typing.ParamSpec, any
documentation strings associated with the parameters should be retained.

Any tool processing typing.Doc objects should interpret the string as a
docstring, and therefore should normalize whitespace as if
inspect.cleandoc() were used.

The string passed to typing.Doc should be of the form that would be a
valid docstring. This means that f-strings and string operations should
not be used. As this cannot be enforced by the Python runtime, tools
should not rely on this behavior.

When tools providing rendering show the raw signature, they could allow
configuring if the whole raw ~typing.Annotated code should be displayed,
but they should default to not include ~typing.Annotated and its
internal code metadata, only the type of the symbols annotated. When
those tools support typing.Doc and rendering in other ways than just a
raw signature, they should show the string value passed to typing.Doc in
a convenient way that shows the relation between the documented symbol
and the documentation string.

Tools providing rendering could allow ways to configure where to show
the parameter documentation and the prose docstring in different ways.
Otherwise, they could simply show the prose docstring first and then the
parameter documentation second.

Examples

Class attributes may be documented:

    from typing import Annotated, Doc

    class User:
        name: Annotated[str, Doc("The user's name")]
        age: Annotated[int, Doc("The user's age")]

        ...

As can function or method parameters and return values:

    from typing import Annotated, Doc

    def create_user(
        name: Annotated[str, Doc("The user's name")],
        age: Annotated[int, Doc("The user's age")],
        cursor: DatabaseConnection | None = None,
    ) -> Annotated[User, Doc("The created user after saving in the database")]:
        """Create a new user in the system.

        It needs the database connection to be already initialized.
        """
        pass

Backwards Compatibility

This proposal is fully backwards compatible with existing code and it
doesn't deprecate existing usage of docstring conventions.

For developers that wish to adopt it before it is available in the
standard library, or to support older versions of Python, they can use
typing_extensions and import and use Doc from there.

For example:

    from typing import Annotated
    from typing_extensions import Doc

    class User:
        name: Annotated[str, Doc("The user's name")]
        age: Annotated[int, Doc("The user's age")]

        ...

Security Implications

There are no known security implications.

How to Teach This

The main mechanism of documentation should continue to be standard
docstrings for prose information, this applies to modules, classes,
functions and methods.

For authors that want to adopt this proposal to add more granularity,
they can use typing.Doc inside of ~typing.Annotated annotations for the
symbols that support it.

Library authors that wish to adopt this proposal while keeping backwards
compatibility with older versions of Python should use
typing_extensions.Doc instead of typing.Doc.

Reference Implementation

typing.Doc is implemented equivalently to:

    class Doc:
        def __init__(self, documentation: str, /):
            self.documentation = documentation

It has been implemented in the typing_extensions package.

Survey of Other languages

Here's a short survey of how other languages document their symbols.

Java

Java functions and their parameters are documented with Javadoc, a
special format for comments put on top of the function definition. This
would be similar to Python current docstring microsyntax conventions
(but only one).

For example:

    /**
    * Returns an Image object that can then be painted on the screen.
    * The url argument must specify an absolute {@link URL}. The name
    * argument is a specifier that is relative to the url argument.
    *

    * This method always returns immediately, whether or not the
    * image exists. When this applet attempts to draw the image on
    * the screen, the data will be loaded. The graphics primitives
    * that draw the image will incrementally paint on the screen.
    *
    * @param  url  an absolute URL giving the base location of the image
    * @param  name the location of the image, relative to the url argument
    * @return      the image at the specified URL
    * @see         Image
    */
    public Image getImage(URL url, String name) {
      try {
        return getImage(new URL(url, name));
      } catch (MalformedURLException e) {
        return null;
      }
    }

JavaScript

Both JavaScript and TypeScript use a similar system to Javadoc.

JavaScript uses JSDoc.

For example:

    /**
    * Represents a book.
    * @constructor
    * @param {string} title - The title of the book.
    * @param {string} author - The author of the book.
    */
    function Book(title, author) {
    }

TypeScript

TypeScript has its own JSDoc reference with some variations.

For example:

    // Parameters may be declared in a variety of syntactic forms
    /**
    * @param {string}  p1 - A string param.
    * @param {string=} p2 - An optional param (Google Closure syntax)
    * @param {string} [p3] - Another optional param (JSDoc syntax).
    * @param {string} [p4="test"] - An optional param with a default value
    * @returns {string} This is the result
    */
    function stringsStringStrings(p1, p2, p3, p4) {
        // TODO
    }

Rust

Rust uses another similar variation of a microsyntax in Doc comments.

But it doesn't have a particular well defined microsyntax structure to
denote what documentation refers to what symbol/parameter other than
what can be inferred from the pure Markdown.

For example:

    #![crate_name = "doc"]

    /// A human being is represented here
    pub struct Person {
       /// A person must have a name, no matter how much Juliet may hate it
       name: String,
    }

    impl Person {
       /// Returns a person with the name given them
       ///
       /// # Arguments
       ///
       /// * `name` - A string slice that holds the name of the person
       ///
       /// # Examples
       ///
       /// ```
       /// // You can have rust code between fences inside the comments
       /// // If you pass --test to `rustdoc`, it will even test it for you!
       /// use doc::Person;
       /// let person = Person::new("name");
       /// ```
       pub fn new(name: &str) -> Person {
          Person {
                name: name.to_string(),
          }
       }

       /// Gives a friendly hello!
       ///
       /// Says "Hello, [name](Person::name)" to the `Person` it is called on.
       pub fn hello(& self) {
          println!("Hello, {}!", self.name);
       }
    }

    fn main() {
       let john = Person::new("John");

       john.hello();
    }

Go Lang

Go also uses a form of Doc Comments.

It doesn't have a well defined microsyntax structure to denote what
documentation refers to which symbol/parameter, but parameters can be
referenced by name without any special syntax or marker, this also means
that ordinary words that could appear in the documentation text should
be avoided as parameter names.

    package strconv

    // Quote returns a double-quoted Go string literal representing s.
    // The returned string uses Go escape sequences (\t, \n, \xFF, \u0100)
    // for control characters and non-printable characters as defined by IsPrint.
    func Quote(s string) string {
       ...
    }

Rejected Ideas

Standardize Current Docstrings

A possible alternative would be to support and try to push as a standard
one of the existing docstring formats. But that would only solve the
standardization.

It wouldn't solve any of the other problems derived from using a
microsyntax inside of a docstring instead of pure Python syntax, the
same as described above in the Rationale - Summary.

Extra Metadata and Decorator

Some ideas before this proposal included having a function doc() instead
of the single class Doc with several parameters to indicate whether an
object is discouraged from use, what exceptions it may raise, etc. To
allow also deprecating functions and classes, it was also expected that
doc() could be used as a decorator. But this functionality is covered by
typing.deprecated() in PEP 702, so it was dropped from this proposal.

A way to declare additional information could still be useful in the
future, but taking early feedback on this idea, all that was postponed
to future proposals.

This also shifted the focus from an all-encompassing function doc() with
multiple parameters to a single Doc class to be used in
~typing.Annotated in a way that could be composed with other future
proposals.

This design change also allows better interoperability with other
proposals like typing.deprecated(), as in the future it could be
considered to allow having typing.deprecated() also in ~typing.Annotated
to deprecate individual parameters, coexisting with Doc.

String Under Definition

A proposed alternative in the discussion is declaring a string under the
definition of a symbol and providing runtime access to those values:

    class User:
        name: str
        "The user's name"
        age: int
        "The user's age"

        ...

This was already proposed and rejected in PEP 224, mainly due to the
ambiguity of how is the string connected with the symbol it's
documenting.

Additionally, there would be no way to provide runtime access to this
value in previous versions of Python.

Plain String in Annotated

In the discussion, it was also suggested to use a plain string inside of
~typing.Annotated:

    from typing import Annotated

    class User:
        name: Annotated[str, "The user's name"]
        age: Annotated[int, "The user's age"]

        ...

But this would create a predefined meaning for any plain string inside
of ~typing.Annotated, and any tool that was using plain strings in them
for any other purpose, which is currently allowed, would now be invalid.

Having an explicit typing.Doc makes it compatible with current valid
uses of ~typing.Annotated.

Another Annotated-Like Type

In the discussion it was suggested to define a new type similar to
~typing.Annotated, it would take the type and a parameter with the
documentation string:

    from typing import Doc

    class User:
        name: Doc[str, "The user's name"]
        age: Doc[int, "The user's age"]

        ...

This idea was rejected as it would only support that use case and would
make it more difficult to combine it with ~typing.Annotated for other
purposes ( e.g. with FastAPI metadata, Pydantic fields, etc.) or adding
additional metadata apart from the documentation string (e.g.
deprecation).

Transferring Documentation from Type aliases

A previous version of this proposal specified that when type aliases
declared with ~typing.Annotated were used, and these type aliases were
used in annotations, the documentation string would be transferred to
the annotated symbol.

For example:

    from typing import Annotated, Doc, TypeAlias

    UserName: TypeAlias = Annotated[str, Doc("The user's name")]

    def create_user(name: UserName): ...

    def delete_user(name: UserName): ...

This was rejected after receiving feedback from the maintainer of one of
the main components used to provide editor support.

Shorthand with Slices

In the discussion, it was suggested to use a shorthand with slices:

    is_approved: Annotated[str: "The status of a PEP."]

Although this is a very clever idea and would remove the need for a new
Doc class, runtime executing of current versions of Python don't allow
it.

At runtime, ~typing.Annotated requires at least two arguments, and it
requires the first argument to be type, it crashes if it is a slice.

Open Issues

Verbosity

The main argument against this would be the increased verbosity.

If the signature was not viewed independently of the documentation and
the body of the function with the docstring was also measured, the total
verbosity would be somewhat similar, as what this proposal does is to
move some of the contents from the docstring in the body to the
signature.

Considering the signature alone, without the body, they could be much
longer than they currently are, they could end up being more than one
page long. In exchange, the equivalent docstrings that currently are
more than one page long would be much shorter.

When comparing the total verbosity, including the signature and the
docstring, the main additional verbosity added by this would be from
using ~typing.Annotated and typing.Doc. If ~typing.Annotated had more
usage, it could make sense to have an improved shorter syntax for it and
for the type of metadata it would carry. But that would only make sense
once ~typing.Annotated is more widely used.

On the other hand, this verbosity would not affect end users as they
would not see the internal code using typing.Doc. The majority of users
would interact with libraries through editors without looking at the
internals, and if anything, they would have tooltips from editors
supporting this proposal.

The cost of dealing with the additional verbosity would mainly be
carried by those library maintainers that use this feature.

This argument could be analogous to the argument against type
annotations in general, as they do indeed increase verbosity, in
exchange for their features. But again, as with type annotations, this
would be optional and only to be used by those that are willing to take
the extra verbosity in exchange for the benefits.

Of course, more advanced users might want to look at the source code of
the libraries and if the authors of those libraries adopted this, those
advanced users would end up having to look at that code with additional
signature verbosity instead of docstring verbosity.

Any authors that decide not to adopt it should be free to continue using
docstrings with any particular format they decide, no docstrings at all,
etc.

Still, there's a high chance that library authors could receive pressure
to adopt this if it became the blessed solution.

Documentation is not Typing

It could also be argued that documentation is not really part of typing,
or that it should live in a different module. Or that this information
should not be part of the signature but live in another place (like the
docstring).

Nevertheless, type annotations in Python could already be considered, by
default, additional metadata: they carry additional information about
variables, parameters, return types, and by default they don't have any
runtime behavior. And this proposal would add one more type of metadata
to them.

It could be argued that this proposal extends the type of information
that type annotations carry, the same way as PEP 702 extends them to
include deprecation information.

~typing.Annotated was added to the standard library precisely to support
adding additional metadata to the annotations, and as the new proposed
Doc class is tightly coupled to ~typing.Annotated, it makes sense for it
to live in the same module. If ~typing.Annotated was moved to another
module, it would make sense to move Doc with it.

Multiple Standards

Another argument against this would be that it would create another
standard, and that there are already several conventions for docstrings.
It could seem better to formalize one of the currently existing
standards.

Nevertheless, as stated above, none of those conventions cover the
general drawbacks of a doctsring-based approach that this proposal
solves naturally.

To see a list of the drawbacks of a docstring-based approach, see the
section above in the Rationale - Summary.

In the same way, it can be seen that, in many cases, a new standard that
takes advantage of new features and solves several problems from
previous methods can be worth having. As is the case with the new
pyproject.toml, dataclass_transform, the new typing pipe/union (|)
operator, and other cases.

Adoption

As this is a new standard proposal, it would only make sense if it had
interest from the community.

Fortunately there's already interest from several mainstream libraries
from several developers and teams, including FastAPI, Typer, SQLModel,
Asyncer (from the author of this proposal), Pydantic, Strawberry
(GraphQL), and others.

There's also interest and support from documentation tools, like
mkdocstrings, which added support even for an earlier version of this
proposal.

All the CPython core developers contacted for early feedback (at least
4) have shown interest and support for this proposal.

Editor developers (VS Code and PyCharm) have shown some interest, while
showing concerns about the signature verbosity of the proposal, although
not about the implementation (which is what would affect them the most).
And they have shown they would consider adding support for this if it
were to become an official standard. In that case, they would only need
to add support for rendering, as support for editing, which is normally
non-existing for other standards, is already there, as they already
support editing standard Python syntax.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 218 Title: Adding a Built-In Set Object Type Author: Greg Wilson
<EMAIL_ADDRESS>Raymond Hettinger<EMAIL_ADDRESS>Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 31-Jul-2000
Python-Version: 2.2 Post-History:

Introduction

This PEP proposes adding a Set module to the standard Python library,
and to then make sets a built-in Python type if that module is widely
used. After explaining why sets are desirable, and why the common idiom
of using dictionaries in their place is inadequate, we describe how we
intend built-in sets to work, and then how the preliminary Set module
will behave. The last section discusses the mutability (or otherwise) of
sets and set elements, and the solution which the Set module will
implement.

Rationale

Sets are a fundamental mathematical structure, and are very commonly
used in algorithm specifications. They are much less frequently used in
implementations, even when they are the "right" structure. Programmers
frequently use lists instead, even when the ordering information in
lists is irrelevant, and by-value lookups are frequent. (Most
medium-sized C programs contain a depressing number of start-to-end
searches through malloc'd vectors to determine whether particular items
are present or not...)

Programmers are often told that they can implement sets as dictionaries
with "don't care" values. Items can be added to these "sets" by
assigning the "don't care" value to them; membership can be tested using
dict.has_key; and items can be deleted using del. However, the other
main operations on sets (union, intersection, and difference) are not
directly supported by this representation, since their meaning is
ambiguous for dictionaries containing key/value pairs.

Proposal

The long-term goal of this PEP is to add a built-in set type to Python.
This type will be an unordered collection of unique values, just as a
dictionary is an unordered collection of key/value pairs.

Iteration and comprehension will be implemented in the obvious ways, so
that:

    for x in S:

will step through the elements of S in arbitrary order, while:

    set(x**2 for x in S)

will produce a set containing the squares of all elements in S,
Membership will be tested using in and not in, and basic set operations
will be implemented by a mixture of overloaded operators:

  ----------- -------------------------------
  |           union
  &           intersection
  ^           symmetric difference
  -           asymmetric difference
  == !=       equality and inequality tests
  < <= >= >   subset and superset tests
  ----------- -------------------------------

and methods:

  ---------------- ----------------------------------------------------------------------------------------------
  S.add(x)         Add "x" to the set.
  S.update(s)      Add all elements of sequence "s" to the set.
  S.remove(x)      Remove "x" from the set. If "x" is not present, this method raises a LookupError exception.
  S.discard(x)     Remove "x" from the set if it is present, or do nothing if it is not.
  S.pop()          Remove and return an arbitrary element, raising a LookupError if the element is not present.
  S.clear()        Remove all elements from this set.
  S.copy()         Make a new set.
  s.issuperset()   Check for a superset relationship.
  s.issubset()     Check for a subset relationship.
  ---------------- ----------------------------------------------------------------------------------------------

and two new built-in conversion functions:

  -------------- ------------------------------------------------------------------------
  set(x)         Create a set containing the elements of the collection "x".
  frozenset(x)   Create an immutable set containing the elements of the collection "x".
  -------------- ------------------------------------------------------------------------

Notes:

1.  We propose using the bitwise operators "|&" for intersection and
    union. While "+" for union would be intuitive, "*" for intersection
    is not (very few of the people asked guessed what it did correctly).
2.  We considered using "+" to add elements to a set, rather than "add".
    However, Guido van Rossum pointed out that "+" is symmetric for
    other built-in types (although "*" is not). Use of "add" will also
    avoid confusion between that operation and set union.

Set Notation

The PEP originally proposed {1,2,3} as the set notation and {-} for the
empty set. Experience with Python 2.3's sets.py showed that the notation
was not necessary. Also, there was some risk of making dictionaries less
instantly recognizable.

It was also contemplated that the braced notation would support set
comprehensions; however, Python 2.4 provided generator expressions which
fully met that need and did so it a more general way. (See PEP 289 for
details on generator expressions).

So, Guido ruled that there would not be a set syntax; however, the issue
could be revisited for Python 3000 (see PEP 3000).

History

To gain experience with sets, a pure python module was introduced in
Python 2.3. Based on that implementation, the set and frozenset types
were introduced in Python 2.4. The improvements are:

-   Better hash algorithm for frozensets
-   More compact pickle format (storing only an element list instead of
    a dictionary of key:value pairs where the value is always True).
-   Use a __reduce__ function so that deep copying is automatic.
-   The BaseSet concept was eliminated.
-   The union_update() method became just update().
-   Auto-conversion between mutable and immutable sets was dropped.
-   The _repr method was dropped (the need is met by the new sorted()
    built-in function).

Tim Peters believes that the class's constructor should take a single
sequence as an argument, and populate the set with that sequence's
elements. His argument is that in most cases, programmers will be
creating sets from pre-existing sequences, so that this case should be
the common one. However, this would require users to remember an extra
set of parentheses when initializing a set with known values:

    >>> Set((1, 2, 3, 4))       # case 1

On the other hand, feedback from a small number of novice Python users
(all of whom were very experienced with other languages) indicates that
people will find a "parenthesis-free" syntax more natural:

    >>> Set(1, 2, 3, 4)         # case 2

Ultimately, we adopted the first strategy in which the initializer takes
a single iterable argument.

Mutability

The most difficult question to resolve in this proposal was whether sets
ought to be able to contain mutable elements. A dictionary's keys must
be immutable in order to support fast, reliable lookup. While it would
be easy to require set elements to be immutable, this would preclude
sets of sets (which are widely used in graph algorithms and other
applications).

Earlier drafts of PEP 218 had only a single set type, but the sets.py
implementation in Python 2.3 has two, Set and ImmutableSet. For Python
2.4, the new built-in types were named set and frozenset which are
slightly less cumbersome.

There are two classes implemented in the "sets" module. Instances of the
Set class can be modified by the addition or removal of elements, and
the ImmutableSet class is "frozen", with an unchangeable collection of
elements. Therefore, an ImmutableSet may be used as a dictionary key or
as a set element, but cannot be updated. Both types of set require that
their elements are immutable, hashable objects. Parallel comments apply
to the "set" and "frozenset" built-in types.

Copyright

This document has been placed in the Public Domain.

license: Public Domain

authors:
Greg Wilson
Raymond Hettinger


text:
PEP: 3149 Title: ABI version tagged .so files Version: $Revision$
Last-Modified: $Date$ Author: Barry Warsaw<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
09-Jul-2010 Python-Version: 3.2 Post-History: 14-Jul-2010, 22-Jul-2010
Resolution:
https://mail.python.org/pipermail/python-dev/2010-September/103408.html

Abstract

PEP 3147 described an extension to Python's import machinery that
improved the sharing of Python source code, by allowing more than one
byte compilation file (.pyc) to be co-located with each source file.

This PEP defines an adjunct feature which allows the co-location of
extension module files (.so) in a similar manner. This optional,
build-time feature will enable downstream distributions of Python to
more easily provide more than one Python major version at a time.

Background

PEP 3147 defined the file system layout for a pure-Python package, where
multiple versions of Python are available on the system. For example,
where the alpha package containing source modules one.py and two.py
exist on a system with Python 3.2 and 3.3, the post-byte compilation
file system layout would be:

    alpha/
        __pycache__/
            __init__.cpython-32.pyc
            __init__.cpython-33.pyc
            one.cpython-32.pyc
            one.cpython-33.pyc
            two.cpython-32.pyc
            two.cpython-33.pyc
        __init__.py
        one.py
        two.py

For packages with extension modules, a similar differentiation is needed
for the module's .so files. Extension modules compiled for different
Python major versions are incompatible with each other due to changes in
the ABI. Different configuration/compilation options for the same Python
version can result in different ABIs (e.g. --with-wide-unicode).

While PEP 384 defines a stable ABI, it will minimize, but not eliminate
extension module incompatibilities between Python builds or major
versions. Thus a mechanism for discriminating extension module file
names is proposed.

Rationale

Linux distributions such as Ubuntu[1] and Debian[2] provide more than
one Python version at the same time to their users. For example, Ubuntu
9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with
Python 2.6 being the default.

In order to share as much as possible between the available Python
versions, these distributions install third party package modules (.pyc
and .so files) into /usr/share/pyshared and symlink to them from
/usr/lib/pythonX.Y/dist-packages. The symlinks exist because in a
pre-PEP 3147 world (i.e < Python 3.2), the .pyc files resulting from
byte compilation by the various installed Pythons will name collide with
each other. For Python versions >= 3.2, all pure-Python packages can be
shared, because the .pyc files will no longer cause file system naming
conflicts. Eliminating these symlinks makes for a simpler, more robust
Python distribution.

A similar situation arises with shared library extensions. Because
extension modules are typically named foo.so for a foo extension module,
these would also name collide if foo was provided for more than one
Python version.

In addition, because different configuration/compilation options for the
same Python version can cause different ABIs to be presented to
extension modules. On POSIX systems for example, the configure options
--with-pydebug, --with-pymalloc, and --with-wide-unicode all change the
ABI. This PEP proposes to encode build-time options in the file name of
the .so extension module files.

PyPy[3] can also benefit from this PEP, allowing it to avoid name
collisions in extension modules built for its API, but with a different
.so tag.

Proposal

The configure/compilation options chosen at Python interpreter
build-time will be encoded in the shared library file name for extension
modules. This "tag" will appear between the module base name and the
operation file system extension for shared libraries.

The following information MUST be included in the shared library file
name:

-   The Python implementation (e.g. cpython, pypy, jython, etc.)
-   The interpreter's major and minor version numbers

These two fields are separated by a hyphen and no dots are to appear
between the major and minor version numbers. E.g. cpython-32.

Python implementations MAY include additional flags in the file name tag
as appropriate. For example, on POSIX systems these flags will also
contribute to the file name:

-   --with-pydebug (flag: d)
-   --with-pymalloc (flag: m)
-   --with-wide-unicode (flag: u)

By default in Python 3.2, configure enables --with-pymalloc so shared
library file names would appear as foo.cpython-32m.so. When the other
two flags are also enabled, the file names would be
foo.cpython-32dmu.so.

The shared library file name tag is used unconditionally; it cannot be
changed. The tag and extension module suffix are available through the
sysconfig modules via the following variables:

    >>> sysconfig.get_config_var('EXT_SUFFIX')
    '.cpython-32mu.so'
    >>> sysconfig.get_config_var('SOABI')
    'cpython-32mu'

Note that $SOABI contains just the tag, while $EXT_SUFFIX includes the
platform extension for shared library files, and is the exact suffix
added to the extension module name.

For an arbitrary package foo, you might see these files when the
distribution package was installed:

    /usr/lib/python/foo.cpython-32m.so
    /usr/lib/python/foo.cpython-33m.so

(These paths are for example purposes only. Distributions are free to
use whatever filesystem layout they choose, and nothing in this PEP
changes the locations where from-source builds of Python are installed.)

Python's dynamic module loader will recognize and import shared library
extension modules with a tag that matches its build-time options. For
backward compatibility, Python will also continue to import untagged
extension modules, e.g. foo.so.

This shared library tag would be used globally for all distutils-based
extension modules, regardless of where on the file system they are
built. Extension modules built by means other than distutils would
either have to calculate the tag manually, or fallback to the non-tagged
.so file name.

Proven approach

The approach described here is already proven, in a sense, on Debian and
Ubuntu system where different extensions are used for debug builds of
Python and extension modules. Debug builds on Windows also already use a
different file extension for dynamic libraries, and in fact encoded (in
a different way than proposed in this PEP) the Python major and minor
version in the .dll file name.

Windows

This PEP only addresses build issues on POSIX systems that use the
configure script. While Windows or other platform support is not
explicitly disallowed under this PEP, platform expertise is needed in
order to evaluate, describe, and implement support on such platforms. It
is not currently clear that the facilities in this PEP are even useful
for Windows.

PEP 384

PEP 384 defines a stable ABI for extension modules. In theory, universal
adoption of PEP 384 would eliminate the need for this PEP because all
extension modules could be compatible with any Python version. In
practice of course, it will be impossible to achieve universal adoption,
and as described above, different build-time flags still affect the ABI.
Thus even with a stable ABI, this PEP may still be necessary. While a
complete specification is reserved for PEP 384, here is a discussion of
the relevant issues.

PEP 384 describes a change to PyModule_Create() where 3 is passed as the
API version if the extension was compiled with Py_LIMITED_API. This
should be formalized into an official macro called PYTHON_ABI_VERSION to
mirror PYTHON_API_VERSION. If and when the ABI changes in an
incompatible way, this version number would be bumped. To facilitate
sharing, Python would be extended to search for extension modules with
the PYTHON_ABI_VERSION number in its name. The prefix abi is reserved
for Python's use.

Thus, an initial implementation of PEP 384, when Python is configured
with the default set of flags, would search for the following file names
when extension module foo is imported (in this order):

    foo.cpython-XYm.so
    foo.abi3.so
    foo.so

The distutils[4] build_ext command would also have to be extended to
compile to shared library files with the abi3 tag, when the module
author indicates that their extension supports that version of the ABI.
This could be done in a backward compatible way by adding a keyword
argument to the Extension class, such as:

    Extension('foo', ['foo.c'], abi=3)

Martin v. Löwis describes his thoughts[5] about the applicability of
this PEP to PEP 384. In summary:

-   --with-pydebug would not be supported by the stable ABI because this
    changes the layout of PyObject, which is an exposed structure.
-   --with-pymalloc has no bearing on the issue.
-   --with-wide-unicode is trickier, though Martin's inclination is to
    force the stable ABI to use a Py_UNICODE that matches the platform's
    wchar_t.

Alternatives

In the initial python-dev thread[6] where this idea was first
introduced, several alternatives were suggested. For completeness they
are listed here, along with the reasons for not adopting them.

Independent directories or symlinks

Debian and Ubuntu could simply add a version-specific directory to
sys.path that would contain just the extension modules for that version
of Python. Or the symlink trick eliminated in PEP 3147 could be retained
for just shared libraries. This approach is rejected because it
propagates the essential complexity that PEP 3147 tries to avoid, and
adds potentially several additional directories to search for all
modules, even when the number of extension modules is much fewer than
the total number of Python packages. For example, builds were made
available both with and without wide unicode, with and without pydebug,
and with and without pymalloc, the total number of directories search
increases substantially.

Don't share packages with extension modules

It has been suggested that Python packages with extension modules not be
shared among all supported Python versions on a distribution. Even with
adoption of PEP 3149, extension modules will have to be compiled for
every supported Python version, so perhaps sharing of such packages
isn't useful anyway. Not sharing packages with extensions though is
infeasible for several reasons.

If a pure-Python package is shared in one version, should it suddenly be
not-shared if the next release adds an extension module for speed? Also,
even though all extension shared libraries will be compiled and
distributed once for every supported Python, there's a big difference
between duplicating the .so files and duplicating all .py files. The
extra size increases the download time for such packages, and more
immediately, increases the space pressures on already constrained
distribution CD-ROMs.

Reference implementation

Work on this code is tracked in a Bazaar branch on Launchpad[7] until
it's ready for merge into Python 3.2. The work-in-progress diff can also
be viewed[8] and is updated automatically as new changes are uploaded.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Ubuntu: <http://www.ubuntu.com>

[2] Debian: <http://www.debian.org>

[3] http://codespeak.net/pypy/dist/pypy/doc/

[4] http://docs.python.org/py3k/distutils/index.html

[5] https://mail.python.org/pipermail/python-dev/2010-August/103330.html

[6] https://mail.python.org/pipermail/python-dev/2010-June/100998.html

[7] https://code.edge.launchpad.net/~barry/python/sovers

[8] https://code.edge.launchpad.net/~barry/python/sovers/+merge/29411


text:
PEP: 8010 Title: The Technical Leader Governance Model Author: Barry
Warsaw<EMAIL_ADDRESS>Status: Rejected Type: Informational Topic:
Governance Content-Type: text/x-rst Created: 24-Aug-2018

Abstract

This PEP proposes a continuation of the singular technical project
leader model, euphemistically called the Benevolent Dictator For Life
(BDFL) model of Python governance, to be henceforth called in this PEP
the Gracious Umpire Influencing Decisions Officer (GUIDO). This change
in name reflects both the expanded view of the GUIDO as final arbiter
for the Python language decision making process in consultation with the
wider development community, and the recognition that "for life" while
perhaps aspirational, is not necessarily in the best interest of the
well-being of either the language or the GUIDO themselves.

This PEP describes:

-   The rationale for maintaining the singular technical leader model
-   The process for how the GUIDO will be selected, elected, retained,
    recalled, and succeeded;
-   The roles of the GUIDO in the Python language evolution process;
-   The term length of service;
-   The relationship of the GUIDO with a Council of Pythonistas (CoP)
    that advise the GUIDO on technical matters;
-   The size, election, and roles of the CoP;
-   The decision delegation process;
-   Any changes to the PEP process to fit the new governance model;

This PEP does not name a new BDFL. Should this model be adopted, it will
be codified in PEP 13 along with the names of all officeholders
described in this PEP.

PEP Rejection

PEP 8010 was rejected by a core developer vote described in PEP 8001 on
Monday, December 17, 2018.

PEP 8016 and the governance model it describes were chosen instead.

Open discussion points

Various tweaks to the parameters of this PEP are allowed during the
governance discussion process, such as the exact size of the CoP, term
lengths of service, and voting procedures. These will be codified by the
time the PEP is ready to be voted on.

The voting procedures and events described in this PEP will default to
the voting method specified in PEP 8001, although as that PEP is still
in discussion at the time of this writing, this is subject to change.

It is allowed, and perhaps even expected, that as experience is gained
with this model, these parameters may be tweaked as future GUIDOs are
named, in order to provide for a smoother governing process.

Why a singular technical leader?

Why this model rather than any other? It comes down to "vision". Design
by committee has many known downsides, leading to a language that
accretes new features based on the varied interests of the contributors
at the time. A famous aphorism is "a camel is a horse designed by
committee". Can a language that is designed by committee "hang
together"? Does it feel like a coherent, self-consistent language where
the rules make sense and are easily remembered?

A singular technical leader can promote that vision more than a
committee can, whether that committee is small (e.g. 3 or 5 persons) or
spans the entire Python community. Every participant will have their own
vision of what "Python" is, and this can lead to indecision or illogical
choices when those individual visions are in conflict. Should CPython be
3x faster or should we preserve the C API? That's a very difficult
question to get consensus on, since neither choice is right or wrong.
But worse than making the wrong decision might be accepting the status
quo because no consensus could be found.

Flexibility

Degrees of flexibility are given to both the GUIDO and CoP by way of
underspecification. This PEP describes how conflicts will be resolved,
but expects all participants, including core developers, community
members, and office holders, to always have the best interest of Python
and its users at heart. The PEP assumes that mutual respect and the best
intentions will always lead to consensus, and that the Code of Conduct
governs all interactions and discussions.

The role of the GUIDO

One of the most important roles of the GUIDO is to provide an
overarching, broad, coherent vision for the evolution of the Python
language, spanning multiple releases. This is especially important when
decision have lasting impact and competing benefits. For example, if
backward incompatible changes to the C API leads to a 2x improvement in
Python performance, different community members will likely advocate
convincingly on both sides of the debate, and a clear consensus may not
emerge. Either choice is equally valid. In consultation with the CoP, it
will be the GUIDO's vision that guides the ultimate decision.

The GUIDO is the ultimate authority for decisions on PEPs and other
issues, including whether any particular change is PEP-worthy. As is the
case today, many --in fact perhaps most-- decisions are handled by
discussion and resolution on the issue tracker, merge requests, and
discussion forums, usually with input or lead by experts in the
particular field. Where this operating procedure works perfectly well,
it can continue unchanged. This also helps reduce the workload on the
CoP and GUIDO, leaving only the most important decisions and broadest
view of the landscape to the central authority.

Similarly, should a particular change be deemed to require a PEP, but
the GUIDO, in consultation with the CoP, identifies experts that have
the full confidence to make the final decision, the GUIDO can name a
Delegate for the PEP. While the GUIDO remains the ultimate authority, it
is expected that the GUIDO will not undermine, and in fact will support
the authority of the Delegate as the final arbiter of the PEP.

The GUIDO has full authority to shut down unproductive discussions,
ideas, and proposals, when it is clear that the proposal runs counter to
the long-term vision for Python. This is done with compassion for the
advocates of the change, but with the health and well-being of all
community members in mind. A toxic discussion on a dead-end proposal
does no one any good, and they can be terminated by fiat.

To sum up: the GUIDO has the authority to make a final pronouncement on
any topic, technical or non-technical, except for changing to the
governance PEP itself.

Authority comes from the community

The GUIDO's authority ultimately resides with the community. A rogue
GUIDO that loses the confidence of the majority of the community can be
recalled and a new vote conducted. This is an exceedingly rare and
unlikely event. This is a sufficient stopgap for the worst-case
scenario, so it should not be undertaken lightly. The GUIDO should not
fear being deposed because of one decision, even if that decision isn't
favored by the majority of Python developers. Recall should be reserved
for actions severely detrimental to the Python language or community.

The Council of Pythonistas (see below) has the responsibility to
initiate a vote of no-confidence.

Length of service and term limits

The GUIDO shall serve for three Python releases, approximately 4.5 years
given the current release cadence. If Python’s release cadence changes,
the length of GUIDO’s term should change to 4.5 years rounded to whole
releases. How the rounding is done is left to the potential release
cadence PEP. After this time, a new election is held according to the
procedures outlined below. There are no term limits, so the GUIDO may
run for re-election for as long as they like.

We expect GUIDOs to serve out their entire term of office, but of
course, Life Happens. Should the GUIDO need to step down before their
term ends, the vacancy will be filled by the process outlined below as
per choosing a new GUIDO. However, the new GUIDO will only serve for the
remainder of the original GUIDO's term, at which time a new election is
conducted. The GUIDO stepping down may continue to serve until their
replacement is selected.

During the transition period, the CoP (see below) may carry out the
GUIDO's duties, however they may also prefer to leave substantive
decisions (such as technical PEP approvals) to the incoming GUIDO.

Choosing a GUIDO

The selection process is triggered whenever a vacancy exists for a new
GUIDO, or when the GUIDO is up for re-election in the normal course of
events. When the selection process is triggered, either by the GUIDO
stepping down, or two months before the end of the GUIDO's regular term,
a new election process begins.

For three weeks prior to the vote, nominations are open. Candidates must
be chosen from the current list of core Python developers. Non-core
developers are ineligible to serve as the GUIDO. Candidates may
self-nominate, but all nominations must be seconded. Nominations and
seconds are conducted as merge requests on a private repository.

Once they accept their nomination, nominees may post short position
statements using the same private repository, and may also post them to
the committers discussion forum. Maybe we'll even have debates! This
phase of the election runs for two weeks.

Core developers then have three weeks to vote, using the process
described in PEP 8001.

The Council of Pythonistas (CoP)

Assisting the GUIDO is a small team of elected Python experts. They
serve on a team of technical committee members. They provide insight and
offer discussion of the choices before the GUIDO. Consultation can be
triggered from either side. For example, if the GUIDO is still undecided
about any particular choice, discussions with the CoP can help clarify
the remaining issues, identify the right questions to ask, and provide
insight into the impact on other users of Python that the GUIDO may not
be as familiar with. The CoP are the GUIDO's trusted advisers, and a
close working relationship is expected.

The CoP shall consist of 3 members, elected from among the core
developers. Their term runs for 3 years and members may run for
re-election as many times as they want. To ensure continuity, CoP
members are elected on a rotating basis; every year, one CoP member is
up for re-election.

In order to bootstrap the stagger for the initial election, the CoP
member with the most votes shall serve for 3 years, the second most
popular vote getter shall serve for 2 years, and CoP member with the
least number of votes shall serve initially for 1 year.

All ties in voting will be broken with a procedure to be determined in
PEP 8001.

The nomination and voting process is similar as with the GUIDO. There is
a three-week nomination period, where self-nominations are allowed and
must be seconded, followed by a period of time for posting position
statements, followed by a vote.

By unanimous decision, the CoP may begin a no-confidence vote on the
GUIDO, triggering the procedure in that section.

No confidence votes

As mentioned above, the CoP may, by unanimous decision, initiate a vote
of no-confidence in the GUIDO. This process should not be undertaken
lightly, but once begun, it triggers up to two votes. In both cases,
voting is done by the same procedure as in PEP 8001, and all core
developers may participate in no confidence votes.

The first vote is whether to recall the current GUIDO or not. Should a
super majority of Python developers vote "no confidence", the GUIDO is
recalled. A second vote is then conducted to select the new GUIDO, in
accordance with the procedures for initial section of this office
holder. During the time in which there is no GUIDO, major decisions are
put on hold, but normal Python operations may of course continue.

Day-to-day operations

The GUIDO is not needed for all -- or even most -- decisions. Python
developers already have plenty of opportunity for delegation,
responsibility, and self-direction. The issue tracker and pull requests
serve exactly the same function as they did before this governance model
was chosen. Most discussions of bug fixes and minor improvements can
just happen on these forums, as they always have.

PEP considerations

The GUIDO, members of the CoP, and anyone else in the Python community
may propose a PEP. Treatment of the prospective PEP is handled the same
regardless of the author of the PEP.

However, in the case of the GUIDO authoring a PEP, an impartial PEP
Delegate should be selected, and given the authority to accept or reject
the PEP. The GUIDO should recuse themselves from the decision making
process. In the case of controversial PEPs where a clear consensus does
not arrive, ultimate authority on PEPs authored by the GUIDO rests with
the CoP.

The PEP propose is further enhanced such that a core developer must
always be chose as the PEP Shepherd. This person ensure that proper
procedure is maintained. The Shepherd must be chosen from among the core
developers. This means that while anyone can author a PEP, all PEPs must
have some level of sponsorship from at least one core developer.

Version History

Version 2

  -   Renamed to "The Technical Leader Governance Model"
  -   "singular leader" -> "singular technical leader"
  -   The adoption of PEP 8001 voting procedures is tentative until that
      PEP is approved
  -   Describe what happens if the GUIDO steps down
  -   Recall votes require a super majority of core devs to succeed

Copyright

This document has been placed in the public domain.


text:
PEP: 11 Title: CPython platform support Author: Martin von Löwis
<EMAIL_ADDRESS>Brett Cannon<EMAIL_ADDRESS>Status: Active
Type: Process Content-Type: text/x-rst Created: 07-Jul-2002
Post-History: 18-Aug-2007, 14-May-2014, 20-Feb-2015, 10-Mar-2022,

Abstract

This PEP documents how an operating system (platform) becomes supported
in CPython, what platforms are currently supported, and documents past
support.

Rationale

Over time, the CPython source code has collected various pieces of
platform-specific code, which, at some point in time, was considered
necessary to use CPython on a specific platform. Without access to this
platform, it is not possible to determine whether this code is still
needed. As a result, this code may either break during CPython's
evolution, or it may become unnecessary as the platforms evolve as well.

Allowing these fragments to grow poses the risk of unmaintainability:
without having experts for a large number of platforms, it is not
possible to determine whether a certain change to the CPython source
code will work on all supported platforms.

To reduce this risk, this PEP specifies what is required for a platform
to be considered supported by CPython as well as providing a procedure
to remove code for platforms with few or no CPython users.

This PEP also lists what platforms are supported by the CPython
interpreter. This lets people know what platforms are directly supported
by the CPython development team.

Support tiers

Platform support is broken down into tiers. Each tier comes with
different requirements which lead to different promises being made about
support.

To be promoted to a tier, steering council support is required and is
expected to be driven by team consensus. Demotion to a lower tier occurs
when the requirements of the current tier are no longer met for a
platform for an extended period of time based on the judgment of the
release manager or steering council. For platforms which no longer meet
the requirements of any tier by b1 of a new feature release, an
announcement will be made to warn the community of the pending removal
of support for the platform (e.g. in the b1 announcement). If the
platform is not brought into line for at least one of the tiers by the
first release candidate, it will be listed as unsupported in this PEP.

Tier 1

-   STATUS
-   CI failures block releases.
-   Changes which would break the main branch are not allowed to be
    merged; any breakage should be fixed or reverted immediately.
-   All core developers are responsible to keep main, and thus these
    platforms, working.
-   Failures on these platforms block a release.

  Target Triple              Notes
  -------------------------- -----------------
  aarch64-apple-darwin       clang
  i686-pc-windows-msvc
  x86_64-pc-windows-msvc
  x86_64-apple-darwin        BSD libc, clang
  x86_64-unknown-linux-gnu   glibc, gcc

Tier 2

-   STATUS
-   Must have a reliable buildbot.
-   At least two core developers are signed up to support the platform.
-   Changes which break any of these platforms are to be fixed or
    reverted within 24 hours.
-   Failures on these platforms block a release.

+-----------------------+--------------------+-----------------------+
| Target Triple         | Notes              | Contacts              |
+=======================+====================+=======================+
| aarc                  | glibc, gcc         | Petr Viktorin, Victor |
| h64-unknown-linux-gnu |                    | Stinner               |
|                       | glibc, clang       |                       |
|                       |                    | Victor Stinner,       |
|                       |                    | Gregory P. Smith      |
+-----------------------+--------------------+-----------------------+
| wasm32-unknown-wasi   | WASI SDK, Wasmtime | Brett Cannon, Eric    |
|                       |                    | Snow                  |
+-----------------------+--------------------+-----------------------+
| x86                   | glibc, clang       | Victor Stinner,       |
| _64-unknown-linux-gnu |                    | Gregory P. Smith      |
+-----------------------+--------------------+-----------------------+

Tier 3

-   STATUS
-   Must have a reliable buildbot.
-   At least one core developer is signed up to support the platform.
-   No response SLA to failures.
-   Failures on these platforms do not block a release.

+----------------------+----------------------+----------------------+
| Target Triple        | Notes                | Contacts             |
+======================+======================+======================+
| a                    |                      | Russell Keith-Magee, |
| arch64-linux-android |                      | Petr Viktorin        |
+----------------------+----------------------+----------------------+
| aar                  |                      | Steve Dower          |
| ch64-pc-windows-msvc |                      |                      |
+----------------------+----------------------+----------------------+
| arm64-apple-ios      | iOS on device        | Russell Keith-Magee, |
|                      |                      | Ned Deily            |
+----------------------+----------------------+----------------------+
| arm64                | iOS on M1 macOS      | Russell Keith-Magee, |
| -apple-ios-simulator | simulator            | Ned Deily            |
+----------------------+----------------------+----------------------+
| armv7l-unk           | Raspberry Pi OS,     | Gregory P. Smith     |
| nown-linux-gnueabihf | glibc, gcc           |                      |
+----------------------+----------------------+----------------------+
| powerpc64            | glibc, clang         | Victor Stinner       |
| le-unknown-linux-gnu |                      |                      |
|                      | glibc, gcc           | Victor Stinner       |
+----------------------+----------------------+----------------------+
| s39                  | glibc, gcc           | Victor Stinner       |
| 0x-unknown-linux-gnu |                      |                      |
+----------------------+----------------------+----------------------+
| x86_64-linux-android |                      | Russell Keith-Magee, |
|                      |                      | Petr Viktorin        |
+----------------------+----------------------+----------------------+
| x8                   | BSD libc, clang      | Victor Stinner       |
| 6_64-unknown-freebsd |                      |                      |
+----------------------+----------------------+----------------------+

All other platforms

Support for a platform may be partial within the code base, such as from
active development around platform support or accidentally. Code changes
to platforms not listed in the above tiers may be rejected or removed
from the code base without a deprecation process if they cause a
maintenance burden or obstruct general improvements.

Platforms not listed here may be supported by the wider Python community
in some way. If your desired platform is not listed above, please
perform a search online to see if someone is already providing support
in some form.

Notes

Microsoft Windows

Windows versions prior to Windows 10 follow Microsoft's Fixed Lifecycle
Policy, with a mainstream support phase for 5 years after release, where
the product is generally commercially available, and an additional 5
year extended support phase, where paid support is still available and
certain bug fixes are released. Extended Security Updates (ESU) is a
paid program available to high-volume enterprise customers as a "last
resort" option to receive certain security updates after extended
support ends. ESU is considered a distinct phase that follows the
expiration of extended support.

Windows 10 and later follow Microsoft's Modern Lifecycle Policy, which
varies per-product, per-version, per-edition and per-channel. Generally,
feature updates (1709, 22H2) occur every 6-12 months and are supported
for 18-36 months; Server and IoT editions, and LTSC channel releases are
supported for 5-10 years, and the latest feature release of a major
version (Windows 10, Windows 11) generally receives new updates for at
least 10 years following release. Microsoft's Windows Lifecycle FAQ has
more specific and up-to-date guidance.

CPython's Windows support currently follows Microsoft's lifecycles. A
new feature release X.Y.0 will support all Windows versions whose
extended support phase has not yet expired. Subsequent bug fix releases
will support the same Windows versions as the original feature release,
even if no longer supported by Microsoft. New versions of Windows
released while CPython is in maintenance mode may be supported at the
discretion of the core team and release manager.

As of 2024, our current interpretation of Microsoft's lifecycles is that
Windows for IoT and embedded systems is out of scope for new CPython
releases, as the intent of those is to avoid feature updates. Windows
Server will usually be the oldest version still receiving free security
fixes, and that will determine the earliest supported client release
with equivalent API version (which will usually be past its
end-of-life).

Each feature release is built by a specific version of Microsoft Visual
Studio. That version should have mainstream support when the release is
made. Developers of extension modules will generally need to use the
same Visual Studio release; they are concerned both with the
availability of the versions they need to use, and with keeping the zoo
of versions small. The CPython source tree will keep unmaintained build
files for older Visual Studio releases, for which patches will be
accepted. Such build files will be removed from the source tree 3 years
after the extended support for the compiler has ended (but continue to
remain available in revision control).

Legacy C Locale

Starting with CPython 3.7.0, *nix platforms are expected to provide at
least one of C.UTF-8 (full locale), C.utf8 (full locale) or UTF-8
(LC_CTYPE-only locale) as an alternative to the legacy C locale.

Any Unicode-related integration problems that occur only in the legacy C
locale and cannot be reproduced in an appropriately configured non-ASCII
locale will be closed as "won't fix".

Unsupporting platforms

If a platform drops out of tiered support, a note must be posted in this
PEP that the platform is no longer actively supported. This note must
include:

-   The name of the system,
-   The first release number that does not support this platform
    anymore, and
-   The first release where the historical support code is actively
    removed.

In some cases, it is not possible to identify the specific list of
systems for which some code is used (e.g. when autoconf tests for
absence of some feature which is considered present on all supported
systems). In this case, the name will give the precise condition
(usually a preprocessor symbol) that will become unsupported.

At the same time, the CPython build must be changed to produce a warning
if somebody tries to install CPython on this platform. On platforms
using autoconf, configure should also be made emit a warning about the
unsupported platform.

This gives potential users of the platform a chance to step forward and
offer maintenance. We do not treat a platform that loses Tier 3 support
any worse than a platform that was never supported.

No-longer-supported platforms

-   Name: MS-DOS, MS-Windows 3.x
    Unsupported in: Python 2.0
    Code removed in: Python 2.1

-   Name: SunOS 4
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: DYNIX
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: dgux
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: Minix
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: Irix 4 and --with-sgi-dl
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: Linux 1
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: Systems defining __d6_pthread_create (configure.in)
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: Systems defining PY_PTHREAD_D4, PY_PTHREAD_D6, or
    PY_PTHREAD_D7 in thread_pthread.h
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: Systems using --with-dl-dld
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: Systems using --without-universal-newlines,
    Unsupported in: Python 2.3
    Code removed in: Python 2.4

-   Name: MacOS 9
    Unsupported in: Python 2.4
    Code removed in: Python 2.4

-   Name: Systems using --with-wctype-functions
    Unsupported in: Python 2.6
    Code removed in: Python 2.6

-   Name: Win9x, WinME, NT4
    Unsupported in: Python 2.6 (warning in 2.5 installer)
    Code removed in: Python 2.6

-   Name: AtheOS
    Unsupported in: Python 2.6 (with "AtheOS" changed to "Syllable")
    Build broken in: Python 2.7 (edit configure to re-enable)
    Code removed in: Python 3.0
    Details: http://www.syllable.org/discussion.php?id=2320

-   Name: BeOS
    Unsupported in: Python 2.6 (warning in configure)
    Build broken in: Python 2.7 (edit configure to re-enable)
    Code removed in: Python 3.0

-   Name: Systems using Mach C Threads
    Unsupported in: Python 3.2
    Code removed in: Python 3.3

-   Name: SunOS lightweight processes (LWP)
    Unsupported in: Python 3.2
    Code removed in: Python 3.3

-   Name: Systems using --with-pth (GNU pth threads)
    Unsupported in: Python 3.2
    Code removed in: Python 3.3

-   Name: Systems using Irix threads
    Unsupported in: Python 3.2
    Code removed in: Python 3.3

-   Name: OSF* systems (issue 8606)
    Unsupported in: Python 3.2
    Code removed in: Python 3.3

-   Name: OS/2 (issue 16135)
    Unsupported in: Python 3.3
    Code removed in: Python 3.4

-   Name: VMS (issue 16136)
    Unsupported in: Python 3.3
    Code removed in: Python 3.4

-   Name: Windows 2000
    Unsupported in: Python 3.3
    Code removed in: Python 3.4

-   Name: Windows systems where COMSPEC points to command.com
    Unsupported in: Python 3.3
    Code removed in: Python 3.4

-   Name: RISC OS
    Unsupported in: Python 3.0 (some code actually removed)
    Code removed in: Python 3.4

-   Name: IRIX
    Unsupported in: Python 3.7
    Code removed in: Python 3.7

-   Name: Systems without multithreading support
    Unsupported in: Python 3.7
    Code removed in: Python 3.7

-   Name: wasm32-unknown-emscripten
    Unsupported in: Python 3.13
    Code removed in: Unknown

Discussions

-   April 2022: Consider adding a Tier 3 to tiered platform support
    (Victor Stinner)
-   March 2022: Proposed tiered platform support (Brett Cannon)
-   February 2015: Update to PEP 11 to clarify garnering platform
    support (Brett Cannon)
-   May 2014: Where is our official policy of what platforms we do
    support? (Brett Cannon)
-   August 2007: PEP 11 update - Call for port maintainers to step
    forward (Skip Montanaro)

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 289 Title: Generator Expressions Author: Raymond Hettinger
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 30-Jan-2002 Python-Version: 2.4 Post-History:
22-Oct-2003

Abstract

This PEP introduces generator expressions as a high performance, memory
efficient generalization of list comprehensions PEP 202 and generators
PEP 255.

Rationale

Experience with list comprehensions has shown their widespread utility
throughout Python. However, many of the use cases do not need to have a
full list created in memory. Instead, they only need to iterate over the
elements one at a time.

For instance, the following summation code will build a full list of
squares in memory, iterate over those values, and, when the reference is
no longer needed, delete the list:

    sum([x*x for x in range(10)])

Memory is conserved by using a generator expression instead:

    sum(x*x for x in range(10))

Similar benefits are conferred on constructors for container objects:

    s = set(word  for line in page  for word in line.split())
    d = dict( (k, func(k)) for k in keylist)

Generator expressions are especially useful with functions like sum(),
min(), and max() that reduce an iterable input to a single value:

    max(len(line)  for line in file  if line.strip())

Generator expressions also address some examples of functionals coded
with lambda:

    reduce(lambda s, a: s + a.myattr, data, 0)
    reduce(lambda s, a: s + a[3], data, 0)

These simplify to:

    sum(a.myattr for a in data)
    sum(a[3] for a in data)

List comprehensions greatly reduced the need for filter() and map().
Likewise, generator expressions are expected to minimize the need for
itertools.ifilter() and itertools.imap(). In contrast, the utility of
other itertools will be enhanced by generator expressions:

    dotproduct = sum(x*y for x,y in itertools.izip(x_vector, y_vector))

Having a syntax similar to list comprehensions also makes it easy to
convert existing code into a generator expression when scaling up
application.

Early timings showed that generators had a significant performance
advantage over list comprehensions. However, the latter were highly
optimized for Py2.4 and now the performance is roughly comparable for
small to mid-sized data sets. As the data volumes grow larger, generator
expressions tend to perform better because they do not exhaust cache
memory and they allow Python to re-use objects between iterations.

BDFL Pronouncements

This PEP is ACCEPTED for Py2.4.

The Details

(None of this is exact enough in the eye of a reader from Mars, but I
hope the examples convey the intention well enough for a discussion in
c.l.py. The Python Reference Manual should contain a 100% exact semantic
and syntactic specification.)

1.  The semantics of a generator expression are equivalent to creating
    an anonymous generator function and calling it. For example:

        g = (x**2 for x in range(10))
        print g.next()

    is equivalent to:

        def __gen(exp):
            for x in exp:
                yield x**2
        g = __gen(iter(range(10)))
        print g.next()

    Only the outermost for-expression is evaluated immediately, the
    other expressions are deferred until the generator is run:

        g = (tgtexp  for var1 in exp1 if exp2 for var2 in exp3 if exp4)

    is equivalent to:

        def __gen(bound_exp):
            for var1 in bound_exp:
                if exp2:
                    for var2 in exp3:
                        if exp4:
                            yield tgtexp
        g = __gen(iter(exp1))
        del __gen

2.  The syntax requires that a generator expression always needs to be
    directly inside a set of parentheses and cannot have a comma on
    either side. With reference to the file Grammar/Grammar in CVS, two
    rules change:

    a)  The rule:

            atom: '(' [testlist] ')'

        changes to:

            atom: '(' [testlist_gexp] ')'

        where testlist_gexp is almost the same as listmaker, but only
        allows a single test after 'for' ... 'in':

            testlist_gexp: test ( gen_for | (',' test)* [','] )

    b)  The rule for arglist needs similar changes.

    This means that you can write:

        sum(x**2 for x in range(10))

    but you would have to write:

        reduce(operator.add, (x**2 for x in range(10)))

    and also:

        g = (x**2 for x in range(10))

    i.e. if a function call has a single positional argument, it can be
    a generator expression without extra parentheses, but in all other
    cases you have to parenthesize it.

    The exact details were checked in to Grammar/Grammar version 1.49.

3.  The loop variable (if it is a simple variable or a tuple of simple
    variables) is not exposed to the surrounding function. This
    facilitates the implementation and makes typical use cases more
    reliable. In some future version of Python, list comprehensions will
    also hide the induction variable from the surrounding code (and, in
    Py2.4, warnings will be issued for code accessing the induction
    variable).

    For example:

        x = "hello"
        y = list(x for x in "abc")
        print x    # prints "hello", not "c"

4.  List comprehensions will remain unchanged. For example:

        [x for x in S]    # This is a list comprehension.
        [(x for x in S)]  # This is a list containing one generator
                          # expression.

    Unfortunately, there is currently a slight syntactic difference. The
    expression:

        [x for x in 1, 2, 3]

    is legal, meaning:

        [x for x in (1, 2, 3)]

    But generator expressions will not allow the former version:

        (x for x in 1, 2, 3)

    is illegal.

    The former list comprehension syntax will become illegal in Python
    3.0, and should be deprecated in Python 2.4 and beyond.

    List comprehensions also "leak" their loop variable into the
    surrounding scope. This will also change in Python 3.0, so that the
    semantic definition of a list comprehension in Python 3.0 will be
    equivalent to list(<generator expression>). Python 2.4 and beyond
    should issue a deprecation warning if a list comprehension's loop
    variable has the same name as a variable used in the immediately
    surrounding scope.

Early Binding versus Late Binding

After much discussion, it was decided that the first (outermost)
for-expression should be evaluated immediately and that the remaining
expressions be evaluated when the generator is executed.

Asked to summarize the reasoning for binding the first expression, Guido
offered[1]:

    Consider sum(x for x in foo()). Now suppose there's a bug in foo()
    that raises an exception, and a bug in sum() that raises an
    exception before it starts iterating over its argument. Which
    exception would you expect to see? I'd be surprised if the one in
    sum() was raised rather the one in foo(), since the call to foo()
    is part of the argument to sum(), and I expect arguments to be
    processed before the function is called.

    OTOH, in sum(bar(x) for x in foo()), where sum() and foo()
    are bugfree, but bar() raises an exception, we have no choice but
    to delay the call to bar() until sum() starts iterating -- that's
    part of the contract of generators. (They do nothing until their
    next() method is first called.)

Various use cases were proposed for binding all free variables when the
generator is defined. And some proponents felt that the resulting
expressions would be easier to understand and debug if bound
immediately.

However, Python takes a late binding approach to lambda expressions and
has no precedent for automatic, early binding. It was felt that
introducing a new paradigm would unnecessarily introduce complexity.

After exploring many possibilities, a consensus emerged that binding
issues were hard to understand and that users should be strongly
encouraged to use generator expressions inside functions that consume
their arguments immediately. For more complex applications, full
generator definitions are always superior in terms of being obvious
about scope, lifetime, and binding[2].

Reduction Functions

The utility of generator expressions is greatly enhanced when combined
with reduction functions like sum(), min(), and max(). The heapq module
in Python 2.4 includes two new reduction functions: nlargest() and
nsmallest(). Both work well with generator expressions and keep no more
than n items in memory at one time.

Acknowledgements

-   Raymond Hettinger first proposed the idea of "generator
    comprehensions" in January 2002.
-   Peter Norvig resurrected the discussion in his proposal for
    Accumulation Displays.
-   Alex Martelli provided critical measurements that proved the
    performance benefits of generator expressions. He also provided
    strong arguments that they were a desirable thing to have.
-   Phillip Eby suggested "iterator expressions" as the name.
-   Subsequently, Tim Peters suggested the name "generator expressions".
-   Armin Rigo, Tim Peters, Guido van Rossum, Samuele Pedroni, Hye-Shik
    Chang and Raymond Hettinger teased out the issues surrounding early
    versus late binding[3].
-   Jiwon Seo single-handedly implemented various versions of the
    proposal including the final version loaded into CVS. Along the way,
    there were periodic code reviews by Hye-Shik Chang and Raymond
    Hettinger. Guido van Rossum made the key design decisions after
    comments from Armin Rigo and newsgroup discussions. Raymond
    Hettinger provided the test suite, documentation, tutorial, and
    examples[4].

References

Copyright

This document has been placed in the public domain.

[1] Discussion over the relative merits of early versus late binding
https://mail.python.org/pipermail/python-dev/2004-April/044555.html

[2] Patch discussion and alternative patches on Source Forge
https://bugs.python.org/issue872326

[3] Discussion over the relative merits of early versus late binding
https://mail.python.org/pipermail/python-dev/2004-April/044555.html

[4] Patch discussion and alternative patches on Source Forge
https://bugs.python.org/issue872326