text:
PEP: 8011 Title: Python Governance Model Lead by Trio of Pythonistas
Author: Mariatta<EMAIL_ADDRESS>Barry Warsaw<EMAIL_ADDRESS>Status: Rejected Type: Informational Topic: Governance Content-Type:
text/x-rst Created: 24-Aug-2018

Abstract

This PEP proposes a governance model for the Core Python development
community, led by a trio of equally authoritative leaders. The Trio of
Pythonistas (ToP, or simply Trio) is tasked with making final decisions
for the language. It differs from PEP 8010 by specifically not proposing
a central singular leader, but instead a group of three people as the
leaders.

This PEP also proposes a formation of specialized workgroups to assist
the leadership trio in making decisions.

This PEP does not name the members of the Trio. Should this model be
adopted, it will be codified in PEP 13 along with the names of all
officeholders described in this PEP.

This PEP describes:

-   The role and responsibilities of the Trio
-   Guidelines of how trio members should be formed
-   Reasoning of the group of three, instead of a singular leader
-   Role and responsibilities of Python core developers to the trio
-   Sustainability considerations
-   Diversity and inclusivity considerations

PEP Rejection

PEP 8011 was rejected by a core developer vote described in PEP 8001 on
Monday, December 17, 2018.

PEP 8016 and the governance model it describes were chosen instead.

Open discussion points

Various tweaks to the parameters of this PEP are allowed during the
governance discussion process, such as the exact responsibilities of the
Trio, term lengths of service, voting procedures, and trio disbandment.
These will be codified by the time the PEP is ready to be voted on.

It is allowed, and perhaps even expected, that as experience is gained
with this model, these parameters may be tweaked in order to provide for
a smoother governing process. The process for tweaking these parameters
will generally be the same voting process as described in PEP 8001.

Roles and responsibilities of the leadership trio

-   Be open, considerate, respectful. In other words, adhering to The
    PSF's code of conduct.
-   Pronounce on PEPs, either as a team, or individually if the other
    trio members agree.
-   Provide vision and leadership for Python, the programming language
    and the community.
-   Understand their own limitation, and seek advice whenever necessary.
-   Provide mentorship to the next generation of leaders.
-   Be a Python core developer
-   Be a voting member of The PSF (one of Contributing / Manager /
    Fellow / Supporter).[1]
-   Understand that Python is not just a language but also a community.
    They need to be aware of issues in Python not just the technical
    aspects, but also other issues in the community.
-   Facilitate the formation of specialized working groups within Core
    Python. See "formation of specialized working groups" section below.
-   Set good example of behavior, culture, and tone to Python community.
    Just as Python looks at and learn from other communities for
    inspiration, other communities will look at Python and learn from
    us.

Authority of the trio

To be clear, in case any dispute arises: the trio has the final
authority to pronounce on PEPs (except for the governance PEP), to
decide whether a particular decision requires a PEP, and to resolve
technical disputes in general. The trio's authority does not include
changing the governance itself, or other non-technical disputes that may
arise; these should be handled through the process described in PEP
8001.

What are NOT considered as the role responsibilities of the trio

The following are not the expected out of the trio, however they can do
these if they wish.

-   They are not always the ones coming up with all the ideas, vision,
    problems to solve, and what not. The trio will be open and accepting
    suggestions from core developers and community.
-   Day to day bug reports do not require the trio to intervene. Any
    core devs are able to make decisions, but will defer to the
    respective focused workgroups, and will eventually defer to the trio
    when there are major disagreements among core developers.
-   Does not run / decide on Python language summit and its logistics.
-   Does not run / decide on Python core sprint and its logistics.
-   Does not handle CoC cases. Those are responsibilities of the PSF CoC
    workgroup, but will speak out if they witness those cases.
-   Does not make decisions about other Python implementations (Cython,
    IronPython, etc).
-   Does not run / decide on Python conferences and its logistics.
-   Not an evangelist of Python. The trio is not expected to
    preach/advertise for Python. They can if they want to, but not
    expected.
-   Not an educator of Python. The trio is not expected to be the ones
    teaching/writing about Python. They can if they want to, but not
    expected.
-   The trio is not expected to be available 24/7, 365 days a year. They
    are free to decide for themselves their availability for Python.
-   Not a PEP editor.

Guidelines for the formation of the trio

The success of this governance model relies on the members of the trio,
and the ability of the trio members to collaborate and work well
together.

The three people need to have similar vision to Python, and each can
have different skills that complement one another.

With such a team, disagreements and conflict should be rare, but can
still happen. We will need to trust the people we select that they are
able to resolve this among themselves.

When it comes to select the members of the trio, instead of nominating
various individuals and choosing the top three, core developers will
nominate trios and vote for groups of threes who they believe can form
this united trio. There is no restriction that an individual can only be
nominated in one slate.

This PEP will not name or nominate anyone into the trio.

Only once this PEP is accepted, any active core developers (who are
eligible to vote) can submit nomination of groups of three.

Once this PEP is accepted and core devs have submitted their
nominations, voting can begin, and the voting mechanism described in PEP
8001 will be used.

Qualities desired out of the trio:

-   Be a Python core developer.
-   Be a voting PSF member (one of Contributing / Manager / Fellow /
    Supporter).[2]
-   Be a member of the community with good standing.
-   Adhere to The PSF's code of conduct (Be open, considerate, and
    respectful).[3]
-   Be willing to accept the said roles and responsibilities.
-   Able to communicate and articulate their thoughts effectively.

The following are not requirements when considering someone into the
trio:

-   "Experience being a BDFL of something" is not a requirement.
-   "Be a genius" is not a requirement.

Diversity and inclusivity

The core Python development team fully supports the Python Software
Foundation’s diversity statement, and welcomes participation and
contribution from people from diverse backgrounds. When nominating
people to be part of the trio, Python core developers will take every
effort into including members from underrepresented group into
consideration.

Ideally, nomination should include and reflect the diversity of core
Python contributors.

Sustainability

Lack of employer support or lack of luxury of free time should not be a
factor when identifying who should be in a trio. If there are
individuals who the core devs have identified as having the necessary
skills for being a member of the trio, but they are unable to do it
because of lack of time, lack of financial support, then we should open
discussion with The PSF or other parties into providing the needed
support.

Additional guidelines

When nominating someone other than yourself, please first ask privately
if they are ok with being nominated, and if they are ok with nominated
in that group of three. This is so people don’t feel pressured to accept
nomination just because it happens publicly.

Why not other governance model

Core Python community are familiar with the singular BDFL model for over
two decades, it was a model that has "worked" for Python. Shifting to a
completely different model all of the sudden, could be disruptive to the
stability of the community. However, the community can continue to
evolve in the future.

If this PEP is chosen, it is not meant to be the only governance model
for Python going forward.

This PEP proposed a transition into a community led by a group of people
(although small), while also introducing the concept of additional
specialized workgroups.

Why not more than three

Too many chefs spoil the soup.

The goal of having a leadership team is for team Python core developers
to be able to come to consensus and decisions. The larger the leadership
team is, the more difficult it will be in coming up with decision.

This is also for the benefit of the members of the trio. Learning to
collaborate with other people in a team is not something that happen
organically and takes a lot of effort. It is expected that members of
the trio will be part of the team for a long-term period. Having to deal
with two other people is probably difficult enough. We want the trio to
be able to do their duties and responsibilities as efficient as
possible.

The more people in the group, the more difficult it is to try to come up
with time to meet, discuss, and coming up with decision.

Roles and responsibilities of Python Core Developers to the trio

-   Be open, considerate, and respectful. In other words, adhere to The
    PSF's Code of Conduct
-   Decisions and pronouncements made by individual members of the trio
    are to be seen as authoritative and coming from the trio.
-   Once the trio has pronounced a decision, core devs will be
    supportive, even if they were not supportive in the beginning
    (before the trio made such decision)
-   Continue with day-to-day decision making in the bug tracker, and
    defer to the trio if there is major disagreement
-   Python core developers do not handle CoC cases, those are
    responsibilities of the CoC workgroup, but will speak out if they
    witness those cases
-   Aware that they are part of the larger Python community, not just
    the technical aspect of it.
-   Be a voting PSF member (one of Contributing / Manager / Fellow /
    Supporter).
-   Set good example of behavior, culture, and tone to Python community.

Term Limit

The trio is not expected to serve for life, however a longer term is
desired. The purpose of longer term service is to avoid unnecessary
churns of needing to “elect”, and to provide stability and consistency
in the language and the community.

Currently, Python release managers hold their position for 5 years (one
release cycle), and that seems to work so far. Therefore, this PEP
proposes that the trio hold their position for 5 years.

Succession planning of the trio (open for discussion)

The trio should notify core devs of their intention to
disband/retire/quit from their roles at least one year in advance, to
allow for them to actively mentor and train the next generation of
successors, and to avoid power vacuum.

The trio do not necessarily have to be the ones choosing who the next
leaders will be.

This PEP does not enforce that the same governance model be chosen for
the next generation. Python as language and community can continue to
evolve. By giving one year advance notice to disband, the trio is giving
the core Python community an opportunity to reflect on the
success/failure of this governance model, and choose a different
governance model if needed.

However, the next governance model and leaders should be chosen/elected
within one year after the trio announced their desire to disband.

If it was decided to continue with this model of governance, the next
generation of trio will be nominated and elected similar to how the
first trio were nominated/chosen.

The trio should act as advisor/mentor to the next generation chosen
leaders for at least X months.

Since future trio will be chosen out of Python core developers, it will
make sense for future Python core developers to possess some but not
necessarily all, qualities of the trio as laid out in this PEP.

Therefore, the guidelines for selecting trio members can also be used as
guidelines when identifying future Python core developers.

Scenario if one member of the trio needs to quit

Effective governance models provide off-ramps or temporary breaks for
leaders who need to step down or pause their leadership service.

What if one member of the chosen trio has to quit, for unforeseen
reasons?

There are several possible options:

-   The remaining duo can select another member to fill in the role
-   The trio can choose to disband, core developers can nominate other
    trios
-   Core developers can choose a different governance model

Since the trio were elected as a slate and so the loss of one breaks
that unit that was elected. Therefore, a new election should be held.

Formation of working groups/area of expertise/ownership (previously BDFL delegate)

(Open for discussion).

Certain areas and topic of Core Python and Python community require
leaders with specific skills of specialty. It will be recommended that
there will be several working groups with more authority in that
specific area to assist the trio in making decisions.

The role of these "specialized work groups/council" is to be the final
decision maker for controversial discussions that arise in their
respective areas.

These working groups should be small (3-5 people), for similar reasons
that the leadership trio is a small group.

These working groups should consist of both Python core developers and
external experts. This is to ensure that decision made does not favor
only Python core developers.

Python Core developers will defer decisions to these working groups on
their respective topic. However these groups will answer/defer to the
trio.

These working groups can be selected and members voted only after this
PEP gets accepted.

If this PEP is accepted, the working group can be decided within a year
or two after the PEP's acceptance.

When selecting members of these special work groups, the trio will take
every effort into including members from underrepresented group into
consideration. Ideally, the workgroup members should include and reflect
the diversity of the wider Python community.

Members of this workgroup do not need to be a Python core developer, but
they need to be at least a basic member of the PSF[4].

These workgroup are active as long as the trio are active.

Several suggested working groups to start:

-   Documentation of CPython
-   Security of CPython
-   Performance of CPython

The workgroup can be seen as having similar role as the previously known
role of "BDFL-delegate" or PEP czars. The difference is, instead of
appointing a single person as decision maker, there will be a small team
of decision makers.

Another difference with the previous "BDFL-delegate" role, the group can
be active as long as the trio is active, as opposed to only when there
is a PEP that requires their expertise.

When the trio disbands, these workgroups are disbanded too.

Why these workgroups are necessary

This is an effort to 'refactor the large role' of the previous Python
BDFL.

Affirmation as being a member of the PSF

This PEP proposes that core developers and the trio members self-certify
themselves as being a member of The PSF.

Being part of the PSF means being part of the Python community, and
support The PSF's mission and diversity statement.

By being a member of The PSF, Python core developers declare their
support for Python and agree to the community Code of Conduct.

For more details of The PSF membership, see: PSF Membership FAQ[5].

Reasoning for choosing the name trio

Not to be confused with Python trio (an async library).

The "trio" is short and easy to pronounce, unlike other words that are
long and can have negative interpretations, like triad, trinity,
triumvirate, threesome, etc.

References

Copyright

This document has been placed in the public domain.

[1] PSF Membership FAQ (https://www.python.org/psf/membership/)

[2] PSF Membership FAQ (https://www.python.org/psf/membership/)

[3] The PSF's Code of Conduct
(https://www.python.org/psf/codeofconduct/)

[4] PSF Membership FAQ (https://www.python.org/psf/membership/)

[5] PSF Membership FAQ (https://www.python.org/psf/membership/)

authors:
Barry Warsaw
Mariatta


text:
PEP: 301 Title: Package Index and Metadata for Distutils Version:
$Revision$ Last-Modified: $Date$ Author: Richard Jones
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic:
Packaging Content-Type: text/x-rst Created: 24-Oct-2002 Python-Version:
2.3 Post-History: 08-Nov-2002

Abstract

This PEP proposes several extensions to the Distutils packaging system
[1]. These enhancements include a central package index server, tools
for submitting package information to the index and extensions to the
package metadata to include Trove[2] information.

This PEP does not address issues of package dependency. It also does not
address storage and download of packages as described in PEP 243. Nor is
it proposing a local database of packages as described in PEP 262.

Existing package repositories such as the Vaults of Parnassus[3],
CPAN[4] and PAUSE[5] will be investigated as prior art in this field.

Rationale

Python programmers have long needed a simple method of discovering
existing modules and systems available for their use. It is arguable
that the existence of these systems for other languages have been a
significant contribution to their popularity. The existence of the
Catalog-SIG, and the many discussions there indicate that there is a
large population of users who recognise this need.

The introduction of the Distutils packaging system to Python simplified
the process of distributing shareable code, and included mechanisms for
the capture of package metadata, but did little with the metadata save
ship it with the package.

An interface to the index should be hosted in the python.org domain,
giving it an air of legitimacy that existing catalog efforts do not
have.

The interface for submitting information to the catalog should be as
simple as possible - hopefully just a one-line command for most users.

Issues of package dependency are not addressed due to the complexity of
such a system. PEP 262 proposes such a system, but as of this writing
the PEP is still unfinished.

Issues of package dissemination (storage on a central server) are not
addressed because they require assumptions about availability of storage
and bandwidth that I am not in a position to make. PEP 243, which is
still being developed, is tackling these issues and many more. This
proposal is considered compatible with, and adjunct to the proposal in
PEP 243.

Specification

The specification takes three parts, the web interface, the Distutils
register command and the Distutils Trove classification.

Web Interface

A web interface is implemented over a simple store. The interface is
available through the python.org domain, either directly or as
packages.python.org.

The store has columns for all metadata fields. The (name, version)
double is used as a uniqueness key. Additional submissions for an
existing (name, version) will result in an update operation.

The web interface implements the following commands/interfaces:

index

    Lists known packages, optionally filtered. An additional HTML page,
    search, presents a form to the user which is used to customise the
    index view. The index will include a browsing interface like that
    presented in the Trove interface design section 4.3. The results
    will be paginated, sorted alphabetically and only showing the most
    recent version. The most recent version information will be
    determined using the Distutils LooseVersion class.

display

    Displays information about the package. All fields are displayed as
    plain text. The "url" (or "home_page") field is hyperlinked.

submit

    Accepts a POST submission of metadata about a package. The "name"
    and "version" fields are mandatory, as they uniquely identify an
    entry in the index. Submit will automatically determine whether to
    create a new entry or update an existing entry. The metadata is
    checked for correctness where appropriate - specifically the Trove
    discriminators are compared with the allowed set. An update will
    update all information about the package based on the new submitted
    information.

    There will also be a submit/edit form that will allow manual
    submission and updating for those who do not use Distutils.

submit_pkg_info

    Accepts a POST submission of a PKG-INFO file and performs the same
    function as the submit interface.

user

    Registers a new user with the index. Requires username, password and
    email address. Passwords will be stored in the index database as SHA
    hashes. If the username already exists in the database:

    1.  If valid HTTP Basic authentication is provided, the password and
        email address are updated with the submission information, or
    2.  If no valid authentication is provided, the user is informed
        that the login is already taken.

    Registration will be a three-step process, involving:

    1.  User submission of details via the Distutils register command or
        through the web,
    2.  Index server sending email to the user's email address with a
        URL to visit to confirm registration with a random one-time key,
        and
    3.  User visits URL with the key and confirms registration.

roles

    An interface for changing user Role assignments.

password_reset

    Using a supplied email address as the key, this resets a user's
    password and sends an email with the new password to the user.

The submit command will require HTTP Basic authentication, preferably
over an HTTPS connection.

The server interface will indicate success or failure of the commands
through a subset of the standard HTTP response codes:

  Code   Meaning        Register command implications
  ------ -------------- -------------------------------------------------------------------------------------------
  200    OK             Everything worked just fine
  400    Bad request    Data provided for submission was malformed
  401    Unauthorised   The username or password supplied were incorrect
  403    Forbidden      User does not have permission to update the package information (not Owner or Maintainer)

User Roles

Three user Roles will be assignable to users:

Owner

    Owns a package name, may assign Maintainer Role for that name. The
    first user to register information about a package is deemed Owner
    of the package name. The Admin user may change this if necessary.
    May submit updates for the package name.

Maintainer

    Can submit and update info for a particular package name.

Admin

    Can assign Owner Role and edit user details. Not specific to a
    package name.

Index Storage (Schema)

The index is stored in a set of relational database tables:

packages

    Lists package names and holds package-level metadata (currently just
    the stable release version)

releases

    Each package has an entry in releases for each version of the
    package that is released. A row holds the bulk of the information
    given in the package's PKG-INFO file. There is one row for each
    package (name, version).

trove_discriminators

    Lists the Trove discriminator text and assigns each one a unique ID.

release_discriminators

    Each entry maps a package (name, version) to a discriminator_id. We
    map to releases instead of packages because the set of
    discriminators may change between releases.

journals

    Holds information about changes to package information in the index.
    Changes to the packages, releases, roles, and release_discriminators
    tables are listed here by package name and version if the change is
    release-specific.

users

    Holds our user database - user name, email address and password.

roles

    Maps user_name and role_name to a package_name.

An additional table, rego_otk holds the One Time Keys generated during
registration and is not interesting in the scope of the index itself.

Distutils register Command

An additional Distutils command, register, is implemented which posts
the package metadata to the central index. The register command
automatically handles user registration; the user is presented with
three options:

1.  login and submit package information
2.  register as a new packager
3.  send password reminder email

On systems where the $HOME environment variable is set, the user will be
prompted at exit to save their username/password to a file in their
$HOME directory in the file .pypirc.

Notification of changes to a package entry will be sent to all users who
have submitted information about the package. That is, the original
submitter and any subsequent updaters.

The register command will include a --verify option which performs a
test submission to the index without actually committing the data. The
index will perform its submission verification checks as usual and
report any errors it would have reported during a normal submission.
This is useful for verifying correctness of Trove discriminators.

Distutils Trove Classification

The Trove concept of discrimination will be added to the metadata set
available to package authors through the new attribute "classifiers".
The list of classifiers will be available through the web, and added to
the package like so:

    setup(
        name = "roundup",
        version = __version__,
        classifiers = [
            'Development Status :: 4 - Beta',
            'Environment :: Console',
            'Environment :: Web Environment',
            'Intended Audience :: End Users/Desktop',
            'Intended Audience :: Developers',
            'Intended Audience :: System Administrators',
            'License :: OSI Approved :: Python Software Foundation License',
            'Operating System :: MacOS :: MacOS X',
            'Operating System :: Microsoft :: Windows',
            'Operating System :: POSIX',
            'Programming Language :: Python',
            'Topic :: Communications :: Email',
            'Topic :: Office/Business',
            'Topic :: Software Development :: Bug Tracking',
        ],
        url = 'http://sourceforge.net/projects/roundup/',
        ...
    )

It was decided that strings would be used for the classification entries
due to the deep nesting that would be involved in a more formal Python
structure.

The original Trove specification that classification namespaces be
separated by slashes ("/") unfortunately collides with many of the names
having slashes in them (e.g. "OS/2"). The double-colon solution (" :: ")
implemented by SourceForge and FreshMeat gets around this limitation.

The list of classification values on the module index has been merged
from FreshMeat and SourceForge (with their permission). This list will
be made available both through the web interface and through the
register command's --list-classifiers option as a text list which may
then be copied to the setup.py file. The register command's --verify
option will check classifiers values against the server's list.

Unfortunately, the addition of the "classifiers" property is not
backwards-compatible. A setup.py file using it will not work under
Python 2.1.3. It is hoped that a bug-fix release of Python 2.2 (most
likely 2.2.3) will relax the argument checking of the setup() command to
allow new keywords, even if they're not actually used. It is preferable
that a warning be produced, rather than a show-stopping error. The use
of the new keyword should be discouraged in situations where the package
is advertised as being compatible with python versions earlier than
2.2.3 or 2.3.

In the PKG-INFO, the classifiers list items will appear as individual
Classifier: entries:

    Name: roundup
    Version: 0.5.2
    Classifier: Development Status :: 4 - Beta
    Classifier: Environment :: Console (Text Based)
                .
                .
    Classifier: Topic :: Software Development :: Bug Tracking
    Url: http://sourceforge.net/projects/roundup/

Implementation

The server is available at:

  http://www.python.org/pypi

The code is available from the SourceForge project:

  http://sourceforge.net/projects/pypi/

The register command has been integrated into Python 2.3.

Rejected Proposals

Originally, the index server was to return custom headers (inspired by
PEP 243):

X-Pypi-Status

    Either "success" or "fail".

X-Pypi-Reason

    A description of the reason for failure, or additional information
    in the case of a success.

However, it has been pointed out[6] that this is a bad scheme to use.

References

Copyright

This document has been placed in the public domain.

Acknowledgements

Anthony Baxter, Martin v. Loewis and David Goodger for encouragement and
feedback during initial drafting.

A.M. Kuchling for support including hosting the second prototype.

Greg Stein for recommending that the register command interpret the HTTP
response codes rather than custom X-PyPI-* headers.

The many participants of the Distutils and Catalog SIGs for their ideas
over the years.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] Distutils packaging system
(http://docs.python.org/library/distutils.html)

[2] Trove (http://www.catb.org/~esr/trove/)

[3] Vaults of Parnassus (http://www.vex.net/parnassus/)

[4] CPAN (http://www.cpan.org/)

[5] PAUSE (http://pause.cpan.org/)

[6] [PEP243] upload status is bogus
(https://mail.python.org/pipermail/distutils-sig/2001-March/002262.html)

license: Public Domain


text:
PEP: 436 Title: The Argument Clinic DSL Version: $Revision$
Last-Modified: $Date$ Author: Larry Hastings<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards
Track Content-Type: text/x-rst Created: 22-Feb-2013 Python-Version: 3.4

Abstract

This document proposes "Argument Clinic", a DSL to facilitate argument
processing for built-in functions in the implementation of CPython.

Rationale and Goals

The primary implementation of Python, "CPython", is written in a mixture
of Python and C. One implementation detail of CPython is what are called
"built-in" functions -- functions available to Python programs but
written in C. When a Python program calls a built-in function and passes
in arguments, those arguments must be translated from Python values into
C values. This process is called "parsing arguments".

As of CPython 3.3, builtin functions nearly always parse their arguments
with one of two functions: the original PyArg_ParseTuple(),[1] and the
more modern PyArg_ParseTupleAndKeywords().[2] The former only handles
positional parameters; the latter also accommodates keyword and
keyword-only parameters, and is preferred for new code.

With either function, the caller specifies the translation for parsing
arguments in a "format string":[3] each parameter corresponds to a
"format unit", a short character sequence telling the parsing function
what Python types to accept and how to translate them into the
appropriate C value for that parameter.

PyArg_ParseTuple() was reasonable when it was first conceived. There
were only a dozen or so of these "format units"; each one was distinct,
and easy to understand and remember. But over the years the PyArg_Parse
interface has been extended in numerous ways. The modern API is complex,
to the point that it is somewhat painful to use. Consider:

-   There are now forty different "format units"; a few are even three
    characters long. This makes it difficult for the programmer to
    understand what the format string says--or even perhaps to parse
    it--without constantly cross-indexing it with the documentation.
-   There are also six meta-format units that may be buried in the
    format string. (They are: "()|$:;".)
-   The more format units are added, the less likely it is the
    implementer can pick an easy-to-use mnemonic for the format unit,
    because the character of choice is probably already in use. In other
    words, the more format units we have, the more obtuse the format
    units become.
-   Several format units are nearly identical to others, having only
    subtle differences. This makes understanding the exact semantics of
    the format string even harder, and can make it difficult to figure
    out exactly which format unit you want.
-   The docstring is specified as a static C string, making it mildly
    bothersome to read and edit since it must obey C string quoting
    rules.
-   When adding a new parameter to a function using
    PyArg_ParseTupleAndKeywords(), it's necessary to touch six different
    places in the code:[4]
    -   Declaring the variable to store the argument.
    -   Passing in a pointer to that variable in the correct spot in
        PyArg_ParseTupleAndKeywords(), also passing in any "length" or
        "converter" arguments in the correct order.
    -   Adding the name of the argument in the correct spot of the
        "keywords" array passed in to PyArg_ParseTupleAndKeywords().
    -   Adding the format unit to the correct spot in the format string.
    -   Adding the parameter to the prototype in the docstring.
    -   Documenting the parameter in the docstring.
-   There is currently no mechanism for builtin functions to provide
    their "signature" information (see inspect.getfullargspec and
    inspect.Signature). Adding this information using a mechanism
    similar to the existing PyArg_Parse functions would require
    repeating ourselves yet again.

The goal of Argument Clinic is to replace this API with a mechanism
inheriting none of these downsides:

-   You need specify each parameter only once.
-   All information about a parameter is kept together in one place.
-   For each parameter, you specify a conversion function; Argument
    Clinic handles the translation from Python value into C value for
    you.
-   Argument Clinic also allows for fine-tuning of argument processing
    behavior with parameterized conversion functions.
-   Docstrings are written in plain text. Function docstrings are
    required; per-parameter docstrings are encouraged.
-   From this, Argument Clinic generates for you all the mundane,
    repetitious code and data structures CPython needs internally. Once
    you've specified the interface, the next step is simply to write
    your implementation using native C types. Every detail of argument
    parsing is handled for you.

Argument Clinic is implemented as a preprocessor. It draws inspiration
for its workflow directly from [Cog] by Ned Batchelder. To use Clinic,
add a block comment to your C source code beginning and ending with
special text strings, then run Clinic on the file. Clinic will find the
block comment, process the contents, and write the output back into your
C source file directly after the comment. The intent is that Clinic's
output becomes part of your source code; it's checked in to revision
control, and distributed with source packages. This means that Python
will still ship ready-to-build. It does complicate development slightly;
in order to add a new function, or modify the arguments or documentation
of an existing function using Clinic, you'll need a working Python 3
interpreter.

Future goals of Argument Clinic include:

-   providing signature information for builtins,
-   enabling alternative implementations of Python to create automated
    library compatibility tests, and
-   speeding up argument parsing with improvements to the generated
    code.

DSL Syntax Summary

The Argument Clinic DSL is specified as a comment embedded in a C file,
as follows. The "Example" column on the right shows you sample input to
the Argument Clinic DSL, and the "Section" column on the left specifies
what each line represents in turn.

Argument Clinic's DSL syntax mirrors the Python def statement, lending
it some familiarity to Python core developers.

    +-----------------------+-----------------------------------------------------------------+
    | Section               | Example                                                         |
    +-----------------------+-----------------------------------------------------------------+
    | Clinic DSL start      | /*[clinic]                                                      |
    | Module declaration    | module module_name                                              |
    | Class declaration     | class module_name.class_name                                    |
    | Function declaration  | module_name.function_name  -> return_annotation                 |
    | Parameter declaration |       name : converter(param=value)                             |
    | Parameter docstring   |           Lorem ipsum dolor sit amet, consectetur               |
    |                       |           adipisicing elit, sed do eiusmod tempor               |
    | Function docstring    | Lorem ipsum dolor sit amet, consectetur adipisicing             |
    |                       | elit, sed do eiusmod tempor incididunt ut labore et             |
    | Clinic DSL end        | [clinic]*/                                                      |
    | Clinic output         | ...                                                             |
    | Clinic output end     | /*[clinic end output:<checksum>]*/                              |
    +-----------------------+-----------------------------------------------------------------+

To give some flavor of the proposed DSL syntax, here are some sample
Clinic code blocks. This first block reflects the normally preferred
style, including blank lines between parameters and per-argument
docstrings. It also includes a user-defined converter (path_t) created
locally:

    /*[clinic]
    os.stat as os_stat_fn -> stat result

       path: path_t(allow_fd=1)
           Path to be examined; can be string, bytes, or open-file-descriptor int.

       *

       dir_fd: OS_STAT_DIR_FD_CONVERTER = DEFAULT_DIR_FD
           If not None, it should be a file descriptor open to a directory,
           and path should be a relative string; path will then be relative to
           that directory.

       follow_symlinks: bool = True
           If False, and the last element of the path is a symbolic link,
           stat will examine the symbolic link itself instead of the file
           the link points to.

    Perform a stat system call on the given path.

    {parameters}

    dir_fd and follow_symlinks may not be implemented
      on your platform.  If they are unavailable, using them will raise a
      NotImplementedError.

    It's an error to use dir_fd or follow_symlinks when specifying path as
      an open file descriptor.

    [clinic]*/

This second example shows a minimal Clinic code block, omitting all
parameter docstrings and non-significant blank lines:

    /*[clinic]
    os.access
       path: path
       mode: int
       *
       dir_fd: OS_ACCESS_DIR_FD_CONVERTER = 1
       effective_ids: bool = False
       follow_symlinks: bool = True
    Use the real uid/gid to test for access to a path.
    Returns True if granted, False otherwise.

    {parameters}

    dir_fd, effective_ids, and follow_symlinks may not be implemented
      on your platform.  If they are unavailable, using them will raise a
      NotImplementedError.

    Note that most operations will use the effective uid/gid, therefore this
      routine can be used in a suid/sgid environment to test if the invoking user
      has the specified access to the path.

    [clinic]*/

This final example shows a Clinic code block handling groups of optional
parameters, including parameters on the left:

    /*[clinic]
    curses.window.addch

       [
       y: int
         Y-coordinate.

       x: int
         X-coordinate.
       ]

       ch: char
         Character to add.

       [
       attr: long
         Attributes for the character.
       ]

       /

    Paint character ch at (y, x) with attributes attr,
    overwriting any character previously painter at that location.
    By default, the character position and attributes are the
    current settings for the window object.
    [clinic]*/

General Behavior Of the Argument Clinic DSL

All lines support # as a line comment delimiter except docstrings. Blank
lines are always ignored.

Like Python itself, leading whitespace is significant in the Argument
Clinic DSL. The first line of the "function" section is the function
declaration. Indented lines below the function declaration declare
parameters, one per line; lines below those that are indented even
further are per-parameter docstrings. Finally, the first line dedented
back to column 0 end parameter declarations and start the function
docstring.

Parameter docstrings are optional; function docstrings are not.
Functions that specify no arguments may simply specify the function
declaration followed by the docstring.

Module and Class Declarations

When a C file implements a module or class, this should be declared to
Clinic. The syntax is simple:

    module module_name

or :

    class module_name.class_name

(Note that these are not actually special syntax; they are implemented
as Directives.)

The module name or class name should always be the full dotted path from
the top-level module. Nested modules and classes are supported.

Function Declaration

The full form of the function declaration is as follows:

    dotted.name [ as legal_c_id ] [ -> return_annotation ]

The dotted name should be the full name of the function, starting with
the highest-level package (e.g. "os.stat" or "curses.window.addch").

The "as legal_c_id" syntax is optional. Argument Clinic uses the name of
the function to create the names of the generated C functions. In some
circumstances, the generated name may collide with other global names in
the C program's namespace. The "as legal_c_id" syntax allows you to
override the generated name with your own; substitute "legal_c_id" with
any legal C identifier. If skipped, the "as" keyword must also be
omitted.

The return annotation is also optional. If skipped, the arrow ("->")
must also be omitted. If specified, the value for the return annotation
must be compatible with ast.literal_eval, and it is interpreted as a
return converter.

Parameter Declaration

The full form of the parameter declaration line as follows:

    name: converter [ (parameter=value [, parameter2=value2]) ] [ = default]

The "name" must be a legal C identifier. Whitespace is permitted between
the name and the colon (though this is not the preferred style).
Whitespace is permitted (and encouraged) between the colon and the
converter.

The "converter" is the name of one of the "converter functions"
registered with Argument Clinic. Clinic will ship with a number of
built-in converters; new converters can also be added dynamically. In
choosing a converter, you are automatically constraining what Python
types are permitted on the input, and specifying what type the output
variable (or variables) will be. Although many of the converters will
resemble the names of C types or perhaps Python types, the name of a
converter may be any legal Python identifier.

If the converter is followed by parentheses, these parentheses enclose
parameter to the conversion function. The syntax mirrors providing
arguments a Python function call: the parameter must always be named, as
if they were "keyword-only parameters", and the values provided for the
parameters will syntactically resemble Python literal values. These
parameters are always optional, permitting all conversion functions to
be called without any parameters. In this case, you may also omit the
parentheses entirely; this is always equivalent to specifying empty
parentheses. The values supplied for these parameters must be compatible
with ast.literal_eval.

The "default" is a Python literal value. Default values are optional; if
not specified you must omit the equals sign too. Parameters which don't
have a default are implicitly required. The default value is dynamically
assigned, "live" in the generated C code, and although it's specified as
a Python value, it's translated into a native C value in the generated C
code. Few default values are permitted, owing to this manual translation
step.

If this were a Python function declaration, a parameter declaration
would be delimited by either a trailing comma or an ending parenthesis.
However, Argument Clinic uses neither; parameter declarations are
delimited by a newline. A trailing comma or right parenthesis is not
permitted.

The first parameter declaration establishes the indent for all parameter
declarations in a particular Clinic code block. All subsequent
parameters must be indented to the same level.

Legacy Converters

For convenience's sake in converting existing code to Argument Clinic,
Clinic provides a set of legacy converters that match PyArg_ParseTuple
format units. They are specified as a C string containing the format
unit. For example, to specify a parameter "foo" as taking a Python "int"
and emitting a C int, you could specify:

    foo : "i"

(To more closely resemble a C string, these must always use double
quotes.)

Although these resemble PyArg_ParseTuple format units, no guarantee is
made that the implementation will call a PyArg_Parse function for
parsing.

This syntax does not support parameters. Therefore, it doesn't support
any of the format units that require input parameters
("O!", "O&", "es", "es#", "et", "et#"). Parameters requiring one of
these conversions cannot use the legacy syntax. (You may still, however,
supply a default value.)

Parameter Docstrings

All lines that appear below and are indented further than a parameter
declaration are the docstring for that parameter. All such lines are
"dedented" until the first line is flush left.

Special Syntax For Parameter Lines

There are four special symbols that may be used in the parameter
section. Each of these must appear on a line by itself, indented to the
same level as parameter declarations. The four symbols are:

*

    Establishes that all subsequent parameters are keyword-only.

[

    Establishes the start of an optional "group" of parameters. Note
    that "groups" may nest inside other "groups". See Functions With
    Positional-Only Parameters below. Note that currently [ is only
    legal for use in functions where all parameters are marked
    positional-only, see / below.

]

    Ends an optional "group" of parameters.

/

    Establishes that all the proceeding arguments are positional-only.
    For now, Argument Clinic does not support functions with both
    positional-only and non-positional-only arguments. Therefore: if /
    is specified for a function, it must currently always be after the
    last parameter. Also, Argument Clinic does not currently support
    default values for positional-only parameters.

(The semantics of / follow a syntax for positional-only parameters in
Python once proposed by Guido.[5] )

Function Docstring

The first line with no leading whitespace after the function declaration
is the first line of the function docstring. All subsequent lines of the
Clinic block are considered part of the docstring, and their leading
whitespace is preserved.

If the string {parameters} appears on a line by itself inside the
function docstring, Argument Clinic will insert a list of all parameters
that have docstrings, each such parameter followed by its docstring. The
name of the parameter is on a line by itself; the docstring starts on a
subsequent line, and all lines of the docstring are indented by two
spaces. (Parameters with no per-parameter docstring are suppressed.) The
entire list is indented by the leading whitespace that appeared before
the {parameters} token.

If the string {parameters} doesn't appear in the docstring, Argument
Clinic will append one to the end of the docstring, inserting a blank
line above it if the docstring does not end with a blank line, and with
the parameter list at column 0.

Converters

Argument Clinic contains a pre-initialized registry of converter
functions. Example converter functions:

int

    Accepts a Python object implementing __int__; emits a C int.

byte

    Accepts a Python int; emits an unsigned char. The integer must be in
    the range [0, 256).

str

    Accepts a Python str object; emits a C char *. Automatically encodes
    the string using the ascii codec.

PyObject

    Accepts any object; emits a C PyObject * without any conversion.

All converters accept the following parameters:

doc_default

    The Python value to use in place of the parameter's actual default
    in Python contexts. In other words: when specified, this value will
    be used for the parameter's default in the docstring, and in the
    Signature. (TBD alternative semantics: If the string is a valid
    Python expression which can be rendered into a Python value using
    eval(), then the result of eval() on it will be used as the default
    in the Signature.) Ignored if there is no default.

required

    Normally any parameter that has a default value is automatically
    optional. A parameter that has "required" set will be considered
    required (non-optional) even if it has a default value. The
    generated documentation will also not show any default value.

Additionally, converters may accept one or more of these optional
parameters, on an individual basis:

annotation

    Explicitly specifies the per-parameter annotation for this
    parameter. Normally it's the responsibility of the conversion
    function to generate the annotation (if any).

bitwise

    For converters that accept unsigned integers. If the Python integer
    passed in is signed, copy the bits directly even if it is negative.

encoding

    For converters that accept str. Encoding to use when encoding a
    Unicode string to a char *.

immutable

    Only accept immutable values.

length

    For converters that accept iterable types. Requests that the
    converter also emit the length of the iterable, passed in to the
    _impl function in a Py_ssize_t variable; its name will be this
    parameter's name appended with "_length".

nullable

    This converter normally does not accept None, but in this case it
    should. If None is supplied on the Python side, the equivalent C
    argument will be NULL. (The _impl argument emitted by this converter
    will presumably be a pointer type.)

types

    A list of strings representing acceptable Python types for this
    object. There are also four strings which represent Python
    protocols:

    -   "buffer"
    -   "mapping"
    -   "number"
    -   "sequence"

zeroes

    For converters that accept string types. The converted value should
    be allowed to have embedded zeroes.

Return Converters

A return converter conceptually performs the inverse operation of a
converter: it converts a native C value into its equivalent Python
value.

Directives

Argument Clinic also permits "directives" in Clinic code blocks.
Directives are similar to pragmas in C; they are statements that modify
Argument Clinic's behavior.

The format of a directive is as follows:

    directive_name [argument [second_argument [ ... ]]]

Directives only take positional arguments.

A Clinic code block must contain either one or more directives, or a
function declaration. It may contain both, in which case all directives
must come before the function declaration.

Internally directives map directly to Python callables. The directive's
arguments are passed directly to the callable as positional arguments of
type str().

Example possible directives include the production, suppression, or
redirection of Clinic output. Also, the "module" and "class" keywords
are implemented as directives in the prototype.

Python Code

Argument Clinic also permits embedding Python code inside C files, which
is executed in-place when Argument Clinic processes the file. Embedded
code looks like this:

    /*[python]

    # this is python code!
    print("/" + "* Hello world! *" + "/")

    [python]*/
    /* Hello world! */
    /*[python end:da39a3ee5e6b4b0d3255bfef95601890afd80709]*/

The "/* Hello world! */" line above was generated by running the Python
code in the preceding comment.

Any Python code is valid. Python code sections in Argument Clinic can
also be used to directly interact with Clinic; see Argument Clinic
Programmatic Interfaces.

Output

Argument Clinic writes its output inline in the C file, immediately
after the section of Clinic code. For "python" sections, the output is
everything printed using builtins.print. For "clinic" sections, the
output is valid C code, including:

-   a #define providing the correct methoddef structure for the function
-   a prototype for the "impl" function -- this is what you'll write to
    implement this function
-   a function that handles all argument processing, which calls your
    "impl" function
-   the definition line of the "impl" function
-   and a comment indicating the end of output.

The intention is that you write the body of your impl function
immediately after the output -- as in, you write a left-curly-brace
immediately after the end-of-output comment and implement builtin in the
body there. (It's a bit strange at first, but oddly convenient.)

Argument Clinic will define the parameters of the impl function for you.
The function will take the "self" parameter passed in originally, all
the parameters you define, and possibly some extra generated parameters
("length" parameters; also "group" parameters, see next section).

Argument Clinic also writes a checksum for the output section. This is a
valuable safety feature: if you modify the output by hand, Clinic will
notice that the checksum doesn't match, and will refuse to overwrite the
file. (You can force Clinic to overwrite with the "-f" command-line
argument; Clinic will also ignore the checksums when using the "-o"
command-line argument.)

Finally, Argument Clinic can also emit the boilerplate definition of the
PyMethodDef array for the defined classes and modules.

Functions With Positional-Only Parameters

A significant fraction of Python builtins implemented in C use the older
positional-only API for processing arguments (PyArg_ParseTuple()). In
some instances, these builtins parse their arguments differently based
on how many arguments were passed in. This can provide some bewildering
flexibility: there may be groups of optional parameters, which must
either all be specified or none specified. And occasionally these groups
are on the left! (A representative example: curses.window.addch().)

Argument Clinic supports these legacy use-cases by allowing you to
specify parameters in groups. Each optional group of parameters is
marked with square brackets. Note that these groups are permitted on the
right or left of any required parameters!

The impl function generated by Clinic will add an extra parameter for
every group, "int group_{left|right}_<x>", where x is a monotonically
increasing number assigned to each group as it builds away from the
required arguments. This argument will be nonzero if the group was
specified on this call, and zero if it was not.

Note that when operating in this mode, you cannot specify default
arguments.

Also, note that it's possible to specify a set of groups to a function
such that there are several valid mappings from the number of arguments
to a valid set of groups. If this happens, Clinic will abort with an
error message. This should not be a problem, as positional-only
operation is only intended for legacy use cases, and all the legacy
functions using this quirky behavior have unambiguous mappings.

Current Status

As of this writing, there is a working prototype implementation of
Argument Clinic available online (though the syntax may be out of date
as you read this).[6] The prototype generates code using the existing
PyArg_Parse APIs. It supports translating to all current format units
except the mysterious "w*". Sample functions using Argument Clinic
exercise all major features, including positional-only argument parsing.

Argument Clinic Programmatic Interfaces

The prototype also currently provides an experimental extension
mechanism, allowing adding support for new types on-the-fly. See
Modules/posixmodule.c in the prototype for an example of its use.

In the future, Argument Clinic is expected to be automatable enough to
allow querying, modification, or outright new construction of function
declarations through Python code. It may even permit dynamically adding
your own custom DSL!

Notes / TBD

-   The API for supplying inspect.Signature metadata for builtins is
    currently under discussion. Argument Clinic will add support for the
    prototype when it becomes viable.

-   Alyssa Coghlan suggests that we a) only support at most one
    left-optional group per function, and b) in the face of ambiguity,
    prefer the left group over the right group. This would solve all our
    existing use cases including range().

-   Optimally we'd want Argument Clinic run automatically as part of the
    normal Python build process. But this presents a bootstrapping
    problem; if you don't have a system Python 3, you need a Python 3
    executable to build Python 3. I'm sure this is a solvable problem,
    but I don't know what the best solution might be. (Supporting this
    will also require a parallel solution for Windows.)

-   On a related note: inspect.Signature has no way of representing
    blocks of arguments, like the left-optional block of y and x for
    curses.window.addch. How far are we going to go in supporting this
    admittedly aberrant parameter paradigm?

-   During the PyCon US 2013 Language Summit, there was discussion of
    having Argument Clinic also generate the actual documentation (in
    ReST, processed by Sphinx) for the function. The logistics of this
    are TBD, but it would require that the docstrings be written in
    ReST, and require that Python ship a ReST -> ascii converter. It
    would be best to come to a decision about this before we begin any
    large-scale conversion of the CPython source tree to using Clinic.

-   Guido proposed having the "function docstring" be hand-written
    inline, in the middle of the output, something like this:

        /*[clinic]
          ... prototype and parameters (including parameter docstrings) go here
        [clinic]*/
        ... some output ...
        /*[clinic docstring start]*/
        ... hand-edited function docstring goes here   <-- you edit this by hand!
        /*[clinic docstring end]*/
        ... more output
        /*[clinic output end]*/

    I tried it this way and don't like it -- I think it's clumsy. I
    prefer that everything you write goes in one place, rather than
    having an island of hand-edited stuff in the middle of the DSL
    output.

-   Argument Clinic does not support automatic tuple unpacking (the
    "(OOO)" style format string for PyArg_ParseTuple().)

-   Argument Clinic removes some dynamism / flexibility. With
    PyArg_ParseTuple() one could theoretically pass in different
    encodings at runtime for the "es"/"et" format units. AFAICT CPython
    doesn't do this itself, however it's possible external users might
    do this. (Trivia: there are no uses of "es" exercised by regrtest,
    and all the uses of "et" exercised are in socketmodule.c, except for
    one in _ssl.c. They're all static, specifying the encoding "idna".)

Acknowledgements

The PEP author wishes to thank Ned Batchelder for permission to
shamelessly rip off his clever design for Cog--"my favorite tool that
I've never gotten to use". Thanks also to everyone who provided feedback
on the [bugtracker issue] and on python-dev. Special thanks to Alyssa
(Nick) Coghlan and Guido van Rossum for a rousing two-hour in-person
deep dive on the topic at PyCon US 2013.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

Cog

    Cog: http://nedbatchelder.com/code/cog/

[1] PyArg_ParseTuple():
http://docs.python.org/3/c-api/arg.html#PyArg_ParseTuple

[2] PyArg_ParseTupleAndKeywords():
http://docs.python.org/3/c-api/arg.html#PyArg_ParseTupleAndKeywords

[3] PyArg_ format units:
http://docs.python.org/3/c-api/arg.html#strings-and-buffers

[4] Keyword parameters for extension functions:
http://docs.python.org/3/extending/extending.html#keyword-parameters-for-extension-functions

[5] Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

[6] Argument Clinic prototype:
https://bitbucket.org/larry/python-clinic/


text:
PEP: 586 Title: Literal Types Author: Michael Lee
<EMAIL_ADDRESS>Ivan Levkivskyi<EMAIL_ADDRESS>Jukka Lehtosalo<EMAIL_ADDRESS>BDFL-Delegate: Guido van Rossum
<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final
Type: Standards Track Topic: Typing Created: 14-Mar-2019 Python-Version:
3.8 Post-History: 14-Mar-2019 Resolution:
<EMAIL_ADDRESS>
typing:literal-types and typing.Literal

Abstract

This PEP proposes adding Literal types to the PEP 484 ecosystem. Literal
types indicate that some expression has literally a specific value. For
example, the following function will accept only expressions that have
literally the value "4":

    from typing import Literal

    def accepts_only_four(x: Literal[4]) -> None:
        pass

    accepts_only_four(4)   # OK
    accepts_only_four(19)  # Rejected

Motivation and Rationale

Python has many APIs that return different types depending on the value
of some argument provided. For example:

-   open(filename, mode) returns either IO[bytes] or IO[Text] depending
    on whether the second argument is something like r or rb.
-   subprocess.check_output(...) returns either bytes or text depending
    on whether the universal_newlines keyword argument is set to True or
    not.

This pattern is also fairly common in many popular 3rd party libraries.
For example, here are just two examples from pandas and numpy
respectively:

-   pandas.concat(...) will return either Series or DataFrame depending
    on whether the axis argument is set to 0 or 1.
-   numpy.unique will return either a single array or a tuple containing
    anywhere from two to four arrays depending on three boolean flag
    values.

The typing issue tracker contains some additional examples and
discussion.

There is currently no way of expressing the type signatures of these
functions: PEP 484 does not include any mechanism for writing signatures
where the return type varies depending on the value passed in. Note that
this problem persists even if we redesign these APIs to instead accept
enums: MyEnum.FOO and MyEnum.BAR are both considered to be of type
MyEnum.

Currently, type checkers work around this limitation by adding ad hoc
extensions for important builtins and standard library functions. For
example, mypy comes bundled with a plugin that attempts to infer more
precise types for open(...). While this approach works for standard
library functions, it’s unsustainable in general: it’s not reasonable to
expect 3rd party library authors to maintain plugins for N different
type checkers.

We propose adding Literal types to address these gaps.

Core Semantics

This section outlines the baseline behavior of literal types.

Core behavior

Literal types indicate that a variable has a specific and concrete
value. For example, if we define some variable foo to have type
Literal[3], we are declaring that foo must be exactly equal to 3 and no
other value.

Given some value v that is a member of type T, the type Literal[v] shall
be treated as a subtype of T. For example, Literal[3] is a subtype of
int.

All methods from the parent type will be directly inherited by the
literal type. So, if we have some variable foo of type Literal[3] it’s
safe to do things like foo + 5 since foo inherits int’s __add__ method.
The resulting type of foo + 5 is int.

This "inheriting" behavior is identical to how we
handle NewTypes <484#newtype-helper-function>.

Equivalence of two Literals

Two types Literal[v1] and Literal[v2] are equivalent when both of the
following conditions are true:

1.  type(v1) == type(v2)
2.  v1 == v2

For example, Literal[20] and Literal[0x14] are equivalent. However,
Literal[0] and Literal[False] is not equivalent despite that 0 == False
evaluates to 'true' at runtime: 0 has type int and False has type bool.

Shortening unions of literals

Literals are parameterized with one or more values. When a Literal is
parameterized with more than one value, it's treated as exactly
equivalent to the union of those types. That is, Literal[v1, v2, v3] is
equivalent to Union[Literal[v1], Literal[v2], Literal[v3]].

This shortcut helps make writing signatures for functions that accept
many different literals more ergonomic — for example, functions like
open(...):

    # Note: this is a simplification of the true type signature.
    _PathType = Union[str, bytes, int]

    @overload
    def open(path: _PathType,
             mode: Literal["r", "w", "a", "x", "r+", "w+", "a+", "x+"],
             ) -> IO[Text]: ...
    @overload
    def open(path: _PathType,
             mode: Literal["rb", "wb", "ab", "xb", "r+b", "w+b", "a+b", "x+b"],
             ) -> IO[bytes]: ...

    # Fallback overload for when the user isn't using literal types
    @overload
    def open(path: _PathType, mode: str) -> IO[Any]: ...

The provided values do not all have to be members of the same type. For
example, Literal[42, "foo", True] is a legal type.

However, Literal must be parameterized with at least one type. Types
like Literal[] or Literal are illegal.

Legal and illegal parameterizations

This section describes what exactly constitutes a legal Literal[...]
type: what values may and may not be used as parameters.

In short, a Literal[...] type may be parameterized by one or more
literal expressions, and nothing else.

Legal parameters for Literal at type check time

Literal may be parameterized with literal ints, byte and unicode
strings, bools, Enum values and None. So for example, all of the
following would be legal:

    Literal[26]
    Literal[0x1A]  # Exactly equivalent to Literal[26]
    Literal[-4]
    Literal["hello world"]
    Literal[b"hello world"]
    Literal[u"hello world"]
    Literal[True]
    Literal[Color.RED]  # Assuming Color is some enum
    Literal[None]

Note: Since the type None is inhabited by just a single value, the types
None and Literal[None] are exactly equivalent. Type checkers may
simplify Literal[None] into just None.

Literal may also be parameterized by other literal types, or type
aliases to other literal types. For example, the following is legal:

    ReadOnlyMode         = Literal["r", "r+"]
    WriteAndTruncateMode = Literal["w", "w+", "wt", "w+t"]
    WriteNoTruncateMode  = Literal["r+", "r+t"]
    AppendMode           = Literal["a", "a+", "at", "a+t"]

    AllModes = Literal[ReadOnlyMode, WriteAndTruncateMode,
                       WriteNoTruncateMode, AppendMode]

This feature is again intended to help make using and reusing literal
types more ergonomic.

Note: As a consequence of the above rules, type checkers are also
expected to support types that look like the following:

    Literal[Literal[Literal[1, 2, 3], "foo"], 5, None]

This should be exactly equivalent to the following type:

    Literal[1, 2, 3, "foo", 5, None]

...and also to the following type:

    Optional[Literal[1, 2, 3, "foo", 5]]

Note: String literal types like Literal["foo"] should subtype either
bytes or unicode in the same way regular string literals do at runtime.

For example, in Python 3, the type Literal["foo"] is equivalent to
Literal[u"foo"], since "foo" is equivalent to u"foo" in Python 3.

Similarly, in Python 2, the type Literal["foo"] is equivalent to
Literal[b"foo"] -- unless the file includes a
from __future__ import unicode_literals import, in which case it would
be equivalent to Literal[u"foo"].

Illegal parameters for Literal at type check time

The following parameters are intentionally disallowed by design:

-   Arbitrary expressions like Literal[3 + 4] or
    Literal["foo".replace("o", "b")].
    -   Rationale: Literal types are meant to be a minimal extension to
        the PEP 484 typing ecosystem and requiring type checkers to
        interpret potentially expressions inside types adds too much
        complexity. Also see Rejected or out-of-scope ideas.
    -   As a consequence, complex numbers like Literal[4 + 3j] and
        Literal[-4 + 2j] are also prohibited. For consistency, literals
        like Literal[4j] that contain just a single complex number are
        also prohibited.
    -   The only exception to this rule is the unary - (minus) for ints:
        types like Literal[-5] are accepted.
-   Tuples containing valid literal types like
    Literal[(1, "foo", "bar")]. The user could always express this type
    as Tuple[Literal[1], Literal["foo"], Literal["bar"]] instead. Also,
    tuples are likely to be confused with the Literal[1, 2, 3] shortcut.
-   Mutable literal data structures like dict literals, list literals,
    or set literals: literals are always implicitly final and immutable.
    So, Literal[{"a": "b", "c": "d"}] is illegal.
-   Any other types: for example, Literal[Path], or
    Literal[some_object_instance] are illegal. This includes typevars:
    if T is a typevar, Literal[T] is not allowed. Typevars can vary over
    only types, never over values.

The following are provisionally disallowed for simplicity. We can
consider allowing them in future extensions of this PEP.

-   Floats: e.g. Literal[3.14]. Representing Literals of infinity or NaN
    in a clean way is tricky; real-world APIs are unlikely to vary their
    behavior based on a float parameter.
-   Any: e.g. Literal[Any]. Any is a type, and Literal[...] is meant to
    contain values only. It is also unclear what Literal[Any] would
    actually semantically mean.

Parameters at runtime

Although the set of parameters Literal[...] may contain at type check
time is very small, the actual implementation of typing.Literal will not
perform any checks at runtime. For example:

    def my_function(x: Literal[1 + 2]) -> int:
        return x * 3

    x: Literal = 3
    y: Literal[my_function] = my_function

The type checker should reject this program: all three uses of Literal
are invalid according to this spec. However, Python itself should
execute this program with no errors.

This is partly to help us preserve flexibility in case we want to expand
the scope of what Literal can be used for in the future, and partly
because it is not possible to detect all illegal parameters at runtime
to begin with. For example, it is impossible to distinguish between
Literal[1 + 2] and Literal[3] at runtime.

Literals, enums, and forward references

One potential ambiguity is between literal strings and forward
references to literal enum members. For example, suppose we have the
type Literal["Color.RED"]. Does this literal type contain a string
literal or a forward reference to some Color.RED enum member?

In cases like these, we always assume the user meant to construct a
literal string. If the user wants a forward reference, they must wrap
the entire literal type in a string -- e.g. "Literal[Color.RED]".

Type inference

This section describes a few rules regarding type inference and
literals, along with some examples.

Backwards compatibility

When type checkers add support for Literal, it's important they do so in
a way that maximizes backwards-compatibility. Type checkers should
ensure that code that used to type check continues to do so after
support for Literal is added on a best-effort basis.

This is particularly important when performing type inference. For
example, given the statement x = "blue", should the inferred type of x
be str or Literal["blue"]?

One naive strategy would be to always assume expressions are intended to
be Literal types. So, x would always have an inferred type of
Literal["blue"] in the example above. This naive strategy is almost
certainly too disruptive -- it would cause programs like the following
to start failing when they previously did not:

    # If a type checker infers 'var' has type Literal[3]
    # and my_list has type List[Literal[3]]...
    var = 3
    my_list = [var]

    # ...this call would be a type-error.
    my_list.append(4)

Another example of when this strategy would fail is when setting fields
in objects:

    class MyObject:
        def __init__(self) -> None:
            # If a type checker infers MyObject.field has type Literal[3]...
            self.field = 3

    m = MyObject()

    # ...this assignment would no longer type check
    m.field = 4

An alternative strategy that does maintain compatibility in every case
would be to always assume expressions are not Literal types unless they
are explicitly annotated otherwise. A type checker using this strategy
would always infer that x is of type str in the first example above.

This is not the only viable strategy: type checkers should feel free to
experiment with more sophisticated inference techniques. This PEP does
not mandate any particular strategy; it only emphasizes the importance
of backwards compatibility.

Using non-Literals in Literal contexts

Literal types follow the existing rules regarding subtyping with no
additional special-casing. For example, programs like the following are
type safe:

    def expects_str(x: str) -> None: ...
    var: Literal["foo"] = "foo"

    # Legal: Literal["foo"] is a subtype of str
    expects_str(var)

This also means non-Literal expressions in general should not
automatically be cast to Literal. For example:

    def expects_literal(x: Literal["foo"]) -> None: ...

    def runner(my_str: str) -> None:
        # ILLEGAL: str is not a subclass of Literal["foo"]
        expects_literal(my_str)

Note: If the user wants their API to support accepting both literals and
the original type -- perhaps for legacy purposes -- they should
implement a fallback overload. See Interactions with overloads.

Interactions with other types and features

This section discusses how Literal types interact with other existing
types.

Intelligent indexing of structured data

Literals can be used to "intelligently index" into structured types like
tuples, NamedTuple, and classes. (Note: this is not an exhaustive list).

For example, type checkers should infer the correct value type when
indexing into a tuple using an int key that corresponds a valid index:

    a: Literal[0] = 0
    b: Literal[5] = 5

    some_tuple: Tuple[int, str, List[bool]] = (3, "abc", [True, False])
    reveal_type(some_tuple[a])   # Revealed type is 'int'
    some_tuple[b]                # Error: 5 is not a valid index into the tuple

We expect similar behavior when using functions like getattr:

    class Test:
        def __init__(self, param: int) -> None:
            self.myfield = param

        def mymethod(self, val: int) -> str: ...

    a: Literal["myfield"]  = "myfield"
    b: Literal["mymethod"] = "mymethod"
    c: Literal["blah"]     = "blah"

    t = Test()
    reveal_type(getattr(t, a))  # Revealed type is 'int'
    reveal_type(getattr(t, b))  # Revealed type is 'Callable[[int], str]'
    getattr(t, c)               # Error: No attribute named 'blah' in Test

Note: See Interactions with Final for a proposal on how we can express
the variable declarations above in a more compact manner.

Interactions with overloads

Literal types and overloads do not need to interact in a special way:
the existing rules work fine.

However, one important use case type checkers must take care to support
is the ability to use a fallback when the user is not using literal
types. For example, consider open:

    _PathType = Union[str, bytes, int]

    @overload
    def open(path: _PathType,
             mode: Literal["r", "w", "a", "x", "r+", "w+", "a+", "x+"],
             ) -> IO[Text]: ...
    @overload
    def open(path: _PathType,
             mode: Literal["rb", "wb", "ab", "xb", "r+b", "w+b", "a+b", "x+b"],
             ) -> IO[bytes]: ...

    # Fallback overload for when the user isn't using literal types
    @overload
    def open(path: _PathType, mode: str) -> IO[Any]: ...

If we were to change the signature of open to use just the first two
overloads, we would break any code that does not pass in a literal
string expression. For example, code like this would be broken:

    mode: str = pick_file_mode(...)
    with open(path, mode) as f:
        # f should continue to be of type IO[Any] here

A little more broadly: we propose adding a policy to typeshed that
mandates that whenever we add literal types to some existing API, we
also always include a fallback overload to maintain
backwards-compatibility.

Interactions with generics

Types like Literal[3] are meant to be just plain old subclasses of int.
This means you can use types like Literal[3] anywhere you could use
normal types, such as with generics.

This means that it is legal to parameterize generic functions or classes
using Literal types:

    A = TypeVar('A', bound=int)
    B = TypeVar('B', bound=int)
    C = TypeVar('C', bound=int)

    # A simplified definition for Matrix[row, column]
    class Matrix(Generic[A, B]):
        def __add__(self, other: Matrix[A, B]) -> Matrix[A, B]: ...
        def __matmul__(self, other: Matrix[B, C]) -> Matrix[A, C]: ...
        def transpose(self) -> Matrix[B, A]: ...

    foo: Matrix[Literal[2], Literal[3]] = Matrix(...)
    bar: Matrix[Literal[3], Literal[7]] = Matrix(...)

    baz = foo @ bar
    reveal_type(baz)  # Revealed type is 'Matrix[Literal[2], Literal[7]]'

Similarly, it is legal to construct TypeVars with value restrictions or
bounds involving Literal types:

    T = TypeVar('T', Literal["a"], Literal["b"], Literal["c"])
    S = TypeVar('S', bound=Literal["foo"])

...although it is unclear when it would ever be useful to construct a
TypeVar with a Literal upper bound. For example, the S TypeVar in the
above example is essentially pointless: we can get equivalent behavior
by using S = Literal["foo"] instead.

Note: Literal types and generics deliberately interact in only very
basic and limited ways. In particular, libraries that want to type check
code containing a heavy amount of numeric or numpy-style manipulation
will almost certainly likely find Literal types as proposed in this PEP
to be insufficient for their needs.

We considered several different proposals for fixing this, but
ultimately decided to defer the problem of integer generics to a later
date. See Rejected or out-of-scope ideas for more details.

Interactions with enums and exhaustiveness checks

Type checkers should be capable of performing exhaustiveness checks when
working Literal types that have a closed number of variants, such as
enums. For example, the type checker should be capable of inferring that
the final else statement must be of type str, since all three values of
the Status enum have already been exhausted:

    class Status(Enum):
        SUCCESS = 0
        INVALID_DATA = 1
        FATAL_ERROR = 2

    def parse_status(s: Union[str, Status]) -> None:
        if s is Status.SUCCESS:
            print("Success!")
        elif s is Status.INVALID_DATA:
            print("The given data is invalid because...")
        elif s is Status.FATAL_ERROR:
            print("Unexpected fatal error...")
        else:
            # 's' must be of type 'str' since all other options are exhausted
            print("Got custom status: " + s)

The interaction described above is not new: it's already
codified within PEP 484 <484#support-for-singleton-types-in-unions>.
However, many type checkers (such as mypy) do not yet implement this due
to the expected complexity of the implementation work.

Some of this complexity will be alleviated once Literal types are
introduced: rather than entirely special-casing enums, we can instead
treat them as being approximately equivalent to the union of their
values and take advantage of any existing logic regarding unions,
exhaustibility, type narrowing, reachability, and so forth the type
checker might have already implemented.

So here, the Status enum could be treated as being approximately
equivalent to
Literal[Status.SUCCESS, Status.INVALID_DATA, Status.FATAL_ERROR] and the
type of s narrowed accordingly.

Interactions with narrowing

Type checkers may optionally perform additional analysis for both enum
and non-enum Literal types beyond what is described in the section
above.

For example, it may be useful to perform narrowing based on things like
containment or equality checks:

    def parse_status(status: str) -> None:
        if status in ("MALFORMED", "ABORTED"):
            # Type checker could narrow 'status' to type
            # Literal["MALFORMED", "ABORTED"] here.
            return expects_bad_status(status)

        # Similarly, type checker could narrow 'status' to Literal["PENDING"]
        if status == "PENDING":
            expects_pending_status(status)

It may also be useful to perform narrowing taking into account
expressions involving Literal bools. For example, we can combine
Literal[True], Literal[False], and overloads to construct "custom type
guards":

    @overload
    def is_int_like(x: Union[int, List[int]]) -> Literal[True]: ...
    @overload
    def is_int_like(x: object) -> bool: ...
    def is_int_like(x): ...

    vector: List[int] = [1, 2, 3]
    if is_int_like(vector):
        vector.append(3)
    else:
        vector.append("bad")   # This branch is inferred to be unreachable

    scalar: Union[int, str]
    if is_int_like(scalar):
        scalar += 3      # Type checks: type of 'scalar' is narrowed to 'int'
    else:
        scalar += "foo"  # Type checks: type of 'scalar' is narrowed to 'str'

Interactions with Final

PEP 591 proposes adding a "Final" qualifier to the typing ecosystem.
This qualifier can be used to declare that some variable or attribute
cannot be reassigned:

    foo: Final = 3
    foo = 4           # Error: 'foo' is declared to be Final

Note that in the example above, we know that foo will always be equal to
exactly 3. A type checker can use this information to deduce that foo is
valid to use in any context that expects a Literal[3]:

    def expects_three(x: Literal[3]) -> None: ...

    expects_three(foo)  # Type checks, since 'foo' is Final and equal to 3

The Final qualifier serves as a shorthand for declaring that a variable
is effectively Literal.

If both this PEP and PEP 591 are accepted, type checkers are expected to
support this shortcut. Specifically, given a variable or attribute
assignment of the form var: Final = value where value is a valid
parameter for Literal[...], type checkers should understand that var may
be used in any context that expects a Literal[value].

Type checkers are not obligated to understand any other uses of Final.
For example, whether or not the following program type checks is left
unspecified:

    # Note: The assignment does not exactly match the form 'var: Final = value'.
    bar1: Final[int] = 3
    expects_three(bar1)  # May or may not be accepted by type checkers

    # Note: "Literal[1 + 2]" is not a legal type.
    bar2: Final = 1 + 2
    expects_three(bar2)  # May or may not be accepted by type checkers

Rejected or out-of-scope ideas

This section outlines some potential features that are explicitly
out-of-scope.

True dependent types/integer generics

This proposal is essentially describing adding a very simplified
dependent type system to the PEP 484 ecosystem. One obvious extension
would be to implement a full-fledged dependent type system that lets
users predicate types based on their values in arbitrary ways. That
would let us write signatures like the below:

    # A vector has length 'n', containing elements of type 'T'
    class Vector(Generic[N, T]): ...

    # The type checker will statically verify our function genuinely does
    # construct a vector that is equal in length to "len(vec1) + len(vec2)"
    # and will throw an error if it does not.
    def concat(vec1: Vector[A, T], vec2: Vector[B, T]) -> Vector[A + B, T]:
        # ...snip...

At the very least, it would be useful to add some form of integer
generics.

Although such a type system would certainly be useful, it’s out of scope
for this PEP: it would require a far more substantial amount of
implementation work, discussion, and research to complete compared to
the current proposal.

It's entirely possible we'll circle back and revisit this topic in the
future: we very likely will need some form of dependent typing along
with other extensions like variadic generics to support popular
libraries like numpy.

This PEP should be seen as a stepping stone towards this goal, rather
than an attempt at providing a comprehensive solution.

Adding more concise syntax

One objection to this PEP is that having to explicitly write
Literal[...] feels verbose. For example, instead of writing:

    def foobar(arg1: Literal[1], arg2: Literal[True]) -> None:
        pass

...it would be nice to instead write:

    def foobar(arg1: 1, arg2: True) -> None:
        pass

Unfortunately, these abbreviations simply will not work with the
existing implementation of typing at runtime. For example, the following
snippet crashes when run using Python 3.7:

    from typing import Tuple

    # Supposed to accept tuple containing the literals 1 and 2
    def foo(x: Tuple[1, 2]) -> None:
        pass

Running this yields the following exception:

    TypeError: Tuple[t0, t1, ...]: each t must be a type. Got 1.

We don’t want users to have to memorize exactly when it’s ok to elide
Literal, so we require Literal to always be present.

A little more broadly, we feel overhauling the syntax of types in Python
is not within the scope of this PEP: it would be best to have that
discussion in a separate PEP, instead of attaching it to this one. So,
this PEP deliberately does not try and innovate Python's type syntax.

Backporting the Literal type

Once this PEP is accepted, the Literal type will need to be backported
for Python versions that come bundled with older versions of the typing
module. We plan to do this by adding Literal to the typing_extensions
3rd party module, which contains a variety of other backported types.

Implementation

The mypy type checker currently has implemented a large subset of the
behavior described in this spec, with the exception of enum Literals and
some of the more complex narrowing interactions described above.

Related work

This proposal was written based on the discussion that took place in the
following threads:

-   Check that literals belong to/are excluded from a set of values
-   Simple dependent types
-   Typing for multi-dimensional arrays

The overall design of this proposal also ended up converging into
something similar to how literal types are handled in TypeScript.

Acknowledgements

Thanks to Mark Mendoza, Ran Benita, Rebecca Chen, and the other members
of typing-sig for their comments on this PEP.

Additional thanks to the various participants in the mypy and typing
issue trackers, who helped provide a lot of the motivation and reasoning
behind this PEP.

Copyright

This document has been placed in the public domain.

authors:
Ivan Levkivskyi
Jukka Lehtosalo
Michael Lee


text:
PEP: 387 Title: Backwards Compatibility Policy Author: Benjamin Peterson
<EMAIL_ADDRESS>PEP-Delegate: Brett Cannon<EMAIL_ADDRESS>Status: Active Type: Process Created: 18-Jun-2009 Post-History:
19-Jun-2009, 12-Jun-2020, 19-Dec-2022, 16-Jun-2023 Replaces: 291

Abstract

This PEP outlines Python's backwards compatibility policy.

Rationale

As one of the most used programming languages today[1], the Python core
language and its standard library play a critical role in millions of
applications and libraries. This is fantastic. However, it means the
development team must be very careful not to break this existing 3rd
party code with new releases.

This PEP takes the perspective that "backwards incompatibility" means
preexisting code ceases to comparatively function after a change. It is
acknowledged that this is not a concrete definition, but the expectation
is people in general understand what is meant by "backwards
incompatibility", and if they are unsure they may ask the Python
development team and/or steering council for guidance.

Backwards Compatibility Rules

This policy applies to all public APIs. These include:

-   Syntax and behavior of these constructs as defined by the reference
    manual.
-   The C-API.
-   Function, class, module, attribute, and method names and types.
-   Given a set of arguments, the return value, side effects, and raised
    exceptions of a function. This does not preclude changes from
    reasonable bug fixes.
-   The position and expected types of arguments and returned values.
-   Behavior of classes with regards to subclasses: the conditions under
    which overridden methods are called.
-   Documented exceptions and the semantics which lead to their raising.
-   Exceptions commonly raised in EAFP scenarios.

Others are explicitly not part of the public API. They can change or be
removed at any time in any way. These include:

-   Function, class, module, attribute, method, and C-API names and
    types that are prefixed by "_" (except special names).
-   Anything documented publicly as being private. Note that if
    something is not documented at all, it is not automatically
    considered private.
-   Imported modules (unless explicitly documented as part of the public
    API; e.g. importing the bacon module in the spam does not
    automatically mean spam.bacon is part of the public API unless it is
    documented as such).
-   Inheritance patterns of internal classes.
-   Test suites. (Anything in the Lib/test directory or test
    subdirectories of packages.)
-   Backward compatibility rules do not apply to any module or API that
    is explicitly documented as Provisional per PEP 411.

Basic policy for backwards compatibility

-   In general, incompatibilities should have a large benefit to
    breakage ratio, and the incompatibility should be easy to resolve in
    affected code. For example, adding an stdlib module with the same
    name as a third party package is generally not acceptable. Adding a
    method or attribute that conflicts with 3rd party code through
    inheritance, however, is likely reasonable.
-   Unless it is going through the deprecation process below, the
    behavior of an API must not change in an incompatible fashion
    between any two consecutive releases. Python's yearly release
    process (PEP 602) means that the deprecation period must last at
    least two years.
-   Similarly a feature cannot be removed without notice between any two
    consecutive releases.
-   For changes that are unable to raise a deprecation warning, consult
    with the steering council.
-   The steering council may grant exceptions to this policy. In
    particular, they may shorten the required deprecation period for a
    feature. Exceptions are only granted for extreme situations such as
    dangerously broken or insecure features or features no one could
    reasonably be depending on (e.g., support for completely obsolete
    platforms).

Soft Deprecation

A soft deprecation can be used when using an API which should no longer
be used to write new code, but it remains safe to continue using it in
existing code. The API remains documented and tested, but will not be
developed further (no enhancement).

The main difference between a "soft" and a (regular) "hard" deprecation
is that the soft deprecation does not imply scheduling the removal of
the deprecated API.

Another difference is that a soft deprecation does not issue a warning:
it's only mentioned in the documentation, whereas usually a "hard"
deprecation issues a DeprecationWarning warning at runtime. The
documentation of a soft deprecation should explain why the API should be
avoided, and if possible propose a replacement.

If the decision is made to deprecate (in the regular sense) a feature
that is currently soft deprecated, the deprecation must follow the
Backwards Compatibility Rules (i.e., there is no exception because the
feature is already soft deprecated).

Making Incompatible Changes

Making an incompatible change is a gradual process performed over
several releases:

1.  Discuss the change. Depending on the degree of incompatibility, this
    could be on the bug tracker, python-dev, python-list, or the
    appropriate SIG. A PEP or similar document may be written. Hopefully
    users of the affected API will pipe up to comment.

2.  Add a warning to the current main branch. If behavior is changing,
    the API may gain a new function or method to perform the new
    behavior; old usage should raise the warning. If an API is being
    removed, simply warn whenever it is entered. DeprecationWarning is
    the usual warning category to use, but PendingDeprecationWarning may
    be used in special cases where the old and new versions of the API
    will coexist for many releases[2]. The warning message should
    include the release the incompatibility is expected to become the
    default and a link to an issue that users can post feedback to. When
    feasible, also change typeshed to add the @deprecated decorator (see
    PEP 702) to the deprecated API, so that users of static type
    checkers have another way to learn about the deprecation.

    For C API, a compiler warning generated by the Py_DEPRECATED macro
    is also acceptable.

3.  Wait for the warning to appear in at least two minor Python versions
    of the same major version, or one minor version in an older major
    version (e.g. for a warning in Python 3.10.0, you either wait until
    at least Python 3.12 or Python 4.0 to make the change).

    It's fine to wait more than two releases, for example:

    -   If the expected maintenance overhead and security risk of the
        deprecated behavior is small (e.g. an old function is
        reimplemented in terms of a new, more general one), it can stay
        indefinitely (or until the situation changes).
    -   If the deprecated feature is replaced by a new one, it should
        generally be removed only after the last Python version without
        the new feature reaches end of support.

4.  See if there's any feedback. Users not involved in the original
    discussions may comment now after seeing the warning. Perhaps
    reconsider.

5.  The behavior change or feature removal may now be made default or
    permanent having reached the declared version. Remove the old
    version and warning.

6.  If a warning cannot be provided to users, consult with the steering
    council.

Changelog

-   2023-Nov-14: Added @deprecated decorator per PEP 702.
-   2023-Jul-03: Added the Soft Deprecation section, as discussed in
    https://discuss.python.org/t/27957.
-   2023-Jun-26: Multiple smaller updates and clarifications, discussed
    in https://discuss.python.org/t/22042.
-   2022-Apr-04: Added explicit notes to ask the Steering Council in
    several exceptional cases.
-   2021-Apr-16: Clarified how long a warning must be emitted before a
    change can be made.
-   2020-Jul-20: Initial accepted version.

References

Copyright

This document has been placed in the public domain.

[1] TIOBE Programming Community Index

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

[2] The warnings module

http://docs.python.org/library/warnings.html