text:
PEP: 735 Title: Dependency Groups in pyproject.toml Author: Stephen
Rosen<EMAIL_ADDRESS>Sponsor: Brett Cannon<EMAIL_ADDRESS>PEP-Delegate: Paul Moore<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/39233 Status: Accepted Type: Standards
Track Topic: Packaging Created: 20-Nov-2023 Post-History: 14-Nov-2023,
20-Nov-2023 Resolution: 10-Oct-2024

Abstract

This PEP specifies a mechanism for storing package requirements in
pyproject.toml files such that they are not included in any built
distribution of the project.

This is suitable for creating named groups of dependencies, similar to
requirements.txt files, which launchers, IDEs, and other tools can find
and identify by name.

The feature defined here is referred to as "Dependency Groups".

Motivation

There are two major use cases for which the Python community has no
standardized answer:

-   How should development dependencies be defined for packages?
-   How should dependencies be defined for projects which do not build
    distributions (non-package projects)?

In support of these two needs, there are two common solutions which are
similar to this proposal:

-   requirements.txt files
-   package extras

Both requirements.txt files and extras have limitations which this
standard seeks to overcome.

Note that the two use cases above describe two different types of
projects which this PEP seeks to support:

-   Python packages, such as libraries
-   non-package projects, such as data science projects

Several motivating use cases are defined in detail in the
Use Cases Appendix <use_cases>.

Limitations of requirements.txt files

Many projects may define one or more requirements.txt files, and may
arrange them either at the project root (e.g. requirements.txt and
test-requirements.txt) or else in a directory (e.g.
requirements/base.txt and requirements/test.txt). However, there are
major issues with the use of requirements files in this way:

-   There is no standardized naming convention such that tools can
    discover or use these files by name.
-   requirements.txt files are not standardized, but instead provide
    options to pip.

As a result, it is difficult to define tool behaviors based on
requirements.txt files. They are not trivial to discover or identify by
name, and their contents may contain a mix of package specifiers and
additional pip options.

The lack of a standard for requirements.txt contents also means they are
not portable to any alternative tools which wish to process them other
than pip.

Additionally, requirements.txt files require a file per dependency list.
For some use-cases, this makes the marginal cost of dependency groupings
high, relative to their benefit. A terser declaration is beneficial to
projects with a number of small groups of dependencies.

In contrast with this, Dependency Groups are defined at a well known
location in pyproject.toml with fully standardized contents. Not only
will they have immediate utility, but they will also serve as a starting
point for future standards.

Limitations of extras

extras are additional package metadata declared in the
[project.optional-dependencies] table. They provide names for lists of
package specifiers which are published as part of a package's metadata,
and which a user can request under that name, as in
pip install 'foo[bar]' to install foo with the bar extra.

Because extras are package metadata, they are not guaranteed to be
statically defined and may require a build system to resolve.
Furthermore, definition of a [project.optional-dependencies] indicates
to many tools that a project is a package, and may drive tool behaviors
such as validation of the [project] table.

For projects which are packages, extras are a common solution for
defining development dependencies, but even under these circumstances
they have downsides:

-   Because an extra defines optional additional dependencies, it is not
    possible to install an extra without installing the current package
    and its dependencies.
-   Because they are user-installable, extras are part of the public
    interface for packages. Because extras are published, package
    developers often are concerned about ensuring that their development
    extras are not confused with user-facing extras.

Rationale

This PEP defines the storage of requirements data in lists within a
[dependency-groups] table. This name was chosen to match the canonical
name of the feature ("Dependency Groups").

This format should be as simple and learnable as possible, having a
format very similar to existing requirements.txt files for many cases.
Each list in [dependency-groups] is defined as a list of package
specifiers. For example:

    [dependency-groups]
    test = ["pytest>7", "coverage"]

There are a number of use cases for requirements.txt files which require
data which cannot be expressed in PEP 508 dependency specifiers. Such
fields are not valid in Dependency Groups. Including many of the data
and fields which pip supports, such as index servers, hashes, and path
dependencies, requires new standards. This standard leaves room for new
standards and developments, but does not attempt to support all valid
requirements.txt contents.

The only exception to this is the -r flag which requirements.txt files
use to include one file in another. Dependency Groups support an
"include" mechanism which is similar in meaning, allowing one dependency
group to extend another.

Dependency Groups have two additional features which are similar to
requirements.txt files:

-   they are not published as distinct metadata in any built
    distribution
-   installation of a dependency group does not imply installation of a
    package's dependencies or the package itself

Use Cases

The following use cases are considered important targets for this PEP.
They are defined in greater detail in the
Use Cases Appendix <use_cases>.

-   Web Applications deployed via a non-python-packaging build process
-   Libraries with unpublished dev dependency groups
-   Data science projects with groups of dependencies but no core
    package
-   Input data to lockfile generation (Dependency Groups should
    generally not be used as a location for locked dependency data)
-   Input data to an environment manager, such as tox, Nox, or Hatch
-   Configurable IDE discovery of test and linter requirements

Regarding Poetry and PDM Dependency Groups

The existing Poetry and PDM tools already offer a feature which each
calls "Dependency Groups". However, absent any standard for specifying
collections of dependencies, each tool defines these in a tool-specific
way, in the relevant sections of the [tool] table.

(PDM also uses extras for some Dependency Groups, and overlaps the
notion heavily with extras.)

This PEP does not support all of the features of Poetry and PDM, which,
like requirements.txt files for pip, support several non-standard
extensions to common dependency specifiers.

It should be possible for such tools to use standardized Dependency
Groups as extensions of their own Dependency Group mechanisms. However,
defining a new data format which replaces the existing Poetry and PDM
solutions is a non-goal. Doing so would require standardizing several
additional features, such as path dependencies, which are supported by
these tools.

Dependency Groups are not Hidden Extras

Dependency Groups are very similar to extras which go unpublished.
However, there are two major features which distinguish them from extras
further:

-   they support non-package projects
-   installation of a Dependency Group does not imply installation of a
    package's dependencies (or the package itself)

Future Compatibility & Invalid Data

Dependency Groups are intended to be extensible in future PEPs. However,
Dependency Groups should also be usable by multiple tools in a single
Python project. With multiple tools using the same data, it is possible
that one implements a future PEP which extends Dependency Groups, while
another does not.

To support users in this case, this PEP defines and recommends
validation behaviors in which tools only examine Dependency Groups which
they are using. This allows multiple tools, using different versions of
Dependency Groups data, to share a single table in pyproject.toml.

Specification

This PEP defines a new section (table) in pyproject.toml files named
dependency-groups. The dependency-groups table contains an arbitrary
number of user-defined keys, each of which has, as its value, a list of
requirements (defined below). These keys must be valid non-normalized
names, and must be normalized before comparisons.

Tools SHOULD prefer to present the original, non-normalized name to
users by default. If duplicate names, after normalization, are
encountered, tools SHOULD emit an error.

Requirement lists under dependency-groups may contain strings, tables
("dicts" in Python), or a mix of strings and tables.

Strings in requirement lists must be valid Dependency Specifiers, as
defined in PEP 508.

Tables in requirement lists must be valid Dependency Object Specifiers.

Dependency Object Specifiers

Dependency Object Specifiers are tables which define zero or more
dependencies.

This PEP standardizes only one type of Dependency Object Specifier, a
"Dependency Group Include". Other types may be added in future
standards.

Dependency Group Include

A Dependency Group Include includes the dependencies of another
Dependency Group in the current Dependency Group.

An include is defined as a table with exactly one key, "include-group",
whose value is a string, the name of another Dependency Group.

For example, {include-group = "test"} is an include which expands to the
contents of the test Dependency Group.

Includes are defined to be exactly equivalent to the contents of the
named Dependency Group, inserted into the current group at the location
of the include. For example, if foo = ["a", "b"] is one group, and
bar = ["c", {include-group = "foo"}, "d"] is another, then bar should
evaluate to ["c", "a", "b", "d"] when Dependency Group Includes are
expanded.

Dependency Group Includes may specify the same package multiple times.
Tools SHOULD NOT deduplicate or otherwise alter the list contents
produced by the include. For example, given the following table:

    [dependency-groups]
    group-a = ["foo"]
    group-b = ["foo>1.0"]
    group-c = ["foo<1.0"]
    all = ["foo", {include-group = "group-a"}, {include-group = "group-b"}, {include-group = "group-c"}]

The resolved value of all SHOULD be
["foo", "foo", "foo>1.0", "foo<1.0"]. Tools should handle such a list
exactly as they would handle any other case in which they are asked to
process the same requirement multiple times with different version
constraints.

Dependency Group Includes may include lists containing Dependency Group
Includes, in which case those includes should be expanded as well.
Dependency Group Includes MUST NOT include cycles, and tools SHOULD
report an error if they detect a cycle.

Example Dependency Groups Table

The following is an example of a partial pyproject.toml which uses this
to define four Dependency Groups: test, docs, typing, and typing-test:

    [dependency-groups]
    test = ["pytest", "coverage"]
    docs = ["sphinx", "sphinx-rtd-theme"]
    typing = ["mypy", "types-requests"]
    typing-test = [{include-group = "typing"}, {include-group = "test"}, "useful-types"]

Note that none of these Dependency Group declarations implicitly install
the current package, its dependencies, or any optional dependencies. Use
of a Dependency Group like test to test a package requires that the
user's configuration or toolchain also installs the current package (.).
For example,

    $TOOL install-dependency-group test
    pip install -e .

could be used (supposing $TOOL is a tool which supports installing
Dependency Groups) to build a testing environment.

This also allows for the docs dependency group to be used without
installing the project as a package:

    $TOOL install-dependency-group docs

Package Building

Build backends MUST NOT include Dependency Group data in built
distributions as package metadata. This means that PKG-INFO in sdists
and METADATA in wheels do not include any referencable fields containing
Dependency Groups.

It is valid to use Dependency Groups in the evaluation of dynamic
metadata, and pyproject.toml files included in sdists will naturally
still contain the [dependency-groups] table. However, the table contents
are not part of a published package's interfaces.

Installing Dependency Groups

Tools which support Dependency Groups are expected to provide new
options and interfaces to allow users to install from Dependency Groups.

No syntax is defined for expressing the Dependency Group of a package,
for two reasons:

-   it would not be valid to refer to the Dependency Groups of a
    third-party package from PyPI (because the data is defined to be
    unpublished)
-   there is not guaranteed to be a current package for Dependency
    Groups -- part of their purpose is to support non-package projects

For example, a possible pip interface for installing Dependency Groups
would be:

    pip install --dependency-groups=test,typing

Note that this is only an example. This PEP does not declare any
requirements for how tools support the installation of Dependency
Groups.

Overlapping Install UX with Extras

Tools MAY choose to provide the same interfaces for installing
Dependency Groups as they do for installing extras.

Note that this specification does not forbid having an extra whose name
matches a Dependency Group.

Users are advised to avoid creating Dependency Groups whose names match
extras. Tools MAY treat such matching as an error.

Validation and Compatibility

Tools supporting Dependency Groups may want to validate data before
using it. However, tools implementing such validation behavior should be
careful to allow for future expansions to this spec, so that they do not
unnecessarily emit errors or warnings in the presence of new syntax.

Tools SHOULD error when evaluating or processing unrecognized data in
Dependency Groups.

Tools SHOULD NOT eagerly validate the list contents of all Dependency
Groups.

This means that in the presence of the following data, most tools will
allow the foo group to be used, and will only error when the bar group
is used:

    [dependency-groups]
    foo = ["pyparsing"]
    bar = [{set-phasers-to = "stun"}]

Linters and Validators may be stricter

Eager validation is discouraged for tools which primarily install or
resolve Dependency Groups. Linters and validation tools may have good
cause to ignore this recommendation.

Reference Implementation

The following Reference Implementation prints the contents of a
Dependency Group to stdout, newline delimited. The output is therefore
valid requirements.txt data.

    import re
    import sys
    import tomllib
    from collections import defaultdict

    from packaging.requirements import Requirement

    def _normalize_name(name: str) -> str:
        return re.sub(r"[-_.]+", "-", name).lower()

    def _normalize_group_names(dependency_groups: dict) -> dict:
        original_names = defaultdict(list)
        normalized_groups = {}

        for group_name, value in dependency_groups.items():
            normed_group_name = _normalize_name(group_name)
            original_names[normed_group_name].append(group_name)
            normalized_groups[normed_group_name] = value

        errors = []
        for normed_name, names in original_names.items():
            if len(names) > 1:
                errors.append(f"{normed_name} ({', '.join(names)})")
        if errors:
            raise ValueError(f"Duplicate dependency group names: {', '.join(errors)}")

        return normalized_groups

    def _resolve_dependency_group(
        dependency_groups: dict, group: str, past_groups: tuple[str, ...] = ()
    ) -> list[str]:
        if group in past_groups:
            raise ValueError(f"Cyclic dependency group include: {group} -> {past_groups}")

        if group not in dependency_groups:
            raise LookupError(f"Dependency group '{group}' not found")

        raw_group = dependency_groups[group]
        if not isinstance(raw_group, list):
            raise ValueError(f"Dependency group '{group}' is not a list")

        realized_group = []
        for item in raw_group:
            if isinstance(item, str):
                # packaging.requirements.Requirement parsing ensures that this is a valid
                # PEP 508 Dependency Specifier
                # raises InvalidRequirement on failure
                Requirement(item)
                realized_group.append(item)
            elif isinstance(item, dict):
                if tuple(item.keys()) != ("include-group",):
                    raise ValueError(f"Invalid dependency group item: {item}")

                include_group = _normalize_name(next(iter(item.values())))
                realized_group.extend(
                    _resolve_dependency_group(
                        dependency_groups, include_group, past_groups + (group,)
                    )
                )
            else:
                raise ValueError(f"Invalid dependency group item: {item}")

        return realized_group

    def resolve(dependency_groups: dict, group: str) -> list[str]:
        if not isinstance(dependency_groups, dict):
            raise TypeError("Dependency Groups table is not a dict")
        if not isinstance(group, str):
            raise TypeError("Dependency group name is not a str")
        return _resolve_dependency_group(dependency_groups, group)

    if __name__ == "__main__":
        with open("pyproject.toml", "rb") as fp:
            pyproject = tomllib.load(fp)

        dependency_groups_raw = pyproject["dependency-groups"]
        dependency_groups = _normalize_group_names(dependency_groups_raw)
        print("\n".join(resolve(pyproject["dependency-groups"], sys.argv[1])))

Backwards Compatibility

At time of writing, the dependency-groups namespace within a
pyproject.toml file is unused. Since the top-level namespace is reserved
for use only by standards specified at packaging.python.org, there are
no direct backwards compatibility concerns.

However, the introduction of the feature has implications for a number
of ecosystem tools, especially those which attempt to support
examination of data in setup.py and requirements.txt.

Audit and Update Tools

A wide range of tools understand Python dependency data as expressed in
requirements.txt files. (e.g., Dependabot, Tidelift, etc)

Such tools inspect dependency data and, in some cases, offer
tool-assisted or fully automated updates. It is our expectation that no
such tools would support the new Dependency Groups at first, and broad
ecosystem support could take many months or even some number of years to
arrive.

As a result, users of Dependency Groups would experience a degradation
in their workflows and tool support at the time that they start using
Dependency Groups. This is true of any new standard for where and how
dependency data are encoded.

Security Implications

This PEP introduces new syntaxes and data formats for specifying
dependency information in projects. However, it does not introduce newly
specified mechanisms for handling or resolving dependencies.

It therefore does not carry security concerns other than those inherent
in any tools which may already be used to install dependencies -- i.e.
malicious dependencies may be specified here, just as they may be
specified in requirements.txt files.

How to Teach This

This feature should be referred to by its canonical name, "Dependency
Groups".

The basic form of usage should be taught as a variant on typical
requirements.txt data. Standard dependency specifiers (PEP 508) can be
added to a named list. Rather than asking pip to install from a
requirements.txt file, either pip or a relevant workflow tool will
install from a named Dependency Group.

For new Python users, they may be taught directly to create a section in
pyproject.toml containing their Dependency Groups, similarly to how they
are currently taught to use requirements.txt files. This also allows new
Python users to learn about pyproject.toml files without needing to
learn about package building. A pyproject.toml file with only
[dependency-groups] and no other tables is valid.

For both new and experienced users, the Dependency Group Includes will
need to be explained. For users with experience using requirements.txt,
this can be described as an analogue for -r. For new users, they should
be taught that an include allows one Dependency Group to extend another.
Similar configuration interfaces and the Python list.extend method may
be used to explain the idea by analogy.

Python users who have used setup.py packaging may be familiar with
common practices which predate pyproject.toml, in which package metadata
is defined dynamically. Requirements loaded from requirements.txt files
and definitions of static lists prior to setup() invocation readily
analogize with Dependency Groups.

Interfaces for Use of Dependency Groups

This specification provides no universal interface for interacting with
Dependency Groups, other than inclusion in a built package via the
project table. This has implications both for tool authors and for
users.

Tool authors should determine how or if Dependency Groups are relevant
to their user stories, and build their own interfaces to fit. For
environment managers, resolvers, installers, and related non-build
tools, they will be able to document that they support "PEP 735
Dependency Groups", but they will be responsible for documenting their
usage modes. For build backends, supporting Dependency Groups will
require support for inclusion from the project table, but set no other
strict requirements.

For users, the primary consequence is that they must consult relevant
tool documentation whenever they wish to use Dependency Groups outside
of package builds. Users should be advised by tools, either through
documentation or runtime warnings or errors, about usages which are
disrecommended or not supported. For example, if a tool wishes to
require that all Dependency Groups are mutually compatible, containing
no contradictory package specifiers, it should document that restriction
and advise users on how to appropriately leverage Dependency Groups for
its purposes.

Rejected Ideas

Why not define each Dependency Group as a table?

If our goal is to allow for future expansion, then defining each
Dependency Group as a subtable, thus enabling us to attach future keys
to each group, allows for the greatest future flexibility.

However, it also makes the structure nested more deeply, and therefore
harder to teach and learn. One of the goals of this PEP is to be an easy
replacement for many requirements.txt use-cases.

Why not define a special string syntax to extend Dependency Specifiers?

Earlier drafts of this specification defined syntactic forms for
Dependency Group Includes and Path Dependencies.

However, there were three major issues with this approach:

-   it complicates the string syntax which must be taught, beyond PEP
    508
-   the resulting strings would always need to be disambiguated from PEP
    508 specifiers, complicating implementations

Why not allow for more non-PEP 508 dependency specifiers?

Several use cases surfaced during discussion which need more expressive
specifiers than are possible with PEP 508.

"Path Dependencies", referring to local paths, and references to
[project.dependencies] were of particular interest.

However, there are no existing standards for these features (excepting
the de-facto standard of pip's implementation details).

As a result, attempting to include these features in this PEP results in
a significant growth in scope, to attempt to standardize these various
features and pip behaviors.

Special attention was devoted to attempting to standardize the
expression of editable installations, as expressed by pip install -e and
PEP 660. However, although the creation of editable installs is
standardized for build backends, the behavior of editables is not
standardized for installers. Inclusion of editables in this PEP requires
that any supporting tool allows for the installation of editables.

Therefore, although Poetry and PDM provide syntaxes for some of these
features, they are considered insufficiently standardized at present for
inclusion in Dependency Groups.

Why is the table not named [run], [project.dependency-groups], ...?

There are many possible names for this concept. It will have to live
alongside the already existing [project.dependencies] and
[project.optional-dependencies] tables, and possibly a new [external]
dependency table as well (at time of writing, PEP 725, which defines the
[external] table, is in progress).

[run] was a leading proposal in earlier discussions, but its proposed
usage centered around a single set of runtime dependencies. This PEP
explicitly outlines multiple groups of dependencies, which makes [run] a
less appropriate fit -- this is not just dependency data for a specific
runtime context, but for multiple contexts.

[project.dependency-groups] would offer a nice parallel with
[project.dependencies] and [project.optional-dependencies], but has
major downsides for non-package projects. [project] requires several
keys to be defined, such as name and version. Using this name would
either require redefining the [project] table to allow for these keys to
be absent, or else would impose a requirement on non-package projects to
define and use these keys. By extension, it would effectively require
any non-package project allow itself to be treated as a package.

Why is pip's planned implementation of --only-deps not sufficient?

pip currently has a feature on the roadmap to add an --only-deps flag.
This flag is intended to allow users to install package dependencies and
extras without installing the current package.

It does not address the needs of non-package projects, nor does it allow
for the installation of an extra without the package dependencies.

Why isn't <environment manager> a solution?

Existing environment managers like tox, Nox, and Hatch already have the
ability to list inlined dependencies as part of their configuration
data. This meets many development dependency needs, and clearly
associates dependency groups with relevant tasks which can be run. These
mechanisms are good but they are not sufficient.

First, they do not address the needs of non-package projects.

Second, there is no standard for other tools to use to access these
data. This has impacts on high-level tools like IDEs and Dependabot,
which cannot support deep integration with these Dependency Groups. (For
example, at time of writing Dependabot will not flag dependencies which
are pinned in tox.ini files.)

Deferred Ideas

Why not support Dependency Group Includes in [project.dependencies] or [project.optional-dependencies]?

Earlier drafts of this specification allowed Dependency Group Includes
to be used in the [project] table. However, there were several issues
raised during community feedback which led to its removal.

Only a small number of additional use cases would be addressed by the
inclusion of Dependency Groups, and it increased the scope of the
specification significantly. In particular, this inclusion would
increase the number of parties impacted by the addition. Many readers of
the [project] table, including build backends, SBOM generators, and
dependency analyzers are implicated by a change to [project] but may
continue to operate as-is in the presence of a new (but unconnected)
[dependency-groups] table.

Separately from the above concern, allowing inclusion of dependency
groups from the [project] table encourages package maintainers to move
dependency metadata out of the current standard location. This
complicates static pyproject.toml metadata and conflicts with the goal
of PEP 621 to store dependency metadata in a single location.

Finally, exclusion of [project] support from this PEP is not final. The
use of includes from that table, or an inclusion syntax from
[dependency-groups] into [project], could be introduced by a future PEP
and considered on its own merits.

Use Cases for Dependency Group Includes From [project]

Although deferred in this PEP, allowing includes from the [project]
table would address several use cases.

In particular, there are cases in which package developers would like to
install only the dependencies of a package, without the package itself.

For example:

-   Specify different environment variables or options when building
    dependencies vs when building the package itself
-   Creating layered container images in which the dependencies are
    isolated from the package being installed
-   Providing the dependencies to analysis environments (e.g., type
    checking) without having to build and install the package itself

For an example of the last case, consider the following sample
pyproject.toml:

    [project]
    dependencies = [{include = "runtime"}]
    [optional-dependencies]
    foo = [{include = "foo"}]
    [dependency-groups]
    runtime = ["a", "b"]
    foo = ["c", "d"]
    typing = ["mypy", {include = "runtime"}, {include = "foo"}]

In this case, a typing group can be defined, with all of the package's
runtime dependencies, but without the package itself. This allows uses
of the typing Dependency Group to skip installation of the package --
not only is this more efficient, but it may reduce the requirements for
testing systems.

Why not support Dependency Group Includes in [build-system.requires]?

Given that we will not allow for [project] usage of Dependency Groups,
[build-system.requires] can be considered in comparison with
[project.dependencies].

There are fewer theoretical usages for build requirements specified in a
group than package requirements. Additionally, the impact of such a
change implicates PEP 517 frontend, which would need to support
Dependency Groups in order to prepare a build environment.

Compared with changes to [project.dependencies] and
[project.optional-dependencies], changing the behaviors of
[build-system.requires] is higher impact and has fewer potential uses.
Therefore, given that this PEP declines to make changes to the [project]
table, changing [build-system] is also deferred.

Why not support a Dependency Group which includes the current project?

Several usage scenarios for dependency groups revolve around installing
a dependency group alongside a package defined in the [project] table.
For example, testing a package involves installing testing dependencies
and the package itself. Additionally, the compatibility of a dependency
group with the main package is a valuable input to lockfile generators.

In such cases, it is desirable for a Dependency Group to declare that it
depends upon the project itself. Example syntaxes from discussions
included {include-project = true} and {include-group = ":project:"}.

However, if a specification is established to extend PEP 508 with Path
Dependencies, this would result in Dependency Groups having two ways of
specifying the main package. For example, if . becomes formally
supported, and {include-project = true} is included in this PEP, then
dependency groups may specify any of the following groups

    [dependency-groups]
    case1 = [{include-project = true}]
    case2 = ["."]
    case3 = [{include-project = true}, "."]
    case4 = [{include-project = false}, "."]

In order to avoid a confusing future in which multiple different options
specify the package defined in pyproject.toml, any syntax for declaring
this relationship is omitted from this PEP.

Appendix A: Prior Art in Non-Python Languages

This section is primarily informational and serves to document how other
language ecosystems solve similar problems.

JavaScript and package.json

In the JavaScript community, packages contain a canonical configuration
and data file, similar in scope to pyproject.toml, at package.json.

Two keys in package.json control dependency data: "dependencies" and
"devDependencies". The role of "dependencies" is effectively the same as
that of [project.dependencies] in pyproject.toml, declaring the direct
dependencies of a package.

"dependencies" data

Dependency data is declared in package.json as a mapping from package
names to version specifiers.

Version specifiers support a small grammar of possible versions, ranges,
and other values, similar to Python's PEP 440 version specifiers.

For example, here is a partial package.json file declaring a few
dependencies:

    {
        "dependencies": {
            "@angular/compiler": "^17.0.2",
            "camelcase": "8.0.0",
            "diff": ">=5.1.0 <6.0.0"
        }
    }

The use of the @ symbol is a scope which declares the package owner, for
organizationally owned packages. "@angular/compiler" therefore declares
a package named compiler grouped under angular ownership.

Dependencies Referencing URLs and Local Paths

Dependency specifiers support a syntax for URLs and Git repositories,
similar to the provisions in Python packaging.

URLs may be used in lieu of version numbers. When used, they implicitly
refer to tarballs of package source code.

Git repositories may be similarly used, including support for committish
specifiers.

Unlike PEP 440, NPM allows for the use of local paths to package source
code directories for dependencies. When these data are added to
package.json via the standard npm install --save command, the path is
normalized to a relative path, from the directory containing
package.json, and prefixed with file:. For example, the following
partial package.json contains a reference to a sibling of the current
directory:

    {
        "dependencies": {
            "my-package": "file:../foo"
        }
    }

The official NPM documentation states that local path dependencies
"should not" be published to public package repositories, but makes no
statement about the inherent validity or invalidity of such dependency
data in published packages.

"devDependencies" data

package.json is permitted to contain a second section named
"devDependencies", in the same format as "dependencies". The
dependencies declared in "devDependencies" are not installed by default
when a package is installed from the package repository (e.g. as part of
a dependency being resolved) but are installed when npm install is run
in the source tree containing package.json.

Just as "dependencies" supports URLs and local paths, so does
"devDependencies".

"peerDependencies" and "optionalDependencies"

There are two additional, related sections in package.json which have
relevance.

"peerDependencies" declares a list of dependencies in the same format as
"dependencies", but with the meaning that these are a compatibility
declaration. For example, the following data declares compatibility with
package foo version 2:

    {
        "peerDependencies": {
            "foo": "2.x"
        }
    }

"optionalDependencies" declares a list of dependencies which should be
installed if possible, but which should not be treated as failures if
they are unavailable. It also uses the same mapping format as
"dependencies".

"peerDependenciesMeta"

"peerDependenciesMeta" is a section which allows for additional control
over how "peerDependencies" are treated.

Warnings about missing dependencies can be disabled by setting packages
to optional in this section, as in the following sample:

    {
        "peerDependencies": {
            "foo": "2.x"
        },
        "peerDependenciesMeta": {
            "foo": {
                "optional": true
            }
        }
    }

--omit and --include

The npm install command supports two options, --omit and --include,
which can control whether "prod", "dev", "optional", or "peer"
dependencies are installed.

The "prod" name refers to dependencies listed under "dependencies".

By default, all four groups are installed when npm install is executed
against a source tree, but these options can be used to control
installation behavior more precisely. Furthermore, these values can be
declared in .npmrc files, allowing per-user and per-project
configurations to control installation behaviors.

Ruby & Ruby Gems

Ruby projects may or may not be intended to produce packages ("gems") in
the Ruby ecosystem. In fact, the expectation is that most users of the
language do not want to produce gems and have no interest in producing
their own packages. Many tutorials do not touch on how to produce
packages, and the toolchain never requires user code to be packaged for
supported use-cases.

Ruby splits requirement specification into two separate files.

-   Gemfile: a dedicated file which only supports requirement data in
    the form of dependency groups
-   <package>.gemspec: a dedicated file for declaring package (gem)
    metadata

The bundler tool, providing the bundle command, is the primary interface
for using Gemfile data.

The gem tool is responsible for building gems from .gemspec data, via
the gem build command.

Gemfiles & bundle

A Gemfile is a Ruby file containing gem directives enclosed in any
number of group declarations. gem directives may also be used outside of
the group declaration, in which case they form an implicitly unnamed
group of dependencies.

For example, the following Gemfile lists rails as a project dependency.
All other dependencies are listed under groups:

    source 'https://rubygems.org'

    gem 'rails'

    group :test do
      gem 'rspec'
    end

    group :lint do
      gem 'rubocop'
    end

    group :docs do
      gem 'kramdown'
      gem 'nokogiri'
    end

If a user executes bundle install with these data, all groups are
installed. Users can deselect groups by creating or modifying a bundler
config in .bundle/config, either manually or via the CLI. For example,
bundle config set --local without 'lint:docs'.

It is not possible, with the above data, to exclude the top-level use of
the 'rails' gem or to refer to that implicit grouping by name.

gemspec and packaged dependency data

A gemspec file is a ruby file containing a Gem::Specification instance
declaration.

Only two fields in a Gem::Specification pertain to package dependency
data. These are add_development_dependency and add_runtime_dependency. A
Gem::Specification object also provides methods for adding dependencies
dynamically, including add_dependency (which adds a runtime dependency).

Here is a variant of the rails.gemspec file, with many fields removed or
shortened to simplify:

    version = '7.1.2'

    Gem::Specification.new do |s|
      s.platform    = Gem::Platform::RUBY
      s.name        = "rails"
      s.version     = version
      s.summary     = "Full-stack web application framework."

      s.license = "MIT"
      s.author   = "David Heinemeier Hansson"

      s.files = ["README.md", "MIT-LICENSE"]

      # shortened from the real 'rails' project
      s.add_dependency "activesupport", version
      s.add_dependency "activerecord",  version
      s.add_dependency "actionmailer",  version
      s.add_dependency "activestorage", version
      s.add_dependency "railties",      version
    end

Note that there is no use of add_development_dependency. Some other
mainstream, major packages (e.g. rubocop) do not use development
dependencies in their gems.

Other projects do use this feature. For example, kramdown makes use of
development dependencies, containing the following specification in its
Rakefile:

    s.add_dependency "rexml"
    s.add_development_dependency 'minitest', '~> 5.0'
    s.add_development_dependency 'rouge', '~> 3.0', '>= 3.26.0'
    s.add_development_dependency 'stringex', '~> 1.5.1'

The purpose of development dependencies is only to declare an implicit
group, as part of the .gemspec, which can then be used by bundler.

For full details, see the gemspec directive in bundler's documentation
on Gemfiles. However, the integration between .gemspec development
dependencies and Gemfile/bundle usage is best understood via an example.

gemspec development dependency example

Consider the following simple project in the form of a Gemfile and
.gemspec. The cool-gem.gemspec file:

    Gem::Specification.new do |s|
      s.author = 'Stephen Rosen'
      s.name = 'cool-gem'
      s.version = '0.0.1'
      s.summary = 'A very cool gem that does cool stuff'
      s.license = 'MIT'

      s.files = []

      s.add_dependency 'rails'
      s.add_development_dependency 'kramdown'
    end

and the Gemfile:

    source 'https://rubygems.org'

    gemspec

The gemspec directive in Gemfile declares a dependency on the local
package, cool-gem, defined in the locally available cool-gem.gemspec
file. It also implicitly adds all development dependencies to a
dependency group named development.

Therefore, in this case, the gemspec directive is equivalent to the
following Gemfile content:

    gem 'cool-gem', :path => '.'

    group :development do
      gem 'kramdown'
    end

Appendix B: Prior Art in Python

In the absence of any prior standard for Dependency Groups, two known
workflow tools, PDM and Poetry, have defined their own solutions.

This section will primarily focus on these two tools as cases of prior
art regarding the definition and use of Dependency Groups in Python.

Projects are Packages

Both PDM and Poetry treat the projects they support as packages. This
allows them to use and interact with standard pyproject.toml metadata
for some of their needs, and allows them to support installation of the
"current project" by doing a build and install using their build
backends.

Effectively, this means that neither Poetry nor PDM supports non-package
projects.

Non-Standard Dependency Specifiers

PDM and Poetry extend PEP 508 dependency specifiers with additional
features which are not part of any shared standard. The two tools use
slightly different approaches to these problems, however.

PDM supports specifying local paths, and editable installs, via a syntax
which looks like a set of arguments to pip install. For example, the
following dependency group includes a local package in editable mode:

    [tool.pdm.dev-dependencies]
    mygroup = ["-e file:///${PROJECT_ROOT}/foo"]

This declares a dependency group mygroup which includes a local editable
install from the foo directory.

Poetry describes dependency groups as tables, mapping package names to
specifiers. For example, the same configuration as the above mygroup
example might appear as follows under Poetry:

    [tool.poetry.group.mygroup]
    foo = { path = "foo", editable = true }

PDM restricts itself to a string syntax, and Poetry introduces tables
which describe dependencies.

Installing and Referring to Dependency Groups

Both PDM and Poetry have tool-specific support for installing dependency
groups. Because both projects support their own lockfile formats, they
also both have the capability to transparently use a dependency group
name to refer to the locked dependency data for that group.

However, neither tool's dependency groups can be referenced natively
from other tools like tox, nox, or pip. Attempting to install a
dependency group under tox, for example, requires an explicit call to
PDM or Poetry to parse their dependency data and do the relevant
installation step.

Appendix C: Use Cases

Web Applications

A web application (e.g. a Django or Flask app) often does not need to
build a distribution, but bundles and ships its source to a deployment
toolchain.

For example, a source code repository may define Python packaging
metadata as well as containerization or other build pipeline metadata
(Dockerfile, etc). The Python application is built by copying the entire
repository into a build context, installing dependencies, and bundling
the result as a machine image or container.

Such applications have dependency groups for the build, but also for
linting, testing, etc. In practice, today, these applications often
define themselves as packages to be able to use packaging tools and
mechanisms like extras to manage their dependency groups. However, they
are not conceptually packages, meant for distribution in sdist or wheel
format.

Dependency Groups allow these applications to define their various
dependencies without relying on packaging metadata, and without trying
to express their needs in packaging terms.

Libraries

Libraries are Python packages which build distributions (sdist and
wheel) and publish them to PyPI.

For libraries, Dependency Groups represent an alternative to extras for
defining groups of development dependencies, with the important
advantages noted above.

A library may define groups for test and typing which allow testing and
type-checking, and therefore rely on the library's own dependencies (as
specified in [project.dependencies]).

Other development needs may not require installation of the package at
all. For example, a lint Dependency Group may be valid and faster to
install without the library, as it only installs tools like black, ruff,
or flake8.

lint and test environments may also be valuable locations to hook in IDE
or editor support. See the case below for a fuller description of such
usage.

Here's an example Dependency Groups table which might be suitable for a
library:

    [dependency-groups]
    test = ["pytest<8", "coverage"]
    typing = ["mypy==1.7.1", "types-requests"]
    lint = ["black", "flake8"]
    typing-test = [{include-group = "typing"}, "pytest<8"]

Note that none of these implicitly install the library itself. It is
therefore the responsibility of any environment management toolchain to
install the appropriate Dependency Groups along with the library when
needed, as in the case of test.

Data Science Projects

Data Science Projects typically take the form of a logical collection of
scripts and utilities for processing and analyzing data, using a common
toolchain. Components may be defined in the Jupyter Notebook format
(ipynb), but rely on the same common core set of utilities.

In such a project, there is no package to build or install. Therefore,
pyproject.toml currently does not offer any solution for dependency
management or declaration.

It is valuable for such a project to be able to define at least one
major grouping of dependencies. For example:

    [dependency-groups]
    main = ["numpy", "pandas", "matplotlib"]

However, it may also be necessary for various scripts to have additional
supporting tools. Projects may even have conflicting or incompatible
tools or tool versions for different components, as they evolve over
time.

Consider the following more elaborate configuration:

    [dependency-groups]
    main = ["numpy", "pandas", "matplotlib"]
    scikit = [{include-group = "main"}, "scikit-learn==1.3.2"]
    scikit-old = [{include-group = "main"}, "scikit-learn==0.24.2"]

This defines scikit and scikit-old as two similar variants of the common
suite of dependencies, pulling in different versions of scikit-learn to
suit different scripts.

This PEP only defines these data. It does not formalize any mechanism
for a Data Science Project (or any other type of project) to install the
dependencies into known environments or associate those environments
with the various scripts. Such combinations of data are left as a
problem for tool authors to solve, and perhaps eventually standardize.

Lockfile Generation

There are a number of tools which generate lockfiles in the Python
ecosystem today. PDM and Poetry each use their own lockfile formats, and
pip-tools generates requirements.txt files with version pins and hashes.

Dependency Groups are not an appropriate place to store lockfiles, as
they lack many of the necessary features. Most notably, they cannot
store hashes, which most lockfile users consider essential.

However, Dependency Groups are a valid input to tools which generate
lockfiles. Furthermore, PDM and Poetry both allow a Dependency Group
name (under their notions of Dependency Groups) to be used to refer to
its locked variant.

Therefore, consider a tool which produces lockfiles, here called $TOOL.
It might be used as follows:

    $TOOL lock --dependency-group=test
    $TOOL install --dependency-group=test --use-locked

All that such a tool needs to do is to ensure that its lockfile data
records the name test in order to support such usage.

The mutual compatibility of Dependency Groups is not guaranteed. For
example, the Data Science example above shows conflicting versions of
scikit-learn. Therefore, installing multiple locked dependency groups in
tandem may require that tools apply additional constraints or generate
additional lockfile data. These problems are considered out of scope for
this PEP.

As two examples of how combinations might be locked:

-   A tool might require that lockfile data be explicitly generated for
    any combination to be considered valid
-   Poetry implements the requirement that all Dependency Groups be
    mutually compatible, and generates only one locked version. (Meaning
    it finds a single solution, rather than a set or matrix of
    solutions.)

Environment Manager Inputs

A common usage in tox, Nox, and Hatch is to install a set of
dependencies into a testing environment.

For example, under tox.ini, type checking dependencies may be defined
inline:

    [testenv:typing]
    deps =
        pyright
        useful-types
    commands = pyright src/

This combination provides a desirable developer experience within a
limited context. Under the relevant environment manager, the
dependencies which are needed for the test environment are declared
alongside the commands which need those dependencies. They are not
published in package metadata, as extras would be, and they are
discoverable for the tool which needs them to build the relevant
environment.

Dependency Groups apply to such usages by effectively "lifting" these
requirements data from a tool-specific location into a more broadly
available one. In the example above, only tox has access to the declared
list of dependencies. Under an implementation supporting dependency
groups, the same data might be available in a Dependency Group:

    [dependency-groups]
    typing = ["pyright", "useful-types"]

The data can then be used under multiple tools. For example, tox might
implement support as dependency_groups = typing, replacing the deps
usage above.

In order for Dependency Groups to be a viable alternative for users of
environment managers, the environment managers will need to support
processing Dependency Groups similarly to how they support inline
dependency declaration.

IDE and Editor Use of Requirements Data

IDE and editor integrations may benefit from conventional or
configurable name definitions for Dependency Groups which are used for
integrations.

There are at least two known scenarios in which it is valuable for an
editor or IDE to be capable of discovering the non-published
dependencies of a project:

-   testing: IDEs such as VS Code support GUI interfaces for running
    particular tests
-   linting: editors and IDEs often support linting and autoformatting
    integrations which highlight or autocorrect errors

These cases could be handled by defining conventional group names like
test, lint, and fix, or by defining configuration mechanisms which allow
the selection of Dependency Groups.

For example, the following pyproject.toml declares the three
aforementioned groups:

    [dependency-groups]
    test = ["pytest", "pytest-timeout"]
    lint = ["flake8", "mypy"]
    fix = ["black", "isort", "pyupgrade"]

This PEP makes no attempt to standardize such names or reserve them for
such uses. IDEs may standardize or may allow users to configure the
group names used for various purposes.

This declaration allows the project author's knowledge of the
appropriate tools for the project to be shared with all editors of that
project.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 279 Title: The enumerate() built-in function Author: Raymond
Hettinger<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 30-Jan-2002 Python-Version: 2.3
Post-History:

Abstract

This PEP introduces a new built-in function, enumerate() to simplify a
commonly used looping idiom. It provides all iterable collections with
the same advantage that iteritems() affords to dictionaries -- a
compact, readable, reliable index notation.

Rationale

Python 2.2 introduced the concept of an iterable interface as proposed
in PEP 234. The iter() factory function was provided as common calling
convention and deep changes were made to use iterators as a unifying
theme throughout Python. The unification came in the form of
establishing a common iterable interface for mappings, sequences, and
file objects.

Generators, as proposed in PEP 255, were introduced as a means for
making it easier to create iterators, especially ones with complex
internal execution or variable states. The availability of generators
makes it possible to improve on the loop counter ideas in PEP 212. Those
ideas provided a clean syntax for iteration with indices and values, but
did not apply to all iterable objects. Also, that approach did not have
the memory friendly benefit provided by generators which do not evaluate
the entire sequence all at once.

The new proposal is to add a built-in function, enumerate() which was
made possible once iterators and generators became available. It
provides all iterables with the same advantage that iteritems() affords
to dictionaries -- a compact, readable, reliable index notation. Like
zip(), it is expected to become a commonly used looping idiom.

This suggestion is designed to take advantage of the existing
implementation and require little additional effort to incorporate. It
is backwards compatible and requires no new keywords. The proposal will
go into Python 2.3 when generators become final and are not imported
from __future__.

BDFL Pronouncements

The new built-in function is ACCEPTED.

Specification for a new built-in

    def enumerate(collection):
       'Generates an indexed series:  (0,coll[0]), (1,coll[1]) ...'
       i = 0
       it = iter(collection)
       while 1:
          yield (i, it.next())
          i += 1

Note A: PEP 212 Loop Counter Iteration discussed several proposals for
achieving indexing. Some of the proposals only work for lists unlike the
above function which works for any generator, xrange, sequence, or
iterable object. Also, those proposals were presented and evaluated in
the world prior to Python 2.2 which did not include generators. As a
result, the non-generator version in PEP 212 had the disadvantage of
consuming memory with a giant list of tuples. The generator version
presented here is fast and light, works with all iterables, and allows
users to abandon the sequence in mid-stream with no loss of computation
effort.

There are other PEPs which touch on related issues: integer iterators,
integer for-loops, and one for modifying the arguments to range and
xrange. The enumerate() proposal does not preclude the other proposals
and it still meets an important need even if those are adopted -- the
need to count items in any iterable. The other proposals give a means of
producing an index but not the corresponding value. This is especially
problematic if a sequence is given which doesn't support random access
such as a file object, generator, or sequence defined with __getitem__.

Note B: Almost all of the PEP reviewers welcomed the function but were
divided as to whether there should be any built-ins. The main argument
for a separate module was to slow the rate of language inflation. The
main argument for a built-in was that the function is destined to be
part of a core programming style, applicable to any object with an
iterable interface. Just as zip() solves the problem of looping over
multiple sequences, the enumerate() function solves the loop counter
problem.

If only one built-in is allowed, then enumerate() is the most important
general purpose tool, solving the broadest class of problems while
improving program brevity, clarity and reliability.

Note C: Various alternative names were discussed:

  --------------- -----------------------------------------------------------------------------------------------------------
  iterindexed()   five syllables is a mouthful
  index()         nice verb but could be confused the .index() method
  indexed()       widely liked however adjectives should be avoided
  indexer()       noun did not read well in a for-loop
  count()         direct and explicit but often used in other contexts
  itercount()     direct, explicit and hated by more than one person
  iteritems()     conflicts with key:value concept for dictionaries
  itemize()       confusing because amap.items() != list(itemize(amap))
  enum()          pithy; less clear than enumerate; too similar to enum in other languages where it has a different meaning
  --------------- -----------------------------------------------------------------------------------------------------------

All of the names involving 'count' had the further disadvantage of
implying that the count would begin from one instead of zero.

All of the names involving 'index' clashed with usage in database
languages where indexing implies a sorting operation rather than linear
sequencing.

Note D: This function was originally proposed with optional start and
stop arguments. GvR pointed out that the function call
enumerate(seqn,4,6) had an alternate, plausible interpretation as a
slice that would return the fourth and fifth elements of the sequence.
To avoid the ambiguity, the optional arguments were dropped even though
it meant losing flexibility as a loop counter. That flexibility was most
important for the common case of counting from one, as in:

    for linenum, line in enumerate(source,1):  print linenum, line

Comments from GvR:

    filter and map should die and be subsumed into list comprehensions,
    not grow more variants. I'd rather introduce built-ins that do
    iterator algebra (e.g. the iterzip that I've often used as an
    example).

    I like the idea of having some way to iterate over a sequence and
    its index set in parallel. It's fine for this to be a built-in.

    I don't like the name "indexed"; adjectives do not make good
    function names. Maybe iterindexed()?

Comments from Ka-Ping Yee:

    I'm also quite happy with everything you proposed ... and the extra
    built-ins (really 'indexed' in particular) are things I have wanted
    for a long time.

Comments from Neil Schemenauer:

    The new built-ins sound okay. Guido may be concerned with increasing
    the number of built-ins too much. You might be better off selling
    them as part of a module. If you use a module then you can add lots
    of useful functions (Haskell has lots of them that we could steal).

Comments for Magnus Lie Hetland:

    I think indexed would be a useful and natural built-in function. I
    would certainly use it a lot. I like indexed() a lot; +1. I'm quite
    happy to have it make PEP 281 obsolete. Adding a separate module for
    iterator utilities seems like a good idea.

Comments from the Community:

    The response to the enumerate() proposal has been close to 100%
    favorable. Almost everyone loves the idea.

Author response:

    Prior to these comments, four built-ins were proposed. After the
    comments, xmap, xfilter and xzip were withdrawn. The one that
    remains is vital for the language and is proposed by itself.
    Indexed() is trivially easy to implement and can be documented in
    minutes. More importantly, it is useful in everyday programming
    which does not otherwise involve explicit use of generators.

    This proposal originally included another function iterzip(). That
    was subsequently implemented as the izip() function in the itertools
    module.

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 516 Title: Build system abstraction for pip/conda etc Author:
Robert Collins<EMAIL_ADDRESS>Nathaniel J. Smith<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Topic:
Packaging Content-Type: text/x-rst Created: 26-Oct-2015 Resolution:
https://mail.python.org/pipermail/distutils-sig/2017-May/030517.html

Abstract

This PEP specifies a programmatic interface for pip[1] and other
distribution or installation tools to use when working with Python
source trees (both the developer tree - e.g. the git tree - and source
distributions).

The programmatic interface allows decoupling of pip from its current
hard dependency on setuptools[2] able for two key reasons:

1.  It enables new build systems that may be much easier to use without
    requiring them to even appear to be setuptools.
2.  It facilitates setuptools itself changing its user interface without
    breaking pip, giving looser coupling.

The interface needed to permit pip to install build systems also enables
pip to install build time requirements for packages which is an
important step in getting pip to full feature parity with the
installation components of easy-install.

As PEP 426 is draft, we cannot utilise the metadata format it defined.
However PEP 427 wheels are in wide use and fairly well specified, so we
have adopted the METADATA format from that for specifying distribution
dependencies and general project metadata. PEP 508 provides a
self-contained language for describing a dependency, which we
encapsulate in a thin JSON schema to describe bootstrap dependencies.

Since Python sdists specified in PEP 314 are also source trees, this PEP
is updating the definition of sdists.

PEP Rejection

The CLI based approach proposed in this PEP has been rejected in favour
of the Python API based approach proposed in PEP 517. The specific CLI
used to communicate with build backends running as isolated subprocesses
will be considered an implementation detail of front-end developer tool
implementations.

Motivation

There is significant pent-up frustration in the Python packaging
ecosystem around the current lock-in between build system and pip.
Breaking that lock-in is better for pip, for setuptools, and for other
build systems like flit [3].

Specification

Overview

Build tools will be located by reading a file pypa.json from the root
directory of the source tree. That file describes how to get the build
tool and the name of the command to run to invoke the tool.

All tools will be expected to conform to a single command line interface
modelled on pip's existing use of the setuptools setup.py interface.

pypa.json

The file pypa.json acts as a neutral configuration file for pip and
other tools that want to build source trees to consult for
configuration. The absence of a pypa.json file in a Python source tree
implies a setuptools or setuptools compatible build system.

The JSON has the following schema. Extra keys are ignored, which permits
the use of pypa.json as a configuration file for other related tools. If
doing that the chosen keys must be namespaced under tools:

    {"tools": {"flit": ["Flits content here"]}}

schema

    The version of the schema. This PEP defines version "1". Defaults to
    "1" when absent. All tools reading the file must error on an
    unrecognised schema version.

bootstrap_requires

    Optional list of PEP 508 dependency specifications that must be
    installed before running the build tool. For instance, if using
    flit, then the requirements might be:

        bootstrap_requires: ["flit"]

build_command

    A mandatory key, this is a list of Python format strings[4]
    describing the command to run. For instance, if using flit then the
    build command might be:

        build_command: ["flit"]

    If using a command which is a runnable module fred:

        build_command: ["{PYTHON}", "-m", "fred"]

Process interface

The command to run is defined by a simple Python format string[5].

This permits build systems with dedicated scripts and those that are
invoked using "python -m somemodule".

Processes will be run with the current working directory set to the root
of the source tree.

When run, processes should not read from stdin - while pip currently
runs build systems with stdin connected to its own stdin, stdout and
stderr are redirected and no communication with the user is possible.

As usual with processes, a non-zero exit status indicates an error.

Available format variables

PYTHON

    The Python interpreter in use. This is important to enable calling
    things which are just Python entry points.

      {PYTHON} -m foo

Available environment variables

These variables are set by the caller of the build system and will
always be available.

PATH

    The standard system path.

PYTHON

    As for format variables.

PYTHONPATH

    Used to control sys.path per the normal Python mechanisms.

Subcommands

There are a number of separate subcommands that build systems must
support. The examples below use a build_command of flit for illustrative
purposes.

build_requires

    Query build requirements. Build requirements are returned as a UTF-8
    encoded JSON document with one key build_requires consisting of a
    list of PEP 508 dependency specifications. Additional keys must be
    ignored. The build_requires command is the only command run without
    setting up a build environment.

    Example command:

        flit build_requires

metadata

    Query project metadata. The metadata and only the metadata should be
    output on stdout in UTF-8 encoding. pip would run metadata just once
    to determine what other packages need to be downloaded and
    installed. The metadata is output as a wheel METADATA file per PEP
    427.

    Note that the metadata generated by the metadata command, and the
    metadata present in a generated wheel must be identical.

    Example command:

        flit metadata

wheel -d OUTPUT_DIR

    Command to run to build a wheel of the project. OUTPUT_DIR will
    point to an existing directory where the wheel should be output.
    Stdout and stderr have no semantic meaning. Only one file should be
    output - if more are output then pip would pick an arbitrary one to
    consume.

    Example command:

        flit wheel -d /tmp/pip-build_1234

develop [--prefix PREFIX]

    Command to do an in-place 'development' installation of the project.
    Stdout and stderr have no semantic meaning.

    Not all build systems will be able to perform develop installs. If a
    build system cannot do develop installs, then it should error when
    run. Note that doing so will cause use operations like
    pip install -e foo to fail.

    The prefix option is used for defining an alternative prefix for the
    installation. While setuptools has --root and --user options, they
    can be done equivalently using --prefix, and pip or other tools that
    accept --root or --user options should translate appropriately.

    The root option is used to define an alternative root within which
    the command should operate.

    For instance:

        flit develop --root /tmp/ --prefix /usr/local

    Should install scripts within /tmp/usr/local/bin, even if the Python
    environment in use reports that the sys.prefix is /usr/ which would
    lead to using /tmp/usr/bin/. Similar logic applies for package files
    etc.

The build environment

Except for the build_requires command, all commands are run within a
build environment. No specific implementation is required, but a build
environment must achieve the following requirements.

1.  All dependencies specified by the project's build_requires must be
    available for import from within $PYTHON.
2.  All command-line scripts provided by the build-required packages
    must be present in $PATH.

A corollary of this is that build systems cannot assume access to any
Python package that is not declared as a build_requires or in the Python
standard library.

Hermetic builds

This specification does not prescribe whether builds should be hermetic
or not. Existing build tools like setuptools will use installed versions
of build time requirements (e.g. setuptools_scm) and only install other
versions on version conflicts or missing dependencies. However its
likely that better consistency can be created by always isolation builds
and using only the specified dependencies.

However, there are nuanced problems there - such as how can users force
the avoidance of a bad version of a build requirement which meets some
packages dependencies. Future PEPs may tackle this problem, but it is
not currently in scope - it does not affect the metadata required to
coordinate between build systems and things that need to do builds, and
thus is not PEP material.

Upgrades

'pypa.json' is versioned to permit future changes without requiring
compatibility.

The sequence for upgrading either of schemas in a new PEP will be:

1.  Issue new PEP defining an updated schema. If the schema is not
    entirely backward compatible then a new version number must be
    defined.
2.  Consumers (e.g. pip) implement support for the new schema version.
3.  Package authors opt into the new schema when they are happy to
    introduce a dependency on the version of 'pip' (and potentially
    other consumers) that introduced support for the new schema version.

The same process will take place for the initial deployment of this
PEP:-the propagation of the capability to use this PEP without a
setuptools shim will be largely gated by the adoption rate of the first
version of pip that supports it.

Static metadata in sdists

This PEP does not tackle the current inability to trust static metadata
in sdists. That is a separate problem to identifying and consuming the
build system that is in use in a source tree, whether it came from an
sdist or not.

Handling of compiler options

Handling of different compiler options is out of scope for this
specification.

pip currently handles compiler options by appending user supplied
strings to the command line it runs when running setuptools. This
approach is sufficient to work with the build system interface defined
in this PEP, with the exception that globally specified options will
stop working globally as different build systems evolve. That problem
can be solved in pip (or conda or other installers) without affecting
interoperability.

In the long term, wheels should be able to express the difference
between wheels built with one compiler or options vs another, and that
is PEP material.

Examples

An example 'pypa.json' for using flit:

    {"bootstrap_requires": ["flit"],
     "build_command": "flit"}

When 'pip' reads this it would prepare an environment with flit in it
before trying to use flit.

Because flit doesn't have setup-requires support today,
flit build_requires would just output a constant string:

    {"build_requires": []}

flit metadata would interrogate flit.ini and marshal the metadata into a
wheel METADATA file and output that on stdout.

flit wheel would need to accept a -d parameter that tells it where to
output the wheel (pip needs this).

Backwards Compatibility

Older pips will remain unable to handle alternative build systems. This
is no worse than the status quo - and individual build system projects
can decide whether to include a shim setup.py or not.

All existing build systems that can product wheels and do develop
installs should be able to run under this abstraction and will only need
a specific adapter for them constructed and published on PyPI.

In the absence of a pypa.json file, tools like pip should assume a
setuptools build system and use setuptools commands directly.

Network effects

Projects that adopt build systems that are not setuptools compatible -
that is that they have no setup.py, or the setup.py doesn't accept
commands that existing tools try to use - will not be installable by
those existing tools.

Where those projects are used by other projects, this effect will
cascade.

In particular, because pip does not handle setup-requires today, any
project (A) that adopts a setuptools-incompatible build system and is
consumed as a setup-requirement by a second project (B) which has not
itself transitioned to having a pypa.json will make B uninstallable by
any version of pip. This is because setup.py in B will trigger
easy-install when 'setup.py egg_info' is run by pip, and that will try
and fail to install A.

As such we recommend that tools which are currently used as
setup-requires either ensure that they keep a setuptools shim or find
their consumers and get them all to upgrade to the use of a pypa.json in
advance of moving themselves. Pragmatically that is impossible, so the
advice is to keep a setuptools shim indefinitely - both for projects
like pbr, setuptools_scm and also projects like numpy.

setuptools shim

It would be possible to write a generic setuptools shim that looks like
setup.py and under the hood uses pypa.json to drive the builds. This is
not needed for pip to use the system, but would allow package authors to
use the new features while still retaining compatibility with older pip
versions.

Rationale

This PEP started with a long mailing list thread on distutils-sig[6].
Subsequent to that an online meeting was held to debug all the positions
folk had. Minutes from that were posted to the list[7].

This specification is a translation of the consensus reached there into
PEP form, along with some arbitrary choices on the minor remaining
questions.

The basic heuristic for the design has been to focus on introducing an
abstraction without requiring development not strictly tied to the
abstraction. Where the gap is small to improvements, or the cost of
using the existing interface is very high, then we've taken on having
the improvement as a dependency, but otherwise deferred such to future
iterations.

We chose wheel METADATA files rather than defining a new specification,
because pip can already handle wheel .dist-info directories which encode
all the necessary data in a METADATA file. PEP 426 can't be used as it's
still draft, and defining a new metadata format, while we should do
that, is a separate problem. Using a directory on disk would not add any
value to the interface (pip has to do that today due to limitations in
the setuptools CLI).

The use of 'develop' as a command is because there is no PEP specifying
the interoperability of things that do what 'setuptools develop' does -
so we'll need to define that before pip can take on the responsibility
for doing the 'develop' step. Once that's done we can issue a successor
PEP to this one.

The use of a command line API rather than a Python API is a little
contentious. Fundamentally anything can be made to work, and the pip
maintainers have spoken strongly in favour of retaining a process based
interface - something that is mature and robust in pip today.

The choice of JSON as a file format is a compromise between several
constraints. Firstly there is no stdlib YAML interpreter, nor one for
any of the other low-friction structured file formats. Secondly,
INIParser is a poor format for a number of reasons, primarily that it
has very minimal structure -but pip's maintainers are not fond of it.
JSON is in the stdlib, has sufficient structure to permit embedding
anything we want in future without requiring embedded DSL's.

Donald suggested using setup.cfg and the existing setuptools command
line rather than inventing something new. While that would permit
interoperability with less visible changes, it requires nearly as much
engineering on the pip side - looking for the new key in setup.cfg,
implementing the non-installed environments to run the build in. And the
desire from other build system authors not to confuse their users by
delivering something that looks like but behaves quite differently to
setuptools seems like a bigger issue than pip learning how to invoke a
custom build tool.

The metadata and wheel commands are required to have consistent metadata
to avoid a race condition that could otherwise happen where pip reads
the metadata, acts on it, and then the resulting wheel has incompatible
requirements. That race is exploited today by packages using PEP 426
environment markers, to work with older pip versions that do not support
environment markers. That exploit is not needed with this PEP, because
either the setuptools shim is in use (with older pip versions), or an
environment marker ready pip is in use. The setuptools shim can take
care of exploiting the difference older pip versions require.

We discussed having an sdist verb. The main driver for this was to make
sure that build systems were able to produce sdists that pip can build -
but this is circular: the whole point of this PEP is to let pip consume
such sdists or VCS source trees reliably and without requiring an
implementation of setuptools. Being able to create new sdists from
existing source trees isn't a thing pip does today, and while there is a
PR to do that as part of building from source, it is contentious and
lacks consensus. Rather than impose a requirement on all build systems,
we are treating it as a YAGNI, and will add such a verb in a future
version of the interface if required. The existing PEP 314 requirements
for sdists still apply, and distutils or setuptools users can use
setup.py sdist to create an sdist. Other tools should create sdists
compatible with PEP 314. Note that pip itself does not require PEP 314
compatibility - it does not use any of the metadata from sdists - they
are treated like source trees from disk or version control.

References

Copyright

This document has been placed in the public domain.

[1] pip, the recommended installer for Python packages
(http://pip.readthedocs.org/en/stable/)

[2] setuptools, the de facto Python package build system
(https://pythonhosted.org/setuptools/)

[3] flit, a simple way to put packages in PyPI
(http://flit.readthedocs.org/en/latest/)

[4] The Python string formatting syntax.
(https://docs.python.org/3.1/library/string.html#format-string-syntax)

[5] The Python string formatting syntax.
(https://docs.python.org/3.1/library/string.html#format-string-syntax)

[6] The kick-off thread.
(https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html)

[7] The minutes.
(https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html)


text:
PEP: 3155 Title: Qualified name for classes and functions Version:
$Revision$ Last-Modified: $Date$ Author: Antoine Pitrou
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 29-Oct-2011 Python-Version: 3.3 Post-History:
Resolution:
https://mail.python.org/pipermail/python-dev/2011-November/114545.html

Rationale

Python's introspection facilities have long had poor support for nested
classes. Given a class object, it is impossible to know whether it was
defined inside another class or at module top-level; and, if the former,
it is also impossible to know in which class it was defined. While use
of nested classes is often considered poor style, the only reason for
them to have second class introspection support is a lousy pun.

Python 3 adds insult to injury by dropping what was formerly known as
unbound methods. In Python 2, given the following definition:

    class C:
        def f():
            pass

you can then walk up from the C.f object to its defining class:

    >>> C.f.im_class
    <class '__main__.C'>

This possibility is gone in Python 3:

    >>> C.f.im_class
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'function' object has no attribute 'im_class'
    >>> dir(C.f)
    ['__annotations__', '__call__', '__class__', '__closure__', '__code__',
    '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__',
    '__eq__', '__format__', '__ge__', '__get__', '__getattribute__',
    '__globals__', '__gt__', '__hash__', '__init__', '__kwdefaults__',
    '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__',
    '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__',
    '__str__', '__subclasshook__']

This limits again the introspection capabilities available to the user.
It can produce actual issues when porting software to Python 3, for
example Twisted Core where the issue of introspecting method objects
came up several times. It also limits pickling support[1].

Proposal

This PEP proposes the addition of a __qualname__ attribute to functions
and classes. For top-level functions and classes, the __qualname__
attribute is equal to the __name__ attribute. For nested classes,
methods, and nested functions, the __qualname__ attribute contains a
dotted path leading to the object from the module top-level. A
function's local namespace is represented in that dotted path by a
component named <locals>.

The repr() and str() of functions and classes is modified to use
__qualname__ rather than __name__.

Example with nested classes

>>> class C: ... def f(): pass ... class D: ... def g(): pass ... >>>
C.__qualname__ 'C' >>> C.f.__qualname__ 'C.f' >>> C.D.__qualname__
'C.D' >>> C.D.g.__qualname__ 'C.D.g'

Example with nested functions

>>> def f(): ... def g(): pass ... return g ... >>> f.__qualname__
'f' >>> f().__qualname__ 'f.<locals>.g'

Limitations

With nested functions (and classes defined inside functions), the dotted
path will not be walkable programmatically as a function's namespace is
not available from the outside. It will still be more helpful to the
human reader than the bare __name__.

As the __name__ attribute, the __qualname__ attribute is computed
statically and it will not automatically follow rebinding.

Discussion

Excluding the module name

As __name__, __qualname__ doesn't include the module name. This makes it
independent of module aliasing and rebinding, and also allows to compute
it at compile time.

Reviving unbound methods

Reviving unbound methods would only solve a fraction of the problems
this PEP solves, at a higher price (an additional object type and an
additional indirection, rather than an additional attribute).

Naming choice

"Qualified name" is the best approximation, as a short phrase, of what
the additional attribute is about. It is not a "full name" or "fully
qualified name" since it (deliberately) does not include the module
name. Calling it a "path" would risk confusion with filesystem paths and
the __file__ attribute.

The first proposal for the attribute name was to call it __qname__ but
many people (who are not aware of previous use of such jargon in e.g.
the XML specification[2]) found it obscure and non-obvious, which is why
the slightly less short and more explicit __qualname__ was finally
chosen.

References

Copyright

This document has been placed in the public domain.

[1] "pickle should support methods": http://bugs.python.org/issue9276

[2] "QName" entry in Wikipedia: http://en.wikipedia.org/wiki/QName


text:
PEP: 624 Title: Remove Py_UNICODE encoder APIs Author: Inada Naoki
<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 06-Jul-2020 Python-Version: 3.11
Post-History: 08-Jul-2020

Abstract

This PEP proposes to remove deprecated Py_UNICODE encoder APIs in Python
3.11:

-   PyUnicode_Encode()
-   PyUnicode_EncodeASCII()
-   PyUnicode_EncodeLatin1()
-   PyUnicode_EncodeUTF7()
-   PyUnicode_EncodeUTF8()
-   PyUnicode_EncodeUTF16()
-   PyUnicode_EncodeUTF32()
-   PyUnicode_EncodeUnicodeEscape()
-   PyUnicode_EncodeRawUnicodeEscape()
-   PyUnicode_EncodeCharmap()
-   PyUnicode_TranslateCharmap()
-   PyUnicode_EncodeDecimal()
-   PyUnicode_TransformDecimalToASCII()

Note

PEP 623 propose to remove Unicode object APIs relating to Py_UNICODE. On
the other hand, this PEP is not relating to Unicode object. These PEPs
are split because they have different motivations and need different
discussions.

Motivation

In general, reducing the number of APIs that have been deprecated for a
long time and have few users is a good idea for not only it improves the
maintainability of CPython, but it also helps API users and other Python
implementations.

Rationale

Deprecated since Python 3.3

Py_UNICODE and APIs using it has been deprecated since Python 3.3.

Inefficient

All of these APIs are implemented using PyUnicode_FromWideChar. So these
APIs are inefficient when user want to encode Unicode object.

Not used widely

When searching from the top 4000 PyPI packages[1], only pyodbc use these
APIs.

-   PyUnicode_EncodeUTF8()
-   PyUnicode_EncodeUTF16()

pyodbc uses these APIs to encode Unicode object into bytes object. So it
is easy to fix it.[2]

Alternative APIs

There are alternative APIs to accept PyObject *unicode instead of
Py_UNICODE *. Users can migrate to them.

+-------------------------------------+--------------------------------------+
| Deprecated API                      | Alternative APIs                     |
+=====================================+======================================+
| PyUnicode_Encode()                  | PyUnicode_AsEncodedString()          |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeASCII()             | PyUnicode_AsASCIIString() (1)        |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeLatin1()            | PyUnicode_AsLatin1String() (1)       |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeUTF7()              | (2)                                  |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeUTF8()              | PyUnicode_AsUTF8String() (1)         |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeUTF16()             | PyUnicode_AsUTF16String() (3)        |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeUTF32()             | PyUnicode_AsUTF32String() (3)        |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeUnicodeEscape()     | PyUnicode_AsUnicodeEscapeString()    |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeRawUnicodeEscape()  | PyUnicode_AsRawUnicodeEscapeString() |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeCharmap()           | PyUnicode_AsCharmapString() (1)      |
+-------------------------------------+--------------------------------------+
| PyUnicode_TranslateCharmap()        | PyUnicode_Translate()                |
+-------------------------------------+--------------------------------------+
| PyUnicode_EncodeDecimal()           |   (4)                                |
+-------------------------------------+--------------------------------------+
| PyUnicode_TransformDecimalToASCII() |   (4)                                |
+-------------------------------------+--------------------------------------+

Notes:

(1)

    const char *errors parameter is missing.

(2)

    There is no public alternative API. But user can use generic
    PyUnicode_AsEncodedString() instead.

(3)

    const char *errors, int byteorder parameters are missing.

(4)

    There is no direct replacement. But Py_UNICODE_TODECIMAL can be used
    instead. CPython uses _PyUnicode_TransformDecimalAndSpaceToASCII for
    converting from Unicode to numbers instead.

Plan

Remove these APIs in Python 3.11. They have been deprecated already.

-   PyUnicode_Encode()
-   PyUnicode_EncodeASCII()
-   PyUnicode_EncodeLatin1()
-   PyUnicode_EncodeUTF7()
-   PyUnicode_EncodeUTF8()
-   PyUnicode_EncodeUTF16()
-   PyUnicode_EncodeUTF32()
-   PyUnicode_EncodeUnicodeEscape()
-   PyUnicode_EncodeRawUnicodeEscape()
-   PyUnicode_EncodeCharmap()
-   PyUnicode_TranslateCharmap()
-   PyUnicode_EncodeDecimal()
-   PyUnicode_TransformDecimalToASCII()

Alternative Ideas

Replace Py_UNICODE* with PyObject*

As described in the "Alternative APIs" section, some APIs don't have
public alternative APIs accepting PyObject *unicode input. And some
public alternative APIs have restrictions like missing errors and
byteorder parameters.

Instead of removing deprecated APIs, we can reuse their names for
alternative public APIs.

Since we have private alternative APIs already, it is just renaming from
private name to public and deprecated names.

+--------------------------+-------------------------------+
| Rename to                | Rename from                   |
+==========================+===============================+
| PyUnicode_EncodeASCII()  |   _PyUnicode_AsASCIIString()  |
+--------------------------+-------------------------------+
| PyUnicode_EncodeLatin1() |   _PyUnicode_AsLatin1String() |
+--------------------------+-------------------------------+
| PyUnicode_EncodeUTF7()   |   _PyUnicode_EncodeUTF7()     |
+--------------------------+-------------------------------+
| PyUnicode_EncodeUTF8()   |   _PyUnicode_AsUTF8String()   |
+--------------------------+-------------------------------+
| PyUnicode_EncodeUTF16()  |   _PyUnicode_EncodeUTF16()    |
+--------------------------+-------------------------------+
| PyUnicode_EncodeUTF32()  |   _PyUnicode_EncodeUTF32()    |
+--------------------------+-------------------------------+

Pros:

-   We have a more consistent API set.

Cons:

-   Backward incompatible.
-   We have more public APIs to maintain for rare use cases.
-   Existing public APIs are enough for most use cases, and
    PyUnicode_AsEncodedString() can be used in other cases.

Replace Py_UNICODE* with Py_UCS4*

We can replace Py_UNICODE with Py_UCS4 and undeprecate these APIs.

UTF-8, UTF-16, UTF-32 encoders support Py_UCS4 internally. So
PyUnicode_EncodeUTF8(), PyUnicode_EncodeUTF16(), and
PyUnicode_EncodeUTF32() can avoid to create a temporary Unicode object.

Pros:

-   We can avoid creating temporary Unicode object when encoding from
    Py_UCS4* into bytes object with UTF-8, UTF-16, UTF-32 codecs.

Cons:

-   Backward incompatible.
-   We have more public APIs to maintain for rare use cases.
-   Other Python implementations that want to support Python/C API need
    to support these APIs too.
-   If we change the Unicode internal representation to UTF-8 in the
    future, we need to keep UCS-4 support only for these APIs.

Replace Py_UNICODE* with wchar_t*

We can replace Py_UNICODE with wchar_t. Since Py_UNICODE is typedef of
wchar_t already, this is status quo.

On platforms where sizeof(wchar_t) == 4, we can avoid to create a
temporary Unicode object when encoding from wchar_t* to bytes objects
using UTF-8, UTF-16, and UTF-32 codec, like the "Replace Py_UNICODE*
with Py_UCS4*" idea.

Pros:

-   Backward compatible.
-   We can avoid creating temporary Unicode object when encode from
    Py_UCS4* into bytes object with UTF-8, UTF-16, UTF-32 codecs on
    platform where sizeof(wchar_t) == 4.

Cons:

-   Although Windows is the most major platform that uses wchar_t
    heavily, these APIs need to create a temporary Unicode object always
    because sizeof(wchar_t) == 2 on Windows.
-   We have more public APIs to maintain for rare use cases.
-   Other Python implementations that want to support Python/C API need
    to support these APIs too.
-   If we change the Unicode internal representation to UTF-8 in the
    future, we need to keep UCS-4 support only for these APIs.

Rejected Ideas

Emit runtime warning

In addition to existing compiler warning, emitting runtime
DeprecationWarning is suggested.

But these APIs doesn't release GIL for now. Emitting a warning from such
APIs is not safe. See this example.

    PyObject *u = PyList_GET_ITEM(list, i);  // u is borrowed reference.
    PyObject *b = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(u),
            PyUnicode_GET_SIZE(u), NULL);
    // Assumes u is still living reference.
    PyObject *t = PyTuple_Pack(2, u, b);
    Py_DECREF(b);
    return t;

If we emit Python warning from PyUnicode_EncodeUTF8(), warning filters
and other threads may change the list and u can be a dangling reference
after PyUnicode_EncodeUTF8() returned.

Discussions

-   [python-dev] Plan to remove Py_UNICODE APis except PEP 623
-   bpo-41123: Remove Py_UNICODE APIs except PEP 623
-   [python-dev] PEP 624: Remove Py_UNICODE encoder APIs

Objections

-   Removing these APIs removes ability to use codec without temporary
    Unicode.
    -   Codecs can not encode Unicode buffer directly without temporary
        Unicode object since Python 3.3. All these APIs creates
        temporary Unicode object for now. So removing them doesn't
        reduce any abilities.
-   Why not remove decoder APIs too?
    -   They are part of stable ABI.
    -   PyUnicode_DecodeASCII() and PyUnicode_DecodeUTF8() are used very
        widely. Deprecating them is not worth enough.
    -   Decoder APIs can decode from byte buffer directly, without
        creating temporary bytes object. On the other hand, encoder APIs
        can not avoid temporary Unicode object.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] Source package list chosen from top 4000 PyPI packages.
(https://github.com/methane/notes/blob/master/2020/wchar-cache/package_list.txt)

[2] pyodbc -- Don't use PyUnicode_Encode API #792
(https://github.com/mkleehammer/pyodbc/pull/792)