text:
PEP: 725 Title: Specifying external dependencies in pyproject.toml
Author: Pradyun Gedam<EMAIL_ADDRESS>Ralf Gommers
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/31888 Status: Draft Type: Standards Track
Topic: Packaging Content-Type: text/x-rst Created: 17-Aug-2023
Post-History: 18-Aug-2023

Abstract

This PEP specifies how to write a project's external, or non-PyPI, build
and runtime dependencies in a pyproject.toml file for packaging-related
tools to consume.

This PEP proposes to add an [external] table to pyproject.toml with
three keys: "build-requires", "host-requires" and "dependencies". These
are for specifying three types of dependencies:

1.  build-requires, build tools to run on the build machine
2.  host-requires, build dependencies needed for host machine but also
    needed at build time.
3.  dependencies, needed at runtime on the host machine but not needed
    at build time.

Cross compilation is taken into account by distinguishing build and host
dependencies. Optional build-time and runtime dependencies are supported
too, in a manner analogies to how that is supported in the [project]
table.

Motivation

Python packages may have dependencies on build tools, libraries,
command-line tools, or other software that is not present on PyPI.
Currently there is no way to express those dependencies in standardized
metadata [1],[2]. Key motivators for this PEP are to:

-   Enable tools to automatically map external dependencies to packages
    in other packaging repositories,
-   Make it possible to include needed dependencies in error messages
    emitting by Python package installers and build frontends,
-   Provide a canonical place for package authors to record this
    dependency information.

Packaging ecosystems like Linux distros, Conda, Homebrew, Spack, and Nix
need full sets of dependencies for Python packages, and have tools like
pyp2spec (Fedora), Grayskull (Conda), and dh_python (Debian) which
attempt to automatically generate dependency metadata for their own
package managers from the metadata in upstream Python packages. External
dependencies are currently handled manually, because there is no
metadata for this in pyproject.toml or any other standard location.
Enabling automating this conversion is a key benefit of this PEP, making
packaging Python packages for distros easier and more reliable. In
addition, the authors envision other types of tools making use of this
information, e.g., dependency analysis tools like Repology, Dependabot
and libraries.io. Software bill of materials (SBOM) generation tools may
also be able to use this information, e.g. for flagging that external
dependencies listed in pyproject.toml but not contained in wheel
metadata are likely vendored within the wheel.

Packages with external dependencies are typically hard to build from
source, and error messages from build failures tend to be hard to
decipher for end users. Missing external dependencies on the end user's
system are the most likely cause of build failures. If installers can
show the required external dependencies as part of their error message,
this may save users a lot of time.

At the moment, information on external dependencies is only captured in
installation documentation of individual packages. It is hard to
maintain for package authors and tends to go out of date. It's also hard
for users and distro packagers to find it. Having a canonical place to
record this dependency information will improve this situation.

This PEP is not trying to specify how the external dependencies should
be used, nor a mechanism to implement a name mapping from names of
individual packages that are canonical for Python projects published on
PyPI to those of other packaging ecosystems. Those topics should be
addressed in separate PEPs.

Rationale

Types of external dependencies

Multiple types of external dependencies can be distinguished:

-   Concrete packages that can be identified by name and have a
    canonical location in another language-specific package repository.
    E.g., Rust packages on crates.io, R packages on CRAN, JavaScript
    packages on the npm registry.
-   Concrete packages that can be identified by name but do not have a
    clear canonical location. This is typically the case for libraries
    and tools written in C, C++, Fortran, CUDA and other low-level
    languages. E.g., Boost, OpenSSL, Protobuf, Intel MKL, GCC.
-   "Virtual" packages, which are names for concepts, types of tools or
    interfaces. These typically have multiple implementations, which are
    concrete packages. E.g., a C++ compiler, BLAS, LAPACK, OpenMP, MPI.

Concrete packages are straightforward to understand, and are a concept
present in virtually every package management system. Virtual packages
are a concept also present in a number of packaging systems -- but not
always, and the details of their implementation varies.

Cross compilation

Cross compilation is not yet (as of August 2023) well-supported by
stdlib modules and pyproject.toml metadata. It is however important when
translating external dependencies to those of other packaging systems
(with tools like pyp2spec). Introducing support for cross compilation
immediately in this PEP is much easier than extending [external] in the
future, hence the authors choose to include this now.

Terminology

This PEP uses the following terminology:

-   build machine: the machine on which the package build process is
    being executed
-   host machine: the machine on which the produced artifact will be
    installed and run
-   build dependency: dependency for building the package that needs to
    be present at build time and itself was built for the build
    machine's OS and architecture
-   host dependency: dependency for building the package that needs to
    be present at build time and itself was built for the host machine's
    OS and architecture

Note that this terminology is not consistent across build and packaging
tools, so care must be taken when comparing build/host dependencies in
pyproject.toml to dependencies from other package managers.

Note that "target machine" or "target dependency" is not used in this
PEP. That is typically only relevant for cross-compiling compilers or
other such advanced scenarios[3],[4] - this is out of scope for this
PEP.

Finally, note that while "dependency" is the term most widely used for
packages needed at build time, the existing key in pyproject.toml for
PyPI build-time dependencies is build-requires. Hence this PEP uses the
keys build-requires and host-requires under [external] for consistency.

Build and host dependencies

Clear separation of metadata associated with the definition of build and
target platforms, rather than assuming that build and target platform
will always be the same, is important[5].

Build dependencies are typically run during the build process - they may
be compilers, code generators, or other such tools. In case the use of a
build dependency implies a runtime dependency, that runtime dependency
does not have to be declared explicitly. For example, when compiling
Fortran code with gfortran into a Python extension module, the package
likely incurs a dependency on the libgfortran runtime library. The
rationale for not explicitly listing such runtime dependencies is
two-fold: (1) it may depend on compiler/linker flags or details of the
build environment whether the dependency is present, and (2) these
runtime dependencies can be detected and handled automatically by tools
like auditwheel.

Host dependencies are typically not run during the build process, but
only used for linking against. This is not a rule though -- it may be
possible or necessary to run a host dependency under an emulator, or
through a custom tool like crossenv. When host dependencies imply a
runtime dependency, that runtime dependency also does not have to be
declared, just like for build dependencies.

When host dependencies are declared and a tool is not cross-compilation
aware and has to do something with external dependencies, the tool MAY
merge the host-requires list into build-requires. This may for example
happen if an installer like pip starts reporting external dependencies
as a likely cause of a build failure when a package fails to build from
an sdist.

Specifying external dependencies

Concrete package specification through PURL

The two types of concrete packages are supported by PURL (Package URL),
which implements a scheme for identifying packages that is meant to be
portable across packaging ecosystems. Its design is:

    scheme:type/namespace/name@version?qualifiers#subpath

The scheme component is a fixed string, pkg, and of the other components
only type and name are required. As an example, a package URL for the
requests package on PyPI would be:

    pkg:pypi/requests

Adopting PURL to specify external dependencies in pyproject.toml solves
a number of problems at once - and there are already implementations of
the specification in Python and multiple languages. PURL is also already
supported by dependency-related tooling like SPDX (see External
Repository Identifiers in the SPDX 2.3 spec), the Open Source
Vulnerability format, and the Sonatype OSS Index; not having to wait
years before support in such tooling arrives is valuable.

For concrete packages without a canonical package manager to refer to,
either pkg:generic/pkg-name can be used, or a direct reference to the
VCS system that the package is maintained in (e.g.,
pkg:github/user-or-org-name/pkg-name). Which of these is more
appropriate is situation-dependent. This PEP recommends using
pkg:generic when the package name is unambiguous and well-known (e.g.,
pkg:generic/git or pkg:generic/openblas), and using the VCS as the PURL
type otherwise.

Virtual package specification

There is no ready-made support for virtual packages in PURL or another
standard. There are a relatively limited number of such dependencies
though, and adopting a scheme similar to PURL but with the virtual:
rather than pkg: scheme seems like it will be understandable and map
well to Linux distros with virtual packages and to the likes of Conda
and Spack.

The two known virtual package types are compiler and interface.

Versioning

Support in PURL for version expressions and ranges beyond a fixed
version is still pending, see the Open Issues section.

Dependency specifiers

Regular Python dependency specifiers (as originally defined in PEP 508)
may be used behind PURLs. PURL qualifiers, which use ? followed by a
package type-specific dependency specifier component, must not be used.
The reason for this is pragmatic: dependency specifiers are already used
for other metadata in pyproject.toml, any tooling that is used with
pyproject.toml is likely to already have a robust implementation to
parse it. And we do not expect to need the extra possibilities that PURL
qualifiers provide (e.g. to specify a Conan or Conda channel, or a
RubyGems platform).

Usage of core metadata fields

The core metadata specification contains one relevant field, namely
Requires-External. This has no well-defined semantics in core metadata
2.1; this PEP chooses to reuse the field for external runtime
dependencies. The core metadata specification does not contain fields
for any metadata in pyproject.toml's [build-system] table. Therefore the
build-requires and host-requires content also does not need to be
reflected in core metadata fields. The optional-dependencies content
from [external] would need to either reuse Provides-Extra or require a
new Provides-External-Extra field. Neither seems desirable.

Differences between sdist and wheel metadata

A wheel may vendor its external dependencies. This happens in particular
when distributing wheels on PyPI or other Python package indexes - and
tools like auditwheel, delvewheel and delocate automate this process. As
a result, a Requires-External entry in an sdist may disappear from a
wheel built from that sdist. It is also possible that a
Requires-External entry remains in a wheel, either unchanged or with
narrower constraints. auditwheel does not vendor certain allow-listed
dependencies, such as OpenGL, by default. In addition, auditwheel and
delvewheel allow a user to manually exclude dependencies via a --exclude
or --no-dll command-line flag. This is used to avoid vendoring large
shared libraries, for example those from CUDA.

Requires-External entries generated from external dependencies in
pyproject.toml in a wheel are therefore allowed to be narrower than
those for the corresponding sdist. They must not be wider, i.e.
constraints must not allow a version of a dependency for a wheel that
isn't allowed for an sdist, nor contain new dependencies that are not
listed in the sdist's metadata at all.

Canonical names of dependencies and -dev(el) split packages

It is fairly common for distros to split a package into two or more
packages. In particular, runtime components are often separately
installable from development components (headers, pkg-config and CMake
files, etc.). The latter then typically has a name with -dev or -devel
appended to the project/library name. This split is the responsibility
of each distro to maintain, and should not be reflected in the
[external] table. It is not possible to specify this in a reasonable way
that works across distros, hence only the canonical name should be used
in [external].

The intended meaning of using a PURL or virtual dependency is "the full
package with the name specified". It will depend on the context in which
the metadata is used whether the split is relevant. For example, if
libffi is a host dependency and a tool wants to prepare an environment
for building a wheel, then if a distro has split off the headers for
libffi into a libffi-devel package then the tool has to install both
libffi and libffi-devel.

Python development headers

Python headers and other build support files may also be split. This is
the same situation as in the section above (because Python is simply a
regular package in distros). However, a python-dev|devel dependency is
special because in pyproject.toml Python itself is an implicit rather
than an explicit dependency. Hence a choice needs to be made here - add
python-dev implicitly, or make each package author add it explicitly
under [external]. For consistency between Python dependencies and
external dependencies, we choose to add it implicitly. Python
development headers must be assumed to be necessary when an [external]
table contains one or more compiler packages.

Specification

If metadata is improperly specified then tools MUST raise an error to
notify the user about their mistake.

Details

Note that pyproject.toml content is in the same format as in PEP 621.

Table name

Tools MUST specify fields defined by this PEP in a table named
[external]. No tools may add fields to this table which are not defined
by this PEP or subsequent PEPs. The lack of an [external] table means
the package either does not have any external dependencies, or the ones
it does have are assumed to be present on the system already.

build-requires/optional-build-requires

-   Format: Array of PURL strings (build-requires) and a table with
    values of arrays of PURL strings (optional-build-requires)
-   Core metadata: N/A

The (optional) external build requirements needed to build the project.

For build-requires, it is a key whose value is an array of strings. Each
string represents a build requirement of the project and MUST be
formatted as either a valid PURL string or a virtual: string.

For optional-build-requires, it is a table where each key specifies an
extra set of build requirements and whose value is an array of strings.
The strings of the arrays MUST be valid PURL strings.

host-requires/optional-host-requires

-   Format: Array of PURL strings (host-requires) and a table with
    values of arrays of PURL strings (optional-host-requires)
-   Core metadata: N/A

The (optional) external host requirements needed to build the project.

For host-requires, it is a key whose value is an array of strings. Each
string represents a host requirement of the project and MUST be
formatted as either a valid PURL string or a virtual: string.

For optional-host-requires, it is a table where each key specifies an
extra set of host requirements and whose value is an array of strings.
The strings of the arrays MUST be valid PURL strings.

dependencies/optional-dependencies

-   Format: Array of PURL strings (dependencies) and a table with values
    of arrays of PURL strings (optional-dependencies)
-   Core metadata: Requires-External, N/A

The (optional) runtime dependencies of the project.

For dependencies, it is a key whose value is an array of strings. Each
string represents a dependency of the project and MUST be formatted as
either a valid PURL string or a virtual: string. Each string maps
directly to a Requires-External entry in the core metadata.

For optional-dependencies, it is a table where each key specifies an
extra and whose value is an array of strings. The strings of the arrays
MUST be valid PURL strings. Optional dependencies do not map to a core
metadata field.

Examples

These examples show what the [external] content for a number of packages
is expected to be.

cryptography 39.0:

    [external]
    build-requires = [
      "virtual:compiler/c",
      "virtual:compiler/rust",
      "pkg:generic/pkg-config",
    ]
    host-requires = [
      "pkg:generic/openssl",
      "pkg:generic/libffi",
    ]

SciPy 1.10:

    [external]
    build-requires = [
      "virtual:compiler/c",
      "virtual:compiler/cpp",
      "virtual:compiler/fortran",
      "pkg:generic/ninja",
      "pkg:generic/pkg-config",
    ]
    host-requires = [
      "virtual:interface/blas",
      "virtual:interface/lapack",  # >=3.7.1 (can't express version ranges with PURL yet)
    ]

Pillow 10.1.0:

    [external]
    build-requires = [
      "virtual:compiler/c",
    ]
    host-requires = [
      "pkg:generic/libjpeg",
      "pkg:generic/zlib",
    ]

    [external.optional-host-requires]
    extra = [
      "pkg:generic/lcms2",
      "pkg:generic/freetype",
      "pkg:generic/libimagequant",
      "pkg:generic/libraqm",
      "pkg:generic/libtiff",
      "pkg:generic/libxcb",
      "pkg:generic/libwebp",
      "pkg:generic/openjpeg",  # add >=2.0 once we have version specifiers
      "pkg:generic/tk",
    ]

NAVis 1.4.0:

    [project.optional-dependencies]
    r = ["rpy2"]

    [external]
    build-requires = [
      "pkg:generic/XCB; platform_system=='Linux'",
    ]

    [external.optional-dependencies]
    nat = [
      "pkg:cran/nat",
      "pkg:cran/nat.nblast",
    ]

Spyder 6.0:

    [external]
    dependencies = [
      "pkg:cargo/ripgrep",
      "pkg:cargo/tree-sitter-cli",
      "pkg:golang/github.com/junegunn/fzf",
    ]

jupyterlab-git 0.41.0:

    [external]
    dependencies = [
      "pkg:generic/git",
    ]

    [external.optional-build-requires]
    dev = [
      "pkg:generic/nodejs",
    ]

PyEnchant 3.2.2:

    [external]
    dependencies = [
      # libenchant is needed on all platforms but only vendored into wheels on
      # Windows, so on Windows the build backend should remove this external
      # dependency from wheel metadata.
      "pkg:github/AbiWord/enchant",
    ]

Backwards Compatibility

There is no impact on backwards compatibility, as this PEP only adds
new, optional metadata. In the absence of such metadata, nothing changes
for package authors or packaging tooling.

Security Implications

There are no direct security concerns as this PEP covers how to
statically define metadata for external dependencies. Any security
issues would stem from how tools consume the metadata and choose to act
upon it.

How to Teach This

External dependencies and if and how those external dependencies are
vendored are topics that are typically not understood in detail by
Python package authors. We intend to start from how an external
dependency is defined, the different ways it can be depended on---from
runtime-only with ctypes or a subprocess call to it being a build
dependency that's linked against---before going into how to declare
external dependencies in metadata. The documentation should make
explicit what is relevant for package authors, and what for distro
packagers.

Material on this topic will be added to the most relevant packaging
tutorials, primarily the Python Packaging User Guide. In addition, we
expect that any build backend that adds support for external
dependencies metadata will include information about that in its
documentation, as will tools like auditwheel.

Reference Implementation

This PEP contains a metadata specification, rather that a code feature -
hence there will not be code implementing the metadata spec as a whole.
However, there are parts that do have a reference implementation:

1.  The [external] table has to be valid TOML and therefore can be
    loaded with tomllib.
2.  The PURL specification, as a key part of this spec, has a Python
    package with a reference implementation for constructing and parsing
    PURLs: packageurl-python.

There are multiple possible consumers and use cases of this metadata,
once that metadata gets added to Python packages. Tested metadata for
all of the top 150 most-downloaded packages from PyPI with published
platform-specific wheels can be found in rgommers/external-deps-build.
This metadata has been validated by using it to build wheels from sdists
patched with that metadata in clean Docker containers.

Rejected Ideas

Specific syntax for external dependencies which are also packaged on PyPI

There are non-Python packages which are packaged on PyPI, such as Ninja,
patchelf and CMake. What is typically desired is to use the system
version of those, and if it's not present on the system then install the
PyPI package for it. The authors believe that specific support for this
scenario is not necessary (or too complex to justify such support); a
dependency provider for external dependencies can treat PyPI as one
possible source for obtaining the package.

Using library and header names as external dependencies

A previous draft PEP ("External dependencies" (2015)) proposed using
specific library and header names as external dependencies. This is too
granular; using package names is a well-established pattern across
packaging ecosystems and should be preferred.

Open Issues

Version specifiers for PURLs

Support in PURL for version expressions and ranges is still pending. The
pull request at vers implementation for PURL seems close to being
merged, at which point this PEP could adopt it.

Versioning of virtual dependencies

Once PURL supports version expressions, virtual dependencies can be
versioned with the same syntax. It must be better specified however what
the version scheme is, because this is not as clear for virtual
dependencies as it is for PURLs (e.g., there can be multiple
implementations, and abstract interfaces may not be unambiguously
versioned). E.g.:

-   OpenMP: has regular MAJOR.MINOR versions of its standard, so would
    look like >=4.5.
-   BLAS/LAPACK: should use the versioning used by Reference LAPACK,
    which defines what the standard APIs are. Uses MAJOR.MINOR.MICRO, so
    would look like >=3.10.0.
-   Compilers: these implement language standards. For C, C++ and
    Fortran these are versioned by year. In order for versions to sort
    correctly, we choose to use the full year (four digits). So "at
    least C99" would be >=1999, and selecting C++14 or Fortran 77 would
    be ==2014 or ==1977 respectively. Other languages may use different
    versioning schemes. These should be described somewhere before they
    are used in pyproject.toml.

A logistical challenge is where to describe the versioning - given that
this will evolve over time, this PEP itself is not the right location
for it. Instead, this PEP should point at that (to be created) location.

Who defines canonical names and canonical package structure?

Similarly to the logistics around versioning is the question about what
names are allowed and where they are described. And then who is in
control of that description and responsible for maintaining it. Our
tentative answer is: there should be a central list for virtual
dependencies and pkg:generic PURLs, maintained as a PyPA project. See
https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/62.
TODO: once that list/project is prototyped, include it in the PEP and
close this open issue.

Syntax for virtual dependencies

The current syntax this PEP uses for virtual dependencies is
virtual:type/name, which is analogous to but not part of the PURL spec.
This open issue discusses supporting virtual dependencies within PURL:
purl-spec#222.

Should a host-requires key be added under [build-system]?

Adding host-requires for host dependencies that are on PyPI in order to
better support name mapping to other packaging systems with support for
cross-compiling may make sense. This issue tracks this topic and has
arguments in favor and against adding host-requires under [build-system]
as part of this PEP.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] The "define native requirements metadata" part of the "Wanting a
singular packaging vision" thread (2022, Discourse):
https://discuss.python.org/t/wanting-a-singular-packaging-tool-vision/21141/92

[2] pypackaging-native: "Native dependencies"
https://pypackaging-native.github.io/key-issues/native-dependencies/

[3] GCC documentation - Configure Terms and History,
https://gcc.gnu.org/onlinedocs/gccint/Configure-Terms.html

[4] Meson documentation - Cross compilation
https://mesonbuild.com/Cross-compilation.html

[5] pypackaging-native: "Cross compilation"
https://pypackaging-native.github.io/key-issues/cross_compilation/


text:
PEP: 3129 Title: Class Decorators Version: $Revision$ Last-Modified:
$Date$ Author: Collin Winter<EMAIL_ADDRESS>Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 01-May-2007
Python-Version: 3.0 Post-History: 07-May-2007

Abstract

This PEP proposes class decorators, an extension to the function and
method decorators introduced in PEP 318.

Rationale

When function decorators were originally debated for inclusion in Python
2.4, class decorators were seen as
obscure and unnecessary <318#motivation> thanks to metaclasses. After
several years' experience with the Python 2.4.x series of releases and
an increasing familiarity with function decorators and their uses, the
BDFL and the community re-evaluated class decorators and recommended
their inclusion in Python 3.0[1].

The motivating use-case was to make certain constructs more easily
expressed and less reliant on implementation details of the CPython
interpreter. While it is possible to express class decorator-like
functionality using metaclasses, the results are generally unpleasant
and the implementation highly fragile[2]. In addition, metaclasses are
inherited, whereas class decorators are not, making metaclasses
unsuitable for some, single class-specific uses of class decorators. The
fact that large-scale Python projects like Zope were going through these
wild contortions to achieve something like class decorators won over the
BDFL.

Semantics

The semantics and design goals of class decorators are the same as for
function decorators (318#current-syntax, 318#design-goals); the only
difference is that you're decorating a class instead of a function. The
following two snippets are semantically identical:

    class A:
      pass
    A = foo(bar(A))

    @foo
    @bar
    class A:
      pass

For a detailed examination of decorators, please refer to PEP 318.

Implementation

Adapting Python's grammar to support class decorators requires modifying
two rules and adding a new rule:

    funcdef: [decorators] 'def' NAME parameters ['->' test] ':' suite

    compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt |
                   with_stmt | funcdef | classdef

need to be changed to :

    decorated: decorators (classdef | funcdef)

    funcdef: 'def' NAME parameters ['->' test] ':' suite

    compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt |
                   with_stmt | funcdef | classdef | decorated

Adding decorated is necessary to avoid an ambiguity in the grammar.

The Python AST and bytecode must be modified accordingly.

A reference implementation[3] has been provided by Jack Diederich.

Acceptance

There was virtually no discussion following the posting of this PEP,
meaning that everyone agreed it should be accepted.

The patch was committed to Subversion as revision 55430.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] https://mail.python.org/pipermail/python-dev/2006-March/062942.html

[2] https://mail.python.org/pipermail/python-dev/2006-March/062888.html

[3] https://bugs.python.org/issue1671208

license: Public Domain


text:
PEP: 562 Title: Module __getattr__ and __dir__ Author: Ivan Levkivskyi
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History:
09-Sep-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-December/151033.html

Abstract

It is proposed to support __getattr__ and __dir__ function defined on
modules to provide basic customization of module attribute access.

Rationale

It is sometimes convenient to customize or otherwise have control over
access to module attributes. A typical example is managing deprecation
warnings. Typical workarounds are assigning __class__ of a module object
to a custom subclass of types.ModuleType or replacing the sys.modules
item with a custom wrapper instance. It would be convenient to simplify
this procedure by recognizing __getattr__ defined directly in a module
that would act like a normal __getattr__ method, except that it will be
defined on module instances. For example:

    # lib.py

    from warnings import warn

    deprecated_names = ["old_function", ...]

    def _deprecated_old_function(arg, other):
        ...

    def __getattr__(name):
        if name in deprecated_names:
            warn(f"{name} is deprecated", DeprecationWarning)
            return globals()[f"_deprecated_{name}"]
        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

    # main.py

    from lib import old_function  # Works, but emits the warning

Another widespread use case for __getattr__ would be lazy submodule
imports. Consider a simple example:

    # lib/__init__.py

    import importlib

    __all__ = ['submod', ...]

    def __getattr__(name):
        if name in __all__:
            return importlib.import_module("." + name, __name__)
        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

    # lib/submod.py

    print("Submodule loaded")
    class HeavyClass:
        ...

    # main.py

    import lib
    lib.submod.HeavyClass  # prints "Submodule loaded"

There is a related proposal PEP 549 that proposes to support instance
properties for a similar functionality. The difference is this PEP
proposes a faster and simpler mechanism, but provides more basic
customization. An additional motivation for this proposal is that PEP
484 already defines the use of module __getattr__ for this purpose in
Python stub files, see 484#stub-files.

In addition, to allow modifying result of a dir() call on a module to
show deprecated and other dynamically generated attributes, it is
proposed to support module level __dir__ function. For example:

    # lib.py

    deprecated_names = ["old_function", ...]
    __all__ = ["new_function_one", "new_function_two", ...]

    def new_function_one(arg, other):
       ...
    def new_function_two(arg, other):
        ...

    def __dir__():
        return sorted(__all__ + deprecated_names)

    # main.py

    import lib

    dir(lib)  # prints ["new_function_one", "new_function_two", "old_function", ...]

Specification

The __getattr__ function at the module level should accept one argument
which is the name of an attribute and return the computed value or raise
an AttributeError:

    def __getattr__(name: str) -> Any: ...

If an attribute is not found on a module object through the normal
lookup (i.e. object.__getattribute__), then __getattr__ is searched in
the module __dict__ before raising an AttributeError. If found, it is
called with the attribute name and the result is returned. Looking up a
name as a module global will bypass module __getattr__. This is
intentional, otherwise calling __getattr__ for builtins will
significantly harm performance.

The __dir__ function should accept no arguments, and return a list of
strings that represents the names accessible on module:

    def __dir__() -> List[str]: ...

If present, this function overrides the standard dir() search on a
module.

The reference implementation for this PEP can be found in[1].

Backwards compatibility and impact on performance

This PEP may break code that uses module level (global) names
__getattr__ and __dir__. (But the language reference explicitly reserves
all undocumented dunder names, and allows "breakage without warning";
see[2].) The performance implications of this PEP are minimal, since
__getattr__ is called only for missing attributes.

Some tools that perform module attributes discovery might not expect
__getattr__. This problem is not new however, since it is already
possible to replace a module with a module subclass with overridden
__getattr__ and __dir__, but with this PEP such problems can occur more
often.

Discussion

Note that the use of module __getattr__ requires care to keep the
referred objects pickleable. For example, the __name__ attribute of a
function should correspond to the name with which it is accessible via
__getattr__:

    def keep_pickleable(func):
        func.__name__ = func.__name__.replace('_deprecated_', '')
        func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
        return func

    @keep_pickleable
    def _deprecated_old_function(arg, other):
        ...

One should be also careful to avoid recursion as one would do with a
class level __getattr__.

To use a module global with triggering __getattr__ (for example if one
wants to use a lazy loaded submodule) one can access it as:

    sys.modules[__name__].some_global

or as:

    from . import some_global

Note that the latter sets the module attribute, thus __getattr__ will be
called only once.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] The reference implementation
(https://github.com/ilevkivskyi/cpython/pull/3/files)

[2] Reserved classes of identifiers
(https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers)


text:
PEP: 707 Title: A simplified signature for __exit__ and __aexit__
Author: Irit Katriel<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/24402 Status: Rejected Type: Standards
Track Content-Type: text/x-rst Created: 18-Feb-2023 Python-Version: 3.12
Post-History: 02-Mar-2023, Resolution:
https://discuss.python.org/t/pep-707-a-simplified-signature-for-exit-and-aexit/24402/46

Rejection Notice

Per the SC:

  We discussed the PEP and have decided to reject it. Our thinking was
  the magic and risk of potential breakage didn’t warrant the benefits.
  We are totally supportive, though, of exploring a potential context
  manager v2 API or __leave__.

Abstract

This PEP proposes to make the interpreter accept context managers whose
~py3.11:object.__exit__ / ~py3.11:object.__aexit__ method takes only a
single exception instance, while continuing to also support the current
(typ, exc, tb) signature for backwards compatibility.

This proposal is part of an ongoing effort to remove the redundancy of
the 3-item exception representation from the language, a relic of
earlier Python versions which now confuses language users while adding
complexity and overhead to the interpreter.

The proposed implementation uses introspection, which is tailored to the
requirements of this use case. The solution ensures the safety of the
new feature by supporting it only in non-ambiguous cases. In particular,
any signature that could accept three arguments is assumed to expect
them.

Because reliable introspection of callables is not currently possible in
Python, the solution proposed here is limited in that only the common
types of single-arg callables will be identified as such, while some of
the more esoteric ones will continue to be called with three arguments.
This imperfect solution was chosen among several imperfect alternatives
in the spirit of practicality. It is my hope that the discussion about
this PEP will explore the other options and lead us to the best way
forward, which may well be to remain with our imperfect status quo.

Motivation

In the past, an exception was represented in many parts of Python by a
tuple of three elements: the type of the exception, its value, and its
traceback. While there were good reasons for this design at the time,
they no longer hold because the type and traceback can now be reliably
deduced from the exception instance. Over the last few years we saw
several efforts to simplify the representation of exceptions.

Since 3.10 in CPython PR #70577, the py3.11:traceback module's functions
accept either a 3-tuple as described above, or just an exception
instance as a single argument.

Internally, the interpreter no longer represents exceptions as a
triplet. This was removed for the handled exception in 3.11 and for the
raised exception in 3.12. As a consequence, several APIs that expose the
triplet can now be replaced by simpler alternatives:

                                                         Legacy API                 Alternative
  ------------------------------------------------------ -------------------------- ---------------------------
  Get handled exception (Python)                         py3.12:sys.exc_info        py3.12:sys.exception
  Get handled exception (C)                              PyErr_GetExcInfo           PyErr_GetHandledException
  Set handled exception (C)                              PyErr_SetExcInfo           PyErr_SetHandledException
  Get raised exception (C)                               PyErr_Fetch                PyErr_GetRaisedException
  Set raised exception (C)                               PyErr_Restore              PyErr_SetRaisedException
  Construct an exception instance from the 3-tuple (C)   PyErr_NormalizeException   N/A

The current proposal is a step in this process, and considers the way
forward for one more case in which the 3-tuple representation has leaked
to the language. The motivation for all this work is twofold.

Simplify the implementation of the language

The simplification gained by reducing the interpreter's internal
representation of the handled exception to a single object was
significant. Previously, the interpreter needed to push onto/pop from
the stack three items whenever it did anything with exceptions. This
increased stack depth (adding pressure on caches and registers) and
complicated some of the bytecodes. Reducing this to one item removed
about 100 lines of code from ceval.c (the interpreter's eval loop
implementation), and it was later followed by the removal of the
POP_EXCEPT_AND_RERAISE opcode which has become simple enough to be
replaced by generic stack manipulation instructions. Micro-benchmarks
showed a speedup of about 10% for catching and raising an exception, as
well as for creating generators. To summarize, removing this redundancy
in Python's internals simplified the interpreter and made it faster.

The performance of invoking __exit__/__aexit__ when leaving a context
manager can be also improved by replacing a multi-arg function call with
a single-arg one. Micro-benchmarks showed that entering and exiting a
context manager with single-arg __exit__ is about 13% faster.

Simplify the language itself

One of the reasons for the popularity of Python is its simplicity. The
py3.11:sys.exc_info triplet is cryptic for new learners, and the
redundancy in it is confusing for those who do understand it.

It will take multiple releases to get to a point where we can think of
deprecating sys.exc_info(). However, we can relatively quickly reach a
stage where new learners do not need to know about it, or about the
3-tuple representation, at least until they are maintaining legacy code.

Rationale

The only reason to object today to the removal of the last remaining
appearances of the 3-tuple from the language is the concerns about
disruption that such changes can bring. The goal of this PEP is to
propose a safe, gradual and minimally disruptive way to make this change
in the case of __exit__, and with this to initiate a discussion of our
options for evolving its method signature.

In the case of the py3.11:traceback module's API, evolving the functions
to have a hybrid signature is relatively straightforward and safe. The
functions take one positional and two optional arguments, and interpret
them according to their types. This is safe when sentinels are used for
default values. The signatures of callbacks, which are defined by the
user's program, are harder to evolve.

The safest option is to make the user explicitly indicate which
signature the callback is expecting, by marking it with an additional
attribute or giving it a different name. For example, we could make the
interpreter look for a __leave__ method on the context manager, and call
it with a single arg if it exists (otherwise, it looks for __exit__ and
continues as it does now). The introspection-based alternative proposed
here intends to make it more convenient for users to write new code,
because they can just use the single-arg version and remain unaware of
the legacy API. However, if the limitations of introspection are found
to be too severe, we should consider an explicit option. Having both
__exit__ and __leave__ around for 5-10 years with similar functionality
is not ideal, but it is an option.

Let us now examine the limitations of the current proposal. It
identifies 2-arg python functions and METH_O C functions as having a
single-arg signature, and assumes that anything else is expecting 3
args. Obviously it is possible to create false negatives for this
heuristic (single-arg callables that it will not identify). Context
managers written in this way won't work, they will continue to fail as
they do now when their __exit__ function will be called with three
arguments.

I believe that it will not be a problem in practice. First, all working
code will continue to work, so this is a limitation on new code rather
than a problem impacting existing code. Second, exotic callable types
are rarely used for __exit__ and if one is needed, it can always be
wrapped by a plain vanilla method that delegates to the callable. For
example, we can write this:

    class C:
       __enter__ = lambda self: self
       __exit__ = ExoticCallable()

as follows:

    class CM:
       __enter__ = lambda self: self
       _exit = ExoticCallable()
       __exit__ = lambda self, exc: CM._exit(exc)

While discussing the real-world impact of the problem in this PEP, it is
worth noting that most __exit__ functions don't do anything with their
arguments. Typically, a context manager is implemented to ensure that
some cleanup actions take place upon exit. It is rarely appropriate for
the __exit__ function to handle exceptions raised within the context,
and they are typically allowed to propagate out of __exit__ to the
calling function. This means that most __exit__ functions do not access
their arguments at all, and we should take this into account when trying
to assess the impact of different solutions on Python's userbase.

Specification

A context manager's __exit__/__aexit__ method can have a single-arg
signature, in which case it is invoked by the interpreter with the
argument equal to an exception instance or None:

    >>> class C:
    ...     def __enter__(self):
    ...         return self
    ...     def __exit__(self, exc):
    ...         print(f'__exit__ called with: {exc!r}')
    ...
    >>> with C():
    ...     pass
    ...
    __exit__ called with: None
    >>> with C():
    ...     1/0
    ...
    __exit__ called with: ZeroDivisionError('division by zero')
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    ZeroDivisionError: division by zero

If __exit__/__aexit__ has any other signature, it is invoked with the
3-tuple (typ, exc, tb) as happens now:

    >>> class C:
    ...     def __enter__(self):
    ...         return self
    ...     def __exit__(self, *exc):
    ...         print(f'__exit__ called with: {exc!r}')
    ...
    >>> with C():
    ...     pass
    ...
    __exit__ called with: (None, None, None)
    >>> with C():
    ...     1/0
    ...
    __exit__ called with: (<class 'ZeroDivisionError'>, ZeroDivisionError('division by zero'), <traceback object at 0x1039cb570>)
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    ZeroDivisionError: division by zero

These __exit__ methods will also be called with a 3-tuple:

    def __exit__(self, typ, *exc):
        pass

    def __exit__(self, typ, exc, tb):
        pass

A reference implementation is provided in CPython PR #101995.

When the interpreter reaches the end of the scope of a context manager,
and it is about to call the relevant __exit__ or __aexit__ function, it
instrospects this function to determine whether it is the single-arg or
the legacy 3-arg version. In the draft PR, this introspection is
performed by the is_legacy___exit__ function:

    static int is_legacy___exit__(PyObject *exit_func) {
        if (PyMethod_Check(exit_func)) {
            PyObject *func = PyMethod_GET_FUNCTION(exit_func);
            if (PyFunction_Check(func)) {
                PyCodeObject *code = (PyCodeObject*)PyFunction_GetCode(func);
                if (code->co_argcount == 2 && !(code->co_flags & CO_VARARGS)) {
                    /* Python method that expects self + one more arg */
                    return false;
                }
            }
        }
        else if (PyCFunction_Check(exit_func)) {
            if (PyCFunction_GET_FLAGS(exit_func) == METH_O) {
                /* C function declared as single-arg */
                return false;
             }
        }
        return true;
    }

It is important to note that this is not a generic introspection
function, but rather one which is specifically designed for our use
case. We know that exit_func is an attribute of the context manager
class (taken from the type of the object that provided __enter__), and
it is typically a function. Furthermore, for this to be useful we need
to identify enough single-arg forms, but not necessarily all of them.
What is critical for backwards compatibility is that we will never
misidentify a legacy exit_func as a single-arg one. So, for example,
__exit__(self, *args) and __exit__(self, exc_type, *args) both have the
legacy form, even though they could be invoked with one arg.

In summary, an exit_func will be invoke with a single arg if:

-   It is a PyMethod with argcount 2 (to count self) and no vararg, or
-   it is a PyCFunction with the METH_O flag.

Note that any performance cost of the introspection can be mitigated via
specialization <659>, so it won't be a problem if we need to make it
more sophisticated than this for some reason.

Backwards Compatibility

All context managers that previously worked will continue to work in the
same way because the interpreter will call them with three args whenever
they can accept three args. There may be context managers that
previously did not work because their exit_func expected one argument,
so the call to __exit__ would have caused a TypeError exception to be
raised, and now the call would succeed. This could theoretically change
the behaviour of existing code, but it is unlikely to be a problem in
practice.

The backwards compatibility concerns will show up in some cases when
libraries try to migrate their context managers from the multi-arg to
the single-arg signature. If __exit__ or __aexit__ is called by any code
other than the interpreter's eval loop, the introspection does not
automatically happen. For example, this will occur where a context
manager is subclassed and its __exit__ method is called directly from
the derived __exit__. Such context managers will need to migrate to the
single-arg version with their users, and may choose to offer a parallel
API rather than breaking the existing one. Alternatively, a superclass
can stay with the signature __exit__(self, *args), and support both one
and three args. Since most context managers do not use the value of the
arguments to __exit__, and simply allow the exception to propagate
onward, this is likely to be the common approach.

Security Implications

I am not aware of any.

How to Teach This

The language tutorial will present the single-arg version, and the
documentation for context managers will include a section on the legacy
signatures of __exit__ and __aexit__.

Reference Implementation

CPython PR #101995 implements the proposal of this PEP.

Rejected Ideas

Support __leave__(self, exc)

It was considered to support a method by a new name, such as __leave__,
with the new signature. This basically makes the programmer explicitly
declare which signature they are intending to use, and avoid the need
for introspection.

Different variations of this idea include different amounts of magic
that can help automate the equivalence between __leave__ and __exit__.
For example, Mark Shannon suggested that the type constructor would add
a default implementation for each of __exit__ and __leave__ whenever one
of them is defined on a class. This default implementation acts as a
trampoline that calls the user's function. This would make inheritance
work seamlessly, as well as the migration from __exit__ to __leave__ for
particular classes. The interpreter would just need to call __leave__,
and that would call __exit__ whenever necessary.

While this suggestion has several advantages over the current proposal,
it has two drawbacks. The first is that it adds a new dunder name to the
data model, and we would end up with two dunders that mean the same
thing, and only slightly differ in their signatures. The second is that
it would require the migration of every __exit__ to __leave__, while
with introspection it would not be necessary to change the many
__exit__(*arg) methods that do not access their args. While it is not as
simple as a grep for __exit__, it is possible to write an AST visitor
that detects __exit__ methods that can accept multiple arguments, and
which do access them.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 720 Title: Cross-compiling Python packages Author: Filipe Laíns
<EMAIL_ADDRESS>PEP-Delegate: Status: Draft Type: Informational
Content-Type: text/x-rst Created: 01-Jul-2023 Python-Version: 3.12

Abstract

This PEP attempts to document the status of cross-compilation of
downstream projects.

It should give an overview of the approaches currently used by
distributors (Linux distros, WASM environment providers, etc.) to
cross-compile downstream projects (3rd party extensions, etc.).

Motivation

We write this PEP to express the challenges in cross-compilation and act
as a supporting document in future improvement proposals.

Analysis

Introduction

There are a couple different approaches being used to tackle this, with
different levels of interaction required from the user, but they all
require a significant amount of effort. This is due to the lack of
standardized cross-compilation infrastructure on the Python packaging
ecosystem, which itself stems from the complexity of cross-builds,
making it a huge undertaking.

Upstream support

Some major projects like CPython, setuptools, etc. provide some support
to help with cross-compilation, but it's unofficial and at a best-effort
basis. For example, the sysconfig module allows overwriting the data
module name via the _PYTHON_SYSCONFIGDATA_NAME environment variable,
something that is required for cross-builds, and setuptools accepts
patches[1] to tweak/fix its logic to be compatible with popular
"environment faking" workflows[2].

The lack of first-party support in upstream projects leads to
cross-compilation being fragile and requiring a significant effort from
users, but at the same time, the lack of standardization makes it harder
for upstreams to improve support as there's no clarity on how this
feature should be provided.

Projects with decent cross-build support

It seems relevant to point out that there are a few modern Python
package build-backends with, at least, decent cross-compilation support,
those being scikit-build and meson-python. Both these projects integrate
external mature build-systems into Python packaging — CMake and Meson,
respectively — so cross-build support is inherited from them.

Downstream approaches

Cross-compilation approaches fall in a spectrum that goes from, by
design, requiring extensive user interaction to (ideally) almost none.
Usually, they'll be based on one of two main strategies, using a
cross-build environment, or faking the target environment.

Cross-build environment

This consists of running the Python interpreter normally and utilizing
the cross-build provided by the projects' build-system. However, as we
saw above, upstream support is lacking, so this approach only works for
a small-ish set of projects. When this fails, the usual strategy is to
patch the build-system code to build use the correct toolchain, system
details, etc.[3].

Since this approach often requires package-specific patching, it
requires a lot of user interaction.

Examples

python-for-android, kivy-ios, etc.

Faking the target environment

Aiming to drop the requirement for user input, a popular approach is
trying to fake the target environment. It generally consists of
monkeypatching the Python interpreter to get it to mimic the interpreter
on the target system, which constitutes of changing many of the sys
module attributes, the sysconfig data, etc. Using this strategy,
build-backends do not need to have any cross-build support, and should
just work without any code changes.

Unfortunately, though, it isn't possible to truly fake the target
environment. There are many reasons for this, one of the main ones being
that it breaks code that actually needs to introspect the running
interpreter. As a result, monkeypatching Python to look like target is
very tricky — to achieve the less amount of breakage, we can only patch
certain aspects of the interpreter. Consequently, build-backends may
need some code changes, but these are generally much smaller than the
previous approach. This is an inherent limitation of the technique,
meaning this strategy still requires some user interaction.

Nonetheless, this strategy still works out-of-the-box with significantly
more projects than the approach above, and requires much less effort in
these cases. It is successful in decreasing the amount of user
interaction needed, even though it doesn't succeed in being generic.

Examples

crossenv, conda-forge, etc.

Environment introspection

As explained above, most build system code is written with the
assumption that the target system is the same as where the build is
occurring, so introspection is usually used to guide the build.

In this section, we try to document most of the ways this is
accomplished. It should give a decent overview of of environment details
that are required by build systems.

+----------------------+----------------------+----------------------+
| Snippet              | Description          | Variance             |
+======================+======================+======================+
|     >>               | Extension (native    | This is              |
| > importlib.machiner | module) suffixes     | imp                  |
| y.EXTENSION_SUFFIXES | supported by this    | lementation-defined, |
|     [                | interpreter.         | but it usually       |
|                      |                      | differs based on the |
|      '.cpython-311-x |                      | implementation,      |
| 86_64-linux-gnu.so', |                      | system architecture, |
|        '.abi3.so',   |                      | build configuration, |
|        '.so',        |                      | Python language      |
|     ]                |                      | version, and         |
|                      |                      | implementation       |
|                      |                      | version — if one     |
|                      |                      | exists.              |
+----------------------+----------------------+----------------------+
|                      | Source (pure-Python) | This is              |
|  >>> importlib.machi | suffixes supported   | imp                  |
| nery.SOURCE_SUFFIXES | by this interpreter. | lementation-defined, |
|     ['.py']          |                      | but it usually       |
|                      |                      | doesn't differ       |
|                      |                      | (outside exotic      |
|                      |                      | implementations or   |
|                      |                      | systems).            |
+----------------------+----------------------+----------------------+
|                      | All module file      | This is              |
|   >>> importlib.mach | suffixes supported   | imp                  |
| inery.all_suffixes() | by this interpreter. | lementation-defined, |
|     [                | It should be the     | but it usually       |
|        '.py',        | union of all         | differs based on the |
|        '.pyc',       | importlib.           | implementation,      |
|                      | machinery.*_SUFFIXES | system architecture, |
|      '.cpython-311-x | attributes.          | build configuration, |
| 86_64-linux-gnu.so', |                      | Python language      |
|        '.abi3.so',   |                      | version, and         |
|        '.so',        |                      | implementation       |
|     ]                |                      | version — if one     |
|                      |                      | exists. See the      |
|                      |                      | entries above for    |
|                      |                      | more information.    |
+----------------------+----------------------+----------------------+
|     >>> sys.abiflags | ABI flags, as        | Differs based on the |
|     ''               | specified in PEP     | build configuration. |
|                      | 3149.                |                      |
+----------------------+----------------------+----------------------+
|                      | C API version.       | Differs based on the |
|  >>> sys.api_version |                      | Python installation. |
|     1013             |                      |                      |
+----------------------+----------------------+----------------------+
|                      | Prefix of the        | Differs based on the |
|  >>> sys.base_prefix | installation-wide    | platform, and        |
|     /usr             | directories where    | installation.        |
|                      | platform independent |                      |
|                      | files are installed. |                      |
+----------------------+----------------------+----------------------+
|     >>>              | Prefix of the        | Differs based on the |
| sys.base_exec_prefix | installation-wide    | platform, and        |
|     /usr             | directories where    | installation.        |
|                      | platform dependent   |                      |
|                      | files are installed. |                      |
+----------------------+----------------------+----------------------+
|                      | Native byte order.   | Differs based on the |
|    >>> sys.byteorder |                      | platform.            |
|     'little'         |                      |                      |
+----------------------+----------------------+----------------------+
|     >>> sys.         | Names of all modules | Differs based on the |
| builtin_module_names | that are compiled    | platform, system     |
|     ('_abc', '_a     | into the Python      | architecture, and    |
| st', '_codecs', ...) | interpreter.         | build configuration. |
+----------------------+----------------------+----------------------+
|                      | Prefix of the        | Differs based on the |
|  >>> sys.exec_prefix | site-specific        | platform,            |
|     /usr             | directories where    | installation, and    |
|                      | platform independent | environment.         |
|                      | files are installed. |                      |
|                      | Because it concerns  |                      |
|                      | the site-specific    |                      |
|                      | directories, in      |                      |
|                      | standard virtual     |                      |
|                      | environment          |                      |
|                      | implementation, it   |                      |
|                      | will be a            |                      |
|                      | virtual-             |                      |
|                      | environment-specific |                      |
|                      | path.                |                      |
+----------------------+----------------------+----------------------+
|                      | Path of the Python   | Differs based on the |
|   >>> sys.executable | interpreter being    | installation.        |
|                      | used.                |                      |
|    '/usr/bin/python' |                      |                      |
+----------------------+----------------------+----------------------+
|     >                | Whether the Python   | Differs based on the |
| >> with open(sys.exe | interpreter is an    | installation.        |
| cutable, 'rb') as f: | ELF file, and the    |                      |
|     ...              | ELF header. This     |                      |
|   header = f.read(4) | approach is          |                      |
|     ..               | something used to    |                      |
| .   if is_elf := (he | identify the target  |                      |
| ader == b'\x7fELF'): | architecture of the  |                      |
|     ...     elf_cl   | installation         |                      |
| ass = int(f.read(1)) | (example).           |                      |
|     ...              |                      |                      |
|     size = {1: 52, 2 |                      |                      |
| : 64}.get(elf_class) |                      |                      |
|                      |                      |                      |
|    ...     elf_heade |                      |                      |
| r = f.read(size - 5) |                      |                      |
+----------------------+----------------------+----------------------+
|                      | Low level            | Differs based on the |
|   >>> sys.float_info | information about    | architecture, and    |
|     sys.float_info(  | the float type, as   | platform.            |
|        max=1.79      | defined by float.h.  |                      |
|<PHONE_NUMBER>3157e+308, |                      |                      |
|        max_exp=1024, |                      |                      |
|                      |                      |                      |
|      max_10_exp=308, |                      |                      |
|        min=2.22      |                      |                      |
|<PHONE_NUMBER>2014e-308, |                      |                      |
|                      |                      |                      |
|       min_exp=-1021, |                      |                      |
|                      |                      |                      |
|     min_10_exp=-307, |                      |                      |
|        dig=15,       |                      |                      |
|        mant_dig=53,  |                      |                      |
|        epsilon=2.    |                      |                      |
|<PHONE_NUMBER>50313e-16, |                      |                      |
|        radix=2,      |                      |                      |
|        rounds=1,     |                      |                      |
|     )                |                      |                      |
+----------------------+----------------------+----------------------+
|     >>> sys.         | Integer representing | Differs based on the |
| getandroidapilevel() | the Android API      | platform.            |
|     21               | level.               |                      |
+----------------------+----------------------+----------------------+
|     >>> sys          | Windows version of   | Differs based on the |
| .getwindowsversion() | the system.          | platform.            |
|     sy               |                      |                      |
| s.getwindowsversion( |                      |                      |
|        major=10,     |                      |                      |
|        minor=0,      |                      |                      |
|        build=19045,  |                      |                      |
|        platform=2,   |                      |                      |
|                      |                      |                      |
|     service_pack='', |                      |                      |
|     )                |                      |                      |
+----------------------+----------------------+----------------------+
|                      | Python version       | Differs based on the |
|   >>> sys.hexversion | encoded as an        | Python language      |
|     0x30b03f0        | integer.             | version.             |
+----------------------+----------------------+----------------------+
|     >>               | Interpreter          | Differs based on the |
| > sys.implementation | implementation       | interpreter          |
|     namespace(       | details.             | implementation,      |
|                      |                      | Python language      |
|      name='cpython', |                      | version, and         |
|        cach          |                      | implementation       |
| e_tag='cpython-311', |                      | version — if one     |
|        versi         |                      | exists. It may also  |
| on=sys.version_info( |                      | include              |
|           major=3,   |                      | ar                   |
|           minor=11,  |                      | chitecture-dependent |
|           micro=3,   |                      | information, so it   |
|           r          |                      | may also differ      |
| eleaselevel='final', |                      | based on the system  |
|           serial=0,  |                      | architecture.        |
|        ),            |                      |                      |
|        h             |                      |                      |
| exversion=0x30b03f0, |                      |                      |
|        _multiarch    |                      |                      |
| ='x86_64-linux-gnu', |                      |                      |
|     )                |                      |                      |
+----------------------+----------------------+----------------------+
|     >>> sys.int_info | Low level            | Differs based on the |
|     sys.int_info(    | information about    | architecture,        |
|                      | Python's internal    | platform,            |
|   bits_per_digit=30, | integer              | implementation,      |
|                      | representation.      | build, and runtime   |
|      sizeof_digit=4, |                      | flags.               |
|        default_      |                      |                      |
| max_str_digits=4300, |                      |                      |
|        str_digits_   |                      |                      |
| check_threshold=640, |                      |                      |
|     )                |                      |                      |
+----------------------+----------------------+----------------------+
|     >>> sys.maxsize  | Maximum value a      | Differs based on the |
|                      | variable of type     | architecture,        |
|   0x7fffffffffffffff | Py_ssize_t can take. | platform, and        |
|                      |                      | implementation.      |
+----------------------+----------------------+----------------------+
|                      | Value of the largest | Differs based on the |
|   >>> sys.maxunicode | Unicode code point.  | implementation, and  |
|     0x10ffff         |                      | on Python versions   |
|                      |                      | older than 3.3, the  |
|                      |                      | build.               |
+----------------------+----------------------+----------------------+
|     >>> sys.platform | Platform identifier. | Differs based on the |
|     linux            |                      | platform.            |
+----------------------+----------------------+----------------------+
|     >>> sys.prefix   | Prefix of the        | Differs based on the |
|     /usr             | site-specific        | platform,            |
|                      | directories where    | installation, and    |
|                      | platform dependent   | environment.         |
|                      | files are installed. |                      |
|                      | Because it concerns  |                      |
|                      | the site-specific    |                      |
|                      | directories, in      |                      |
|                      | standard virtual     |                      |
|                      | environment          |                      |
|                      | implementation, it   |                      |
|                      | will be a            |                      |
|                      | virtual-             |                      |
|                      | environment-specific |                      |
|                      | path.                |                      |
+----------------------+----------------------+----------------------+
|                      | Platform-specific    | Differs based on the |
|   >>> sys.platlibdir | library directory.   | platform, and        |
|     lib              |                      | vendor.              |
+----------------------+----------------------+----------------------+
|                      | Python language      | Differs if the       |
| >>> sys.version_info | version implemented  | target Python        |
|                      | by the interpreter.  | version is not the   |
|    sys.version_info( |                      | same[8].             |
|        major=3,      |                      |                      |
|        minor=11,     |                      |                      |
|        micro=3,      |                      |                      |
|        r             |                      |                      |
| eleaselevel='final', |                      |                      |
|        serial=0,     |                      |                      |
|     )                |                      |                      |
+----------------------+----------------------+----------------------+
|                      | Information about    | Differs based on the |
|  >>> sys.thread_info | the thread           | platform, and        |
|     sys.thread_info( | implementation.      | implementation.      |
|                      |                      |                      |
|      name='pthread', |                      |                      |
|                      |                      |                      |
|    lock='semaphore', |                      |                      |
|                      |                      |                      |
| version='NPTL 2.37', |                      |                      |
|     )                |                      |                      |
+----------------------+----------------------+----------------------+
|     >>> sys.winver   | Version number used  | Differs based on the |
|     3.8-32           | to form Windows      | platform, and        |
|                      | registry keys.       | implementation.      |
+----------------------+----------------------+----------------------+
|     >>> sysconf      | Python distribution  | This is              |
| ig.get_config_vars() | configuration        | imp                  |
|     { ... }          | variables. It        | lementation-defined, |
|     >>> sysconfig    | includes a set of    | but it usually       |
| .get_config_var(...) | variables[9] — like  | differs between      |
|     ...              | prefix, exec_prefix, | non-identical        |
|                      | etc. — based on the  | builds. Please refer |
|                      | running context[10], | to the sysconfig     |
|                      | and may include some | configuration        |
|                      | extra variables      | variables table for  |
|                      | based on the Python  | a overview of the    |
|                      | implementation and   | different            |
|                      | system.              | configuration        |
|                      |                      | variable that are    |
|                      | In CPython and most  | usually present.     |
|                      | other                |                      |
|                      | implementations that |                      |
|                      | use the same         |                      |
|                      | build-system, the    |                      |
|                      | "extra" variables    |                      |
|                      | mention above are:   |                      |
|                      | on POSIX, all        |                      |
|                      | variables from the   |                      |
|                      | Makefile used to     |                      |
|                      | build the            |                      |
|                      | interpreter, and on  |                      |
|                      | Windows, it usually  |                      |
|                      | only includes a      |                      |
|                      | small subset of the  |                      |
|                      | those[11] — like     |                      |
|                      | EXT_SUFFIX, BINDIR,  |                      |
|                      | etc.                 |                      |
+----------------------+----------------------+----------------------+

CPython (and similar)

  --------------------------------------------------------------------------------------------------------
  Name                 Example Value                      Description                Variance
  -------------------- ---------------------------------- -------------------------- ---------------------
  SOABI                cpython-311-x86_64-linux-gnu       ABI string — defined by    Differs based on the
                                                          PEP 3149.                  implementation,
                                                                                system architecture,
                                                                                Python language
                                                                                version, and
                                                                                implementation
                                                                                version — if one
                                                                                exists.

  SHLIB_SUFFIX         .so                                Shared library suffix.     Differs based on the
                                                                                platform.

  EXT_SUFFIX           .cpython-311-x86_64-linux-gnu.so   Interpreter-specific       Differs based on the
                                                          Python extension (native   implementation,
                                                          module) suffix — generally system architecture,
                                                          defined as                 Python language
                                                          .{SOABI}.{SHLIB_SUFFIX}.   version, and
                                                                                implementation
                                                                                version — if one
                                                                                exists.

  LDLIBRARY            libpython3.11.so                   Shared libpython library   Differs based on the
                                                          name — if available. If    implementation,
                                                          unavailable[12], the       system architecture,
                                                          variable will be empty, if build configuration,
                                                          available, the library     Python language
                                                          should be located in       version, and
                                                          LIBDIR.                    implementation
                                                                                version — if one
                                                                                exists.

  PY3LIBRARY           libpython3.so                      Shared Python 3 only       Differs based on the
                                                          (major version bound       implementation,
                                                          only)[13] libpython        system architecture,
                                                          library name — if          build configuration,
                                                          available. If              Python language
                                                          unavailable[14], the       version, and
                                                          variable will be empty, if implementation
                                                          available, the library     version — if one
                                                          should be located in       exists.
                                                          LIBDIR.

  LIBRARY              libpython3.11.a                    Static libpython library   Differs based on the
                                                          name — if available. If    implementation,
                                                          unavailable[15], the       system architecture,
                                                          variable will be empty, if build configuration,
                                                          available, the library     Python language
                                                          should be located in       version, and
                                                          LIBDIR.                    implementation
                                                                                version — if one
                                                                                exists.

  Py_DEBUG             0                                  Whether this is a debug    Differs based on the
                                                          build.                     build configuration.

  WITH_PYMALLOC        1                                  Whether this build has     Differs based on the
                                                          pymalloc support.          build configuration.

  Py_TRACE_REFS        0                                  Whether reference tracing  Differs based on the
                                                          (debug build only) is      build configuration.
                                                          enabled.

  Py_UNICODE_SIZE                                         Size of the Py_UNICODE     Differs based on the
                                                          object, in bytes. This     build configuration.
                                                          variable is only present
                                                          in CPython versions older
                                                          than 3.3, and was commonly
                                                          used to detect if the
                                                          build uses UCS2 or UCS4
                                                          for unicode objects —
                                                          before PEP 393.

  Py_ENABLE_SHARED     1                                  Whether a shared libpython Differs based on the
                                                          is available.              build configuration.

  PY_ENABLE_SHARED     1                                  Whether a shared libpython Differs based on the
                                                          is available.              build configuration.

  CC                   gcc                                The C compiler used to     Differs based on the
                                                          build the Python           build configuration.
                                                          distribution.

  CXX                  g++                                The C compiler used to     Differs based on the
                                                          build the Python           build configuration.
                                                          distribution.

  CFLAGS               -DNDEBUG -g -fwrapv ...            The C compiler flags used  Differs based on the
                                                          to build the Python        build configuration.
                                                          distribution.

  py_version           3.11.3                             Full form of the Python    Differs based on the
                                                          version.                   Python language
                                                                                version.

  py_version_short     3.11                               Custom form of the Python  Differs based on the
                                                          version, containing only   Python language
                                                          the major and minor        version.
                                                          numbers.

  py_version_nodot     311                                Custom form of the Python  Differs based on the
                                                          version, containing only   Python language
                                                          the major and minor        version.
                                                          numbers, and no dots.

  prefix               /usr                               Same as sys.prefix, please Differs based on the
                                                          refer to the entry in      platform,
                                                          table above.               installation, and
                                                                                environment.

  base                 /usr                               Same as sys.prefix, please Differs based on the
                                                          refer to the entry in      platform,
                                                          table above.               installation, and
                                                                                environment.

  exec_prefix          /usr                               Same as sys.exec_prefix,   Differs based on the
                                                          please refer to the entry  platform,
                                                          in table above.            installation, and
                                                                                environment.

  platbase             /usr                               Same as sys.exec_prefix,   Differs based on the
                                                          please refer to the entry  platform,
                                                          in table above.            installation, and
                                                                                environment.

  installed_base       /usr                               Same as sys.base_prefix,   Differs based on the
                                                          please refer to the entry  platform, and
                                                          in table above.            installation.

  installed_platbase   /usr                               Same as                    Differs based on the
                                                          sys.base_exec_prefix,      platform, and
                                                          please refer to the entry  installation.
                                                          in table above.

  platlibdir           lib                                Same as sys.platlibdir,    Differs based on the
                                                          please refer to the entry  platform, and vendor.
                                                          in table above.

  SIZEOF_*             4                                  Size of a certain C type   Differs based on the
                                                          (double, float, etc.).     system architecture,
                                                                                and build details.
  --------------------------------------------------------------------------------------------------------

  : sysconfig configuration variables

Relevant Information

There are some bits of information required by build systems — eg.
platform particularities — scattered across many places, and it often is
difficult to identify code with assumptions based on them. In this
section, we try to document the most relevant cases.

When should extensions be linked against libpython?

Short answer

    Yes, on Windows. No on POSIX platforms, except Android, Cygwin, and
    other Windows-based POSIX-like platforms.

When building extensions for dynamic loading, depending on the target
platform, they may need to be linked against libpython.

On Windows, extensions need to link against libpython, because all
symbols must be resolvable at link time. POSIX-like platforms based on
Windows — like Cygwin, MinGW, or MSYS — will also require linking
against libpython.

On most POSIX platforms, it is not necessary to link against libpython,
as the symbols will already be available in the due to the interpreter —
or, when embedding, the executable/library in question — already linking
to libpython. Not linking an extension module against libpython will
allow it to be loaded by static Python builds, so when possible, it is
desirable to do so (see GH-65735).

This might not be the case on all POSIX platforms, so make sure you
check. One example is Android, where only the main executable and
LD_PRELOAD entries are considered to be RTLD_GLOBAL (meaning
dependencies are RTLD_LOCAL) [16], which causes the libpython symbols be
unavailable when loading the extension.

What are prefix, exec_prefix, base_prefix, and base_exec_prefix?

These are sys attributes set in the Python initialization that describe
the running environment. They refer to the prefix of directories where
installation/environment files are installed, according to the table
below.

  Name               Target files                             Environment Scope
  ------------------ ---------------------------------------- -------------------
  prefix             platform independent (eg. pure Python)   site-specific
  exec_prefix        platform dependent (eg. native code)     site-specific
  base_prefix        platform independent (eg. pure Python)   installation-wide
  base_exec_prefix   platform dependent (eg. native code)     installation-wide

Because the site-specific prefixes will be different inside virtual
environments, checking sys.prexix != sys.base_prefix is commonly used to
check if we are in a virtual environment.

Case studies

crossenv

Description

    Virtual Environments for Cross-Compiling Python Extension Modules.

URL

    https://github.com/benfogle/crossenv

crossenv is a tool to create a virtual environment with a monkeypatched
Python installation that tries to emulate the target machine in certain
scenarios. More about this approach can be found in the Faking the
target environment section.

conda-forge

Description

    A community-led collection of recipes, build infrastructure and
    distributions for the conda package manager.

URL

    https://conda-forge.org/

XXX: Jaime will write a quick summary once the PEP draft is public.

XXX Uses a modified crossenv.

Yocto Project

Description

    The Yocto Project is an open source collaboration project that helps
    developers create custom Linux-based systems regardless of the
    hardware architecture.

URL

    https://www.yoctoproject.org/

XXX: Sent email to the mailing list.

TODO

Buildroot

Description

    Buildroot is a simple, efficient and easy-to-use tool to generate
    embedded Linux systems through cross-compilation.

URL

    https://buildroot.org/

TODO

Pyodide

Description

    Pyodide is a Python distribution for the browser and Node.js based
    on WebAssembly.

URL

    https://pyodide.org/en/stable/

XXX: Hood should review/expand this section.

Pyodide is a provides a Python distribution compiled to WebAssembly
using the Emscripten toolchain.

It patches several aspects of the CPython installation and some external
components. A custom package manager — micropip — supporting both Pure
and wasm32/Emscripten wheels, is also provided as a part of the
distribution. On top of this, a repo with a selected set of 3rd party
packages is also provided and enabled by default.

Beeware

Description

    BeeWare allows you to write your app in Python and release it on
    multiple platforms.

URL

    https://beeware.org/

TODO

python-for-android

Description

    Turn your Python application into an Android APK.

URL

    https://github.com/kivy/python-for-android

resource https://github.com/Android-for-Python/Android-for-Python-Users

python-for-android is a tool to package Python apps on Android. It
creates a Python distribution with your app and its dependencies.

Pure-Python dependencies are handled automatically and in a generic way,
but native dependencies need recipes. A set of recipes for popular
dependencies is provided, but users need to provide their own recipes
for any other native dependencies.

kivy-ios

Description

    Toolchain for compiling Python / Kivy / other libraries for iOS.

URL

    https://github.com/kivy/kivy-ios

kivy-ios is a tool to package Python apps on iOS. It provides a
toolchain to build a Python distribution with your app and its
dependencies, as well as a CLI to create and manage Xcode projects that
integrate with the toolchain.

It uses the same approach as python-for-android (also maintained by the
Kivy project) for app dependencies — pure-Python dependencies are
handled automatically, but native dependencies need recipes, and the
project provides recipes for popular dependencies.

AidLearning

Description

    AI, Android, Linux, ARM: AI application development platform based
    on Android+Linux integrated ecology.

URL

    https://github.com/aidlearning/AidLearning-FrameWork

TODO

QPython

Description

    QPython is the Python engine for android.

URL

    https://github.com/qpython-android/qpython

TODO

pyqtdeploy

Description

    pyqtdeploy is a tool for deploying PyQt applications.

URL

    https://www.riverbankcomputing.com/software/pyqtdeploy/

contact
https://www.riverbankcomputing.com/pipermail/pyqt/2023-May/thread.html
contacted Phil, the maintainer

TODO

Chaquopy

Description

    Chaquopy provides everything you need to include Python components
    in an Android app.

URL

    https://chaquo.com/chaquopy/

TODO

EDK II

Description

    EDK II is a modern, feature-rich, cross-platform firmware
    development environment for the UEFI and PI specifications.

URL

    https://github.com/tianocore/edk2-libc/tree/master/AppPkg/Applications/Python

TODO

ActivePython

Description

    Commercial-grade, quality-assured Python distribution focusing on
    easy installation and cross-platform compatibility on Windows,
    Linux, Mac OS X, Solaris, HP-UX and AIX.

URL

    https://www.activestate.com/products/python/

TODO

Termux

Description

    Termux is an Android terminal emulator and Linux environment app
    that works directly with no rooting or setup required.

URL

    https://termux.dev/en/

TODO

[1] At the time of writing (Jun 2023), setuptools' compiler interface
code, the component that most of affects cross-compilation, is developed
on the pypa/distutils repository, which gets periodically synced to the
setuptools repository.

[2] We specifically mention popular workflows, because this is not
standardized. Though, many of the most popular implementations
(crossenv, conda-forge's build system, etc.) work similarly, and this is
what we are referring to here. For clarity, the implementations we are
referring to here could be described as crossenv-style.

[3] The scope of the build-system patching varies between users and
usually depends on the their goal — some (eg. Linux distributions) may
patch the build-system to support cross-builds, while others might
hardcode compiler paths and system information in the build-system, to
simply make the build work.

[4] Ideally, you want to perform cross-builds with the same Python
version and implementation, however, this is often not the case. It
should not be very problematic as long as the major and minor versions
don't change.

[5] The set of config variables that will always be present mostly
consists of variables needed to calculate the installation scheme paths.

[6] The context we refer here consists of the "path initialization",
which is a process that happens in the interpreter startup and is
responsible for figuring out which environment it is being run — eg.
global environment, virtual environment, etc. — and setting sys.prefix
and other attributes accordingly.

[7] This is because Windows builds may not use the Makefile, and instead
use the Visual Studio build system. A subset of the most relevant
Makefile variables is provided to make user code that uses them simpler.

[8] Ideally, you want to perform cross-builds with the same Python
version and implementation, however, this is often not the case. It
should not be very problematic as long as the major and minor versions
don't change.

[9] The set of config variables that will always be present mostly
consists of variables needed to calculate the installation scheme paths.

[10] The context we refer here consists of the "path initialization",
which is a process that happens in the interpreter startup and is
responsible for figuring out which environment it is being run — eg.
global environment, virtual environment, etc. — and setting sys.prefix
and other attributes accordingly.

[11] This is because Windows builds may not use the Makefile, and
instead use the Visual Studio build system. A subset of the most
relevant Makefile variables is provided to make user code that uses them
simpler.

[12] Due to Python bring compiled without shared or static libpython
support, respectively.

[13] This is the libpython library that users of the stable ABI should
link against, if they need to link against libpython.

[14] Due to Python bring compiled without shared or static libpython
support, respectively.

[15] Due to Python bring compiled without shared or static libpython
support, respectively.

[16] Refer to dlopen's man page for more information.