text:
PEP: 590 Title: Vectorcall: a fast calling protocol for CPython Author:
Mark Shannon<EMAIL_ADDRESS>Jeroen Demeyer<EMAIL_ADDRESS>BDFL-Delegate: Petr Viktorin<EMAIL_ADDRESS>Status: Accepted Type:
Standards Track Created: 29-Mar-2019 Python-Version: 3.8 Post-History:

python:vectorcall

Abstract

This PEP introduces a new C API to optimize calls of objects. It
introduces a new "vectorcall" protocol and calling convention. This is
based on the "fastcall" convention, which is already used internally by
CPython. The new features can be used by any user-defined extension
class.

Most of the new API is private in CPython 3.8. The plan is to finalize
semantics and make it public in Python 3.9.

NOTE: This PEP deals only with the Python/C API, it does not affect the
Python language or standard library.

Motivation

The choice of a calling convention impacts the performance and
flexibility of code on either side of the call. Often there is tension
between performance and flexibility.

The current tp_call[1] calling convention is sufficiently flexible to
cover all cases, but its performance is poor. The poor performance is
largely a result of having to create intermediate tuples, and possibly
intermediate dicts, during the call. This is mitigated in CPython by
including special-case code to speed up calls to Python and builtin
functions. Unfortunately, this means that other callables such as
classes and third party extension objects are called using the slower,
more general tp_call calling convention.

This PEP proposes that the calling convention used internally for Python
and builtin functions is generalized and published so that all calls can
benefit from better performance. The new proposed calling convention is
not fully general, but covers the large majority of calls. It is
designed to remove the overhead of temporary object creation and
multiple indirections.

Another source of inefficiency in the tp_call convention is that it has
one function pointer per class, rather than per object. This is
inefficient for calls to classes as several intermediate objects need to
be created. For a class cls, at least one intermediate object is created
for each call in the sequence type.__call__, cls.__new__, cls.__init__.

This PEP proposes an interface for use by extension modules. Such
interfaces cannot effectively be tested, or designed, without having the
consumers in the loop. For that reason, we provide private
(underscore-prefixed) names. The API may change (based on consumer
feedback) in Python 3.9, where we expect it to be finalized, and the
underscores removed.

Specification

The function pointer type

Calls are made through a function pointer taking the following
parameters:

-   PyObject *callable: The called object
-   PyObject *const *args: A vector of arguments
-   size_t nargs: The number of arguments plus the optional flag
    PY_VECTORCALL_ARGUMENTS_OFFSET (see below)
-   PyObject *kwnames: Either NULL or a tuple with the names of the
    keyword arguments

This is implemented by the function pointer type:
typedef PyObject *(*vectorcallfunc)(PyObject *callable, PyObject *const *args, size_t nargs, PyObject *kwnames);

Changes to the PyTypeObject struct

The unused slot printfunc tp_print is replaced with
tp_vectorcall_offset. It has the type Py_ssize_t. A new tp_flags flag is
added, _Py_TPFLAGS_HAVE_VECTORCALL, which must be set for any class that
uses the vectorcall protocol.

If _Py_TPFLAGS_HAVE_VECTORCALL is set, then tp_vectorcall_offset must be
a positive integer. It is the offset into the object of the vectorcall
function pointer of type vectorcallfunc. This pointer may be NULL, in
which case the behavior is the same as if _Py_TPFLAGS_HAVE_VECTORCALL
was not set.

The tp_print slot is reused as the tp_vectorcall_offset slot to make it
easier for external projects to backport the vectorcall protocol to
earlier Python versions. In particular, the Cython project has shown
interest in doing that (see
https://mail.python.org/pipermail/python-dev/2018-June/153927.html).

Descriptor behavior

One additional type flag is specified: Py_TPFLAGS_METHOD_DESCRIPTOR.

Py_TPFLAGS_METHOD_DESCRIPTOR should be set if the callable uses the
descriptor protocol to create a bound method-like object. This is used
by the interpreter to avoid creating temporary objects when calling
methods (see _PyObject_GetMethod and the LOAD_METHOD/CALL_METHOD
opcodes).

Concretely, if Py_TPFLAGS_METHOD_DESCRIPTOR is set for type(func), then:

-   func.__get__(obj, cls)(*args, **kwds) (with obj not None) must be
    equivalent to func(obj, *args, **kwds).
-   func.__get__(None, cls)(*args, **kwds) must be equivalent to
    func(*args, **kwds).

There are no restrictions on the object func.__get__(obj, cls). The
latter is not required to implement the vectorcall protocol.

The call

The call takes the form
((vectorcallfunc)(((char *)o)+offset))(o, args, n, kwnames) where offset
is Py_TYPE(o)->tp_vectorcall_offset. The caller is responsible for
creating the kwnames tuple and ensuring that there are no duplicates in
it.

n is the number of positional arguments plus possibly the
PY_VECTORCALL_ARGUMENTS_OFFSET flag.

PY_VECTORCALL_ARGUMENTS_OFFSET

The flag PY_VECTORCALL_ARGUMENTS_OFFSET should be added to n if the
callee is allowed to temporarily change args[-1]. In other words, this
can be used if args points to argument 1 in the allocated vector. The
callee must restore the value of args[-1] before returning.

Whenever they can do so cheaply (without allocation), callers are
encouraged to use PY_VECTORCALL_ARGUMENTS_OFFSET. Doing so will allow
callables such as bound methods to make their onward calls cheaply. The
bytecode interpreter already allocates space on the stack for the
callable, so it can use this trick at no additional cost.

See[2] for an example of how PY_VECTORCALL_ARGUMENTS_OFFSET is used by a
callee to avoid allocation.

For getting the actual number of arguments from the parameter n, the
macro PyVectorcall_NARGS(n) must be used. This allows for future changes
or extensions.

New C API and changes to CPython

The following functions or macros are added to the C API:

-   PyObject *_PyObject_Vectorcall(PyObject *obj, PyObject *const *args, size_t nargs, PyObject *keywords):
    Calls obj with the given arguments. Note that nargs may include the
    flag PY_VECTORCALL_ARGUMENTS_OFFSET. The actual number of positional
    arguments is given by PyVectorcall_NARGS(nargs). The argument
    keywords is a tuple of keyword names or NULL. An empty tuple has the
    same effect as passing NULL. This uses either the vectorcall
    protocol or tp_call internally; if neither is supported, an
    exception is raised.
-   PyObject *PyVectorcall_Call(PyObject *obj, PyObject *tuple, PyObject *dict):
    Call the object (which must support vectorcall) with the old *args
    and **kwargs calling convention. This is mostly meant to put in the
    tp_call slot.
-   Py_ssize_t PyVectorcall_NARGS(size_t nargs): Given a vectorcall
    nargs argument, return the actual number of arguments. Currently
    equivalent to nargs & ~PY_VECTORCALL_ARGUMENTS_OFFSET.

Subclassing

Extension types inherit the type flag _Py_TPFLAGS_HAVE_VECTORCALL and
the value tp_vectorcall_offset from the base class, provided that they
implement tp_call the same way as the base class. Additionally, the flag
Py_TPFLAGS_METHOD_DESCRIPTOR is inherited if tp_descr_get is implemented
the same way as the base class.

Heap types never inherit the vectorcall protocol because that would not
be safe (heap types can be changed dynamically). This restriction may be
lifted in the future, but that would require special-casing __call__ in
type.__setattribute__.

Finalizing the API

The underscore in the names _PyObject_Vectorcall and
_Py_TPFLAGS_HAVE_VECTORCALL indicates that this API may change in minor
Python versions. When finalized (which is planned for Python 3.9), they
will be renamed to PyObject_Vectorcall and Py_TPFLAGS_HAVE_VECTORCALL.
The old underscore-prefixed names will remain available as aliases.

The new API will be documented as normal, but will warn of the above.

Semantics for the other names introduced in this PEP
(PyVectorcall_NARGS, PyVectorcall_Call, Py_TPFLAGS_METHOD_DESCRIPTOR,
PY_VECTORCALL_ARGUMENTS_OFFSET) are final.

Internal CPython changes

Changes to existing classes

The function, builtin_function_or_method, method_descriptor, method,
wrapper_descriptor, method-wrapper classes will use the vectorcall
protocol (not all of these will be changed in the initial
implementation).

For builtin_function_or_method and method_descriptor (which use the
PyMethodDef data structure), one could implement a specific vectorcall
wrapper for every existing calling convention. Whether or not it is
worth doing that remains to be seen.

Using the vectorcall protocol for classes

For a class cls, creating a new instance using cls(xxx) requires
multiple calls. At least one intermediate object is created for each
call in the sequence type.__call__, cls.__new__, cls.__init__. So it
makes a lot of sense to use vectorcall for calling classes. This really
means implementing the vectorcall protocol for type. Some of the most
commonly used classes will use this protocol, probably range, list, str,
and type.

The PyMethodDef protocol and Argument Clinic

Argument Clinic[3] automatically generates wrapper functions around
lower-level callables, providing safe unboxing of primitive types and
other safety checks. Argument Clinic could be extended to generate
wrapper objects conforming to the new vectorcall protocol. This will
allow execution to flow from the caller to the Argument Clinic generated
wrapper and thence to the hand-written code with only a single
indirection.

Third-party extension classes using vectorcall

To enable call performance on a par with Python functions and built-in
functions, third-party callables should include a vectorcallfunc
function pointer, set tp_vectorcall_offset to the correct value and add
the _Py_TPFLAGS_HAVE_VECTORCALL flag. Any class that does this must
implement the tp_call function and make sure its behaviour is consistent
with the vectorcallfunc function. Setting tp_call to PyVectorcall_Call
is sufficient.

Performance implications of these changes

This PEP should not have much impact on the performance of existing code
(neither in the positive nor the negative sense). It is mainly meant to
allow efficient new code to be written, not to make existing code
faster.

Nevertheless, this PEP optimizes for METH_FASTCALL functions.
Performance of functions using METH_VARARGS will become slightly worse.

Stable ABI

Nothing from this PEP is added to the stable ABI (PEP 384).

Alternative Suggestions

bpo-29259

PEP 590 is close to what was proposed in bpo-29259[4]. The main
difference is that this PEP stores the function pointer in the instance
rather than in the class. This makes more sense for implementing
functions in C, where every instance corresponds to a different C
function. It also allows optimizing type.__call__, which is not possible
with bpo-29259.

PEP 576 and PEP 580

Both PEP 576 and PEP 580 are designed to enable 3rd party objects to be
both expressive and performant (on a par with CPython objects). The
purpose of this PEP is provide a uniform way to call objects in the
CPython ecosystem that is both expressive and as performant as possible.

This PEP is broader in scope than PEP 576 and uses variable rather than
fixed offset function-pointers. The underlying calling convention is
similar. Because PEP 576 only allows a fixed offset for the function
pointer, it would not allow the improvements to any objects with
constraints on their layout.

PEP 580 proposes a major change to the PyMethodDef protocol used to
define builtin functions. This PEP provides a more general and simpler
mechanism in the form of a new calling convention. This PEP also extends
the PyMethodDef protocol, but merely to formalise existing conventions.

Other rejected approaches

A longer, 6 argument, form combining both the vector and optional tuple
and dictionary arguments was considered. However, it was found that the
code to convert between it and the old tp_call form was overly
cumbersome and inefficient. Also, since only 4 arguments are passed in
registers on x64 Windows, the two extra arguments would have
non-negligible costs.

Removing any special cases and making all calls use the tp_call form was
also considered. However, unless a much more efficient way was found to
create and destroy tuples, and to a lesser extent dictionaries, then it
would be too slow.

Acknowledgements

Victor Stinner for developing the original "fastcall" calling convention
internally to CPython. This PEP codifies and extends his work.

References

Reference implementation

A minimal implementation can be found at
https://github.com/markshannon/cpython/tree/vectorcall-minimal

Copyright

This document has been placed in the public domain.

[1] tp_call/PyObject_Call calling convention
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_call

[2] Using PY_VECTORCALL_ARGUMENTS_OFFSET in callee
https://github.com/markshannon/cpython/blob/815cc1a30d85cdf2e3d77d21224db7055a1f07cb/Objects/classobject.c#L53

[3] Argument Clinic https://docs.python.org/3/howto/clinic.html

[4] Add tp_fastcall to PyTypeObject: support FASTCALL calling convention
for all callable objects, https://bugs.python.org/issue29259

authors:
Jeroen Demeyer
Mark Shannon


text:
PEP: 206 Title: Python Advanced Library Author: A.M. Kuchling
<EMAIL_ADDRESS>Status: Withdrawn Type: Informational Created: 14-Jul-2000
Post-History:

Introduction

This PEP describes the Python Advanced Library, a collection of
high-quality and frequently-used third party extension modules.

Batteries Included Philosophy

The Python source distribution has long maintained the philosophy of
"batteries included" -- having a rich and versatile standard library
which is immediately available, without making the user download
separate packages. This gives the Python language a head start in many
projects.

However, the standard library modules aren't always the best choices for
a job. Some library modules were quick hacks (e.g. calendar, commands),
some were designed poorly and are now near-impossible to fix (cgi), and
some have been rendered obsolete by other, more complete modules
(binascii offers the same features as the binhex, uu, base64 modules).
This PEP describes a list of third-party modules that make Python more
competitive for various application domains, forming the Python Advanced
Library.

The deliverable is a set of scripts that will retrieve, build, and
install the packages for a particular application domain. The Python
Package Index now contains enough information to let software
automatically find packages and download them, so the time is ripe to
implement this.

Currently this document doesn't suggest removing modules from the
standard library that are superseded by a third-party module. That's
difficult to do because it entails many backward-compatibility problems,
so it's not worth bothering with now.

Please suggest additional domains of interest.

Domain: Web tasks

XML parsing: ElementTree + SAX.

URL retrieval: libcurl? other possibilities?

HTML parsing: mxTidy? HTMLParser?

Async network I/O: Twisted

RDF parser: ???

HTTP serving: ???

HTTP cookie processing: ???

Web framework: A WSGI gateway, perhaps? Paste?

Graphics: PIL, Chaco.

Domain: Scientific Programming

Numeric: Numeric, SciPy

Graphics: PIL, Chaco.

Domain: Application Development

GUI toolkit: ???

Graphics: Reportlab for PDF generation.

Domain: Education

Graphics: PyGame

Software covered by the GNU General Public License

Some of these third-party modules are covered by the GNU General Public
License and the GNU Lesser General Public License. Providing a script to
download and install such packages, or even assembling all these
packages into a single tarball or CD-ROM, shouldn't cause any
difficulties with the GPL, under the "mere aggregation" clause of the
license.

Open Issues

What other application domains are important?

Should this just be a set of Ubuntu or Debian packages? Compiling things
such as PyGame can be very complicated and may be too difficult to
automate.

Acknowledgements

The PEP is based on an earlier draft PEP by Moshe Zadka, titled "2.0
Batteries Included."

license: Public Domain


text:
PEP: 699 Title: Remove private dict version field added in PEP 509
Author: Ken Jin<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-699-remove-private-dict-version-field-added-in-pep-509/19724
Status: Accepted Type: Standards Track Content-Type: text/x-rst Created:
03-Oct-2022 Python-Version: 3.12 Post-History: 05-Oct-2022 Replaces: 509
Resolution:
https://discuss.python.org/t/pep-699-remove-private-dict-version-field-added-in-pep-509/19724/13

Abstract

PEP 509 introduced a private ma_version_tag field for dictionaries to
allow optimizations in CPython and extension libraries. This PEP
proposes to rescind PEP 509 and declare the field an implementation
detail, as it has already been superseded by alternatives. This will
further allow the field to be reused for future optimization.

Motivation

PEP 509 introduced the ma_version_tag field to dictionaries. This 64-bit
field stored a version counter that updates every time a dictionary was
modified. The original proposal was to use this version counter as a
guard for optimizations.

Since CPython 3.11, this field has become unused by internal
optimization efforts. PEP 659-specialized instructions use other methods
of verifying that certain optimizations are safe.

To enable further optimizations in CPython, this PEP proposes that the
ma_version_tag field no longer conform to the PEP 509 specification.
This will allow the CPython developers to store other optimization
information, such as dictionary write watchers.

Rationale

This PEP does not specify what the field may be used for in the future.
This is intentional, as implementation details are subject to change,
and the field should be used only for internal consumption by CPython.

Specification

This specification rescinds that in PEP 509. The ma_version_tag field of
the Python dict class is declared to be an internal implementation
detail and may be removed altogether, or may have a different
representation. C extensions should not rely on this field.

Backwards Compatibility

Certain extensions use ma_version_tag for fast dictionary or globals
lookups. For example, Cython uses the field for fast dynamic module
variable lookups.

This PEP proposes to emit a compiler warning when accessing
ma_version_tag. After two consecutive version releases with warnings,
ma_version_tag will be removed, in line with PEP 387.

The biggest user the author could find for this field was Cython.
Discussions with a Cython maintainer indicated that removing support for
it from Cython is trivial.

Security Implications

PEP 509 was concerned with integer overflow. However, this PEP does not
introduce any additional security concerns.

Rejected Ideas

A possible alternative is to preserve the field for backwards
compatibility. This PEP rejects that alternative as future optimizations
will consume more memory, and the field was always considered private
and undocumented aside from the PEP, with no backward compatibility
guarantees. Dictionaries in Python are commonly used, and any increase
in their memory consumption will adversely affect Python’s performance.

Reference Implementation

A reference implementation can be found in python/cpython#101193.

Special Thanks

Thanks to C.A.M. Gerlach for edits and wording changes to this document.
Thanks also to Mark Shannon and Kumar Aditya for providing possible
implementations.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 315 Title: Enhanced While Loop Version: $Revision$ Last-Modified:
$Date$ Author: Raymond Hettinger<EMAIL_ADDRESS>W Isaac Carroll
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 25-Apr-2003 Python-Version: 2.5
Post-History:

Abstract

This PEP proposes adding an optional "do" clause to the beginning of the
while loop to make loop code clearer and reduce errors caused by code
duplication.

Notice

Rejected; see[1].

This PEP has been deferred since 2006; see[2].

Subsequent efforts to revive the PEP in April 2009 did not meet with
success because no syntax emerged that could compete with the following
form:

    while True:
        <setup code>
        if not <condition>:
            break
        <loop body>

A syntax alternative to the one proposed in the PEP was found for a
basic do-while loop but it gained little support because the condition
was at the top:

    do ... while <cond>:
        <loop body>

Users of the language are advised to use the while-True form with an
inner if-break when a do-while loop would have been appropriate.

Motivation

It is often necessary for some code to be executed before each
evaluation of the while loop condition. This code is often duplicated
outside the loop, as setup code that executes once before entering the
loop:

    <setup code>
    while <condition>:
        <loop body>
        <setup code>

The problem is that duplicated code can be a source of errors if one
instance is changed but the other is not. Also, the purpose of the
second instance of the setup code is not clear because it comes at the
end of the loop.

It is possible to prevent code duplication by moving the loop condition
into a helper function, or an if statement in the loop body. However,
separating the loop condition from the while keyword makes the behavior
of the loop less clear:

    def helper(args):
        <setup code>
        return <condition>

    while helper(args):
        <loop body>

This last form has the additional drawback of requiring the loop's else
clause to be added to the body of the if statement, further obscuring
the loop's behavior:

    while True:
        <setup code>
        if not <condition>: break
        <loop body>

This PEP proposes to solve these problems by adding an optional clause
to the while loop, which allows the setup code to be expressed in a
natural way:

    do:
        <setup code>
    while <condition>:
        <loop body>

This keeps the loop condition with the while keyword where it belongs,
and does not require code to be duplicated.

Syntax

The syntax of the while statement :

    while_stmt : "while" expression ":" suite
             ["else" ":" suite]

is extended as follows:

    while_stmt : ["do" ":" suite]
                "while" expression ":" suite
                ["else" ":" suite]

Semantics of break and continue

In the do-while loop the break statement will behave the same as in the
standard while loop: It will immediately terminate the loop without
evaluating the loop condition or executing the else clause.

A continue statement in the do-while loop jumps to the while condition
check.

In general, when the while suite is empty (a pass statement), the
do-while loop and break and continue statements should match the
semantics of do-while in other languages.

Likewise, when the do suite is empty, the do-while loop and break and
continue statements should match behavior found in regular while loops.

Future Statement

Because of the new keyword "do", the statement :

    from __future__ import do_while

will initially be required to use the do-while form.

Implementation

The first implementation of this PEP can compile the do-while loop as an
infinite loop with a test that exits the loop.

References

Copyright

This document is placed in the public domain.

[1] Guido van Rossum, PEP 315: do-while
https://mail.python.org/pipermail/python-ideas/2013-June/021610.html

[2] Raymond Hettinger, release plan for 2.5 ?
https://mail.python.org/pipermail/python-dev/2006-February/060718.html

authors:
Raymond Hettinger
W Isaac Carroll


text:
PEP: 690 Title: Lazy Imports Author: Germán Méndez Bravo
<EMAIL_ADDRESS>Carl Meyer<EMAIL_ADDRESS>Sponsor: Barry
Warsaw<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-690-lazy-imports/15474 Status: Rejected
Type: Standards Track Content-Type: text/x-rst Created: 29-Apr-2022
Python-Version: 3.12 Post-History: 03-May-2022, 03-May-2022 Resolution:
https://discuss.python.org/t/pep-690-lazy-imports-again/19661/26

Abstract

This PEP proposes a feature to transparently defer the finding and
execution of imported modules until the moment when an imported object
is first used. Since Python programs commonly import many more modules
than a single invocation of the program is likely to use in practice,
lazy imports can greatly reduce the overall number of modules loaded,
improving startup time and memory usage. Lazy imports also mostly
eliminate the risk of import cycles.

Motivation

Common Python code style prefers <8#imports> imports at module level, so
they don't have to be repeated within each scope the imported object is
used in, and to avoid the inefficiency of repeated execution of the
import system at runtime. This means that importing the main module of a
program typically results in an immediate cascade of imports of most or
all of the modules that may ever be needed by the program.

Consider the example of a Python command line program (CLI) with a
number of subcommands. Each subcommand may perform different tasks,
requiring the import of different dependencies. But a given invocation
of the program will only execute a single subcommand, or possibly none
(i.e. if just --help usage info is requested). Top-level eager imports
in such a program will result in the import of many modules that will
never be used at all; the time spent (possibly compiling and) executing
these modules is pure waste.

To improve startup time, some large Python CLIs make imports lazy by
manually placing imports inline into functions to delay imports of
expensive subsystems. This manual approach is labor-intensive and
fragile; one misplaced import or refactor can easily undo painstaking
optimization work.

The Python standard library already includes built-in support for lazy
imports, via importlib.util.LazyLoader. There are also third-party
packages such as demandimport. These provide a "lazy module object"
which delays its own import until first attribute access. This is not
sufficient to make all imports lazy: imports such as
from foo import a, b will still eagerly import the module foo since they
immediately access an attribute from it. It also imposes noticeable
runtime overhead on every module attribute access, since it requires a
Python-level __getattr__ or __getattribute__ implementation.

Authors of scientific Python packages have also made extensive use of
lazy imports to allow users to write e.g. import scipy as sp and then
easily access many different submodules with e.g. sp.linalg, without
requiring all the many submodules to be imported up-front. SPEC 1
codifies this practice in the form of a lazy_loader library that can be
used explicitly in a package __init__.py to provide lazily accessible
submodules.

Users of static typing also have to import names for use in type
annotations that may never be used at runtime (if PEP 563 or possibly in
future PEP 649 are used to avoid eager runtime evaluation of
annotations). Lazy imports are very attractive in this scenario to avoid
overhead of unneeded imports.

This PEP proposes a more general and comprehensive solution for lazy
imports that can encompass all of the above use cases and does not
impose detectable overhead in real-world use. The implementation in this
PEP has already demonstrated startup time improvements up to 70% and
memory-use reductions up to 40% on real-world Python CLIs.

Lazy imports also eliminate most import cycles. With eager imports,
"false cycles" can easily occur which are fixed by simply moving an
import to the bottom of a module or inline into a function, or switching
from from foo import bar to import foo. With lazy imports, these
"cycles" just work. The only cycles which will remain are those where
two modules actually each use a name from the other at module level;
these "true" cycles are only fixable by refactoring the classes or
functions involved.

Rationale

The aim of this feature is to make imports transparently lazy. "Lazy"
means that the import of a module (execution of the module body and
addition of the module object to sys.modules) should not occur until the
module (or a name imported from it) is actually referenced during
execution. "Transparent" means that besides the delayed import (and
necessarily observable effects of that, such as delayed import side
effects and changes to sys.modules), there is no other observable change
in behavior: the imported object is present in the module namespace as
normal and is transparently loaded whenever first used: its status as a
"lazy imported object" is not directly observable from Python or from C
extension code.

The requirement that the imported object be present in the module
namespace as usual, even before the import has actually occurred, means
that we need some kind of "lazy object" placeholder to represent the
not-yet-imported object. The transparency requirement dictates that this
placeholder must never be visible to Python code; any reference to it
must trigger the import and replace it with the real imported object.

Given the possibility that Python (or C extension) code may pull objects
directly out of a module __dict__, the only way to reliably prevent
accidental leakage of lazy objects is to have the dictionary itself be
responsible to ensure resolution of lazy objects on lookup.

When a lookup finds that the key references a lazy object, it resolves
the lazy object immediately before returning it. To avoid side effects
mutating dictionaries midway through iteration, all lazy objects in a
dictionary are resolved prior to starting an iteration; this could incur
a performance penalty when using bulk iterations (iter(dict),
reversed(dict), dict.__reversed__(), dict.keys(), iter(dict.keys()) and
reversed(dict.keys())). To avoid this performance penalty on the vast
majority of dictionaries, which never contain any lazy objects, we steal
a bit from the dk_kind field for a new dk_lazy_imports flag to keep
track of whether a dictionary may contain lazy objects or not.

This implementation comprehensively prevents leakage of lazy objects,
ensuring they are always resolved to the real imported object before
anyone can get hold of them for any use, while avoiding any significant
performance impact on dictionaries in general.

Specification

Lazy imports are opt-in, and they can be globally enabled either via a
new -L flag to the Python interpreter, or via a call to a new
importlib.set_lazy_imports() function. This function takes two
arguments, a boolean enabled and an excluding container. If enabled is
true, lazy imports will be turned on from that point forward. If it is
false, they will be turned off from that point forward. (Use of the
excluding keyword is discussed below under "Per-module opt-out.")

When the flag -L is passed to the Python interpreter, a new
sys.flags.lazy_imports is set to True, otherwise it exists as False.
This flag is used to propagate -L to new Python subprocesses.

The flag in sys.flags.lazy_imports does not necessarily reflect the
current status of lazy imports, only whether the interpreter was started
with the -L option. Actual current status of whether lazy imports are
enabled or not at any moment can be retrieved using
importlib.is_lazy_imports_enabled(), which will return True if lazy
imports are enabled at the call point or False otherwise.

When lazy imports are enabled, the loading and execution of all (and
only) top-level imports is deferred until the imported name is first
used. This could happen immediately (e.g. on the very next line after
the import statement) or much later (e.g. while using the name inside a
function being called by some other code at some later time.)

For these top level imports, there are two contexts which will make them
eager (not lazy): imports inside try / except / finally or with blocks,
and star imports (from foo import *.) Imports inside exception-handling
blocks (this includes with blocks, since those can also "catch" and
handle exceptions) remain eager so that any exceptions arising from the
import can be handled. Star imports must remain eager since performing
the import is the only way to know which names should be added to the
namespace.

Imports inside class definitions or inside functions/methods are not
"top level" and are never lazy.

Dynamic imports using __import__() or importlib.import_module() are also
never lazy.

Lazy imports state (i.e. whether they have been enabled, and any
excluded modules; see below) is per-interpreter, but global within the
interpreter (i.e. all threads will be affected).

Example

Say we have a module spam.py:

    # simulate some work
    import time
    time.sleep(10)
    print("spam loaded")

And a module eggs.py which imports it:

    import spam
    print("imports done")

If we run python -L eggs.py, the spam module will never be imported
(because it is never referenced after the import), "spam loaded" will
never be printed, and there will be no 10 second delay.

But if eggs.py simply references the name spam after importing it, that
will be enough to trigger the import of spam.py:

    import spam
    print("imports done")
    spam

Now if we run python -L eggs.py, we will see the output "imports done"
printed first, then a 10 second delay, and then "spam loaded" printed
after that.

Of course, in real use cases (especially with lazy imports), it's not
recommended to rely on import side effects like this to trigger real
work. This example is just to clarify the behavior of lazy imports.

Another way to explain the effect of lazy imports is that it is as if
each lazy import statement had instead been written inline in the source
code immediately before each use of the imported name. So one can think
of lazy imports as similar to transforming this code:

    import foo

    def func1():
        return foo.bar()

    def func2():
        return foo.baz()

To this:

    def func1():
        import foo
        return foo.bar()

    def func2():
        import foo
        return foo.baz()

This gives a good sense of when the import of foo will occur under lazy
imports, but lazy import is not really equivalent to this code
transformation. There are several notable differences:

-   Unlike in the latter code, under lazy imports the name foo still
    does exist in the module's global namespace, and can be imported or
    referenced by other modules that import this one. (Such references
    would also trigger the import.)
-   The runtime overhead of lazy imports is much lower than the latter
    code; after the first reference to the name foo which triggers the
    import, subsequent references will have zero import system overhead;
    they are indistinguishable from a normal name reference.

In a sense, lazy imports turn the import statement into just a
declaration of an imported name or names, to later be fully resolved
when referenced.

An import in the style from foo import bar can also be made lazy. When
the import occurs, the name bar will be added to the module namespace as
a lazy import. The first reference to bar will import foo and resolve
bar to foo.bar.

Intended usage

Since lazy imports are a potentially-breaking semantic change, they
should be enabled only by the author or maintainer of a Python
application, who is prepared to thoroughly test the application under
the new semantics, ensure it behaves as expected, and opt-out any
specific imports as needed (see below). Lazy imports should not be
enabled speculatively by the end user of a Python application with any
expectation of success.

It is the responsibility of the application developer enabling lazy
imports for their application to opt-out any library imports that turn
out to need to be eager for their application to work correctly; it is
not the responsibility of library authors to ensure that their library
behaves exactly the same under lazy imports.

The documentation of the feature, the -L flag, and the new importlib
APIs will be clear about the intended usage and the risks of adoption
without testing.

Implementation

Lazy imports are represented internally by a "lazy import" object. When
a lazy import occurs (say import foo or from foo import bar), the key
"foo" or "bar" is immediately added to the module namespace dictionary,
but with its value set to an internal-only "lazy import" object that
preserves all the necessary metadata to execute the import later.

A new boolean flag in PyDictKeysObject (dk_lazy_imports) is set to
signal that this particular dictionary may contain lazy import objects.
This flag is only used to efficiently resolve all lazy objects in "bulk"
operations, when a dictionary may contain lazy objects.

Anytime a key is looked up in a dictionary to extract its value, the
value is checked to see if it is a lazy import object. If so, the lazy
object is immediately resolved, the relevant imported modules executed,
the lazy import object is replaced in the dictionary (if possible) by
the actual imported value, and the resolved value is returned from the
lookup function. A dictionary could mutate as part of an import side
effect while resolving a lazy import object. In this case it is not
possible to efficiently replace the key value with the resolved object.
In this case, the lazy import object will gain a cached pointer to the
resolved object. On next access that cached reference will be returned
and the lazy import object will be replaced in the dict with the
resolved value.

Because this is all handled internally by the dictionary implementation,
lazy import objects can never escape from the module namespace to become
visible to Python code; they are always resolved at their first
reference. No stub, dummy or thunk objects are ever visible to Python
code or placed in sys.modules. If a module is imported lazily, no entry
for it will appear in sys.modules at all until it is actually imported
on first reference.

If two different modules (moda and modb) both contain a lazy import foo,
each module's namespace dictionary will have an independent lazy import
object under the key "foo", delaying import of the same foo module. This
is not a problem. When there is first a reference to, say, moda.foo, the
module foo will be imported and placed in sys.modules as usual, and the
lazy object under the key moda.__dict__["foo"] will be replaced by the
actual module foo. At this point modb.__dict__["foo"] will remain a lazy
import object. When modb.foo is later referenced, it will also try to
import foo. This import will find the module already present in
sys.modules, as is normal for subsequent imports of the same module in
Python, and at this point will replace the lazy import object at
modb.__dict__["foo"] with the actual module foo.

There are two cases in which a lazy import object can escape a
dictionary:

-   Into another dictionary: to preserve the performance of bulk-copy
    operations like dict.update() and dict.copy(), they do not check for
    or resolve lazy import objects. However, if the source dict has the
    dk_lazy_imports flag set that indicates it may contain lazy objects,
    that flag will be passed on to the updated/copied dictionary. This
    still ensures that the lazy import object can't escape into Python
    code without being resolved.
-   Through the garbage collector: lazy imported objects are still
    Python objects and live within the garbage collector; as such, they
    can be collected and seen via e.g. gc.get_objects(). If a lazy
    object becomes visible to Python code in this way, it is opaque and
    inert; it has no useful methods or attributes. A repr() of it would
    be shown as something like: <lazy_object 'fully.qualified.name'>.

When a lazy object is added to a dictionary, the flag dk_lazy_imports is
set. Once set, the flag is only cleared if all lazy import objects in
the dictionary are resolved, e.g. prior to dictionary iteration.

All dictionary iteration methods involving values (such as dict.items(),
dict.values(), PyDict_Next() etc.) will attempt to resolve all lazy
import objects in the dictionary prior to starting the iteration. Since
only (some) module namespace dictionaries will ever have dk_lazy_imports
set, the extra overhead of resolving all lazy import objects inside a
dictionary is only paid by those dictionaries that need it. Minimizing
the overhead on normal non-lazy dictionaries is the sole purpose of the
dk_lazy_imports flag.

PyDict_Next will attempt to resolve all lazy import objects the first
time position 0 is accessed, and those imports could fail with
exceptions. Since PyDict_Next cannot set an exception, PyDict_Next will
return 0 immediately in this case, and any exception will be printed to
stderr as an unraisable exception.

For this reason, this PEP introduces PyDict_NextWithError, which works
in the same way as PyDict_Next, but which can set an error when
returning 0 and this should be checked via PyErr_Occurred() after the
call.

The eagerness of imports within try / except / with blocks or within
class or function bodies is handled in the compiler via a new
EAGER_IMPORT_NAME opcode that always imports eagerly. Top-level imports
use IMPORT_NAME, which may be lazy or eager depending on -L and/or
importlib.set_lazy_imports().

Debugging

Debug logging from python -v will include logging whenever an import
statement has been encountered but execution of the import will be
deferred.

Python's -X importtime feature for profiling import costs adapts
naturally to lazy imports; the profiled time is the time spent actually
importing.

Although lazy import objects are not generally visible to Python code,
in some debugging cases it may be useful to check from Python code
whether the value at a given key in a given dictionary is a lazy import
object, without triggering its resolution. For this purpose,
importlib.is_lazy_import() can be used:

    from importlib import is_lazy_import

    import foo

    is_lazy_import(globals(), "foo")

    foo

    is_lazy_import(globals(), "foo")

In this example, if lazy imports have been enabled the first call to
is_lazy_import will return True and the second will return False.

Per-module opt-out

Due to the backwards compatibility issues mentioned below, it may be
necessary for an application using lazy imports to force some imports to
be eager.

In first-party code, since imports inside a try or with block are never
lazy, this can be easily accomplished:

    try:  # force these imports to be eager
        import foo
        import bar
    finally:
        pass

This PEP proposes to add a new importlib.eager_imports() context
manager, so the above technique can be less verbose and doesn't require
comments to clarify its intent:

    from importlib import eager_imports

    with eager_imports():
        import foo
        import bar

Since imports within context managers are always eager, the
eager_imports() context manager can just be an alias to a null context
manager. The context manager's effect is not transitive: foo and bar
will be imported eagerly, but imports within those modules will still
follow the usual laziness rules.

The more difficult case can occur if an import in third-party code that
can't easily be modified must be forced to be eager. For this purpose,
importlib.set_lazy_imports() takes a second optional keyword-only
excluding argument, which can be set to a container of module names
within which all imports will be eager:

    from importlib import set_lazy_imports

    set_lazy_imports(excluding=["one.mod", "another"])

The effect of this is also shallow: all imports within one.mod will be
eager, but not imports in all modules imported by one.mod.

The excluding parameter of set_lazy_imports() can be a container of any
kind that will be checked to see whether it contains a module name. If
the module name is contained in the object, imports within it will be
eager. Thus, arbitrary opt-out logic can be encoded in a __contains__
method:

    import re
    from importlib import set_lazy_imports

    class Checker:
        def __contains__(self, name):
            return re.match(r"foo\.[^.]+\.logger", name)

    set_lazy_imports(excluding=Checker())

If Python was executed with the -L flag, then lazy imports will already
be globally enabled, and the only effect of calling
set_lazy_imports(True, excluding=...) will be to globally set the eager
module names/callback. If set_lazy_imports(True) is called with no
excluding argument, the exclusion list/callback will be cleared and all
eligible imports (module-level imports not in try/except/with, and not
import *) will be lazy from that point forward.

This opt-out system is designed to maintain the possibility of local
reasoning about the laziness of an import. You only need to see the code
of one module, and the excluding argument to set_lazy_imports, if any,
to know whether a given import will be eager or lazy.

Testing

The CPython test suite will pass with lazy imports enabled (with some
tests skipped). One buildbot should run the test suite with lazy imports
enabled.

C API

For authors of C extension modules, the proposed public C API is as
follows:

  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  C API                                                                       Python API
  --------------------------------------------------------------------------- -----------------------------------------------------------------------------------------------------
  PyObject *PyImport_SetLazyImports(PyObject *enabled, PyObject *excluding)   importlib.set_lazy_imports(enabled: bool = True, *, excluding: typing.Container[str] | None = None)

  int PyDict_IsLazyImport(PyObject *dict, PyObject *name)                     importlib.is_lazy_import(dict: typing.Dict[str, object], name: str) -> bool

  int PyImport_IsLazyImportsEnabled()                                         importlib.is_lazy_imports_enabled() -> bool

  void PyDict_ResolveLazyImports(PyObject *dict)

  PyDict_NextWithError()
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

-   void PyDict_ResolveLazyImports(PyObject *dict) resolves all lazy
    objects in a dictionary, if any. To be used prior calling
    PyDict_NextWithError() or PyDict_Next().
-   PyDict_NextWithError(), works the same way as PyDict_Next(), with
    the exception it propagates any errors to the caller by returning 0
    and setting an exception. The caller should use PyErr_Occurred() to
    check for any errors.

Backwards Compatibility

This proposal preserves full backwards compatibility when the feature is
disabled, which is the default.

Even when enabled, most code will continue to work normally without any
observable change (other than improved startup time and memory usage.)
Namespace packages are not affected: they work just as they do
currently, except lazily.

In some existing code, lazy imports could produce currently unexpected
results and behaviors. The problems that we may see when enabling lazy
imports in an existing codebase are related to:

Import Side Effects

Import side effects that would otherwise be produced by the execution of
imported modules during the execution of import statements will be
deferred until the imported objects are used.

These import side effects may include:

-   code executing any side-effecting logic during import;
-   relying on imported submodules being set as attributes in the parent
    module.

A relevant and typical affected case is the click library for building
Python command-line interfaces. If e.g. cli = click.group() is defined
in main.py, and sub.py imports cli from main and adds subcommands to it
via decorator (@cli.command(...)), but the actual cli() call is in
main.py, then lazy imports may prevent the subcommands from being
registered, since in this case Click is depending on side effects of the
import of sub.py. In this case the fix is to ensure the import of sub.py
is eager, e.g. by using the importlib.eager_imports() context manager.

Dynamic Paths

There could be issues related to dynamic Python import paths;
particularly, adding (and then removing after the import) paths from
sys.path:

    sys.path.insert(0, "/path/to/foo/module")
    import foo
    del sys.path[0]
    foo.Bar()

In this case, with lazy imports enabled, the import of foo will not
actually occur while the addition to sys.path is present.

An easy fix for this (which also improves the code style and ensures
cleanup) would be to place the sys.path modifications in a context
manager. This resolves the issue, since imports inside a with block are
always eager.

Deferred Exceptions

Exceptions that occur during a lazy import bubble up and erase the
partially-constructed module(s) from sys.modules, just as exceptions
during normal import do.

Since errors raised during a lazy import will occur later than they
would if the import were eager (i.e. wherever the name is first
referenced), it is also possible that they could be accidentally caught
by exception handlers that did not expect the import to be running
within their try block, leading to confusion.

Drawbacks

Downsides of this PEP include:

-   It provides a subtly incompatible semantics for the behavior of
    Python imports. This is a potential burden on library authors who
    may be asked by their users to support both semantics, and is one
    more possibility for Python users/readers to be aware of.
-   Some popular Python coding patterns (notably centralized registries
    populated by a decorator) rely on import side effects and may
    require explicit opt-out to work as expected with lazy imports.
-   Exceptions can be raised at any point while accessing names
    representing lazy imports, this could lead to confusion and
    debugging of unexpected exceptions.

Lazy import semantics are already possible and even supported today in
the Python standard library, so these drawbacks are not newly introduced
by this PEP. So far, existing usage of lazy imports by some applications
has not proven a problem. But this PEP could make the usage of lazy
imports more popular, potentially exacerbating these drawbacks.

These drawbacks must be weighed against the significant benefits offered
by this PEP's implementation of lazy imports. Ultimately these costs
will be higher if the feature is widely used; but wide usage also
indicates the feature provides a lot of value, perhaps justifying the
costs.

Security Implications

Deferred execution of code could produce security concerns if process
owner, shell path, sys.path, or other sensitive environment or
contextual states change between the time the import statement is
executed and the time the imported object is first referenced.

Performance Impact

The reference implementation has shown that the feature has negligible
performance impact on existing real-world codebases (Instagram Server,
several CLI programs at Meta, Jupyter notebooks used by Meta
researchers), while providing substantial improvements to startup time
and memory usage.

The reference implementation shows no measurable change in aggregate
performance on the pyperformance benchmark suite.

How to Teach This

Since the feature is opt-in, beginners should not encounter it by
default. Documentation of the -L flag and importlib.set_lazy_imports()
can clarify the behavior of lazy imports.

The documentation should also clarify that opting into lazy imports is
opting into a non-standard semantics for Python imports, which could
cause Python libraries to break in unexpected ways. The responsibility
to identify these breakages and work around them with an opt-out (or
stop using lazy imports) rests entirely with the person choosing to
enable lazy imports for their application, not with the library author.
Python libraries are under no obligation to support lazy import
semantics. Politely reporting an incompatibility may be useful to the
library author, but they may choose to simply say their library does not
support use with lazy imports, and this is a valid choice.

Some best practices to deal with some of the issues that could arise and
to better take advantage of lazy imports are:

-   Avoid relying on import side effects. Perhaps the most common
    reliance on import side effects is the registry pattern, where
    population of some external registry happens implicitly during the
    importing of modules, often via decorators. Instead, the registry
    should be built via an explicit call that does a discovery process
    to find decorated functions or classes in explicitly nominated
    modules.
-   Always import needed submodules explicitly, don't rely on some other
    import to ensure a module has its submodules as attributes. That is,
    unless there is an explicit from . import bar in foo/__init__.py,
    always do import foo.bar; foo.bar.Baz, not import foo; foo.bar.Baz.
    The latter only works (unreliably) because the attribute foo.bar is
    added as a side effect of foo.bar being imported somewhere else.
    With lazy imports this may not always happen in time.
-   Avoid using star imports, as those are always eager.

Reference Implementation

The initial implementation is available as part of Cinder. This
reference implementation is in use within Meta and has proven to achieve
improvements in startup time (and total runtime for some applications)
in the range of 40%-70%, as well as significant reduction in memory
footprint (up to 40%), thanks to not needing to execute imports that end
up being unused in the common flow.

An updated reference implementation based on CPython main branch is also
available.

Rejected Ideas

Wrapping deferred exceptions

To reduce the potential for confusion, exceptions raised in the course
of executing a lazy import could be replaced by a LazyImportError
exception (a subclass of ImportError), with a __cause__ set to the
original exception.

Ensuring that all lazy import errors are raised as LazyImportError would
reduce the likelihood that they would be accidentally caught and
mistaken for a different expected exception. However, in practice we
have seen cases, e.g. inside tests, where failing modules raise
unittest.SkipTest exception and this too would end up being wrapped in
LazyImportError, making such tests fail because the true exception type
is hidden. The drawbacks here seem to outweigh the hypothetical case
where unexpected deferred exceptions are caught by mistake.

Per-module opt-in

A per-module opt-in using future imports (i.e.
from __future__ import lazy_imports) does not make sense because
__future__ imports are not feature flags, they are for transition to
behaviors which will become default in the future. It is not clear if
lazy imports will ever make sense as the default behavior, so we should
not promise this with a __future__ import.

There are other cases where a library might desire to locally opt-in to
lazy imports for a particular module; e.g. a lazy top-level __init__.py
for a large library, to make its subcomponents accessible as lazy
attributes. For now, to keep the feature simpler, this PEP chooses to
focus on the "application" use case and does not address the library use
case. The underlying laziness mechanism introduced in this PEP could be
used in the future to address this use case as well.

Explicit syntax for individual lazy imports

If the primary objective of lazy imports were solely to work around
import cycles and forward references, an explicitly-marked syntax for
particular targeted imports to be lazy would make a lot of sense. But in
practice it would be very hard to get robust startup time or memory use
benefits from this approach, since it would require converting most
imports within your code base (and in third-party dependencies) to use
the lazy import syntax.

It would be possible to aim for a "shallow" laziness where only the
top-level imports of subsystems from the main module are made explicitly
lazy, but then imports within the subsystems are all eager. This is
extremely fragile, though -- it only takes one mis-placed import to undo
the carefully constructed shallow laziness. Globally enabling lazy
imports, on the other hand, provides in-depth robust laziness where you
always pay only for the imports you use.

There may be use cases (e.g. for static typing) where
individually-marked lazy imports are desirable to avoid forward
references, but the perf/memory benefits of globally lazy imports are
not needed. Since this is a different set of motivating use cases and
requires new syntax, we prefer not to include it in this PEP. Another
PEP could build on top of this implementation and propose the additional
syntax.

Environment variable to enable lazy imports

Providing an environment variable opt-in lends itself too easily to
abuse of the feature. It may seem tempting for a Python user to, for
instance, globally set the environment variable in their shell in the
hopes of speeding up all the Python programs they run. This usage with
untested programs is likely to lead to spurious bug reports and
maintenance burden for the authors of those tools. To avoid this, we
choose not to provide an environment variable opt-in at all.

Removing the -L flag

We do provide the -L CLI flag, which could in theory be abused in a
similar way by an end user running an individual Python program that is
run with python somescript.py or python -m somescript (rather than
distributed via Python packaging tools). But the potential scope for
misuse is much less with -L than an environment variable, and -L is
valuable for some applications to maximize startup time benefits by
ensuring that all imports from the start of a process will be lazy, so
we choose to keep it.

It is already the case that running arbitrary Python programs with
command line flags they weren't intended to be used with (e.g. -s, -S,
-E, or -I) can have unexpected and breaking results. -L is nothing new
in this regard.

Half-lazy imports

It would be possible to eagerly run the import loader to the point of
finding the module source, but then defer the actual execution of the
module and creation of the module object. The advantage of this would be
that certain classes of import errors (e.g. a simple typo in the module
name) would be caught eagerly instead of being deferred to the use of an
imported name.

The disadvantage would be that the startup time benefits of lazy imports
would be significantly reduced, since unused imports would still require
a filesystem stat() call, at least. It would also introduce a possibly
non-obvious split between which import errors are raised eagerly and
which are delayed, when lazy imports are enabled.

This idea is rejected for now on the basis that in practice, confusion
about import typos has not been an observed problem with the reference
implementation. Generally delayed imports are not delayed forever, and
errors show up soon enough to be caught and fixed (unless the import is
truly unused.)

Another possible motivation for half-lazy imports would be to allow
modules themselves to control via some flag whether they are imported
lazily or eagerly. This is rejected both on the basis that it requires
half-lazy imports, giving up some of the performance benefits of import
laziness, and because in general modules do not decide how or when they
are imported, the module importing them decides that. There isn't clear
rationale for this PEP to invert that control; instead it just provides
more options for the importing code to make the decision.

Lazy dynamic imports

It would be possible to add a lazy=True or similar option to
__import__() and/or importlib.import_module(), to enable them to perform
lazy imports. That idea is rejected in this PEP for lack of a clear use
case. Dynamic imports are already far outside the PEP 8 code style
recommendations for imports, and can easily be made precisely as lazy as
desired by placing them at the desired point in the code flow. These
aren't commonly used at module top level, which is where lazy imports
applies.

Deep eager-imports override

The proposed importlib.eager_imports() context manager and excluded
modules in the importlib.set_lazy_imports(excluding=...) override all
have shallow effects: they only force eagerness for the location they
are applied to, not transitively. It would be possible to provide a
deep/transitive version of one or both. That idea is rejected in this
PEP because the implementation would be complex (taking into account
threads and async code), experience with the reference implementation
has not shown it to be necessary, and because it prevents local
reasoning about laziness of imports.

A deep override can lead to confusing behavior because the
transitively-imported modules may be imported from multiple locations,
some of which use the "deep eager override" and some of which don't.
Thus those modules may still be imported lazily initially, if they are
first imported from a location that doesn't have the override.

With deep overrides it is not possible to locally reason about whether a
given import will be lazy or eager. With the behavior specified in this
PEP, such local reasoning is possible.

Making lazy imports the default behavior

Making lazy imports the default/sole behavior of Python imports, instead
of opt-in, would have some long-term benefits, in that library authors
would (eventually) no longer need to consider the possibility of both
semantics.

However, the backwards-incompatibilies are such that this could only be
considered over a long time frame, with a __future__ import. It is not
at all clear that lazy imports should become the default import
semantics for Python.

This PEP takes the position that the Python community needs more
experience with lazy imports before considering making it the default
behavior, so that is entirely left to a possible future PEP.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

authors:
Carl Meyer
Germán Méndez Bravo


text:
PEP: 655 Title: Marking individual TypedDict items as required or
potentially-missing Author: David Foster <david at dafoster.net>
Sponsor: Guido van Rossum <guido at python.org> Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Typing Created: 30-Jan-2021
Python-Version: 3.11 Post-History: 31-Jan-2021, 11-Feb-2021,
20-Feb-2021, 26-Feb-2021, 17-Jan-2022, 28-Jan-2022 Resolution:
<EMAIL_ADDRESS>
typing:required-notrequired, typing.Required and typing.NotRequired

Abstract

PEP 589 defines notation for declaring a TypedDict with all required
keys and notation for defining a TypedDict with
all potentially-missing keys <589#totality>, however it does not provide
a mechanism to declare some keys as required and others as
potentially-missing. This PEP introduces two new notations: Required[],
which can be used on individual items of a TypedDict to mark them as
required, and NotRequired[], which can be used on individual items to
mark them as potentially-missing.

This PEP makes no Python grammar changes. Correct usage of required and
potentially-missing keys of TypedDicts is intended to be enforced only
by static type checkers and need not be enforced by Python itself at
runtime.

Motivation

It is not uncommon to want to define a TypedDict with some keys that are
required and others that are potentially-missing. Currently the only way
to define such a TypedDict is to declare one TypedDict with one value
for total and then inherit it from another TypedDict with a different
value for total:

    class _MovieBase(TypedDict):  # implicitly total=True
        title: str

    class Movie(_MovieBase, total=False):
        year: int

Having to declare two different TypedDict types for this purpose is
cumbersome.

This PEP introduces two new type qualifiers, typing.Required and
typing.NotRequired, which allow defining a single TypedDict with a mix
of both required and potentially-missing keys:

    class Movie(TypedDict):
        title: str
        year: NotRequired[int]

This PEP also makes it possible to define TypedDicts in the
alternative functional syntax <589#alternative-syntax> with a mix of
required and potentially-missing keys, which is not currently possible
at all because the alternative syntax does not support inheritance:

    Actor = TypedDict('Actor', {
        'name': str,
        # "in" is a keyword, so the functional syntax is necessary
        'in': NotRequired[List[str]],
    })

Rationale

One might think it unusual to propose notation that prioritizes marking
required keys rather than potentially-missing keys, as is customary in
other languages like TypeScript:

    interface Movie {
        title: string;
        year?: number;  // ? marks potentially-missing keys
    }

The difficulty is that the best word for marking a potentially-missing
key, Optional[], is already used in Python for a completely different
purpose: marking values that could be either of a particular type or
None. In particular the following does not work:

    class Movie(TypedDict):
        ...
        year: Optional[int]  # means int|None, not potentially-missing!

Attempting to use any synonym of “optional” to mark potentially-missing
keys (like Missing[]) would be too similar to Optional[] and be easy to
confuse with it.

Thus it was decided to focus on positive-form phrasing for required keys
instead, which is straightforward to spell as Required[].

Nevertheless it is common for folks wanting to extend a regular
(total=True) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
not required and potentially-missing, and so we also allow the
NotRequired[] form for that case.

Specification

The typing.Required type qualifier is used to indicate that a variable
declared in a TypedDict definition is a required key:

    class Movie(TypedDict, total=False):
        title: Required[str]
        year: int

Additionally the typing.NotRequired type qualifier is used to indicate
that a variable declared in a TypedDict definition is a
potentially-missing key:

    class Movie(TypedDict):  # implicitly total=True
        title: str
        year: NotRequired[int]

It is an error to use Required[] or NotRequired[] in any location that
is not an item of a TypedDict. Type checkers must enforce this
restriction.

It is valid to use Required[] and NotRequired[] even for items where it
is redundant, to enable additional explicitness if desired:

    class Movie(TypedDict):
        title: Required[str]  # redundant
        year: NotRequired[int]

It is an error to use both Required[] and NotRequired[] at the same
time:

    class Movie(TypedDict):
        title: str
        year: NotRequired[Required[int]]  # ERROR

Type checkers must enforce this restriction. The runtime implementations
of Required[] and NotRequired[] may also enforce this restriction.

The alternative functional syntax <589#alternative-syntax> for TypedDict
also supports Required[] and NotRequired[]:

    Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})

Interaction with total=False

Any PEP 589-style TypedDict declared with total=False is equivalent to a
TypedDict with an implicit total=True definition with all of its keys
marked as NotRequired[].

Therefore:

    class _MovieBase(TypedDict):  # implicitly total=True
        title: str

    class Movie(_MovieBase, total=False):
        year: int

is equivalent to:

    class _MovieBase(TypedDict):
        title: str

    class Movie(_MovieBase):
        year: NotRequired[int]

Interaction with Annotated[]

Required[] and NotRequired[] can be used with Annotated[], in any
nesting order:

    class Movie(TypedDict):
        title: str
        year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]]  # ok

    class Movie(TypedDict):
        title: str
        year: Annotated[NotRequired[int], ValueRange(-9999, 9999)]  # ok

In particular allowing Annotated[] to be the outermost annotation for an
item allows better interoperability with non-typing uses of annotations,
which may always want Annotated[] as the outermost annotation. [1]

Runtime behavior

Interaction with get_type_hints()

typing.get_type_hints(...) applied to a TypedDict will by default strip
out any Required[] or NotRequired[] type qualifiers, since these
qualifiers are expected to be inconvenient for code casually
introspecting type annotations.

typing.get_type_hints(..., include_extras=True) however will retain
Required[] and NotRequired[] type qualifiers, for advanced code
introspecting type annotations that wishes to preserve all annotations
in the original source:

    class Movie(TypedDict):
        title: str
        year: NotRequired[int]

    assert get_type_hints(Movie) == \
        {'title': str, 'year': int}
    assert get_type_hints(Movie, include_extras=True) == \
        {'title': str, 'year': NotRequired[int]}

Interaction with get_origin() and get_args()

typing.get_origin() and typing.get_args() will be updated to recognize
Required[] and NotRequired[]:

    assert get_origin(Required[int]) is Required
    assert get_args(Required[int]) == (int,)

    assert get_origin(NotRequired[int]) is NotRequired
    assert get_args(NotRequired[int]) == (int,)

Interaction with __required_keys__ and __optional_keys__

An item marked with Required[] will always appear in the
__required_keys__ for its enclosing TypedDict. Similarly an item marked
with NotRequired[] will always appear in __optional_keys__.

    assert Movie.__required_keys__ == frozenset({'title'})
    assert Movie.__optional_keys__ == frozenset({'year'})

Backwards Compatibility

No backward incompatible changes are made by this PEP.

How to Teach This

To define a TypedDict where most keys are required and some are
potentially-missing, define a single TypedDict as normal (without the
total keyword) and mark those few keys that are potentially-missing with
NotRequired[].

To define a TypedDict where most keys are potentially-missing and a few
are required, define a total=False TypedDict and mark those few keys
that are required with Required[].

If some items accept None in addition to a regular value, it is
recommended that the TYPE|None notation be preferred over Optional[TYPE]
for marking such item values, to avoid using Required[] or NotRequired[]
alongside Optional[] within the same TypedDict definition:

Yes:

    from __future__ import annotations  # for Python 3.7-3.9

    class Dog(TypedDict):
        name: str
        owner: NotRequired[str|None]

Okay (required for Python 3.5.3-3.6):

    class Dog(TypedDict):
        name: str
        owner: 'NotRequired[str|None]'

No:

    class Dog(TypedDict):
        name: str
        # ick; avoid using both Optional and NotRequired
        owner: NotRequired[Optional[str]]

Usage in Python <3.11

If your code supports Python <3.11 and wishes to use Required[] or
NotRequired[] then it should use typing_extensions.TypedDict rather than
typing.TypedDict because the latter will not understand (Not)Required[].
In particular __required_keys__ and __optional_keys__ on the resulting
TypedDict type will not be correct:

Yes (Python 3.11+ only):

    from typing import NotRequired, TypedDict

    class Dog(TypedDict):
        name: str
        owner: NotRequired[str|None]

Yes (Python <3.11 and 3.11+):

    from __future__ import annotations  # for Python 3.7-3.9

    from typing_extensions import NotRequired, TypedDict  # for Python <3.11 with (Not)Required

    class Dog(TypedDict):
        name: str
        owner: NotRequired[str|None]

No (Python <3.11 and 3.11+):

    from typing import TypedDict  # oops: should import from typing_extensions instead
    from typing_extensions import NotRequired

    class Movie(TypedDict):
        title: str
        year: NotRequired[int]

    assert Movie.__required_keys__ == frozenset({'title', 'year'})  # yikes
    assert Movie.__optional_keys__ == frozenset()  # yikes

Reference Implementation

The mypy 0.930, pyright 1.1.117, and pyanalyze 0.4.0 type checkers
support Required and NotRequired.

A reference implementation of the runtime component is provided in the
typing_extensions module.

Rejected Ideas

Special syntax around the key of a TypedDict item

    class MyThing(TypedDict):
        opt1?: str  # may not exist, but if exists, value is string
        opt2: Optional[str]  # always exists, but may have None value

This notation would require Python grammar changes and it is not
believed that marking TypedDict items as required or potentially-missing
would meet the high bar required to make such grammar changes.

    class MyThing(TypedDict):
        Optional[opt1]: str  # may not exist, but if exists, value is string
        opt2: Optional[str]  # always exists, but may have None value

This notation causes Optional[] to take on different meanings depending
on where it is positioned, which is inconsistent and confusing.

Also, “let’s just not put funny syntax before the colon.”[2]

Marking required or potentially-missing keys with an operator

We could use unary + as shorthand to mark a required key, unary - to
mark a potentially-missing key, or unary ~ to mark a key with
opposite-of-normal totality:

    class MyThing(TypedDict, total=False):
        req1: +int    # + means a required key, or Required[]
        opt1: str
        req2: +float

    class MyThing(TypedDict):
        req1: int
        opt1: -str    # - means a potentially-missing key, or NotRequired[]
        req2: float

    class MyThing(TypedDict):
        req1: int
        opt1: ~str    # ~ means a opposite-of-normal-totality key
        req2: float

Such operators could be implemented on type via the __pos__, __neg__ and
__invert__ special methods without modifying the grammar.

It was decided that it would be prudent to introduce long-form notation
(i.e. Required[] and NotRequired[]) before introducing any short-form
notation. Future PEPs may reconsider introducing this or other
short-form notation options.

Note when reconsidering introducing this short-form notation that +, -,
and ~ already have existing meanings in the Python typing world:
covariant, contravariant, and invariant:

    >>> from typing import TypeVar
    >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V'))
    (+T, -U, ~V)

Marking absence of a value with a special constant

We could introduce a new type-level constant which signals the absence
of a value when used as a union member, similar to JavaScript’s
undefined type, perhaps called Missing:

    class MyThing(TypedDict):
        req1: int
        opt1: str|Missing
        req2: float

Such a Missing constant could also be used for other scenarios such as
the type of a variable which is only conditionally defined:

    class MyClass:
        attr: int|Missing

        def __init__(self, set_attr: bool) -> None:
            if set_attr:
                self.attr = 10

    def foo(set_attr: bool) -> None:
        if set_attr:
            attr = 10
        reveal_type(attr)  # int|Missing

Misalignment with how unions apply to values

However this use of ...|Missing, equivalent to Union[..., Missing],
doesn’t align well with what a union normally means: Union[...] always
describes the type of a value that is present. By contrast missingness
or non-totality is a property of a variable instead. Current precedent
for marking properties of a variable include Final[...] and
ClassVar[...], which the proposal for Required[...] is aligned with.

Misalignment with how unions are subdivided

Furthermore the use of Union[..., Missing] doesn’t align with the usual
ways that union values are broken down: Normally you can eliminate
components of a union type using isinstance checks:

    class Packet:
        data: Union[str, bytes]

    def send_data(packet: Packet) -> None:
        if isinstance(packet.data, str):
            reveal_type(packet.data)  # str
            packet_bytes = packet.data.encode('utf-8')
        else:
            reveal_type(packet.data)  # bytes
            packet_bytes = packet.data
        socket.send(packet_bytes)

However if we were to allow Union[..., Missing] you’d either have to
eliminate the Missing case with hasattr for object attributes:

    class Packet:
        data: Union[str, Missing]

    def send_data(packet: Packet) -> None:
        if hasattr(packet, 'data'):
            reveal_type(packet.data)  # str
            packet_bytes = packet.data.encode('utf-8')
        else:
            reveal_type(packet.data)  # Missing? error?
            packet_bytes = b''
        socket.send(packet_bytes)

or a check against locals() for local variables:

    def send_data(packet_data: Optional[str]) -> None:
        packet_bytes: Union[str, Missing]
        if packet_data is not None:
            packet_bytes = packet.data.encode('utf-8')

        if 'packet_bytes' in locals():
            reveal_type(packet_bytes)  # bytes
            socket.send(packet_bytes)
        else:
            reveal_type(packet_bytes)  # Missing? error?

or a check via other means, such as against globals() for global
variables:

    warning: Union[str, Missing]
    import sys
    if sys.version_info < (3, 6):
        warning = 'Your version of Python is unsupported!'

    if 'warning' in globals():
        reveal_type(warning)  # str
        print(warning)
    else:
        reveal_type(warning)  # Missing? error?

Weird and inconsistent. Missing is not really a value at all; it’s an
absence of definition and such an absence should be treated specially.

Difficult to implement

Eric Traut from the Pyright type checker team has stated that
implementing a Union[..., Missing]-style notation would be difficult.[3]

Introduces a second null-like value into Python

Defining a new Missing type-level constant would be very close to
introducing a new Missing value-level constant at runtime, creating a
second null-like runtime value in addition to None. Having two different
null-like constants in Python (None and Missing) would be confusing.
Many newcomers to JavaScript already have difficulty distinguishing
between its analogous constants null and undefined.

Replace Optional with Nullable. Repurpose Optional to mean “optional item”.

Optional[] is too ubiquitous to deprecate, although use of it may fade
over time in favor of the T|None notation specified by PEP 604.

Change Optional to mean “optional item” in certain contexts instead of “nullable”

Consider the use of a special flag on a TypedDict definition to alter
the interpretation of Optional inside the TypedDict to mean “optional
item” rather than its usual meaning of “nullable”:

    class MyThing(TypedDict, optional_as_missing=True):
        req1: int
        opt1: Optional[str]

or:

    class MyThing(TypedDict, optional_as_nullable=False):
        req1: int
        opt1: Optional[str]

This would add more confusion for users because it would mean that in
some contexts the meaning of Optional[] is different than in other
contexts, and it would be easy to overlook the flag.

Various synonyms for “potentially-missing item”

-   Omittable – too easy to confuse with optional
-   OptionalItem, OptionalKey – two words; too easy to confuse with
    optional
-   MayExist, MissingOk – two words
-   Droppable – too similar to Rust’s Drop, which means something
    different
-   Potential – too vague
-   Open – sounds like applies to an entire structure rather then to an
    item
-   Excludable
-   Checked

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] https://bugs.python.org/issue46491

[2]<EMAIL_ADDRESS>
[3] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6WCIWPBZ54BOJPG56KXVSLZK6/


text:
PEP: 656 Title: Platform Tag for Linux Distributions Using Musl Author:
Tzu-ping Chung<EMAIL_ADDRESS>Sponsor: Brett Cannon
<EMAIL_ADDRESS>PEP-Delegate: Paul Moore<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/7165 Status: Final Type:
Standards Track Topic: Packaging Content-Type: text/x-rst Created:
17-Mar-2021 Post-History: 17-Mar-2021, 18-Apr-2021 Resolution:
https://discuss.python.org/t/7165/32

Abstract

This PEP proposes a new platform tag series musllinux for binary Python
package distributions for a Python installation that depends on musl on
a Linux distribution. The tag works similarly to the "perennial
manylinux" platform tags specified in PEP 600, but targeting platforms
based on musl instead.

Motivation

With the wide use of containers, distributions such as Alpine Linux
[alpine], have been gaining more popularity than ever. Many of them
based on musl [musl], a different libc implementation from glibc, and
therefore cannot use the existing manylinux platform tags. This means
that Python package projects cannot deploy binary distributions on PyPI
for them. Users of such projects demand build constraints from those
projects, putting unnecessary burden on project maintainers.

Rationale

According to the documentation, musl has a stable ABI, and maintains
backwards compatibility [musl-compatibility] [compare-libcs], so a
binary compiled against an earlier version of musl is guaranteed to run
against a newer musl runtime [musl-compat-ml]. Therefore, we use a
scheme similar to the glibc-version-based manylinux tags, but against
musl versions instead of glibc.

Logic behind the new platform tag largely follows PEP 600 ("perennial
manylinux"), and requires wheels using this tag make similar promises.
Please refer to PEP 600 for more details on rationale and reasoning
behind the design.

The musllinux platform tags only apply to Python interpreters
dynamically linked against the musl libc and executed on the runtime
shared library, on a Linux operating system. Statically linked
interpreters or mixed builds with other libc implementations (such as
glibc) are out of scope and not supported by platform tags defined in
this document. Such interpreters should not claim compatibility with
musllinux platform tags.

Specification

Tags using the new scheme will take the form:

    musllinux_${MUSLMAJOR}_${MUSLMINOR}_${ARCH}

This tag promises the wheel works on any mainstream Linux distribution
that uses musl version ${MUSLMAJOR}.${MUSLMINOR}, following the
perennial design. All other system-level dependency requirements rely on
the community's definition to the intentionally vague "mainstream"
description introduced in PEP 600. A wheel may make use of newer system
dependencies when all mainstream distributions using the specified musl
version provide the dependency by default; once all mainstream
distributions on the musl version ship a certain dependency version by
default, users relying on older versions are automatically removed from
the coverage of that musllinux tag.

Reading the musl version

The musl version values can be obtained by executing the musl libc
shared library the Python interpreter is currently running on, and
parsing the output:

    import re
    import subprocess

    def get_musl_major_minor(so: str) -> tuple[int, int] | None:
        """Detect musl runtime version.

        Returns a two-tuple ``(major, minor)`` that indicates musl
        library's version, or ``None`` if the given libc .so does not
        output expected information.

        The libc library should output something like this to stderr::

            musl libc (x86_64)
            Version 1.2.2
            Dynamic Program Loader
        """
        proc = subprocess.run([so], stderr=subprocess.PIPE, text=True)
        lines = (line.strip() for line in proc.stderr.splitlines())
        lines = [line for line in lines if line]
        if len(lines) < 2 or lines[0][:4] != "musl":
            return None
        match = re.match(r"Version (\d+)\.(\d+)", lines[1])
        if match:
            return (int(match.group(1)), int(match.group(2)))
        return None

There are currently two possible ways to find the musl library's
location that a Python interpreter is running on, either with the system
ldd command [ldd], or by parsing the PT_INTERP section's value from the
executable's ELF header [elf].

Formatting the tag

Distributions using the tag make similar promises to those described in
PEP 600, including:

1.  The distribution works on any mainstream Linux distributions with
    musl version ${MUSLMAJOR}.${MUSLMINOR} or later.
2.  The distribution's ${ARCH} matches the return value of
    sysconfig.get_platform() on the host system, replacing period (.)
    and hyphen (-) characters with underscores (_), as outlined in PEP
    425 and PEP 427.

Example values:

    musllinux_1_1_x86_64   # musl 1.1 running on x86-64.
    musllinux_1_2_aarch64  # musl 1.2 running on ARM 64-bit.

The value can be formatted with the following Python code:

    import sysconfig

    def format_musllinux(musl_version: tuple[int, int]) -> str:
        os_name, sep, arch = sysconfig.get_platform().partition("-")
        assert os_name == "linux" and sep, "Not a Linux"
        arch = arch.replace(".", "_").replace("-", "_")
        return f"musllinux_{musl_version[0]}_{musl_version[1]}_{arch}"

Recommendations to package indexes

It is recommended for Python package repositories, including PyPI, to
accept platform tags matching the following regular expression:

    musllinux_([0-9]+)_([0-9]+)_([^.-]+)

Python package repositories may impose additional requirements to reject
Wheels with known issues, including but not limited to:

-   A musllinux_1_1 wheel containing symbols only available in musl 1.2
    or later.
-   Wheel that depends on external libraries not considered generally
    available to the intended audience of the package index.
-   A platform tag claiming compatibility to a non-existent musl version
    (like musllinux_9000_0).

Such policies are ultimately up to individual package repositories. It
is not the author's intention to impose restrictions to the maintainers.

Backwards Compatibility

There are no backwards compatibility concerns in this PEP.

Rejected Ideas

Create a platform tag based specifically for Alpine Linux

Past experience on the manylinux tag series shows this approach would be
too costly time-wise. The author feels the "works well with others" rule
both is more inclusive and works well enough in practice.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

alpine

    https://alpinelinux.org/

compare-libcs

    https://www.etalabs.net/compare_libcs.html

elf

    https://refspecs.linuxfoundation.org/elf/elf.pdf

ldd

    https://www.unix.com/man-page/posix/1/ldd/

musl

    https://musl.libc.org

musl-compat-ml

<EMAIL_ADDRESS>
musl-compatibility

    https://wiki.musl-libc.org/compatibility.html