text:
PEP: 649 Title: Deferred Evaluation Of Annotations Using Descriptors
Author: Larry Hastings<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/
Status: Accepted Type: Standards Track Topic: Typing Content-Type:
text/x-rst Created: 11-Jan-2021 Python-Version: 3.14 Post-History:
11-Jan-2021, 12-Apr-2021, 18-Apr-2021, 09-Aug-2021, 20-Oct-2021,
20-Oct-2021, 17-Nov-2021, 15-Mar-2022, 23-Nov-2022, 07-Feb-2023,
11-Apr-2023, Replaces: 563 Resolution: 08-May-2023

Abstract

Annotations are a Python technology that allows expressing type
information and other metadata about Python functions, classes, and
modules. But Python's original semantics for annotations required them
to be eagerly evaluated, at the time the annotated object was bound.
This caused chronic problems for static type analysis users using "type
hints", due to forward-reference and circular-reference problems.

Python solved this by accepting PEP 563, incorporating a new approach
called "stringized annotations" in which annotations were automatically
converted into strings by Python. This solved the forward-reference and
circular-reference problems, and also fostered intriguing new uses for
annotation metadata. But stringized annotations in turn caused chronic
problems for runtime users of annotations.

This PEP proposes a new and comprehensive third approach for
representing and computing annotations. It adds a new internal mechanism
for lazily computing annotations on demand, via a new object method
called __annotate__. This approach, when combined with a novel technique
for coercing annotation values into alternative formats, solves all the
above problems, supports all existing use cases, and should foster
future innovations in annotations.

Overview

This PEP adds a new dunder attribute to the objects that support
annotations--functions, classes, and modules. The new attribute is
called __annotate__, and is a reference to a function which computes and
returns that object's annotations dict.

At compile time, if the definition of an object includes annotations,
the Python compiler will write the expressions computing the annotations
into its own function. When run, the function will return the
annotations dict. The Python compiler then stores a reference to this
function in __annotate__ on the object.

Furthermore, __annotations__ is redefined to be a "data descriptor"
which calls this annotation function once and caches the result.

This mechanism delays the evaluation of annotations expressions until
the annotations are examined, which solves many circular reference
problems.

This PEP also defines new functionality for two functions in the Python
standard library: inspect.get_annotations and typing.get_type_hints. The
functionality is accessed via a new keyword-only parameter, format.
format allows the user to request the annotations from these functions
in a specific format. Format identifiers are always predefined integer
values. The formats defined by this PEP are:

-   inspect.VALUE = 1

    The default value. The function will return the conventional Python
    values for the annotations. This format is identical to the return
    value for these functions under Python 3.11.

-   inspect.FORWARDREF = 2

    The function will attempt to return the conventional Python values
    for the annotations. However, if it encounters an undefined name, or
    a free variable that has not yet been associated with a value, it
    dynamically creates a proxy object (a ForwardRef) that substitutes
    for that value in the expression, then continues evaluation. The
    resulting dict may contain a mixture of proxies and real values. If
    all real values are defined at the time the function is called,
    inspect.FORWARDREF and inspect.VALUE produce identical results.

-   inspect.SOURCE = 3

    The function will produce an annotation dictionary where the values
    have been replaced by strings containing the original source code
    for the annotation expressions. These strings may only be
    approximate, as they may be reverse-engineered from another format,
    rather than preserving the original source code, but the differences
    will be minor.

If accepted, this PEP would supersede PEP 563, and PEP 563's behavior
would be deprecated and eventually removed.

Comparison Of Annotation Semantics

Note

The code presented in this section is simplified for clarity, and is
intentionally inaccurate in some critical aspects. This example is
intended merely to communicate the high-level concepts involved without
getting lost in the details. But readers should note that the actual
implementation is quite different in several important ways. See the
Implementation section later in this PEP for a far more accurate
description of what this PEP proposes from a technical level.

Consider this example code:

    def foo(x: int = 3, y: MyType = None) -> float:
        ...
    class MyType:
        ...
    foo_y_annotation = foo.__annotations__['y']

As we see here, annotations are available at runtime through an
__annotations__ attribute on functions, classes, and modules. When
annotations are specified on one of these objects, __annotations__ is a
dictionary mapping the names of the fields to the value specified as
that field's annotation.

The default behavior in Python is to evaluate the expressions for the
annotations, and build the annotations dict, at the time the function,
class, or module is bound. At runtime the above code actually works
something like this:

    annotations = {'x': int, 'y': MyType, 'return': float}
    def foo(x = 3, y = "abc"):
        ...
    foo.__annotations__ = annotations
    class MyType:
        ...
    foo_y_annotation = foo.__annotations__['y']

The crucial detail here is that the values int, MyType, and float are
looked up at the time the function object is bound, and these values are
stored in the annotations dict. But this code doesn't run—it throws a
NameError on the first line, because MyType hasn't been defined yet.

PEP 563's solution is to decompile the expressions back into strings
during compilation and store those strings as the values in the
annotations dict. The equivalent runtime code would look something like
this:

    annotations = {'x': 'int', 'y': 'MyType', 'return': 'float'}
    def foo(x = 3, y = "abc"):
        ...
    foo.__annotations__ = annotations
    class MyType:
        ...
    foo_y_annotation = foo.__annotations__['y']

This code now runs successfully. However, foo_y_annotation is no longer
a reference to MyType, it is the string 'MyType'. To turn the string
into the real value MyType, the user would need to evaluate the string
using eval, inspect.get_annotations, or typing.get_type_hints.

This PEP proposes a third approach, delaying the evaluation of the
annotations by computing them in their own function. If this PEP was
active, the generated code would work something like this:

    class function:
        # __annotations__ on a function object is already a
        # "data descriptor" in Python, we're just changing
        # what it does
        @property
        def __annotations__(self):
            return self.__annotate__()

    # ...

    def annotate_foo():
        return {'x': int, 'y': MyType, 'return': float}
    def foo(x = 3, y = "abc"):
        ...
    foo.__annotate__ = annotate_foo
    class MyType:
       ...
    foo_y_annotation = foo.__annotations__['y']

The important change is that the code constructing the annotations dict
now lives in a function—here, called annotate_foo(). But this function
isn't called until we ask for the value of foo.__annotations__, and we
don't do that until after the definition of MyType. So this code also
runs successfully, and foo_y_annotation now has the correct value--the
class MyType--even though MyType wasn't defined until after the
annotation was defined.

Mistaken Rejection Of This Approach In November 2017

During the early days of discussion around PEP 563, in a November 2017
thread in comp.lang.python-dev, the idea of using code to delay the
evaluation of annotations was briefly discussed. At the time the
technique was termed an "implicit lambda expression".

Guido van Rossum—Python's BDFL at the time—replied, asserting that these
"implicit lambda expression" wouldn't work, because they'd only be able
to resolve symbols at module-level scope:

  IMO the inability of referencing class-level definitions from
  annotations on methods pretty much kills this idea.

https://mail.python.org/pipermail/python-dev/2017-November/150109.html

This led to a short discussion about extending lambda-ized annotations
for methods to be able to refer to class-level definitions, by
maintaining a reference to the class-level scope. This idea, too, was
quickly rejected.

PEP 563 summarizes the above discussion
<563#keeping-the-ability-to-use-function-local-state-when-defining-annotations>

The approach taken by this PEP doesn't suffer from these restrictions.
Annotations can access module-level definitions, class-level
definitions, and even local and free variables.

Motivation

A History Of Annotations

Python 3.0 shipped with a new syntax feature, "annotations", defined in
PEP 3107. This allowed specifying a Python value that would be
associated with a parameter of a Python function, or with the value that
function returns. Said another way, annotations gave Python users an
interface to provide rich metadata about a function parameter or return
value, for example type information. All the annotations for a function
were stored together in a new attribute __annotations__, in an
"annotation dict" that mapped parameter names (or, in the case of the
return annotation, using the name 'return') to their Python value.

In an effort to foster experimentation, Python intentionally didn't
define what form this metadata should take, or what values should be
used. User code began experimenting with this new facility almost
immediately. But popular libraries that make use of this functionality
were slow to emerge.

After years of little progress, the BDFL chose a particular approach for
expressing static type information, called type hints, as defined in PEP
484. Python 3.5 shipped with a new typing module which quickly became
very popular.

Python 3.6 added syntax to annotate local variables, class attributes,
and module attributes, using the approach proposed in PEP 526. Static
type analysis continued to grow in popularity.

However, static type analysis users were increasingly frustrated by an
inconvenient problem: forward references. In classic Python, if a class
C depends on a later-defined class D, it's normally not a problem,
because user code will usually wait until both are defined before trying
to use either. But annotations added a new complication, because they
were computed at the time the annotated object (function, class, or
module) was bound. If methods on class C are annotated with type D, and
these annotation expressions are computed at the time that the method is
bound, D may not be defined yet. And if methods in D are also annotated
with type C, you now have an unresolvable circular reference problem.

Initially, static type users worked around this problem by defining
their problematic annotations as strings. This worked because a string
containing the type hint was just as usable for the static type analysis
tool. And users of static type analysis tools rarely examine the
annotations at runtime, so this representation wasn't itself an
inconvenience. But manually stringizing type hints was clumsy and
error-prone. Also, code bases were adding more and more annotations,
which consumed more and more CPU time to create and bind.

To solve these problems, the BDFL accepted PEP 563, which added a new
feature to Python 3.7: "stringized annotations". It was activated with a
future import:

    from __future__ import annotations

Normally, annotation expressions were evaluated at the time the object
was bound, with their values being stored in the annotations dict. When
stringized annotations were active, these semantics changed: instead, at
compile time, the compiler converted all annotations in that module into
string representations of their source code--thus, automatically turning
the users's annotations into strings, obviating the need to manually
stringize them as before. PEP 563 suggested users could evaluate this
string with eval if the actual value was needed at runtime.

(From here on out, this PEP will refer to the classic semantics of PEP
3107 and PEP 526, where the values of annotation expressions are
computed at the time the object is bound, as "stock" semantics, to
differentiate them from the new PEP 563 "stringized" annotation
semantics.)

The Current State Of Annotation Use Cases

Although there are many specific use cases for annotations, annotation
users in the discussion around this PEP tended to fall into one of these
four categories.

Static typing users

Static typing users use annotations to add type information to their
code. But they largely don't examine the annotations at runtime.
Instead, they use static type analysis tools (mypy, pytype) to examine
their source tree and determine whether or not their code is using types
consistently. This is almost certainly the most popular use case for
annotations today.

Many of the annotations use type hints, a la PEP 484 (and many
subsequent PEPs). Type hints are passive objects, mere representation of
type information; they don't do any actual work. Type hints are often
parameterized with other types or other type hints. Since they're
agnostic about what these actual values are, type hints work fine with
ForwardRef proxy objects. Users of static type hints discovered that
extensive type hinting under stock semantics often created large-scale
circular reference and circular import problems that could be difficult
to solve. PEP 563 was designed specifically to solve this problem, and
the solution worked great for these users. The difficulty of rendering
stringized annotations into real values largely didn't inconvenience
these users because of how infrequently they examine annotations at
runtime.

Static typing users often combine PEP 563 with the
if typing.TYPE_CHECKING idiom to prevent their type hints from being
loaded at runtime. This means they often aren't able to evaluate their
stringized annotations and produce real values at runtime. On the rare
occasion that they do examine annotations at runtime, they often forgo
eval, instead using lexical analysis directly on the stringized
annotations.

Under this PEP, static typing users will probably prefer FORWARDREF or
SOURCE format.

Runtime annotation users

Runtime annotation users use annotations as a means of expressing rich
metadata about their functions and classes, which they use as input to
runtime behavior. Specific use cases include runtime type verification
(Pydantic) and glue logic to expose Python APIs in another domain
(FastAPI, Typer). The annotations may or may not be type hints.

As runtime annotation users examine annotations at runtime, they were
traditionally better served with stock semantics. This use case is
largely incompatible with PEP 563, particularly with the
if typing.TYPE_CHECKING idiom.

Under this PEP, runtime annotation users will most likely prefer VALUE
format, though some (e.g. if they evaluate annotations eagerly in a
decorator and want to support forward references) may also use
FORWARDREF format.

Wrappers

Wrappers are functions or classes that wrap user functions or classes
and add functionality. Examples of this would be ~dataclasses.dataclass,
functools.partial, attrs, and wrapt.

Wrappers are a distinct subcategory of runtime annotation users.
Although they do use annotations at runtime, they may or may not
actually examine the annotations of the objects they wrap--it depends on
the functionality the wrapper provides. As a rule they should propagate
the annotations of the wrapped object to the wrapper they create,
although it's possible they may modify those annotations.

Wrappers were generally designed to work well under stock semantics.
Whether or not they work well under PEP 563 semantics depends on the
degree to which they examine the wrapped object's annotations. Often
wrappers don't care about the value per se, only needing specific
information about the annotations. Even so, PEP 563 and the
if typing.TYPE_CHECKING idiom can make it difficult for wrappers to
reliably determine the information they need at runtime. This is an
ongoing, chronic problem. Under this PEP, wrappers will probably prefer
FORWARDREF format for their internal logic. But the wrapped objects need
to support all formats for their users.

Documentation

PEP 563 stringized annotations were a boon for tools that mechanically
construct documentation.

Stringized type hints make for excellent documentation; type hints as
expressed in source code are often succinct and readable. However, at
runtime these same type hints can produce value at runtime whose repr is
a sprawling, nested, unreadable mess. Thus documentation users were
well-served by PEP 563 but poorly served with stock semantics.

Under this PEP, documentation users are expected to use SOURCE format.

Motivation For This PEP

Python's original semantics for annotations made its use for static type
analysis painful due to forward reference problems. PEP 563 solved the
forward reference problem, and many static type analysis users became
happy early adopters of it. But its unconventional solution created new
problems for two of the above cited use cases: runtime annotation users,
and wrappers.

First, stringized annotations didn't permit referencing local or free
variables, which meant many useful, reasonable approaches to creating
annotations were no longer viable. This was particularly inconvenient
for decorators that wrap existing functions and classes, as these
decorators often use closures.

Second, in order for eval to correctly look up globals in a stringized
annotation, you must first obtain a reference to the correct module. But
class objects don't retain a reference to their globals. PEP 563
suggests looking up a class's module by name in sys.modules—a surprising
requirement for a language-level feature.

Additionally, complex but legitimate constructions can make it difficult
to determine the correct globals and locals dicts to give to eval to
properly evaluate a stringized annotation. Even worse, in some
situations it may simply be infeasible.

For example, some libraries (e.g. typing.TypedDict, dataclasses) wrap a
user class, then merge all the annotations from all that class's base
classes together into one cumulative annotations dict. If those
annotations were stringized, calling eval on them later may not work
properly, because the globals dictionary used for the eval will be the
module where the user class was defined, which may not be the same
module where the annotation was defined. However, if the annotations
were stringized because of forward-reference problems, calling eval on
them early may not work either, due to the forward reference not being
resolvable yet. This has proved to be difficult to reconcile; of the
three bug reports linked to below, only one has been marked as fixed.

-   https://github.com/python/cpython/issues/89687
-   https://github.com/python/cpython/issues/85421
-   https://github.com/python/cpython/issues/90531

Even with proper globals and locals, eval can be unreliable on
stringized annotations. eval can only succeed if all the symbols
referenced in an annotations are defined. If a stringized annotation
refers to a mixture of defined and undefined symbols, a simple eval of
that string will fail. This is a problem for libraries with that need to
examine the annotation, because they can't reliably convert these
stringized annotations into real values.

-   Some libraries (e.g. dataclasses) solved this by foregoing real
    values and performing lexical analysis of the stringized annotation,
    which requires a lot of work to get right.
-   Other libraries still suffer with this problem, which can produce
    surprising runtime behavior.
    https://github.com/python/cpython/issues/97727

Also, eval() is slow, and it isn't always available; it's sometimes
removed for space reasons on certain platforms. eval() on MicroPython
doesn't support the locals argument, which makes converting stringized
annotations into real values at runtime even harder.

Finally, PEP 563 requires Python implementations to stringize their
annotations. This is surprising behavior—unprecedented for a
language-level feature, with a complicated implementation, that must be
updated whenever a new operator is added to the language.

These problems motivated the research into finding a new approach to
solve the problems facing annotations users, resulting in this PEP.

Implementation

Observed semantics for annotations expressions

For any object o that supports annotations, provided that all names
evaluated in the annotations expressions are bound before o is defined
and never subsequently rebound, o.__annotations__ will produce an
identical annotations dict both when "stock" semantics are active and
when this PEP is active. In particular, name resolution will be
performed identically in both scenarios.

When this PEP is active, the value of o.__annotations__ won't be
calculated until the first time o.__annotations__ itself is evaluated.
All evaluation of the annotation expressions is delayed until this
moment, which also means that

-   names referenced in the annotations expressions will use their
    current value at this moment, and
-   if evaluating the annotations expressions raises an exception, that
    exception will be raised at this moment.

Once o.__annotations__ is successfully calculated for the first time,
this value is cached and will be returned by future requests for
o.__annotations__.

------------------------------------------------------------------------

Python supports annotations on three different types: functions,
classes, and modules. This PEP modifies the semantics on all three of
these types in a similar way.

First, this PEP adds a new "dunder" attribute, __annotate__.
__annotate__ must be a "data descriptor", implementing all three
actions: get, set, and delete. The __annotate__ attribute is always
defined, and may only be set to either None or to a callable.
(__annotate__ cannot be deleted.) If an object has no annotations,
__annotate__ should be initialized to None, rather than to a function
that returns an empty dict.

The __annotate__ data descriptor must have dedicated storage inside the
object to store the reference to its value. The location of this storage
at runtime is an implementation detail. Even if it's visible to Python
code, it should still be considered an internal implementation detail,
and Python code should prefer to interact with it only via the
__annotate__ attribute.

The callable stored in __annotate__ must accept a single required
positional argument called format, which will always be an int (or a
subclass of int). It must either return a dict (or subclass of dict) or
raise NotImplementedError().

Here's a formal definition of __annotate__, as it will appear in the
"Magic methods" section of the Python Language Reference:

  __annotate__(format: int) -> dict

  Returns a new dictionary object mapping attribute/parameter names to
  their annotation values.

  Takes a format parameter specifying the format in which annotations
  values should be provided. Must be one of the following:

  inspect.VALUE (equivalent to the int constant 1)

    Values are the result of evaluating the annotation expressions.

  inspect.FORWARDREF (equivalent to the int constant 2)

    Values are real annotation values (as per inspect.VALUE format) for
    defined values, and ForwardRef proxies for undefined values. Real
    objects may be exposed to, or contain references to, ForwardRef
    proxy objects.

  inspect.SOURCE (equivalent to the int constant 3)

    Values are the text string of the annotation as it appears in the
    source code. May only be approximate; whitespace may be normalized,
    and constant values may be optimized. It's possible the exact values
    of these strings could change in future version of Python.

  If an __annotate__ function doesn't support the requested format, it
  must raise NotImplementedError(). __annotate__ functions must always
  support 1 (inspect.VALUE) format; they must not raise
  NotImplementedError() when called with format=1.

  When called with format=1, an __annotate__ function may raise
  NameError; it must not raise NameError when called requesting any
  other format.

  If an object doesn't have any annotations, __annotate__ should
  preferably be set to None (it can't be deleted), rather than set to a
  function that returns an empty dict.

When the Python compiler compiles an object with annotations, it
simultaneously compiles the appropriate annotate function. This
function, called with the single positional argument inspect.VALUE,
computes and returns the annotations dict as defined on that object. The
Python compiler and runtime work in concert to ensure that the function
is bound to the appropriate namespaces:

-   For functions and classes, the globals dictionary will be the module
    where the object was defined. If the object is itself a module, its
    globals dictionary will be its own dict.
-   For methods on classes, and for classes, the locals dictionary will
    be the class dictionary.
-   If the annotations refer to free variables, the closure will be the
    appropriate closure tuple containing cells for free variables.

Second, this PEP requires that the existing __annotations__ must be a
"data descriptor", implementing all three actions: get, set, and delete.
__annotations__ must also have its own internal storage it uses to cache
a reference to the annotations dict:

-   Class and module objects must cache the annotations dict in their
    __dict__, using the key __annotations__. This is required for
    backwards compatibility reasons.
-   For function objects, storage for the annotations dict cache is an
    implementation detail. It's preferably internal to the function
    object and not visible in Python.

This PEP defines semantics on how __annotations__ and __annotate__
interact, for all three types that implement them. In the following
examples, fn represents a function, cls represents a class, mod
represents a module, and o represents an object of any of these three
types:

-   When o.__annotations__ is evaluated, and the internal storage for
    o.__annotations__ is unset, and o.__annotate__ is set to a callable,
    the getter for o.__annotations__ calls o.__annotate__(1), then
    caches the result in its internal storage and returns the result.
    -   To explicitly clarify one question that has come up multiple
        times: this o.__annotations__ cache is the only caching
        mechanism defined in this PEP. There are no other caching
        mechanisms defined in this PEP. The __annotate__ functions
        generated by the Python compiler explicitly don't cache any of
        the values they compute.
-   Setting o.__annotate__ to a callable invalidates the cached
    annotations dict.
-   Setting o.__annotate__ to None has no effect on the cached
    annotations dict.
-   Deleting o.__annotate__ raises TypeError. __annotate__ must always
    be set; this prevents unannotated subclasses from inheriting the
    __annotate__ method of one of their base classes.
-   Setting o.__annotations__ to a legal value automatically sets
    o.__annotate__ to None.
    -   Setting cls.__annotations__ or mod.__annotations__ to None
        otherwise works like any other attribute; the attribute is set
        to None.
    -   Setting fn.__annotations__ to None invalidates the cached
        annotations dict. If fn.__annotations__ doesn't have a cached
        annotations value, and fn.__annotate__ is None, the
        fn.__annotations__ data descriptor creates, caches, and returns
        a new empty dict. (This is for backwards compatibility with PEP
        3107 semantics.)

Changes to allowable annotations syntax

__annotate__ now delays the evaluation of annotations until
__annotations__ is referenced in the future. It also means annotations
are evaluated in a new function, rather than in the original context
where the object they were defined on was bound. There are four
operators with significant runtime side-effects that were permitted in
stock semantics, but are disallowed when
from __future__ import annotations is active, and will have to be
disallowed when this PEP is active:

-   :=
-   yield
-   yield from
-   await

Changes to inspect.get_annotations and typing.get_type_hints

(This PEP makes frequent reference to these two functions. In the future
it will refer to them collectively as "the helper functions", as they
help user code work with annotations.)

These two functions extract and return the annotations from an object.
inspect.get_annotations returns the annotations unchanged; for the
convenience of static typing users, typing.get_type_hints makes some
modifications to the annotations before it returns them.

This PEP adds a new keyword-only parameter to these two functions,
format. format specifies what format the values in the annotations dict
should be returned in. The format parameter on these two functions
accepts the same values as the format parameter on the __annotate__
magic method defined above; however, these format parameters also have a
default value of inspect.VALUE.

When either __annotations__ or __annotate__ is updated on an object, the
other of those two attributes is now out-of-date and should also either
be updated or deleted (set to None, in the case of __annotate__ which
cannot be deleted). In general, the semantics established in the
previous section ensure that this happens automatically. However,
there's one case which for all practical purposes can't be handled
automatically: when the dict cached by o.__annotations__ is itself
modified, or when mutable values inside that dict are modified.

Since this can't be handled in code, it must be handled in
documentation. This PEP proposes amending the documentation for
inspect.get_annotations (and similarly for typing.get_type_hints) as
follows:

  If you directly modify the __annotations__ dict on an object, by
  default these changes may not be reflected in the dictionary returned
  by inspect.get_annotations when requesting either SOURCE or FORWARDREF
  format on that object. Rather than modifying the __annotations__ dict
  directly, consider replacing that object's __annotate__ method with a
  function computing the annotations dict with your desired values.
  Failing that, it's best to overwrite the object's __annotate__ method
  with None to prevent inspect.get_annotations from generating stale
  results for SOURCE and FORWARDREF formats.

The stringizer and the fake globals environment

As originally proposed, this PEP supported many runtime annotation user
use cases, and many static type user use cases. But this was
insufficient--this PEP could not be accepted until it satisfied all
extant use cases. This became a longtime blocker of this PEP until Carl
Meyer proposed the "stringizer" and the "fake globals" environment as
described below. These techniques allow this PEP to support both the
FORWARDREF and SOURCE formats, ably satisfying all remaining uses cases.

In a nutshell, this technique involves running a
Python-compiler-generated __annotate__ function in an exotic runtime
environment. Its normal globals dict is replaced with what's called a
"fake globals" dict. A "fake globals" dict is a dict with one important
difference: every time you "get" a key from it that isn't mapped, it
creates, caches, and returns a new value for that key (as per the
__missing__ callback for a dictionary). That value is a an instance of a
novel type referred to as a "stringizer".

A "stringizer" is a Python class with highly unusual behavior. Every
stringizer is initialized with its "value", initially the name of the
missing key in the "fake globals" dict. The stringizer then implements
every Python "dunder" method used to implement operators, and the value
returned by that method is a new stringizer whose value is a text
representation of that operation.

When these stringizers are used in expressions, the result of the
expression is a new stringizer whose name textually represents that
expression. For example, let's say you have a variable f, which is a
reference to a stringizer initialized with the value 'f'. Here are some
examples of operations you could perform on f and the values they would
return:

    >>> f
    Stringizer('f')
    >>> f + 3
    Stringizer('f + 3')
    >> f["key"]
    Stringizer('f["key"]')

Bringing it all together: if we run a Python-generated __annotate__
function, but we replace its globals with a "fake globals" dict, all
undefined symbols it references will be replaced with stringizer proxy
objects representing those symbols, and any operations performed on
those proxies will in turn result in proxies representing that
expression. This allows __annotate__ to complete, and to return an
annotations dict, with stringizer instances standing in for names and
entire expressions that could not have otherwise been evaluated.

In practice, the "stringizer" functionality will be implemented in the
ForwardRef object currently defined in the typing module. ForwardRef
will be extended to implement all stringizer functionality; it will also
be extended to support evaluating the string it contains, to produce the
real value (assuming all symbols referenced are defined). This means the
ForwardRef object will retain references to the appropriate "globals",
"locals", and even "closure" information needed to evaluate the
expression.

This technique is the core of how inspect.get_annotations supports
FORWARDREF and SOURCE formats. Initially, inspect.get_annotations will
call the object's __annotate__ method requesting the desired format. If
that raises NotImplementedError, inspect.get_annotations will construct
a "fake globals" environment, then call the object's __annotate__
method.

-   inspect.get_annotations produces SOURCE format by creating a new
    empty "fake globals" dict, binding it to the object's __annotate__
    method, calling that requesting VALUE format, and then extracting
    the string "value" from each ForwardRef object in the resulting
    dict.
-   inspect.get_annotations produces FORWARDREF format by creating a new
    empty "fake globals" dict, pre-populating it with the current
    contents of the __annotate__ method's globals dict, binding the
    "fake globals" dict to the object's __annotate__ method, calling
    that requesting VALUE format, and returning the result.

This entire technique works because the __annotate__ functions generated
by the compiler are controlled by Python itself, and are simple and
predictable. They're effectively a single return statement, computing
and returning the annotations dict. Since most operations needed to
compute an annotation are implemented in Python using dunder methods,
and the stringizer supports all the relevant dunder methods, this
approach is a reliable, practical solution.

However, it's not reasonable to attempt this technique with just any
__annotate__ method. This PEP assumes that third-party libraries may
implement their own __annotate__ methods, and those functions would
almost certainly work incorrectly when run in this "fake globals"
environment. For that reason, this PEP allocates a flag on code objects,
one of the unused bits in co_flags, to mean "This code object can be run
in a 'fake globals' environment." This makes the "fake globals"
environment strictly opt-in, and it's expected that only __annotate__
methods generated by the Python compiler will set it.

The weakness in this technique is in handling operators which don't
directly map to dunder methods on an object. These are all operators
that implement some manner of flow control, either branching or
iteration:

-   Short-circuiting or
-   Short-circuiting and
-   Ternary operator (the if / then operator)
-   Generator expressions
-   List / dict / set comprehensions
-   Iterable unpacking

As a rule these techniques aren't used in annotations, so it doesn't
pose a problem in practice. However, the recent addition of TypeVarTuple
to Python does use iterable unpacking. The dunder methods involved
(__iter__ and __next__) don't permit distinguishing between iteration
use cases; in order to correctly detect which use case was involved,
mere "fake globals" and a "stringizer" wouldn't be sufficient; this
would require a custom bytecode interpreter designed specifically around
producing SOURCE and FORWARDREF formats.

Thankfully there's a shortcut that will work fine: the stringizer will
simply assume that when its iteration dunder methods are called, it's in
service of iterator unpacking being performed by TypeVarTuple. It will
hard-code this behavior. This means no other technique using iteration
will work, but in practice this won't inconvenience real-world use
cases.

Finally, note that the "fake globals" environment will also require
constructing a matching "fake locals" dictionary, which for FORWARDREF
format will be pre-populated with the relevant locals dict. The "fake
globals" environment will also have to create a fake "closure", a tuple
of ForwardRef objects pre-created with the names of the free variables
referenced by the __annotate__ method.

ForwardRef proxies created from __annotate__ methods that reference free
variables will map the names and closure values of those free variables
into the locals dictionary, to ensure that eval uses the correct values
for those names.

Compiler-generated __annotate__ functions

As mentioned in the previous section, the __annotate__ functions
generated by the compiler are simple. They're mainly a single return
statement, computing and returning the annotations dict.

However, the protocol for inspect.get_annotations to request either
FORWARDREF or SOURCE format requires first asking the __annotate__
method to produce it. __annotate__ methods generated by the Python
compiler won't support either of these formats and will raise
NotImplementedError().

Third-party __annotate__ functions

Third-party classes and functions will likely need to implement their
own __annotate__ methods, so that downstream users of those objects can
take full advantage of annotations. In particular, wrappers will likely
need to transform the annotation dicts produced by the wrapped object:
adding, removing, or modifying the dictionary in some way.

Most of the time, third-party code will implement their __annotate__
methods by calling inspect.get_annotations on some existing upstream
object. For example, wrappers will likely request the annotations dict
for their wrapped object, in the format that was requested from them,
then modify the returned annotations dict as appropriate and return
that. This allows third-party code to leverage the "fake globals"
technique without having to understand or participate in it.

Third-party libraries that support both pre- and post-PEP-649 versions
of Python will have to innovate their own best practices on how to
support both. One sensible approach would be for their wrapper to always
support __annotate__, then call it requesting VALUE format and store the
result as the __annotations__ on their wrapper object. This would
support pre-649 Python semantics, and be forward-compatible with
post-649 semantics.

Pseudocode

Here's high-level pseudocode for inspect.get_annotations:

    def get_annotations(o, format):
        if format == VALUE:
            return dict(o.__annotations__)

        if format == FORWARDREF:
            try:
                return dict(o.__annotations__)
            except NameError:
                pass

        if not hasattr(o.__annotate__):
            return {}

        c_a = o.__annotate__
        try:
            return c_a(format)
        except NotImplementedError:
            if not can_be_called_with_fake_globals(c_a):
                return {}
            c_a_with_fake_globals = make_fake_globals_version(c_a, format)
            return c_a_with_fake_globals(VALUE)

Here's what a Python compiler-generated __annotate__ method might look
like if it was written in Python:

    def __annotate__(self, format):
        if format != 1:
            raise NotImplementedError()
        return { ... }

Here's how a third-party wrapper class might implement __annotate__. In
this example, the wrapper works like functools.partial, pre-binding one
parameter of the wrapped callable, which for simplicity must be named
arg:

    def __annotate__(self, format):
        ann = inspect.get_annotations(self.wrapped_fn, format)
        if 'arg' in ann:
            del ann['arg']
        return ann

Other modifications to the Python runtime

This PEP does not dictate exactly how it should be implemented; that is
left up to the language implementation maintainers. However, the best
implementation of this PEP may require adding additional information to
existing Python objects, which is implicitly condoned by the acceptance
of this PEP.

For example, it may be necessary to add a __globals__ attribute to class
objects, so that the __annotate__ function for that class can be lazily
bound, only on demand. Also, __annotate__ functions defined on methods
defined in a class may need to retain a reference to the class's
__dict__, in order to correctly evaluate names bound in that class. It's
expected that the CPython implementation of this PEP will include both
those new attributes.

All such new information added to existing Python objects should be done
with "dunder" attributes, as they will of course be implementation
details.

Interactive REPL Shell

The semantics established in this PEP also hold true when executing code
in Python's interactive REPL shell, except for module annotations in the
interactive module (__main__) itself. Since that module is never
"finished", there's no specific point where we can compile the
__annotate__ function.

For the sake of simplicity, in this case we forego delayed evaluation.
Module-level annotations in the REPL shell will continue to work exactly
as they do with "stock semantics", evaluating immediately and setting
the result directly inside the __annotations__ dict.

Annotations On Local Variables Inside Functions

Python supports syntax for local variable annotations inside functions.
However, these annotations have no runtime effect--they're discarded at
compile-time. Therefore, this PEP doesn't need to do anything to support
them, the same as stock semantics and PEP 563.

Prototype

The original prototype implementation of this PEP can be found here:

https://github.com/larryhastings/co_annotations/

As of this writing, the implementation is severely out of date; it's
based on Python 3.10 and implements the semantics of the first draft of
this PEP, from early 2021. It will be updated shortly.

Performance Comparison

Performance with this PEP is generally favorable. There are four
scenarios to consider:

-   the runtime cost when annotations aren't defined,
-   the runtime cost when annotations are defined but not referenced,
    and
-   the runtime cost when annotations are defined and referenced as
    objects.
-   the runtime cost when annotations are defined and referenced as
    strings.

We'll examine each of these scenarios in the context of all three
semantics for annotations: stock, PEP 563, and this PEP.

When there are no annotations, all three semantics have the same runtime
cost: zero. No annotations dict is created and no code is generated for
it. This requires no runtime processor time and consumes no memory.

When annotations are defined but not referenced, the runtime cost of
Python with this PEP is roughly the same as PEP 563, and improved over
stock. The specifics depend on the object being annotated:

-   With stock semantics, the annotations dict is always built, and set
    as an attribute of the object being annotated.
-   In PEP 563 semantics, for function objects, a precompiled constant
    (a specially constructed tuple) is set as an attribute of the
    function. For class and module objects, the annotations dict is
    always built and set as an attribute of the class or module.
-   With this PEP, a single object is set as an attribute of the object
    being annotated. Most of the time, this object is a constant (a code
    object), but when the annotations require a class namespace or
    closure, this object will be a tuple constructed at binding time.

When annotations are both defined and referenced as objects, code using
this PEP should be much faster than PEP 563, and be as fast or faster
than stock. PEP 563 semantics requires invoking eval() for every value
inside an annotations dict which is enormously slow. And the
implementation of this PEP generates measurably more efficient bytecode
for class and module annotations than stock semantics; for function
annotations, this PEP and stock semantics should be about the same
speed.

The one case where this PEP will be noticeably slower than PEP 563 is
when annotations are requested as strings; it's hard to beat "they are
already strings." But stringified annotations are intended for online
documentation use cases, where performance is less likely to be a key
factor.

Memory use should also be comparable in all three scenarios across all
three semantic contexts. In the first and third scenarios, memory usage
should be roughly equivalent in all cases. In the second scenario, when
annotations are defined but not referenced, using this PEP's semantics
will mean the function/class/module will store one unused code object
(possibly bound to an unused function object); with the other two
semantics, they'll store one unused dictionary or constant tuple.

Backwards Compatibility

Backwards Compatibility With Stock Semantics

This PEP preserves nearly all existing behavior of annotations from
stock semantics:

-   The format of the annotations dict stored in the __annotations__
    attribute is unchanged. Annotations dicts contain real values, not
    strings as per PEP 563.
-   Annotations dicts are mutable, and any changes to them are
    preserved.
-   The __annotations__ attribute can be explicitly set, and any legal
    value set this way will be preserved.
-   The __annotations__ attribute can be deleted using the del
    statement.

Most code that works with stock semantics should continue to work when
this PEP is active without any modification necessary. But there are
exceptions, as follows.

First, there's a well-known idiom for accessing class annotations which
may not work correctly when this PEP is active. The original
implementation of class annotations had what can only be called a bug:
if a class didn't define any annotations of its own, but one of its base
classes did define annotations, the class would "inherit" those
annotations. This behavior was never desirable, so user code found a
workaround: instead of accessing the annotations on the class directly
via cls.__annotations__, code would access the class's annotations via
its dict as in cls.__dict__.get("__annotations__", {}). This idiom
worked because classes stored their annotations in their __dict__, and
accessing them this way avoided the lookups in the base classes. The
technique relied on implementation details of CPython, so it was never
supported behavior--though it was necessary. However, when this PEP is
active, a class may have annotations defined but hasn't yet called
__annotate__ and cached the result, in which case this approach would
lead to mistakenly assuming the class didn't have annotations. In any
case, the bug was fixed as of Python 3.10, and the idiom should no
longer be used. Also as of Python 3.10, there's an Annotations HOWTO
that defines best practices for working with annotations; code that
follows these guidelines will work correctly even when this PEP is
active, because it suggests using different approaches to get
annotations from class objects based on the Python version the code runs
under.

Since delaying the evaluation of annotations until they are introspected
changes the semantics of the language, it's observable from within the
language. Therefore it's possible to write code that behaves differently
based on whether annotations are evaluated at binding time or at access
time, e.g.

    mytype = str
    def foo(a:mytype): pass
    mytype = int
    print(foo.__annotations__['a'])

This will print <class 'str'> with stock semantics and <class 'int'>
when this PEP is active. This is therefore a backwards-incompatible
change. However, this example is poor programming style, so this change
seems acceptable.

There are two uncommon interactions possible with class and module
annotations that work with stock semantics that would no longer work
when this PEP was active. These two interactions would have to be
prohibited. The good news is, neither is common, and neither is
considered good practice. In fact, they're rarely seen outside of
Python's own regression test suite. They are:

-   Code that sets annotations on module or class attributes from inside
    any kind of flow control statement. It's currently possible to set
    module and class attributes with annotations inside an if or try
    statement, and it works as one would expect. It's untenable to
    support this behavior when this PEP is active.
-   Code in module or class scope that references or modifies the local
    __annotations__ dict directly. Currently, when setting annotations
    on module or class attributes, the generated code simply creates a
    local __annotations__ dict, then adds mappings to it as needed. It's
    possible for user code to directly modify this dict, though this
    doesn't seem to be an intentional feature. Although it would be
    possible to support this after a fashion once this PEP was active,
    the semantics would likely be surprising and wouldn't make anyone
    happy.

Note that these are both also pain points for static type checkers, and
are unsupported by those tools. It seems reasonable to declare that both
are at the very least unsupported, and their use results in undefined
behavior. It might be worth making a small effort to explicitly prohibit
them with compile-time checks.

Finally, if this PEP is active, annotation values shouldn't use the
if / else ternary operator. Although this will work correctly when
accessing o.__annotations__ or requesting inspect.VALUE from a helper
function, the boolean expression may not compute correctly with
inspect.FORWARDREF when some names are defined, and would be far less
correct with inspect.SOURCE.

Backwards Compatibility With PEP 563 Semantics

PEP 563 changed the semantics of annotations. When its semantics are
active, annotations must assume they will be evaluated in module-level
or class-level scope. They may no longer refer directly to local
variables in the current function or an enclosing function. This PEP
removes that restriction, and annotations may refer any local variable.

PEP 563 requires using eval (or a helper function like
typing.get_type_hints or inspect.get_annotations that uses eval for you)
to convert stringized annotations into their "real" values. Existing
code that activates stringized annotations, and calls eval() directly to
convert the strings back into real values, can simply remove the eval()
call. Existing code using a helper function would continue to work
unchanged, though use of those functions may become optional.

Static typing users often have modules that only contain inert type hint
definitions--but no live code. These modules are only needed when
running static type checking; they aren't used at runtime. But under
stock semantics, these modules have to be imported in order for the
runtime to evaluate and compute the annotations. Meanwhile, these
modules often caused circular import problems that could be difficult or
even impossible to solve. PEP 563 allowed users to solve these circular
import problems by doing two things. First, they activated PEP 563 in
their modules, which meant annotations were constant strings, and didn't
require the real symbols to be defined in order for the annotations to
be computable. Second, this permitted users to only import the
problematic modules in an if typing.TYPE_CHECKING block. This allowed
the static type checkers to import the modules and the type definitions
inside, but they wouldn't be imported at runtime. So far, this approach
will work unchanged when this PEP is active; if typing.TYPE_CHECKING is
supported behavior.

However, some codebases actually did examine their annotations at
runtime, even when using the if typing.TYPE_CHECKING technique and not
importing definitions used in their annotations. These codebases
examined the annotation strings without evaluating them, instead relying
on identity checks or simple lexical analysis on the strings.

This PEP supports these techniques too. But users will need to port
their code to it. First, user code will need to use
inspect.get_annotations or typing.get_type_hints to access the
annotations; they won't be able to simply get the __annotations__
attribute from their object. Second, they will need to specify either
inspect.FORWARDREF or inspect.SOURCE for the format when calling that
function. This means the helper function can succeed in producing the
annotations dict, even when not all the symbols are defined. Code
expecting stringized annotations should work unmodified with
inspect.SOURCE formatted annotations dicts; however, users should
consider switching to inspect.FORWARDREF, as it may make their analysis
easier.

Similarly, PEP 563 permitted use of class decorators on annotated
classes in a way that hadn't previously been possible. Some class
decorators (e.g. dataclasses) examine the annotations on the class.
Because class decorators using the @ decorator syntax are run before the
class name is bound, they can cause unsolvable circular-definition
problems. If you annotate attributes of a class with references to the
class itself, or annotate attributes in multiple classes with circular
references to each other, you can't decorate those classes with the @
decorator syntax using decorators that examine the annotations. PEP 563
allowed this to work, as long as the decorators examined the strings
lexically and didn't use eval to evaluate them (or handled the NameError
with further workarounds). When this PEP is active, decorators will be
able to compute the annotations dict in inspect.SOURCE or
inspect.FORWARDREF format using the helper functions. This will permit
them to analyze annotations containing undefined symbols, in the format
they prefer.

Early adopters of PEP 563 discovered that "stringized" annotations were
useful for automatically-generated documentation. Users experimented
with this use case, and Python's pydoc has expressed some interest in
this technique. This PEP supports this use case; the code generating the
documentation will have to be updated to use a helper function to access
the annotations in inspect.SOURCE format.

Finally, the warnings about using the if / else ternary operator in
annotations apply equally to users of PEP 563. It currently works for
them, but could produce incorrect results when requesting some formats
from the helper functions.

If this PEP is accepted, PEP 563 will be deprecated and eventually
removed. To facilitate this transition for early adopters of PEP 563,
who now depend on its semantics, inspect.get_annotations and
typing.get_type_hints will implement a special affordance.

The Python compiler won't generate annotation code objects for objects
defined in a module where PEP 563 semantics are active, even if this PEP
is accepted. So, under normal circumstances, requesting inspect.SOURCE
format from a helper function would return an empty dict. As an
affordance, to facilitate the transition, if the helper functions detect
that an object was defined in a module with PEP 563 active, and the user
requests inspect.SOURCE format, they'll return the current value of the
__annotations__ dict, which in this case will be the stringized
annotations. This will allow PEP 563 users who lexically analyze
stringized annotations to immediately change over to requesting
inspect.SOURCE format from the helper functions, which will hopefully
smooth their transition away from PEP 563.

Rejected Ideas

"Just store the strings"

One proposed idea for supporting SOURCE format was for the Python
compiler to emit the actual source code for the annotation values
somewhere, and to furnish that when the user requested SOURCE format.

This idea wasn't rejected so much as categorized as "not yet". We
already know we need to support FORWARDREF format, and that technique
can be adapted to support SOURCE format in just a few lines. There are
many unanswered questions about this approach:

-   Where would we store the strings? Would they always be loaded when
    the annotated object was created, or would they be lazy-loaded on
    demand? If so, how would the lazy-loading work?
-   Would the "source code" include the newlines and comments of the
    original? Would it preserve all whitespace, including indents and
    extra spaces used purely for formatting?

It's possible we'll revisit this topic in the future, if improving the
fidelity of SOURCE values to the original source code is judged
sufficiently important.

Acknowledgements

Thanks to Carl Meyer, Barry Warsaw, Eric V. Smith, Mark Shannon, Jelle
Ziljstra, and Guido van Rossum for ongoing feedback and encouragement.

Particular thanks to several individuals who contributed key ideas that
became some of the best aspects of this proposal:

-   Carl Meyer suggested the "stringizer" technique that made FORWARDREF
    and SOURCE formats possible, which allowed making forward progress
    on this PEP possible after a year of languishing due to
    seemingly-unfixable problems. He also suggested the affordance for
    PEP 563 users where inspect.SOURCE will return the stringized
    annotations, and many more suggestions besides. Carl was also the
    primary correspondent in private email threads discussing this PEP,
    and was a tireless resource and voice of sanity. This PEP would
    almost certainly not have been accepted it were it not for Carl's
    contributions.
-   Mark Shannon suggested building the entire annotations dict inside a
    single code object, and only binding it to a function on demand.
-   Guido van Rossum suggested that __annotate__ functions should
    duplicate the name visibility rules of annotations under "stock"
    semantics.
-   Jelle Zijlstra contributed not only feedback--but code!

References

-   https://github.com/larryhastings/co_annotations/issues
-   https://discuss.python.org/t/two-polls-on-how-to-revise-pep-649/23628
-   https://discuss.python.org/t/a-massive-pep-649-update-with-some-major-course-corrections/25672

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 539 Title: A New C-API for Thread-Local Storage in CPython Version:
$Revision$ Last-Modified: $Date$ Author: Erik M. Bray, Masayuki Yamamoto
BDFL-Delegate: Alyssa Coghlan Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 20-Dec-2016 Python-Version: 3.7
Post-History: 16-Dec-2016, 31-Aug-2017, 08-Sep-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-September/149358.html

Abstract

The proposal is to add a new Thread Local Storage (TLS) API to CPython
which would supersede use of the existing TLS API within the CPython
interpreter, while deprecating the existing API. The new API is named
the "Thread Specific Storage (TSS) API" (see Rationale for Proposed
Solution for the origin of the name).

Because the existing TLS API is only used internally (it is not
mentioned in the documentation, and the header that defines it,
pythread.h, is not included in Python.h either directly or indirectly),
this proposal probably only affects CPython, but might also affect other
interpreter implementations (PyPy?) that implement parts of the CPython
API.

This is motivated primarily by the fact that the old API uses int to
represent TLS keys across all platforms, which is neither
POSIX-compliant, nor portable in any practical sense[1].

Note

Throughout this document the acronym "TLS" refers to Thread Local
Storage and should not be confused with "Transportation Layer Security"
protocols.

Specification

The current API for TLS used inside the CPython interpreter consists of
6 functions:

    PyAPI_FUNC(int) PyThread_create_key(void)
    PyAPI_FUNC(void) PyThread_delete_key(int key)
    PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value)
    PyAPI_FUNC(void *) PyThread_get_key_value(int key)
    PyAPI_FUNC(void) PyThread_delete_key_value(int key)
    PyAPI_FUNC(void) PyThread_ReInitTLS(void)

These would be superseded by a new set of analogous functions:

    PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key)
    PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t *key)
    PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value)
    PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key)

The specification also adds a few new features:

-   A new type Py_tss_t--an opaque type the definition of which may
    depend on the underlying TLS implementation. It is defined:

        typedef struct {
            int _is_initialized;
            NATIVE_TSS_KEY_T _key;
        } Py_tss_t;

    where NATIVE_TSS_KEY_T is a macro whose value depends on the
    underlying native TLS implementation (e.g. pthread_key_t).

-   An initializer for Py_tss_t variables, Py_tss_NEEDS_INIT.

-   Three new functions:

        PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void)
        PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key)
        PyAPI_FUNC(int) PyThread_tss_is_created(Py_tss_t *key)

    The first two are needed for dynamic (de-)allocation of a Py_tss_t,
    particularly in extension modules built with Py_LIMITED_API, where
    static allocation of this type is not possible due to its
    implementation being opaque at build time. A value returned by
    PyThread_tss_alloc is in the same state as a value initialized with
    Py_tss_NEEDS_INIT, or NULL in the case of dynamic allocation
    failure. The behavior of PyThread_tss_free involves calling
    PyThread_tss_delete preventively, or is a no-op if the value pointed
    to by the key argument is NULL. PyThread_tss_is_created returns
    non-zero if the given Py_tss_t has been initialized (i.e. by
    PyThread_tss_create).

The new TSS API does not provide functions which correspond to
PyThread_delete_key_value and PyThread_ReInitTLS, because these
functions were needed only for CPython's now defunct built-in TLS
implementation; that is the existing behavior of these functions is
treated as follows: PyThread_delete_key_value(key) is equivalent to
PyThread_set_key_value(key, NULL), and PyThread_ReInitTLS() is a
no-op[2].

The new PyThread_tss_ functions are almost exactly analogous to their
original counterparts with a few minor differences: Whereas
PyThread_create_key takes no arguments and returns a TLS key as an int,
PyThread_tss_create takes a Py_tss_t* as an argument and returns an int
status code. The behavior of PyThread_tss_create is undefined if the
value pointed to by the key argument is not initialized by
Py_tss_NEEDS_INIT. The returned status code is zero on success and
non-zero on failure. The meanings of non-zero status codes are not
otherwise defined by this specification.

Similarly the other PyThread_tss_ functions are passed a Py_tss_t*
whereas previously the key was passed by value. This change is
necessary, as being an opaque type, the Py_tss_t type could
hypothetically be almost any size. This is especially necessary for
extension modules built with Py_LIMITED_API, where the size of the type
is not known. Except for PyThread_tss_free, the behaviors of
PyThread_tss_ are undefined if the value pointed to by the key argument
is NULL.

Moreover, because of the use of Py_tss_t instead of int, there are
behaviors in the new API which differ from the existing API with regard
to key creation and deletion. PyThread_tss_create can be called
repeatedly on the same key--calling it on an already initialized key is
a no-op and immediately returns success. Similarly for calling
PyThread_tss_delete with an uninitialized key.

The behavior of PyThread_tss_delete is defined to change the key's
initialization state to "uninitialized"--this allows, for example,
statically allocated keys to be reset to a sensible state when
restarting the CPython interpreter without terminating the process (e.g.
embedding Python in an application)[3].

The old PyThread_*_key* functions will be marked as deprecated in the
documentation, but will not generate runtime deprecation warnings.

Additionally, on platforms where sizeof(pthread_key_t) != sizeof(int),
PyThread_create_key will return immediately with a failure status, and
the other TLS functions will all be no-ops on such platforms.

Comparison of API Specification

+-------------------+-----------------------+-----------------------+
| API               | Thread Local Storage  | Thread Specific       |
|                   | (TLS)                 | Storage (TSS)         |
+===================+=======================+=======================+
| Version           | Existing              | New                   |
+-------------------+-----------------------+-----------------------+
| Key Type          | int                   | Py_tss_t (opaque      |
|                   |                       | type)                 |
+-------------------+-----------------------+-----------------------+
| Handle Native Key | cast to int           | conceal into internal |
|                   |                       | field                 |
+-------------------+-----------------------+-----------------------+
| Function Argument | int                   | Py_tss_t *            |
+-------------------+-----------------------+-----------------------+
| Features          | -   create key        | -   create key        |
|                   | -   delete key        | -   delete key        |
|                   | -   set value         | -   set value         |
|                   | -   get value         | -   get value         |
|                   | -   delete value      | -   (set NULL         |
|                   | -   reinitialize keys |     instead)[7]       |
|                   |     (after fork)      | -   (unnecessary)[8]  |
|                   |                       | -   dynamically       |
|                   |                       |     (de-)allocate key |
|                   |                       | -   check key's       |
|                   |                       |     initialization    |
|                   |                       |     state             |
+-------------------+-----------------------+-----------------------+
| Key Initializer   | (-1 as key creation   | Py_tss_NEEDS_INIT     |
|                   | failure)              |                       |
+-------------------+-----------------------+-----------------------+
| Requirement       | native threads (since | native threads        |
|                   | CPython 3.7[9])       |                       |
+-------------------+-----------------------+-----------------------+
| Restriction       | No support for        | Unable to statically  |
|                   | platforms where       | allocate keys when    |
|                   | native TLS key is     | Py_LIMITED_API is     |
|                   | defined in a way that | defined.              |
|                   | cannot be safely cast |                       |
|                   | to int.               |                       |
+-------------------+-----------------------+-----------------------+

Example

With the proposed changes, a TSS key is initialized like:

    static Py_tss_t tss_key = Py_tss_NEEDS_INIT;
    if (PyThread_tss_create(&tss_key)) {
        /* ... handle key creation failure ... */
    }

The initialization state of the key can then be checked like:

    assert(PyThread_tss_is_created(&tss_key));

The rest of the API is used analogously to the old API:

    int the_value = 1;
    if (PyThread_tss_get(&tss_key) == NULL) {
        PyThread_tss_set(&tss_key, (void *)&the_value);
        assert(PyThread_tss_get(&tss_key) != NULL);
    }
    /* ... once done with the key ... */
    PyThread_tss_delete(&tss_key);
    assert(!PyThread_tss_is_created(&tss_key));

When Py_LIMITED_API is defined, a TSS key must be dynamically allocated:

    static Py_tss_t *ptr_key = PyThread_tss_alloc();
    if (ptr_key == NULL) {
        /* ... handle key allocation failure ... */
    }
    assert(!PyThread_tss_is_created(ptr_key));
    /* ... once done with the key ... */
    PyThread_tss_free(ptr_key);
    ptr_key = NULL;

Platform Support Changes

A new "Native Thread Implementation" section will be added to PEP 11
that states:

-   As of CPython 3.7, all platforms are required to provide a native
    thread implementation (such as pthreads or Windows) to implement the
    TSS API. Any TSS API problems that occur in an implementation
    without native threads will be closed as "won't fix".

Motivation

The primary problem at issue here is the type of the keys (int) used for
TLS values, as defined by the original PyThread TLS API.

The original TLS API was added to Python by GvR back in 1997, and at the
time the key used to represent a TLS value was an int, and so it has
been to the time of writing. This used CPython's own TLS implementation
which long remained unused, largely unchanged, in Python/thread.c.
Support for implementation of the API on top of native thread
implementations (pthreads and Windows) was added much later, and the
built-in implementation has been deemed no longer necessary and has
since been removed[10].

The problem with the choice of int to represent a TLS key, is that while
it was fine for CPython's own TLS implementation, and happens to be
compatible with Windows (which uses DWORD for the analogous data), it is
not compatible with the POSIX standard for the pthreads API, which
defines pthread_key_t as an opaque type not further defined by the
standard (as with Py_tss_t described above)[11]. This leaves it up to
the underlying implementation how a pthread_key_t value is used to look
up thread-specific data.

This has not generally been a problem for Python's API, as it just
happens that on Linux pthread_key_t is defined as an unsigned int, and
so is fully compatible with Python's TLS API--pthread_key_t's created by
pthread_create_key can be freely cast to int and back (well, not
exactly, even this has some limitations as pointed out by issue #22206).

However, as issue #25658 points out, there are at least some platforms
(namely Cygwin, CloudABI, but likely others as well) which have
otherwise modern and POSIX-compliant pthreads implementations, but are
not compatible with Python's API because their pthread_key_t is defined
in a way that cannot be safely cast to int. In fact, the possibility of
running into this problem was raised by MvL at the time pthreads TLS was
added[12].

It could be argued that PEP 11 makes specific requirements for
supporting a new, not otherwise officially-support platform (such as
CloudABI), and that the status of Cygwin support is currently dubious.
However, this creates a very high barrier to supporting platforms that
are otherwise Linux- and/or POSIX-compatible and where CPython might
otherwise "just work" except for this one hurdle. CPython itself imposes
this implementation barrier by way of an API that is not compatible with
POSIX (and in fact makes invalid assumptions about pthreads).

Rationale for Proposed Solution

The use of an opaque type (Py_tss_t) to key TLS values allows the API to
be compatible with all present (POSIX and Windows) and future (C11?)
native TLS implementations supported by CPython, as it allows the
definition of Py_tss_t to depend on the underlying implementation.

Since the existing TLS API has been available in the limited API[13] for
some platforms (e.g. Linux), CPython makes an effort to provide the new
TSS API at that level likewise. Note, however, that the Py_tss_t
definition becomes to be an opaque struct when Py_LIMITED_API is
defined, because exposing NATIVE_TSS_KEY_T as part of the limited API
would prevent us from switching native thread implementation without
rebuilding extension modules.

A new API must be introduced, rather than changing the function
signatures of the current API, in order to maintain backwards
compatibility. The new API also more clearly groups together these
related functions under a single name prefix, PyThread_tss_. The "tss"
in the name stands for "thread-specific storage", and was influenced by
the naming and design of the "tss" API that is part of the C11 threads
API[14]. However, this is in no way meant to imply compatibility with or
support for the C11 threads API, or signal any future intention of
supporting C11--it's just the influence for the naming and design.

The inclusion of the special initializer Py_tss_NEEDS_INIT is required
by the fact that not all native TLS implementations define a sentinel
value for uninitialized TLS keys. For example, on Windows a TLS key is
represented by a DWORD (unsigned int) and its value must be treated as
opaque[15]. So there is no unsigned integer value that can be safely
used to represent an uninitialized TLS key on Windows. Likewise, POSIX
does not specify a sentinel for an uninitialized pthread_key_t, instead
relying on the pthread_once interface to ensure that a given TLS key is
initialized only once per-process. Therefore, the Py_tss_t type contains
an explicit ._is_initialized that can indicate the key's initialization
state independent of the underlying implementation.

Changing PyThread_create_key to immediately return a failure status on
systems using pthreads where sizeof(int) != sizeof(pthread_key_t) is
intended as a sanity check: Currently, PyThread_create_key may report
initial success on such systems, but attempts to use the returned key
are likely to fail. Although in practice this failure occurs earlier in
the interpreter initialization, it's better to fail immediately at the
source of problem (PyThread_create_key) rather than sometime later when
use of an invalid key is attempted. In other words, this indicates
clearly that the old API is not supported on platforms where it cannot
be used reliably, and that no effort will be made to add such support.

Rejected Ideas

-   Do nothing: The status quo is fine because it works on Linux, and
    platforms wishing to be supported by CPython should follow the
    requirements of PEP 11. As explained above, while this would be a
    fair argument if CPython were being to asked to make changes to
    support particular quirks or features of a specific platform, in
    this case it is a quirk of CPython that prevents it from being used
    to its full potential on otherwise POSIX-compliant platforms. The
    fact that the current implementation happens to work on Linux is a
    happy accident, and there's no guarantee that this will never
    change.
-   Affected platforms should just configure Python --without-threads:
    this is no longer an option as the --without-threads option has been
    removed for Python 3.7[16].
-   Affected platforms should use CPython's built-in TLS implementation
    instead of a native TLS implementation: This is a more acceptable
    alternative to the previous idea, and in fact there had been a patch
    to do just that[17]. However, the built-in implementation being
    "slower and clunkier" in general than native implementations still
    needlessly hobbles performance on affected platforms. At least one
    other module (tracemalloc) is also broken if Python is built without
    a native TLS implementation. This idea also cannot be adopted
    because the built-in implementation has since been removed.
-   Keep the existing API, but work around the issue by providing a
    mapping from pthread_key_t values to int values. A couple attempts
    were made at this ([18],[19]), but this injects needless complexity
    and overhead into performance-critical code on platforms that are
    not currently affected by this issue (such as Linux). Even if use of
    this workaround were made conditional on platform compatibility, it
    introduces platform-specific code to maintain, and still has the
    problem of the previous rejected ideas of needlessly hobbling
    performance on affected platforms.

Implementation

An initial version of a patch[20] is available on the bug tracker for
this issue. Since the migration to GitHub, its development has continued
in the pep539-tss-api feature branch[21] in Masayuki Yamamoto's fork of
the CPython repository on GitHub. A work-in-progress PR is available
at[22].

This reference implementation covers not only the new API implementation
features, but also the client code updates needed to replace the
existing TLS API with the new TSS API.

Copyright

This document has been placed in the public domain.

References and Footnotes

[1] http://bugs.python.org/issue25658

[2] https://bugs.python.org/msg298342

[3] https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx

[4] https://bugs.python.org/msg298342

[5] https://bugs.python.org/msg298342

[6] http://bugs.python.org/issue30832

[7] https://bugs.python.org/msg298342

[8] https://bugs.python.org/msg298342

[9] http://bugs.python.org/issue30832

[10] http://bugs.python.org/issue30832

[11] http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_key_create.html

[12] https://bugs.python.org/msg116292

[13] It is also called as "stable ABI" (PEP 384)

[14] http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf#page=404

[15] https://msdn.microsoft.com/en-us/library/windows/desktop/ms686801(v=vs.85).aspx

[16] https://bugs.python.org/issue31370

[17] http://bugs.python.org/file45548/configure-pthread_key_t.patch

[18] http://bugs.python.org/file44269/issue25658-1.patch

[19] http://bugs.python.org/file44303/key-constant-time.diff

[20] http://bugs.python.org/file46379/pythread-tss-3.patch

[21] https://github.com/python/cpython/compare/master...ma8ma:pep539-tss-api

[22] https://github.com/python/cpython/pull/1362

license: Public Domain

authors:
Erik M. Bray
Masayuki Yamamoto


text:
PEP: 265 Title: Sorting Dictionaries by Value Author: Grant Griffin
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type:
text/x-rst Created: 08-Aug-2001 Python-Version: 2.2 Post-History:

Abstract

This PEP suggests a "sort by value" operation for dictionaries. The
primary benefit would be in terms of "batteries included" support for a
common Python idiom which, in its current form, is both difficult for
beginners to understand and cumbersome for all to implement.

BDFL Pronouncement

This PEP is rejected because the need for it has been largely fulfilled
by Py2.4's sorted() builtin function:

    >>> sorted(d.iteritems(), key=itemgetter(1), reverse=True)
    [('b', 23), ('d', 17), ('c', 5), ('a', 2), ('e', 1)]

or for just the keys:

    sorted(d, key=d.__getitem__, reverse=True)
    ['b', 'd', 'c', 'a', 'e']

Also, Python 2.5's heapq.nlargest() function addresses the common use
case of finding only a few of the highest valued items:

    >>> nlargest(2, d.iteritems(), itemgetter(1))
    [('b', 23), ('d', 17)]

Motivation

A common use of dictionaries is to count occurrences by setting the
value of d[key] to 1 on its first occurrence, then increment the value
on each subsequent occurrence. This can be done several different ways,
but the get() method is the most succinct:

    d[key] = d.get(key, 0) + 1

Once all occurrences have been counted, a common use of the resulting
dictionary is to print the occurrences in occurrence-sorted order, often
with the largest value first.

This leads to a need to sort a dictionary's items by value. The
canonical method of doing so in Python is to first use d.items() to get
a list of the dictionary's items, then invert the ordering of each
item's tuple from (key, value) into (value, key), then sort the list;
since Python sorts the list based on the first item of the tuple, the
list of (inverted) items is therefore sorted by value. If desired, the
list can then be reversed, and the tuples can be re-inverted back to
(key, value). (However, in my experience, the inverted tuple ordering is
fine for most purposes, e.g. printing out the list.)

For example, given an occurrence count of:

    >>> d = {'a':2, 'b':23, 'c':5, 'd':17, 'e':1}

we might do:

    >>> items = [(v, k) for k, v in d.items()]
    >>> items.sort()
    >>> items.reverse()             # so largest is first
    >>> items = [(k, v) for v, k in items]

resulting in:

    >>> items
    [('b', 23), ('d', 17), ('c', 5), ('a', 2), ('e', 1)]

which shows the list in by-value order, largest first. (In this case,
'b' was found to have the most occurrences.)

This works fine, but is "hard to use" in two aspects. First, although
this idiom is known to veteran Pythoneers, it is not at all obvious to
newbies -- either in terms of its algorithm (inverting the ordering of
item tuples) or its implementation (using list comprehensions -- which
are an advanced Python feature.) Second, it requires having to
repeatedly type a lot of "grunge", resulting in both tedium and
mistakes.

We therefore would rather Python provide a method of sorting
dictionaries by value which would be both easy for newbies to understand
(or, better yet, not to have to understand) and easier for all to use.

Rationale

As Tim Peters has pointed out, this sort of thing brings on the problem
of trying to be all things to all people. Therefore, we will limit its
scope to try to hit "the sweet spot". Unusual cases (e.g. sorting via a
custom comparison function) can, of course, be handled "manually" using
present methods.

Here are some simple possibilities:

The items() method of dictionaries can be augmented with new parameters
having default values that provide for full backwards-compatibility:

    (1) items(sort_by_values=0, reversed=0)

or maybe just:

    (2) items(sort_by_values=0)

since reversing a list is easy enough.

Alternatively, items() could simply let us control the (key, value)
order:

    (3) items(values_first=0)

Again, this is fully backwards-compatible. It does less work than the
others, but it at least eases the most complicated/tricky part of the
sort-by-value problem: inverting the order of item tuples. Using this is
very simple:

    items = d.items(1)
    items.sort()
    items.reverse()         # (if desired)

The primary drawback of the preceding three approaches is the additional
overhead for the parameter-less items() case, due to having to process
default parameters. (However, if one assumes that items() gets used
primarily for creating sort-by-value lists, this is not really a
drawback in practice.)

Alternatively, we might add a new dictionary method which somehow
embodies "sorting". This approach offers two advantages. First, it
avoids adding overhead to the items() method. Second, it is perhaps more
accessible to newbies: when they go looking for a method for sorting
dictionaries, they hopefully run into this one, and they will not have
to understand the finer points of tuple inversion and list sorting to
achieve sort-by-value.

To allow the four basic possibilities of sorting by key/value and in
forward/reverse order, we could add this method:

    (4) sorted_items(by_value=0, reversed=0)

I believe the most common case would actually be by_value=1, reversed=1,
but the defaults values given here might lead to fewer surprises by
users: sorted_items() would be the same as items() followed by sort().

Finally (as a last resort), we could use:

    (5) items_sorted_by_value(reversed=0)

Implementation

The proposed dictionary methods would necessarily be implemented in C.
Presumably, the implementation would be fairly simple since it involves
just adding a few calls to Python's existing machinery.

Concerns

Aside from the run-time overhead already addressed in possibilities 1
through 3, concerns with this proposal probably will fall into the
categories of "feature bloat" and/or "code bloat". However, I believe
that several of the suggestions made here will result in quite minimal
bloat, resulting in a good tradeoff between bloat and "value added".

Tim Peters has noted that implementing this in C might not be
significantly faster than implementing it in Python today. However, the
major benefits intended here are "accessibility" and "ease of use", not
"speed". Therefore, as long as it is not noticeably slower (in the case
of plain items(), speed need not be a consideration.

References

A related thread called "counting occurrences" appeared on
comp.lang.python in August, 2001. This included examples of approaches
to systematizing the sort-by-value problem by implementing it as
reusable Python functions and classes.

Copyright

This document has been placed in the public domain.


text:
PEP: 746 Title: Type checking Annotated metadata Author: Adrian Garcia
Badaracco<EMAIL_ADDRESS>Sponsor: Jelle Zijlstra
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-746-typedmetadata-for-type-checking-of-pep-593-annotated/53834
Status: Draft Type: Standards Track Topic: Typing Created: 20-May-2024
Python-Version: 3.14 Post-History: 20-May-2024

Abstract

This PEP proposes a mechanism for type checking metadata that uses the
typing.Annotated type. Metadata objects that implement the new
__supports_annotated_base__ protocol will be type checked by static type
checkers to ensure that the metadata is valid for the given type.

Motivation

PEP 593 introduced Annotated as a way to attach runtime metadata to
types. In general, the metadata is not meant for static type checkers,
but even so, it is often useful to be able to check that the metadata
makes sense for the given type.

Take the first example in PEP 593, which uses Annotated to attach
serialization information to a field:

    class Student(struct2.Packed):
        name: Annotated[str, struct2.ctype("<10s")]

Here, the struct2.ctype("<10s") metadata is meant to be used by a
serialization library to serialize the field. Such libraries can only
serialize a subset of types: it would not make sense to write, for
example, Annotated[list[str], struct2.ctype("<10s")]. Yet the type
system provides no way to enforce this. The metadata are completely
ignored by type checkers.

This use case comes up in libraries like pydantic and msgspec, which use
Annotated to attach validation and conversion information to fields or
fastapi, which uses Annotated to mark parameters as extracted from
headers, query strings or dependency injection.

Specification

This PEP introduces a protocol that can be used by static and runtime
type checkers to validate the consistency between Annotated metadata and
a given type. Objects that implement this protocol have an attribute
called __supports_annotated_base__ that specifies whether the metadata
is valid for a given type:

    class Int64:
        __supports_annotated_base__: int

The attribute may also be marked as a ClassVar to avoid interaction with
dataclasses:

    from dataclasses import dataclass
    from typing import ClassVar

    @dataclass
    class Gt:
        value: int
        __supports_annotated_base__: ClassVar[int]

When a static type checker encounters a type expression of the form
Annotated[T, M1, M2, ...], it should enforce that for each metadata
element in M1, M2, ..., one of the following is true:

-   The metadata element evaluates to an object that does not have a
    __supports_annotated_base__ attribute; or
-   The metadata element evaluates to an object M that has a
    __supports_annotated_base__ attribute; and T is assignable to the
    type of M.__supports_annotated_base__.

To support generic Gt metadata, one might write:

    from typing import Protocol

    class SupportsGt[T](Protocol):
        def __gt__(self, __other: T) -> bool:
            ...

    class Gt[T]:
        __supports_annotated_base__: ClassVar[SupportsGt[T]]

        def __init__(self, value: T) -> None:
            self.value = value

    x1: Annotated[int, Gt(0)] = 1  # OK
    x2: Annotated[str, Gt(0)] = 0  # type checker error: str is not assignable to SupportsGt[int]
    x3: Annotated[int, Gt(1)] = 0  # OK for static type checkers; runtime type checkers may flag this

Backwards Compatibility

Metadata that does not implement the protocol will be considered valid
for all types, so no breaking changes are introduced for existing code.
The new checks only apply to metadata objects that explicitly implement
the protocol specified by this PEP.

Security Implications

None.

How to Teach This

This protocol is intended mostly for libraries that provide Annotated
metadata; end users of those libraries are unlikely to need to implement
the protocol themselves. The protocol should be mentioned in the
documentation for typing.Annotated and in the typing specification.

Reference Implementation

None yet.

Rejected ideas

Introducing a type variable instead of a generic class

We considered using a special type variable,
AnnotatedT = TypeVar("AnnotatedT"), to represent the type T of the inner
type in Annotated; metadata would be type checked against this type
variable. However, this would require using the old type variable syntax
(before PEP 695), which is now a discouraged feature. In addition, this
would use type variables in an unusual way that does not fit well with
the rest of the type system.

Introducing a new type to typing.py that all metadata objects should subclass

A previous version of this PEP suggested adding a new generic base
class, TypedMetadata[U], that metadata objects would subclass. If a
metadata object is a subclass of TypedMetadata[U], then type checkers
would check that the annotation's base type is assignable to U. However,
this mechanism does not integrate as well with the rest of the language;
Python does not generally use marker base classes. In addition, it
provides less flexibility than the current proposal: it would not allow
overloads, and it would require metadata objects to add a new base
class, which may make their runtime implementation more complex.

Using a method instead of an attribute for __supports_annotated_base__

We considered using a method instead of an attribute for the protocol,
so that this method can be used at runtime to check the validity of the
metadata and to support overloads or returning boolean literals.
However, using a method adds boilerplate to the implementation and the
value of the runtime use cases or more complex scenarios involving
overloads and returning boolean literals was not clear.

Acknowledgments

We thank Eric Traut for suggesting the idea of using a protocol and
implementing provisional support in Pyright. Thank you to Jelle Zijlstra
for sponsoring this PEP.

Copyright

This document has been placed in the public domain.