text:
PEP: 653 Title: Precise Semantics for Pattern Matching Author: Mark
Shannon<EMAIL_ADDRESS>Status: Draft Type: Standards Track
Content-Type: text/x-rst Created: 09-Feb-2021 Post-History: 18-Feb-2021

Abstract

This PEP proposes a semantics for pattern matching that respects the
general concept of PEP 634, but is more precise, easier to reason about,
and should be faster.

The object model will be extended with two special (dunder) attributes,
__match_container__ and __match_class__, in addition to the
__match_args__ attribute from PEP 634, to support pattern matching. Both
of these new attributes must be integers and __match_args__ is required
to be a tuple of unique strings.

With this PEP:

-   The semantics of pattern matching will be clearer, so that patterns
    are easier to reason about.
-   It will be possible to implement pattern matching in a more
    efficient fashion.
-   Pattern matching will be more usable for complex classes, by
    allowing classes some more control over which patterns they match.

Motivation

Pattern matching in Python, as described in PEP 634, is to be added to
Python 3.10. Unfortunately, PEP 634 is not as precise about the
semantics as it could be, nor does it allow classes sufficient control
over how they match patterns.

Precise semantics

PEP 634 explicitly includes a section on undefined behavior. Large
amounts of undefined behavior may be acceptable in a language like C,
but in Python it should be kept to a minimum. Pattern matching in Python
can be defined more precisely without losing expressiveness or
performance.

Improved control over class matching

PEP 634 delegates the decision over whether a class is a sequence or
mapping to collections.abc. Not all classes that could be considered
sequences are registered as subclasses of collections.abc.Sequence. This
PEP allows them to match sequence patterns, without the full
collections.abc.Sequence machinery.

PEP 634 privileges some builtin classes with a special form of matching,
the "self" match. For example the pattern list(x) matches a list and
assigns the list to x. By allowing classes to choose which kinds of
pattern they match, other classes can use this form as well.

For example, using sympy, we might want to write:

    # a*a == a**2
    case Mul(args=[Symbol(a), Symbol(b)]) if a == b:
        return Pow(a, 2)

Which requires the sympy class Symbol to "self" match. For sympy to
support this pattern with PEP 634 is possible, but a bit tricky. With
this PEP it can be implemented very easily[1].

Robustness

With this PEP, access to attributes during pattern matching becomes well
defined and deterministic. This makes pattern matching less error prone
when matching objects with hidden side effects, such as
object-relational mappers. Objects will have more control over their own
deconstruction, which can help prevent unintended consequences should
attribute access have side-effects.

PEP 634 relies on the collections.abc module when determining which
patterns a value can match, implicitly importing it if necessary. This
PEP will eliminate surprising import errors and misleading audit events
from those imports.

Efficient implementation

The semantics proposed in this PEP will allow efficient implementation,
partly as a result of having precise semantics and partly from using the
object model.

With precise semantics, it is possible to reason about what code
transformations are correct, and thus apply optimizations effectively.

Because the object model is a core part of Python, implementations
already handle special attribute lookup efficiently. Looking up a
special attribute is much faster than performing a subclass test on an
abstract base class.

Rationale

The object model and special methods are at the core of the Python
language. Consequently, implementations support them well. Using special
attributes for pattern matching allows pattern matching to be
implemented in a way that integrates well with the rest of the
implementation, and is thus easier to maintain and is likely to perform
better.

A match statement performs a sequence of pattern matches. In general,
matching a pattern has three parts:

1.  Can the value match this kind of pattern?
2.  When deconstructed, does the value match this particular pattern?
3.  Is the guard true?

To determine whether a value can match a particular kind of pattern, we
add the __match_container__ and __match_class__ attributes. This allows
the kind of a value to be determined in a efficient fashion.

Specification

Additions to the object model

The __match_container__ and __match_class__ attributes will be added to
object. __match_container__ should be overridden by classes that want to
match mapping or sequence patterns. __match_class__ should be overridden
by classes that want to change the default behavior when matching class
patterns.

__match_container__ must be an integer and should be exactly one of
these:

    0
    MATCH_SEQUENCE = 1
    MATCH_MAPPING = 2

MATCH_SEQUENCE is used to indicate that instances of the class can match
sequence patterns.

MATCH_MAPPING is used to indicate that instances of the class can match
mapping patterns.

__match_class__ must be an integer and should be exactly one of these:

    0
    MATCH_SELF = 8

MATCH_SELF is used to indicate that for a single positional argument
class pattern, the subject will be used and not deconstructed.

Note

In the rest of this document, we will refer to the above values by name
only. Symbolic constants will be provided both for Python and C, and the
values will never be changed.

object will have the following values for the special attributes:

    __match_container__ = 0
    __match_class__= 0
    __match_args__ = ()

These special attributes will be inherited as normal.

If __match_args__ is overridden, then it is required to hold a tuple of
unique strings. It may be empty.

Note

__match_args__ will be automatically generated for dataclasses and named
tuples, as specified in PEP 634.

The pattern matching implementation is not required to check that any of
these attributes behave as specified. If the value of
__match_container__, __match_class__ or __match_args__ is not as
specified, then the implementation may raise any exception, or match the
wrong pattern. Of course, implementations are free to check these
properties and provide meaningful error messages if they can do so
efficiently.

Semantics of the matching process

In the following, all variables of the form $var are temporary variables
and are not visible to the Python program. They may be visible via
introspection, but that is an implementation detail and should not be
relied on. The pseudo-statement FAIL is used to signify that matching
failed for this pattern and that matching should move to the next
pattern. If control reaches the end of the translation without reaching
a FAIL, then it has matched, and following patterns are ignored.

Variables of the form $ALL_CAPS are meta-variables holding a syntactic
element, they are not normal variables. So, $VARS = $items is not an
assignment of $items to $VARS, but an unpacking of $items into the
variables that $VARS holds. For example, with the abstract syntax
case [$VARS]:, and the concrete syntax case[a, b]: then $VARS would hold
the variables (a, b), not the values of those variables.

The pseudo-function QUOTE takes a variable and returns the name of that
variable. For example, if the meta-variable $VAR held the variable foo
then QUOTE($VAR) == "foo".

All additional code listed below that is not present in the original
source will not trigger line events, conforming to PEP 626.

Preamble

Before any patterns are matched, the expression being matched is
evaluated:

    match expr:

translates to:

    $value = expr

Capture patterns

Capture patterns always match, so the irrefutable match:

    case capture_var:

translates to:

    capture_var = $value

Wildcard patterns

Wildcard patterns always match, so:

    case _:

translates to:

    # No code -- Automatically matches

Literal Patterns

The literal pattern:

    case LITERAL:

translates to:

    if $value != LITERAL:
        FAIL

except when the literal is one of None, True or False , when it
translates to:

    if $value is not LITERAL:
        FAIL

Value Patterns

The value pattern:

    case value.pattern:

translates to:

    if $value != value.pattern:
        FAIL

Sequence Patterns

A pattern not including a star pattern:

    case [$VARS]:

translates to:

    $kind = type($value).__match_container__
    if $kind != MATCH_SEQUENCE:
        FAIL
    if len($value) != len($VARS):
        FAIL
    $VARS = $value

Example:[2]

A pattern including a star pattern:

    case [$VARS]

translates to:

    $kind = type($value).__match_container__
    if $kind != MATCH_SEQUENCE:
        FAIL
    if len($value) < len($VARS):
        FAIL
    $VARS = $value # Note that $VARS includes a star expression.

Example:[3]

Mapping Patterns

A pattern not including a double-star pattern:

    case {$KEYWORD_PATTERNS}:

translates to:

    $sentinel = object()
    $kind = type($value).__match_container__
    if $kind != MATCH_MAPPING:
        FAIL
    # $KEYWORD_PATTERNS is a meta-variable mapping names to variables.
    for $KEYWORD in $KEYWORD_PATTERNS:
        $tmp = $value.get(QUOTE($KEYWORD), $sentinel)
        if $tmp is $sentinel:
            FAIL
        $KEYWORD_PATTERNS[$KEYWORD] = $tmp

Example:[4]

A pattern including a double-star pattern:

    case {$KEYWORD_PATTERNS, **$DOUBLE_STARRED_PATTERN}:

translates to:

    $kind = type($value).__match_container__
    if $kind != MATCH_MAPPING:
        FAIL
    # $KEYWORD_PATTERNS is a meta-variable mapping names to variables.
    $tmp = dict($value)
    if not $tmp.keys() >= $KEYWORD_PATTERNS.keys():
        FAIL:
    for $KEYWORD in $KEYWORD_PATTERNS:
        $KEYWORD_PATTERNS[$KEYWORD] = $tmp.pop(QUOTE($KEYWORD))
    $DOUBLE_STARRED_PATTERN = $tmp

Example:[5]

Class Patterns

Class pattern with no arguments:

    case ClsName():

translates to:

    if not isinstance($value, ClsName):
        FAIL

Class pattern with a single positional pattern:

    case ClsName($VAR):

translates to:

    $kind = type($value).__match_class__
    if $kind == MATCH_SELF:
        if not isinstance($value, ClsName):
            FAIL
        $VAR = $value
    else:
        As other positional-only class pattern

Positional-only class pattern:

    case ClsName($VARS):

translates to:

    if not isinstance($value, ClsName):
        FAIL
    $attrs = ClsName.__match_args__
    if len($attr) < len($VARS):
        raise TypeError(...)
    try:
        for i, $VAR in enumerate($VARS):
            $VAR = getattr($value, $attrs[i])
    except AttributeError:
        FAIL

Example:[6]

Class patterns with all keyword patterns:

    case ClsName($KEYWORD_PATTERNS):

translates to:

    if not isinstance($value, ClsName):
        FAIL
    try:
        for $KEYWORD in $KEYWORD_PATTERNS:
            $tmp = getattr($value, QUOTE($KEYWORD))
            $KEYWORD_PATTERNS[$KEYWORD] = $tmp
    except AttributeError:
        FAIL

Example:[7]

Class patterns with positional and keyword patterns:

    case ClsName($VARS, $KEYWORD_PATTERNS):

translates to:

    if not isinstance($value, ClsName):
        FAIL
    $attrs = ClsName.__match_args__
    if len($attr) < len($VARS):
        raise TypeError(...)
    $pos_attrs = $attrs[:len($VARS)]
    try:
        for i, $VAR in enumerate($VARS):
            $VAR = getattr($value, $attrs[i])
        for $KEYWORD in $KEYWORD_PATTERNS:
            $name = QUOTE($KEYWORD)
            if $name in pos_attrs:
                raise TypeError(...)
            $KEYWORD_PATTERNS[$KEYWORD] = getattr($value, $name)
    except AttributeError:
        FAIL

Example:[8]

Nested patterns

The above specification assumes that patterns are not nested. For nested
patterns the above translations are applied recursively by introducing
temporary capture patterns.

For example, the pattern:

    case [int(), str()]:

translates to:

    $kind = type($value).__match_class__
    if $kind != MATCH_SEQUENCE:
        FAIL
    if len($value) != 2:
        FAIL
    $value_0, $value_1 = $value
    #Now match on temporary values
    if not isinstance($value_0, int):
        FAIL
    if not isinstance($value_1, str):
        FAIL

Guards

Guards translate to a test following the rest of the translation:

    case pattern if guard:

translates to:

    [translation for pattern]
    if not guard:
        FAIL

Non-conforming special attributes

All classes should ensure that the the values of __match_container__,
__match_class__ and __match_args__ follow the specification. Therefore,
implementations can assume, without checking, that the following are
true:

    __match_container__ == 0 or __match_container__ == MATCH_SEQUENCE or __match_container__ == MATCH_MAPPING
    __match_class__ == 0 or __match_class__ == MATCH_SELF

and that __match_args__ is a tuple of unique strings.

Values of the special attributes for classes in the standard library

For the core builtin container classes __match_container__ will be:

-   list: MATCH_SEQUENCE
-   tuple: MATCH_SEQUENCE
-   dict: MATCH_MAPPING
-   bytearray: 0
-   bytes: 0
-   str: 0

Named tuples will have __match_container__ set to MATCH_SEQUENCE.

-   All other standard library classes for which
    issubclass(cls, collections.abc.Mapping) is true will have
    __match_container__ set to MATCH_MAPPING.
-   All other standard library classes for which
    issubclass(cls, collections.abc.Sequence) is true will have
    __match_container__ set to MATCH_SEQUENCE.

For the following builtin classes __match_class__ will be set to
MATCH_SELF:

-   bool
-   bytearray
-   bytes
-   float
-   frozenset
-   int
-   set
-   str
-   list
-   tuple
-   dict

Legal optimizations

The above semantics implies a lot of redundant effort and copying in the
implementation. However, it is possible to implement the above semantics
efficiently by employing semantic preserving transformations on the
naive implementation.

When performing matching, implementations are allowed to treat the
following functions and methods as pure:

For any class supporting MATCH_SEQUENCE:

    * ``cls.__len__()``
    * ``cls.__getitem__()``

For any class supporting MATCH_MAPPING:

    * ``cls.get()`` (Two argument form only)

Implementations are allowed to make the following assumptions:

-   isinstance(obj, cls) can be freely replaced with
    issubclass(type(obj), cls) and vice-versa.
-   isinstance(obj, cls) will always return the same result for any
    (obj, cls) pair and repeated calls can thus be elided.
-   Reading any of __match_container__, __match_class__ or
    __match_args__ is a pure operation, and may be cached.
-   Sequences, that is any class for which
    __match_container__ == MATCH_SEQUENCE is not zero, are not modified
    by iteration, subscripting or calls to len(). Consequently, those
    operations can be freely substituted for each other where they would
    be equivalent when applied to an immutable sequence.
-   Mappings, that is any class for which
    __match_container__ == MATCH_MAPPING is not zero, will not capture
    the second argument of the get() method. So, the $sentinel value may
    be freely re-used.

In fact, implementations are encouraged to make these assumptions, as it
is likely to result in significantly better performance.

Security Implications

None.

Implementation

The naive implementation that follows from the specification will not be
very efficient. Fortunately, there are some reasonably straightforward
transformations that can be used to improve performance. Performance
should be comparable to the implementation of PEP 634 (at time of
writing) by the release of 3.10. Further performance improvements may
have to wait for the 3.11 release.

Possible optimizations

The following is not part of the specification, but guidelines to help
developers create an efficient implementation.

Splitting evaluation into lanes

Since the first step in matching each pattern is check to against the
kind, it is possible to combine all the checks against kind into a
single multi-way branch at the beginning of the match. The list of cases
can then be duplicated into several "lanes" each corresponding to one
kind. It is then trivial to remove unmatchable cases from each lane.
Depending on the kind, different optimization strategies are possible
for each lane. Note that the body of the match clause does not need to
be duplicated, just the pattern.

Sequence patterns

This is probably the most complex to optimize and the most profitable in
terms of performance. Since each pattern can only match a range of
lengths, often only a single length, the sequence of tests can be
rewritten in as an explicit iteration over the sequence, attempting to
match only those patterns that apply to that sequence length.

For example:

    case []:
        A
    case [x]:
        B
    case [x, y]:
        C
    case other:
        D

Can be compiled roughly as:

    # Choose lane
    $i = iter($value)
    for $0 in $i:
        break
    else:
        A
        goto done
    for $1 in $i:
        break
    else:
        x = $0
        B
        goto done
    for $2 in $i:
        del $0, $1, $2
        break
    else:
        x = $0
        y = $1
        C
        goto done
    other = $value
    D
    done:

Mapping patterns

The best strategy here is probably to form a decision tree based on the
size of the mapping and which keys are present. There is no point
repeatedly testing for the presence of a key. For example:

    match obj:
        case {a:x, b:y}:
            W
        case {a:x, c:y}:
            X
        case {a:x, b:_, c:y}:
            Y
        case other:
            Z

If the key "a" is not present when checking for case X, there is no need
to check it again for Y.

The mapping lane can be implemented, roughly as:

    # Choose lane
    if len($value) == 2:
        if "a" in $value:
            if "b" in $value:
                x = $value["a"]
                y = $value["b"]
                goto W
            if "c" in $value:
                x = $value["a"]
                y = $value["c"]
                goto X
    elif len($value) == 3:
        if "a" in $value and "b" in $value:
            x = $value["a"]
            y = $value["c"]
            goto Y
    other = $value
    goto Z

Summary of differences between this PEP and PEP 634

The changes to the semantics can be summarized as:

-   Requires __match_args__ to be a tuple of strings, not just a
    sequence. This make pattern matching a bit more robust and
    optimizable as __match_args__ can be assumed to be immutable.
-   Selecting the kind of container patterns that can be matched uses
    cls.__match_container__ instead of
    issubclass(cls, collections.abc.Mapping) and
    issubclass(cls, collections.abc.Sequence).
-   Allows classes to opt out of deconstruction altogether, if
    necessary, but setting __match_class__ = 0.
-   The behavior when matching patterns is more precisely defined, but
    is otherwise unchanged.

There are no changes to syntax. All examples given in the PEP 636
tutorial should continue to work as they do now.

Rejected Ideas

Using attributes from the instance's dictionary

An earlier version of this PEP only used attributes from the instance's
dictionary when matching a class pattern with __match_class__ was the
default value. The intent was to avoid capturing bound-methods and other
synthetic attributes. However, this also mean that properties were
ignored.

For the class:

    class C:
        def __init__(self):
            self.a = "a"
        @property
        def p(self):
            ...
        def m(self):
            ...

Ideally we would match the attributes "a" and "p", but not "m". However,
there is no general way to do that, so this PEP now follows the
semantics of PEP 634.

Lookup of __match_args__ on the subject not the pattern

An earlier version of this PEP looked up __match_args__ on the class of
the subject and not the class specified in the pattern. This has been
rejected for a few reasons:

    * Using the class specified in the pattern is more amenable to optimization and can offer better performance.
    * Using the class specified in the pattern has the potential to provide better error reporting is some cases.
    * Neither approach is perfect, both have odd corner cases. Keeping the status quo minimizes disruption.

Combining __match_class__ and __match_container__ into a single value

An earlier version of this PEP combined __match_class__ and
__match_container__ into a single value, __match_kind__. Using a single
value has a small advantage in terms of performance, but is likely to
result in unintended changes to container matching when overriding class
matching behavior, and vice versa.

Deferred Ideas

The original version of this PEP included the match kind
MATCH_POSITIONAL and special method __deconstruct__ which would allow
classes full control over their matching. This is important for
libraries like sympy.

For example, using sympy, we might want to write:

    # sin(x)**2 + cos(x)**2 == 1
    case Add(Pow(sin(a), 2), Pow(cos(b), 2)) if a == b:
        return 1

For sympy to support the positional patterns with current pattern
matching is possible, but is tricky. With these additional features it
can be implemented easily[9].

This idea will feature in a future PEP for 3.11. However, it is too late
in the 3.10 development cycle for such a change.

Having a separate value to reject all class matches

In an earlier version of this PEP, there was a distinct value for
__match_class__ that allowed classes to not match any class pattern that
would have required deconstruction. However, this would become redundant
once MATCH_POSITIONAL is introduced, and complicates the specification
for an extremely rare case.

Code examples

    class Symbol:
        __match_class__ = MATCH_SELF

This:

    case [a, b] if a is b:

translates to:

    $kind = type($value).__match_container__
    if $kind != MATCH_SEQUENCE:
        FAIL
    if len($value) != 2:
        FAIL
    a, b = $value
    if not a is b:
        FAIL

This:

    case [a, *b, c]:

translates to:

    $kind = type($value).__match_container__
    if $kind != MATCH_SEQUENCE:
        FAIL
    if len($value) < 2:
        FAIL
    a, *b, c = $value

This:

    case {"x": x, "y": y} if x > 2:

translates to:

    $kind = type($value).__match_container__
    if $kind != MATCH_MAPPING:
        FAIL
    $tmp = $value.get("x", $sentinel)
    if $tmp is $sentinel:
        FAIL
    x = $tmp
    $tmp = $value.get("y", $sentinel)
    if $tmp is $sentinel:
        FAIL
    y = $tmp
    if not x > 2:
        FAIL

This:

    case {"x": x, "y": y, **z}:

translates to:

    $kind = type($value).__match_container__
    if $kind != MATCH_MAPPING:
        FAIL
    $tmp = dict($value)
    if not $tmp.keys() >= {"x", "y"}:
        FAIL
    x = $tmp.pop("x")
    y = $tmp.pop("y")
    z = $tmp

This:

    match ClsName(x, y):

translates to:

    if not isinstance($value, ClsName):
        FAIL
    $attrs = ClsName.__match_args__
    if len($attr) < 2:
        FAIL
    try:
        x = getattr($value, $attrs[0])
        y = getattr($value, $attrs[1])
    except AttributeError:
        FAIL

This:

    match ClsName(a=x, b=y):

translates to:

    if not isinstance($value, ClsName):
        FAIL
    try:
        x = $value.a
        y = $value.b
    except AttributeError:
        FAIL

This:

    match ClsName(x, a=y):

translates to:

    if not isinstance($value, ClsName):
        FAIL
    $attrs = ClsName.__match_args__
    if len($attr) < 1:
        raise TypeError(...)
    $positional_names = $attrs[:1]
    try:
        x = getattr($value, $attrs[0])
        if "a" in $positional_names:
            raise TypeError(...)
        y = $value.a
    except AttributeError:
        FAIL

    class Basic:
        __match_class__ = MATCH_POSITIONAL
        def __deconstruct__(self):
            return self._args

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]


text:
PEP: 561 Title: Distributing and Packaging Type Information Author:
Ethan Smith<EMAIL_ADDRESS>Status: Final Type: Standards Track
Topic: Packaging, Typing Content-Type: text/x-rst Created: 09-Sep-2017
Python-Version: 3.7 Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017,
26-Oct-2017, 12-Apr-2018

typing:packaging-typed-libraries

Abstract

PEP 484 introduced type hinting to Python, with goals of making typing
gradual and easy to adopt. Currently, typing information must be
distributed manually. This PEP provides a standardized means to leverage
existing tooling to package and distribute type information with minimal
work and an ordering for type checkers to resolve modules and collect
this information for type checking.

Rationale

Currently, package authors wish to distribute code that has inline type
information. Additionally, maintainers would like to distribute stub
files to keep Python 2 compatibility while using newer annotation
syntax. However, there is no standard method to distribute packages with
type information. Also, if one wished to ship stub files privately the
only method available would be via setting MYPYPATH or the equivalent to
manually point to stubs. If the package can be released publicly, it can
be added to typeshed[1]. However, this does not scale and becomes a
burden on the maintainers of typeshed. In addition, it ties bug fixes in
stubs to releases of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this
section <484#storing-and-distributing-stub-files> the PEP recommends
using shared/typehints/pythonX.Y/ for shipping stub files. However,
manually adding a path to stub files for each third party library does
not scale. The simplest approach people have taken is to add
site-packages to their MYPYPATH, but this causes type checkers to fail
on packages that are highly dynamic (e.g. sqlalchemy and Django).

Definition of Terms

The definition of "MAY", "MUST", and "SHOULD", and "SHOULD NOT" are to
be interpreted as described in 2119.

"inline" - the types are part of the runtime code using PEP 526 and PEP
3107 syntax (the filename ends in .py).

"stubs" - files containing only type information, empty of runtime code
(the filename ends in .pyi).

"Distributions" are the packaged files which are used to publish and
distribute a release. (PEP 426)

"Module" a file containing Python runtime code or stubbed type
information.

"Package" a directory or directories that namespace Python modules.
(Note the distinction between packages and distributions. While most
distributions are named after the one package they install, some
distributions install multiple packages.)

Specification

There are several motivations and methods of supporting typing in a
package. This PEP recognizes three types of packages that users of
typing wish to create:

1.  The package maintainer would like to add type information inline.
2.  The package maintainer would like to add type information via stubs.
3.  A third party or package maintainer would like to share stub files
    for a package, but the maintainer does not want to include them in
    the source of the package.

This PEP aims to support all three scenarios and make them simple to add
to packaging and deployment.

The two major parts of this specification are the packaging
specifications and the resolution order for resolving module type
information. The type checking spec is meant to replace the
shared/typehints/pythonX.Y/
spec of PEP 484 <484#storing-and-distributing-stub-files>.

New third party stub libraries SHOULD distribute stubs via the third
party packaging methods proposed in this PEP in place of being added to
typeshed. Typeshed will remain in use, but if maintainers are found,
third party stubs in typeshed MAY be split into their own package.

Packaging Type Information

In order to make packaging and distributing type information as simple
and easy as possible, packaging and distribution is done through
existing frameworks.

Package maintainers who wish to support type checking of their code MUST
add a marker file named py.typed to their package supporting typing.
This marker applies recursively: if a top-level package includes it, all
its sub-packages MUST support type checking as well. To have this file
installed with the package, maintainers can use existing packaging
options such as package_data in distutils, shown below.

Distutils option example:

    setup(
        ...,
        package_data = {
            'foopkg': ['py.typed'],
        },
        ...,
        )

For namespace packages (see PEP 420), the py.typed file should be in the
submodules of the namespace, to avoid conflicts and for clarity.

This PEP does not support distributing typing information as part of
module-only distributions or single-file modules within namespace
packages.

The single-file module should be refactored into a package and indicate
that the package supports typing as described above.

Stub-only Packages

For package maintainers wishing to ship stub files containing all of
their type information, it is preferred that the *.pyi stubs are
alongside the corresponding *.py files. However, the stubs can also be
put in a separate package and distributed separately. Third parties can
also find this method useful if they wish to distribute stub files. The
name of the stub package MUST follow the scheme foopkg-stubs for type
stubs for the package named foopkg. Note that for stub-only packages
adding a py.typed marker is not needed since the name *-stubs is enough
to indicate it is a source of typing information.

Third parties seeking to distribute stub files are encouraged to contact
the maintainer of the package about distribution alongside the package.
If the maintainer does not wish to maintain or package stub files or
type information inline, then a third party stub-only package can be
created.

In addition, stub-only distributions SHOULD indicate which version(s) of
the runtime package are supported by indicating the runtime
distribution's version(s) through normal dependency data. For example,
the stub package flyingcircus-stubs can indicate the versions of the
runtime flyingcircus distribution it supports through install_requires
in distutils-based tools, or the equivalent in other packaging tools.
Note that in pip 9.0, if you update flyingcircus-stubs, it will update
flyingcircus. In pip 9.0, you can use the
--upgrade-strategy=only-if-needed flag. In pip 10.0 this is the default
behavior.

For namespace packages (see PEP 420), stub-only packages should use the
-stubs suffix on only the root namespace package. All stub-only
namespace packages should omit __init__.pyi files. py.typed marker files
are not necessary for stub-only packages, but similarly to packages with
inline types, if used, they should be in submodules of the namespace to
avoid conflicts and for clarity.

For example, if the pentagon and hexagon are separate distributions
installing within the namespace package shapes.polygons The
corresponding types-only distributions should produce packages laid out
as follows:

    shapes-stubs
    └── polygons
        └── pentagon
            └── __init__.pyi

    shapes-stubs
    └── polygons
        └── hexagon
            └── __init__.pyi

Type Checker Module Resolution Order

The following is the order in which type checkers supporting this PEP
SHOULD resolve modules containing type information:

1.  Stubs or Python source manually put in the beginning of the path.
    Type checkers SHOULD provide this to allow the user complete control
    of which stubs to use, and to patch broken stubs/inline types from
    packages. In mypy the $MYPYPATH environment variable can be used for
    this.
2.  User code - the files the type checker is running on.
3.  Stub packages - these packages SHOULD supersede any installed inline
    package. They can be found at foopkg-stubs for package foopkg.
4.  Packages with a py.typed marker file - if there is nothing
    overriding the installed package, and it opts into type checking,
    the types bundled with the package SHOULD be used (be they in .pyi
    type stub files or inline in .py files).
5.  Typeshed (if used) - Provides the stdlib types and several third
    party libraries.

If typecheckers identify a stub-only namespace package without the
desired module in step 3, they should continue to step 4/5. Typecheckers
should identify namespace packages by the absence of __init__.pyi. This
allows different subpackages to independently opt for inline vs
stub-only.

Type checkers that check a different Python version than the version
they run on MUST find the type information in the
site-packages/dist-packages of that Python version. This can be queried
e.g. pythonX.Y -c 'import site; print(site.getsitepackages())'. It is
also recommended that the type checker allow for the user to point to a
particular Python binary, in case it is not in the path.

Partial Stub Packages

Many stub packages will only have part of the type interface for
libraries completed, especially initially. For the benefit of type
checking and code editors, packages can be "partial". This means modules
not found in the stub package SHOULD be searched for in parts four and
five of the module resolution order above, namely inline packages and
typeshed.

Type checkers should merge the stub package and runtime package or
typeshed directories. This can be thought of as the functional
equivalent of copying the stub package into the same directory as the
corresponding runtime package or typeshed folder and type checking the
combined directory structure. Thus type checkers MUST maintain the
normal resolution order of checking *.pyi before *.py files.

If a stub package distribution is partial it MUST include partial\n in a
py.typed file. For stub-packages distributing within a namespace package
(see PEP 420), the py.typed file should be in the submodules of the
namespace.

Type checkers should treat namespace packages within stub-packages as
incomplete since multiple distributions may populate them. Regular
packages within namespace packages in stub-package distributions are
considered complete unless a py.typed with partial\n is included.

Implementation

The proposed scheme of indicating support for typing is completely
backwards compatible, and requires no modification to package tooling. A
sample package with inline types is available [typed_package], as well
as a [stub_package]. A sample package checker [pkg_checker] which reads
the metadata of installed packages and reports on their status as either
not typed, inline typed, or a stub package.

The mypy type checker has an implementation of PEP 561 searching which
can be read about in the mypy docs[2].

[numpy-stubs] is an example of a real stub-only package for the numpy
distribution.

Acknowledgements

This PEP would not have been possible without the ideas, feedback, and
support of Ivan Levkivskyi, Jelle Zijlstra, Alyssa Coghlan, Daniel F
Moisset, Andrey Vlasovskikh, Nathaniel Smith, and Guido van Rossum.

Version History

-   2023-01-13

      -   Clarify that the 4th step of the Module Resolution Order <mro>
          applies to any package with a py.typed marker file (and not
          just inline packages).

-   2021-09-20

      -   Clarify expectations and typechecker behavior for stub-only
          namespace packages
      -   Clarify handling of single-file modules within namespace
          packages.

-   2018-07-09

      -   Add links to sample stub-only packages

-   2018-06-19

      -   Partial stub packages can look at typeshed as well as runtime
          packages

-   2018-05-15

      -   Add partial stub package spec.

-   2018-04-09

      -   Add reference to mypy implementation
      -   Clarify stub package priority.

-   2018-02-02

      -   Change stub-only package suffix to be -stubs not _stubs.
      -   Note that py.typed is not needed for stub-only packages.
      -   Add note about pip and upgrading stub packages.

-   2017-11-12

      -   Rewritten to use existing tooling only
      -   No need to indicate kind of type information in metadata
      -   Name of marker file changed from .typeinfo to py.typed

-   2017-11-10

      -   Specification re-written to use package metadata instead of
          distribution metadata.
      -   Removed stub-only packages and merged into third party
          packages spec.
      -   Removed suggestion for typecheckers to consider checking
          runtime versions
      -   Implementations updated to reflect PEP changes.

-   2017-10-26

      -   Added implementation references.
      -   Added acknowledgements and version history.

-   2017-10-06

      -   Rewritten to use .distinfo/METADATA over a distutils specific
          command.
      -   Clarify versioning of third party stub packages.

-   2017-09-11

      -   Added information about current solutions and typeshed.
      -   Clarify rationale.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

numpy-stubs

    Stubs for numpy (https://github.com/numpy/numpy-stubs)

pkg_checker

    Sample package checker (https://github.com/ethanhs/check_typedpkg)

stub_package

    A stub-only package (https://github.com/ethanhs/stub-package)

typed_package

    Sample typed package
    (https://github.com/ethanhs/sample-typed-package)

[1] Typeshed (https://github.com/python/typeshed)

[2] Example implementation in a type checker
(https://mypy.readthedocs.io/en/latest/installed_packages.html)

license: Public Domain


text:
PEP: 673 Title: Self Type Author: Pradeep Kumar Srinivasan
<EMAIL_ADDRESS>James Hilton-Balfe<EMAIL_ADDRESS>Sponsor: Jelle Zijlstra<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Typing
Created: 10-Nov-2021 Python-Version: 3.11 Post-History: 17-Nov-2021
Resolution:
<EMAIL_ADDRESS>
typing:self and typing.Self

Abstract

This PEP introduces a simple and intuitive way to annotate methods that
return an instance of their class. This behaves the same as the
TypeVar-based approach specified in PEP 484 but is more concise and
easier to follow.

Motivation

A common use case is to write a method that returns an instance of the
same class, usually by returning self.

    class Shape:
        def set_scale(self, scale: float):
            self.scale = scale
            return self

    Shape().set_scale(0.5)  # => should be Shape

One way to denote the return type is to specify it as the current class,
say, Shape. Using the method makes the type checker infer the type
Shape, as expected.

    class Shape:
        def set_scale(self, scale: float) -> Shape:
            self.scale = scale
            return self

    Shape().set_scale(0.5)  # => Shape

However, when we call set_scale on a subclass of Shape, the type checker
still infers the return type to be Shape. This is problematic in
situations such as the one shown below, where the type checker will
return an error because we are trying to use attributes or methods not
present on the base class.

    class Circle(Shape):
        def set_radius(self, r: float) -> Circle:
            self.radius = r
            return self

    Circle().set_scale(0.5)  # *Shape*, not Circle
    Circle().set_scale(0.5).set_radius(2.7)
    # => Error: Shape has no attribute set_radius

The present workaround for such instances is to define a TypeVar with
the base class as the bound and use it as the annotation for the self
parameter and the return type:

    from typing import TypeVar

    TShape = TypeVar("TShape", bound="Shape")

    class Shape:
        def set_scale(self: TShape, scale: float) -> TShape:
            self.scale = scale
            return self

    class Circle(Shape):
        def set_radius(self, radius: float) -> Circle:
            self.radius = radius
            return self

    Circle().set_scale(0.5).set_radius(2.7)  # => Circle

Unfortunately, this is verbose and unintuitive. Because self is usually
not explicitly annotated, the above solution doesn't immediately come to
mind, and even if it does, it is very easy to go wrong by forgetting
either the bound on the TypeVar(bound="Shape") or the annotation for
self.

This difficulty means that users often give up and either use fallback
types like Any or just omit the type annotation completely, both of
which make the code less safe.

We propose a more intuitive and succinct way of expressing the above
intention. We introduce a special form Self that stands for a type
variable bound to the encapsulating class. For situations such as the
one above, the user simply has to annotate the return type as Self:

    from typing import Self

    class Shape:
        def set_scale(self, scale: float) -> Self:
            self.scale = scale
            return self

    class Circle(Shape):
        def set_radius(self, radius: float) -> Self:
            self.radius = radius
            return self

By annotating the return type as Self, we no longer have to declare a
TypeVar with an explicit bound on the base class. The return type Self
mirrors the fact that the function returns self and is easier to
understand.

As in the above example, the type checker will correctly infer the type
of Circle().set_scale(0.5) to be Circle, as expected.

Usage statistics

We analyzed popular open-source projects and found that patterns like
the above were used about 40% as often as popular types like dict or
Callable. For example, in typeshed alone, such “Self” types are used 523
times, compared to 1286 uses of dict and 1314 uses of Callable as of
October 2021. This suggests that a Self type will be used quite often
and users will benefit a lot from the simpler approach above.

Users of Python types have also frequently requested this feature, both
on the proposal doc and on GitHub.

Specification

Use in Method Signatures

Self used in the signature of a method is treated as if it were a
TypeVar bound to the class.

    from typing import Self

    class Shape:
        def set_scale(self, scale: float) -> Self:
            self.scale = scale
            return self

is treated equivalently to:

    from typing import TypeVar

    SelfShape = TypeVar("SelfShape", bound="Shape")

    class Shape:
        def set_scale(self: SelfShape, scale: float) -> SelfShape:
            self.scale = scale
            return self

This works the same for a subclass too:

    class Circle(Shape):
        def set_radius(self, radius: float) -> Self:
            self.radius = radius
            return self

which is treated equivalently to:

    SelfCircle = TypeVar("SelfCircle", bound="Circle")

    class Circle(Shape):
        def set_radius(self: SelfCircle, radius: float) -> SelfCircle:
            self.radius = radius
            return self

One implementation strategy is to simply desugar the former to the
latter in a preprocessing step. If a method uses Self in its signature,
the type of self within a method will be Self. In other cases, the type
of self will remain the enclosing class.

Use in Classmethod Signatures

The Self type annotation is also useful for classmethods that return an
instance of the class that they operate on. For example, from_config in
the following snippet builds a Shape object from a given config.

    class Shape:
        def __init__(self, scale: float) -> None: ...

        @classmethod
        def from_config(cls, config: dict[str, float]) -> Shape:
            return cls(config["scale"])

However, this means that Circle.from_config(...) is inferred to return a
value of type Shape, when in fact it should be Circle:

    class Circle(Shape):
        def circumference(self) -> float: ...

    shape = Shape.from_config({"scale": 7.0})
    # => Shape

    circle = Circle.from_config({"scale": 7.0})
    # => *Shape*, not Circle

    circle.circumference()
    # Error: `Shape` has no attribute `circumference`

The current workaround for this is unintuitive and error-prone:

    Self = TypeVar("Self", bound="Shape")

    class Shape:
        @classmethod
        def from_config(
            cls: type[Self], config: dict[str, float]
        ) -> Self:
            return cls(config["scale"])

We propose using Self directly:

    from typing import Self

    class Shape:
        @classmethod
        def from_config(cls, config: dict[str, float]) -> Self:
            return cls(config["scale"])

This avoids the complicated cls: type[Self] annotation and the TypeVar
declaration with a bound. Once again, the latter code behaves
equivalently to the former code.

Use in Parameter Types

Another use for Self is to annotate parameters that expect instances of
the current class:

    Self = TypeVar("Self", bound="Shape")

    class Shape:
        def difference(self: Self, other: Self) -> float: ...

        def apply(self: Self, f: Callable[[Self], None]) -> None: ...

We propose using Self directly to achieve the same behavior:

    from typing import Self

    class Shape:
        def difference(self, other: Self) -> float: ...

        def apply(self, f: Callable[[Self], None]) -> None: ...

Note that specifying self: Self is harmless, so some users may find it
more readable to write the above as:

    class Shape:
        def difference(self: Self, other: Self) -> float: ...

Use in Attribute Annotations

Another use for Self is to annotate attributes. One example is where we
have a LinkedList whose elements must be subclasses of the current
class.

    from dataclasses import dataclass
    from typing import Generic, TypeVar

    T = TypeVar("T")

    @dataclass
    class LinkedList(Generic[T]):
        value: T
        next: LinkedList[T] | None = None

    # OK
    LinkedList[int](value=1, next=LinkedList[int](value=2))
    # Not OK
    LinkedList[int](value=1, next=LinkedList[str](value="hello"))

However, annotating the next attribute as LinkedList[T] allows invalid
constructions with subclasses:

    @dataclass
    class OrdinalLinkedList(LinkedList[int]):
        def ordinal_value(self) -> str:
            return as_ordinal(self.value)

    # Should not be OK because LinkedList[int] is not a subclass of
    # OrdinalLinkedList, # but the type checker allows it.
    xs = OrdinalLinkedList(value=1, next=LinkedList[int](value=2))

    if xs.next:
        print(xs.next.ordinal_value())  # Runtime Error.

We propose expressing this constraint using next: Self | None:

    from typing import Self

    @dataclass
    class LinkedList(Generic[T]):
        value: T
        next: Self | None = None

    @dataclass
    class OrdinalLinkedList(LinkedList[int]):
        def ordinal_value(self) -> str:
            return as_ordinal(self.value)

    xs = OrdinalLinkedList(value=1, next=LinkedList[int](value=2))
    # Type error: Expected OrdinalLinkedList, got LinkedList[int].

    if xs.next is not None:
        xs.next = OrdinalLinkedList(value=3, next=None)  # OK
        xs.next = LinkedList[int](value=3, next=None)  # Not OK

The code above is semantically equivalent to treating each attribute
containing a Self type as a property that returns that type:

    from dataclasses import dataclass
    from typing import Any, Generic, TypeVar

    T = TypeVar("T")
    Self = TypeVar("Self", bound="LinkedList")

    class LinkedList(Generic[T]):
        value: T

        @property
        def next(self: Self) -> Self | None:
            return self._next

        @next.setter
        def next(self: Self, next: Self | None) -> None:
            self._next = next

    class OrdinalLinkedList(LinkedList[int]):
        def ordinal_value(self) -> str:
            return str(self.value)

Use in Generic Classes

Self can also be used in generic class methods:

    class Container(Generic[T]):
        value: T
        def set_value(self, value: T) -> Self: ...

This is equivalent to writing:

    Self = TypeVar("Self", bound="Container[Any]")

    class Container(Generic[T]):
        value: T
        def set_value(self: Self, value: T) -> Self: ...

The behavior is to preserve the type argument of the object on which the
method was called. When called on an object with concrete type
Container[int], Self is bound to Container[int]. When called with an
object of generic type Container[T], Self is bound to Container[T]:

    def object_with_concrete_type() -> None:
        int_container: Container[int]
        str_container: Container[str]
        reveal_type(int_container.set_value(42))  # => Container[int]
        reveal_type(str_container.set_value("hello"))  # => Container[str]

    def object_with_generic_type(
        container: Container[T], value: T,
    ) -> Container[T]:
        return container.set_value(value)  # => Container[T]

The PEP doesn’t specify the exact type of self.value within the method
set_value. Some type checkers may choose to implement Self types using
class-local type variables with
Self = TypeVar(“Self”, bound=Container[T]), which will infer a precise
type T. However, given that class-local type variables are not a
standardized type system feature, it is also acceptable to infer Any for
self.value. We leave this up to the type checker.

Note that we reject using Self with type arguments, such as Self[int].
This is because it creates ambiguity about the type of the self
parameter and introduces unnecessary complexity:

    class Container(Generic[T]):
        def foo(
            self, other: Self[int], other2: Self,
        ) -> Self[str]:  # Rejected
            ...

In such cases, we recommend using an explicit type for self:

    class Container(Generic[T]):
        def foo(
            self: Container[T],
            other: Container[int],
            other2: Container[T]
        ) -> Container[str]: ...

Use in Protocols

Self is valid within Protocols, similar to its use in classes:

    from typing import Protocol, Self

    class ShapeProtocol(Protocol):
        scale: float

        def set_scale(self, scale: float) -> Self:
            self.scale = scale
            return self

is treated equivalently to:

    from typing import TypeVar

    SelfShape = TypeVar("SelfShape", bound="ShapeProtocol")

    class ShapeProtocol(Protocol):
        scale: float

        def set_scale(self: SelfShape, scale: float) -> SelfShape:
            self.scale = scale
            return self

See PEP 544
<544#self-types-in-protocols> for details on the behavior of TypeVars
bound to protocols.

Checking a class for compatibility with a protocol: If a protocol uses
Self in methods or attribute annotations, then a class Foo is considered
compatible with the protocol if its corresponding methods and attribute
annotations use either Self or Foo or any of Foo’s subclasses. See the
examples below:

    from typing import Protocol

    class ShapeProtocol(Protocol):
        def set_scale(self, scale: float) -> Self: ...

    class ReturnSelf:
        scale: float = 1.0

        def set_scale(self, scale: float) -> Self:
            self.scale = scale
            return self

    class ReturnConcreteShape:
        scale: float = 1.0

        def set_scale(self, scale: float) -> ReturnConcreteShape:
            self.scale = scale
            return self

    class BadReturnType:
        scale: float = 1.0

        def set_scale(self, scale: float) -> int:
            self.scale = scale
            return 42

    class ReturnDifferentClass:
        scale: float = 1.0

        def set_scale(self, scale: float) -> ReturnConcreteShape:
            return ReturnConcreteShape(...)

    def accepts_shape(shape: ShapeProtocol) -> None:
        y = shape.set_scale(0.5)
        reveal_type(y)

    def main() -> None:
        return_self_shape: ReturnSelf
        return_concrete_shape: ReturnConcreteShape
        bad_return_type: BadReturnType
        return_different_class: ReturnDifferentClass

        accepts_shape(return_self_shape)  # OK
        accepts_shape(return_concrete_shape)  # OK
        accepts_shape(bad_return_type)  # Not OK
        # Not OK because it returns a non-subclass.
        accepts_shape(return_different_class)

Valid Locations for Self

A Self annotation is only valid in class contexts, and will always refer
to the encapsulating class. In contexts involving nested classes, Self
will always refer to the innermost class.

The following uses of Self are accepted:

    class ReturnsSelf:
        def foo(self) -> Self: ... # Accepted

        @classmethod
        def bar(cls) -> Self:  # Accepted
            return cls()

        def __new__(cls, value: int) -> Self: ...  # Accepted

        def explicitly_use_self(self: Self) -> Self: ...  # Accepted

        # Accepted (Self can be nested within other types)
        def returns_list(self) -> list[Self]: ...

        # Accepted (Self can be nested within other types)
        @classmethod
        def return_cls(cls) -> type[Self]:
            return cls

    class Child(ReturnsSelf):
        # Accepted (we can override a method that uses Self annotations)
        def foo(self) -> Self: ...

    class TakesSelf:
        def foo(self, other: Self) -> bool: ...  # Accepted

    class Recursive:
        # Accepted (treated as an @property returning ``Self | None``)
        next: Self | None

    class CallableAttribute:
        def foo(self) -> int: ...

        # Accepted (treated as an @property returning the Callable type)
        bar: Callable[[Self], int] = foo

    class HasNestedFunction:
        x: int = 42

        def foo(self) -> None:

            # Accepted (Self is bound to HasNestedFunction).
            def nested(z: int, inner_self: Self) -> Self:
                print(z)
                print(inner_self.x)
                return inner_self

            nested(42, self)  # OK

    class Outer:
        class Inner:
            def foo(self) -> Self: ...  # Accepted (Self is bound to Inner)

The following uses of Self are rejected.

    def foo(bar: Self) -> Self: ...  # Rejected (not within a class)

    bar: Self  # Rejected (not within a class)

    class Foo:
        # Rejected (Self is treated as unknown).
        def has_existing_self_annotation(self: T) -> Self: ...

    class Foo:
        def return_concrete_type(self) -> Self:
            return Foo()  # Rejected (see FooChild below for rationale)

    class FooChild(Foo):
        child_value: int = 42

        def child_method(self) -> None:
            # At runtime, this would be Foo, not FooChild.
            y = self.return_concrete_type()

            y.child_value
            # Runtime error: Foo has no attribute child_value

    class Bar(Generic[T]):
        def bar(self) -> T: ...

    class Baz(Bar[Self]): ...  # Rejected

We reject type aliases containing Self. Supporting Self outside class
definitions can require a lot of special-handling in type checkers.
Given that it also goes against the rest of the PEP to use Self outside
a class definition, we believe the added convenience of aliases is not
worth it:

    TupleSelf = Tuple[Self, Self]  # Rejected

    class Alias:
        def return_tuple(self) -> TupleSelf:  # Rejected
            return (self, self)

Note that we reject Self in staticmethods. Self does not add much value
since there is no self or cls to return. The only possible use cases
would be to return a parameter itself or some element from a container
passed in as a parameter. These don’t seem worth the additional
complexity.

    class Base:
        @staticmethod
        def make() -> Self:  # Rejected
            ...

        @staticmethod
        def return_parameter(foo: Self) -> Self:  # Rejected
            ...

Likewise, we reject Self in metaclasses. Self in this PEP consistently
refers to the same type (that of self). But in metaclasses, it would
have to refer to different types in different method signatures. For
example, in __mul__, Self in the return type would refer to the
implementing class Foo, not the enclosing class MyMetaclass. But, in
__new__, Self in the return type would refer to the enclosing class
MyMetaclass. To avoid confusion, we reject this edge case.

    class MyMetaclass(type):
        def __new__(cls, *args: Any) -> Self:  # Rejected
            return super().__new__(cls, *args)

        def __mul__(cls, count: int) -> list[Self]:  # Rejected
            return [cls()] * count

    class Foo(metaclass=MyMetaclass): ...

Runtime behavior

Because Self is not subscriptable, we propose an implementation similar
to typing.NoReturn.

    @_SpecialForm
    def Self(self, params):
        """Used to spell the type of "self" in classes.

        Example::

          from typing import Self

          class ReturnsSelf:
              def parse(self, data: bytes) -> Self:
                  ...
                  return self

        """
        raise TypeError(f"{self} is not subscriptable")

Rejected Alternatives

Allow the Type Checker to Infer the Return Type

One proposal is to leave the Self type implicit and let the type checker
infer from the body of the method that the return type must be the same
as the type of the self parameter:

    class Shape:
        def set_scale(self, scale: float):
            self.scale = scale
            return self  # Type checker infers that we are returning self

We reject this because Explicit Is Better Than Implicit. Beyond that,
the above approach will fail for type stubs, which don’t have method
bodies to analyze.

Reference Implementations

Mypy: Proof of concept implementation in Mypy.

Pyright: v1.1.184

Runtime implementation of Self: PR.

Resources

Similar discussions on a Self type in Python started in Mypy around
2016: Mypy issue #1212 - SelfType or another way to spell "type of
self". However, the approach ultimately taken there was the bounded
TypeVar approach shown in our "before" examples. Other issues that
discuss this include Mypy issue #2354 - Self types in generic classes.

Pradeep made a concrete proposal at the PyCon Typing Summit 2021:

    recorded talk, slides.

James brought up the proposal independently on typing-sig: Typing-sig
thread.

Other languages have similar ways to express the type of the enclosing
class:

-   TypeScript has the this type (TypeScript docs)
-   Rust has the Self type (Rust docs)

Thanks to the following people for their feedback on the PEP:

Jia Chen, Rebecca Chen, Sergei Lebedev, Kaylynn Morgan, Tuomas Suutari,
Eric Traut, Alex Waygood, Shannon Zhu, and Никита Соболев

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 344 Title: Exception Chaining and Embedded Tracebacks Version:
$Revision$ Last-Modified: $Date$ Author: Ka-Ping Yee Status: Superseded
Type: Standards Track Content-Type: text/x-rst Created: 12-May-2005
Python-Version: 2.5 Post-History:

Numbering Note

This PEP has been renumbered to PEP 3134. The text below is the last
version submitted under the old number.

Abstract

This PEP proposes three standard attributes on exception instances: the
__context__ attribute for implicitly chained exceptions, the __cause__
attribute for explicitly chained exceptions, and the __traceback__
attribute for the traceback. A new raise ... from statement sets the
__cause__ attribute.

Motivation

During the handling of one exception (exception A), it is possible that
another exception (exception B) may occur. In today's Python (version
2.4), if this happens, exception B is propagated outward and exception A
is lost. In order to debug the problem, it is useful to know about both
exceptions. The __context__ attribute retains this information
automatically.

Sometimes it can be useful for an exception handler to intentionally
re-raise an exception, either to provide extra information or to
translate an exception to another type. The __cause__ attribute provides
an explicit way to record the direct cause of an exception.

In today's Python implementation, exceptions are composed of three
parts: the type, the value, and the traceback. The sys module, exposes
the current exception in three parallel variables, exc_type, exc_value,
and exc_traceback, the sys.exc_info() function returns a tuple of these
three parts, and the raise statement has a three-argument form accepting
these three parts. Manipulating exceptions often requires passing these
three things in parallel, which can be tedious and error-prone.
Additionally, the except statement can only provide access to the value,
not the traceback. Adding the __traceback__ attribute to exception
values makes all the exception information accessible from a single
place.

History

Raymond Hettinger[1] raised the issue of masked exceptions on Python-Dev
in January 2003 and proposed a PyErr_FormatAppend() function that C
modules could use to augment the currently active exception with more
information. Brett Cannon[2] brought up chained exceptions again in June
2003, prompting a long discussion.

Greg Ewing[3] identified the case of an exception occurring in a finally
block during unwinding triggered by an original exception, as distinct
from the case of an exception occurring in an except block that is
handling the original exception.

Greg Ewing[4] and Guido van Rossum[5], and probably others, have
previously mentioned adding a traceback attribute to Exception
instances. This is noted in PEP 3000.

This PEP was motivated by yet another recent Python-Dev reposting of the
same ideas[6][7].

Rationale

The Python-Dev discussions revealed interest in exception chaining for
two quite different purposes. To handle the unexpected raising of a
secondary exception, the exception must be retained implicitly. To
support intentional translation of an exception, there must be a way to
chain exceptions explicitly. This PEP addresses both.

Several attribute names for chained exceptions have been suggested on
Python-Dev[8], including cause, antecedent, reason, original, chain,
chainedexc, xc_chain, excprev, previous and precursor. For an explicitly
chained exception, this PEP suggests __cause__ because of its specific
meaning. For an implicitly chained exception, this PEP proposes the name
__context__ because the intended meaning is more specific than temporal
precedence but less specific than causation: an exception occurs in the
context of handling another exception.

This PEP suggests names with leading and trailing double-underscores for
these three attributes because they are set by the Python VM. Only in
very special cases should they be set by normal assignment.

This PEP handles exceptions that occur during except blocks and finally
blocks in the same way. Reading the traceback makes it clear where the
exceptions occurred, so additional mechanisms for distinguishing the two
cases would only add unnecessary complexity.

This PEP proposes that the outermost exception object (the one exposed
for matching by except clauses) be the most recently raised exception
for compatibility with current behaviour.

This PEP proposes that tracebacks display the outermost exception last,
because this would be consistent with the chronological order of
tracebacks (from oldest to most recent frame) and because the actual
thrown exception is easier to find on the last line.

To keep things simpler, the C API calls for setting an exception will
not automatically set the exception's __context__. Guido van Rossum has
expressed concerns with making such changes[9].

As for other languages, Java and Ruby both discard the original
exception when another exception occurs in a catch/rescue or
finally/ensure clause. Perl 5 lacks built-in structured exception
handling. For Perl 6, RFC number 88[10] proposes an exception mechanism
that implicitly retains chained exceptions in an array named @@. In that
RFC, the most recently raised exception is exposed for matching, as in
this PEP; also, arbitrary expressions (possibly involving @@) can be
evaluated for exception matching.

Exceptions in C# contain a read-only InnerException property that may
point to another exception. Its documentation[11] says that "When an
exception X is thrown as a direct result of a previous exception Y, the
InnerException property of X should contain a reference to Y." This
property is not set by the VM automatically; rather, all exception
constructors take an optional innerException argument to set it
explicitly. The __cause__ attribute fulfills the same purpose as
InnerException, but this PEP proposes a new form of raise rather than
extending the constructors of all exceptions. C# also provides a
GetBaseException method that jumps directly to the end of the
InnerException chain; this PEP proposes no analog.

The reason all three of these attributes are presented together in one
proposal is that the __traceback__ attribute provides convenient access
to the traceback on chained exceptions.

Implicit Exception Chaining

Here is an example to illustrate the __context__ attribute:

    def compute(a, b):
        try:
            a/b
        except Exception, exc:
            log(exc)

    def log(exc):
        file = open('logfile.txt')  # oops, forgot the 'w'
        print >>file, exc
        file.close()

Calling compute(0, 0) causes a ZeroDivisionError. The compute() function
catches this exception and calls log(exc), but the log() function also
raises an exception when it tries to write to a file that wasn't opened
for writing.

In today's Python, the caller of compute() gets thrown an IOError. The
ZeroDivisionError is lost. With the proposed change, the instance of
IOError has an additional __context__ attribute that retains the
ZeroDivisionError.

The following more elaborate example demonstrates the handling of a
mixture of finally and except clauses:

    def main(filename):
        file = open(filename)       # oops, forgot the 'w'
        try:
            try:
                compute()
            except Exception, exc:
                log(file, exc)
            finally:
                file.clos()         # oops, misspelled 'close'

    def compute():
        1/0

    def log(file, exc):
        try:
            print >>file, exc       # oops, file is not writable
        except:
            display(exc)

    def display(exc):
        print ex                    # oops, misspelled 'exc'

Calling main() with the name of an existing file will trigger four
exceptions. The ultimate result will be an AttributeError due to the
misspelling of clos, whose __context__ points to a NameError due to the
misspelling of ex, whose __context__ points to an IOError due to the
file being read-only, whose __context__ points to a ZeroDivisionError,
whose __context__ attribute is None.

The proposed semantics are as follows:

1.  Each thread has an exception context initially set to None.
2.  Whenever an exception is raised, if the exception instance does not
    already have a __context__ attribute, the interpreter sets it equal
    to the thread's exception context.
3.  Immediately after an exception is raised, the thread's exception
    context is set to the exception.
4.  Whenever the interpreter exits an except block by reaching the end
    or executing a return, yield, continue, or break statement, the
    thread's exception context is set to None.

Explicit Exception Chaining

The __cause__ attribute on exception objects is always initialized to
None. It is set by a new form of the raise statement:

    raise EXCEPTION from CAUSE

which is equivalent to:

    exc = EXCEPTION
    exc.__cause__ = CAUSE
    raise exc

In the following example, a database provides implementations for a few
different kinds of storage, with file storage as one kind. The database
designer wants errors to propagate as DatabaseError objects so that the
client doesn't have to be aware of the storage-specific details, but
doesn't want to lose the underlying error information:

    class DatabaseError(StandardError):
        pass

    class FileDatabase(Database):
        def __init__(self, filename):
            try:
                self.file = open(filename)
            except IOError, exc:
                raise DatabaseError('failed to open') from exc

If the call to open() raises an exception, the problem will be reported
as a DatabaseError, with a __cause__ attribute that reveals the IOError
as the original cause.

Traceback Attribute

The following example illustrates the __traceback__ attribute:

    def do_logged(file, work):
        try:
            work()
        except Exception, exc:
            write_exception(file, exc)
            raise exc

    from traceback import format_tb

    def write_exception(file, exc):
        ...
        type = exc.__class__
        message = str(exc)
        lines = format_tb(exc.__traceback__)
        file.write(... type ... message ... lines ...)
        ...

In today's Python, the do_logged() function would have to extract the
traceback from sys.exc_traceback or sys.exc_info()[12] and pass both the
value and the traceback to write_exception(). With the proposed change,
write_exception() simply gets one argument and obtains the exception
using the __traceback__ attribute.

The proposed semantics are as follows:

1.  Whenever an exception is caught, if the exception instance does not
    already have a __traceback__ attribute, the interpreter sets it to
    the newly caught traceback.

Enhanced Reporting

The default exception handler will be modified to report chained
exceptions. The chain of exceptions is traversed by following the
__cause__ and __context__ attributes, with __cause__ taking priority. In
keeping with the chronological order of tracebacks, the most recently
raised exception is displayed last; that is, the display begins with the
description of the innermost exception and backs up the chain to the
outermost exception. The tracebacks are formatted as usual, with one of
the lines:

    The above exception was the direct cause of the following exception:

or

    During handling of the above exception, another exception occurred:

between tracebacks, depending whether they are linked by __cause__ or
__context__ respectively. Here is a sketch of the procedure:

    def print_chain(exc):
        if exc.__cause__:
            print_chain(exc.__cause__)
            print '\nThe above exception was the direct cause...'
        elif exc.__context__:
            print_chain(exc.__context__)
            print '\nDuring handling of the above exception, ...'
        print_exc(exc)

In the traceback module, the format_exception, print_exception,
print_exc, and print_last functions will be updated to accept an
optional chain argument, True by default. When this argument is True,
these functions will format or display the entire chain of exceptions as
just described. When it is False, these functions will format or display
only the outermost exception.

The cgitb module should also be updated to display the entire chain of
exceptions.

C API

The PyErr_Set* calls for setting exceptions will not set the __context__
attribute on exceptions. PyErr_NormalizeException will always set the
traceback attribute to its tb argument and the __context__ and __cause__
attributes to None.

A new API function, PyErr_SetContext(context), will help C programmers
provide chained exception information. This function will first
normalize the current exception so it is an instance, then set its
__context__ attribute. A similar API function, PyErr_SetCause(cause),
will set the __cause__ attribute.

Compatibility

Chained exceptions expose the type of the most recent exception, so they
will still match the same except clauses as they do now.

The proposed changes should not break any code unless it sets or uses
attributes named __context__, __cause__, or __traceback__ on exception
instances. As of 2005-05-12, the Python standard library contains no
mention of such attributes.

Open Issue: Extra Information

Walter Dörwald[13] expressed a desire to attach extra information to an
exception during its upward propagation without changing its type. This
could be a useful feature, but it is not addressed by this PEP. It could
conceivably be addressed by a separate PEP establishing conventions for
other informational attributes on exceptions.

Open Issue: Suppressing Context

As written, this PEP makes it impossible to suppress __context__, since
setting exc.__context__ to None in an except or finally clause will only
result in it being set again when exc is raised.

Open Issue: Limiting Exception Types

To improve encapsulation, library implementors may want to wrap all
implementation-level exceptions with an application-level exception. One
could try to wrap exceptions by writing this:

    try:
        ... implementation may raise an exception ...
    except:
        import sys
        raise ApplicationError from sys.exc_value

or this

    try:
        ... implementation may raise an exception ...
    except Exception, exc:
        raise ApplicationError from exc

but both are somewhat flawed. It would be nice to be able to name the
current exception in a catch-all except clause, but that isn't addressed
here. Such a feature would allow something like this:

    try:
        ... implementation may raise an exception ...
    except *, exc:
        raise ApplicationError from exc

Open Issue: yield

The exception context is lost when a yield statement is executed;
resuming the frame after the yield does not restore the context.
Addressing this problem is out of the scope of this PEP; it is not a new
problem, as demonstrated by the following example:

    >>> def gen():
    ...     try:
    ...         1/0
    ...     except:
    ...         yield 3
    ...         raise
    ...
    >>> g = gen()
    >>> g.next()
    3
    >>> g.next()
    TypeError: exceptions must be classes, instances, or strings
    (deprecated), not NoneType

Open Issue: Garbage Collection

The strongest objection to this proposal has been that it creates cycles
between exceptions and stack frames[14]. Collection of cyclic garbage
(and therefore resource release) can be greatly delayed:

    >>> try:
    >>>     1/0
    >>> except Exception, err:
    >>>     pass

will introduce a cycle from err -> traceback -> stack frame -> err,
keeping all locals in the same scope alive until the next GC happens.

Today, these locals would go out of scope. There is lots of code which
assumes that "local" resources -- particularly open files -- will be
closed quickly. If closure has to wait for the next GC, a program (which
runs fine today) may run out of file handles.

Making the __traceback__ attribute a weak reference would avoid the
problems with cyclic garbage. Unfortunately, it would make saving the
Exception for later (as unittest does) more awkward, and it would not
allow as much cleanup of the sys module.

A possible alternate solution, suggested by Adam Olsen, would be to
instead turn the reference from the stack frame to the err variable into
a weak reference when the variable goes out of scope[15].

Possible Future Compatible Changes

These changes are consistent with the appearance of exceptions as a
single object rather than a triple at the interpreter level.

-   If PEP 340 or PEP 343 is accepted, replace the three (type, value,
    traceback) arguments to __exit__ with a single exception argument.
-   Deprecate sys.exc_type, sys.exc_value, sys.exc_traceback, and
    sys.exc_info() in favour of a single member, sys.exception.
-   Deprecate sys.last_type, sys.last_value, and sys.last_traceback in
    favour of a single member, sys.last_exception.
-   Deprecate the three-argument form of the raise statement in favour
    of the one-argument form.
-   Upgrade cgitb.html() to accept a single value as its first argument
    as an alternative to a (type, value, traceback) tuple.

Possible Future Incompatible Changes

These changes might be worth considering for Python 3000.

-   Remove sys.exc_type, sys.exc_value, sys.exc_traceback, and
    sys.exc_info().
-   Remove sys.last_type, sys.last_value, and sys.last_traceback.
-   Replace the three-argument sys.excepthook with a one-argument API,
    and changing the cgitb module to match.
-   Remove the three-argument form of the raise statement.
-   Upgrade traceback.print_exception to accept an exception argument
    instead of the type, value, and traceback arguments.

Acknowledgements

Brett Cannon, Greg Ewing, Guido van Rossum, Jeremy Hylton, Phillip J.
Eby, Raymond Hettinger, Walter Dörwald, and others.

References

Copyright

This document has been placed in the public domain.

[1] Raymond Hettinger, "Idea for avoiding exception masking"
https://mail.python.org/pipermail/python-dev/2003-January/032492.html

[2] Brett Cannon explains chained exceptions
https://mail.python.org/pipermail/python-dev/2003-June/036063.html

[3] Greg Ewing points out masking caused by exceptions during finally
https://mail.python.org/pipermail/python-dev/2003-June/036290.html

[4] Greg Ewing suggests storing the traceback in the exception object
https://mail.python.org/pipermail/python-dev/2003-June/036092.html

[5] Guido van Rossum mentions exceptions having a traceback attribute
https://mail.python.org/pipermail/python-dev/2005-April/053060.html

[6] Ka-Ping Yee, "Tidier Exceptions"
https://mail.python.org/pipermail/python-dev/2005-May/053671.html

[7] Ka-Ping Yee, "Chained Exceptions"
https://mail.python.org/pipermail/python-dev/2005-May/053672.html

[8] Brett Cannon explains chained exceptions
https://mail.python.org/pipermail/python-dev/2003-June/036063.html

[9] Guido van Rossum discusses automatic chaining in PyErr_Set*
https://mail.python.org/pipermail/python-dev/2003-June/036180.html

[10] Tony Olensky, "Omnibus Structured Exception/Error Handling
Mechanism" http://dev.perl.org/perl6/rfc/88.html

[11] MSDN .NET Framework Library, "Exception.InnerException Property"
http://msdn.microsoft.com/library/en-us/cpref/html/frlrfsystemexceptionclassinnerexceptiontopic.asp

[12] Brett Cannon explains chained exceptions
https://mail.python.org/pipermail/python-dev/2003-June/036063.html

[13] Walter Dörwald suggests wrapping exceptions to add details
https://mail.python.org/pipermail/python-dev/2003-June/036148.html

[14] Guido van Rossum restates the objection to cyclic trash
https://mail.python.org/pipermail/python-3000/2007-January/005322.html

[15] Adam Olsen suggests using a weakref from stack frame to exception
https://mail.python.org/pipermail/python-3000/2007-January/005363.html


text:
PEP: 242 Title: Numeric Kinds Author: Paul F. Dubois<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track Created: 17-Mar-2001
Python-Version: 2.2 Post-History: 17-Apr-2001

The kinds module will not be added to the standard library.

There was no opposition to the proposal but only mild interest in using
it, not enough to justify adding the module to the standard library.
Instead, it will be made available as a separate distribution item at
the Numerical Python site. At the next release of Numerical Python, it
will no longer be a part of the Numeric distribution.

Abstract

This proposal gives the user optional control over the precision and
range of numeric computations so that a computation can be written once
and run anywhere with at least the desired precision and range. It is
backward compatible with existing code. The meaning of decimal literals
is clarified.

Rationale

Currently it is impossible in every language except Fortran 90 to write
a program in a portable way that uses floating point and gets roughly
the same answer regardless of platform -- or refuses to compile if that
is not possible. Python currently has only one floating point type,
equal to a C double in the C implementation.

No type exists corresponding to single or quad floats. It would
complicate the language to try to introduce such types directly and
their subsequent use would not be portable. This proposal is similar to
the Fortran 90 "kind" solution, adapted to the Python environment. With
this facility an entire calculation can be switched from one level of
precision to another by changing a single line. If the desired precision
does not exist on a particular machine, the program will fail rather
than get the wrong answer. Since coding in this style would involve an
early call to the routine that will fail, this is the next best thing to
not compiling.

Supported Kinds of Ints and Floats

Complex numbers are treated separately below, since Python can be built
without them.

Each Python compiler may define as many "kinds" of integer and floating
point numbers as it likes, except that it must support at least two
kinds of integer corresponding to the existing int and long, and must
support at least one kind of floating point number, equivalent to the
present float.

The range and precision of these required kinds are processor dependent,
as at present, except for the "long integer" kind, which can hold an
arbitrary integer.

The built-in functions int(), long(), and float() convert inputs to
these default kinds as they do at present. (Note that a Unicode string
is actually a different "kind" of string and that a sufficiently
knowledgeable person might be able to expand this PEP to cover that
case.)

Within each type (integer, floating) the compiler supports a
linearly-ordered set of kinds, with the ordering determined by the
ability to hold numbers of an increased range and/or precision.

Kind Objects

Two new standard functions are defined in a module named "kinds". They
return callable objects called kind objects. Each int or floating kind
object f has the signature result = f(x), and each complex kind object
has the signature result = f(x, y=0.).

int_kind(n)

    For an integer argument n >= 1, return a callable object whose
    result is an integer kind that will hold an integer number in the
    open interval (-10**n, 10**n). The kind object accepts arguments
    that are integers including longs. If n == 0, returns the kind
    object corresponding to the Python literal 0.

float_kind(nd, n)

    For nd >= 0 and n >= 1, return a callable object whose result is a
    floating point kind that will hold a floating-point number with at
    least nd digits of precision and a base-10 exponent in the closed
    interval [-n, n]. The kind object accepts arguments that are integer
    or float.

    If nd and n are both zero, returns the kind object corresponding to
    the Python literal 0.0.

The compiler will return a kind object corresponding to the least of its
available set of kinds for that type that has the desired properties. If
no kind with the desired qualities exists in a given implementation an
OverflowError exception is thrown. A kind function converts its argument
to the target kind, but if the result does not fit in the target kind's
range, an OverflowError exception is thrown.

Besides their callable behavior, kind objects have attributes giving the
traits of the kind in question.

1.  name is the name of the kind. The standard kinds are called int,
    long, double.
2.  typecode is a single-letter string that would be appropriate for use
    with Numeric or module array to form an array of this kind. The
    standard types' typecodes are 'i', 'O', 'd' respectively.
3.  Integer kinds have these additional attributes: MAX, equal to the
    maximum permissible integer of this kind, or None for the long kind.
    MIN, equal to the most negative permissible integer of this kind, or
    None for the long kind.
4.  Float kinds have these additional attributes whose properties are
    equal to the corresponding value for the corresponding C type in the
    standard header file "float.h". MAX, MIN, DIG, MANT_DIG, EPSILON,
    MAX_EXP, MAX_10_EXP, MIN_EXP, MIN_10_EXP, RADIX, ROUNDS (==
    FLT_RADIX, FLT_ROUNDS in float.h). These values are of type integer
    except for MAX, MIN, and EPSILON, which are of the Python floating
    type to which the kind corresponds.

Attributes of Module kinds

int_kinds is a list of the available integer kinds, sorted from lowest
to highest kind. By definition, int_kinds[-1] is the long kind.

float_kinds is a list of the available floating point kinds, sorted from
lowest to highest kind.

default_int_kind is the kind object corresponding to the Python literal
0

default_long_kind is the kind object corresponding to the Python literal
0L

default_float_kind is the kind object corresponding to the Python
literal 0.0

Complex Numbers

If supported, complex numbers have real and imaginary parts that are
floating-point numbers with the same kind. A Python compiler must
support a complex analog of each floating point kind it supports, if it
supports complex numbers at all.

If complex numbers are supported, the following are available in module
kinds:

complex_kind(nd, n)

    Return a callable object whose result is a complex kind that will
    hold a complex number each of whose components (.real, .imag) is of
    kind float_kind(nd, n). The kind object will accept one argument
    that is of any integer, real, or complex kind, or two arguments,
    each integer or real.

complex_kinds is a list of the available complex kinds, sorted from
lowest to highest kind.

default_complex_kind is the kind object corresponding to the Python
literal 0.0j. The name of this kind is doublecomplex, and its typecode
is 'D'.

Complex kind objects have these addition attributes:

floatkind is the kind object of the corresponding float type.

Examples

In module myprecision.py:

    import kinds
    tinyint = kinds.int_kind(1)
    single = kinds.float_kind(6, 90)
    double = kinds.float_kind(15, 300)
    csingle = kinds.complex_kind(6, 90)

In the rest of my code:

    from myprecision import tinyint, single, double, csingle
    n = tinyint(3)
    x = double(1.e20)
    z = 1.2
    # builtin float gets you the default float kind, properties unknown
    w = x * float(x)
    # but in the following case we know w has kind "double".
    w = x * double(z)

    u = csingle(x + z * 1.0j)
    u2 = csingle(x+z, 1.0)

Note how that entire code can then be changed to a higher precision by
changing the arguments in myprecision.py.

Comment: note that you aren't promised that single != double; but you
are promised that double(1.e20) will hold a number with 15 decimal
digits of precision and a range up to 10**300 or that the float_kind
call will fail.

Open Issues

No open issues have been raised at this time.

Copyright

This document has been placed in the public domain.