text:
PEP: 677 Title: Callable Type Syntax Author: Steven Troxler
<EMAIL_ADDRESS>Pradeep Kumar Srinivasan
<EMAIL_ADDRESS>Sponsor: Guido van Rossum <guido at python.org>
Discussions-To<EMAIL_ADDRESS>Status: Rejected Type: Standards
Track Topic: Typing Content-Type: text/x-rst Created: 13-Dec-2021
Python-Version: 3.11 Post-History: 16-Dec-2021 Resolution:
<EMAIL_ADDRESS>
Abstract

This PEP introduces a concise and friendly syntax for callable types,
supporting the same functionality as typing.Callable but with an arrow
syntax inspired by the syntax for typed function signatures. This allows
types like Callable[[int, str], bool] to be written as
(int, str) -> bool.

The proposed syntax supports all the functionality provided by
typing.Callable and typing.Concatenate, and is intended to work as a
drop-in replacement.

Motivation

One way to make code safer and easier to analyze is by making sure that
functions and classes are well-typed. In Python we have type
annotations, the framework for which is defined in PEP 484, to provide
type hints that can find bugs as well as helping with editor tooling
like tab completion, static analysis tooling, and code review.

Consider the following untyped code:

    def flat_map(func, l):
        out = []
        for element in l:
            out.extend(func(element))
        return out

    def wrap(x: int) -> list[int]:
        return [x]

    def add(x: int, y: int) -> int:
        return x + y

    flat_map(wrap, [1, 2, 3])  # no runtime error, output is [1, 2, 3]
    flat_map(add, [1, 2, 3])   # runtime error: `add` expects 2 arguments, got 1

We can add types to this example to detect the runtime error:

    from typing import Callable

    def flat_map(
        func: Callable[[int], list[int]],
        l: list[int]
    ) -> list[int]:
        ....

    ...

    flat_map(wrap, [1, 2, 3])  # type checks okay, output is [1, 2, 3]
    flat_map(add, [1, 2, 3])   # type check error

There are a few usability challenges with Callable we can see here:

-   It is verbose, particularly for more complex function signatures.
-   It relies on two levels of nested brackets, unlike any other generic
    type. This can be especially hard to read when some of the type
    parameters are themselves generic types.
-   The bracket structure is not visually similar to how function
    signatures are written.
-   It requires an explicit import, unlike many of the other most common
    types like list and dict.

Possibly as a result, programmers often fail to write complete Callable
types. Such untyped or partially-typed callable types do not check the
parameter types or return types of the given callable and thus negate
the benefits of static typing. For example, they might write this:

    from typing import Callable

    def flat_map(
        func: Callable[..., Any],
        l: list[int]
    ) -> list[int]:
        ....

    ...

    flat_map(add, [1, 2, 3])  # oops, no type check error!

There's some partial type information here - we at least know that func
needs to be callable. But we've dropped too much type information for
type checkers to find the bug.

With our proposal, the example looks like this:

    def flat_map(
        func: (int) -> list[int],
        l: list[int]
    ) -> list[int]:
        out = []
        for element in l:
            out.extend(f(element))
        return out

    ...

The type (int) -> list[int] is more concise, uses an arrow similar to
the one indicating a return type in a function header, avoids nested
brackets, and does not require an import.

Rationale

The Callable type is widely used. For example, as of October 2021 it was
the fifth most common complex type in typeshed, after Optional, Tuple,
Union, and List.

The others have had their syntax improved and the need for imports
eliminated by either PEP 604 or PEP 585:

-   typing.Optional[int] is written int | None
-   typing.Union[int, str] is written int | str
-   typing.List[int] is written list[int]
-   typing.Tuple[int, str] is written tuple[int, str]

The typing.Callable type is used almost as often as these other types,
is more complicated to read and write, and still requires an import and
bracket-based syntax.

In this proposal, we chose to support all the existing semantics of
typing.Callable, without adding support for new features. We made this
decision after examining how frequently each feature might be used in
existing typed and untyped open-source code. We determined that the vast
majority of use cases are covered.

We considered adding support for named, optional, and variadic
arguments. However, we decided against including these features, as our
analysis showed they are infrequently used. When they are really needed,
it is possible to type these using callback protocols.

An Arrow Syntax for Callable Types

We are proposing a succinct, easy-to-use syntax for typing.Callable that
looks similar to function headers in Python. Our proposal closely
follows syntax used by several popular languages such as Typescript,
Kotlin, and Scala.

Our goals are that:

-   Callable types using this syntax will be easier to learn and use,
    particularly for developers with experience in other languages.
-   Library authors will be more likely to use expressive types for
    callables that enable type checkers to better understand code and
    find bugs, as in the decorator example above.

Consider this simplified real-world example from a web server, written
using the existing typing.Callable:

    from typing import Awaitable, Callable
    from app_logic import Response, UserSetting

    def customize_response(
        response: Response,
        customizer: Callable[[Response, list[UserSetting]], Awaitable[Response]]
    ) -> Response:
       ...

With our proposal, this code can be abbreviated to:

    from app_logic import Response, UserSetting

    def customize_response(
        response: Response,
        customizer: async (Response, list[UserSetting]) -> Response,
    ) -> Response:
        ...

This is shorter and requires fewer imports. It also has far less nesting
of square brackets - only one level, as opposed to three in the original
code.

Compact Syntax for ParamSpec

A particularly common case where library authors leave off type
information for callables is when defining decorators. Consider the
following:

    from typing import Any, Callable

    def with_retries(
        f: Callable[..., Any]
    ) -> Callable[..., Any]:
        def wrapper(retry_once, *args, **kwargs):
            if retry_once:
                try: return f(*args, **kwargs)
                except Exception: pass
            return f(*args, **kwargs)
        return wrapper

    @with_retries
    def f(x: int) -> int:
        return x

    f(y=10)  # oops - no type error!

In the code above, it is clear that the decorator should produce a
function whose signature is like that of the argument f other than an
additional retry_once argument. But the use of ... prevents a type
checker from seeing this and alerting a user that f(y=10) is invalid.

With PEP 612 it is possible to type decorators like this correctly as
follows:

    from typing import Any, Callable, Concatenate, ParamSpec, TypeVar

    R = TypeVar("R")
    P = ParamSpec("P")

    def with_retries(
        f: Callable[P, R]
    ) -> Callable[Concatenate[bool, P] R]:
        def wrapper(retry_once: bool, *args: P.args, **kwargs: P.kwargs) -> R:
            ...
        return wrapper

    ...

With our proposed syntax, the properly-typed decorator example becomes
concise and the type representations are visually descriptive:

    from typing import Any, ParamSpec, TypeVar

    R = TypeVar("R")
    P = ParamSpec("P")

    def with_retries(
        f: (**P) -> R
    ) -> (bool, **P) -> R:
        ...

Comparing to Other Languages

Many popular programming languages use an arrow syntax similar to the
one we are proposing here.

TypeScript

In TypeScript, function types are expressed in a syntax almost the same
as the one we are proposing, but the arrow token is => and arguments
have names:

    (x: int, y: str) => bool

The names of the arguments are not actually relevant to the type. So,
for example, this is the same callable type:

    (a: int, b: str) => bool

Kotlin

Function types in Kotlin permit an identical syntax to the one we are
proposing, for example:

    (Int, String) -> Bool

It also optionally allows adding names to the arguments, for example:

    (x: Int, y: String) -> Bool

As in TypeScript, the argument names (if provided) are just there for
documentation and are not part of the type itself.

Scala

Scala uses the => arrow for function types. Other than that, their
syntax is the same as the one we are proposing, for example:

    (Int, String) => Bool

Scala, like Python, has the ability to provide function arguments by
name. Function types can optionally include names, for example:

    (x: Int, y: String) => Bool

Unlike in TypeScript and Kotlin, these names are part of the type if
provided - any function implementing the type must use the same names.
This is similar to the extended syntax proposal we describe in our
Rejected Alternatives section.

Function Definitions vs Callable Type Annotations

In all of the languages listed above, type annotations for function
definitions use a : rather than a ->. For example, in TypeScript a
simple add function looks like this:

    function higher_order(fn: (a: string) => string): string {
      return fn("Hello, World");
    }

Scala and Kotlin use essentially the same : syntax for return
annotations. The : makes sense in these languages because they all use :
for type annotations of parameters and variables, and the use for
function return types is similar.

In Python we use : to denote the start of a function body and -> for
return annotations. As a result, even though our proposal is
superficially the same as these other languages the context is
different. There is potential for more confusion in Python when reading
function definitions that include callable types.

This is a key concern for which we are seeking feedback with our draft
PEP; one idea we have floated is to use => instead to make it easier to
differentiate.

The ML Language Family

Languages in the ML family, including F#, OCaml, and Haskell, all use ->
to represent function types. All of them use a parentheses-free syntax
with multiple arrows, for example in Haskell:

    Integer -> String -> Bool

The use of multiple arrows, which differs from our proposal, makes sense
for languages in this family because they use automatic currying of
function arguments, which means that a multi-argument function behaves
like a single-argument function returning a function.

Specification

Typing Behavior

Type checkers should treat the new syntax with exactly the same
semantics as typing.Callable.

As such, a type checker should treat the following pairs exactly the
same:

    from typing import Awaitable, Callable, Concatenate, ParamSpec, TypeVarTuple

    P = ParamSpec("P")
    Ts = TypeVarTuple('Ts')

    f0: () -> bool
    f0: Callable[[], bool]

    f1: (int, str) -> bool
    f1: Callable[[int, str], bool]

    f2: (...) -> bool
    f2: Callable[..., bool]

    f3: async (str) -> str
    f3: Callable[[str], Awaitable[str]]

    f4: (**P) -> bool
    f4: Callable[P, bool]

    f5: (int, **P) -> bool
    f5: Callable[Concatenate[int, P], bool]

    f6: (*Ts) -> bool
    f6: Callable[[*Ts], bool]

    f7: (int, *Ts, str) -> bool
    f7: Callable[[int, *Ts, str], bool]

Grammar and AST

The proposed new syntax can be described by these AST changes to
Parser/Python.asdl:

    expr = <prexisting_expr_kinds>
         | AsyncCallableType(callable_type_arguments args, expr returns)
         | CallableType(callable_type_arguments args, expr returns)

    callable_type_arguments = AnyArguments
                            | ArgumentsList(expr* posonlyargs)
                            | Concatenation(expr* posonlyargs, expr param_spec)

Here are our proposed changes to the `Python Grammar
<https://docs.python.org/3/reference/grammar.htm>`:

    expression:
        | disjunction disjunction 'else' expression
        | callable_type_expression
        | disjunction
        | lambdef

    callable_type_expression:
        | callable_type_arguments '->' expression
        | ASYNC callable_type_arguments '->' expression

    callable_type_arguments:
        | '(' '...' [','] ')'
        | '(' callable_type_positional_argument*  ')'
        | '(' callable_type_positional_argument* callable_type_param_spec ')'

    callable_type_positional_argument:
        | !'...' expression ','
        | !'...' expression &')'

    callable_type_param_spec:
        | '**' expression ','
        | '**' expression &')'

If PEP 646 is accepted, we intend to include support for unpacked types
in two ways. To support the "star-for-unpack" syntax proposed in PEP
646, we will modify the grammar for callable_type_positional_argument as
follows:

    callable_type_positional_argument:
        | !'...' expression ','
        | !'...' expression &')'
        | '*' expression ','
        | '*' expression &')'

With this change, a type of the form (int, *Ts) -> bool should evaluate
the AST form:

    CallableType(
        ArgumentsList(Name("int"), Starred(Name("Ts")),
        Name("bool")
    )

and be treated by type checkers as equivalent to or
Callable[[int, *Ts], bool] or Callable[[int, Unpack[Ts]], bool].

Implications of the Grammar

Precedence of ->

-> binds less tightly than other operators, both inside types and in
function signatures, so the following two callable types are equivalent:

    (int) -> str | bool
    (int) -> (str | bool)

-> associates to the right, both inside types and in function
signatures. So the following pairs are equivalent:

    (int) -> (str) -> bool
    (int) -> ((str) -> bool)

    def f() -> (int, str) -> bool: pass
    def f() -> ((int, str) -> bool): pass

    def f() -> (int) -> (str) -> bool: pass
    def f() -> ((int) -> ((str) -> bool)): pass

Because operators bind more tightly than ->, parentheses are required
whenever an arrow type is intended to be inside an argument to an
operator like |:

    (int) -> () -> int | () -> bool      # syntax error!
    (int) -> (() -> int) | (() -> bool)  # okay

We discussed each of these behaviors and believe they are desirable:

-   Union types (represented by A | B according to PEP 604) are valid in
    function signature returns, so we need to allow operators in the
    return position for consistency.
-   Given that operators bind more tightly than -> it is correct that a
    type like bool | () -> bool must be a syntax error. We should be
    sure the error message is clear because this may be a common
    mistake.
-   Associating -> to the right, rather than requiring explicit
    parentheses, is consistent with other languages like TypeScript and
    respects the principle that valid expressions should normally be
    substitutable when possible.

async Keyword

All of the binding rules still work for async callable types:

    (int) -> async (float) -> str | bool
    (int) -> (async (float) -> (str | bool))

    def f() -> async (int, str) -> bool: pass
    def f() -> (async (int, str) -> bool): pass

    def f() -> async (int) -> async (str) -> bool: pass
    def f() -> (async (int) -> (async (str) -> bool)): pass

Trailing Commas

-   Following the precedent of function signatures, putting a comma in
    an empty arguments list is illegal: (,) -> bool is a syntax error.

-   Again following precedent, trailing commas are otherwise always
    permitted:

        ((int,) -> bool == (int) -> bool
        ((int, **P,) -> bool == (int, **P) -> bool
        ((...,) -> bool) == ((...) -> bool)

Allowing trailing commas also gives autoformatters more flexibility when
splitting callable types across lines, which is always legal following
standard python whitespace rules.

Disallowing ... as an Argument Type

Under normal circumstances, any valid expression is permitted where we
want a type annotation and ... is a valid expression. This is never
semantically valid and all type checkers would reject it, but the
grammar would allow it if we did not explicitly prevent this.

Since ... is meaningless as a type and there are usability concerns, our
grammar rules it out and the following is a syntax error:

    (int, ...) -> bool

We decided that there were compelling reasons to do this:

-   The semantics of (...) -> bool are different from (T) -> bool for
    any valid type T: (...) is a special form indicating AnyArguments
    whereas T is a type parameter in the arguments list.
-   ... is used as a placeholder default value to indicate an optional
    argument in stubs and callback protocols. Allowing it in the
    position of a type could easily lead to confusion and possibly bugs
    due to typos.
-   In the tuple generic type, we special-case ... to mean "more of the
    same", e.g. a tuple[int, ...] means a tuple with one or more
    integers. We do not use ... in a a similar way in callable types, so
    to prevent misunderstandings it makes sense to prevent this.

Incompatibility with other possible uses of * and **

The use of **P for supporting PEP 612 ParamSpec rules out any future
proposal using a bare **<some_type> to type kwargs. This seems
acceptable because:

-   If we ever do want such a syntax, it would be clearer to require an
    argument name anyway. This would also make the type look more
    similar to a function signature. In other words, if we ever support
    typing kwargs in callable types, we would prefer
    (int, **kwargs: str) rather than (int, **str).
-   PEP 646 unpacking syntax would rule out using *<some_type> for args.
    The kwargs case is similar enough that this rules out a bare
    **<some_type> anyway.

Compatibility with Arrow-Based Lambda Syntax

To the best of our knowledge there is no active discussion of
arrow-style lambda syntax that we are aware of, but it is nonetheless
worth considering what possibilities would be ruled out by adopting this
proposal.

It would be incompatible with this proposal to adopt the same a
parenthesized ->-based arrow syntax for lambdas, e.g. (x, y) -> x + y
for lambda x, y: x + y.

Our view is that if we want arrow syntax for lambdas in the future, it
would be a better choice to use =>, e.g. (x, y) => x + y. Many languages
use the same arrow token for both lambdas and callable types, but Python
is unique in that types are expressions and have to evaluate to runtime
values. Our view is that this merits using separate tokens, and given
the existing use of -> for return types in function signatures it would
be more coherent to use -> for callable types and => for lambdas.

Runtime Behavior

The new AST nodes need to evaluate to runtime types, and we have two
goals for the behavior of these runtime types:

-   They should expose a structured API that is descriptive and powerful
    enough to be compatible with extending the type to include new
    features like named and variadic arguments.
-   They should also expose an API that is backward-compatible with
    typing.Callable.

Evaluation and Structured API

We intend to create new builtin types to which the new AST nodes will
evaluate, exposing them in the types module.

Our plan is to expose a structured API as if they were defined as
follows:

    class CallableType:
        is_async: bool
        arguments: Ellipsis | tuple[CallableTypeArgument]
        return_type: object

    class CallableTypeArgument:
        kind: CallableTypeArgumentKind
        annotation: object

    @enum.global_enum
    class CallableTypeArgumentKind(enum.IntEnum):
        POSITIONAL_ONLY: int = ...
        PARAM_SPEC: int = ...

The evaluation rules are expressed in terms of the following pseudocode:

    def evaluate_callable_type(
        callable_type: ast.CallableType | ast.AsyncCallableType:
    ) -> CallableType:
        return CallableType(
           is_async=isinstance(callable_type, ast.AsyncCallableType),
           arguments=_evaluate_arguments(callable_type.arguments),
           return_type=evaluate_expression(callable_type.returns),
        )

    def _evaluate_arguments(arguments):
        match arguments:
            case ast.AnyArguments():
                return Ellipsis
            case ast.ArgumentsList(posonlyargs):
                return tuple(
                    _evaluate_arg(arg) for arg in args
                )
            case ast.ArgumentsListConcatenation(posonlyargs, param_spec):
                return tuple(
                    *(evaluate_arg(arg) for arg in args),
                    _evaluate_arg(arg=param_spec, kind=PARAM_SPEC)
                )
            if isinstance(arguments, Any
        return Ellipsis

    def _evaluate_arg(arg, kind=POSITIONAL_ONLY):
        return CallableTypeArgument(
            kind=POSITIONAL_ONLY,
            annotation=evaluate_expression(value)
        )

Backward-Compatible API

To get backward compatibility with the existing types.Callable API,
which relies on fields __args__ and __parameters__, we can define them
as if they were written in terms of the following:

    import itertools
    import typing

    def get_args(t: CallableType) -> tuple[object]:
        return_type_arg = (
            typing.Awaitable[t.return_type]
            if t.is_async
            else t.return_type
        )
        arguments = t.arguments
        if isinstance(arguments, Ellipsis):
            argument_args = (Ellipsis,)
        else:
            argument_args = (arg.annotation for arg in arguments)
        return (
            *arguments_args,
            return_type_arg
        )

    def get_parameters(t: CallableType) -> tuple[object]:
        out = []
        for arg in get_args(t):
            if isinstance(arg, typing.ParamSpec):
                out.append(t)
            else:
                out.extend(arg.__parameters__)
        return tuple(out)

Additional Behaviors of types.CallableType

As with the A | B syntax for unions introduced in PEP 604:

-   The __eq__ method should treat equivalent typing.Callable values as
    equal to values constructed using the builtin syntax, and otherwise
    should behave like the __eq__ of typing.Callable.
-   The __repr__ method should produce an arrow syntax representation
    that, when evaluated, gives us back an equal types.CallableType
    instance.

Rejected Alternatives

Many of the alternatives we considered would have been more expressive
than typing.Callable, for example adding support for describing
signatures that include named, optional, and variadic arguments.

To determine which features we most needed to support with a callable
type syntax, we did an extensive analysis of existing projects:

-   stats on the use of the Callable type;
-   stats on how untyped and partially-typed callbacks are actually
    used.

We decided on a simple proposal with improved syntax for the existing
Callable type because the vast majority of callbacks can be correctly
described by the existing typing.Callable semantics:

-   Positional parameters: By far the most important case to handle well
    is simple callable types with positional parameters, such as
    (int, str) -> bool
-   ParamSpec and Concatenate: The next most important feature is good
    support for PEP 612 ParamSpec and Concatenate types like
    (**P) -> bool and (int, **P) -> bool. These are common primarily
    because of the heavy use of decorator patterns in python code.
-   TypeVarTuples: The next most important feature, assuming PEP 646 is
    accepted, is for unpacked types which are common because of cases
    where a wrapper passes along *args to some other function.

Features that other, more complicated proposals would support account
for fewer than 2% of the use cases we found. These are already
expressible using callback protocols, and since they are uncommon we
decided that it made more sense to move forward with a simpler syntax.

Extended Syntax Supporting Named and Optional Arguments

Another alternative was for a compatible but more complex syntax that
could express everything in this PEP but also named, optional, and
variadic arguments. In this “extended” syntax proposal the following
types would have been equivalent:

    class Function(typing.Protocol):
        def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
            ...

    Function = (int, y: float, *, z: bool = ..., **kwargs: str) -> bool

Advantages of this syntax include: - Most of the advantages of the
proposal in this PEP (conciseness, PEP 612 support, etc) -Furthermore,
the ability to handle named, optional, and variadic arguments

We decided against proposing it for the following reasons:

-   The implementation would have been more difficult, and usage stats
    demonstrate that fewer than 3% of use cases would benefit from any
    of the added features.

-   The group that debated these proposals was split down the middle
    about whether these changes are desirable:

    -   On the one hand, they make callable types more expressive. On
        the other hand, they could easily confuse users who have not
        read the full specification of callable type syntax.
    -   We believe the simpler syntax proposed in this PEP, which
        introduces no new semantics and closely mimics syntax in other
        popular languages like Kotlin, Scala, and TypesScript, is much
        less likely to confuse users.

-   We intend to implement the current proposal in a way that is
    forward-compatible with the more complicated extended syntax. If the
    community decides after more experience and discussion that we want
    the additional features, it should be straightforward to propose
    them in the future.

-   Even a full extended syntax cannot replace the use of callback
    protocols for overloads. For example, no closed form of callable
    type could express a function that maps bools to bools and ints to
    floats, like this callback protocol.:

        from typing import overload, Protocol

        class OverloadedCallback(Protocol)

          @overload
          def __call__(self, x: int) -> float: ...

          @overload
          def __call__(self, x: bool) -> bool: ...

          def __call__(self, x: int | bool) -> float | bool: ...

        f: OverloadedCallback = ...
        f(True)  # bool
        f(3)     # float

We confirmed that the current proposal is forward-compatible with
extended syntax by implementing a grammar and AST for this extended
syntax on top of our reference implementation of this PEP's grammar.

Syntax Closer to Function Signatures

One alternative we had floated was a syntax much more similar to
function signatures.

In this proposal, the following types would have been equivalent:

    class Function(typing.Protocol):
        def f(self, x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool:
            ...

    Function = (x: int, /, y: float, *, z: bool = ..., **kwargs: str) -> bool

The benefits of this proposal would have included:

-   Perfect syntactic consistency between signatures and callable types.
-   Support for more features of function signatures (named, optional,
    variadic args) that this PEP does not support.

Key downsides that led us to reject the idea include the following:

-   A large majority of use cases only use positional-only arguments.
    This syntax would be more verbose for that use case, both because of
    requiring argument names and an explicit /, for example
    (int, /) -> bool where our proposal allows (int) -> bool
-   The requirement for explicit / for positional-only arguments has a
    high risk of causing frequent bugs - which often would not be
    detected by unit tests - where library authors would accidentally
    use types with named arguments.
-   Our analysis suggests that support for ParamSpec is key, but the
    scoping rules laid out in PEP 612 would have made this difficult.

Other Proposals Considered

Functions-as-Types

An idea we looked at very early on was to allow using functions as
types. The idea is allowing a function to stand in for its own call
signature, with roughly the same semantics as the __call__ method of
callback protocols:

    def CallableType(
        positional_only: int,
        /,
        named: str,
        *args: float,
        keyword_only: int = ...,
        **kwargs: str
    ) -> bool: ...

    f: CallableType = ...
    f(5, 6.6, 6.7, named=6, x="hello", y="world")  # typechecks as bool

This may be a good idea, but we do not consider it a viable replacement
for callable types:

-   It would be difficult to handle ParamSpec, which we consider a
    critical feature to support.
-   When using functions as types, the callable types are not
    first-class values. Instead, they require a separate, out-of-line
    function definition to define a type alias
-   It would not support more features than callback protocols, and
    seems more like a shorter way to write them than a replacement for
    Callable.

Hybrid keyword-arrow Syntax

In the Rust language, a keyword fn is used to indicate functions in much
the same way as Python's def, and callable types are indicated using a
hybrid arrow syntax Fn(i64, String) -> bool.

We could use the def keyword in callable types for Python, for example
our two-parameter boolean function could be written as
def(int, str) -> bool. But we think this might confuse readers into
thinking def(A, B) -> C is a lambda, particularly because Javascript's
function keyword is used in both named and anonymous functions.

Parenthesis-Free Syntax

We considered a parentheses-free syntax that would have been even more
concise:

    int, str -> bool

We decided against it because this is not visually as similar to
existing function header syntax. Moreover, it is visually similar to
lambdas, which bind names with no parentheses: lambda x, y: x == y.

Requiring Outer Parentheses

A concern with the current proposal is readability, particularly when
callable types are used in return type position which leads to multiple
top-level -> tokens, for example:

    def make_adder() -> (int) -> int:
        return lambda x: x + 1

We considered a few ideas to prevent this by changing rules about
parentheses. One was to move the parentheses to the outside, so that a
two-argument boolean function is written (int, str -> bool). With this
change, the example above becomes:

    def make_adder() -> (int -> int):
        return lambda x: x + 1

This makes the nesting of many examples that are difficult to follow
clear, but we rejected it because

-   Currently in Python commas bind very loosely, which means it might
    be common to misread (int, str -> bool) as a tuple whose first
    element is an int, rather than a two-parameter callable type.
-   It is not very similar to function header syntax, and one of our
    goals was familiar syntax inspired by function headers.
-   This syntax may be more readable for deaply nested callables like
    the one above, but deep nesting is not very common. Encouraging
    extra parentheses around callable types in return position via a
    style guide would have most of the readability benefit without the
    downsides.

We also considered requiring parentheses on both the parameter list and
the outside, e.g. ((int, str) -> bool). With this change, the example
above becomes:

    def make_adder() -> ((int) -> int):
        return lambda x: x + 1

We rejected this change because:

-   The outer parentheses only help readability in some cases, mostly
    when a callable type is used in return position. In many other cases
    they hurt readability rather than helping.
-   We agree that it might make sense to encourage outer parentheses in
    several cases, particularly callable types in function return
    annotations. But
    -   We believe it is more appropriate to encourage this in style
        guides, linters, and autoformatters than to bake it into the
        parser and throw syntax errors.

    -   Moreover, if a type is complicated enough that readability is a
        concern we can always use type aliases, for example:

            IntToIntFunction: (int) -> int

            def make_adder() -> IntToIntFunction:
                return lambda x: x + 1

Making -> bind tighter than |

In order to allow both -> and | tokens in type expressions we had to
choose precedence. In the current proposal, this is a function returning
an optional boolean:

    (int, str) -> bool | None  # equivalent to (int, str) -> (bool | None)

We considered having -> bind tighter so that instead the expression
would parse as ((int, str) -> bool) | None. There are two advantages to
this:

-   It means we no would longer have to treat None | (int, str) -> bool
    as a syntax error.
-   Looking at typeshed today, optional callable arguments are very
    common because using None as a default value is a standard Python
    idiom. Having -> bind tighter would make these easier to write.

We decided against this for a few reasons:

-   The function header def f() -> int | None: ... is legal and
    indicates a function returning an optional int. To be consistent
    with function headers, callable types should do the same.
-   TypeScript is the other popular language we know of that uses both
    -> and | tokens in type expressions, and they have | bind tighter.
    While we do not have to follow their lead, we prefer to do so.
-   We do acknowledge that optional callable types are common and having
    | bind tighter forces extra parentheses, which makes these types
    harder to write. But code is read more often than written, and we
    believe that requiring the outer parentheses for an optional
    callable type like ((int, str) -> bool) | None is preferable for
    readability.

Introducing type-strings

Another idea was adding a new “special string” syntax and putting the
type inside of it, for example t”(int, str) -> bool”. We rejected this
because it is not as readable, and seems out of step with guidance from
the Steering Council on ensuring that type expressions do not diverge
from the rest of Python's syntax.

Improving Usability of the Indexed Callable Type

If we do not want to add new syntax for callable types, we could look at
how to make the existing type easier to read. One proposal would be to
make the builtin callable function indexable so that it could be used as
a type:

    callable[[int, str], bool]

This change would be analogous to PEP 585 that made built in collections
like list and dict usable as types, and would make imports more
convenient, but it wouldn't help readability of the types themselves
much.

In order to reduce the number of brackets needed in complex callable
types, it would be possible to allow tuples for the argument list:

    callable[(int, str), bool]

This actually is a significant readability improvement for
multi-argument functions, but the problem is that it makes callables
with one arguments, which are the most common arity, hard to write:
because (x) evaluates to x, they would have to be written like
callable[(int,), bool]. We find this awkward.

Moreover, none of these ideas help as much with reducing verbosity as
the current proposal, nor do they introduce as strong a visual cue as
the -> between the parameter types and the return type.

Alternative Runtime Behaviors

The hard requirements on our runtime API are that:

-   It must preserve backward compatibility with typing.Callable via
    __args__ and __params__.
-   It must provide a structured API, which should be extensible if in
    the future we try to support named and variadic arguments.

Alternative APIs

We considered having the runtime data types.CallableType use a more
structured API where there would be separate fields for posonlyargs and
param_spec. The current proposal was was inspired by the
inspect.Signature type.

We use "argument" in our field and type names, unlike "parameter" as in
inspect.Signature, in order to avoid confusion with the
callable_type.__parameters__ field from the legacy API that refers to
type parameters rather than callable parameters.

Using the plain return type in __args__ for async types

It is debatable whether we are required to preserve backward
compatibility of __args__ for async callable types like
async (int) -> str. The reason is that one could argue they are not
expressible directly using typing.Callable, and therefore it would be
fine to set __args__ as (int, int) rather than
(int, typing.Awaitable[int]).

But we believe this would be problematic. By preserving the appearance
of a backward-compatible API while actually breaking its semantics on
async types, we would cause runtime type libraries that attempt to
interpret Callable using __args__ to fail silently.

It is for this reason that we automatically wrap the return type in
Awaitable.

Backward Compatibility

This PEP proposes a major syntax improvement over typing.Callable, but
the static semantics are the same.

As such, the only thing we need for backward compatibility is to ensure
that types specified via the new syntax behave the same as equivalent
typing.Callable and typing.Concatenate values they intend to replace.

There is no particular interaction between this proposal and
from __future__ import annotations - just like any other type annotation
it will be unparsed to a string at module import, and
typing.get_type_hints should correctly evaluate the resulting strings in
cases where that is possible.

This is discussed in more detail in the Runtime Behavior section.

Reference Implementation

We have a working implementation of the AST and Grammar with tests
verifying that the grammar proposed here has the desired behaviors.

The runtime behavior is not yet implemented. As discussed in the Runtime
Behavior portion of the spec we have a detailed plan for both a
backward-compatible API and a more structured API in a separate doc
where we are also open to discussion and alternative ideas.

Open Issues

Details of the Runtime API

We have attempted to provide a complete behavior specification in the
Runtime Behavior section of this PEP.

But there are probably more details that we will not realize we need to
define until we build a full reference implementation.

Optimizing SyntaxError messages

The current reference implementation has a fully-functional parser and
all edge cases presented here have been tested.

But there are some known cases where the errors are not as informative
as we would like. For example, because (int, ...) -> bool is illegal but
(int, ...) is a valid tuple, we currently produce a syntax error
flagging the -> as the problem even though the real cause of the error
is using ... as an argument type.

This is not part of the specification per se but is an important detail
to address in our implementation. The solution will likely involve
adding invalid_.* rules to python.gram and customizing error messages.

Resources

Background and History

PEP 484 specifies
<484#suggested-syntax-for-python-2-7-and-straddling-code> a very similar
syntax for function type hint comments for use in code that needs to
work on Python 2.7. For example:

    def f(x, y):
        # type: (int, str) -> bool
        ...

At that time we used indexing operations to specify generic types like
typing.Callable because we decided not to add syntax for types. However,
we have since begun to do so, e.g. with PEP 604.

Maggie proposed better callable type syntax as part of a larger
presentation on typing simplifications at the PyCon Typing Summit 2021.

Steven brought up this proposal on typing-sig. We had several meetings
to discuss alternatives, and this presentation led us to the current
proposal.

Pradeep brought this proposal to python-dev for feedback.

Acknowledgments

Thanks to the following people for their feedback on the PEP and help
planning the reference implementation:

Alex Waygood, Eric Traut, Guido van Rossum, James Hilton-Balfe, Jelle
Zijlstra, Maggie Moss, Tuomas Suutari, Shannon Zhu.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 620 Title: Hide implementation details from the C API Author:
Victor Stinner<EMAIL_ADDRESS>Status: Withdrawn Type: Standards
Track Content-Type: text/x-rst Created: 19-Jun-2020 Python-Version: 3.12

Abstract

Introduce C API incompatible changes to hide implementation details.

Once most implementation details will be hidden, evolution of CPython
internals would be less limited by C API backward compatibility issues.
It will be way easier to add new features.

It becomes possible to experiment with more advanced optimizations in
CPython than just micro-optimizations, like tagged pointers.

Define a process to reduce the number of broken C extensions.

The implementation of this PEP is expected to be done carefully over
multiple Python versions. It already started in Python 3.7 and most
changes are already completed. The Process to reduce the number of
broken C extensions dictates the rhythm.

PEP withdrawn

This PEP was withdrawn by its author since the scope is too broad and
the work is distributed over multiple Python versions, which makes it
difficult to make a decision on the overall PEP. It was split into new
PEPs with narrower and better defined scopes, like PEP 670.

Motivation

The C API blocks CPython evolutions

Adding or removing members of C structures is causing multiple backward
compatibility issues.

Adding a new member breaks the stable ABI (PEP 384), especially for
types declared statically (e.g. static PyTypeObject MyType = {...};). In
Python 3.4, the PEP 442 "Safe object finalization" added the tp_finalize
member at the end of the PyTypeObject structure. For ABI backward
compatibility, a new Py_TPFLAGS_HAVE_FINALIZE type flag was required to
announce if the type structure contains the tp_finalize member. The flag
was removed in Python 3.8 (bpo-32388).

The PyTypeObject.tp_print member, deprecated since Python 3.0 released
in 2009, has been removed in the Python 3.8 development cycle. But the
change broke too many C extensions and had to be reverted before 3.8
final release. Finally, the member was removed again in Python 3.9.

C extensions rely on the ability to access structure members, indirectly
through the C API, or even directly. Modifying structures like
PyListObject cannot be even considered.

The PyTypeObject structure is the one which evolved the most, simply
because there was no other way to evolve CPython than modifying it.

A C extension can technically dereference a PyObject* pointer and access
PyObject members. This prevents experiments like tagged pointers
(storing small values as PyObject* which does not point to a valid
PyObject structure).

Replacing Python garbage collector with a tracing garbage collector
would also need to remove PyObject.ob_refcnt reference counter, whereas
currently Py_INCREF() and Py_DECREF() macros access directly to
PyObject.ob_refcnt.

Same CPython design since 1990: structures and reference counting

When the CPython project was created, it was written with one principle:
keep the implementation simple enough so it can be maintained by a
single developer. CPython complexity grew a lot and many
micro-optimizations have been implemented, but CPython core design has
not changed.

Members of PyObject and PyTupleObject structures have not changed since
the "Initial revision" commit (1990):

    #define OB_HEAD \
        unsigned int ob_refcnt; \
        struct _typeobject *ob_type;

    typedef struct _object {
        OB_HEAD
    } object;

    typedef struct {
        OB_VARHEAD
        object *ob_item[1];
    } tupleobject;

Only names changed: object was renamed to PyObject and tupleobject was
renamed to PyTupleObject.

CPython still tracks Python objects lifetime using reference counting
internally and for third party C extensions (through the Python C API).

All Python objects must be allocated on the heap and cannot be moved.

Why is PyPy more efficient than CPython?

The PyPy project is a Python implementation which is 4.2x faster than
CPython on average. PyPy developers chose to not fork CPython, but start
from scratch to have more freedom in terms of optimization choices.

PyPy does not use reference counting, but a tracing garbage collector
which moves objects. Objects can be allocated on the stack (or even not
at all), rather than always having to be allocated on the heap.

Objects layouts are designed with performance in mind. For example, a
list strategy stores integers directly as integers, rather than objects.

Moreover, PyPy also has a JIT compiler which emits fast code thanks to
the efficient PyPy design.

PyPy bottleneck: the Python C API

While PyPy is way more efficient than CPython to run pure Python code,
it is as efficient or slower than CPython to run C extensions.

Since the C API requires PyObject* and allows to access directly
structure members, PyPy has to associate a CPython object to PyPy
objects and maintain both consistent. Converting a PyPy object to a
CPython object is inefficient. Moreover, reference counting also has to
be implemented on top of PyPy tracing garbage collector.

These conversions are required because the Python C API is too close to
the CPython implementation: there is no high-level abstraction. For
example, structures members are part of the public C API and nothing
prevents a C extension to get or set directly PyTupleObject.ob_item[0]
(the first item of a tuple).

See Inside cpyext: Why emulating CPython C API is so Hard (Sept 2018) by
Antonio Cuni for more details.

Rationale

Hide implementation details

Hiding implementation details from the C API has multiple advantages:

-   It becomes possible to experiment with more advanced optimizations
    in CPython than just micro-optimizations. For example, tagged
    pointers, and replace the garbage collector with a tracing garbage
    collector which can move objects.
-   Adding new features in CPython becomes easier.
-   PyPy should be able to avoid conversions to CPython objects in more
    cases: keep efficient PyPy objects.
-   It becomes easier to implement the C API for a new Python
    implementation.
-   More C extensions will be compatible with Python implementations
    other than CPython.

Relationship with the limited C API

The PEP 384 "Defining a Stable ABI" is implemented in Python 3.4. It
introduces the "limited C API": a subset of the C API. When the limited
C API is used, it becomes possible to build a C extension only once and
use it on multiple Python versions: that's the stable ABI.

The main limitation of the PEP 384 is that C extensions have to opt-in
for the limited C API. Only very few projects made this choice, usually
to ease distribution of binaries, especially on Windows.

This PEP moves the C API towards the limited C API.

Ideally, the C API will become the limited C API and all C extensions
will use the stable ABI, but this is out of this PEP scope.

Specification

Summary

-   (Completed) Reorganize the C API header files: create
    Include/cpython/ and Include/internal/ subdirectories.
-   (Completed) Move private functions exposing implementation details
    to the internal C API.
-   (Completed) Convert macros to static inline functions.
-   (Completed) Add new functions Py_SET_TYPE(), Py_SET_REFCNT() and
    Py_SET_SIZE(). The Py_TYPE(), Py_REFCNT() and Py_SIZE() macros
    become functions which cannot be used as l-value.
-   (Completed) New C API functions must not return borrowed references.
-   (In Progress) Provide pythoncapi_compat.h header file.
-   (In Progress) Make structures opaque, add getter and setter
    functions.
-   (Not Started) Deprecate PySequence_Fast_ITEMS().
-   (Not Started) Convert PyTuple_GET_ITEM() and PyList_GET_ITEM()
    macros to static inline functions.

Reorganize the C API header files

The first consumer of the C API was Python itself. There is no clear
separation between APIs which must not be used outside Python, and API
which are public on purpose.

Header files must be reorganized in 3 API:

-   Include/ directory is the limited C API: no implementation details,
    structures are opaque. C extensions using it get a stable ABI.
-   Include/cpython/ directory is the CPython C API: less "portable"
    API, depends more on the Python version, expose some implementation
    details, few incompatible changes can happen.
-   Include/internal/ directory is the internal C API: implementation
    details, incompatible changes are likely at each Python release.

The creation of the Include/cpython/ directory is fully backward
compatible. Include/cpython/ header files cannot be included directly
and are included automatically by Include/ header files when the
Py_LIMITED_API macro is not defined.

The internal C API is installed and can be used for specific usage like
debuggers and profilers which must access structures members without
executing code. C extensions using the internal C API are tightly
coupled to a Python version and must be recompiled at each Python
version.

STATUS: Completed (in Python 3.8)

The reorganization of header files started in Python 3.7 and was
completed in Python 3.8:

-   bpo-35134: Add a new Include/cpython/ subdirectory for the "CPython
    API" with implementation details.
-   bpo-35081: Move internal headers to Include/internal/

Move private functions to the internal C API

Private functions which expose implementation details must be moved to
the internal C API.

If a C extension relies on a CPython private function which exposes
CPython implementation details, other Python implementations have to
re-implement this private function to support this C extension.

STATUS: Completed (in Python 3.9)

Private functions moved to the internal C API in Python 3.8:

-   _PyObject_GC_TRACK(), _PyObject_GC_UNTRACK()

Macros and functions excluded from the limited C API in Python 3.9:

-   _PyObject_SIZE(), _PyObject_VAR_SIZE()
-   PyThreadState_DeleteCurrent()
-   PyFPE_START_PROTECT(), PyFPE_END_PROTECT()
-   _Py_NewReference(), _Py_ForgetReference()
-   _PyTraceMalloc_NewReference()
-   _Py_GetRefTotal()

Private functions moved to the internal C API in Python 3.9:

-   GC functions like _Py_AS_GC(), _PyObject_GC_IS_TRACKED() and
    _PyGCHead_NEXT()
-   _Py_AddToAllObjects() (not exported)
-   _PyDebug_PrintTotalRefs(), _Py_PrintReferences(),
    _Py_PrintReferenceAddresses() (not exported)

Public "clear free list" functions moved to the internal C API and
renamed to private functions in Python 3.9:

-   PyAsyncGen_ClearFreeLists()
-   PyContext_ClearFreeList()
-   PyDict_ClearFreeList()
-   PyFloat_ClearFreeList()
-   PyFrame_ClearFreeList()
-   PyList_ClearFreeList()
-   PyTuple_ClearFreeList()
-   Functions simply removed:
    -   PyMethod_ClearFreeList() and PyCFunction_ClearFreeList(): bound
        method free list removed in Python 3.9.
    -   PySet_ClearFreeList(): set free list removed in Python 3.4.
    -   PyUnicode_ClearFreeList(): Unicode free list removed in Python
        3.3.

Convert macros to static inline functions

Converting macros to static inline functions has multiple advantages:

-   Functions have well defined parameter types and return type.
-   Functions can use variables with a well defined scope (the
    function).
-   Debugger can be put breakpoints on functions and profilers can
    display the function name in the call stacks. In most cases, it
    works even when a static inline function is inlined.
-   Functions don't have macros pitfalls.

Converting macros to static inline functions should only impact very few
C extensions that use macros in unusual ways.

For backward compatibility, functions must continue to accept any type,
not only PyObject*, to avoid compiler warnings, since most macros cast
their parameters to PyObject*.

Python 3.6 requires C compilers to support static inline functions: the
PEP 7 requires a subset of C99.

STATUS: Completed (in Python 3.9)

Macros converted to static inline functions in Python 3.8:

-   Py_INCREF(), Py_DECREF()
-   Py_XINCREF(), Py_XDECREF()
-   PyObject_INIT(), PyObject_INIT_VAR()
-   _PyObject_GC_TRACK(), _PyObject_GC_UNTRACK(), _Py_Dealloc()

Macros converted to regular functions in Python 3.9:

-   Py_EnterRecursiveCall(), Py_LeaveRecursiveCall() (added to the
    limited C API)
-   PyObject_INIT(), PyObject_INIT_VAR()
-   PyObject_GET_WEAKREFS_LISTPTR()
-   PyObject_CheckBuffer()
-   PyIndex_Check()
-   PyObject_IS_GC()
-   PyObject_NEW() (alias to PyObject_New()), PyObject_NEW_VAR() (alias
    to PyObject_NewVar())
-   PyType_HasFeature() (always call PyType_GetFlags())
-   Py_TRASHCAN_BEGIN_CONDITION() and Py_TRASHCAN_END() macros now call
    functions which hide implementation details, rather than accessing
    directly members of the PyThreadState structure.

Make structures opaque

The following structures of the C API become opaque:

-   PyInterpreterState
-   PyThreadState
-   PyGC_Head
-   PyTypeObject
-   PyObject and PyVarObject
-   PyTypeObject
-   All types which inherit from PyObject or PyVarObject

C extensions must use getter or setter functions to get or set structure
members. For example, tuple->ob_item[0] must be replaced with
PyTuple_GET_ITEM(tuple, 0).

To be able to move away from reference counting, PyObject must become
opaque. Currently, the reference counter PyObject.ob_refcnt is exposed
in the C API. All structures must become opaque, since they "inherit"
from PyObject. For, PyFloatObject inherits from PyObject:

    typedef struct {
        PyObject ob_base;
        double ob_fval;
    } PyFloatObject;

Making PyObject fully opaque requires converting Py_INCREF() and
Py_DECREF() macros to function calls. This change has an impact on
performance. It is likely to be one of the very last changes when making
structures opaque.

Making PyTypeObject structure opaque breaks C extensions declaring types
statically (e.g. static PyTypeObject MyType = {...};). C extensions must
use PyType_FromSpec() to allocate types on the heap instead. Using heap
types has other advantages like being compatible with subinterpreters.
Combined with PEP 489 "Multi-phase extension module initialization", it
makes a C extension behavior closer to a Python module, like allowing to
create more than one module instance.

Making PyThreadState structure opaque requires adding getter and setter
functions for members used by C extensions.

STATUS: In Progress (started in Python 3.8)

The PyInterpreterState structure was made opaque in Python 3.8
(bpo-35886) and the PyGC_Head structure (bpo-40241) was made opaque in
Python 3.9.

Issues tracking the work to prepare the C API to make following
structures opaque:

-   PyObject: bpo-39573
-   PyTypeObject: bpo-40170
-   PyFrameObject: bpo-40421
    -   Python 3.9 adds PyFrame_GetCode() and PyFrame_GetBack() getter
        functions, and moves PyFrame_GetLineNumber to the limited C API.
-   PyThreadState: bpo-39947
    -   Python 3.9 adds 3 getter functions: PyThreadState_GetFrame(),
        PyThreadState_GetID(), PyThreadState_GetInterpreter().

Disallow using Py_TYPE() as l-value

The Py_TYPE() function gets an object type, its PyObject.ob_type member.
It is implemented as a macro which can be used as an l-value to set the
type: Py_TYPE(obj) = new_type. This code relies on the assumption that
PyObject.ob_type can be modified directly. It prevents making the
PyObject structure opaque.

New setter functions Py_SET_TYPE(), Py_SET_REFCNT() and Py_SET_SIZE()
are added and must be used instead.

The Py_TYPE(), Py_REFCNT() and Py_SIZE() macros must be converted to
static inline functions which can not be used as l-value.

For example, the Py_TYPE() macro:

    #define Py_TYPE(ob)             (((PyObject*)(ob))->ob_type)

becomes:

    #define _PyObject_CAST_CONST(op) ((const PyObject*)(op))

    static inline PyTypeObject* _Py_TYPE(const PyObject *ob) {
        return ob->ob_type;
    }

    #define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob))

STATUS: Completed (in Python 3.10)

New functions Py_SET_TYPE(), Py_SET_REFCNT() and Py_SET_SIZE() were
added to Python 3.9.

In Python 3.10, Py_TYPE(), Py_REFCNT() and Py_SIZE() can no longer be
used as l-value and the new setter functions must be used instead.

New C API functions must not return borrowed references

When a function returns a borrowed reference, Python cannot track when
the caller stops using this reference.

For example, if the Python list type is specialized for small integers,
store directly "raw" numbers rather than Python objects,
PyList_GetItem() has to create a temporary Python object. The problem is
to decide when it is safe to delete the temporary object.

The general guidelines is to avoid returning borrowed references for new
C API functions.

No function returning borrowed references is scheduled for removal by
this PEP.

STATUS: Completed (in Python 3.9)

In Python 3.9, new C API functions returning Python objects only return
strong references:

-   PyFrame_GetBack()
-   PyFrame_GetCode()
-   PyObject_CallNoArgs()
-   PyObject_CallOneArg()
-   PyThreadState_GetFrame()

Avoid functions returning PyObject**

The PySequence_Fast_ITEMS() function gives a direct access to an array
of PyObject* objects. The function is deprecated in favor of
PyTuple_GetItem() and PyList_GetItem().

PyTuple_GET_ITEM() can be abused to access directly the
PyTupleObject.ob_item member:

    PyObject **items = &PyTuple_GET_ITEM(0);

The PyTuple_GET_ITEM() and PyList_GET_ITEM() macros are converted to
static inline functions to disallow that.

STATUS: Not Started

New pythoncapi_compat.h header file

Making structures opaque requires modifying C extensions to use getter
and setter functions. The practical issue is how to keep support for old
Python versions which don't have these functions.

For example, in Python 3.10, it is no longer possible to use Py_TYPE()
as an l-value. The new Py_SET_TYPE() function must be used instead:

    #if PY_VERSION_HEX >= 0x030900A4
        Py_SET_TYPE(&MyType, &PyType_Type);
    #else
        Py_TYPE(&MyType) = &PyType_Type;
    #endif

This code may ring a bell to developers who ported their Python code
base from Python 2 to Python 3.

Python will distribute a new pythoncapi_compat.h header file which
provides new C API functions to old Python versions. Example:

    #if PY_VERSION_HEX < 0x030900A4
    static inline void
    _Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
    {
        ob->ob_type = type;
    }
    #define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
    #endif  // PY_VERSION_HEX < 0x030900A4

Using this header file, Py_SET_TYPE() can be used on old Python versions
as well.

Developers can copy this file in their project, or even to only
copy/paste the few functions needed by their C extension.

STATUS: In Progress (implemented but not distributed by CPython yet)

The pythoncapi_compat.h header file is currently developed at:
https://github.com/pythoncapi/pythoncapi_compat

Process to reduce the number of broken C extensions

Process to reduce the number of broken C extensions when introducing C
API incompatible changes listed in this PEP:

-   Estimate how many popular C extensions are affected by the
    incompatible change.
-   Coordinate with maintainers of broken C extensions to prepare their
    code for the future incompatible change.
-   Introduce the incompatible changes in Python. The documentation must
    explain how to port existing code. It is recommended to merge such
    changes at the beginning of a development cycle to have more time
    for tests.
-   Changes which are the most likely to break a large number of C
    extensions should be announced on the capi-sig mailing list to
    notify C extensions maintainers to prepare their project for the
    next Python.
-   If the change breaks too many projects, reverting the change should
    be discussed, taking in account the number of broken packages, their
    importance in the Python community, and the importance of the
    change.

The coordination usually means reporting issues to the projects, or even
proposing changes. It does not require waiting for a new release
including fixes for every broken project.

Since more and more C extensions are written using Cython, rather
directly using the C API, it is important to ensure that Cython is
prepared in advance for incompatible changes. It gives more time for C
extension maintainers to release a new version with code generated with
the updated Cython (for C extensions distributing the code generated by
Cython).

Future incompatible changes can be announced by deprecating a function
in the documentation and by annotating the function with
Py_DEPRECATED(). But making a structure opaque and preventing the usage
of a macro as l-value cannot be deprecated with Py_DEPRECATED().

The important part is coordination and finding a balance between CPython
evolutions and backward compatibility. For example, breaking a random,
old, obscure and unmaintained C extension on PyPI is less severe than
breaking numpy.

If a change is reverted, we move back to the coordination step to better
prepare the change. Once more C extensions are ready, the incompatible
change can be reconsidered.

Version History

-   Version 3, June 2020: PEP rewritten from scratch. Python now
    distributes a new pythoncapi_compat.h header and a process is
    defined to reduce the number of broken C extensions when introducing
    C API incompatible changes listed in this PEP.
-   Version 2, April 2020: PEP: Modify the C API to hide implementation
    details.
-   Version 1, July 2017: PEP: Hide implementation details in the C API
    sent to python-ideas

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 380 Title: Syntax for Delegating to a Subgenerator Version:
$Revision$ Last-Modified: $Date$ Author: Gregory Ewing
<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.3
Post-History: Resolution:
https://mail.python.org/pipermail/python-dev/2011-June/112010.html

Abstract

A syntax is proposed for a generator to delegate part of its operations
to another generator. This allows a section of code containing 'yield'
to be factored out and placed in another generator. Additionally, the
subgenerator is allowed to return with a value, and the value is made
available to the delegating generator.

The new syntax also opens up some opportunities for optimisation when
one generator re-yields values produced by another.

PEP Acceptance

Guido officially accepted the PEP on 26th June, 2011.

Motivation

A Python generator is a form of coroutine, but has the limitation that
it can only yield to its immediate caller. This means that a piece of
code containing a yield cannot be factored out and put into a separate
function in the same way as other code. Performing such a factoring
causes the called function to itself become a generator, and it is
necessary to explicitly iterate over this second generator and re-yield
any values that it produces.

If yielding of values is the only concern, this can be performed without
much difficulty using a loop such as

    for v in g:
        yield v

However, if the subgenerator is to interact properly with the caller in
the case of calls to send(), throw() and close(), things become
considerably more difficult. As will be seen later, the necessary code
is very complicated, and it is tricky to handle all the corner cases
correctly.

A new syntax will be proposed to address this issue. In the simplest use
cases, it will be equivalent to the above for-loop, but it will also
handle the full range of generator behaviour, and allow generator code
to be refactored in a simple and straightforward way.

Proposal

The following new expression syntax will be allowed in the body of a
generator:

    yield from <expr>

where <expr> is an expression evaluating to an iterable, from which an
iterator is extracted. The iterator is run to exhaustion, during which
time it yields and receives values directly to or from the caller of the
generator containing the yield from expression (the "delegating
generator").

Furthermore, when the iterator is another generator, the subgenerator is
allowed to execute a return statement with a value, and that value
becomes the value of the yield from expression.

The full semantics of the yield from expression can be described in
terms of the generator protocol as follows:

-   Any values that the iterator yields are passed directly to the
    caller.
-   Any values sent to the delegating generator using send() are passed
    directly to the iterator. If the sent value is None, the iterator's
    __next__() method is called. If the sent value is not None, the
    iterator's send() method is called. If the call raises
    StopIteration, the delegating generator is resumed. Any other
    exception is propagated to the delegating generator.
-   Exceptions other than GeneratorExit thrown into the delegating
    generator are passed to the throw() method of the iterator. If the
    call raises StopIteration, the delegating generator is resumed. Any
    other exception is propagated to the delegating generator.
-   If a GeneratorExit exception is thrown into the delegating
    generator, or the close() method of the delegating generator is
    called, then the close() method of the iterator is called if it has
    one. If this call results in an exception, it is propagated to the
    delegating generator. Otherwise, GeneratorExit is raised in the
    delegating generator.
-   The value of the yield from expression is the first argument to the
    StopIteration exception raised by the iterator when it terminates.
-   return expr in a generator causes StopIteration(expr) to be raised
    upon exit from the generator.

Enhancements to StopIteration

For convenience, the StopIteration exception will be given a value
attribute that holds its first argument, or None if there are no
arguments.

Formal Semantics

Python 3 syntax is used in this section.

1.  The statement :

        RESULT = yield from EXPR

    is semantically equivalent to :

        _i = iter(EXPR)
        try:
            _y = next(_i)
        except StopIteration as _e:
            _r = _e.value
        else:
            while 1:
                try:
                    _s = yield _y
                except GeneratorExit as _e:
                    try:
                        _m = _i.close
                    except AttributeError:
                        pass
                    else:
                        _m()
                    raise _e
                except BaseException as _e:
                    _x = sys.exc_info()
                    try:
                        _m = _i.throw
                    except AttributeError:
                        raise _e
                    else:
                        try:
                            _y = _m(*_x)
                        except StopIteration as _e:
                            _r = _e.value
                            break
                else:
                    try:
                        if _s is None:
                            _y = next(_i)
                        else:
                            _y = _i.send(_s)
                    except StopIteration as _e:
                        _r = _e.value
                        break
        RESULT = _r

2.  In a generator, the statement :

        return value

    is semantically equivalent to :

        raise StopIteration(value)

    except that, as currently, the exception cannot be caught by except
    clauses within the returning generator.

3.  The StopIteration exception behaves as though defined thusly:

        class StopIteration(Exception):

            def __init__(self, *args):
                if len(args) > 0:
                    self.value = args[0]
                else:
                    self.value = None
                Exception.__init__(self, *args)

Rationale

The Refactoring Principle

The rationale behind most of the semantics presented above stems from
the desire to be able to refactor generator code. It should be possible
to take a section of code containing one or more yield expressions, move
it into a separate function (using the usual techniques to deal with
references to variables in the surrounding scope, etc.), and call the
new function using a yield from expression.

The behaviour of the resulting compound generator should be, as far as
reasonably practicable, the same as the original unfactored generator in
all situations, including calls to __next__(), send(), throw() and
close().

The semantics in cases of subiterators other than generators has been
chosen as a reasonable generalization of the generator case.

The proposed semantics have the following limitations with regard to
refactoring:

-   A block of code that catches GeneratorExit without subsequently
    re-raising it cannot be factored out while retaining exactly the
    same behaviour.
-   Factored code may not behave the same way as unfactored code if a
    StopIteration exception is thrown into the delegating generator.

With use cases for these being rare to non-existent, it was not
considered worth the extra complexity required to support them.

Finalization

There was some debate as to whether explicitly finalizing the delegating
generator by calling its close() method while it is suspended at a
yield from should also finalize the subiterator. An argument against
doing so is that it would result in premature finalization of the
subiterator if references to it exist elsewhere.

Consideration of non-refcounting Python implementations led to the
decision that this explicit finalization should be performed, so that
explicitly closing a factored generator has the same effect as doing so
to an unfactored one in all Python implementations.

The assumption made is that, in the majority of use cases, the
subiterator will not be shared. The rare case of a shared subiterator
can be accommodated by means of a wrapper that blocks throw() and
close() calls, or by using a means other than yield from to call the
subiterator.

Generators as Threads

A motivation for generators being able to return values concerns the use
of generators to implement lightweight threads. When using generators in
that way, it is reasonable to want to spread the computation performed
by the lightweight thread over many functions. One would like to be able
to call a subgenerator as though it were an ordinary function, passing
it parameters and receiving a returned value.

Using the proposed syntax, a statement such as :

    y = f(x)

where f is an ordinary function, can be transformed into a delegation
call :

    y = yield from g(x)

where g is a generator. One can reason about the behaviour of the
resulting code by thinking of g as an ordinary function that can be
suspended using a yield statement.

When using generators as threads in this way, typically one is not
interested in the values being passed in or out of the yields. However,
there are use cases for this as well, where the thread is seen as a
producer or consumer of items. The yield from expression allows the
logic of the thread to be spread over as many functions as desired, with
the production or consumption of items occurring in any subfunction, and
the items are automatically routed to or from their ultimate source or
destination.

Concerning throw() and close(), it is reasonable to expect that if an
exception is thrown into the thread from outside, it should first be
raised in the innermost generator where the thread is suspended, and
propagate outwards from there; and that if the thread is terminated from
outside by calling close(), the chain of active generators should be
finalised from the innermost outwards.

Syntax

The particular syntax proposed has been chosen as suggestive of its
meaning, while not introducing any new keywords and clearly standing out
as being different from a plain yield.

Optimisations

Using a specialised syntax opens up possibilities for optimisation when
there is a long chain of generators. Such chains can arise, for
instance, when recursively traversing a tree structure. The overhead of
passing __next__() calls and yielded values down and up the chain can
cause what ought to be an O(n) operation to become, in the worst case,
O(n**2).

A possible strategy is to add a slot to generator objects to hold a
generator being delegated to. When a __next__() or send() call is made
on the generator, this slot is checked first, and if it is nonempty, the
generator that it references is resumed instead. If it raises
StopIteration, the slot is cleared and the main generator is resumed.

This would reduce the delegation overhead to a chain of C function calls
involving no Python code execution. A possible enhancement would be to
traverse the whole chain of generators in a loop and directly resume the
one at the end, although the handling of StopIteration is more
complicated then.

Use of StopIteration to return values

There are a variety of ways that the return value from the generator
could be passed back. Some alternatives include storing it as an
attribute of the generator-iterator object, or returning it as the value
of the close() call to the subgenerator. However, the proposed mechanism
is attractive for a couple of reasons:

-   Using a generalization of the StopIteration exception makes it easy
    for other kinds of iterators to participate in the protocol without
    having to grow an extra attribute or a close() method.
-   It simplifies the implementation, because the point at which the
    return value from the subgenerator becomes available is the same
    point at which the exception is raised. Delaying until any later
    time would require storing the return value somewhere.

Rejected Ideas

Some ideas were discussed but rejected.

Suggestion: There should be some way to prevent the initial call to
__next__(), or substitute it with a send() call with a specified value,
the intention being to support the use of generators wrapped so that the
initial __next__() is performed automatically.

Resolution: Outside the scope of the proposal. Such generators should
not be used with yield from.

Suggestion: If closing a subiterator raises StopIteration with a value,
return that value from the close() call to the delegating generator.

The motivation for this feature is so that the end of a stream of values
being sent to a generator can be signalled by closing the generator. The
generator would catch GeneratorExit, finish its computation and return a
result, which would then become the return value of the close() call.

Resolution: This usage of close() and GeneratorExit would be
incompatible with their current role as a bail-out and clean-up
mechanism. It would require that when closing a delegating generator,
after the subgenerator is closed, the delegating generator be resumed
instead of re-raising GeneratorExit. But this is not acceptable, because
it would fail to ensure that the delegating generator is finalised
properly in the case where close() is being called for cleanup purposes.

Signalling the end of values to a consumer is better addressed by other
means, such as sending in a sentinel value or throwing in an exception
agreed upon by the producer and consumer. The consumer can then detect
the sentinel or exception and respond by finishing its computation and
returning normally. Such a scheme behaves correctly in the presence of
delegation.

Suggestion: If close() is not to return a value, then raise an exception
if StopIteration with a non-None value occurs.

Resolution: No clear reason to do so. Ignoring a return value is not
considered an error anywhere else in Python.

Criticisms

Under this proposal, the value of a yield from expression would be
derived in a very different way from that of an ordinary yield
expression. This suggests that some other syntax not containing the word
yield might be more appropriate, but no acceptable alternative has so
far been proposed. Rejected alternatives include call, delegate and
gcall.

It has been suggested that some mechanism other than return in the
subgenerator should be used to establish the value returned by the
yield from expression. However, this would interfere with the goal of
being able to think of the subgenerator as a suspendable function, since
it would not be able to return values in the same way as other
functions.

The use of an exception to pass the return value has been criticised as
an "abuse of exceptions", without any concrete justification of this
claim. In any case, this is only one suggested implementation; another
mechanism could be used without losing any essential features of the
proposal.

It has been suggested that a different exception, such as
GeneratorReturn, should be used instead of StopIteration to return a
value. However, no convincing practical reason for this has been put
forward, and the addition of a value attribute to StopIteration
mitigates any difficulties in extracting a return value from a
StopIteration exception that may or may not have one. Also, using a
different exception would mean that, unlike ordinary functions, 'return'
without a value in a generator would not be equivalent to 'return None'.

Alternative Proposals

Proposals along similar lines have been made before, some using the
syntax yield * instead of yield from. While yield * is more concise, it
could be argued that it looks too similar to an ordinary yield and the
difference might be overlooked when reading code.

To the author's knowledge, previous proposals have focused only on
yielding values, and thereby suffered from the criticism that the
two-line for-loop they replace is not sufficiently tiresome to write to
justify a new syntax. By dealing with the full generator protocol, this
proposal provides considerably more benefit.

Additional Material

Some examples of the use of the proposed syntax are available, and also
a prototype implementation based on the first optimisation outlined
above.

Examples and Implementation

A version of the implementation updated for Python 3.3 is available from
tracker issue #11682

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 448 Title: Additional Unpacking Generalizations Author: Joshua
Landau<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
29-Jun-2013 Python-Version: 3.5 Post-History:

Abstract

This PEP proposes extended usages of the * iterable unpacking operator
and ** dictionary unpacking operators to allow unpacking in more
positions, an arbitrary number of times, and in function calls and
displays.

Function calls are proposed to support an arbitrary number of unpackings
rather than just one:

    >>> print(*[1], *[2], 3)
    1 2 3
    >>> dict(**{'x': 1}, y=2, **{'z': 3})
    {'x': 1, 'y': 2, 'z': 3}

Unpacking is proposed to be allowed inside tuple, list, set, and
dictionary displays:

    >>> *range(4), 4
    (0, 1, 2, 3, 4)
    >>> [*range(4), 4]
    [0, 1, 2, 3, 4]
    >>> {*range(4), 4}
    {0, 1, 2, 3, 4}
    >>> {'x': 1, **{'y': 2}}
    {'x': 1, 'y': 2}

In dictionaries, later values will always override earlier ones:

    >>> {'x': 1, **{'x': 2}}
    {'x': 2}

    >>> {**{'x': 2}, 'x': 1}
    {'x': 1}

This PEP does not include unpacking operators inside list, set and
dictionary comprehensions although this has not been ruled out for
future proposals.

Rationale

Current usage of the * iterable unpacking operator features unnecessary
restrictions that can harm readability.

Unpacking multiple times has an obvious rationale. When you want to
unpack several iterables into a function definition or follow an unpack
with more positional arguments, the most natural way would be to write:

    function(**kw_arguments, **more_arguments)

    function(*arguments, argument)

Simple examples where this is useful are print and str.format. Instead,
you could be forced to write:

    kwargs = dict(kw_arguments)
    kwargs.update(more_arguments)
    function(**kwargs)

    args = list(arguments)
    args.append(arg)
    function(*args)

or, if you know to do so:

    from collections import ChainMap
    function(**ChainMap(more_arguments, arguments))

    from itertools import chain
    function(*chain(args, [arg]))

which add unnecessary line-noise and, with the first methods, causes
duplication of work.

There are two primary rationales for unpacking inside of containers.
Firstly there is a symmetry of assignment, where
fst, *other, lst = elems and elems = fst, *other, lst are approximate
inverses, ignoring the specifics of types. This, in effect, simplifies
the language by removing special cases.

Secondly, it vastly simplifies types of "addition" such as combining
dictionaries, and does so in an unambiguous and well-defined way:

    combination = {**first_dictionary, "x": 1, "y": 2}

instead of:

    combination = first_dictionary.copy()
    combination.update({"x": 1, "y": 2})

which is especially important in contexts where expressions are
preferred. This is also useful as a more readable way of summing
iterables into a list, such as my_list + list(my_tuple) + list(my_range)
which is now equivalent to just [*my_list, *my_tuple, *my_range].

Specification

Function calls may accept an unbounded number of * and ** unpackings.
There will be no restriction of the order of positional arguments with
relation to * unpackings nor any restriction of the order of keyword
arguments with relation to ** unpackings.

Function calls continue to have the restriction that keyword arguments
must follow positional arguments and ** unpackings must additionally
follow * unpackings.

Currently, if an argument is given multiple times — such as a positional
argument given both positionally and by keyword — a TypeError is raised.
This remains true for duplicate arguments provided through multiple **
unpackings, e.g. f(**{'x': 2}, **{'x': 3}), except that the error will
be detected at runtime.

A function looks like this:

    function(
        argument or *args, argument or *args, ...,
        kwargument or *args, kwargument or *args, ...,
        kwargument or **kwargs, kwargument or **kwargs, ...
    )

Tuples, lists, sets and dictionaries will allow unpacking. This will act
as if the elements from unpacked items were inserted in order at the
site of unpacking, much as happens in unpacking in a function-call.
Dictionaries require ** unpacking; all the others require * unpacking.

The keys in a dictionary remain in a right-to-left priority order, so
{**{'a': 1}, 'a': 2, **{'a': 3}} evaluates to {'a': 3}. There is no
restriction on the number or position of unpackings.

Disadvantages

The allowable orders for arguments in a function call are more
complicated than before. The simplest explanation for the rules may be
"positional arguments precede keyword arguments and ** unpacking; *
unpacking precedes ** unpacking".

Whilst *elements, = iterable causes elements to be a list,
elements = *iterable, causes elements to be a tuple. The reason for this
may confuse people unfamiliar with the construct.

Concerns have been raised about the unexpected difference between
duplicate keys in dictionaries being allowed but duplicate keys in
function call syntax raising an error. Although this is already the case
with current syntax, this proposal might exacerbate the issue. It
remains to be seen how much of an issue this is in practice.

Variations

The PEP originally considered whether the ordering of argument types in
a function call (positional, keyword, * or **) could become less strict.
This met little support so the idea was shelved.

Earlier iterations of this PEP allowed unpacking operators inside list,
set, and dictionary comprehensions as a flattening operator over
iterables of containers:

    >>> ranges = [range(i) for i in range(5)]
    >>> [*item for item in ranges]
    [0, 0, 1, 0, 1, 2, 0, 1, 2, 3]

    >>> {*item for item in ranges}
    {0, 1, 2, 3}

This was met with a mix of strong concerns about readability and mild
support. In order not to disadvantage the less controversial aspects of
the PEP, this was not accepted with the rest of the proposal.

Unbracketed comprehensions in function calls, such as f(x for x in it),
are already valid. These could be extended to:

    f(*x for x in it) == f((*x for x in it))
    f(**x for x in it) == f({**x for x in it})

However, it wasn't clear if this was the best behaviour or if it should
unpack into the arguments of the call to f. Since this is likely to be
confusing and is of only very marginal utility, it is not included in
this PEP. Instead, these will throw a SyntaxError and comprehensions
with explicit brackets should be used instead.

Approval

This PEP was accepted by Guido on February 25, 2015[1].

Implementation

An implementation for Python 3.5 is found at Issue 2292 on bug
tracker[2]. This currently includes support for unpacking inside
comprehensions, which should be removed.

References

[3] Discussion on Python-ideas list, "list / array comprehensions
extension", Alexander Heger
(https://mail.python.org/pipermail/python-ideas/2011-December/013097.html)

Copyright

This document has been placed in the public domain.

[1] PEP accepted, "PEP 448 review", Guido van Rossum
(https://mail.python.org/pipermail/python-dev/2015-February/138564.html)

[2] Issue 2292, "Missing *-unpacking generalizations", Thomas Wouters
(https://github.com/python/cpython/issues/46545)