text:
PEP: 646 Title: Variadic Generics Author: Mark Mendoza
<EMAIL_ADDRESS>Matthew Rahtz<EMAIL_ADDRESS>Pradeep
Kumar Srinivasan<EMAIL_ADDRESS>Vincent Siles<EMAIL_ADDRESS>Sponsor: Guido van Rossum<EMAIL_ADDRESS>Status: Final Type:
Standards Track Topic: Typing Content-Type: text/x-rst Created:
16-Sep-2020 Python-Version: 3.11 Post-History: 07-Oct-2020, 23-Dec-2020,
29-Dec-2020 Resolution:
<EMAIL_ADDRESS>
typing:typevartuple and typing.TypeVarTuple

Abstract

PEP 484 introduced TypeVar, enabling creation of generics parameterised
with a single type. In this PEP, we introduce TypeVarTuple, enabling
parameterisation with an arbitrary number of types - that is, a variadic
type variable, enabling variadic generics. This enables a wide variety
of use cases. In particular, it allows the type of array-like structures
in numerical computing libraries such as NumPy and TensorFlow to be
parameterised with the array shape, enabling static type checkers to
catch shape-related bugs in code that uses these libraries.

Acceptance

This PEP was accepted for Python 3.11, with the caveat that details
around multiple unpackings in a type expression aren't specified
precisely. This gives individual type checkers some leeway, but can be
tightened in future PEPs.

Motivation

Variadic generics have long been a requested feature, for a myriad of
use cases[1]. One particular use case - a use case with potentially
large impact, and the main case this PEP targets - concerns typing in
numerical libraries.

In the context of numerical computation with libraries such as NumPy and
TensorFlow, the shape of variables is often just as important as the
variable type. For example, consider the following function which
converts a batch[2] of videos to grayscale:

    def to_gray(videos: Array): ...

From the signature alone, it is not obvious what shape of array[3] we
should pass for the videos argument. Possibilities include, for example,

  batch × time × height × width × channels

and

  time × batch × channels × height × width.[4]

This is important for three reasons:

-   Documentation. Without the required shape being clear in the
    signature, the user must hunt in the docstring or the code in
    question to determine what the input/output shape requirements are.
-   Catching shape bugs before runtime. Ideally, use of incorrect shapes
    should be an error we can catch ahead of time using static analysis.
    (This is particularly important for machine learning code, where
    iteration times can be slow.)
-   Preventing subtle shape bugs. In the worst case, use of the wrong
    shape will result in the program appearing to run fine, but with a
    subtle bug that can take days to track down. (See this exercise in a
    popular machine learning tutorial for a particularly pernicious
    example.)

Ideally, we should have some way of making shape requirements explicit
in type signatures. Multiple proposals[5][6] [7] have suggested the use
of the standard generics syntax for this purpose. We would write:

    def to_gray(videos: Array[Time, Batch, Height, Width, Channels]): ...

However, note that arrays can be of arbitrary rank - Array as used above
is generic in an arbitrary number of axes. One way around this would be
to use a different Array class for each rank...

    Axis1 = TypeVar('Axis1')
    Axis2 = TypeVar('Axis2')

    class Array1(Generic[Axis1]): ...

    class Array2(Generic[Axis1, Axis2]): ...

...but this would be cumbersome, both for users (who would have to
sprinkle 1s and 2s and so on throughout their code) and for the authors
of array libraries (who would have to duplicate implementations
throughout multiple classes).

Variadic generics are necessary for an Array that is generic in an
arbitrary number of axes to be cleanly defined as a single class.

Summary Examples

Cutting right to the chase, this PEP allows an Array class that is
generic in its shape (and datatype) to be defined using a
newly-introduced arbitrary-length type variable, TypeVarTuple, as
follows:

    from typing import TypeVar, TypeVarTuple

    DType = TypeVar('DType')
    Shape = TypeVarTuple('Shape')

    class Array(Generic[DType, *Shape]):

        def __abs__(self) -> Array[DType, *Shape]: ...

        def __add__(self, other: Array[DType, *Shape]) -> Array[DType, *Shape]: ...

Such an Array can be used to support a number of different kinds of
shape annotations. For example, we can add labels describing the
semantic meaning of each axis:

    from typing import NewType

    Height = NewType('Height', int)
    Width = NewType('Width', int)

    x: Array[float, Height, Width] = Array()

We could also add annotations describing the actual size of each axis:

    from typing import Literal as L

    x: Array[float, L[480], L[640]] = Array()

For consistency, we use semantic axis annotations as the basis of the
examples in this PEP, but this PEP is agnostic about which of these two
(or possibly other) ways of using Array is preferable; that decision is
left to library authors.

(Note also that for the rest of this PEP, for conciseness of example, we
use a simpler version of Array which is generic only in the shape - not
the data type.)

Specification

In order to support the above use cases, we introduce TypeVarTuple. This
serves as a placeholder not for a single type but for a tuple of types.

In addition, we introduce a new use for the star operator: to 'unpack'
TypeVarTuple instances and tuple types such as Tuple[int, str].
Unpacking a TypeVarTuple or tuple type is the typing equivalent of
unpacking a variable or a tuple of values.

Type Variable Tuples

In the same way that a normal type variable is a stand-in for a single
type such as int, a type variable tuple is a stand-in for a tuple type
such as Tuple[int, str].

Type variable tuples are created with:

    from typing import TypeVarTuple

    Ts = TypeVarTuple('Ts')

Using Type Variable Tuples in Generic Classes

Type variable tuples behave like a number of individual type variables
packed in a Tuple. To understand this, consider the following example:

    Shape = TypeVarTuple('Shape')

    class Array(Generic[*Shape]): ...

    Height = NewType('Height', int)
    Width = NewType('Width', int)
    x: Array[Height, Width] = Array()

The Shape type variable tuple here behaves like Tuple[T1, T2], where T1
and T2 are type variables. To use these type variables as type
parameters of Array, we must unpack the type variable tuple using the
star operator: *Shape. The signature of Array then behaves as if we had
simply written class Array(Generic[T1, T2]): ....

In contrast to Generic[T1, T2], however, Generic[*Shape] allows us to
parameterise the class with an arbitrary number of type parameters. That
is, in addition to being able to define rank-2 arrays such as
Array[Height, Width], we could also define rank-3 arrays, rank-4 arrays,
and so on:

    Time = NewType('Time', int)
    Batch = NewType('Batch', int)
    y: Array[Batch, Height, Width] = Array()
    z: Array[Time, Batch, Height, Width] = Array()

Using Type Variable Tuples in Functions

Type variable tuples can be used anywhere a normal TypeVar can. This
includes class definitions, as shown above, as well as function
signatures and variable annotations:

    class Array(Generic[*Shape]):

        def __init__(self, shape: Tuple[*Shape]):
            self._shape: Tuple[*Shape] = shape

        def get_shape(self) -> Tuple[*Shape]:
            return self._shape

    shape = (Height(480), Width(640))
    x: Array[Height, Width] = Array(shape)
    y = abs(x)  # Inferred type is Array[Height, Width]
    z = x + x   #        ...    is Array[Height, Width]

Type Variable Tuples Must Always be Unpacked

Note that in the previous example, the shape argument to __init__ was
annotated as Tuple[*Shape]. Why is this necessary - if Shape behaves
like Tuple[T1, T2, ...], couldn't we have annotated the shape argument
as Shape directly?

This is, in fact, deliberately not possible: type variable tuples must
always be used unpacked (that is, prefixed by the star operator). This
is for two reasons:

-   To avoid potential confusion about whether to use a type variable
    tuple in a packed or unpacked form ("Hmm, should I write '-> Shape',
    or '-> Tuple[Shape]', or '-> Tuple[*Shape]'...?")
-   To improve readability: the star also functions as an explicit
    visual indicator that the type variable tuple is not a normal type
    variable.

Unpack for Backwards Compatibility

Note that the use of the star operator in this context requires a
grammar change, and is therefore available only in new versions of
Python. To enable use of type variable tuples in older versions of
Python, we introduce the Unpack type operator that can be used in place
of the star operator:

    # Unpacking using the star operator in new versions of Python
    class Array(Generic[*Shape]): ...

    # Unpacking using ``Unpack`` in older versions of Python
    class Array(Generic[Unpack[Shape]]): ...

Variance, Type Constraints and Type Bounds: Not (Yet) Supported

To keep this PEP minimal, TypeVarTuple does not yet support
specification of:

-   Variance (e.g. TypeVar('T', covariant=True))
-   Type constraints (TypeVar('T', int, float))
-   Type bounds (TypeVar('T', bound=ParentClass))

We leave the decision of how these arguments should behave to a future
PEP, when variadic generics have been tested in the field. As of this
PEP, type variable tuples are invariant.

Type Variable Tuple Equality

If the same TypeVarTuple instance is used in multiple places in a
signature or class, a valid type inference might be to bind the
TypeVarTuple to a Tuple of a Union of types:

    def foo(arg1: Tuple[*Ts], arg2: Tuple[*Ts]): ...

    a = (0,)
    b = ('0',)
    foo(a, b)  # Can Ts be bound to Tuple[int | str]?

We do not allow this; type unions may not appear within the Tuple. If a
type variable tuple appears in multiple places in a signature, the types
must match exactly (the list of type parameters must be the same length,
and the type parameters themselves must be identical):

    def pointwise_multiply(
        x: Array[*Shape],
        y: Array[*Shape]
    ) -> Array[*Shape]: ...

    x: Array[Height]
    y: Array[Width]
    z: Array[Height, Width]
    pointwise_multiply(x, x)  # Valid
    pointwise_multiply(x, y)  # Error
    pointwise_multiply(x, z)  # Error

Multiple Type Variable Tuples: Not Allowed

As of this PEP, only a single type variable tuple may appear in a type
parameter list:

    class Array(Generic[*Ts1, *Ts2]): ...  # Error

The reason is that multiple type variable tuples make it ambiguous which
parameters get bound to which type variable tuple: :

    x: Array[int, str, bool]  # Ts1 = ???, Ts2 = ???

Type Concatenation

Type variable tuples don't have to be alone; normal types can be
prefixed and/or suffixed:

    Shape = TypeVarTuple('Shape')
    Batch = NewType('Batch', int)
    Channels = NewType('Channels', int)

    def add_batch_axis(x: Array[*Shape]) -> Array[Batch, *Shape]: ...
    def del_batch_axis(x: Array[Batch, *Shape]) -> Array[*Shape]: ...
    def add_batch_channels(
      x: Array[*Shape]
    ) -> Array[Batch, *Shape, Channels]: ...

    a: Array[Height, Width]
    b = add_batch_axis(a)      # Inferred type is Array[Batch, Height, Width]
    c = del_batch_axis(b)      # Array[Height, Width]
    d = add_batch_channels(a)  # Array[Batch, Height, Width, Channels]

Normal TypeVar instances can also be prefixed and/or suffixed:

    T = TypeVar('T')
    Ts = TypeVarTuple('Ts')

    def prefix_tuple(
        x: T,
        y: Tuple[*Ts]
    ) -> Tuple[T, *Ts]: ...

    z = prefix_tuple(x=0, y=(True, 'a'))
    # Inferred type of z is Tuple[int, bool, str]

Unpacking Tuple Types

We mentioned that a TypeVarTuple stands for a tuple of types. Since we
can unpack a TypeVarTuple, for consistency, we also allow unpacking a
tuple type. As we shall see, this also enables a number of interesting
features.

Unpacking Concrete Tuple Types

Unpacking a concrete tuple type is analogous to unpacking a tuple of
values at runtime. Tuple[int, *Tuple[bool, bool], str] is equivalent to
Tuple[int, bool, bool, str].

Unpacking Unbounded Tuple Types

Unpacking an unbounded tuple preserves the unbounded tuple as it is.
That is, *Tuple[int, ...] remains *Tuple[int, ...]; there's no simpler
form. This enables us to specify types such as
Tuple[int, *Tuple[str, ...], str] - a tuple type where the first element
is guaranteed to be of type int, the last element is guaranteed to be of
type str, and the elements in the middle are zero or more elements of
type str. Note that Tuple[*Tuple[int, ...]] is equivalent to
Tuple[int, ...].

Unpacking unbounded tuples is also useful in function signatures where
we don't care about the exact elements and don't want to define an
unnecessary TypeVarTuple:

    def process_batch_channels(
        x: Array[Batch, *Tuple[Any, ...], Channels]
    ) -> None:
        ...

    x: Array[Batch, Height, Width, Channels]
    process_batch_channels(x)  # OK
    y: Array[Batch, Channels]
    process_batch_channels(y)  # OK
    z: Array[Batch]
    process_batch_channels(z)  # Error: Expected Channels.

We can also pass a *Tuple[int, ...] wherever a *Ts is expected. This is
useful when we have particularly dynamic code and cannot state the
precise number of dimensions or the precise types for each of the
dimensions. In those cases, we can smoothly fall back to an unbounded
tuple:

    y: Array[*Tuple[Any, ...]] = read_from_file()

    def expect_variadic_array(
        x: Array[Batch, *Shape]
    ) -> None: ...

    expect_variadic_array(y)  # OK

    def expect_precise_array(
        x: Array[Batch, Height, Width, Channels]
    ) -> None: ...

    expect_precise_array(y)  # OK

Array[*Tuple[Any, ...]] stands for an array with an arbitrary number of
dimensions of type Any. This means that, in the call to
expect_variadic_array, Batch is bound to Any and Shape is bound to
Tuple[Any, ...]. In the call to expect_precise_array, the variables
Batch, Height, Width, and Channels are all bound to Any.

This allows users to handle dynamic code gracefully while still
explicitly marking the code as unsafe (by using
y: Array[*Tuple[Any, ...]]). Otherwise, users would face noisy errors
from the type checker every time they tried to use the variable y, which
would hinder them when migrating a legacy code base to use TypeVarTuple.

Multiple Unpackings in a Tuple: Not Allowed

As with TypeVarTuples, only one <Multiple Type Variable Tuples: Not
Allowed_> unpacking may appear in a tuple:

    x: Tuple[int, *Ts, str, *Ts2]  # Error
    y: Tuple[int, *Tuple[int, ...], str, *Tuple[str, ...]]  # Error

*args as a Type Variable Tuple

PEP 484 states that when a type annotation is provided for *args, every
argument must be of the type annotated. That is, if we specify *args to
be type int, then all arguments must be of type int. This limits our
ability to specify the type signatures of functions that take
heterogeneous argument types.

If *args is annotated as a type variable tuple, however, the types of
the individual arguments become the types in the type variable tuple:

    Ts = TypeVarTuple('Ts')

    def args_to_tuple(*args: *Ts) -> Tuple[*Ts]: ...

    args_to_tuple(1, 'a')  # Inferred type is Tuple[int, str]

In the above example, Ts is bound to Tuple[int, str]. If no arguments
are passed, the type variable tuple behaves like an empty tuple,
Tuple[()].

As usual, we can unpack any tuple types. For example, by using a type
variable tuple inside a tuple of other types, we can refer to prefixes
or suffixes of the variadic argument list. For example:

    # os.execle takes arguments 'path, arg0, arg1, ..., env'
    def execle(path: str, *args: *Tuple[*Ts, Env]) -> None: ...

Note that this is different to

    def execle(path: str, *args: *Ts, env: Env) -> None: ...

as this would make env a keyword-only argument.

Using an unpacked unbounded tuple is equivalent to the
484#arbitrary-argument-lists-and-default-argument-values behavior of
*args: int, which accepts zero or more values of type int:

    def foo(*args: *Tuple[int, ...]) -> None: ...

    # equivalent to:
    def foo(*args: int) -> None: ...

Unpacking tuple types also allows more precise types for heterogeneous
*args. The following function expects an int at the beginning, zero or
more str values, and a str at the end:

    def foo(*args: *Tuple[int, *Tuple[str, ...], str]) -> None: ...

For completeness, we mention that unpacking a concrete tuple allows us
to specify *args of a fixed number of heterogeneous types:

    def foo(*args: *Tuple[int, str]) -> None: ...

    foo(1, "hello")  # OK

Note that, in keeping with the rule that type variable tuples must
always be used unpacked, annotating *args as being a plain type variable
tuple instance is not allowed:

    def foo(*args: Ts): ...  # NOT valid

*args is the only case where an argument can be annotated as *Ts
directly; other arguments should use *Ts to parameterise something else,
e.g. Tuple[*Ts]. If *args itself is annotated as Tuple[*Ts], the old
behaviour still applies: all arguments must be a Tuple parameterised
with the same types.

    def foo(*args: Tuple[*Ts]): ...

    foo((0,), (1,))    # Valid
    foo((0,), (1, 2))  # Error
    foo((0,), ('1',))  # Error

Finally, note that a type variable tuple may not be used as the type of
**kwargs. (We do not yet know of a use case for this feature, so we
prefer to leave the ground fresh for a potential future PEP.)

    # NOT valid
    def foo(**kwargs: *Ts): ...

Type Variable Tuples with Callable

Type variable tuples can also be used in the arguments section of a
Callable:

    class Process:
      def __init__(
        self,
        target: Callable[[*Ts], None],
        args: Tuple[*Ts],
      ) -> None: ...

    def func(arg1: int, arg2: str) -> None: ...

    Process(target=func, args=(0, 'foo'))  # Valid
    Process(target=func, args=('foo', 0))  # Error

Other types and normal type variables can also be prefixed/suffixed to
the type variable tuple:

    T = TypeVar('T')

    def foo(f: Callable[[int, *Ts, T], Tuple[T, *Ts]]): ...

The behavior of a Callable containing an unpacked item, whether the item
is a TypeVarTuple or a tuple type, is to treat the elements as if they
were the type for *args. So, Callable[[*Ts], None] is treated as the
type of the function:

    def foo(*args: *Ts) -> None: ...

Callable[[int, *Ts, T], Tuple[T, *Ts]] is treated as the type of the
function:

    def foo(*args: *Tuple[int, *Ts, T]) -> Tuple[T, *Ts]: ...

Behaviour when Type Parameters are not Specified

When a generic class parameterised by a type variable tuple is used
without any type parameters, it behaves as if the type variable tuple
was substituted with Tuple[Any, ...]:

    def takes_any_array(arr: Array): ...

    # equivalent to:
    def takes_any_array(arr: Array[*Tuple[Any, ...]]): ...

    x: Array[Height, Width]
    takes_any_array(x)  # Valid
    y: Array[Time, Height, Width]
    takes_any_array(y)  # Also valid

This enables gradual typing: existing functions accepting, for example,
a plain TensorFlow Tensor will still be valid even if Tensor is made
generic and calling code passes a Tensor[Height, Width].

This also works in the opposite direction:

    def takes_specific_array(arr: Array[Height, Width]): ...

    z: Array
    # equivalent to Array[*Tuple[Any, ...]]

    takes_specific_array(z)

(For details, see the section on Unpacking Unbounded Tuple Types.)

This way, even if libraries are updated to use types like
Array[Height, Width], users of those libraries won't be forced to also
apply type annotations to all of their code; users still have a choice
about what parts of their code to type and which parts to not.

Aliases

Generic aliases can be created using a type variable tuple in a similar
way to regular type variables:

    IntTuple = Tuple[int, *Ts]
    NamedArray = Tuple[str, Array[*Ts]]

    IntTuple[float, bool]  # Equivalent to Tuple[int, float, bool]
    NamedArray[Height]     # Equivalent to Tuple[str, Array[Height]]

As this example shows, all type parameters passed to the alias are bound
to the type variable tuple.

Importantly for our original Array example (see Summary Examples), this
allows us to define convenience aliases for arrays of a fixed shape or
datatype:

    Shape = TypeVarTuple('Shape')
    DType = TypeVar('DType')
    class Array(Generic[DType, *Shape]):

    # E.g. Float32Array[Height, Width, Channels]
    Float32Array = Array[np.float32, *Shape]

    # E.g. Array1D[np.uint8]
    Array1D = Array[DType, Any]

If an explicitly empty type parameter list is given, the type variable
tuple in the alias is set empty:

    IntTuple[()]    # Equivalent to Tuple[int]
    NamedArray[()]  # Equivalent to Tuple[str, Array[()]]

If the type parameter list is omitted entirely, the unspecified type
variable tuples are treated as Tuple[Any, ...] (similar to Behaviour
when Type Parameters are not Specified):

    def takes_float_array_of_any_shape(x: Float32Array): ...
    x: Float32Array[Height, Width] = Array()
    takes_float_array_of_any_shape(x)  # Valid

    def takes_float_array_with_specific_shape(
        y: Float32Array[Height, Width]
    ): ...
    y: Float32Array = Array()
    takes_float_array_with_specific_shape(y)  # Valid

Normal TypeVar instances can also be used in such aliases:

    T = TypeVar('T')
    Foo = Tuple[T, *Ts]

    # T bound to str, Ts to Tuple[int]
    Foo[str, int]
    # T bound to float, Ts to Tuple[()]
    Foo[float]
    # T bound to Any, Ts to an Tuple[Any, ...]
    Foo

Substitution in Aliases

In the previous section, we only discussed simple usage of generic
aliases in which the type arguments were just simple types. However, a
number of more exotic constructions are also possible.

Type Arguments can be Variadic

First, type arguments to generic aliases can be variadic. For example, a
TypeVarTuple can be used as a type argument:

    Ts1 = TypeVar('Ts1')
    Ts2 = TypeVar('Ts2')

    IntTuple = Tuple[int, *Ts1]
    IntFloatTuple = IntTuple[float, *Ts2]  # Valid

Here, *Ts1 in the IntTuple alias is bound to Tuple[float, *Ts2],
resulting in an alias IntFloatTuple equivalent to
Tuple[int, float, *Ts2].

Unpacked arbitrary-length tuples can also be used as type arguments,
with similar effects:

    IntFloatsTuple = IntTuple[*Tuple[float, ...]]  # Valid

Here, *Ts1 is bound to *Tuple[float, ...], resulting in IntFloatsTuple
being equivalent to Tuple[int, *Tuple[float, ...]]: a tuple consisting
of an int then zero or more floats.

Variadic Arguments Require Variadic Aliases

Variadic type arguments can only be used with generic aliases that are
themselves variadic. For example:

    T = TypeVar('T')

    IntTuple = Tuple[int, T]

    IntTuple[str]                 # Valid
    IntTuple[*Ts]                 # NOT valid
    IntTuple[*Tuple[float, ...]]  # NOT valid

Here, IntTuple is a non-variadic generic alias that takes exactly one
type argument. Hence, it cannot accept *Ts or *Tuple[float, ...] as type
arguments, because they represent an arbitrary number of types.

Aliases with Both TypeVars and TypeVarTuples

In Aliases, we briefly mentioned that aliases can be generic in both
TypeVars and TypeVarTuples:

    T = TypeVar('T')
    Foo = Tuple[T, *Ts]

    Foo[str, int]         # T bound to str, Ts to Tuple[int]
    Foo[str, int, float]  # T bound to str, Ts to Tuple[int, float]

In accordance with Multiple Type Variable Tuples: Not Allowed, at most
one TypeVarTuple may appear in the type parameters to an alias. However,
a TypeVarTuple can be combined with an arbitrary number of TypeVars,
both before and after:

    T1 = TypeVar('T1')
    T2 = TypeVar('T2')
    T3 = TypeVar('T3')

    Tuple[*Ts, T1, T2]      # Valid
    Tuple[T1, T2, *Ts]      # Valid
    Tuple[T1, *Ts, T2, T3]  # Valid

In order to substitute these type variables with supplied type
arguments, any type variables at the beginning or end of the type
parameter list first consume type arguments, and then any remaining type
arguments are bound to the TypeVarTuple:

    Shrubbery = Tuple[*Ts, T1, T2]

    Shrubbery[str, bool]              # T2=bool,  T1=str,   Ts=Tuple[()]
    Shrubbery[str, bool, float]       # T2=float, T1=bool,  Ts=Tuple[str]
    Shrubbery[str, bool, float, int]  # T2=int,   T1=float, Ts=Tuple[str, bool]

    Ptang = Tuple[T1, *Ts, T2, T3]

    Ptang[str, bool, float]       # T1=str, T3=float, T2=bool,  Ts=Tuple[()]
    Ptang[str, bool, float, int]  # T1=str, T3=int,   T2=float, Ts=Tuple[bool]

Note that the minimum number of type arguments in such cases is set by
the number of TypeVars:

    Shrubbery[int]  # Not valid; Shrubbery needs at least two type arguments

Splitting Arbitrary-Length Tuples

A final complication occurs when an unpacked arbitrary-length tuple is
used as a type argument to an alias consisting of both TypeVars and a
TypeVarTuple:

    Elderberries = Tuple[*Ts, T1]
    Hamster = Elderberries[*Tuple[int, ...]]  # valid

In such cases, the arbitrary-length tuple is split between the TypeVars
and the TypeVarTuple. We assume the arbitrary-length tuple contains at
least as many items as there are TypeVars, such that individual
instances of the inner type - here int - are bound to any TypeVars
present. The 'rest' of the arbitrary-length tuple - here
*Tuple[int, ...], since a tuple of arbitrary length minus two items is
still arbitrary-length -is bound to the TypeVarTuple.

Here, therefore, Hamster is equivalent to Tuple[*Tuple[int, ...], int]:
a tuple consisting of zero or more ints, then a final int.

Of course, such splitting only occurs if necessary. For example, if we
instead did:

    Elderberries[*Tuple[int, ...], str]

Then splitting would not occur; T1 would be bound to str, and Ts to
*Tuple[int, ...].

In particularly awkward cases, a TypeVarTuple may consume both a type
and a part of an arbitrary-length tuple type:

    Elderberries[str, *Tuple[int, ...]]

Here, T1 is bound to int, and Ts is bound to
Tuple[str, *Tuple[int, ...]]. This expression is therefore equivalent to
Tuple[str, *Tuple[int, ...], int]: a tuple consisting of a str, then
zero or more ints, ending with an int.

TypeVarTuples Cannot be Split

Finally, although any arbitrary-length tuples in the type argument list
can be split between the type variables and the type variable tuple, the
same is not true of TypeVarTuples in the argument list:

    Ts1 = TypeVarTuple('Ts1')
    Ts2 = TypeVarTuple('Ts2')

    Camelot = Tuple[T, *Ts1]
    Camelot[*Ts2]  # NOT valid

This is not possible because, unlike in the case of an unpacked
arbitrary-length tuple, there is no way to 'peer inside' the
TypeVarTuple to see what its individual types are.

Overloads for Accessing Individual Types

For situations where we require access to each individual type in the
type variable tuple, overloads can be used with individual TypeVar
instances in place of the type variable tuple:

    Shape = TypeVarTuple('Shape')
    Axis1 = TypeVar('Axis1')
    Axis2 = TypeVar('Axis2')
    Axis3 = TypeVar('Axis3')

    class Array(Generic[*Shape]):

      @overload
      def transpose(
        self: Array[Axis1, Axis2]
      ) -> Array[Axis2, Axis1]: ...

      @overload
      def transpose(
        self: Array[Axis1, Axis2, Axis3]
      ) -> Array[Axis3, Axis2, Axis1]: ...

(For array shape operations in particular, having to specify overloads
for each possible rank is, of course, a rather cumbersome solution.
However, it's the best we can do without additional type manipulation
mechanisms. We plan to introduce these in a future PEP.)

Rationale and Rejected Ideas

Shape Arithmetic

Considering the use case of array shapes in particular, note that as of
this PEP, it is not yet possible to describe arithmetic transformations
of array dimensions - for example,
def repeat_each_element(x: Array[N]) -> Array[2*N]. We consider this
out-of-scope for the current PEP, but plan to propose additional
mechanisms that will enable this in a future PEP.

Supporting Variadicity Through Aliases

As noted in the introduction, it is possible to avoid variadic generics
by simply defining aliases for each possible number of type parameters:

    class Array1(Generic[Axis1]): ...
    class Array2(Generic[Axis1, Axis2]): ...

However, this seems somewhat clumsy - it requires users to unnecessarily
pepper their code with 1s, 2s, and so on for each rank necessary.

Construction of TypeVarTuple

TypeVarTuple began as ListVariadic, based on its naming in an early
implementation in Pyre.

We then changed this to TypeVar(list=True), on the basis that a) it
better emphasises the similarity to TypeVar, and b) the meaning of
'list' is more easily understood than the jargon of 'variadic'.

Once we'd decided that a variadic type variable should behave like a
Tuple, we also considered TypeVar(bound=Tuple), which is similarly
intuitive and accomplishes most what we wanted without requiring any new
arguments to TypeVar. However, we realised this may constrain us in the
future, if for example we want type bounds or variance to function
slightly differently for variadic type variables than what the semantics
of TypeVar might otherwise imply. Also, we may later wish to support
arguments that should not be supported by regular type variables (such
as arbitrary_len[8]).

We therefore settled on TypeVarTuple.

Unspecified Type Parameters: Tuple vs TypeVarTuple

In order to support gradual typing, this PEP states that both of the
following examples should type-check correctly:

    def takes_any_array(x: Array): ...
    x: Array[Height, Width]
    takes_any_array(x)

    def takes_specific_array(y: Array[Height, Width]): ...
    y: Array
    takes_specific_array(y)

Note that this is in contrast to the behaviour of the only
currently-existing variadic type in Python, Tuple:

    def takes_any_tuple(x: Tuple): ...
    x: Tuple[int, str]
    takes_any_tuple(x)  # Valid

    def takes_specific_tuple(y: Tuple[int, str]): ...
    y: Tuple
    takes_specific_tuple(y)  # Error

The rules for Tuple were deliberately chosen such that the latter case
is an error: it was thought to be more likely that the programmer has
made a mistake than that the function expects a specific kind of Tuple
but the specific kind of Tuple passed is unknown to the type checker.
Additionally, Tuple is something of a special case, in that it is used
to represent immutable sequences. That is, if an object's type is
inferred to be an unparameterised Tuple, it is not necessarily because
of incomplete typing.

In contrast, if an object's type is inferred to be an unparameterised
Array, it is much more likely that the user has simply not yet fully
annotated their code, or that the signature of a shape-manipulating
library function cannot yet be expressed using the typing system and
therefore returning a plain Array is the only option. We rarely deal
with arrays of truly arbitrary shape; in certain cases, some parts of
the shape will be arbitrary - for example, when dealing with sequences,
the first two parts of the shape are often 'batch' and 'time' - but we
plan to support these cases explicitly in a future PEP with a syntax
such as Array[Batch, Time, ...].

We therefore made the decision to have variadic generics other than
Tuple behave differently, in order to give the user more flexibility in
how much of their code they wish to annotate, and to enable
compatibility between old unannotated code and new versions of libraries
which do use these type annotations.

Alternatives

It should be noted that the approach outlined in this PEP to solve the
issue of shape checking in numerical libraries is not the only approach
possible. Examples of lighter-weight alternatives based on runtime
checking include ShapeGuard[9], tsanley[10], and PyContracts[11].

While these existing approaches improve significantly on the default
situation of shape checking only being possible through lengthy and
verbose assert statements, none of them enable static analysis of shape
correctness. As mentioned in Motivation, this is particularly desirable
for machine learning applications where, due to library and
infrastructure complexity, even relatively simple programs must suffer
long startup times; iterating by running the program until it crashes,
as is necessary with these existing runtime-based approaches, can be a
tedious and frustrating experience.

Our hope with this PEP is to begin to codify generic type annotations as
an official, language-supported way of dealing with shape correctness.
With something of a standard in place, in the long run, this will
hopefully enable a thriving ecosystem of tools for analysing and
verifying shape properties of numerical computing programs.

Grammar Changes

This PEP requires two grammar changes.

Change 1: Star Expressions in Indexes

The first grammar change enables use of star expressions in index
operations (that is, within square brackets), necessary to support
star-unpacking of TypeVarTuples:

    DType = TypeVar('DType')
    Shape = TypeVarTuple('Shape')
    class Array(Generic[DType, *Shape]):
        ...

Before:

    slices:
        | slice !','
        | ','.slice+ [',']

After:

    slices:
        | slice !','
        | ','.(slice | starred_expression)+ [',']

As with star-unpacking in other contexts, the star operator calls
__iter__ on the callee, and adds the contents of the resulting iterator
to the argument passed to __getitem__. For example, if we do
foo[a, *b, c], and b.__iter__ produces an iterator yielding d and e,
foo.__getitem__ would receive (a, d, e, c).

To put it another way, note that x[..., *a, ...] produces the same
result as x[(..., *a, ...)] (with any slices i:j in ... replaced with
slice(i, j), with the one edge case that x[*a] becomes x[(*a,)]).

TypeVarTuple Implementation

With this grammar change, TypeVarTuple is implemented as follows. Note
that this implementation is useful only for the benefit of a) correct
repr() and b) runtime analysers; static analysers would not use the
implementation.

    class TypeVarTuple:
        def __init__(self, name):
            self._name = name
            self._unpacked = UnpackedTypeVarTuple(name)
        def __iter__(self):
            yield self._unpacked
        def __repr__(self):
            return self._name

    class UnpackedTypeVarTuple:
        def __init__(self, name):
            self._name = name
        def __repr__(self):
            return '*' + self._name

Implications

This grammar change implies a number of additional changes in behaviour
not required by this PEP. We choose to allow these additional changes
rather than disallowing them at a syntax level in order to keep the
syntax change as small as possible.

First, the grammar change enables star-unpacking of other structures,
such as lists, within indexing operations:

    idxs = (1, 2)
    array_slice = array[0, *idxs, -1]  # Equivalent to [0, 1, 2, -1]
    array[0, *idxs, -1] = array_slice  # Also allowed

Second, more than one instance of a star-unpack can occur within an
index:

    array[*idxs_to_select, *idxs_to_select]  # Equivalent to array[1, 2, 1, 2]

Note that this PEP disallows multiple unpacked TypeVarTuples within a
single type parameter list. This requirement would therefore need to be
implemented in type checking tools themselves rather than at the syntax
level.

Third, slices may co-occur with starred expressions:

    array[3:5, *idxs_to_select]  # Equivalent to array[3:5, 1, 2]

However, note that slices involving starred expressions are still
invalid:

    # Syntax error
    array[*idxs_start:*idxs_end]

Change 2: *args as a TypeVarTuple

The second change enables use of *args: *Ts in function definitions.

Before:

    star_etc:
    | '*' param_no_default param_maybe_default* [kwds]
    | '*' ',' param_maybe_default+ [kwds]
    | kwds

After:

    star_etc:
    | '*' param_no_default param_maybe_default* [kwds]
    | '*' param_no_default_star_annotation param_maybe_default* [kwds]  # New
    | '*' ',' param_maybe_default+ [kwds]
    | kwds

Where:

    param_no_default_star_annotation:
    | param_star_annotation ',' TYPE_COMMENT?
    | param_star_annotation TYPE_COMMENT? &')'

    param_star_annotation: NAME star_annotation

    star_annotation: ':' star_expression

We also need to deal with the star_expression that results from this
construction. Normally, a star_expression occurs within the context of
e.g. a list, so a star_expression is handled by essentially calling
iter() on the starred object, and inserting the results of the resulting
iterator into the list at the appropriate place. For *args: *Ts,
however, we must process the star_expression in a different way.

We do this by instead making a special case for the star_expression
resulting from *args: *Ts, emitting code equivalent to
[annotation_value] = [*Ts]. That is, we create an iterator from Ts by
calling Ts.__iter__, fetch a single value from the iterator, verify that
the iterator is exhausted, and set that value as the annotation value.
This results in the unpacked TypeVarTuple being set directly as the
runtime annotation for *args:

    >>> Ts = TypeVarTuple('Ts')
    >>> def foo(*args: *Ts): pass
    >>> foo.__annotations__
    {'args': *Ts}
    # *Ts is the repr() of Ts._unpacked, an instance of UnpackedTypeVarTuple

This allows the runtime annotation to be consistent with an AST
representation that uses a Starred node for the annotations of args - in
turn important for tools that rely on the AST such as mypy to correctly
recognise the construction:

    >>> print(ast.dump(ast.parse('def foo(*args: *Ts): pass'), indent=2))
    Module(
      body=[
        FunctionDef(
          name='foo',
          args=arguments(
            posonlyargs=[],
            args=[],
            vararg=arg(
              arg='args',
              annotation=Starred(
                value=Name(id='Ts', ctx=Load()),
                ctx=Load())),
            kwonlyargs=[],
            kw_defaults=[],
            defaults=[]),
          body=[
            Pass()],
          decorator_list=[])],
      type_ignores=[])

Note that the only scenario in which this grammar change allows *Ts to
be used as a direct annotation (rather than being wrapped in e.g.
Tuple[*Ts]) is *args. Other uses are still invalid:

    x: *Ts                 # Syntax error
    def foo(x: *Ts): pass  # Syntax error

Implications

As with the first grammar change, this change also has a number of side
effects. In particular, the annotation of *args could be set to a
starred object other than a TypeVarTuple - for example, the following
nonsensical annotations are possible:

    >>> foo = [1]
    >>> def bar(*args: *foo): pass
    >>> bar.__annotations__
    {'args': 1}

    >>> foo = [1, 2]
    >>> def bar(*args: *foo): pass
    ValueError: too many values to unpack (expected 1)

Again, prevention of such annotations will need to be done by, say,
static checkers, rather than at the level of syntax.

Alternatives (Why Not Just Use Unpack?)

If these grammar changes are considered too burdensome, there are two
alternatives.

The first would be to support change 1 but not change 2. Variadic
generics are more important to us than the ability to annotate *args.

The second alternative would be to use ``Unpack`` instead, requiring no
grammar changes. However, we regard this as a suboptimal solution for
two reasons:

-   Readability. class Array(Generic[DType, Unpack[Shape]]) is a bit of
    a mouthful; the flow of reading is interrupted by length of Unpack
    and the extra set of square brackets.
    class Array(Generic[DType, *Shape]) is much easier to skim, while
    still marking Shape as special.
-   Intuitiveness. We think a user is more likely to intuitively
    understand the meaning of *Ts - especially when they see that Ts is
    a TypeVar**Tuple** - than the meaning of Unpack[Ts]. (This assumes
    the user is familiar with star-unpacking in other contexts; if the
    user is reading or writing code that uses variadic generics, this
    seems reasonable.)

If even change 1 is thought too significant a change, therefore, it
might be better for us to reconsider our options before going ahead with
this second alternative.

Backwards Compatibility

The Unpack version of the PEP should be back-portable to previous
versions of Python.

Gradual typing is enabled by the fact that unparameterised variadic
classes are compatible with an arbitrary number of type parameters. This
means that if existing classes are made generic, a) all existing
(unparameterised) uses of the class will still work, and b)
parameterised and unparameterised versions of the class can be used
together (relevant if, for example, library code is updated to use
parameters while user code is not, or vice-versa).

Reference Implementation

Two reference implementations of type-checking functionality exist: one
in Pyre, as of v0.9.0, and one in Pyright, as of v1.1.108.

A preliminary implementation of the Unpack version of the PEP in CPython
is available in cpython/23527. A preliminary version of the version
using the star operator, based on an early implementation of PEP 637, is
also available at mrahtz/cpython/pep637+646.

Appendix A: Shape Typing Use Cases

To give this PEP additional context for those particularly interested in
the array typing use case, in this appendix we expand on the different
ways this PEP can be used for specifying shape-based subtypes.

Use Case 1: Specifying Shape Values

The simplest way to parameterise array types is using Literal type
parameters - e.g. Array[Literal[64], Literal[64]].

We can attach names to each parameter using normal type variables:

    K = TypeVar('K')
    N = TypeVar('N')

    def matrix_vector_multiply(x: Array[K, N], y: Array[N]) -> Array[K]: ...

    a: Array[Literal[64], Literal[32]]
    b: Array[Literal[32]]
    matrix_vector_multiply(a, b)
    # Result is Array[Literal[64]]

Note that such names have a purely local scope. That is, the name K is
bound to Literal[64] only within matrix_vector_multiply. To put it
another way, there's no relationship between the value of K in different
signatures. This is important: it would be inconvenient if every axis
named K were constrained to have the same value throughout the entire
program.

The disadvantage of this approach is that we have no ability to enforce
shape semantics across different calls. For example, we can't address
the problem mentioned in Motivation: if one function returns an array
with leading dimensions 'Time × Batch', and another function takes the
same array assuming leading dimensions 'Batch × Time', we have no way of
detecting this.

The main advantage is that in some cases, axis sizes really are what we
care about. This is true for both simple linear algebra operations such
as the matrix manipulations above, but also in more complicated
transformations such as convolutional layers in neural networks, where
it would be of great utility to the programmer to be able to inspect the
array size after each layer using static analysis. To aid this, in the
future we would like to explore possibilities for additional type
operators that enable arithmetic on array shapes - for example:

    def repeat_each_element(x: Array[N]) -> Array[Mul[2, N]]: ...

Such arithmetic type operators would only make sense if names such as N
refer to axis size.

Use Case 2: Specifying Shape Semantics

A second approach (the one that most of the examples in this PEP are
based around) is to forgo annotation with actual axis size, and instead
annotate axis type.

This would enable us to solve the problem of enforcing shape properties
across calls. For example:

    # lib.py

    class Batch: pass
    class Time: pass

    def make_array() -> Array[Batch, Time]: ...

    # user.py

    from lib import Batch, Time

    # `Batch` and `Time` have the same identity as in `lib`,
    # so must take array as produced by `lib.make_array`
    def use_array(x: Array[Batch, Time]): ...

Note that in this case, names are global (to the extent that we use the
same Batch type in different place). However, because names refer only
to axis types, this doesn't constrain the value of certain axes to be
the same through (that is, this doesn't constrain all axes named Height
to have a value of, say, 480 throughout).

The argument for this approach is that in many cases, axis type is the
more important thing to verify; we care more about which axis is which
than what the specific size of each axis is.

It also does not preclude cases where we wish to describe shape
transformations without knowing the type ahead of time. For example, we
can still write:

    K = TypeVar('K')
    N = TypeVar('N')

    def matrix_vector_multiply(x: Array[K, N], y: Array[N]) -> Array[K]: ...

We can then use this with:

    class Batch: pass
    class Values: pass

    batch_of_values: Array[Batch, Values]
    value_weights: Array[Values]
    matrix_vector_multiply(batch_of_values, value_weights)
    # Result is Array[Batch]

The disadvantages are the inverse of the advantages from use case 1. In
particular, this approach does not lend itself well to arithmetic on
axis types: Mul[2, Batch] would be as meaningless as 2 * int.

Discussion

Note that use cases 1 and 2 are mutually exclusive in user code. Users
can verify size or semantic type but not both.

As of this PEP, we are agnostic about which approach will provide most
benefit. Since the features introduced in this PEP are compatible with
both approaches, however, we leave the door open.

Why Not Both?

Consider the following 'normal' code:

    def f(x: int): ...

Note that we have symbols for both the value of the thing (x) and the
type of the thing (int). Why can't we do the same with axes? For
example, with an imaginary syntax, we could write:

    def f(array: Array[TimeValue: TimeType]): ...

This would allow us to access the axis size (say, 32) through the symbol
TimeValue and the type through the symbol TypeType.

This might even be possible using existing syntax, through a second
level of parameterisation:

    def f(array: array[TimeValue[TimeType]]): ..

However, we leave exploration of this approach to the future.

Appendix B: Shaped Types vs Named Axes

An issue related to those addressed by this PEP concerns axis selection.
For example, if we have an image stored in an array of shape 64×64x3, we
might wish to convert to black-and-white by computing the mean over the
third axis, mean(image, axis=2). Unfortunately, the simple typo axis=1
is difficult to spot and will produce a result that means something
completely different (all while likely allowing the program to keep on
running, resulting in a bug that is serious but silent).

In response, some libraries have implemented so-called 'named tensors'
(in this context, 'tensor' is synonymous with 'array'), in which axes
are selected not by index but by label - e.g.
mean(image, axis='channels').

A question we are often asked about this PEP is: why not just use named
tensors? The answer is that we consider the named tensors approach
insufficient, for two main reasons:

-   Static checking of shape correctness is not possible. As mentioned
    in Motivation, this is a highly desirable feature in machine
    learning code where iteration times are slow by default.
-   Interface documentation is still not possible with this approach. If
    a function should only be willing to take array arguments that have
    image-like shapes, this cannot be stipulated with named tensors.

Additionally, there's the issue of poor uptake. At the time of writing,
named tensors have only been implemented in a small number of numerical
computing libraries. Possible explanations for this include difficulty
of implementation (the whole API must be modified to allow selection by
axis name instead of index), and lack of usefulness due to the fact that
axis ordering conventions are often strong enough that axis names
provide little benefit (e.g. when working with images, 3D tensors are
basically always height × width × channels). However, ultimately we are
still uncertain why this is the case.

Can the named tensors approach be combined with the approach we advocate
for in this PEP? We're not sure. One area of overlap is that in some
contexts, we could do, say:

    Image: Array[Height, Width, Channels]
    im: Image
    mean(im, axis=Image.axes.index(Channels)

Ideally, we might write something like
im: Array[Height=64, Width=64, Channels=3] -but this won't be possible
in the short term, due to the rejection of PEP 637. In any case, our
attitude towards this is mostly "Wait and see what happens before taking
any further steps".

Footnotes

Endorsements

Variadic generics have a wide range of uses. For the fraction of that
range involving numerical computing, how likely is it that relevant
libraries will actually make use of the features proposed in this PEP?

We reached out to a number of people with this question, and received
the following endorsements.

From Stephan Hoyer, member of the NumPy Steering Council: [12]

  I just wanted to thank Matthew & Pradeep for writing this PEP and for
  clarifications to the broader context of PEP 646 for array typing in
  https://github.com/python/peps/pull/1904.

  As someone who is heavily involved in the Python numerical computing
  community (e.g., NumPy, JAX, Xarray), but who is not so familiar with
  the details of Python's type system, it is reassuring to see that a
  broad range of use-cases related to type checking of named axes &
  shapes have been considered, and could build upon the infrastructure
  in this PEP.

  Type checking for shapes is something the NumPy community is very
  interested in -- there are more thumbs up on the relevant issue on
  NumPy's GitHub than any others
  (https://github.com/numpy/numpy/issues/7370) and we recently added a
  "typing" module that is under active development.

  It will certainly require experimentation to figure out the best ways
  to use type checking for ndarrays, but this PEP looks like an
  excellent foundation for such work.

From Bas van Beek, who has worked on preliminary support for
shape-generics in NumPy:

  I very much share Stephan's opinion here and look forward to
  integrating the new PEP 646 variadics into numpy.

  In the context of numpy (and tensor typing general): the typing of
  array shapes is a fairly complicated subject and the introduction of
  variadics will likely play in big role in laying its foundation, as it
  allows for the expression of both dimensioability as well as basic
  shape manipulation.

  All in all, I'm very interested in where both PEP 646 and future PEPs
  will take us and look forward to further developments.

From Dan Moldovan, a Senior Software Engineer on the TensorFlow Dev Team
and author of the TensorFlow RFC, TensorFlow Canonical Type System:[13]

  I'd be interested in using this the mechanisms defined in this PEP to
  define rank-generic Tensor types in TensorFlow, which are important in
  specifying tf.function signatures in a Pythonic way, using type
  annotations (rather than the custom input_signature mechanism we have
  today - see this issue:
  https://github.com/tensorflow/tensorflow/issues/31579). Variadic
  generics are among the last few missing pieces to create an elegant
  set of type definitions for tensors and shapes.

(For the sake of transparency - we also reached out to folks from a
third popular numerical computing library, PyTorch, but did not receive
a statement of endorsement from them. Our understanding is that although
they are interested in some of the same issues - e.g. static shape
inference - they are currently focusing on enabling this through a DSL
rather than the Python type system.)

Acknowledgements

Thank you to Alfonso Castaño, Antoine Pitrou, Bas v.B., David Foster,
Dimitris Vardoulakis, Eric Traut, Guido van Rossum, Jia Chen, Lucio
Fernandez-Arjona, Nikita Sobolev, Peilonrayz, Rebecca Chen, Sergei
Lebedev, and Vladimir Mikulik for helpful feedback and suggestions on
drafts of this PEP.

Thank you especially to Lucio for suggesting the star syntax (which has
made multiple aspects of this proposal much more concise and intuitive),
and to Stephan Hoyer and Dan Moldovan for their endorsements.

Resources

Discussions on variadic generics in Python started in 2016 with Issue
193 on the python/typing GitHub repository[14].

Inspired by this discussion, Ivan Levkivskyi made a concrete proposal at
PyCon 2019, summarised in notes on 'Type system improvements'[15] and
'Static typing of Python numeric stack'[16].

Expanding on these ideas, Mark Mendoza and Vincent Siles gave a
presentation on 'Variadic Type Variables for Decorators and Tensors'[17]
at the 2019 Python Typing Summit.

Discussion over how type substitution in generic aliases should behave
took place in cpython#91162.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] Python typing issue #193:
https://github.com/python/typing/issues/193

[2] 'Batch' is machine learning parlance for 'a number of'.

[3] We use the term 'array' to refer to a matrix with an arbitrary
number of dimensions. In NumPy, the corresponding class is the ndarray;
in TensorFlow, the Tensor; and so on.

[4] If the shape begins with 'batch × time', then videos_batch[0][1]
would select the second frame of the first video. If the shape begins
with 'time × batch', then videos_batch[1][0] would select the same
frame.

[5] Ivan Levkivskyi, 'Static typing of Python numeric stack', PyCon
2019:
https://paper.dropbox.com/doc/Static-typing-of-Python-numeric-stack-summary-6ZQzTkgN6e0oXko8fEWwN

[6] Stephan Hoyer, 'Ideas for array shape typing in Python':
https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY/edit

[7] Matthew Rahtz et al., 'Shape annotation syntax proposal':
https://docs.google.com/document/d/1But-hjet8-djv519HEKvBN6Ik2lW3yu0ojZo6pG9osY/edit

[8] Discussion on Python typing-sig mailing list:
<EMAIL_ADDRESS>
[9] ShapeGuard: https://github.com/Qwlouse/shapeguard

[10] tsanley: https://github.com/ofnote/tsanley

[11] PyContracts: https://github.com/AndreaCensi/contracts

[12]<EMAIL_ADDRESS>
[13]<EMAIL_ADDRESS>
[14] Python typing issue #193:
https://github.com/python/typing/issues/193

[15] Ivan Levkivskyi, 'Type system improvements', PyCon 2019:
https://paper.dropbox.com/doc/Type-system-improvements-HHOkniMG9WcCgS0LzXZAe

[16] Ivan Levkivskyi, 'Static typing of Python numeric stack', PyCon
2019:
https://paper.dropbox.com/doc/Static-typing-of-Python-numeric-stack-summary-6ZQzTkgN6e0oXko8fEWwN

[17] Mark Mendoza, 'Variadic Type Variables for Decorators and Tensors',
Python Typing Summit 2019:
https://github.com/facebook/pyre-check/blob/ae85c0c6e99e3bbfc92ec55104bfdc5b9b3097b2/docs/Variadic_Type_Variables_for_Decorators_and_Tensors.pdf


text:
PEP: 737 Title: C API to format a type fully qualified name Author:
Victor Stinner<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-737-unify-type-name-formatting/39872
Status: Final Type: Standards Track Created: 29-Nov-2023 Python-Version:
3.13 Post-History: 29-Nov-2023 Resolution:
https://discuss.python.org/t/pep-737-unify-type-name-formatting/39872/60

Abstract

Add new convenient C APIs to format a type fully qualified name. No
longer format type names differently depending on how types are
implemented.

Recommend using the type fully qualified name in error messages and in
__repr__() methods in new C code. Recommend not truncating type names in
new C code.

Add %T, %#T, %N and %#N formats to PyUnicode_FromFormat() to format the
fully qualified, respectively, of an object type and of a type.

Make C code safer by avoiding borrowed reference which can lead to
crashes. The new C API is compatible with the limited C API.

Rationale

Standard library

In the Python standard library, formatting a type name or the type name
of an object is a common operation to format an error message and to
implement a __repr__() method. There are different ways to format a type
name which give different outputs.

Example with the datetime.timedelta type:

-   The type short name (type.__name__) and the type qualified name
    (type.__qualname__) are 'timedelta'.
-   The type module (type.__module__) is 'datetime'.
-   The type fully qualified name is 'datetime.timedelta'.
-   The type representation (repr(type)) contains the fully qualified
    name: <class 'datetime.timedelta'>.

Python code

In Python, type.__name__ gets the type short name, whereas
f"{type.__module__}.{type.__qualname__}" formats the type "fully
qualified name". Usually, type(obj) or obj.__class__ are used to get the
type of the object obj. Sometimes, the type name is put between quotes.

Examples:

-   raise TypeError("str expected, not %s" % type(value).__name__)
-   raise TypeError("can't serialize %s" % self.__class__.__name__)
-   name = "%s.%s" % (obj.__module__, obj.__qualname__)

Qualified names were added to types (type.__qualname__) in Python 3.3 by
PEP 3155 "Qualified name for classes and functions".

C code

In C, the most common way to format a type name is to get the
PyTypeObject.tp_name member of the type. Example:

    PyErr_Format(PyExc_TypeError, "globals must be a dict, not %.100s",
                 Py_TYPE(globals)->tp_name);

The type "fully qualified name" is used in a few places:
PyErr_Display(), type.__repr__() implementation, and sys.unraisablehook
implementation.

Using Py_TYPE(obj)->tp_name is preferred since it is more convenient
than calling PyType_GetQualName() which requires Py_DECREF(). Moreover,
PyType_GetQualName() was only added recently, in Python 3.11.

Some functions use %R (repr(type)) to format a type name, the output
contains the type fully qualified name. Example:

    PyErr_Format(PyExc_TypeError,
                 "calling %R should have returned an instance "
                 "of BaseException, not %R",
                 type, Py_TYPE(value));

Using PyTypeObject.tp_name is inconsistent with Python

The PyTypeObject.tp_name member is different depending on the type
implementation:

-   Static types and heap types in C: tp_name is the type fully
    qualified name.
-   Python class: tp_name is the type short name (type.__name__).

So using Py_TYPE(obj)->tp_name to format an object type name gives a
different output depending if a type is implemented in C or in Python.

It goes against PEP 399 "Pure Python/C Accelerator Module Compatibility
Requirements" principles which recommends code behaves the same way if
written in Python or in C.

Example:

    $ python3.12
    >>> import _datetime; c_obj = _datetime.date(1970, 1, 1)
    >>> import _pydatetime; py_obj = _pydatetime.date(1970, 1, 1)
    >>> my_list = list(range(3))

    >>> my_list[c_obj]  # C type
    TypeError: list indices must be integers or slices, not datetime.date

    >>> my_list[py_obj]  # Python type
    TypeError: list indices must be integers or slices, not date

The error message contains the type fully qualified name (datetime.date)
if the type is implemented in C, or the type short name (date) if the
type is implemented in Python.

Limited C API

The Py_TYPE(obj)->tp_name code cannot be used with the limited C API,
since the PyTypeObject members are excluded from the limited C API.

The type name should be read using PyType_GetName(),
PyType_GetQualName() and PyType_GetModule() functions which are less
convenient to use.

Truncating type names in C

In 1998, when the PyErr_Format() function was added, the implementation
used a fixed buffer of 500 bytes. The function had the following
comment:

    /* Caller is responsible for limiting the format */

In 2001, the function was modified to allocate a dynamic buffer on the
heap. Too late, the practice of truncating type names, like using the
%.100s format, already became a habit, and developers forgot why type
names are truncated. In Python, type names are not truncated.

Truncating type names in C but not in Python goes against PEP 399 "Pure
Python/C Accelerator Module Compatibility Requirements" principles which
recommends code behaves the same way if written in Python or in C.

See the issue: Replace %.100s by %s in PyErr_Format(): the arbitrary
limit of 500 bytes is outdated (2011).

Specification

-   Add PyType_GetFullyQualifiedName() function.
-   Add PyType_GetModuleName() function.
-   Add formats to PyUnicode_FromFormat().
-   Recommend using the type fully qualified name in error messages and
    in __repr__() methods in new C code.
-   Recommend not truncating type names in new C code.

Add PyType_GetFullyQualifiedName() function

Add the PyType_GetFullyQualifiedName() function to get the fully
qualified name of a type: similar to
f"{type.__module__}.{type.__qualname__}", or type.__qualname__ if
type.__module__ is not a string or is equal to "builtins" or is equal to
"__main__".

API:

    PyObject* PyType_GetFullyQualifiedName(PyTypeObject *type)

On success, return a new reference to the string. On error, raise an
exception and return NULL.

Add PyType_GetModuleName() function

Add the PyType_GetModuleName() function to get the module name of a type
(type.__module__ string). API:

    PyObject* PyType_GetModuleName(PyTypeObject *type)

On success, return a new reference to the string. On error, raise an
exception and return NULL.

Add formats to PyUnicode_FromFormat()

Add the following formats to PyUnicode_FromFormat():

-   %N formats the fully qualified name of a type, similar to
    PyType_GetFullyQualifiedName(type); N stands for type Name.
-   %T formats the type fully qualified name of an object's type,
    similar to PyType_GetFullyQualifiedName(Py_TYPE(obj)); T stands for
    object Type.
-   %#N and %#T: the alternative form uses the colon separator (:),
    instead of the dot separator (.), between the module name and the
    qualified name.

For example, the existing code using tp_name:

    PyErr_Format(PyExc_TypeError,
                 "__format__ must return a str, not %.200s",
                 Py_TYPE(result)->tp_name);

can be replaced with the %T format:

    PyErr_Format(PyExc_TypeError,
                 "__format__ must return a str, not %T", result);

Advantages of the updated code:

-   Safer C code: avoid Py_TYPE() which returns a borrowed reference.
-   The PyTypeObject.tp_name member is no longer read explicitly: the
    code becomes compatible with the limited C API.
-   The formatted type name no longer depends on the type
    implementation.
-   The type name is no longer truncated.

Note: The %T format is used by time.strftime(), but not by printf().

Formats Summary

  C object   C type   Format
  ---------- -------- ---------------------------------------------
  %T         %N       Type fully qualified name.
  %#T        %#N      Type fully qualified name, colon separator.

Recommend using the type fully qualified name

The type fully qualified name is recommended in error messages and in
__repr__() methods in new C code.

In non-trivial applications, it is likely to have two types with the
same short name defined in two different modules, especially with
generic names. Using the fully qualified name helps identifying the type
in an unambiguous way.

Recommend not truncating type names

Type names should not be truncated in new C code. For example, the
%.100s format should be avoided: use the %s format instead (or %T format
in C).

Implementation

-   Pull request: Add type.__fully_qualified_name__ attribute.
-   Pull request: Add %T format to PyUnicode_FromFormat().

Backwards Compatibility

Changes proposed in this PEP are backward compatible.

Adding new C APIs has no effect on the backward compatibility. Existing
C APIs are left unchanged. No Python API is changed.

Replacing the type short name with the type fully qualified name is only
recommended in new C code. No longer truncating type names is only
recommended in new C code. Existing code should be left unchanged and so
remains backward compatible. There is no recommendation for Python code.

Rejected Ideas

Add type.__fully_qualified_name__ attribute

Add type.__fully_qualified_name__ read-only attribute, the fully
qualified name of a type: similar to
f"{type.__module__}.{type.__qualname__}", or type.__qualname__ if
type.__module__ is not a string or is equal to "builtins" or is equal to
"__main__".

The type.__repr__() is left unchanged, it only omits the module if the
module is equal to "builtins".

This change was rejected by the Steering Council:

  We can see the usefulness of the C API changes proposed by the PEP and
  would likely accept those changes as is.

  We see less justification for the Python level changes. We especially
  question the need for __fully_qualified_name__.

Thomas Wouters added:

  If there really is a desire for formatting types the exact same way
  the C API does it, a utility function would make more sense to me,
  personally, than type.__format__, but I think the SC could be
  persuaded given some concrete use-cases.

Add type.__format__() method

Add type.__format__() method with the following formats:

-   N formats the type fully qualified name
    (type.__fully_qualified_name__); N stands for Name.
-   #N (alternative form) formats the type fully qualified name using
    the colon (:) separator, instead of the dot separator (.), between
    the module name and the qualified name.

Examples using f-string:

    >>> import datetime
    >>> f"{datetime.timedelta:N}"  # fully qualified name
    'datetime.timedelta'
    >>> f"{datetime.timedelta:#N}" # fully qualified name, colon separator
    'datetime:timedelta'

The colon (:) separator used by the #N format eliminates guesswork when
you want to import the name, see pkgutil.resolve_name(),
python -m inspect command line interface, and setuptools entry points.

This change was rejected by the Steering Council.

Change str(type)

The type.__str__() method can be modified to format a type name
differently. For example, it can return the type fully qualified name.

The problem is that it's a backward incompatible change. For example,
enum, functools, optparse, pdb and xmlrpc.server modules of the standard
library have to be updated. test_dataclasses, test_descrtut and
test_cmd_line_script tests have to be updated as well.

See the pull request: type(str) returns the fully qualified name.

Add !t formatter to get an object type

Use f"{obj!t:T}" to format type(obj).__fully_qualified_name__, similar
to f"{type(obj):T}".

When the !t formatter was proposed in 2018, Eric Smith was strongly
opposed to this; Eric is the author of the f-string PEP 498 "Literal
String Interpolation".

Add formats to str % args

It was proposed to add formats to format a type name in str % arg. For
example, add the %T format to format a type fully qualified name.

Nowadays, f-strings are preferred for new code.

Other ways to format type names in C

The printf() function supports multiple size modifiers: hh (char), h
(short), l (long), ll (long long), z (size_t), t (ptrdiff_t) and j
(intmax_t). The PyUnicode_FromFormat() function supports most of them.

Proposed formats using h and hh length modifiers:

-   %hhT formats type.__name__.
-   %hT formats type.__qualname__.
-   %T formats type.__fully_qualified_name__.

Length modifiers are used to specify the C type of the argument, not to
change how an argument is formatted. The alternate form (#) changes how
an argument is formatted. Here the argument C type is always PyObject*.

Other proposed formats:

-   %Q
-   %t.
-   %lT formats type.__fully_qualified_name__.
-   %Tn formats type.__name__.
-   %Tq formats type.__qualname__.
-   %Tf formats type.__fully_qualified_name__.

Having more options to format type names can lead to inconsistencies
between different modules and make the API more error prone.

About the %t format, printf() now uses t as a length modifier for
ptrdiff_t argument.

The following APIs to be used to format a type:

  C API                    Python API          Format
  ------------------------ ------------------- ----------------------
  PyType_GetName()         type.__name__       Type short name.
  PyType_GetQualName()     type.__qualname__   Type qualified name.
  PyType_GetModuleName()   type.__module__     Type module name.

Use %T format with Py_TYPE(): pass a type

It was proposed to pass a type to the %T format, like:

    PyErr_Format(PyExc_TypeError, "object type name: %T", Py_TYPE(obj));

The Py_TYPE() functions returns a borrowed reference. Just to format an
error, using a borrowed reference to a type looks safe. In practice, it
can lead to crash. Example:

    import gc
    import my_cext

    class ClassA:
        pass

    def create_object():
         class ClassB:
              def __repr__(self):
                    self.__class__ = ClassA
                    gc.collect()
                    return "ClassB repr"
         return ClassB()

    obj = create_object()
    my_cext.func(obj)

where my_cext.func() is a C function which calls:

    PyErr_Format(PyExc_ValueError,
                 "Unexpected value %R of type %T",
                 obj, Py_TYPE(obj));

PyErr_Format() is called with a borrowed reference to ClassB. When
repr(obj) is called by the %R format, the last reference to ClassB is
removed and the class is deallocated. When the %T format is proceed,
Py_TYPE(obj) is already a dangling pointer and Python does crash.

Other proposed APIs to get a type fully qualified name

-   Add type.__fullyqualname__ attribute: name without underscore
    between words. Several dunders, including some of the most recently
    added ones, include an underscore in the word: __class_getitem__,
    __release_buffer__, __type_params__, __init_subclass__ and
    __text_signature__.
-   Add type.__fqn__ attribute: FQN name stands for Fully Qualified
    Name.
-   Add type.fully_qualified_name() method. Methods added to type are
    inherited by all types and so can affect existing code.
-   Add a function to the inspect module. Need to import the inspect
    module to use it.

Include the __main__ module in the type fully qualified name

Format type.__fully_qualified_name__ as
f"{type.__module__}.{type.__qualname__}", or type.__qualname__ if
type.__module__ is not a string or is equal to "builtins". Do not treat
the __main__ module differently: include it in the name.

Existing code such as type.__repr__(), collections.abc and unittest
modules format a type name with f'{obj.__module__}.{obj.__qualname__}'
and only omit the module part if the module is equal to builtins.

Only the traceback and pdb modules also omit the module if it's equal to
"builtins" or "__main__".

The type.__fully_qualified_name__ attribute omits the __main__ module to
produce shorter names for a common case: types defined in a script run
with python script.py. For debugging, the repr() function can be used on
a type, it includes the __main__ module in the type name. Or use
f"{type.__module__}.{type.__qualname__}" format to always include the
module name, even for the "builtins" module.

Example of script:

    class MyType:
        pass

    print(f"name: {MyType.__fully_qualified_name__}")
    print(f"repr: {repr(MyType)}")

Output:

    name: MyType
    repr: <class '__main__.MyType'>

Discussions

-   Discourse: PEP 737 – Unify type name formatting (2023).
-   Discourse: Enhance type name formatting when raising an exception:
    add %T format in C, and add type.__fullyqualname__ (2023).
-   Issue: PyUnicode_FromFormat(): Add %T format to format the type name
    of an object (2023).
-   Issue: C API: Investigate how the PyTypeObject members can be
    removed from the public C API (2023).
-   python-dev thread: bpo-34595: How to format a type name? (2018).
-   Issue: PyUnicode_FromFormat(): add %T format for an object type name
    (2018).
-   Issue: Replace %.100s by %s in PyErr_Format(): the arbitrary limit
    of 500 bytes is outdated (2011).

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

license: Public Domain


text:
PEP: 276 Title: Simple Iterator for ints Author: Jim Althoff
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 12-Nov-2001 Python-Version: 2.3
Post-History:

Abstract

Python 2.1 added new functionality to support iterators (PEP 234).
Iterators have proven to be useful and convenient in many coding
situations. It is noted that the implementation of Python's for-loop
control structure uses the iterator protocol as of release 2.1. It is
also noted that Python provides iterators for the following builtin
types: lists, tuples, dictionaries, strings, and files. This PEP
proposes the addition of an iterator for the builtin type int
(types.IntType). Such an iterator would simplify the coding of certain
for-loops in Python.

BDFL Pronouncement

This PEP was rejected on 17 June 2005 with a note to python-dev.

Much of the original need was met by the enumerate() function which was
accepted for Python 2.3.

Also, the proposal both allowed and encouraged misuses such as:

    >>> for i in 3: print i
    0
    1
    2

Likewise, it was not helpful that the proposal would disable the syntax
error in statements like:

    x, = 1

Specification

Define an iterator for types.intType (i.e., the builtin type "int") that
is returned from the builtin function "iter" when called with an
instance of types.intType as the argument.

The returned iterator has the following behavior:

-   Assume that object i is an instance of types.intType (the builtin
    type int) and that i > 0

-   iter(i) returns an iterator object

-   said iterator object iterates through the sequence of ints
    0,1,2,...,i-1

    Example:

      iter(5) returns an iterator object that iterates through the
      sequence of ints 0,1,2,3,4

-   if i <= 0, iter(i) returns an "empty" iterator, i.e., one that
    throws StopIteration upon the first call of its "next" method

In other words, the conditions and semantics of said iterator is
consistent with the conditions and semantics of the range() and xrange()
functions.

Note that the sequence 0,1,2,...,i-1 associated with the int i is
considered "natural" in the context of Python programming because it is
consistent with the builtin indexing protocol of sequences in Python.
Python lists and tuples, for example, are indexed starting at 0 and
ending at len(object)-1 (when using positive indices). In other words,
such objects are indexed with the sequence 0,1,2,...,len(object)-1

Rationale

A common programming idiom is to take a collection of objects and apply
some operation to each item in the collection in some established
sequential order. Python provides the "for in" looping control structure
for handling this common idiom. Cases arise, however, where it is
necessary (or more convenient) to access each item in an "indexed"
collection by iterating through each index and accessing each item in
the collection using the corresponding index.

For example, one might have a two-dimensional "table" object where one
requires the application of some operation to the first column of each
row in the table. Depending on the implementation of the table it might
not be possible to access first each row and then each column as
individual objects. It might, rather, be possible to access a cell in
the table using a row index and a column index. In such a case it is
necessary to use an idiom where one iterates through a sequence of
indices (indexes) in order to access the desired items in the table.
(Note that the commonly used DefaultTableModel class in
Java-Swing-Jython has this very protocol).

Another common example is where one needs to process two or more
collections in parallel. Another example is where one needs to access,
say, every second item in a collection.

There are many other examples where access to items in a collection is
facilitated by a computation on an index thus necessitating access to
the indices rather than direct access to the items themselves.

Let's call this idiom the "indexed for-loop" idiom. Some programming
languages provide builtin syntax for handling this idiom. In Python the
common convention for implementing the indexed for-loop idiom is to use
the builtin range() or xrange() function to generate a sequence of
indices as in, for example:

    for rowcount in range(table.getRowCount()):
        print table.getValueAt(rowcount, 0)

or

    for rowcount in xrange(table.getRowCount()):
        print table.getValueAt(rowcount, 0)

From time to time there are discussions in the Python community about
the indexed for-loop idiom. It is sometimes argued that the need for
using the range() or xrange() function for this design idiom is:

-   Not obvious (to new-to-Python programmers),
-   Error prone (easy to forget, even for experienced Python
    programmers)
-   Confusing and distracting for those who feel compelled to understand
    the differences and recommended usage of xrange() vis-a-vis range()
-   Unwieldy, especially when combined with the len() function, i.e.,
    xrange(len(sequence))
-   Not as convenient as equivalent mechanisms in other languages,
-   Annoying, a "wart", etc.

And from time to time proposals are put forth for ways in which Python
could provide a better mechanism for this idiom. Recent examples include
PEP 204, "Range Literals", and PEP 212, "Loop Counter Iteration".

Most often, such proposal include changes to Python's syntax and other
"heavyweight" changes.

Part of the difficulty here is that advocating new syntax implies a
comprehensive solution for "general indexing" that has to include
aspects like:

-   starting index value
-   ending index value
-   step value
-   open intervals versus closed intervals versus half opened intervals

Finding a new syntax that is comprehensive, simple, general, Pythonic,
appealing to many, easy to implement, not in conflict with existing
structures, not excessively overloading of existing structures, etc. has
proven to be more difficult than one might anticipate.

The proposal outlined in this PEP tries to address the problem by
suggesting a simple "lightweight" solution that helps the most common
case by using a proven mechanism that is already available (as of Python
2.1): namely, iterators.

Because for-loops already use "iterator" protocol as of Python 2.1,
adding an iterator for types.IntType as proposed in this PEP would
enable by default the following shortcut for the indexed for-loop idiom:

    for rowcount in table.getRowCount():
        print table.getValueAt(rowcount, 0)

The following benefits for this approach vis-a-vis the current mechanism
of using the range() or xrange() functions are claimed to be:

-   Simpler,
-   Less cluttered,
-   Focuses on the problem at hand without the need to resort to
    secondary implementation-oriented functions (range() and xrange())

And compared to other proposals for change:

-   Requires no new syntax
-   Requires no new keywords
-   Takes advantage of the new and well-established iterator mechanism

And generally:

-   Is consistent with iterator-based "convenience" changes already
    included (as of Python 2.1) for other builtin types such as: lists,
    tuples, dictionaries, strings, and files.

Backwards Compatibility

The proposed mechanism is generally backwards compatible as it calls for
neither new syntax nor new keywords. All existing, valid Python programs
should continue to work unmodified.

However, this proposal is not perfectly backwards compatible in the
sense that certain statements that are currently invalid would, under
the current proposal, become valid.

Tim Peters has pointed out two such examples:

1)  The common case where one forgets to include range() or xrange(),
    for example:

        for rowcount in table.getRowCount():
            print table.getValueAt(rowcount, 0)

    in Python 2.2 raises a TypeError exception.

    Under the current proposal, the above statement would be valid and
    would work as (presumably) intended. Presumably, this is a good
    thing.

    As noted by Tim, this is the common case of the "forgotten range"
    mistake (which one currently corrects by adding a call to range() or
    xrange()).

2)  The (hopefully) very uncommon case where one makes a typing mistake
    when using tuple unpacking. For example:

        x, = 1

    in Python 2.2 raises a TypeError exception.

    Under the current proposal, the above statement would be valid and
    would set x to 0. The PEP author has no data as to how common this
    typing error is nor how difficult it would be to catch such an error
    under the current proposal. He imagines that it does not occur
    frequently and that it would be relatively easy to correct should it
    happen.

Issues

Extensive discussions concerning PEP 276 on the Python interest mailing
list suggests a range of opinions: some in favor, some neutral, some
against. Those in favor tend to agree with the claims above of the
usefulness, convenience, ease of learning, and simplicity of a simple
iterator for integers.

Issues with PEP 276 include:

-   Using range/xrange is fine as is.

    Response: Some posters feel this way. Other disagree.

-   Some feel that iterating over the sequence "0, 1, 2, ..., n-1" for
    an integer n is not intuitive. "for i in 5:" is considered (by some)
    to be "non-obvious", for example. Some dislike this usage because it
    doesn't have "the right feel". Some dislike it because they believe
    that this type of usage forces one to view integers as a sequences
    and this seems wrong to them. Some dislike it because they prefer to
    view for-loops as dealing with explicit sequences rather than with
    arbitrary iterators.

    Response: Some like the proposed idiom and see it as simple,
    elegant, easy to learn, and easy to use. Some are neutral on this
    issue. Others, as noted, dislike it.

-   Is it obvious that iter(5) maps to the sequence 0,1,2,3,4?

    Response: Given, as noted above, that Python has a strong convention
    for indexing sequences starting at 0 and stopping at (inclusively)
    the index whose value is one less than the length of the sequence,
    it is argued that the proposed sequence is reasonably intuitive to
    the Python programmer while being useful and practical. More
    importantly, it is argued that once learned this convention is very
    easy to remember. Note that the doc string for the range function
    makes a reference to the natural and useful association between
    range(n) and the indices for a list whose length is n.

-   Possible ambiguity

        for i in 10: print i

    might be mistaken for

        for i in (10,): print i

    Response: This is exactly the same situation with strings in current
    Python (replace 10 with 'spam' in the above, for example).

-   Too general: in the newest releases of Python there are contexts --
    as with for-loops -- where iterators are called implicitly. Some
    fear that having an iterator invoked for an integer in one of the
    context (excluding for-loops) might lead to unexpected behavior and
    bugs. The "x, = 1" example noted above is an a case in point.

    Response: From the author's perspective the examples of the above
    that were identified in the PEP 276 discussions did not appear to be
    ones that would be accidentally misused in ways that would lead to
    subtle and hard-to-detect errors.

    In addition, it seems that there is a way to deal with this issue by
    using a variation of what is outlined in the specification section
    of this proposal. Instead of adding an __iter__ method to class int,
    change the for-loop handling code to convert (in essence) from

        for i in n:  # when isinstance(n,int) is 1

    to

        for i in xrange(n):

    This approach gives the same results in a for-loop as an __iter__
    method would but would prevent iteration on integer values in any
    other context. Lists and tuples, for example, don't have __iter__
    and are handled with special code. Integer values would be one more
    special case.

-   "i in n" seems very unnatural.

    Response: Some feel that "i in len(mylist)" would be easily
    understandable and useful. Some don't like it, particularly when a
    literal is used as in "i in 5". If the variant mentioned in the
    response to the previous issue is implemented, this issue is moot.
    If not, then one could also address this issue by defining a
    __contains__ method in class int that would always raise a
    TypeError. This would then make the behavior of "i in n" identical
    to that of current Python.

-   Might dissuade newbies from using the indexed for-loop idiom when
    the standard "for item in collection:" idiom is clearly better.

    Response: The standard idiom is so nice when it fits that it needs
    neither extra "carrot" nor "stick". On the other hand, one does
    notice cases of overuse/misuse of the standard idiom (due, most
    likely, to the awkwardness of the indexed for-loop idiom), as in:

        for item in sequence:
            print sequence.index(item)

-   Why not propose even bigger changes?

The majority of disagreement with PEP 276 came from those who favor much
larger changes to Python to address the more general problem of
specifying a sequence of integers where such a specification is general
enough to handle the starting value, ending value, and stepping value of
the sequence and also addresses variations of open, closed, and
half-open (half-closed) integer intervals. Many suggestions of such were
discussed.

These include:

-   adding Haskell-like notation for specifying a sequence of integers
    in a literal list,
-   various uses of slicing notation to specify sequences,
-   changes to the syntax of for-in loops to allow the use of relational
    operators in the loop header,
-   creation of an integer-interval class along with methods that
    overload relational operators or division operators to provide
    "slicing" on integer-interval objects,
-   and more.

It should be noted that there was much debate but not an overwhelming
consensus for any of these larger-scale suggestions.

Clearly, PEP 276 does not propose such a large-scale change and instead
focuses on a specific problem area. Towards the end of the discussion
period, several posters expressed favor for the narrow focus and
simplicity of PEP 276 vis-a-vis the more ambitious suggestions that were
advanced. There did appear to be consensus for the need for a PEP for
any such larger-scale, alternative suggestion. In light of this
recognition, details of the various alternative suggestions are not
discussed here further.

Implementation

An implementation is not available at this time but is expected to be
straightforward. The author has implemented a subclass of int with an
__iter__ method (written in Python) as a means to test out the ideas in
this proposal, however.

Copyright

This document has been placed in the public domain.


text:
PEP: 400 Title: Deprecate codecs.StreamReader and codecs.StreamWriter
Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner
<EMAIL_ADDRESS>Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 28-May-2011 Python-Version: 3.3

Abstract

io.TextIOWrapper and codecs.StreamReaderWriter offer the same API [1].
TextIOWrapper has more features and is faster than StreamReaderWriter.
Duplicate code means that bugs should be fixed twice and that we may
have subtle differences between the two implementations.

The codecs module was introduced in Python 2.0 (see the PEP 100). The io
module was introduced in Python 2.6 and 3.0 (see the PEP 3116), and
reimplemented in C in Python 2.7 and 3.1.

PEP Deferral

Further exploration of the concepts covered in this PEP has been
deferred for lack of a current champion interested in promoting the
goals of the PEP and collecting and incorporating feedback, and with
sufficient available time to do so effectively.

Motivation

When the Python I/O model was updated for 3.0, the concept of a
"stream-with-known-encoding" was introduced in the form of
io.TextIOWrapper. As this class is critical to the performance of
text-based I/O in Python 3, this module has an optimised C version which
is used by CPython by default. Many corner cases in handling buffering,
stateful codecs and universal newlines have been dealt with since the
release of Python 3.0.

This new interface overlaps heavily with the legacy codecs.StreamReader,
codecs.StreamWriter and codecs.StreamReaderWriter interfaces that were
part of the original codec interface design in PEP 100. These interfaces
are organised around the principle of an encoding with an associated
stream (i.e. the reverse of arrangement in the io module), so the
original PEP 100 design required that codec writers provide appropriate
StreamReader and StreamWriter implementations in addition to the core
codec encode() and decode() methods. This places a heavy burden on codec
authors providing these specialised implementations to correctly handle
many of the corner cases (see Appendix A) that have now been dealt with
by io.TextIOWrapper. While deeper integration between the codec and the
stream allows for additional optimisations in theory, these
optimisations have in practice either not been carried out and else the
associated code duplication means that the corner cases that have been
fixed in io.TextIOWrapper are still not handled correctly in the various
StreamReader and StreamWriter implementations.

Accordingly, this PEP proposes that:

-   codecs.open() be updated to delegate to the builtin open() in Python
    3.3;
-   the legacy codecs.Stream* interfaces, including the streamreader and
    streamwriter attributes of codecs.CodecInfo be deprecated in Python
    3.3.

Rationale

StreamReader and StreamWriter issues

-   StreamReader is unable to translate newlines.
-   StreamWriter doesn't support "line buffering" (flush if the input
    text contains a newline).
-   StreamReader classes of the CJK encodings (e.g. GB18030) only
    supports UNIX newlines ('\n').
-   StreamReader and StreamWriter are stateful codecs but don't expose
    functions to control their state (getstate() or setstate()). Each
    codec has to handle corner cases, see Appendix A.
-   StreamReader and StreamWriter are very similar to IncrementalReader
    and IncrementalEncoder, some code is duplicated for stateful codecs
    (e.g. UTF-16).
-   Each codec has to reimplement its own StreamReader and StreamWriter
    class, even if it's trivial (just call the encoder/decoder).
-   codecs.open(filename, "r") creates an io.TextIOWrapper object.
-   No codec implements an optimized method in StreamReader or
    StreamWriter based on the specificities of the codec.

Issues in the bug tracker:

-   Issue #5445 (2009-03-08): codecs.StreamWriter.writelines problem
    when passed generator
-   Issue #7262: (2009-11-04): codecs.open() + eol (windows)
-   Issue #8260 (2010-03-29): When I use codecs.open(...) and
    f.readline() follow up by f.read() return bad result
-   Issue #8630 (2010-05-05): Keepends param in codec readline(s)
-   Issue #10344 (2010-11-06): codecs.readline doesn't care buffering
-   Issue #11461 (2011-03-10): Reading UTF-16 with codecs.readline()
    breaks on surrogate pairs
-   Issue #12446 (2011-06-30): StreamReader Readlines behavior odd
-   Issue #12508 (2011-07-06): Codecs Anomaly
-   Issue #12512 (2011-07-07): codecs: StreamWriter issues with stateful
    codecs after a seek or with append mode
-   Issue #12513 (2011-07-07): codec.StreamReaderWriter: issues with
    interlaced read-write

TextIOWrapper features

-   TextIOWrapper supports any kind of newline, including translating
    newlines (to UNIX newlines), to read and write.
-   TextIOWrapper reuses codecs incremental encoders and decoders (no
    duplication of code).
-   The io module (TextIOWrapper) is faster than the codecs module
    (StreamReader). It is implemented in C, whereas codecs is
    implemented in Python.
-   TextIOWrapper has a readahead algorithm which speeds up small reads:
    read character by character or line by line (io is 10x through 25x
    faster than codecs on these operations).
-   TextIOWrapper has a write buffer.
-   TextIOWrapper.tell() is optimized.
-   TextIOWrapper supports random access (read+write) using a single
    class which permit to optimize interlaced read-write (but no such
    optimization is implemented).

TextIOWrapper issues

-   Issue #12215 (2011-05-30): TextIOWrapper: issues with interlaced
    read-write

Possible improvements of StreamReader and StreamWriter

By adding codec state read/write functions to the StreamReader and
StreamWriter classes, it will become possible to fix issues with
stateful codecs in a base class instead of in each stateful StreamReader
and StreamWriter classes.

It would be possible to change StreamReader and StreamWriter to make
them use IncrementalDecoder and IncrementalEncoder.

A codec can implement variants which are optimized for the specific
encoding or intercept certain stream methods to add functionality or
improve the encoding/decoding performance. TextIOWrapper cannot
implement such optimization, but TextIOWrapper uses incremental encoders
and decoders and uses read and write buffers, so the overhead of
incomplete inputs is low or nul.

A lot more could be done for other variable length encoding codecs, e.g.
UTF-8, since these often have problems near the end of a read due to
missing bytes. The UTF-32-BE/LE codecs could simply multiply the
character position by 4 to get the byte position.

Usage of StreamReader and StreamWriter

These classes are rarely used directly, but indirectly using
codecs.open(). They are not used in Python 3 standard library (except in
the codecs module).

Some projects implement their own codec with StreamReader and
StreamWriter, but don't use these classes.

Backwards Compatibility

Keep the public API, codecs.open

codecs.open() can be replaced by the builtin open() function. open() has
a similar API but has also more options. Both functions return file-like
objects (same API).

codecs.open() was the only way to open a text file in Unicode mode until
Python 2.6. Many Python 2 programs uses this function. Removing
codecs.open() implies more work to port programs from Python 2 to Python
3, especially projects using the same code base for the two Python
versions (without using 2to3 program).

codecs.open() is kept for backward compatibility with Python 2.

Deprecate StreamReader and StreamWriter

Instantiating StreamReader or StreamWriter must emit a
DeprecationWarning in Python 3.3. Defining a subclass doesn't emit a
DeprecationWarning.

codecs.open() will be changed to reuse the builtin open() function
(TextIOWrapper) to read-write text files.

Alternative Approach

An alternative to the deprecation of the codecs.Stream* classes is to
rename codecs.open() to codecs.open_stream(), and to create a new
codecs.open() function reusing open() and so io.TextIOWrapper.

Appendix A: Issues with stateful codecs

It is difficult to use correctly a stateful codec with a stream. Some
cases are supported by the codecs module, while io has no more known bug
related to stateful codecs. The main difference between the codecs and
the io module is that bugs have to be fixed in StreamReader and/or
StreamWriter classes of each codec for the codecs module, whereas bugs
can be fixed only once in io.TextIOWrapper. Here are some examples of
issues with stateful codecs.

Stateful codecs

Python supports the following stateful codecs:

-   cp932
-   cp949
-   cp950
-   euc_jis_2004
-   euc_jisx2003
-   euc_jp
-   euc_kr
-   gb18030
-   gbk
-   hz
-   iso2022_jp
-   iso2022_jp_1
-   iso2022_jp_2
-   iso2022_jp_2004
-   iso2022_jp_3
-   iso2022_jp_ext
-   iso2022_kr
-   shift_jis
-   shift_jis_2004
-   shift_jisx0213
-   utf_8_sig
-   utf_16
-   utf_32

Read and seek(0)

    with open(filename, 'w', encoding='utf-16') as f:
        f.write('abc')
        f.write('def')
        f.seek(0)
        assert f.read() == 'abcdef'
        f.seek(0)
        assert f.read() == 'abcdef'

The io and codecs modules support this usecase correctly.

seek(n)

    with open(filename, 'w', encoding='utf-16') as f:
        f.write('abc')
        pos = f.tell()
    with open(filename, 'w', encoding='utf-16') as f:
        f.seek(pos)
        f.write('def')
        f.seek(0)
        f.write('###')
    with open(filename, 'r', encoding='utf-16') as f:
        assert f.read() == '###def'

The io module supports this usecase, whereas codecs fails because it
writes a new BOM on the second write (issue #12512).

Append mode

    with open(filename, 'w', encoding='utf-16') as f:
        f.write('abc')
    with open(filename, 'a', encoding='utf-16') as f:
        f.write('def')
    with open(filename, 'r', encoding='utf-16') as f:
        assert f.read() == 'abcdef'

The io module supports this usecase, whereas codecs fails because it
writes a new BOM on the second write (issue #12512).

Links

-   PEP 100: Python Unicode Integration <100>
-   PEP 3116: New I/O <3116>
-   Issue #8796: Deprecate codecs.open()
-   [python-dev] Deprecate codecs.open() and StreamWriter/StreamReader

Copyright

This document has been placed in the public domain.

Footnotes

[1] StreamReaderWriter has two more attributes than TextIOWrapper,
reader and writer.