text:
PEP: 728 Title: TypedDict with Typed Extra Items Author: Zixuan James Li
<EMAIL_ADDRESS>Sponsor: Jelle Zijlstra
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443
Status: Draft Type: Standards Track Topic: Typing Content-Type:
text/x-rst Created: 12-Sep-2023 Python-Version: 3.13 Post-History:
09-Feb-2024,

Abstract

This PEP proposes a way to limit extra items for ~typing.TypedDict using
a closed argument and to type them with the special __extra_items__ key.
This addresses the need to define closed TypedDict type or to type a
subset of keys that might appear in a dict while permitting additional
items of a specified type.

Motivation

A typing.TypedDict type can annotate the value type of each known item
in a dictionary. However, due to structural subtyping, a TypedDict can
have extra items that are not visible through its type. There is
currently no way to restrict the types of items that might be present in
the TypedDict type's structural subtypes.

Defining a Closed TypedDict Type

The current behavior of TypedDict prevents users from defining a closed
TypedDict type when it is expected that the type contains no additional
items.

Due to the possible presence of extra items, type checkers cannot infer
more precise return types for .items() and .values() on a TypedDict.
This can also be resolved by defining a closed TypedDict type.

Another possible use case for this is a sound way to enable type
narrowing with the in check:

    class Movie(TypedDict):
        name: str
        director: str

    class Book(TypedDict):
        name: str
        author: str

    def fun(entry: Movie | Book) -> None:
        if "author" in entry:
            reveal_type(entry)  # Revealed type is 'Movie | Book'

Nothing prevents a dict that is structurally compatible with Movie to
have the author key, and under the current specification it would be
incorrect for the type checker to narrow its type.

Allowing Extra Items of a Certain Type

For supporting API interfaces or legacy codebase where only a subset of
possible keys are known, it would be useful to explicitly expect
additional keys of certain value types.

However, the typing spec is more restrictive on type checking the
construction of a TypedDict, preventing users from doing this:

    class MovieBase(TypedDict):
        name: str

    def fun(movie: MovieBase) -> None:
        # movie can have extra items that are not visible through MovieBase
        ...

    movie: MovieBase = {"name": "Blade Runner", "year": 1982}  # Not OK
    fun({"name": "Blade Runner", "year": 1982})  # Not OK

While the restriction is enforced when constructing a TypedDict, due to
structural subtyping, the TypedDict may have extra items that are not
visible through its type. For example:

    class Movie(MovieBase):
        year: int

    movie: Movie = {"name": "Blade Runner", "year": 1982}
    fun(movie)  # OK

It is not possible to acknowledge the existence of the extra items
through in checks and access them without breaking type safety, even
though they might exist from arbitrary structural subtypes of MovieBase:

    def g(movie: MovieBase) -> None:
        if "year" in movie:
            reveal_type(movie["year"])  # Error: TypedDict 'MovieBase' has no key 'year'

Some workarounds have already been implemented in response to the need
to allow extra keys, but none of them is ideal. For mypy,
--disable-error-code=typeddict-unknown-key suppresses type checking
error specifically for unknown keys on TypedDict. This sacrifices type
safety over flexibility, and it does not offer a way to specify that the
TypedDict type expects additional keys compatible with a certain type.

Support Additional Keys for Unpack

PEP 692 adds a way to precisely annotate the types of individual keyword
arguments represented by **kwargs using TypedDict with Unpack. However,
because TypedDict cannot be defined to accept arbitrary extra items, it
is not possible to allow additional keyword arguments that are not known
at the time the TypedDict is defined.

Given the usage of pre-PEP 692 type annotation for **kwargs in existing
codebases, it will be valuable to accept and type extra items on
TypedDict so that the old typing behavior can be supported in
combination with the new Unpack construct.

Rationale

A type that allows extra items of type str on a TypedDict can be loosely
described as the intersection between the TypedDict and
Mapping[str, str].

Index Signatures in TypeScript achieve this:

    type Foo = {
        a: string
        [key: string]: string
    }

This proposal aims to support a similar feature without introducing
general intersection of types or syntax changes, offering a natural
extension to the existing type consistency rules.

We propose that we add an argument closed to TypedDict. Similar to
total, only a literal True or False value is allowed. When closed=True
is used in the TypedDict type definition, we give the dunder attribute
__extra_items__ a special meaning: extra items are allowed, and their
types should be compatible with the value type of __extra_items__.

If closed=True is set, but there is no __extra_items__ key, the
TypedDict is treated as if it contained an item __extra_items__: Never.

Note that __extra_items__ on the same TypedDict type definition will
remain as a regular item if closed=True is not used.

Different from index signatures, the types of the known items do not
need to be consistent with the value type of __extra_items__.

There are some advantages to this approach:

-   Inheritance works naturally. __extra_items__ defined on a TypedDict
    will also be available to its subclasses.
-   We can build on top of the type consistency rules defined in the
    typing spec. __extra_items__ can be treated as a pseudo-item in
    terms of type consistency.
-   There is no need to introduce a grammar change to specify the type
    of the extra items.
-   We can precisely type the extra items without making __extra_items__
    the union of known items.
-   We do not lose backwards compatibility as __extra_items__ still can
    be used as a regular key.

Specification

This specification is structured to parallel PEP 589 to highlight
changes to the original TypedDict specification.

If closed=True is specified, extra items are treated as non-required
items having the same type of __extra_items__ whose keys are allowed
when determining supported and unsupported operations.

Using TypedDict Types

Assuming that closed=True is used in the TypedDict type definition.

For a TypedDict type that has the special __extra_items__ key, during
construction, the value type of each unknown item is expected to be
non-required and compatible with the value type of __extra_items__. For
example:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: bool

    a: Movie = {"name": "Blade Runner", "novel_adaptation": True}  # OK
    b: Movie = {
        "name": "Blade Runner",
        "year": 1982,  # Not OK. 'int' is incompatible with 'bool'
    }

In this example, __extra_items__: bool does not mean that Movie has a
required string key "__extra_items__" whose value type is bool. Instead,
it specifies that keys other than "name" have a value type of bool and
are non-required.

The alternative inline syntax is also supported:

    Movie = TypedDict("Movie", {"name": str, "__extra_items__": bool}, closed=True)

Accessing extra keys is allowed. Type checkers must infer its value type
from the value type of __extra_items__:

    def f(movie: Movie) -> None:
        reveal_type(movie["name"])              # Revealed type is 'str'
        reveal_type(movie["novel_adaptation"])  # Revealed type is 'bool'

When a TypedDict type defines __extra_items__ without closed=True,
closed defaults to False and the key is assumed to be a regular key:

    class Movie(TypedDict):
        name: str
        __extra_items__: bool

    a: Movie = {"name": "Blade Runner", "novel_adaptation": True}  # Not OK. Unexpected key 'novel_adaptation'
    b: Movie = {
        "name": "Blade Runner",
        "__extra_items__": True,  # OK
    }

For such non-closed TypedDict types, it is assumed that they allow
non-required extra items of value type ReadOnly[object] during
inheritance or type consistency checks. However, extra keys found during
construction should still be rejected by the type checker.

closed is not inherited through subclassing:

    class MovieBase(TypedDict, closed=True):
        name: str
        __extra_items__: ReadOnly[str | None]

    class Movie(MovieBase):
        __extra_items__: str  # A regular key

    a: Movie = {"name": "Blade Runner", "__extra_items__": None}  # Not OK. 'None' is incompatible with 'str'
    b: Movie = {
        "name": "Blade Runner",
        "__extra_items__": "A required regular key",
        "other_extra_key": None,
    }  # OK

Here, "__extra_items__" in a is a regular key defined on Movie where its
value type is narrowed from ReadOnly[str | None] to str,
"other_extra_key" in b is an extra key whose value type must be
consistent with the value type of "__extra_items__" defined on
MovieBase.

Interaction with Totality

It is an error to use Required[] or NotRequired[] with the special
__extra_items__ item. total=False and total=True have no effect on
__extra_items__ itself.

The extra items are non-required, regardless of the totality of the
TypedDict. Operations that are available to NotRequired items should
also be available to the extra items:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: int

    def f(movie: Movie) -> None:
        del movie["name"]  # Not OK
        del movie["year"]  # OK

Interaction with Unpack

For type checking purposes, Unpack[TypedDict] with extra items should be
treated as its equivalent in regular parameters, and the existing rules
for function parameters still apply:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: int

    def f(**kwargs: Unpack[Movie]) -> None: ...

    # Should be equivalent to
    def f(*, name: str, **kwargs: int) -> None: ...

Interaction with PEP 705

When the special __extra_items__ item is annotated with ReadOnly[], the
extra items on the TypedDict have the properties of read-only items.
This interacts with inheritance rules specified in
PEP 705 <705#Inheritance>.

Notably, if the TypedDict type declares __extra_items__ to be read-only,
a subclass of the TypedDict type may redeclare __extra_items__'s value
type or additional non-extra items' value type.

Because a non-closed TypedDict type implicitly allows non-required extra
items of value type ReadOnly[object], its subclass can override the
special __extra_items__ with more specific types.

More details are discussed in the later sections.

Inheritance

When the TypedDict type is defined as closed=False (the default),
__extra_items__ should behave and be inherited the same way a regular
key would. A regular __extra_items__ key can coexist with the special
__extra_items__ and both should be inherited when subclassing.

We assume that closed=True whenever __extra_items__ is mentioned for the
rest of this section.

__extra_items__ is inherited the same way as a regular key: value_type
item. As with the other keys, the same rules from the typing spec and
PEP 705 <705#inheritance> apply. We interpret the existing rules in the
context of __extra_items__.

We need to reinterpret the following rule to define how __extra_items__
interacts with it:

  -   Changing a field type of a parent TypedDict class in a subclass is
      not allowed.

First, it is not allowed to change the value type of __extra_items__ in
a subclass unless it is declared to be ReadOnly in the superclass:

    class Parent(TypedDict, closed=True):
        __extra_items__: int | None

    class Child(Parent, closed=True):
        __extra_items__: int  # Not OK. Like any other TypedDict item, __extra_items__'s type cannot be changed

Second, __extra_items__: T effectively defines the value type of any
unnamed items accepted to the TypedDict and marks them as non-required.
Thus, the above restriction applies to any additional items defined in a
subclass. For each item added in a subclass, all of the following
conditions should apply:

-   If __extra_items__ is read-only
    -   The item can be either required or non-required
    -   The item's value type is consistent with T
-   If __extra_items__ is not read-only
    -   The item is non-required
    -   The item's value type is consistent with T
    -   T is consistent with the item's value type
-   If __extra_items__ is not redeclared, the subclass inherits it
    as-is.

For example:

    class MovieBase(TypedDict, closed=True):
        name: str
        __extra_items__: int | None

    class AdaptedMovie(MovieBase):  # Not OK. 'bool' is not consistent with 'int | None'
        adapted_from_novel: bool

    class MovieRequiredYear(MovieBase):  # Not OK. Required key 'year' is not known to 'Parent'
        year: int | None

    class MovieNotRequiredYear(MovieBase):  # Not OK. 'int | None' is not consistent with 'int'
        year: NotRequired[int]

    class MovieWithYear(MovieBase):  # OK
        year: NotRequired[int | None]

Due to this nature, an important side effect allows us to define a
TypedDict type that disallows additional items:

    class MovieFinal(TypedDict, closed=True):
        name: str
        __extra_items__: Never

Here, annotating __extra_items__ with typing.Never specifies that there
can be no other keys in MovieFinal other than the known ones. Because of
its potential common use, this is equivalent to:

    class MovieFinal(TypedDict, closed=True):
        name: str

where we implicitly assume the __extra_items__: Never field by default
if only closed=True is specified.

Type Consistency

In addition to the set S of keys of the explicitly defined items, a
TypedDict type that has the item __extra_items__: T is considered to
have an infinite set of items that all satisfy the following conditions:

-   If __extra_items__ is read-only
    -   The key's value type is consistent with T
    -   The key is not in S.
-   If __extra_items__ is not read-only
    -   The key is non-required
    -   The key's value type is consistent with T
    -   T is consistent with the key's value type
    -   The key is not in S.

For type checking purposes, let __extra_items__ be a non-required
pseudo-item to be included whenever "for each ... item/key" is stated in
the existing type consistency rules from PEP 705 <705#type-consistency>,
and we modify it as follows:

  A TypedDict type A is consistent with TypedDict B if A is structurally
  compatible with B. This is true if and only if all of the following
  are satisfied:

  -   For each item in B, A has the corresponding key, unless the item
      in B is read-only, not required, and of top value type
      (ReadOnly[NotRequired[object]]). [Edit: Otherwise, if the
      corresponding key with the same name cannot be found in ``A``,
      "__extra_items__" is considered the corresponding key.]
  -   For each item in B, if A has the corresponding key [Edit: or
      "__extra_items__"], the corresponding value type in A is
      consistent with the value type in B.
  -   For each non-read-only item in B, its value type is consistent
      with the corresponding value type in A. [Edit: if the
      corresponding key with the same name cannot be found in ``A``,
      "__extra_items__" is considered the corresponding key.]
  -   For each required key in B, the corresponding key is required in
      A. For each non-required key in B, if the item is not read-only in
      B, the corresponding key is not required in A. [Edit: if the
      corresponding key with the same name cannot be found in ``A``,
      "__extra_items__" is considered to be non-required as the
      corresponding key.]

The following examples illustrate these checks in action.

__extra_items__ puts various restrictions on additional items for type
consistency checks:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: int | None

    class MovieDetails(TypedDict, closed=True):
        name: str
        year: NotRequired[int]
        __extra_items__: int | None

    details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003}
    movie: Movie = details  # Not OK. While 'int' is consistent with 'int | None',
                            # 'int | None' is not consistent with 'int'

    class MovieWithYear(TypedDict, closed=True):
        name: str
        year: int | None
        __extra_items__: int | None

    details: MovieWithYear = {"name": "Kill Bill Vol. 1", "year": 2003}
    movie: Movie = details  # Not OK. 'year' is not required in 'Movie',
                            # so it shouldn't be required in 'MovieWithYear' either

Because "year" is absent in Movie, __extra_items__ is considered the
corresponding key. "year" being required violates the rule "For each
required key in B, the corresponding key is required in A".

When __extra_items__ is defined to be read-only in a TypedDict type, it
is possible for an item to have a narrower type than __extra_items__'s
value type:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: ReadOnly[str | int]

    class MovieDetails(TypedDict, closed=True):
        name: str
        year: NotRequired[int]
        __extra_items__: int

    details: MovieDetails = {"name": "Kill Bill Vol. 2", "year": 2004}
    movie: Movie = details  # OK. 'int' is consistent with 'str | int'.

This behaves the same way as PEP 705 specified if
year: ReadOnly[str | int] is an item defined in Movie.

__extra_items__ as a pseudo-item follows the same rules that other items
have, so when both TypedDicts contain __extra_items__, this check is
naturally enforced:

    class MovieExtraInt(TypedDict, closed=True):
        name: str
        __extra_items__: int

    class MovieExtraStr(TypedDict, closed=True):
        name: str
        __extra_items__: str

    extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
    extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""}
    extra_int = extra_str  # Not OK. 'str' is inconsistent with 'int' for item '__extra_items__'
    extra_str = extra_int  # Not OK. 'int' is inconsistent with 'str' for item '__extra_items__'

A non-closed TypedDict type implicitly allows non-required extra keys of
value type ReadOnly[object]. This allows to apply the type consistency
rules between this type and a closed TypedDict type:

    class MovieNotClosed(TypedDict):
        name: str

    extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
    not_closed: MovieNotClosed = {"name": "No Country for Old Men"}
    extra_int = not_closed  # Not OK. 'ReadOnly[object]' implicitly on 'MovieNotClosed' is not consistent with 'int' for item '__extra_items__'
    not_closed = extra_int  # OK

Interaction with Constructors

TypedDicts that allow extra items of type T also allow arbitrary keyword
arguments of this type when constructed by calling the class object:

    class OpenMovie(TypedDict):
        name: str

    OpenMovie(name="No Country for Old Men")  # OK
    OpenMovie(name="No Country for Old Men", year=2007)  # Not OK. Unrecognized key

    class ExtraMovie(TypedDict, closed=True):
        name: str
        __extra_items__: int

    ExtraMovie(name="No Country for Old Men")  # OK
    ExtraMovie(name="No Country for Old Men", year=2007)  # OK
    ExtraMovie(
        name="No Country for Old Men",
        language="English",
    )  # Not OK. Wrong type for extra key

    # This implies '__extra_items__: Never',
    # so extra keyword arguments produce an error
    class ClosedMovie(TypedDict, closed=True):
        name: str

    ClosedMovie(name="No Country for Old Men")  # OK
    ClosedMovie(
        name="No Country for Old Men",
        year=2007,
    )  # Not OK. Extra items not allowed

Interaction with Mapping[KT, VT]

A TypedDict type can be consistent with Mapping[KT, VT] types other than
Mapping[str, object] as long as the union of value types on the
TypedDict type is consistent with VT. It is an extension of this rule
from the typing spec:

  -   A TypedDict with all int values is not consistent with
      Mapping[str, int], since there may be additional non-int values
      not visible through the type, due to structural subtyping. These
      can be accessed using the values() and items() methods in Mapping

For example:

    class MovieExtraStr(TypedDict, closed=True):
        name: str
        __extra_items__: str

    extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""}
    str_mapping: Mapping[str, str] = extra_str  # OK

    int_mapping: Mapping[str, int] = extra_int  # Not OK. 'int | str' is not consistent with 'int'
    int_str_mapping: Mapping[str, int | str] = extra_int  # OK

Furthermore, type checkers should be able to infer the precise return
types of values() and items() on such TypedDict types:

    def fun(movie: MovieExtraStr) -> None:
        reveal_type(movie.items())  # Revealed type is 'dict_items[str, str]'
        reveal_type(movie.values())  # Revealed type is 'dict_values[str, str]'

Interaction with dict[KT, VT]

Note that because the presence of __extra_items__ on a closed TypedDict
type prohibits additional required keys in its structural subtypes, we
can determine if the TypedDict type and its structural subtypes will
ever have any required key during static analysis.

The TypedDict type is consistent with dict[str, VT] if all items on the
TypedDict type satisfy the following conditions:

-   VT is consistent with the value type of the item
-   The value type of the item is consistent with VT
-   The item is not read-only.
-   The item is not required.

For example:

    class IntDict(TypedDict, closed=True):
        __extra_items__: int

    class IntDictWithNum(IntDict):
        num: NotRequired[int]

    def f(x: IntDict) -> None:
        v: dict[str, int] = x  # OK
        v.clear()  # OK

    not_required_num: IntDictWithNum = {"num": 1, "bar": 2}
    regular_dict: dict[str, int] = not_required_num  # OK
    f(not_required_num)  # OK

In this case, methods that are previously unavailable on a TypedDict are
allowed:

    not_required_num.clear()  # OK

    reveal_type(not_required_num.popitem())  # OK. Revealed type is tuple[str, int]

However, dict[str, VT] is not necessarily consistent with a TypedDict
type, because such dict can be a subtype of dict:

    class CustomDict(dict[str, int]):
        ...

    not_a_regular_dict: CustomDict = {"num": 1}
    int_dict: IntDict = not_a_regular_dict  # Not OK

How to Teach This

The choice of the spelling "__extra_items__" is intended to make this
feature more understandable to new users compared to shorter
alternatives like "__extra__".

Details of this should be documented in both the typing spec and the
typing documentation.

Backwards Compatibility

Because __extra_items__ remains as a regular key if closed=True is not
specified, no existing codebase will break due to this change.

If the proposal is accepted, none of __required_keys__,
__optional_keys__, __readonly_keys__ and __mutable_keys__ should include
"__extra_items__" defined on the same TypedDict type when closed=True is
specified.

Note that closed as a keyword argument does not collide with the keyword
arguments alternative to define keys with the functional syntax that
allows things like TD = TypedDict("TD", foo=str, bar=int), because it is
scheduled to be removed in Python 3.13.

Because this is a type-checking feature, it can be made available to
older versions as long as the type checker supports it.

Rejected Ideas

Allowing Extra Items without Specifying the Type

extra=True was originally proposed for defining a TypedDict that accepts
extra items regardless of the type, like how total=True works:

    class TypedDict(extra=True):
        pass

Because it did not offer a way to specify the type of the extra items,
the type checkers will need to assume that the type of the extra items
is Any, which compromises type safety. Furthermore, the current behavior
of TypedDict already allows untyped extra items to be present in
runtime, due to structural subtyping. closed=True plays a similar role
in the current proposal.

Supporting TypedDict(extra=type)

During the discussion of the PEP, there were strong objections against
adding another place where types are passed as values instead of
annotations from some authors of type checkers. While this design is
potentially viable, there are also several partially addressable
concerns to consider.

-   Usability of forward reference As in the functional syntax, using a
    quoted type or a type alias will be required when SomeType is a
    forward reference. This is already a requirement for the functional
    syntax, so implementations can potentially reuse that piece of
    logic, but this is still extra work that the closed=True proposal
    doesn't have.
-   Concerns about using type as a value Whatever is not allowed as the
    value type in the functional syntax should not be allowed as the
    argument for extra either. While type checkers might be able to
    reuse this check, it still needs to be somehow special-cased for the
    class-based syntax.
-   How to teach Notably, the extra=type often gets brought up due to it
    being an intuitive solution for the use case, so it is potentially
    simpler to learn than the less obvious solution. However, the more
    common used case only requires closed=True, and the other drawbacks
    mentioned earlier outweigh what is need to teach the usage of the
    special key.

Support Extra Items with Intersection

Supporting intersections in Python's type system requires a lot of
careful consideration, and it can take a long time for the community to
reach a consensus on a reasonable design.

Ideally, extra items in TypedDict should not be blocked by work on
intersections, nor does it necessarily need to be supported through
intersections.

Moreover, the intersection between Mapping[...] and TypedDict is not
equivalent to a TypedDict type with the proposed __extra_items__ special
item, as the value type of all known items in TypedDict needs to satisfy
the is-subtype-of relation with the value type of Mapping[...].

Requiring Type Compatibility of the Known Items with __extra_items__

__extra_items__ restricts the value type for keys that are unknown to
the TypedDict type. So the value type of any known item is not
necessarily consistent with __extra_items__'s type, and
__extra_items__'s type is not necessarily consistent with the value
types of all known items.

This differs from TypeScript's Index Signatures syntax, which requires
all properties' types to match the string index's type. For example:

    interface MovieWithExtraNumber {
        name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'.
        [index: string]: number
    }

    interface MovieWithExtraNumberOrString {
        name: string // OK
        [index: string]: number | string
    }

This is a known limitation discussed in TypeScript's issue tracker,
where it is suggested that there should be a way to exclude the defined
keys from the index signature so that it is possible to define a type
like MovieWithExtraNumber.

Reference Implementation

This proposal is supported in pyright 1.1.352, and pyanalyze 0.12.0.

Acknowledgments

Thanks to Jelle Zijlstra for sponsoring this PEP and providing review
feedback, Eric Traut who proposed the original design this PEP iterates
on, and Alice Purcell for offering their perspective as the author of
PEP 705.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 545 Title: Python Documentation Translations Author: Julien Palard
<EMAIL_ADDRESS>Inada Naoki<EMAIL_ADDRESS>Victor Stinner
<EMAIL_ADDRESS>Status: Final Type: Process Content-Type:
text/x-rst Created: 04-Mar-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-May/147957.html

Abstract

The intent of this PEP is to make existing translations of the Python
Documentation more accessible and discoverable. By doing so, we hope to
attract and motivate new translators and new translations.

Translated documentation will be hosted on python.org. Examples of two
active translation teams:

-   http://docs.python.org/fr/: French
-   http://docs.python.org/ja/: Japanese

http://docs.python.org/en/ will redirect to http://docs.python.org/.

Sources of translated documentation will be hosted in the Python
organization on GitHub: https://github.com/python/. Contributors will
have to accept a Documentation Contribution Agreement.

Motivation

On the French #python-fr IRC channel on freenode, it's not rare to meet
people who don't speak English and so are unable to read the Python
official documentation. Python wants to be widely available to all users
in any language: this is also why Python 3 supports any non-ASCII
identifiers: 3131#rationale

There are at least 4 groups of people who are translating the Python
documentation to their native language (French[1][2][3], Japanese[4][5],
Spanish[6], Hungarian[7][8]) even though their translations are not
visible on d.p.o. Other, less visible and less organized groups, are
also translating the documentation, we've heard of Russian[9], Chinese
and Korean. Others we haven't found yet might also exist. This PEP
defines rules describing how to move translations on docs.python.org so
they can easily be found by developers, newcomers and potential
translators.

The Japanese team has (as of March 2017) translated ~80% of the
documentation, the French team ~20%. French translation went from 6% to
23% in 2016[10] with 7 contributors[11], proving a translation team can
be faster than the rate the documentation mutates.

Quoting Xiang Zhang about Chinese translations:

  I have seen several groups trying to translate part of our official
  doc. But their efforts are disperse and quickly become lost because
  they are not organized to work towards a single common result and
  their results are hold anywhere on the Web and hard to find. An
  official one could help ease the pain.

Rationale

Translation

Issue tracker

Considering that issues opened about translations may be written in the
translation language, which can be considered noise but at least is
inconsistent, issues should be placed outside bugs.python.org (b.p.o).

As all translation must have their own GitHub project (see Repository
for Po Files), they must use the associated GitHub issue tracker.

Considering the noise induced by translation issues redacted in any
languages which may beyond every warnings land in b.p.o, triage will
have to be done. Considering that translations already exist and are not
actually a source of noise in b.p.o, an unmanageable amount of work is
not to be expected. Considering that Xiang Zhang and Victor Stinner are
already triaging, and Julien Palard is willing to help on this task,
noise on b.p.o is not to be expected.

Also, language team coordinators (see Language Team) should help with
triaging b.p.o by properly indicating, in the language of the issue
author if required, the right issue tracker.

Branches

Translation teams should focus on last stable versions, and use tools
(scripts, translation memory, …) to automatically translate what is done
in one branch to other branches.

Note

Translation memories are a kind of database of previously translated
paragraphs, even removed ones. See also Sphinx Internationalization.

The three currently stable branches that will be translated are[12]:
2.7, 3.5, and 3.6. The scripts to build the documentation of older
branches needs to be modified to support translation[13], whereas these
branches now only accept security-only fixes.

The development branch (main) should have a lower translation priority
than stable branches. But docsbuild-scripts should build it anyway so it
is possible for a team to work on it to be ready for the next release.

Hosting

Domain Name, Content negotiation and URL

Different translations can be identified by changing one of the
following: Country Code Top Level Domain (CCTLD), path segment,
subdomain or content negotiation.

Buying a CCTLD for each translations is expensive, time-consuming, and
sometimes almost impossible when already registered, this solution
should be avoided.

Using subdomains like "es.docs.python.org" or "docs.es.python.org" is
possible but confusing ("is it es.docs.python.org or
docs.es.python.org?"). Hyphens in subdomains like pt-br.doc.python.org
is uncommon and SEOMoz[14] correlated the presence of hyphens as a
negative factor. Usage of underscores in subdomain is prohibited by the
1123, section 2.1. Finally, using subdomains means creating TLS
certificates for each language. This not only requires more maintenance
but will also cause issues in language switcher if, as for version
switcher, we want a preflight to check if the translation exists in the
given version: preflight will probably be blocked by same-origin-policy.
Wildcard TLS certificates are very expensive.

Using content negotiation (HTTP headers Accept-Language in the request
and Vary: Accept-Language) leads to a bad user experience where they
can't easily change the language. According to Mozilla: "This header is
a hint to be used when the server has no way of determining the language
via another way, like a specific URL, that is controlled by an explicit
user decision."[15]. As we want to be able to easily change the
language, we should not use the content negotiation as a main language
determination, so we need something else.

Last solution is to use the URL path, which looks readable, allows for
an easy switch from a language to another, and nicely accepts hyphens.
Typically something like: "docs.python.org/de/" or, by using a hyphen:
"docs.python.org/pt-BR/".

As for the version, sphinx-doc does not support compiling for multiple
languages, so we'll have full builds rooted under a path, exactly like
we're already doing with versions.

So we can have "docs.python.org/de/3.6/" or "docs.python.org/3.6/de/". A
question that arises is: "Does the language contain multiple versions or
does the version contain multiple languages?". As versions exist in any
case and translations for a given version may or may not exist, we may
prefer "docs.python.org/3.6/de/", but doing so scatters languages
everywhere. Having "/de/3.6/" is clearer, meaning: "everything under
/de/ is written in German". Having the version at the end is also a
habit taken by readers of the documentation: they like to easily change
the version by changing the end of the path.

So we should use the following pattern:
"docs.python.org/LANGUAGE_TAG/VERSION/".

The current documentation is not moved to "/en/", instead
"docs.python.org/en/" will redirect to "docs.python.org".

Language Tag

A common notation for language tags is the IETF Language Tag <5646> [16]
based on ISO 639, although gettext uses ISO 639 tags with underscores
(ex: pt_BR) instead of dashes to join tags[17] (ex: pt-BR). Examples of
IETF Language Tags: fr (French), ja (Japanese), pt-BR (Orthographic
formulation of 1943 -Official in Brazil).

It is more common to see dashes instead of underscores in URLs[18], so
we should use IETF language tags, even if sphinx uses gettext
internally: URLs are not meant to leak the underlying implementation.

It's uncommon to see capitalized letters in URLs, and docs.python.org
doesn't use any, so it may hurt readability by attracting the eye on it,
like in: "https://docs.python.org/pt-BR/3.6/library/stdtypes.html".
5646#section-2.1.1 (Tags for Identifying Languages (IETF)) section-2.1
states that tags are not case sensitive. As the RFC allows lower case,
and it enhances readability, we should use lowercased tags like pt-br.

We may drop the region subtag when it does not add distinguishing
information, for example: "de-DE" or "fr-FR". (Although it might make
sense, respectively meaning "German as spoken in Germany" and "French as
spoken in France"). But when the region subtag actually adds
information, for example "pt-BR" or "Portuguese as spoken in Brazil", it
should be kept.

So we should use IETF language tags, lowercased, like /fr/, /pt-br/,
/de/ and so on.

Fetching And Building Translations

Currently docsbuild-scripts are building the documentation[19]. These
scripts should be modified to fetch and build translations.

Building new translations is like building new versions so, while we're
adding complexity it is not that much.

Two steps should be configurable distinctively: Building a new language,
and adding it to the language switcher. This allows a transition step
between "we accepted the language" and "it is translated enough to be
made public". During this step, translators can review their
modifications on d.p.o without having to build the documentation
locally.

From the translation repositories, only the .po files should be opened
by the docsbuild-script to keep the attack surface and probable bug
sources at a minimum. This means no translation can patch sphinx to
advertise their translation tool. (This specific feature should be
handled by sphinx anyway[20]).

Community

Mailing List

The doc-sig mailing list will be used to discuss cross-language changes
on translated documentation.

There is also the i18n-sig list but it's more oriented towards i18n APIs
[21] than translating the Python documentation.

Chat

Due to the Python community being highly active on IRC, we should create
a new IRC channel on freenode, typically #python-doc for consistency
with the mailing list name.

Each language coordinator can organize their own team, even by choosing
another chat system if the local usage asks for it. As local teams will
write in their native languages, we don't want each team in a single
channel. It's also natural for the local teams to reuse their local
channels like "#python-fr" for French translators.

Repository for PO Files

Considering that each translation team may want to use different
translation tools, and that those tools should easily be synchronized
with git, all translations should expose their .po files via a git
repository.

Considering that each translation will be exposed via git repositories,
and that Python has migrated to GitHub, translations will be hosted on
GitHub.

For consistency and discoverability, all translations should be in the
same GitHub organization and named according to a common pattern.

Given that we want translations to be official, and that Python already
has a GitHub organization, translations should be hosted as projects of
the Python GitHub organization.

For consistency, translation repositories should be called
python-docs-LANGUAGE_TAG[22], using the language tag used in paths:
without region subtag if redundant, and lowercased.

The docsbuild-scripts may enforce this rule by refusing to fetch outside
of the Python organization or a wrongly named repository.

The CLA bot may be used on the translation repositories, but with a
limited effect as local coordinators may synchronize themselves with
translations from an external tool, like transifex, and lose track of
who translated what in the process.

Versions can be hosted on different repositories, different directories
or different branches. Storing them on different repositories will
probably pollute the Python GitHub organization. As it is typical and
natural to use branches to separate versions, branches should be used to
do so.

Translation tools

Most of the translation work is actually done on Transifex[23].

Other tools may be used later like https://pontoon.mozilla.org/ and
http://zanata.org/.

Documentation Contribution Agreement

Documentation does require a license from the translator, as it involves
creativity in the expression of the ideas.

There's multiple solutions, quoting Van Lindberg from the PSF asked
about the subject:

  1.  Docs should either have the copyright assigned or be under CCO. A
      permissive software license (like Apache or MIT) would also get
      the job done, although it is not quite fit for task.
  2.  The translators should either sign an agreement or submit a
      declaration of the license with the translation.
  3.  We should have in the project page an invitation for people to
      contribute under a defined license, with acceptance defined by
      their act of contribution. Such as:

  "By posting this project on Transifex and inviting you to participate,
  we are proposing an agreement that you will provide your translation
  for the PSF's use under the CC0 license. In return, you may noted that
  you were the translator for the portion you translate. You signify
  acceptance of this agreement by submitting your work to the PSF for
  inclusion in the documentation."

It looks like having a "Documentation Contribution Agreement" is the
most simple thing we can do as we can use multiple ways (GitHub bots,
invitation page, …) in different context to ensure contributors are
agreeing with it.

Language Team

Each language team should have one coordinator responsible for:

-   Managing the team.
-   Choosing and managing the tools the team will use (chat, mailing
    list, …).
-   Ensure contributors understand and agree with the documentation
    contribution agreement.
-   Ensure quality (grammar, vocabulary, consistency, filtering spam,
    ads, …).
-   Redirect issues posted on b.p.o to the correct GitHub issue tracker
    for the language.

Alternatives

Simplified English

It would be possible to introduce a "simplified English" version like
Wikipedia did[24], as discussed on python-dev[25], targeting English
learners and children.

Pros: It yields a single translation, theoretically readable by everyone
and reviewable by current maintainers.

Cons: Subtle details may be lost, and translators from English to
English may be hard to find as stated by Wikipedia:

> The main English Wikipedia has 5 million articles, written by nearly
140K active users; the Swedish Wikipedia is almost as big, 3M articles
from only 3K active users; but the Simple English Wikipedia has just
123K articles and 871 active users. That's fewer articles than
Esperanto!

Changes

Get a Documentation Contribution Agreement

The Documentation Contribution Agreement have to be written by the PSF,
then listed at https://www.python.org/psf/contrib/ and have its own page
like https://www.python.org/psf/contrib/doc-contrib-form/.

Migrate GitHub Repositories

We (authors of this PEP) already own French and Japanese Git
repositories, so moving them to the Python documentation organization
will not be a problem. We'll however be following the New Translation
Procedure.

Setup a GitHub bot for Documentation Contribution Agreement

To help ensuring contributors from GitHub have signed the Documentation
Contribution Agreement, We can setup the "The Knights Who Say Ni" GitHub
bot customized for this agreement on the migrated repositories[26].

Patch docsbuild-scripts to Compile Translations

Docsbuild-script must be patched to:

-   List the language tags to build along with the branches to build.
-   List the language tags to display in the language switcher.
-   Find translation repositories by formatting
    github.com:python/python-docs-{language_tag}.git (See Repository for
    Po Files)
-   Build translations for each branch and each language.

Patched docsbuild-scripts must only open .po files from translation
repositories.

List coordinators in the devguide

Add a page or a section with an empty list of coordinators to the
devguide, each new coordinator will be added to this list.

Create sphinx-doc Language Switcher

Highly similar to the version switcher, a language switcher must be
implemented. This language switcher must be configurable to hide or show
a given language.

The language switcher will only have to update or add the language
segment to the path like the current version switcher does. Unlike the
version switcher, no preflight are required as destination page always
exists (translations does not add or remove pages). Untranslated (but
existing) pages still exists, they should however be rendered as so, see
Enhance Rendering of Untranslated and Fuzzy Translations.

Update sphinx-doc Version Switcher

The patch_url function of the version switcher in version_switch.js have
to be updated to understand and allow the presence of the language
segment in the path.

Enhance Rendering of Untranslated and Fuzzy Translations

It's an opened sphinx issue[27], but we'll need it so we'll have to work
on it. Translated, fuzzy, and untranslated paragraphs should be
differentiated. (Fuzzy paragraphs have to warn the reader what he's
reading may be out of date.)

New Translation Procedure

Designate a Coordinator

The first step is to designate a coordinator, see Language Team, The
coordinator must sign the CLA.

The coordinator should be added to the list of translation coordinators
on the devguide.

Create GitHub Repository

Create a repository named "python-docs-{LANGUAGE_TAG}" (IETF language
tag, without redundant region subtag, with a dash, and lowercased.) on
the Python GitHub organization (See Repository For Po Files.), and grant
the language coordinator push rights to this repository.

Setup the Documentation Contribution Agreement

The README file should clearly show the following Documentation
Contribution Agreement:

    NOTE REGARDING THE LICENSE FOR TRANSLATIONS: Python's documentation is
    maintained using a global network of volunteers. By posting this
    project on Transifex, GitHub, and other public places, and inviting
    you to participate, we are proposing an agreement that you will
    provide your improvements to Python's documentation or the translation
    of Python's documentation for the PSF's use under the CC0 license
    (available at
    `https://creativecommons.org/publicdomain/zero/1.0/legalcode`_). In
    return, you may publicly claim credit for the portion of the
    translation you contributed and if your translation is accepted by the
    PSF, you may (but are not required to) submit a patch including an
    appropriate annotation in the Misc/ACKS or TRANSLATORS file. Although
    nothing in this Documentation Contribution Agreement obligates the PSF
    to incorporate your textual contribution, your participation in the
    Python community is welcomed and appreciated.

    You signify acceptance of this agreement by submitting your work to
    the PSF for inclusion in the documentation.

Add support for translations in docsbuild-scripts

As soon as the translation hits its first commits, update the
docsbuild-scripts configuration to build the translation (but not
displaying it in the language switcher).

Add Translation to the Language Switcher

As soon as the translation hits:

-   100% of bugs.html with proper links to the language repository issue
    tracker.
-   100% of tutorial.
-   100% of library/functions (builtins).

the translation can be added to the language switcher.

Previous Discussions

[Python-ideas] Cross link documentation translations (January, 2016)

[Python-Dev] Translated Python documentation (February 2016)

[Python-ideas] https://docs.python.org/fr/ ? (March 2016)

References

[2] [Doc-SIG] Localization of Python docs
(https://mail.python.org/pipermail/doc-sig/2013-September/003948.html)

Copyright

This document has been placed in the public domain.

[1] French translation (https://www.afpy.org/doc/python/)

[2] French translation on Gitea
(https://git.afpy.org/AFPy/python-docs-fr)

[3] French mailing list
(http://lists.afpy.org/mailman/listinfo/traductions)

[4] Japanese translation (http://docs.python.jp/3/)

[5] Japanese translation on GitHub
(https://github.com/python-doc-ja/python-doc-ja)

[6] Spanish translation
(https://docs.python.org/es/3/tutorial/index.html)

[7] Python-oktató
(http://web.archive.org/web/20170526080729/http://harp.pythonanywhere.com/python_doc/tutorial/index.html)

[8] The Python-hu Archives
(https://mail.python.org/pipermail/python-hu/)

[9] Документация Python 2.7!
(http://python-lab.ru/documentation/index.html)

[10] French translation progression (https://mdk.fr/pycon2016/#/11)

[11] French translation contributors
(https://github.com/AFPy/python_doc_fr/graphs/contributors?from=2016-01-01&to=2016-12-31&type=c)

[12] Passing options to sphinx from Doc/Makefile
(https://github.com/python/cpython/commit/57acb82d275ace9d9d854b156611e641f68e9e7c)

[13] Passing options to sphinx from Doc/Makefile
(https://github.com/python/cpython/commit/57acb82d275ace9d9d854b156611e641f68e9e7c)

[14] Domains - SEO Best Practices | Moz
(https://moz.com/learn/seo/domain)

[15] Accept-Language
(https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language)

[16] IETF language tag (https://en.wikipedia.org/wiki/IETF_language_tag)

[17] GNU Gettext manual, section 2.3.1: Locale Names
(https://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html)

[18] Semantic URL: Slug (https://en.wikipedia.org/wiki/Clean_URL#Slug)

[19] Docsbuild-scripts GitHub repository
(https://github.com/python/docsbuild-scripts/)

[20] i18n: Highlight untranslated paragraphs
(https://github.com/sphinx-doc/sphinx/issues/1246)

[21] [I18n-sig] Hello Python members, Do you have any idea about Python
documents?
(https://mail.python.org/pipermail/i18n-sig/2013-September/002130.html)

[22] [Python-Dev] Translated Python documentation: doc vs docs
(https://mail.python.org/pipermail/python-dev/2017-February/147472.html)

[23] Python-doc on Transifex
(https://www.transifex.com/python-doc/public/)

[24] Wikipedia: Simple English
(https://simple.wikipedia.org/wiki/Main_Page)

[25] Python-dev discussion about simplified English
(https://mail.python.org/pipermail/python-dev/2017-February/147446.html)

[26] [Python-Dev] PEP 545: Python Documentation Translations
(https://mail.python.org/pipermail/python-dev/2017-April/147752.html)

[27] i18n: Highlight untranslated paragraphs
(https://github.com/sphinx-doc/sphinx/issues/1246)

license: Public Domain


text:
PEP: 520 Title: Preserving Class Attribute Definition Order Version:
$Revision$ Last-Modified: $Date$ Author: Eric Snow
<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 07-Jun-2016 Python-Version: 3.6
Post-History: 07-Jun-2016, 11-Jun-2016, 20-Jun-2016, 24-Jun-2016
Resolution:
https://mail.python.org/pipermail/python-dev/2016-June/145442.html

Note

Since compact dict has landed in 3.6, __definition_order__ has been
removed. cls.__dict__ now mostly accomplishes the same thing instead.

Abstract

The class definition syntax is ordered by its very nature. Class
attributes defined there are thus ordered. Aside from helping with
readability, that ordering is sometimes significant. If it were
automatically available outside the class definition then the attribute
order could be used without the need for extra boilerplate (such as
metaclasses or manually enumerating the attribute order). Given that
this information already exists, access to the definition order of
attributes is a reasonable expectation. However, currently Python does
not preserve the attribute order from the class definition.

This PEP changes that by preserving the order in which attributes are
introduced in the class definition body. That order will now be
preserved in the __definition_order__ attribute of the class. This
allows introspection of the original definition order, e.g. by class
decorators.

Additionally, this PEP requires that the default class definition
namespace be ordered (e.g. OrderedDict) by default. The long-lived class
namespace (__dict__) will remain a dict.

Motivation

The attribute order from a class definition may be useful to tools that
rely on name order. However, without the automatic availability of the
definition order, those tools must impose extra requirements on users.
For example, use of such a tool may require that your class use a
particular metaclass. Such requirements are often enough to discourage
use of the tool.

Some tools that could make use of this PEP include:

-   documentation generators
-   testing frameworks
-   CLI frameworks
-   web frameworks
-   config generators
-   data serializers
-   enum factories (my original motivation)

Background

When a class is defined using a class statement, the class body is
executed within a namespace. Currently that namespace defaults to dict.
If the metaclass defines __prepare__() then the result of calling it is
used for the class definition namespace.

After the execution completes, the definition namespace is copied into a
new dict. Then the original definition namespace is discarded. The new
copy is stored away as the class's namespace and is exposed as __dict__
through a read-only proxy.

The class attribute definition order is represented by the insertion
order of names in the definition namespace. Thus, we can have access to
the definition order by switching the definition namespace to an ordered
mapping, such as collections.OrderedDict. This is feasible using a
metaclass and __prepare__, as described above. In fact, exactly this is
by far the most common use case for using __prepare__.

At that point, the only missing thing for later access to the definition
order is storing it on the class before the definition namespace is
thrown away. Again, this may be done using a metaclass. However, this
means that the definition order is preserved only for classes that use
such a metaclass. There are two practical problems with that:

First, it requires the use of a metaclass. Metaclasses introduce an
extra level of complexity to code and in some cases (e.g. conflicts) are
a problem. So reducing the need for them is worth doing when the
opportunity presents itself. PEP 422 and PEP 487 discuss this at length.
We have such an opportunity by using an ordered mapping (e.g.
OrderedDict for CPython at least) for the default class definition
namespace, virtually eliminating the need for __prepare__().

Second, only classes that opt in to using the OrderedDict-based
metaclass will have access to the definition order. This is problematic
for cases where universal access to the definition order is important.

Specification

Part 1:

-   all classes have a __definition_order__ attribute
-   __definition_order__ is a tuple of identifiers (or None)
-   __definition_order__ is always set:
    1.  during execution of the class body, the insertion order of names
        into the class definition namespace is stored in a tuple
    2.  if __definition_order__ is defined in the class body then it
        must be a tuple of identifiers or None; any other value will
        result in TypeError
    3.  classes that do not have a class definition (e.g. builtins) have
        their __definition_order__ set to None
    4.  classes for which __prepare__() returned something other than
        OrderedDict (or a subclass) have their __definition_order__ set
        to None (except where #2 applies)

Not changing:

-   dir() will not depend on __definition_order__
-   descriptors and custom __getattribute__ methods are unconstrained
    regarding __definition_order__

Part 2:

-   the default class definition namespace is now an ordered mapping
    (e.g. OrderdDict)
-   cls.__dict__ does not change, remaining a read-only proxy around
    dict

Note that Python implementations which have an ordered dict won't need
to change anything.

The following code demonstrates roughly equivalent semantics for both
parts 1 and 2:

    class Meta(type):
        @classmethod
        def __prepare__(cls, *args, **kwargs):
            return OrderedDict()

    class Spam(metaclass=Meta):
        ham = None
        eggs = 5
        __definition_order__ = tuple(locals())

Why a tuple?

Use of a tuple reflects the fact that we are exposing the order in which
attributes on the class were defined. Since the definition is already
complete by the time __definition_order__ is set, the content and order
of the value won't be changing. Thus we use a type that communicates
that state of immutability.

Why not a read-only attribute?

There are some valid arguments for making __definition_order__ a
read-only attribute (like cls.__dict__ is). Most notably, a read-only
attribute conveys the nature of the attribute as "complete", which is
exactly correct for __definition_order__. Since it represents the state
of a particular one-time event (execution of the class definition body),
allowing the value to be replaced would reduce confidence that the
attribute corresponds to the original class body. Furthermore, often an
immutable-by-default approach helps to make data easier to reason about.

However, in this case there still isn't a strong reason to counter the
well-worn precedent found in Python. Per Guido:

    I don't see why it needs to be a read-only attribute. There are
    very few of those -- in general we let users play around with
    things unless we have a hard reason to restrict assignment (e.g.
    the interpreter's internal state could be compromised). I don't
    see such a hard reason here.

Also, note that a writeable __definition_order__ allows dynamically
created classes (e.g. by Cython) to still have __definition_order__
properly set. That could certainly be handled through specific
class-creation tools, such as type() or the C-API, without the need to
lose the semantics of a read-only attribute. However, with a writeable
attribute it's a moot point.

Why not "__attribute_order__"?

__definition_order__ is centered on the class definition body. The use
cases for dealing with the class namespace (__dict__) post-definition
are a separate matter. __definition_order__ would be a significantly
misleading name for a feature focused on more than class definition.

Why not ignore "dunder" names?

Names starting and ending with "__" are reserved for use by the
interpreter. In practice they should not be relevant to the users of
__definition_order__. Instead, for nearly everyone they would only be
clutter, causing the same extra work (filtering out the dunder names)
for the majority. In cases where a dunder name is significant, the class
definition could manually set __definition_order__, making the common
case simpler.

However, leaving dunder names out of __definition_order__ means that
their place in the definition order would be unrecoverably lost.
Dropping dunder names by default may inadvertently cause problems for
classes that use dunder names unconventionally. In this case it's better
to play it safe and preserve all the names from the class definition.
This isn't a big problem since it is easy to filter out dunder names:

    (name for name in cls.__definition_order__
          if not (name.startswith('__') and name.endswith('__')))

In fact, in some application contexts there may be other criteria on
which similar filtering would be applied, such as ignoring any name
starting with "_", leaving out all methods, or including only
descriptors. Ultimately dunder names aren't a special enough case to be
treated exceptionally.

Note that a couple of dunder names (__name__ and __qualname__) are
injected by default by the compiler. So they will be included even
though they are not strictly part of the class definition body.

Why None instead of an empty tuple?

A key objective of adding __definition_order__ is to preserve
information in class definitions which was lost prior to this PEP. One
consequence is that __definition_order__ implies an original class
definition. Using None allows us to clearly distinguish classes that do
not have a definition order. An empty tuple clearly indicates a class
that came from a definition statement but did not define any attributes
there.

Why None instead of not setting the attribute?

The absence of an attribute requires more complex handling than None
does for consumers of __definition_order__.

Why constrain manually set values?

If __definition_order__ is manually set in the class body then it will
be used. We require it to be a tuple of identifiers (or None) so that
consumers of __definition_order__ may have a consistent expectation for
the value. That helps maximize the feature's usefulness.

We could also allow an arbitrary iterable for a manually set
__definition_order__ and convert it into a tuple. However, not all
iterables infer a definition order (e.g. set). So we opt in favor of
requiring a tuple.

Why not hide __definition_order__ on non-type objects?

Python doesn't make much effort to hide class-specific attributes during
lookup on instances of classes. While it may make sense to consider
__definition_order__ a class-only attribute, hidden during lookup on
objects, setting precedent in that regard is beyond the goals of this
PEP.

What about __slots__?

__slots__ will be added to __definition_order__ like any other name in
the class definition body. The actual slot names will not be added to
__definition_order__ since they aren't set as names in the definition
namespace.

Why is __definition_order__ even necessary?

Since the definition order is not preserved in __dict__, it is lost once
class definition execution completes. Classes could explicitly set the
attribute as the last thing in the body. However, then independent
decorators could only make use of classes that had done so. Instead,
__definition_order__ preserves this one bit of info from the class body
so that it is universally available.

Support for C-API Types

Arguably, most C-defined Python types (e.g. built-in, extension modules)
have a roughly equivalent concept of a definition order. So conceivably
__definition_order__ could be set for such types automatically. This PEP
does not introduce any such support. However, it does not prohibit it
either. However, since __definition_order__ can be set at any time
through normal attribute assignment, it does not need any special
treatment in the C-API.

The specific cases:

-   builtin types
-   PyType_Ready
-   PyType_FromSpec

Compatibility

This PEP does not break backward compatibility, except in the case that
someone relies strictly on dict as the class definition namespace. This
shouldn't be a problem since issubclass(OrderedDict, dict) is true.

Changes

In addition to the class syntax, the following expose the new behavior:

-   builtins.__build_class__
-   types.prepare_class
-   types.new_class

Also, the 3-argument form of builtins.type() will allow inclusion of
__definition_order__ in the namespace that gets passed in. It will be
subject to the same constraints as when __definition_order__ is
explicitly defined in the class body.

Other Python Implementations

Pending feedback, the impact on Python implementations is expected to be
minimal. All conforming implementations are expected to set
__definition_order__ as described in this PEP.

Implementation

The implementation is found in the tracker.

Alternatives

An Order-preserving cls.__dict__

Instead of storing the definition order in __definition_order__, the
now-ordered definition namespace could be copied into a new OrderedDict.
This would then be used as the mapping proxied as __dict__. Doing so
would mostly provide the same semantics.

However, using OrderedDict for __dict__ would obscure the relationship
with the definition namespace, making it less useful.

Additionally, (in the case of OrderedDict specifically) doing this would
require significant changes to the semantics of the concrete dict C-API.

There has been some discussion about moving to a compact dict
implementation which would (mostly) preserve insertion order. However
the lack of an explicit __definition_order__ would still remain as a
pain point.

A "namespace" Keyword Arg for Class Definition

PEP 422 <422#order-preserving-classes> introduced a new "namespace"
keyword arg to class definitions that effectively replaces the need to
__prepare__(). However, the proposal was withdrawn in favor of the
simpler PEP 487.

A stdlib Metaclass that Implements __prepare__() with OrderedDict

This has all the same problems as writing your own metaclass. The only
advantage is that you don't have to actually write this metaclass. So it
doesn't offer any benefit in the context of this PEP.

Set __definition_order__ at Compile-time

Each class's __qualname__ is determined at compile-time. This same
concept could be applied to __definition_order__. The result of
composing __definition_order__ at compile-time would be nearly the same
as doing so at run-time.

Comparative implementation difficulty aside, the key difference would be
that at compile-time it would not be practical to preserve definition
order for attributes that are set dynamically in the class body (e.g.
locals()[name] = value). However, they should still be reflected in the
definition order. One possible resolution would be to require class
authors to manually set __definition_order__ if they define any class
attributes dynamically.

Ultimately, the use of OrderedDict at run-time or compile-time discovery
is almost entirely an implementation detail.

References

-   Original discussion
-   Follow-up 1
-   Follow-up 2
-   Alyssa (Nick) Coghlan's concerns about mutability

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 662 Title: Editable installs via virtual wheels Author: Bernát
Gábor<EMAIL_ADDRESS>Sponsor: Brett Cannon<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/discuss-tbd-editable-installs-by-gaborbernat/9071
Status: Rejected Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 28-May-2021 Post-History: Resolution:
https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450

Abstract

This document describes extensions to the build backend and frontend
communication (as introduced by PEP 517) to allow projects to be
installed in editable mode by introducing virtual wheels.

Motivation

During development, many Python users prefer to install their libraries
so that changes to the underlying source code and resources are
automatically reflected in subsequent interpreter invocations without an
additional installation step. This mode is usually called "development
mode" or "editable installs". Currently, there is no standardized way to
accomplish this, as it was explicitly left out of PEP 517 due to the
complexity of the actual observed behaviors.

At the moment, users to get this behaviour perform one of the following:

-   For just Python code by adding the relevant source directories to
    sys.path (configurable from the command line interface via the
    PYTHONPATH environment variable). Note in this case, the users have
    to install the project dependencies themselves, and entry points or
    project metadata are not generated.
-   setuptools provides the setup.py develop mechanism: that installs a
    pth file that injects the project root onto the sys.path at
    interpreter startup time, generates the project metadata, and also
    installs project dependencies. pip exposes calling this mechanism
    via the pip install -e command-line interface.
-   flit provides the flit install --symlink command that symlinks the
    project files into the interpreters purelib folder, generates the
    project metadata, and also installs dependencies. Note, this allows
    supporting resource files too.

As these examples shows an editable install can be achieved in multiple
ways and at the moment there's no standard way of doing it. Furthermore,
it's not clear whose responsibility it is to achieve and define what an
editable installation is:

1.  allow the build backend to define and materialize it,
2.  allow the build frontend to define and materialize it,
3.  explicitly define and standardize one method from the possible
    options.

The author of this PEP believes there's no one size fits all solution
here, each method of achieving editable effect has its pros and cons.
Therefore this PEP rejects option three as it's unlikely for the
community to agree on a single solution. Furthermore, question remains
as to whether the frontend or the build backend should own this
responsibility. PEP 660 proposes the build backend to own this, while
the current PEP proposes primarily the frontend, but still allows the
backend to take take control if it wants to do so.

Rationale

PEP 517 deferred "Editable installs" because this would have delayed
further its adoption, and there wasn't an agreement on how editable
installs should be achieved. Due to the popularity of the setuptools and
pip projects, the status quo prevailed, and the backend could achieve
editable mode by providing a setup.py develop implementation, which the
user could trigger via pip install -e. By defining an editable interface
between the build backend and frontend, we can eliminate the setup.py
file and their current communication method.

Terminology and goals

This PEP aims to delineate the frontend and the backend roles clearly
and give the developers of each the maximum ability to provide valuable
features to their users. In this proposal, the backend's role is to
prepare the project for an editable installation, and then provide
enough information to the frontend so that the frontend can manifest and
enforce the editable installation.

The information the backend provides to the frontend is a wheel that
follows the existing specification within PEP 427. The wheel metadata
about the archive itself ({distribution}-{version}.dist-info/WHEEL) must
also contain the key Editable with value of true.

However, instead of providing the project files within the wheel, it
must provide an editable.json file (at the root level of the wheel) that
defines the files to be exposed by the frontend. The content of this
file is formulated as a mapping of absolute source tree paths to
relative target interpreter destination paths within a scheme mapping.

A wheel that satisfies the previous two paragraphs is a virtual wheel.
The frontend's role is to take the virtual wheel and install the project
in editable mode. The way it achieves this is entirely up to the
frontend and is considered implementation detail.

The editable installation mode implies that the source code of the
project being installed is available in a local directory. Once the
project is installed in editable mode, some changes to the project code
in the local source tree will become effective without the need for a
new installation step. At a minimum, changes to the text of
non-generated files that existed at the installation time should be
reflected upon the subsequent import of the package.

Some kinds of changes, such as adding or modifying entry points or new
dependencies, require a new installation step to become effective. These
changes are typically made in build backend configuration files (such as
pyproject.toml). This requirement is consistent with the general user
expectation that such modifications will only become effective after
re-installation.

While users expect editable installations to behave identically to
standard installations, this may not always be possible and may be in
tension with other user expectations. Depending on how a frontend
implements the editable mode, some differences may be visible, such as
the presence of additional files (compared to a typical installation),
either in the source tree or the interpreter's installation path.

Frontends should seek to minimize differences between the behavior of
editable and standard installations and document known differences.

For reference, a non-editable installation works as follows:

1.  The developer is using a tool, we'll call it here the frontend, to
    drive the project development (e.g., pip). When the user wants to
    trigger a package build and installation of a project, they'll
    communicate with the frontend.
2.  The frontend uses a build frontend to trigger the build of a wheel
    (e.g., build). The build frontend uses PEP 517 to communicate with
    the build backend (e.g. setuptools) - with the build backend
    installed into a PEP 518 environment. Once invoked, the backend
    returns a wheel.
3.  The frontend takes the wheel and feeds it to an installer (e.g.,
    installer) to install the wheel into the target Python interpreter.

The Mechanism

This PEP adds two optional hooks to the PEP 517 backend interface. One
of the hooks is used to specify the build dependencies of an editable
install. The other hook returns the necessary information via the build
frontend the frontend needs to create an editable install.

get_requires_for_build_editable

    def get_requires_for_build_editable(config_settings=None):
        ...

This hook MUST return an additional sequence of strings containing PEP
508 dependency specifications, above and beyond those specified in the
pyproject.toml file. The frontend must ensure that these dependencies
are available in the build environment in which the build_editable hook
is called.

If not defined, the default implementation is equivalent to returning
[].

prepare_metadata_for_build_editable

    def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
        ...

Must create a .dist-info directory containing wheel metadata inside the
specified metadata_directory (i.e., creates a directory like
{metadata_directory}/{package}-{version}.dist-info/). This directory
MUST be a valid .dist-info directory as defined in the wheel
specification, except that it need not contain RECORD or signatures. The
hook MAY also create other files inside this directory, and a build
frontend MUST preserve, but otherwise ignore, such files; the intention
here is that in cases where the metadata depends on build-time
decisions, the build backend may need to record these decisions in some
convenient format for re-use by the actual wheel-building step.

This must return the basename (not the full path) of the .dist-info
directory it creates, as a unicode string.

If a build frontend needs this information and the method is not
defined, it should call build_editable and look at the resulting
metadata directly.

build_editable

    def build_editable(self, wheel_directory, config_settings=None,
                        metadata_directory=None):
        ...

Must build a .whl file, and place it in the specified wheel_directory.
It must return the basename (not the full path) of the .whl file it
creates, as a unicode string. The wheel file must be of type virtual
wheel as defined under the terminology section.

If the build frontend has previously called
prepare_metadata_for_build_editable and depends on the wheel resulting
from this call to have metadata matching this earlier call, then it
should provide the path to the created .dist-info directory as the
metadata_directory argument. If this argument is provided, then
build_editable MUST produce a wheel with identical metadata. The
directory passed in by the build frontend MUST be identical to the
directory created by prepare_metadata_for_build_editable, including any
unrecognized files it created.

Backends which do not provide the prepare_metadata_for_build_editable
hook may either silently ignore the metadata_directory parameter to
build_editable, or else raise an exception when it is set to anything
other than None.

The source directory may be read-only, in such cases the backend may
raise an error that the frontend can display to the user. The backend
may store intermediate artifacts in cache locations or temporary
directories. The presence or absence of any caches should not make a
material difference to the final result of the build.

The content of the editable.json MUST pass against the following JSON
schema:

For example:

    {
       "version": 1,
       "scheme": {
          "purelib": {"/src/tree/a.py": "tree/a.py"},
          "platlib": {},
          "data": {"/src/tree/py.typed": "tree/py.typed"},
          "headers": {},
          "scripts": {}
       }
    }

The scheme paths map from project source absolute paths to target
directory relative paths. We allow backends to change the project layout
from the project source directory to what the interpreter will see by
using the mapping.

For example if the backend returns "purelib": {"/me/project/src": ""}
this would mean that expose all files and modules within /me/project/src
at the root of the purelib path within the target interpreter.

Build frontend requirements

The build frontend is responsible for setting up the environment for the
build backend to generate the virtual wheel. All recommendations from
PEP 517 for the build wheel hook applies here too.

Frontend requirements

The frontend must install the virtual wheel exactly as defined within
PEP 427. Furthermore is responsible for also installing the files
defined within the editable.json file. The manner in which it does is
left up to the frontend, and is encouraged for the frontend to
communicate with the user exactly the method chosen, and what
limitations that solution will have.

The frontend must create a direct_url.json file in the .dist-info
directory of the installed distribution, in compliance with PEP 610. The
url value must be a file:// URL pointing to the project directory (i.e.,
the directory containing pyproject.toml), and the dir_info value must be
{'editable': true}.

The frontend can rely on the prepare_metadata_for_build_editable hook
when installing in editable mode.

If the frontend concludes it cannot achieve an editable installation
with the information provided by the build backend it should fail and
raise an error to clarify to the user why not.

The frontend might implement one or more editable installation
mechanisms and can leave it up to the user the choose one that its
optimal to the use case of the user. For example, pip could add an
editable mode flag, and allow the user to choose between pth files or
symlinks ( pip install -e . --editable-mode=pth vs
pip install -e . --editable-mode=symlink).

Example editable implementations

To show how this PEP might be used, we'll now present a few case
studies. Note the offered solutions are purely for illustration purpose
and are not normative for the frontend/backend.

Add the source tree as is to the interpreter

This is one of the simplest implementations, it will add the source tree
as is into the interpreters scheme paths, the editable.json within the
virtual wheel might look like:

    {
      {"version": 1, "scheme": {"purelib": {"<project dir>": "<project dir>"}}}
    }

The frontend then could either:

-   Add the source directory onto the target interpreters sys.path
    during startup of it. This is done by creating a pth file into the
    target interpreters purelib folder. setuptools does this today and
    is what pip install -e translate too. This solution is fast and
    cross-platform compatible. However, this puts the entire source tree
    onto the system, potentially exposing modules that would not be
    available in a standard installation case.
-   Symlink the folder, or the individual files within it. This method
    is what flit does via its flit install --symlink. This solution
    requires the current platform to support symlinks. Still, it allows
    potentially to symlink individual files, which could solve the
    problem of including files that should be excluded from the source
    tree.

Using custom importers

For a more robust and more dynamic collaboration between the build
backend and the target interpreter, we can take advantage of the import
system allowing the registration of custom importers. See PEP 302 for
more details and editables as an example of this. The backend can
generate a new importer during the editable build (or install it as an
additional dependency) and register it at interpreter startup by adding
a pth file.

    {
       "version": 1,
       "scheme": {
          "purelib": {
            "<project dir>/.editable/_register_importer.pth": "<project dir>/_register_importer.pth".
            "<project dir>/.editable/_editable_importer.py": "<project dir>/_editable_importer.py"
          }
        }
      }
    }

The backend here registered a hook that is called whenever a new module
is imported, allowing dynamic and on-demand functionality. Potential use
cases where this is useful:

-   Expose a source folder, but honor module excludes: the backend may
    generate an import hook that consults the exclusion table before
    allowing a source file loader to discover a file in the source
    directory or not.
-   For a project, let there be two modules, A.py and B.py. These are
    two separate files in the source directory; however, while building
    a wheel, they are merged into one mega file project.py. In this
    case, with this PEP, the backend could generate an import hook that
    reads the source files at import time and merges them in memory
    before materializing it as a module.
-   Automatically update out-of-date C-extensions: the backend may
    generate an import hook that checks the last modified timestamp for
    a C-extension source file. If it is greater than the current
    C-extension binary, trigger an update by calling the compiler before
    import.

Rejected ideas

This PEP competes with PEP 660 and rejects that proposal because we
think the mechanism of achieving an editable installation should be
within the frontend rather than the build backend. Furthermore, this
approach allows the ecosystem to use alternative means to accomplish the
editable installation effect (e.g., insert path on sys.path or symlinks
instead of just implying the loose wheel mode from the backend described
by that PEP).

Prominently, PEP 660 does not allow using symlinks to expose code and
data files without also extending the wheel file standard with symlink
support. It's not clear how the wheel format could be extended to
support symlinks that refer not to files within the wheel itself, but
files only available on the local disk. It's important to note that the
backend itself (or backend generated code) must not generate these
symlinks (e.g., at interpreter startup time) as that would conflict with
the frontends book keeping of what files need to be uninstalled.

Finally, PEP 660 adds support only for purelib and platlib files. It
purposefully avoids supporting other types of information that the wheel
format supports: include, data and scripts. With this path the frontend
can support these on a best effort basis via the symlinks mechanism
(though this feature is not universally available - on Windows require
enablement). We believe its beneficial to add best effort support for
these file types, rather than exclude the possibility of supporting them
at all.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.