text:
PEP: 749 Title: Implementing PEP 649 Author: Jelle Zijlstra
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-749-implementing-pep-649/54974 Status:
Draft Type: Standards Track Topic: Typing Requires: 649 Created:
28-May-2024 Python-Version: 3.14 Post-History: 04-Jun-2024

Abstract

This PEP supplements PEP 649 by providing various tweaks and additions
to its specification:

-   from __future__ import annotations (PEP 563) will continue to exist
    with its current behavior at least until Python 3.13 reaches its
    end-of-life. Subsequently, it will be deprecated and eventually
    removed.
-   A new standard library module, annotationlib, is added to provide
    tooling for annotations. It will include the get_annotations()
    function, an enum for annotation formats, a ForwardRef class, and a
    helper function for calling __annotate__ functions.
-   Annotations in the REPL are lazily evaluated, just like other
    module-level annotations.
-   We specify the behavior of wrapper objects that provide annotations,
    such as classmethod and code that uses functools.wraps.
-   There will not be a code flag for marking __annotate__ functions
    that can be run in a "fake globals" environment. Instead, we add a
    fourth format, VALUE_WITH_FAKE_GLOBALS, to allow third-party
    implementors of annotate functions to indicate what formats they
    support.
-   Deleting the __annotations__ attribute directly will also clear
    __annotate__.
-   We add functionality to allow evaluating type alias values and type
    parameter bounds and defaults (which were added by PEP 695 and
    PEP 696) using PEP 649-like semantics.
-   The SOURCE format is renamed to STRING to improve clarity and reduce
    the risk of user confusion.

Motivation

PEP 649 provides an excellent framework for creating better semantics
for annotations in Python. It solves a common pain point for users of
annotations, including those using static type hints as well as those
using runtime typing, and it makes the language more elegant and
powerful. The PEP was originally proposed in 2021 for Python 3.10, and
it was accepted in 2023. However, the implementation took longer than
anticipated, and now the PEP is expected to be implemented in Python
3.14.

I have started working on the implementation of the PEP in CPython. I
found that the PEP leaves some areas underspecified, and some of its
decisions in corner cases are questionable. This new PEP proposes
several changes and additions to the specification to address these
issues.

This PEP supplements rather than supersedes PEP 649. The changes
proposed here should make the overall user experience better, but they
do not change the general framework of the earlier PEP.

The future of from __future__ import annotations

PEP 563 previously introduced the future import
from __future__ import annotations, which changes all annotations to
strings. PEP 649 proposes an alternative approach that does not require
this future import, and states:

  If this PEP is accepted, PEP 563 will be deprecated and eventually
  removed.

However, the PEP does not provide a detailed plan for this deprecation.

There is some previous discussion of this topic on Discourse (note that
in the linked post I proposed something different from what is proposed
here).

Specification

We suggest the following deprecation plan:

-   In Python 3.14, from __future__ import annotations will continue to
    work as it did before, converting annotations into strings.
    -   If the future import is active, the __annotate__ function of
        objects with annotations will return the annotations as strings
        when called with the VALUE format, reflecting the behavior of
        __annotations__.
-   Sometime after the last release that did not support PEP 649
    semantics (expected to be 3.13) reaches its end-of-life,
    from __future__ import annotations is deprecated. Compiling any code
    that uses the future import will emit a DeprecationWarning. This
    will happen no sooner than the first release after Python 3.13
    reaches its end-of-life, but the community may decide to wait
    longer.
-   After at least two releases, the future import is removed, and
    annotations are always evaluated as per PEP 649. Code that continues
    to use the future import will raise a SyntaxError, similar to any
    other undefined future import.

Rejected alternatives

Immediately make the future import a no-op: We considered applying PEP
649 semantics to all code in Python 3.14, making the future import a
no-op. However, this would break code that works in 3.13 under the
following set of conditions:

-   __future__ import annotations is active
-   There are annotations that rely on forward references
-   Annotations are eagerly evaluated at import time, for example by a
    metaclass or class or function decorator. For example, this
    currently applies to the released version of
    typing_extensions.TypedDict.

This is expected to be a common pattern, so we cannot afford to break
such code during the upgrade from 3.13 to 3.14.

Such code would still break when the future import is eventually
removed. However, this is many years in the future, giving affected
decorators plenty of time to update their code.

Immediately deprecate the future import: Instead of waiting until Python
3.13 reaches its end-of-life, we could immediately start emitting
warnings when the future import is used. However, many libraries are
already using from __future__ import annotations as an elegant way to
enable unrestricted forward references in their annotations. If we
deprecate the future import immediately, it would be impossible for
these libraries to use unrestricted forward references on all supported
Python versions while avoiding deprecation warnings: unlike other
features deprecated from the standard library, a __future__ import must
be the first statement in a given module, meaning it would be impossible
to only conditionally import __future__.annotations on Python 3.13 and
lower. (The necessary sys.version_info check would count as a statement
preceding the __future__ import.)

Keep the future import around forever: We could also decide to keep the
future import indefinitely. However, this would permanently bifurcate
the behavior of the Python language. This is undesirable; the language
should have only a single set of semantics, not two permanently
different modes.

Make the future import a no-op in the future: Instead of eventually
making from __future__ import annotations a SyntaxError, we could make
it do nothing instead at some point after Python 3.13 reaches its
end-of-life. This still has some of the same issues outlined above
around making it a no-op now, although the ecosystem would have had much
longer to adapt. It is better to have users explicitly remove the future
import from their code in the future once they have confirmed they do
not rely on stringized annotations.

New annotationlib module

PEP 649 proposes to add tooling related to annotations to the inspect
module. However, that module is rather large, has direct or indirect
dependencies on at least 35 other standard library modules, and is so
slow to import that other standard library modules are often discouraged
from importing it. Furthermore, we anticipate adding more tools in
addition to the inspect.get_annotations function and the VALUE,
FORWARDREF, and SOURCE formats.

A new standard library module provides a logical home for this
functionality and also enables us to add more tooling that is useful for
consumers of annotations.

Rationale

PEP 649 indicates that !typing.ForwardRef should be used to implement
the FORWARDREF format in inspect.get_annotations. However, the existing
implementation of !typing.ForwardRef is intertwined with the rest of the
!typing module, and it would not make sense to add !typing-specific
behavior to the generic get_annotations() function. Furthermore,
!typing.ForwardRef is a problematic class: it is public and documented,
but the documentation lists no attributes or methods for it.
Nonetheless, third-party libraries make use of some of its undocumented
attributes. For instance, Pydantic and Typeguard use the _evaluate
method; beartype and pyanalyze use the __forward_arg__ attribute.

We replace the existing but poorly specified !typing.ForwardRef with a
new class, annotationlib.ForwardRef. It is designed to be mostly
compatible with existing uses of the !typing.ForwardRef class, but
without the behaviors specific to the !typing module. For compatibility
with existing users, we keep the private _evaluate method, but mark it
as deprecated. It delegates to a new public function in the !typing
module, typing.evaluate_forward_ref, that is designed to evaluate
forward references in a way that is specific to type hints.

We add a function annotationlib.call_annotate_function as a helper for
calling __annotate__ functions. This is a useful building block when
implementing functionality that needs to partially evaluate annotations
while a class is being constructed. For example, the implementation of
typing.NamedTuple needs to retrieve the annotations from a class
namespace dictionary before the namedtuple class itself can be
constructed, because the annotations determine what fields exist on the
namedtuple.

Specification

A new module, annotationlib, is added to the standard library. Its aim
is to provide tooling for introspecting and wrapping annotations.

The exact contents of the module are not yet specified. We will add
support for PEP 649 semantics to standard library functionality that
uses annotations, such as dataclasses and typing.TypedDict, and use the
experience to inform the design of the new module.

The module will contain the following functionality:

-   get_annotations(): A function that returns the annotations of a
    function, module, or class. This will replace
    inspect.get_annotations. The latter will delegate to the new
    function. It may eventually be deprecated, but to minimize
    disruption, we do not propose an immediate deprecation.
-   get_annotate_function(): A function that returns the __annotate__
    function of an object, if it has one, or None if it does not. This
    is usually equivalent to accessing the .__annotate__ attribute,
    except in the presence of metaclasses (see
    below <pep749-metaclasses>).
-   Format: an enum that contains the possible formats of annotations.
    This will replace the VALUE, FORWARDREF, and SOURCE formats in
    PEP 649. PEP 649 proposed to make these values global members of the
    inspect module; we prefer to place them within an enum. We propose
    to add a fourth format, VALUE_WITH_FAKE_GLOBALS (see below).
-   ForwardRef: a class representing a forward reference; it may be
    returned by get_annotations() when the format is FORWARDREF. The
    existing class typing.ForwardRef will become an alias of this class.
    Its members include:
    -   __forward_arg__: the string argument of the forward reference
    -   evaluate(globals=None, locals=None, type_params=None, owner=None):
        a method that attempts to evaluate the forward reference. The
        ForwardRef object may hold a reference to the globals and other
        namespaces of the object that it originated from. If so, these
        namespaces may be used to evaluate the forward reference. The
        owner argument may be the object that holds the original
        annotation, such as the class or module object; it is used to
        extract the globals and locals namespaces if these are not
        provided.
    -   _evaluate(), with the same interface as the existing
        ForwardRef._evaluate method. It will be undocumented and
        immediately deprecated. It is provided for compatibility with
        existing users of typing.ForwardRef.
-   call_annotate_function(func: Callable, format: Format): a helper for
    calling an __annotate__ function with a given format. If the
    function does not support this format, call_annotate_function() will
    set up a "fake globals" environment, as described in PEP 649, and
    use that environment to return the desired annotations format.
-   call_evaluate_function(func: Callable | None, format: Format):
    similar to call_annotate_function, but does not rely on the function
    returning an annotations dictionary. This is intended to be used for
    evaluating deferred attributes introduced by PEP 695 and PEP 696;
    see below for details. func may be None for convenience; if None is
    passed, the function also returns None.
-   annotations_to_string(annotations: dict[str, object]) -> dict[str, str]:
    a function that converts each value in an annotations dictionary to
    a string representation. This is useful for implementing the SOURCE
    format in cases where the original source is not available, such as
    in the functional syntax for typing.TypedDict.
-   value_to_string(value: object) -> str: a function that converts a
    single value to a string representation. This is used by
    annotations_to_string. It uses repr() for most values, but for types
    it returns the fully qualified name. It is also useful as a helper
    for the repr() of a number of objects in the typing and
    collections.abc modules.

A new function is also added to the !typing module,
typing.evaluate_forward_ref. This function is a wrapper around the
ForwardRef.evaluate method, but it performs additional work that is
specific to type hints. For example, it recurses into complex types and
evaluates additional forward references within these types.

Contrary to PEP 649, the annotation formats (VALUE, FORWARDREF, and
SOURCE) will not be added as global members of the inspect module. The
only recommended way to refer to these constants will be as
annotationlib.Format.VALUE.

Open issues

What should this module be called? Some ideas:

-   annotations: The most obvious name, but it may cause confusion with
    the existing from __future__ import annotations, because users may
    have both import annotations and from __future__ import annotations
    in the same module. The use of a common word as the name will make
    the module harder to search for. There is a PyPI package
    annotations, but it had only a single release in 2015 and looks
    abandoned.
-   annotation (in the singular): Similar, but does not cause confusion
    with the future import. There is an abandoned PyPI package
    annotation, but it apparently never released any artifacts.
-   annotools: Analogous to itertools and functools, but "anno" is a
    less obvious abbreviation than "iter" or "func". As of this writing,
    there is no PyPI package with this name.
-   annotationtools: A more explicit version. There is a PyPI package
    annotationtools, which had a release in 2023.
-   annotation_tools: A variation of the above but without a PyPI
    conflict. However, no other public standard library module has an
    underscore in its name.
-   annotationslib: Analogous to tomllib, pathlib, and importlib. There
    is no PyPI package with this name.
-   annotationlib: Similar to the above, but one character shorter and
    subjectively reads better. Also not taken on PyPI.

Rejected alternatives

Add the functionality to the inspect module: As described above, the
inspect module is already quite large, and its import time is
prohibitive for some use cases.

Add the functionality to the typing module: While annotations are mostly
used for typing, they may also be used for other purposes. We prefer to
keep a clean separation between functionality for introspecting
annotations and functionality that is exclusively meant for type hints.

Add the functionality to the types module: The types module is meant for
functionality related to types, and annotations can exist on functions
and modules, not only on types.

Develop this functionality in a third-party package: The functionality
in this new module will be pure Python code, and it is possible to
implement a third-party package that provides the same functionality by
interacting directly with __annotate__ functions generated by the
interpreter. However, the functionality of the proposed new module will
certainly be useful in the standard library itself (e.g., for
implementing dataclasses and typing.NamedTuple), so it makes sense to
include it in the standard library.

Add this functionality to a private module: It would be possible to
initially develop the module in a private standard library module (e.g.,
_annotations), and publicize it only after we have gained more
experience with the API. However, we already know that we will need
parts of this module for the standard library itself (e.g., for
implementing !dataclasses and !typing.NamedTuple). Even if we make it
private, the module will inevitably get used by third-party users. It is
preferable to start with a clear, documented API from the beginning, to
enable third-party users to support PEP 649 semantics as thoroughly as
the standard library. The module will immediately be used in other parts
of the standard library, ensuring that it covers a reasonable set of use
cases.

Behavior of the REPL

PEP 649 specifies the following behavior of the interactive REPL:

  For the sake of simplicity, in this case we forego delayed evaluation.
  Module-level annotations in the REPL shell will continue to work
  exactly as they do with “stock semantics”, evaluating immediately and
  setting the result directly inside the __annotations__ dict.

There are several problems with this proposed behavior. It makes the
REPL the only context where annotations are still evaluated immediately,
which is confusing for users and complicates the language.

It also makes the implementation of the REPL more complex, as it needs
to ensure that all statements are compiled in "interactive" mode, even
if their output does not need to be displayed. (This matters if there
are multiple statements in a single line evaluated by the REPL.)

Most importantly, this breaks some plausible use cases that
inexperienced users could run into. A user might write the following in
a file:

    a: X | None = None
    class X: ...

Under PEP 649 this would work fine: X is not yet defined when it is used
in the annotation for a, but the annotation is lazily evaluated.
However, if a user were to paste this same code into the REPL and
execute it line by line, it would throw a NameError, because the name X
is not yet defined.

This topic was previously discussed on Discourse.

Specification

We propose to treat the interactive console like any other module-level
code, and make annotations lazily evaluated. This makes the language
more consistent and avoids subtle behavior changes between modules and
the REPL.

Because the REPL is evaluated line by line, we would generate a new
__annotate__ function for every evaluated statement in the global scope
that contains annotations. Whenever a line containing annotations is
evaluated, the previous __annotate__ function is lost:

    >>> x: int
    >>> __annotate__(1)
    {'x': <class 'int'>}
    >>> y: str
    >>> __annotate__(1)
    {'y': <class 'str'>}
    >>> z: doesntexist
    >>> __annotate__(1)
    Traceback (most recent call last):
    File "<python-input-5>", line 1, in <module>
        __annotate__(1)
        ~~~~~~~~~~~~^^^
    File "<python-input-4>", line 1, in __annotate__
        z: doesntexist
           ^^^^^^^^^^^
    NameError: name 'doesntexist' is not defined

There will be no __annotations__ key in the global namespace of the
REPL. In module namespaces, this key is created lazily when the
__annotations__ descriptor of the module object is accessed, but in the
REPL there is no such module object.

Classes and functions defined within the REPL will also work like any
other classes, so evaluation of their annotations will be deferred. It
is possible to access the __annotations__ and __annotate__ attributes or
use the annotationlib module to introspect the annotations.

Wrappers that provide __annotations__

Several objects in the standard library and elsewhere provide
annotations for their wrapped object. PEP 649 does not specify how such
wrappers should behave.

Specification

Wrappers that provide annotations should be designed with the following
goals in mind:

-   Evaluation of __annotations__ should be deferred for as long as
    possible, consistent with the behavior of built-in functions,
    classes, and modules.
-   Backward compatibility with the behavior prior to the implementation
    of PEP 649 should be preserved.
-   The __annotate__ and __annotations__ attributes should both be
    supplied with semantics consistent to those of the wrapped object.

More specifically:

-   functools.update_wrapper (and therefore functools.wraps) will copy
    only the __annotate__ attribute from the wrapped object to the
    wrapper. The __annotations__ descriptor on the wrapper function will
    use the copied __annotate__.
-   The constructors for classmethod and staticmethod currently copy the
    __annotations__ attribute from the wrapped object to the wrapper.
    They will instead have writable attributes for __annotate__ and
    __annotations__. Reading these attributes will retrieve the
    corresponding attribute from the underlying callable and cache it in
    the wrapper's __dict__. Writing to these attributes will directly
    update the __dict__, without affecting the wrapped callable.

Annotations and metaclasses

Testing of the initial implementation of this PEP revealed serious
problems with the interaction between metaclasses and class annotations.

Pre-existing bugs

We found several bugs in the existing behavior of __annotations__ on
classes while investigating the behaviors to be specified in this PEP.
Fixing these bugs on Python 3.13 and earlier is outside the scope of
this PEP, but they are noted here to explain the corner cases that need
to be dealt with.

For context, on Python 3.10 through 3.13 the __annotations__ dictionary
is placed in the class namespace if the class has any annotations. If it
does not, there is no __annotations__ class dictionary key when the
class is created, but accessing cls.__annotations__ invokes a descriptor
defined on type that returns an empty dictionary and stores it in the
class dictionary. Static types <static-types> are an exception: they
never have annotations, and accessing .__annotations__ raises
AttributeError. On Python 3.9 and earlier, the behavior was different;
see gh-88067.

The following code fails identically on Python 3.10 through 3.13:

    class Meta(type): pass

    class X(metaclass=Meta):
        a: str

    class Y(X): pass

    Meta.__annotations__  # important
    assert Y.__annotations__ == {}, Y.__annotations__  # fails: {'a': <class 'str'>}

If the annotations on the metaclass Meta are accessed before the
annotations on Y, then the annotations for the base class X are leaked
to Y. However, if the metaclass's annotations are not accessed (i.e.,
the line Meta.__annotations__ above is removed), then the annotations
for Y are correctly empty.

Similarly, annotations from annotated metaclasses leak to unannotated
classes that are instances of the metaclass:

    class Meta(type):
        a: str

    class X(metaclass=Meta):
        pass

    assert X.__annotations__ == {}, X.__annotations__  # fails: {'a': <class 'str'>}

The reason for these behaviors is that if the metaclass contains an
__annotations__ entry in its class dictionary, this prevents instances
of the metaclass from using the __annotations__ data descriptor on the
base type class. In the first case, accessing Meta.__annotations__ sets
Meta.__dict__["__annotations__"] = {} as a side effect. Then, looking up
the __annotations__ attribute on Y first sees the metaclass attribute,
but skips it because it is a data descriptor. Next, it looks in the
class dictionaries of the classes in its method resolution order (MRO),
finds X.__annotations__, and returns it. In the second example, there
are no annotations anywhere in the MRO, so type.__getattribute__ falls
back to returning the metaclass attribute.

Metaclass behavior with PEP 649

With PEP 649, the behavior of accessing the .__annotations__ attribute
on classes when metaclasses are involved becomes even more erratic,
because now __annotations__ is only lazily added to the class dictionary
even for classes with annotations. The new __annotate__ attribute is
also lazily created on classes without annotations, which causes further
misbehaviors when metaclasses are involved.

The cause of these problems is that we set the __annotate__ and
__annotations__ class dictionary entries only under some circumstances,
and rely on descriptors defined on type to fill them in if they are not
set. When normal attribute lookup is used, this approach breaks down in
the presence of metaclasses, because entries in the metaclass's own
class dictionary can render the descriptors invisible.

While we considered several approaches that would allow
cls.__annotations__ and cls.__annotate__ to work reliably when cls is a
type with a custom metaclass, any such approach would expose significant
complexity to advanced users. Instead, we recommend a simpler approach
that confines the complexity to the annotationlib module: in
annotationlib.get_annotations, we bypass normal attribute lookup by
using the type.__annotations__ descriptor directly.

Specification

Users should always use annotationlib.get_annotations to access the
annotations of a class object, and annotationlib.get_annotate_function
to access the __annotate__ function. These functions will return only
the class's own annotations, even when metaclasses are involved.

The behavior of accessing the __annotations__ and __annotate__
attributes on classes with a metaclass other than builtins.type is
unspecified. The documentation should warn against direct use of these
attributes and recommend using the annotationlib module instead.

Similarly, the presence of __annotations__ and __annotate__ keys in the
class dictionary is an implementation detail and should not be relied
upon.

Rejected alternatives

We considered two broad approaches for dealing with the behavior of the
__annotations__ and __annotate__ entries in classes:

-   Ensure that the entry is always present in the class dictionary,
    even if it is empty or has not yet been evaluated. This means we do
    not have to rely on the descriptors defined on type to fill in the
    field, and therefore the metaclass's attributes will not interfere.
    (Prototype in gh-120719.)
-   Ensure that the entry is never present in the class dictionary, or
    at least never added by logic in the language core. This means that
    the descriptors on type will always be used, without interference
    from the metaclass. (Prototype in gh-120816.)

Alex Waygood suggested an implementation using the first approach. When
a heap type (such as a class created through the class statement) is
created, cls.__dict__["__annotations__"] is set to a special descriptor.
On __get__, the descriptor evaluates the annotations by calling
__annotate__ and returning the result. The annotations dictionary is
cached within the descriptor instance. The descriptor also behaves like
a mapping, so that code that uses cls.__dict__["__annotations__"] will
still usually work: treating the object as a mapping will evaluate the
annotations and behave as if the descriptor itself was the annotations
dictionary. (Code that assumes that cls.__dict__["__annotations__"] is
specifically an instance of dict may break, however.)

This approach is also straightforward to implement for __annotate__:
this attribute is already always set for classes with annotations, and
we can set it explicitly to None for classes without annotations.

While this approach would fix the known edge cases with metaclasses, it
introduces significant complexity to all classes, including a new
built-in type (for the annotations descriptor) with unusual behavior.

The alternative approach would be to never set
__dict__["__annotations__"] and use some other storage to store the
cached annotations. This behavior change would have to apply even to
classes defined under from __future__ import annotations, because
otherwise there could be buggy behavior if a class is defined without
from __future__ import annotations but its metaclass does have the
future enabled. As PEP 649 previously noted, removing __annotations__
from class dictionaries also has backwards compatibility implications:
cls.__dict__.get("__annotations__") is a common idiom to retrieve
annotations.

This approach would also mean that accessing .__annotations__ on an
instance of an annotated class no longer works. While this behavior is
not documented, it is a long-standing feature of Python and is relied
upon by some users.

Adding the VALUE_WITH_FAKE_GLOBALS format

PEP 649 specifies:

  This PEP assumes that third-party libraries may implement their own
  __annotate__ methods, and those functions would almost certainly work
  incorrectly when run in this "fake globals" environment. For that
  reason, this PEP allocates a flag on code objects, one of the unused
  bits in co_flags, to mean "This code object can be run in a 'fake
  globals' environment." This makes the "fake globals" environment
  strictly opt-in, and it's expected that only __annotate__ methods
  generated by the Python compiler will set it.

However, this mechanism couples the implementation with low-level
details of the code object. The code object flags are CPython-specific
and the documentation explicitly warns <inspect-module-co-flags> against
relying on the values.

Larry Hastings suggested an alternative approach that does not rely on
code flags: a fourth format, VALUE_WITH_FAKE_GLOBALS. Compiler-generated
annotate functions would support only the VALUE and
VALUE_WITH_FAKE_GLOBALS formats, both of which are implemented
identically. The standard library would use the VALUE_WITH_FAKE_GLOBALS
format when invoking an annotate function in one of the special "fake
globals" environments.

This approach is useful as a forward-compatible mechanism for adding new
annotation formats in the future. Users who manually write annotate
functions should raise NotImplementedError if the
VALUE_WITH_FAKE_GLOBALS format is requested, so the standard library
will not call the manually written annotate function with "fake
globals", which could have unpredictable results.

Specification

An additional format, FAKE_GLOBALS_VALUE, is added to the Format enum in
the annotationlib module, with value equal to 2. (As a result, the
values of the other formats will shift relative to PEP 649: FORWARDREF
will be 3 and SOURCE will be 4.)

Compiler-generated annotate functions will support this format and
return the same value as they would return for the VALUE format. The
standard library will pass this format to the __annotate__ function when
it is called in a "fake globals" environment, as used to implement the
FORWARDREF and SOURCE formats. All public functions in the annotationlib
module that accept a format argument will raise NotImplementedError if
the format is FAKE_GLOBALS_VALUE.

Third-party code that implements __annotate__ functions should raise
NotImplementedError if the FAKE_GLOBALS_VALUE format is passed and the
function is not prepared to be run in a "fake globals" environment. This
should be mentioned in the data model documentation for __annotate__.

Effect of deleting __annotations__

PEP 649 specifies:

  Setting o.__annotations__ to a legal value automatically sets
  o.__annotate__ to None.

However, the PEP does not say what happens if the __annotations__
attribute is deleted (using del). It seems most consistent that deleting
the attribute will also delete __annotate__.

Specification

Deleting the __annotations__ attribute on functions, modules, and
classes results in setting __annotate__ to None.

Deferred evaluation of PEP 695 and 696 objects

Since PEP 649 was written, Python 3.12 and 3.13 gained support for
several new features that also use deferred evaluation, similar to the
behavior this PEP proposes for annotations:

-   The value of type aliases created through the type statement (PEP
    695)
-   The bound and constraints of typing.TypeVar objects created through
    the syntax for generics (PEP 695)
-   The default value of typing.TypeVar, ParamSpec, and
    typing.TypeVarTuple objects (PEP 696)

Currently, these objects use deferred evaluation, but there is no direct
access to the function object used for deferred evaluation. To enable
the same kind of introspection that is now possible for annotations, we
propose to expose the internal function objects, allowing users to
evaluate them using the FORWARDREF and SOURCE formats.

Specification

We will add the following new attributes:

-   evaluate_value on typing.TypeAliasType
-   evaluate_bound, evaluate_constraints, and evaluate_default on
    typing.TypeVar
-   evaluate_default on typing.ParamSpec
-   evaluate_default on typing.TypeVarTuple

Except for evaluate_value, these attributes may be None if the object
does not have a bound, constraints, or default. Otherwise, the attribute
is a callable, similar to an __annotate__ function, that takes a single
integer argument and returns the evaluated value. Unlike __annotate__
functions, these callables return a single value, not a dictionary of
annotations. These attributes are read-only.

Usually, users would use these attributes in combinations with
annotationlib.call_evaluate_function. For example, to get a TypeVar's
bound in SOURCE format, one could write
annotationlib.call_evaluate_function(T.evaluate_bound, annotationlib.Format.SOURCE).

Behavior of dataclass field types

One consequence of the deferred evaluation of annotations is that
dataclasses can use forward references in their annotations:

    >>> from dataclasses import dataclass
    >>> @dataclass
    ... class D:
    ...     x: undefined
    ...

However, the FORWARDREF format leaks into the field types of the
dataclass:

    >>> fields(D)[0].type
    ForwardRef('undefined')

We considered a change where the .type attribute of a field object would
trigger evaluation of annotations, so that the field type could contain
actual values in the case of forward references that were defined after
the dataclass itself was created, but before the field type is accessed.
However, this would also mean that accessing .type could now run
arbitrary code in the annotation, and potentially throws errors such as
NameError.

Therefore, we consider it more user-friendly to keep the ForwardRef
object in the type, and document that users who want to resolve forward
references can use the ForwardRef.evaluate method.

If use cases come up in the future, we could add additional
functionality, such as a new method that re-evaluates the annotation
from scratch.

Renaming SOURCE to STRING

The SOURCE format is meant for tools that need to show a human-readable
format that is close to the original source code. However, we cannot
retrieve the original source in __annotate__ functions, and in some
cases, we have __annotate__ functions in Python code that do not have
access to the original code. For example, this applies to
dataclasses.make_dataclass and the call-based syntax for
typing.TypedDict.

This makes the name SOURCE a bit of a misnomer. The goal of the format
should indeed be to recreate the source, but the name is likely to
mislead users in practice. A more neutral name would emphasize that the
format returns an annotation dictionary with only strings. We suggest
STRING.

Specification

The SOURCE format is renamed to STRING. To reiterate the changes in this
PEP, the four supported formats are now:

-   VALUE: the default format, which evaluates the annotations and
    returns the resulting values.
-   VALUE_WITH_FAKE_GLOBALS: for internal use; should be handled like
    VALUE by annotate functions that support execution with fake
    globals.
-   FORWARDREF: replaces undefined names with ForwardRef objects.
-   STRING: returns strings, attempts to recreate code close to the
    original source.

Miscellaneous implementation details

PEP 649 goes into considerable detail on some aspects of the
implementation. To avoid confusion, we describe a few aspects where the
current implementation differs from that described in the PEP. However,
these details are not guaranteed to hold in the future, and they may
change without notice in the future, unless they are documented in the
language reference.

Supported operations on ForwardRef objects

The SOURCE format is implemented by the "stringizer" technique, where
the globals dictionary of a function is augmented so that every lookup
results in a special object that can be used to reconstruct the
operations that are performed on the object.

PEP 649 specifies:

  In practice, the "stringizer" functionality will be implemented in the
  ForwardRef object currently defined in the typing module. ForwardRef
  will be extended to implement all stringizer functionality; it will
  also be extended to support evaluating the string it contains, to
  produce the real value (assuming all symbols referenced are defined).

However, this is likely to lead to confusion in practice. An object that
implements stringizer functionality must implement almost all special
methods, including __getattr__ and __eq__, to return a new stringizer.
Such an object is confusing to work with: all operations succeed, but
they are likely to return different objects than the user expects.

The current implementation instead implements only a few useful methods
on the ForwardRef class. During the evaluation of annotations, an
instance of a private stringizer class is used instead of ForwardRef.
After evaluation completes, the implementation of the FORWARDREF format
converts these internal objects into ForwardRef objects.

Signature of __annotate__ functions

PEP 649 specifies the signature of __annotate__ functions as:

  __annotate__(format: int) -> dict

However, using format as a parameter name could lead to collisions if an
annotation uses a symbol named format. The parameter should be
positional-only and have a name that cannot be a legal identifier in
order to avoid this problem.

The current implementation uses the name .format with a leading dot, but
the exact name should be considered an implementation detail and cannot
be relied upon.

The documentation may still use the name format for simplicity.

Backwards Compatibility

PEP 649 provides a thorough discussion of the backwards compatibility
implications on existing code that uses either stock or PEP 563
semantics.

However, there is another set of compatibility problems: new code that
is written assuming PEP 649 semantics, but uses existing tools that
eagerly evaluate annotations. For example, consider a dataclass-like
class decorator @annotator that retrieves the annotated fields in the
class it decorates, either by accessing __annotations__ directly or by
calling inspect.get_annotations.

Once PEP 649 is implemented, code like this will work fine:

    class X:
        y: Y

    class Y: pass

But this will not, unless @annotator is changed to use the new
FORWARDREF format:

    @annotator
    class X:
        y: Y

    class Y: pass

This is not strictly a backwards compatibility issue, since no
previously working code would break; before PEP 649, this code would
have raised NameError at runtime. In a sense, it is no different from
any other new Python feature that needs to be supported by third-party
libraries. Nevertheless, it is a serious issue for libraries that
perform introspection, and it is important that we make it as easy as
possible for libraries to support the new semantics in a
straightforward, user-friendly way.

We will update those parts of the standard library that are affected by
this problem, and we propose to add commonly useful functionality to the
new annotationlib module, so third-party tools can use the same set of
tools.

Security Implications

None.

How to Teach This

The semantics of PEP 649, as modified by this PEP, should largely be
intuitive for users who add annotations to their code. We eliminate the
need for manually adding quotes around annotations that require forward
references, a major source of confusion for users.

For advanced users who need to introspect annotations, the story becomes
more complex. The documentation of the new annotationlib module will
serve as a reference for users who need to interact programmatically
with annotations.

Reference Implementation

The in-progress PR #119891 implements much of this PEP.

Open Issues

We may discover additional areas where PEP 649 needs clarification or
amendment as we make progress on implementing it. Readers are encouraged
to follow the CPython issue tracking the implementation of the PEP and
try out the draft implementation. Any feedback may be incorporated into
future versions of this PEP.

Acknowledgments

First of all, I thank Larry Hastings for writing PEP 649. This PEP
modifies some of his initial decisions, but the overall design is still
his.

I thank Carl Meyer and Alex Waygood for feedback on early drafts of this
PEP. Alex Waygood, Alyssa Coghlan, and David Ellis provided insightful
feedback and suggestions on the interaction between metaclasses and
__annotations__. Larry Hastings also provided useful feedback on this
PEP.

Appendix

Which expressions can be stringified?

PEP 649 acknowledges that the stringifier cannot handle all expressions.
Now that we have a draft implementation, we can be more precise about
the expressions that can and cannot be handled. Below is a list of all
expressions in the Python AST that can and cannot be recovered by the
stringifier. The full list should probably not be added to the
documentation, but creating it is a useful exercise.

First, the stringifier of course cannot recover any information that is
not present in the compiled code, including comments, whitespace,
parenthesization, and operations that get simplified by the AST
optimizer.

Second, the stringifier can intercept almost all operations that involve
names looked up in some scope, but it cannot intercept operations that
operate fully on constants. As a corollary, this also means it is not
safe to request the SOURCE format on untrusted code: Python is powerful
enough that it is possible to achieve arbitrary code execution even with
no access to any globals or builtins. For example:

    >>> def f(x: (1).__class__.__base__.__subclasses__()[-1].__init__.__builtins__["print"]("Hello world")): pass
    ...
    >>> annotationlib.get_annotations(f, format=annotationlib.Format.SOURCE)
    Hello world
    {'x': 'None'}

(This particular example worked for me on the current implementation of
a draft of this PEP; the exact code may not keep working in the future.)

The following are supported (sometimes with caveats):

-   BinOp
-   UnaryOp
    -   Invert (~), UAdd (+), and USub (-) are supported
    -   Not (not) is not supported
-   Dict (except when using ** unpacking)
-   Set
-   Compare
    -   Eq and NotEq are supported
    -   Lt, LtE, Gt, and GtE are supported, but the operand may be
        flipped
    -   Is, IsNot, In, and NotIn are not supported
-   Call (except when using ** unpacking)
-   Constant (though not the exact representation of the constant; for
    example, escape sequences in strings are lost; hexadecimal numbers
    are converted to decimal)
-   Attribute (assuming the value is not a constant)
-   Subscript (assuming the value is not a constant)
-   Starred (* unpacking)
-   Name
-   List
-   Tuple
-   Slice

The following are unsupported, but throw an informative error when
encountered by the stringifier:

-   FormattedValue (f-strings; error is not detected if conversion
    specifiers like !r are used)
-   JoinedStr (f-strings)

The following are unsupported and result in incorrect output:

-   BoolOp (and and or)
-   IfExp
-   Lambda
-   ListComp
-   SetComp
-   DictComp
-   GeneratorExp

The following are disallowed in annotation scopes and therefore not
relevant:

-   NamedExpr (:=)
-   Await
-   Yield
-   YieldFrom

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 246 Title: Object Adaptation Author: Alex Martelli
<EMAIL_ADDRESS>Clark C. Evans<EMAIL_ADDRESS>Status:
Rejected Type: Standards Track Content-Type: text/x-rst Created:
21-Mar-2001 Python-Version: 2.5 Post-History: 29-Mar-2001, 10-Jan-2005

Rejection Notice

I'm rejecting this PEP. Something much better is about to happen; it's
too early to say exactly what, but it's not going to resemble the
proposal in this PEP too closely so it's better to start a new PEP. GvR.

Abstract

This proposal puts forth an extensible cooperative mechanism for the
adaptation of an incoming object to a context which expects an object
supporting a specific protocol (say a specific type, class, or
interface).

This proposal provides a built-in "adapt" function that, for any object
X and any protocol Y, can be used to ask the Python environment for a
version of X compliant with Y. Behind the scenes, the mechanism asks
object X: "Are you now, or do you know how to wrap yourself to provide,
a supporter of protocol Y?". And, if this request fails, the function
then asks protocol Y: "Does object X support you, or do you know how to
wrap it to obtain such a supporter?" This duality is important, because
protocols can be developed after objects are, or vice-versa, and this
PEP lets either case be supported non-invasively with regard to the
pre-existing component[s].

Lastly, if neither the object nor the protocol know about each other,
the mechanism may check a registry of adapter factories, where callables
able to adapt certain objects to certain protocols can be registered
dynamically. This part of the proposal is optional: the same effect
could be obtained by ensuring that certain kinds of protocols and/or
objects can accept dynamic registration of adapter factories, for
example via suitable custom metaclasses. However, this optional part
allows adaptation to be made more flexible and powerful in a way that is
not invasive to either protocols or other objects, thereby gaining for
adaptation much the same kind of advantage that Python standard
library's "copy_reg" module offers for serialization and persistence.

This proposal does not specifically constrain what a protocol is, what
"compliance to a protocol" exactly means, nor what precisely a wrapper
is supposed to do. These omissions are intended to leave this proposal
compatible with both existing categories of protocols, such as the
existing system of type and classes, as well as the many concepts for
"interfaces" as such which have been proposed or implemented for Python,
such as the one in PEP 245, the one in Zope3[1], or the ones discussed
in the BDFL's Artima blog in late 2004 and early 2005[2]. However, some
reflections on these subjects, intended to be suggestive and not
normative, are also included.

Motivation

Currently there is no standardized mechanism in Python for checking if
an object supports a particular protocol. Typically, existence of
certain methods, particularly special methods such as __getitem__, is
used as an indicator of support for a particular protocol. This
technique works well for a few specific protocols blessed by the BDFL
(Benevolent Dictator for Life). The same can be said for the alternative
technique based on checking 'isinstance' (the built-in class
"basestring" exists specifically to let you use 'isinstance' to check if
an object "is a [built-in] string"). Neither approach is easily and
generally extensible to other protocols, defined by applications and
third party frameworks, outside of the standard Python core.

Even more important than checking if an object already supports a given
protocol can be the task of obtaining a suitable adapter (wrapper or
proxy) for the object, if the support is not already there. For example,
a string does not support the file protocol, but you can wrap it into a
StringIO instance to obtain an object which does support that protocol
and gets its data from the string it wraps; that way, you can pass the
string (suitably wrapped) to subsystems which require as their arguments
objects that are readable as files. Unfortunately, there is currently no
general, standardized way to automate this extremely important kind of
"adaptation by wrapping" operations.

Typically, today, when you pass objects to a context expecting a
particular protocol, either the object knows about the context and
provides its own wrapper or the context knows about the object and wraps
it appropriately. The difficulty with these approaches is that such
adaptations are one-offs, are not centralized in a single place of the
users code, and are not executed with a common technique, etc. This lack
of standardization increases code duplication with the same adapter
occurring in more than one place or it encourages classes to be
re-written instead of adapted. In either case, maintainability suffers.

It would be very nice to have a standard function that can be called
upon to verify an object's compliance with a particular protocol and
provide for a wrapper if one is readily available --all without having
to hunt through each library's documentation for the incantation
appropriate to that particular, specific case.

Requirements

When considering an object's compliance with a protocol, there are
several cases to be examined:

a)  When the protocol is a type or class, and the object has exactly
    that type or is an instance of exactly that class (not a subclass).
    In this case, compliance is automatic.
b)  When the object knows about the protocol, and either considers
    itself compliant, or knows how to wrap itself suitably.
c)  When the protocol knows about the object, and either the object
    already complies or the protocol knows how to suitably wrap the
    object.
d)  When the protocol is a type or class, and the object is a member of
    a subclass. This is distinct from the first case (a) above, since
    inheritance (unfortunately) does not necessarily imply
    substitutability, and thus must be handled carefully.
e)  When the context knows about the object and the protocol and knows
    how to adapt the object so that the required protocol is satisfied.
    This could use an adapter registry or similar approaches.

The fourth case above is subtle. A break of substitutability can occur
when a subclass changes a method's signature, or restricts the domains
accepted for a method's argument ("co-variance" on arguments types), or
extends the co-domain to include return values which the base class may
never produce ("contra-variance" on return types). While compliance
based on class inheritance should be automatic, this proposal allows an
object to signal that it is not compliant with a base class protocol.

If Python gains some standard "official" mechanism for interfaces,
however, then the "fast-path" case (a) can and should be extended to the
protocol being an interface, and the object an instance of a type or
class claiming compliance with that interface. For example, if the
"interface" keyword discussed in[3] is adopted into Python, the "fast
path" of case (a) could be used, since instantiable classes implementing
an interface would not be allowed to break substitutability.

Specification

This proposal introduces a new built-in function, adapt(), which is the
basis for supporting these requirements.

The adapt() function has three parameters:

-   obj, the object to be adapted
-   protocol, the protocol requested of the object
-   alternate, an optional object to return if the object could not be
    adapted

A successful result of the adapt() function returns either the object
passed obj, if the object is already compliant with the protocol, or a
secondary object wrapper, which provides a view of the object compliant
with the protocol. The definition of wrapper is deliberately vague, and
a wrapper is allowed to be a full object with its own state if
necessary. However, the design intention is that an adaptation wrapper
should hold a reference to the original object it wraps, plus (if
needed) a minimum of extra state which it cannot delegate to the wrapper
object.

An excellent example of adaptation wrapper is an instance of StringIO
which adapts an incoming string to be read as if it was a textfile: the
wrapper holds a reference to the string, but deals by itself with the
"current point of reading" (from where in the wrapped strings will the
characters for the next, e.g., "readline" call come from), because it
cannot delegate it to the wrapped object (a string has no concept of
"current point of reading" nor anything else even remotely related to
that concept).

A failure to adapt the object to the protocol raises an AdaptationError
(which is a subclass of TypeError), unless the alternate parameter is
used, in this case the alternate argument is returned instead.

To enable the first case listed in the requirements, the adapt()
function first checks to see if the object's type or the object's class
are identical to the protocol. If so, then the adapt() function returns
the object directly without further ado.

To enable the second case, when the object knows about the protocol, the
object must have a __conform__() method. This optional method takes two
arguments:

-   self, the object being adapted
-   protocol, the protocol requested

Just like any other special method in today's Python, __conform__ is
meant to be taken from the object's class, not from the object itself
(for all objects, except instances of "classic classes" as long as we
must still support the latter). This enables a possible 'tp_conform'
slot to be added to Python's type objects in the future, if desired.

The object may return itself as the result of __conform__ to indicate
compliance. Alternatively, the object also has the option of returning a
wrapper object compliant with the protocol. If the object knows it is
not compliant although it belongs to a type which is a subclass of the
protocol, then __conform__ should raise a LiskovViolation exception (a
subclass of AdaptationError). Finally, if the object cannot determine
its compliance, it should return None to enable the remaining
mechanisms. If __conform__ raises any other exception, "adapt" just
propagates it.

To enable the third case, when the protocol knows about the object, the
protocol must have an __adapt__() method. This optional method takes two
arguments:

-   self, the protocol requested
-   obj, the object being adapted

If the protocol finds the object to be compliant, it can return obj
directly. Alternatively, the method may return a wrapper compliant with
the protocol. If the protocol knows the object is not compliant although
it belongs to a type which is a subclass of the protocol, then __adapt__
should raise a LiskovViolation exception (a subclass of
AdaptationError). Finally, when compliance cannot be determined, this
method should return None to enable the remaining mechanisms. If
__adapt__ raises any other exception, "adapt" just propagates it.

The fourth case, when the object's class is a sub-class of the protocol,
is handled by the built-in adapt() function. Under normal circumstances,
if "isinstance(object, protocol)" then adapt() returns the object
directly. However, if the object is not substitutable, either the
__conform__() or __adapt__() methods, as above mentioned, may raise an
LiskovViolation (a subclass of AdaptationError) to prevent this default
behavior.

If none of the first four mechanisms worked, as a last-ditch attempt,
'adapt' falls back to checking a registry of adapter factories, indexed
by the protocol and the type of obj, to meet the fifth case. Adapter
factories may be dynamically registered and removed from that registry
to provide "third party adaptation" of objects and protocols that have
no knowledge of each other, in a way that is not invasive to either the
object or the protocols.

Intended Use

The typical intended use of adapt is in code which has received some
object X "from the outside", either as an argument or as the result of
calling some function, and needs to use that object according to a
certain protocol Y. A "protocol" such as Y is meant to indicate an
interface, usually enriched with some semantics constraints (such as are
typically used in the "design by contract" approach), and often also
some pragmatical expectation (such as "the running time of a certain
operation should be no worse than O(N)", or the like); this proposal
does not specify how protocols are designed as such, nor how or whether
compliance to a protocol is checked, nor what the consequences may be of
claiming compliance but not actually delivering it (lack of "syntactic"
compliance -- names and signatures of methods -- will often lead to
exceptions being raised; lack of "semantic" compliance may lead to
subtle and perhaps occasional errors [imagine a method claiming to be
threadsafe but being in fact subject to some subtle race condition, for
example]; lack of "pragmatic" compliance will generally lead to code
that runs correctly, but too slowly for practical use, or sometimes to
exhaustion of resources such as memory or disk space).

When protocol Y is a concrete type or class, compliance to it is
intended to mean that an object allows all of the operations that could
be performed on instances of Y, with "comparable" semantics and
pragmatics. For example, a hypothetical object X that is a singly-linked
list should not claim compliance with protocol 'list', even if it
implements all of list's methods: the fact that indexing X[n] takes time
O(n), while the same operation would be O(1) on a list, makes a
difference. On the other hand, an instance of StringIO.StringIO does
comply with protocol 'file', even though some operations (such as those
of module 'marshal') may not allow substituting one for the other
because they perform explicit type-checks: such type-checks are "beyond
the pale" from the point of view of protocol compliance.

While this convention makes it feasible to use a concrete type or class
as a protocol for purposes of this proposal, such use will often not be
optimal. Rarely will the code calling 'adapt' need ALL of the features
of a certain concrete type, particularly for such rich types as file,
list, dict; rarely can all those features be provided by a wrapper with
good pragmatics, as well as syntax and semantics that are really the
same as a concrete type's.

Rather, once this proposal is accepted, a design effort needs to start
to identify the essential characteristics of those protocols which are
currently used in Python, particularly within the standard library, and
to formalize them using some kind of "interface" construct (not
necessarily requiring any new syntax: a simple custom metaclass would
let us get started, and the results of the effort could later be
migrated to whatever "interface" construct is eventually accepted into
the Python language). With such a palette of more formally designed
protocols, the code using 'adapt' will be able to ask for, say,
adaptation into "a filelike object that is readable and seekable", or
whatever else it specifically needs with some decent level of
"granularity", rather than too-generically asking for compliance to the
'file' protocol.

Adaptation is NOT "casting". When object X itself does not conform to
protocol Y, adapting X to Y means using some kind of wrapper object Z,
which holds a reference to X, and implements whatever operation Y
requires, mostly by delegating to X in appropriate ways. For example, if
X is a string and Y is 'file', the proper way to adapt X to Y is to make
a StringIO(X), NOT to call file(X) [which would try to open a file named
by X].

Numeric types and protocols may need to be an exception to this
"adaptation is not casting" mantra, however.

Guido's "Optional Static Typing: Stop the Flames" Blog Entry

A typical simple use case of adaptation would be:

    def f(X):
        X = adapt(X, Y)
        # continue by using X according to protocol Y

In[4], the BDFL has proposed introducing the syntax:

    def f(X: Y):
        # continue by using X according to protocol Y

to be a handy shortcut for exactly this typical use of adapt, and, as a
basis for experimentation until the parser has been modified to accept
this new syntax, a semantically equivalent decorator:

    @arguments(Y)
    def f(X):
        # continue by using X according to protocol Y

These BDFL ideas are fully compatible with this proposal, as are other
of Guido's suggestions in the same blog.

Reference Implementation and Test Cases

The following reference implementation does not deal with classic
classes: it consider only new-style classes. If classic classes need to
be supported, the additions should be pretty clear, though a bit messy
(x.__class__ vs type(x), getting boundmethods directly from the object
rather than from the type, and so on).

    -----------------------------------------------------------------
    adapt.py
    -----------------------------------------------------------------
    class AdaptationError(TypeError):
        pass
    class LiskovViolation(AdaptationError):
        pass

    _adapter_factory_registry = {}

    def registerAdapterFactory(objtype, protocol, factory):
        _adapter_factory_registry[objtype, protocol] = factory

    def unregisterAdapterFactory(objtype, protocol):
        del _adapter_factory_registry[objtype, protocol]

    def _adapt_by_registry(obj, protocol, alternate):
        factory = _adapter_factory_registry.get((type(obj), protocol))
        if factory is None:
            adapter = alternate
        else:
            adapter = factory(obj, protocol, alternate)
        if adapter is AdaptationError:
            raise AdaptationError
        else:
            return adapter

    def adapt(obj, protocol, alternate=AdaptationError):

        t = type(obj)

        # (a) first check to see if object has the exact protocol
        if t is protocol:
           return obj

        try:
            # (b) next check if t.__conform__ exists & likes protocol
            conform = getattr(t, '__conform__', None)
            if conform is not None:
                result = conform(obj, protocol)
                if result is not None:
                    return result

            # (c) then check if protocol.__adapt__ exists & likes obj
            adapt = getattr(type(protocol), '__adapt__', None)
            if adapt is not None:
                result = adapt(protocol, obj)
                if result is not None:
                    return result
        except LiskovViolation:
            pass
        else:
            # (d) check if object is instance of protocol
            if isinstance(obj, protocol):
                return obj

        # (e) last chance: try the registry
        return _adapt_by_registry(obj, protocol, alternate)

    -----------------------------------------------------------------
    test.py
    -----------------------------------------------------------------
    from adapt import AdaptationError, LiskovViolation, adapt
    from adapt import registerAdapterFactory, unregisterAdapterFactory
    import doctest

    class A(object):
        '''
        >>> a = A()
        >>> a is adapt(a, A)   # case (a)
        True
        '''

    class B(A):
        '''
        >>> b = B()
        >>> b is adapt(b, A)   # case (d)
        True
        '''

    class C(object):
        '''
        >>> c = C()
        >>> c is adapt(c, B)   # case (b)
        True
        >>> c is adapt(c, A)   # a failure case
        Traceback (most recent call last):
            ...
        AdaptationError
        '''
        def __conform__(self, protocol):
            if protocol is B:
                return self

    class D(C):
        '''
        >>> d = D()
        >>> d is adapt(d, D)   # case (a)
        True
        >>> d is adapt(d, C)   # case (d) explicitly blocked
        Traceback (most recent call last):
            ...
        AdaptationError
        '''
        def __conform__(self, protocol):
            if protocol is C:
                raise LiskovViolation

    class MetaAdaptingProtocol(type):
        def __adapt__(cls, obj):
            return cls.adapt(obj)

    class AdaptingProtocol:
        __metaclass__ = MetaAdaptingProtocol
        @classmethod
        def adapt(cls, obj):
            pass

    class E(AdaptingProtocol):
        '''
        >>> a = A()
        >>> a is adapt(a, E)   # case (c)
        True
        >>> b = A()
        >>> b is adapt(b, E)   # case (c)
        True
        >>> c = C()
        >>> c is adapt(c, E)   # a failure case
        Traceback (most recent call last):
            ...
        AdaptationError
        '''
        @classmethod
        def adapt(cls, obj):
            if isinstance(obj, A):
                return obj

    class F(object):
        pass

    def adapt_F_to_A(obj, protocol, alternate):
        if isinstance(obj, F) and issubclass(protocol, A):
            return obj
        else:
            return alternate

    def test_registry():
        '''
        >>> f = F()
        >>> f is adapt(f, A)   # a failure case
        Traceback (most recent call last):
            ...
        AdaptationError
        >>> registerAdapterFactory(F, A, adapt_F_to_A)
        >>> f is adapt(f, A)   # case (e)
        True
        >>> unregisterAdapterFactory(F, A)
        >>> f is adapt(f, A)   # a failure case again
        Traceback (most recent call last):
            ...
        AdaptationError
        >>> registerAdapterFactory(F, A, adapt_F_to_A)
        '''

    doctest.testmod()

Relationship To Microsoft's QueryInterface

Although this proposal has some similarities to Microsoft's (COM)
QueryInterface, it differs by a number of aspects.

First, adaptation in this proposal is bi-directional, allowing the
interface (protocol) to be queried as well, which gives more dynamic
abilities (more Pythonic). Second, there is no special "IUnknown"
interface which can be used to check or obtain the original unwrapped
object identity, although this could be proposed as one of those
"special" blessed interface protocol identifiers. Third, with
QueryInterface, once an object supports a particular interface it must
always there after support this interface; this proposal makes no such
guarantee, since, in particular, adapter factories can be dynamically
added to the registried and removed again later.

Fourth, implementations of Microsoft's QueryInterface must support a
kind of equivalence relation -- they must be reflexive, symmetrical, and
transitive, in specific senses. The equivalent conditions for protocol
adaptation according to this proposal would also represent desirable
properties:

    # given, to start with, a successful adaptation:
    X_as_Y = adapt(X, Y)

    # reflexive:
    assert adapt(X_as_Y, Y) is X_as_Y

    # transitive:
    X_as_Z = adapt(X, Z, None)
    X_as_Y_as_Z = adapt(X_as_Y, Z, None)
    assert (X_as_Y_as_Z is None) == (X_as_Z is None)

    # symmetrical:
    X_as_Z_as_Y = adapt(X_as_Z, Y, None)
    assert (X_as_Y_as_Z is None) == (X_as_Z_as_Y is None)

However, while these properties are desirable, it may not be possible to
guarantee them in all cases. QueryInterface can impose their equivalents
because it dictates, to some extent, how objects, interfaces, and
adapters are to be coded; this proposal is meant to be not necessarily
invasive, usable and to "retrofit" adaptation between two frameworks
coded in mutual ignorance of each other without having to modify either
framework.

Transitivity of adaptation is in fact somewhat controversial, as is the
relationship (if any) between adaptation and inheritance.

The latter would not be controversial if we knew that inheritance always
implies Liskov substitutability, which, unfortunately we don't. If some
special form, such as the interfaces proposed in [5], could indeed
ensure Liskov substitutability, then for that kind of inheritance, only,
we could perhaps assert that if X conforms to Y and Y inherits from Z
then X conforms to Z... but only if substitutability was taken in a very
strong sense to include semantics and pragmatics, which seems doubtful.
(For what it's worth: in QueryInterface, inheritance does not require
nor imply conformance). This proposal does not include any "strong"
effects of inheritance, beyond the small ones specifically detailed
above.

Similarly, transitivity might imply multiple "internal" adaptation
passes to get the result of adapt(X, Z) via some intermediate Y,
intrinsically like adapt(adapt(X, Y), Z), for some suitable and
automatically chosen Y. Again, this may perhaps be feasible under
suitably strong constraints, but the practical implications of such a
scheme are still unclear to this proposal's authors. Thus, this proposal
does not include any automatic or implicit transitivity of adaptation,
under whatever circumstances.

For an implementation of the original version of this proposal which
performs more advanced processing in terms of transitivity, and of the
effects of inheritance, see Phillip J. Eby's PyProtocols[6]. The
documentation accompanying PyProtocols is well worth studying for its
considerations on how adapters should be coded and used, and on how
adaptation can remove any need for typechecking in application code.

Questions and Answers

-   Q: What benefit does this proposal provide?

    A: The typical Python programmer is an integrator, someone who is
    connecting components from various suppliers. Often, to interface
    between these components, one needs intermediate adapters. Usually
    the burden falls upon the programmer to study the interface exposed
    by one component and required by another, determine if they are
    directly compatible, or develop an adapter. Sometimes a supplier may
    even include the appropriate adapter, but even then searching for
    the adapter and figuring out how to deploy the adapter takes time.

    This technique enables suppliers to work with each other directly,
    by implementing __conform__ or __adapt__ as necessary. This frees
    the integrator from making their own adapters. In essence, this
    allows the components to have a simple dialogue among themselves.
    The integrator simply connects one component to another, and if the
    types don't automatically match an adapting mechanism is built-in.

    Moreover, thanks to the adapter registry, a "fourth party" may
    supply adapters to allow interoperation of frameworks which are
    totally unaware of each other, non-invasively, and without requiring
    the integrator to do anything more than install the appropriate
    adapter factories in the registry at start-up.

    As long as libraries and frameworks cooperate with the adaptation
    infrastructure proposed here (essentially by defining and using
    protocols appropriately, and calling 'adapt' as needed on arguments
    received and results of call-back factory functions), the
    integrator's work thereby becomes much simpler.

    For example, consider SAX1 and SAX2 interfaces: there is an adapter
    required to switch between them. Normally, the programmer must be
    aware of this; however, with this adaptation proposal in place, this
    is no longer the case --indeed, thanks to the adapter registry, this
    need may be removed even if the framework supplying SAX1 and the one
    requiring SAX2 are unaware of each other.

-   Q: Why does this have to be built-in, can't it be standalone?

    A: Yes, it does work standalone. However, if it is built-in, it has
    a greater chance of usage. The value of this proposal is primarily
    in standardization: having libraries and frameworks coming from
    different suppliers, including the Python standard library, use a
    single approach to adaptation. Furthermore:

    0.  The mechanism is by its very nature a singleton.
    1.  If used frequently, it will be much faster as a built-in.
    2.  It is extensible and unassuming.
    3.  Once 'adapt' is built-in, it can support syntax extensions and
        even be of some help to a type inference system.

-   Q: Why the verbs __conform__ and __adapt__?

    A: conform, verb intransitive

    1.  To correspond in form or character; be similar.
    2.  To act or be in accord or agreement; comply.
    3.  To act in accordance with current customs or modes.

    adapt, verb transitive

    1.  To make suitable to or fit for a specific use or situation.

    Source: The American Heritage Dictionary of the English Language,
    Third Edition

Backwards Compatibility

There should be no problem with backwards compatibility unless someone
had used the special names __conform__ or __adapt__ in other ways, but
this seems unlikely, and, in any case, user code should never use
special names for non-standard purposes.

This proposal could be implemented and tested without changes to the
interpreter.

Credits

This proposal was created in large part by the feedback of the talented
individuals on the main Python mailing lists and the type-sig list. To
name specific contributors (with apologies if we missed anyone!),
besides the proposal's authors: the main suggestions for the proposal's
first versions came from Paul Prescod, with significant feedback from
Robin Thomas, and we also borrowed ideas from Marcin 'Qrczak' Kowalczyk
and Carlos Ribeiro.

Other contributors (via comments) include Michel Pelletier, Jeremy
Hylton, Aahz Maruch, Fredrik Lundh, Rainer Deyke, Timothy Delaney, and
Huaiyu Zhu. The current version owes a lot to discussions with (among
others) Phillip J. Eby, Guido van Rossum, Bruce Eckel, Jim Fulton, and
Ka-Ping Yee, and to study and reflection of their proposals,
implementations, and documentation about use and adaptation of
interfaces and protocols in Python.

References and Footnotes

Copyright

This document has been placed in the public domain.

[1] http://www.zope.org/Wikis/Interfaces/FrontPage

[2] http://www.artima.com/weblogs/index.jsp?blogger=guido

[3] http://www.artima.com/weblogs/index.jsp?blogger=guido

[4] http://www.artima.com/weblogs/viewpost.jsp?thread=87182

[5] http://www.artima.com/weblogs/viewpost.jsp?thread=87182

[6] http://peak.telecommunity.com/PyProtocols.html

license: Public Domain