text:
PEP: 240 Title: Adding a Rational Literal to Python Author: Christopher
A. Craig<EMAIL_ADDRESS>Moshe Zadka<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
11-Mar-2001 Python-Version: 2.2 Post-History: 16-Mar-2001

Abstract

A different PEP <239> suggests adding a builtin rational type to Python.
This PEP suggests changing the ddd.ddd float literal to a rational in
Python, and modifying non-integer division to return it.

BDFL Pronouncement

This PEP is rejected. The needs outlined in the rationale section have
been addressed to some extent by the acceptance of PEP 327 for decimal
arithmetic. Guido also noted, "Rational arithmetic was the default
'exact' arithmetic in ABC and it did not work out as expected". See the
python-dev discussion on 17 June 2005[1].

Rationale

Rational numbers are useful for exact and unsurprising arithmetic. They
give the correct results people have been taught in various math
classes. Making the "obvious" non-integer type one with more predictable
semantics will surprise new programmers less than using floating point
numbers. As quite a few posts on c.l.py and on<EMAIL_ADDRESS>have
shown, people often get bit by strange semantics of floating point
numbers: for example, round(0.98, 2) still gives 0.97999999999999998.

Proposal

Literals conforming to the regular expression '\d*.\d*' will be rational
numbers.

Backwards Compatibility

The only backwards compatible issue is the type of literals mentioned
above. The following migration is suggested:

1.  The next Python after approval will allow
    from __future__ import rational_literals to cause all such literals
    to be treated as rational numbers.
2.  Python 3.0 will have a warning, turned on by default, about such
    literals in the absence of a __future__ statement. The warning
    message will contain information about the __future__ statement, and
    indicate that to get floating point literals, they should be
    suffixed with "e0".
3.  Python 3.1 will have the warning turned off by default. This warning
    will stay in place for 24 months, at which time the literals will be
    rationals and the warning will be removed.

Common Objections

Rationals are slow and memory intensive! (Relax, I'm not taking floats
away, I'm just adding two more characters. 1e0 will still be a float)

Rationals must present themselves as a decimal float or they will be
horrible for users expecting decimals (i.e. str(.5) should return '.5'
and not '1/2'). This means that many rationals must be truncated at some
point, which gives us a new loss of precision.

References

Copyright

This document has been placed in the public domain.

[1] Raymond Hettinger, Propose rejection of PEPs 239 and 240 -- a
builtin rational type and rational literals
https://mail.python.org/pipermail/python-dev/2005-June/054281.html

authors:
Christopher A. Craig
Moshe Zadka


text:
PEP: 267 Title: Optimized Access to Module Namespaces Author: Jeremy
Hylton<EMAIL_ADDRESS>Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 23-May-2001 Python-Version: 2.2
Post-History:

Deferral

While this PEP is a nice idea, no-one has yet emerged to do the work of
hashing out the differences between this PEP, PEP 266 and PEP 280.
Hence, it is being deferred.

Abstract

This PEP proposes a new implementation of global module namespaces and
the builtin namespace that speeds name resolution. The implementation
would use an array of object pointers for most operations in these
namespaces. The compiler would assign indices for global variables and
module attributes at compile time.

The current implementation represents these namespaces as dictionaries.
A global name incurs a dictionary lookup each time it is used; a builtin
name incurs two dictionary lookups, a failed lookup in the global
namespace and a second lookup in the builtin namespace.

This implementation should speed Python code that uses module-level
functions and variables. It should also eliminate awkward coding styles
that have evolved to speed access to these names.

The implementation is complicated because the global and builtin
namespaces can be modified dynamically in ways that are impossible for
the compiler to detect. (Example: A module's namespace is modified by a
script after the module is imported.) As a result, the implementation
must maintain several auxiliary data structures to preserve these
dynamic features.

Introduction

This PEP proposes a new implementation of attribute access for module
objects that optimizes access to module variables known at compile time.
The module will store these variables in an array and provide an
interface to lookup attributes using array offsets. For globals,
builtins, and attributes of imported modules, the compiler will generate
code that uses the array offsets for fast access.

[describe the key parts of the design: dlict, compiler support, stupid
name trick workarounds, optimization of other module's globals]

The implementation will preserve existing semantics for module
namespaces, including the ability to modify module namespaces at runtime
in ways that affect the visibility of builtin names.

DLict design

The namespaces are implemented using a data structure that has sometimes
gone under the name dlict. It is a dictionary that has numbered slots
for some dictionary entries. The type must be implemented in C to
achieve acceptable performance. The new type-class unification work
should make this fairly easy. The DLict will presumably be a subclass of
dictionary with an alternate storage module for some keys.

A Python implementation is included here to illustrate the basic design:

    """A dictionary-list hybrid"""

    import types

    class DLict:
        def __init__(self, names):
            assert isinstance(names, types.DictType)
            self.names = {}
            self.list = [None] * size
            self.empty = [1] * size
            self.dict = {}
            self.size = 0

        def __getitem__(self, name):
            i = self.names.get(name)
            if i is None:
                return self.dict[name]
            if self.empty[i] is not None:
                raise KeyError, name
            return self.list[i]

        def __setitem__(self, name, val):
            i = self.names.get(name)
            if i is None:
                self.dict[name] = val
            else:
                self.empty[i] = None
                self.list[i] = val
                self.size += 1

        def __delitem__(self, name):
            i = self.names.get(name)
            if i is None:
                del self.dict[name]
            else:
                if self.empty[i] is not None:
                    raise KeyError, name
                self.empty[i] = 1
                self.list[i] = None
                self.size -= 1

        def keys(self):
            if self.dict:
                return self.names.keys() + self.dict.keys()
            else:
                return self.names.keys()

        def values(self):
            if self.dict:
                return self.names.values() + self.dict.values()
            else:
                return self.names.values()

        def items(self):
            if self.dict:
                return self.names.items()
            else:
                return self.names.items() + self.dict.items()

        def __len__(self):
            return self.size + len(self.dict)

        def __cmp__(self, dlict):
            c = cmp(self.names, dlict.names)
            if c != 0:
                return c
            c = cmp(self.size, dlict.size)
            if c != 0:
                return c
            for i in range(len(self.names)):
                c = cmp(self.empty[i], dlict.empty[i])
            if c != 0:
                return c
            if self.empty[i] is None:
                c = cmp(self.list[i], dlict.empty[i])
                if c != 0:
                    return c
            return cmp(self.dict, dlict.dict)

        def clear(self):
            self.dict.clear()
            for i in range(len(self.names)):
                if self.empty[i] is None:
                    self.empty[i] = 1
                    self.list[i] = None

        def update(self):
            pass

        def load(self, index):
            """dlict-special method to support indexed access"""
            if self.empty[index] is None:
                return self.list[index]
            else:
                raise KeyError, index # XXX might want reverse mapping

        def store(self, index, val):
            """dlict-special method to support indexed access"""
            self.empty[index] = None
            self.list[index] = val

        def delete(self, index):
            """dlict-special method to support indexed access"""
            self.empty[index] = 1
            self.list[index] = None

Compiler issues

The compiler currently collects the names of all global variables in a
module. These are names bound at the module level or bound in a class or
function body that declares them to be global.

The compiler would assign indices for each global name and add the names
and indices of the globals to the module's code object. Each code object
would then be bound irrevocably to the module it was defined in. (Not
sure if there are some subtle problems with this.)

For attributes of imported modules, the module will store an indirection
record. Internally, the module will store a pointer to the defining
module and the offset of the attribute in the defining module's global
variable array. The offset would be initialized the first time the name
is looked up.

Runtime model

The PythonVM will be extended with new opcodes to access globals and
module attributes via a module-level array.

A function object would need to point to the module that defined it in
order to provide access to the module-level global array.

For module attributes stored in the dlict (call them static attributes),
the get/delattr implementation would need to track access to these
attributes using the old by-name interface. If a static attribute is
updated dynamically, e.g.:

    mod.__dict__["foo"] = 2

The implementation would need to update the array slot instead of the
backup dict.

Backwards compatibility

The dlict will need to maintain meta-information about whether a slot is
currently used or not. It will also need to maintain a pointer to the
builtin namespace. When a name is not currently used in the global
namespace, the lookup will have to fail over to the builtin namespace.

In the reverse case, each module may need a special accessor function
for the builtin namespace that checks to see if a global shadowing the
builtin has been added dynamically. This check would only occur if there
was a dynamic change to the module's dlict, i.e. when a name is bound
that wasn't discovered at compile-time.

These mechanisms would have little if any cost for the common case
whether a module's global namespace is not modified in strange ways at
runtime. They would add overhead for modules that did unusual things
with global names, but this is an uncommon practice and probably one
worth discouraging.

It may be desirable to disable dynamic additions to the global namespace
in some future version of Python. If so, the new implementation could
provide warnings.

Related PEPs

PEP 266, Optimizing Global Variable/Attribute Access, proposes a
different mechanism for optimizing access to global variables as well as
attributes of objects. The mechanism uses two new opcodes TRACK_OBJECT
and UNTRACK_OBJECT to create a slot in the local variables array that
aliases the global or object attribute. If the object being aliases is
rebound, the rebind operation is responsible for updating the aliases.

The objecting tracking approach applies to a wider range of objects than
just module. It may also have a higher runtime cost, because each
function that uses a global or object attribute must execute extra
opcodes to register its interest in an object and unregister on exit;
the cost of registration is unclear, but presumably involves a
dynamically resizable data structure to hold a list of callbacks.

The implementation proposed here avoids the need for registration,
because it does not create aliases. Instead it allows functions that
reference a global variable or module attribute to retain a pointer to
the location where the original binding is stored. A second advantage is
that the initial lookup is performed once per module rather than once
per function call.

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 747 Title: Annotating Type Forms Author: David Foster <david at
dafoster.net>, Eric Traut <erictr at microsoft.com> Sponsor: Jelle
Zijlstra <jelle.zijlstra at gmail.com> Discussions-To:
https://discuss.python.org/t/pep-747-typeexpr-type-hint-for-a-type-expression/55984
Status: Draft Type: Standards Track Topic: Typing Created: 27-May-2024
Python-Version: 3.14 Post-History: 19-Apr-2024, 04-May-2024, 17-Jun-2024

Abstract

Type expressions <typing:type-expression> provide a standardized way to
specify types in the Python type system. When a type expression is
evaluated at runtime, the resulting type form object encodes the
information supplied in the type expression. This enables a variety of
use cases including runtime type checking, introspection, and
metaprogramming.

Such use cases have proliferated, but there is currently no way to
accurately annotate functions that accept type form objects. Developers
are forced to use an overly-wide type like object, which makes some use
cases impossible and generally reduces type safety. This PEP addresses
this limitation by introducing a new special form typing.TypeForm.

This PEP makes no changes to the Python grammar. Correct usage of
TypeForm is intended to be enforced only by type checkers, not by the
Python runtime.

Motivation

A function that operates on type form objects must understand how type
expression details are encoded in these objects. For example, int | str,
"int | str", list[int], and MyTypeAlias are all valid type expressions,
and they evaluate to instances of types.UnionType, builtins.str,
types.GenericAlias, and typing.TypeAliasType, respectively.

There is currently no way to indicate to a type checker that a function
accepts type form objects and knows how to work with them. TypeForm
addresses this limitation. For example, here is a function that checks
whether a value is assignable to a specified type and returns None if it
is not:

    def trycast[T](typx: TypeForm[T], value: object) -> T | None: ...

The use of TypeForm and the type variable T describes a relationship
between the type form passed to parameter typx and the function's return
type.

TypeForm can also be used with typing:typeis to define custom type
narrowing behaviors:

    def isassignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]: ...

    request_json: object = ...
    if isassignable(request_json, MyTypedDict):
        assert_type(request_json, MyTypedDict)  # Type of variable is narrowed

The isassignable function implements something like an enhanced
isinstance check. This is useful for validating whether a value decoded
from JSON conforms to a particular structure of nested TypedDicts,
lists, unions, Literals, or any other type form that can be described
with a type expression. This kind of check was alluded to in
PEP 589 <589#using-typeddict-types> but could not be implemented without
TypeForm.

Why not type[C]?

One might think that type[C] would suffice for these use cases. However,
only class objects (instances of the builtins.type class) are assignable
to type[C]. Many type form objects do not meet this requirement:

    def trycast[T](typx: type[T], value: object) -> T | None: ...

    trycast(str, 'hi')  # OK
    trycast(Literal['hi'], 'hi')  # Type violation
    trycast(str | None, 'hi')  # Type violation
    trycast(MyProtocolClass, obj)  # Type violation

TypeForm use cases

A survey of Python libraries reveals several categories of functions
that would benefit from TypeForm:

-   Assignability checkers:
    -   Determines whether a value is assignable to a specified type
    -   Pattern 1:
        def is_assignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]
    -   Pattern 2:
        def is_match[T](value: object, typx: TypeForm[T]) -> TypeGuard[T]
    -   Examples: beartype.is_bearable, trycast.isassignable,
        typeguard.check_type, xdsl.isa
-   Converters:
    -   If a value is assignable to (or coercible to) a specified type,
        a converter returns the value narrowed to (or coerced to) that
        type. Otherwise, an exception is raised.

    -   Pattern 1:

            def convert[T](value: object, typx: TypeForm[T]) -> T

        -   Examples: cattrs.BaseConverter.structure, trycast.checkcast,
            typedload.load

    -   Pattern 2:

            class Converter[T]:
                def __init__(self, typx: TypeForm[T]) -> None: ...
                def convert(self, value: object) -> T: ...

        -   Examples: pydantic.TypeAdapter(T).validate_python,
            mashumaro.JSONDecoder(T).decode
-   Typed field definitions:
    -   Pattern:

            class Field[T]:
                value_type: TypeForm[T]

    -   Examples: attrs.make_class, dataclasses.make_dataclass[1],
        openapify

The survey also identified some introspection functions that accept
runtime type forms as input. Today, these functions are annotated with
object:

-   General introspection operations:
    -   Pattern: def get_annotation_info(typx: object) -> object
    -   Examples: typing.{get_origin, get_args},
        typing_inspect.{is*_type, get_origin, get_parameters}

These functions accept values evaluated from arbitrary annotation
expressions, not just type expressions, so they cannot be altered to use
TypeForm.

Specification

When a type expression is evaluated at runtime, the resulting value is a
type form object. This value encodes the information supplied in the
type expression, and it represents the type described by that type
expression.

TypeForm is a special form that, when used in a type expression,
describes a set of type form objects. It accepts a single type argument,
which must be a valid type expression. TypeForm[T] describes the set of
all type form objects that represent the type T or types that are
assignable to <typing:assignable> T. For example, TypeForm[str | None]
describes the set of all type form objects that represent a type
assignable to str | None:

    ok1: TypeForm[str | None] = str | None  # OK
    ok2: TypeForm[str | None] = str   # OK
    ok3: TypeForm[str | None] = None  # OK
    ok4: TypeForm[str | None] = Literal[None]  # OK
    ok5: TypeForm[str | None] = Optional[str]  # OK
    ok6: TypeForm[str | None] = "str | None"  # OK
    ok7: TypeForm[str | None] = Any  # OK

    err1: TypeForm[str | None] = str | int  # Error
    err2: TypeForm[str | None] = list[str | None]  # Error

By this same definition, TypeForm[Any] describes a type form object that
represents the type Any or any type that is assignable to Any. Since all
types in the Python type system are assignable to Any, TypeForm[Any]
describes the set of all type form objects evaluated from all valid type
expressions.

The type expression TypeForm, with no type argument provided, is
equivalent to TypeForm[Any].

Implicit TypeForm Evaluation

When a static type checker encounters an expression that follows all of
the syntactic, semantic and contextual rules for a type expression as
detailed in the typing spec, the evaluated type of this expression
should be assignable to TypeForm[T] if the type it describes is
assignable to T.

For example, if a static type checker encounters the expression
str | None, it may normally evaluate its type as UnionType because it
produces a runtime value that is an instance of types.UnionType.
However, because this expression is a valid type expression, it is also
assignable to the type TypeForm[str | None]:

    v1_actual: UnionType = str | None  # OK
    v1_type_form: TypeForm[str | None] = str | None  # OK

    v2_actual: type = list[int]  # OK
    v2_type_form: TypeForm = list[int]  # OK

The Annotated special form is allowed in type expressions, so it can
also appear in an expression that is assignable to TypeForm. Consistent
with the typing spec's rules for Annotated, a static type checker may
choose to ignore any Annotated metadata that it does not understand:

    v3: TypeForm[int | str] = Annotated[int | str, "metadata"]  # OK
    v4: TypeForm[Annotated[int | str, "metadata"]] = int | str  # OK

A string literal expression containing a valid type expression should
likewise be assignable to TypeForm:

    v5: TypeForm[set[str]] = "set[str]"  # OK

Expressions that violate one or more of the syntactic, semantic, or
contextual rules for type expressions should not evaluate to a TypeForm
type. The rules for type expression validity are explained in detail
within the typing spec, so they are not repeated here:

    bad1: TypeForm = tuple()  # Error: Call expression not allowed in type expression
    bad2: TypeForm = (1, 2)  # Error: Tuple expression not allowed in type expression
    bad3: TypeForm = 1  # Non-class object not allowed in type expression
    bad4: TypeForm = Self  # Error: Self not allowed outside of a class
    bad5: TypeForm = Literal[var]  # Error: Variable not allowed in type expression
    bad6: TypeForm = Literal[f""]  # Error: f-strings not allowed in type expression
    bad7: TypeForm = ClassVar[int]  # Error: ClassVar not allowed in type expression
    bad8: TypeForm = Required[int]  # Error: Required not allowed in type expression
    bad9: TypeForm = Final[int]  # Error: Final not allowed in type expression
    bad10: TypeForm = Unpack[Ts]  # Error: Unpack not allowed in this context
    bad11: TypeForm = Optional  # Error: Invalid use of Optional special form
    bad12: TypeForm = T  # Error if T is an out-of-scope TypeVar
    bad13: TypeForm = "int + str"  # Error: invalid quoted type expression

Explicit TypeForm Evaluation

TypeForm also acts as a function that can be called with a single
argument. Type checkers should validate that this argument is a valid
type expression:

    x1 = TypeForm(str | None)
    reveal_type(v1)  # Revealed type is "TypeForm[str | None]"

    x2 = TypeForm("list[int]")
    revealed_type(v2)  # Revealed type is "TypeForm[list[int]]"

    x3 = TypeForm('type(1)')  # Error: invalid type expression

At runtime the TypeForm(...) callable simply returns the value passed to
it.

This explicit syntax serves two purposes. First, it documents the
developer's intent to use the value as a type form object. Second,
static type checkers validate that all rules for type expressions are
followed:

    x4 = type(int)  # No error, evaluates to "type[int]"

    x5 = TypeForm(type(int))  # Error: call not allowed in type expression

Assignability

TypeForm has a single type parameter, which is covariant. That means
TypeForm[B] is assignable to TypeForm[A] if B is assignable to A:

    def get_type_form() -> TypeForm[int]: ...

    t1: TypeForm[int | str] = get_type_form()  # OK
    t2: TypeForm[str] = get_type_form()  # Error

type[T] is a subtype of TypeForm[T], which means that type[B] is
assignable to TypeForm[A] if B is assignable to A:

    def get_type() -> type[int]: ...

    t3: TypeForm[int | str] = get_type()  # OK
    t4: TypeForm[str] = get_type()  # Error

TypeForm is a subtype of object and is assumed to have all of the
attributes and methods of object.

Backward Compatibility

This PEP clarifies static type checker behaviors when evaluating type
expressions in "value expression" contexts (that is, contexts where type
expressions are not mandated by the typing spec). In the absence of a
TypeForm type annotation, existing type evaluation behaviors persist, so
no backward compatibility issues are anticipated. For example, if a
static type checker previously evaluated the type of expression
str | None as UnionType, it will continue to do so unless this
expression is assigned to a variable or parameter whose type is
annotated as TypeForm.

How to Teach This

Type expressions are used in annotations to describe which values are
accepted by a function parameter, returned by a function, or stored in a
variable:

    parameter type   return type
    |                |
    v                v
    def plus(n1: int, n2: int) -> int:
    sum: int = n1 + n2
    ^
    |
    variable type

    return sum

Type expressions evaluate to valid type form objects at runtime and can
be assigned to variables and manipulated like any other data in a
program:

    a variable                   a type expression
    |                            |
    v                            v
    int_type_form: TypeForm = int | None
                    ^
                    |
                    the type of a type form object

TypeForm[] is how you spell the type of a type form object, which is a
runtime representation of a type.

TypeForm is similar to type, but type is compatible only with class
objects like int, str, list, or MyClass. TypeForm accommodates any type
form that can be expressed using a valid type expression, including
those with brackets (list[int]), union operators (int | None), and
special forms (Any, LiteralString, Never, etc.).

Most programmers will not define their own functions that accept a
TypeForm parameter or return a TypeForm value. It is more common to pass
a type form object to a library function that knows how to decode and
use such objects.

For example, the isassignable function in the trycast library can be
used like Python's built-in isinstance function to check whether a value
matches the shape of a particular type. isassignable accepts any type
form object as input.

-   Yes:

        from trycast import isassignable

        if isassignable(some_object, MyTypedDict):  # OK: MyTypedDict is a TypeForm[]
            ...

-   No:

        if isinstance(some_object, MyTypedDict):  # ERROR: MyTypedDict is not a type[]
            ...

Advanced Examples

If you want to write your own runtime type checker or a function that
manipulates type form objects as values at runtime, this section
provides examples of how such a function can use TypeForm.

Introspecting type form objects

Functions like typing.get_origin and typing.get_args can be used to
extract components of some type form objects.

    import typing

    def strip_annotated_metadata(typx: TypeForm[T]) -> TypeForm[T]:
        if typing.get_origin(typx) is typing.Annotated:
            typx = cast(TypeForm[T], typing.get_args(typx)[0])
        return typx

isinstance and is can also be used to distinguish between different
kinds of type form objects:

    import types
    import typing

    def split_union(typx: TypeForm) -> tuple[TypeForm, ...]:
        if isinstance(typ, types.UnionType):  # X | Y
            return cast(tuple[TypeForm, ...], typing.get_args(typ))
        if typing.get_origin(typ) is typing.Union:  # Union[X, Y]
            return cast(tuple[TypeForm, ...], typing.get_args(typ))
        if typ in (typing.Never, typing.NoReturn,):
            return ()
        return (typ,)

Combining with a type variable

TypeForm can be parameterized by a type variable that is used elsewhere
within the same function definition:

    def as_instance[T](typx: TypeForm[T]) -> T | None:
        return typ() if isinstance(typ, type) else None

Combining with type

Both TypeForm and type can be parameterized by the same type variable
within the same function definition:

    def as_type[T](typx: TypeForm[T]) -> type[T] | None:
        return typ if isinstance(typ, type) else None

Combining with TypeIs and TypeGuard

A type variable can also be used by a TypeIs or TypeGuard return type:

    def isassignable[T](value: object, typx: TypeForm[T]) -> TypeIs[T]: ...

    count: int | str = ...
    if isassignable(count, int):
        assert_type(count, int)
    else:
        assert_type(count, str)

Challenges When Accepting All TypeForms

A function that takes an arbitrary TypeForm as input must support a
variety of possible type form objects. Such functions are not easy to
write.

-   New special forms are introduced with each new Python version, and
    special handling may be required for each one.
-   Quoted annotations[2] (like 'list[str]') must be parsed (to
    something like list[str]).
-   Resolving quoted forward references inside type expressions is
    typically done with eval(), which is difficult to use in a safe way.
-   Recursive types like IntTree = list[int | 'IntTree'] are difficult
    to resolve.
-   User-defined generic types (like Django’s QuerySet[User]) can
    introduce non-standard behaviors that require runtime support.

Reference Implementation

Pyright (version 1.1.379) provides a reference implementation for
TypeForm.

Mypy contributors also plan to implement support for TypeForm.

A reference implementation of the runtime component is provided in the
typing_extensions module.

Rejected Ideas

Alternative names

Alternate names were considered for TypeForm. TypeObject and TypeType
were deemed too generic. TypeExpression and TypeExpr were also
considered, but these were considered confusing because these objects
are not themselves "expressions" but rather the result of evaluating a
type expression.

Widen type[C] to support all type expressions

type was designed to describe class objects, subclasses of the type
class. A value with the type type is assumed to be instantiable through
a constructor call. Widening the meaning of type to represent arbitrary
type form objects would present backward compatibility problems and
would eliminate a way to describe the set of values limited to
subclasses of type.

Accept arbitrary annotation expressions

Certain special forms act as type qualifiers and can be used in some but
not all annotation contexts:

For example. the type qualifier Final can be used as a variable type but
not as a parameter type or a return type:

    some_const: Final[str] = ...  # OK

    def foo(not_reassignable: Final[object]): ...  # Error: Final not allowed here

    def nonsense() -> Final[object]: ...  # Error: Final not alowed here

With the exception of Annotated, type qualifiers are not allowed in type
expressions. TypeForm is limited to type expressions because its
assignability rules are based on the assignability rules for types. It
is nonsensical to ask whether Final[int] is assignable to int because
the former is not a valid type expression.

Functions that wish to operate on objects that are evaluated from
annotation expressions can continue to accept such inputs as object
parameters.

Pattern matching on type forms

It was asserted that some functions may wish to pattern match on the
interior of type expressions in their signatures.

One use case is to allow a function to explicitly enumerate all the
specific kinds of type expressions it supports as input. Consider the
following possible pattern matching syntax:

    @overload
    def checkcast(typx: TypeForm[AT=Annotated[T, *A]], value: str) -> T: ...
    @overload
    def checkcast(typx: TypeForm[UT=Union[*Ts]], value: str) -> Union[*Ts]: ...
    @overload
    def checkcast(typx: type[C], value: str) -> C: ...
    # ... (more)

All functions observed in the wild that conceptually accept type form
objects generally try to support all kinds of type expressions, so it
doesn’t seem valuable to enumerate a particular subset.

Additionally, the above syntax isn’t precise enough to fully describe
the input constraints for a typical function in the wild. For example,
many functions do not support type expressions with quoted
subexpressions like list['Movie'].

A second use case for pattern matching is to explicitly match an
Annotated form to extract the interior type argument and strip away any
metadata:

    def checkcast(
        typx: TypeForm[T] | TypeForm[AT=Annotated[T, *A]],
        value: object
    ) -> T:

However, Annotated[T, metadata] is already treated equivalent to T by
static type checkers. There’s no additional value in being explicit
about this behavior. The example above could more simply be written as
the equivalent:

    def checkcast(typx: TypeForm[T], value: object) -> T:

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] dataclass.make_dataclass allows the type qualifier InitVar[...], so
TypeForm cannot be used in this case.

[2] Quoted annotations are expected to become less common starting in
Python 3.14 when deferred annotations <649> is implemented. However,
code written for earlier Python versions relies on quoted annotations
and will need to be supported for several years.

authors:
David Foster
Eric Traut


text:
PEP: 205 Title: Weak References Version: $Revision$ Last-Modified:
$Date$ Author: Fred L. Drake, Jr<EMAIL_ADDRESS>Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 14-Jul-2000
Python-Version: 2.1 Post-History: 11-Jan-2001

Motivation

There are two basic applications for weak references which have been
noted by Python programmers: object caches and reduction of pain from
circular references.

Caches (weak dictionaries)

There is a need to allow objects to be maintained that represent
external state, mapping a single instance to the external reality, where
allowing multiple instances to be mapped to the same external resource
would create unnecessary difficulty maintaining synchronization among
instances. In these cases, a common idiom is to support a cache of
instances; a factory function is used to return either a new or existing
instance.

The difficulty in this approach is that one of two things must be
tolerated: either the cache grows without bound, or there needs to be
explicit management of the cache elsewhere in the application. The later
can be very tedious and leads to more code than is really necessary to
solve the problem at hand, and the former can be unacceptable for
long-running processes or even relatively short processes with
substantial memory requirements.

-   External objects that need to be represented by a single instance,
    no matter how many internal users there are. This can be useful for
    representing files that need to be written back to disk in whole
    rather than locked & modified for every use.
-   Objects that are expensive to create, but may be needed by multiple
    internal consumers. Similar to the first case, but not necessarily
    bound to external resources, and possibly not an issue for shared
    state. Weak references are only useful in this case if there is some
    flavor of "soft" references or if there is a high likelihood that
    users of individual objects will overlap in lifespan.

Circular references

-   DOMs require a huge amount of circular (to parent & document nodes)
    references, but these could be eliminated using a weak dictionary
    mapping from each node to its parent. This might be especially
    useful in the context of something like xml.dom.pulldom, allowing
    the .unlink() operation to become a no-op.

This proposal is divided into the following sections:

-   Proposed Solution
-   Implementation Strategy
-   Possible Applications
-   Previous Weak Reference Work in Python
-   Weak References in Java

The full text of one early proposal is included as an appendix since it
does not appear to be available on the net.

Aspects of the Solution Space

There are two distinct aspects to the weak references problem:

-   Invalidation of weak references
-   Presentation of weak references to Python code

Invalidation

Past approaches to weak reference invalidation have often hinged on
storing a strong reference and being able to examine all the instances
of weak reference objects, and invalidating them when the reference
count of their referent goes to one (indicating that the reference
stored by the weak reference is the last remaining reference). This has
the advantage that the memory management machinery in Python need not
change, and that any type can be weakly referenced.

The disadvantage of this approach to invalidation is that it assumes
that the management of the weak references is called sufficiently
frequently that weakly-referenced objects are noticed within a
reasonably short time frame; since this means a scan over some data
structure to invalidate references, an operation which is O(N) on the
number of weakly referenced objects, this is not effectively amortized
for any single object which is weakly referenced. This also assumes that
the application is calling into code which handles weakly-referenced
objects with some frequency, which makes weak-references less attractive
for library code.

An alternate approach to invalidation is that the de-allocation code to
be aware of the possibility of weak references and make a specific call
into the weak-reference management code to all invalidation whenever an
object is deallocated. This requires a change in the tp_dealloc handler
for weakly-referencable objects; an additional call is needed at the
"top" of the handler for objects which support weak-referencing, and an
efficient way to map from an object to a chain of weak references for
that object is needed as well.

Presentation

Two ways that weak references are presented to the Python layer have
been as explicit reference objects upon which some operation is required
in order to retrieve a usable reference to the underlying object, and
proxy objects which masquerade as the original objects as much as
possible.

Reference objects are easy to work with when some additional layer of
object management is being added in Python; references can be checked
for liveness explicitly, without having to invoke operations on the
referents and catching some special exception raised when an invalid
weak reference is used.

However, a number of users favor the proxy approach simply because the
weak reference looks so much like the original object.

Proposed Solution

Weak references should be able to point to any Python object that may
have substantial memory size (directly or indirectly), or hold
references to external resources (database connections, open files,
etc.).

A new module, weakref, will contain new functions used to create weak
references. weakref.ref() will create a "weak reference object" and
optionally attach a callback which will be called when the object is
about to be finalized. weakref.mapping() will create a "weak
dictionary". A third function, weakref.proxy(), will create a proxy
object that behaves somewhat like the original object.

A weak reference object will allow access to the referenced object if it
hasn't been collected and to determine if the object still exists in
memory. Retrieving the referent is done by calling the reference object.
If the referent is no longer alive, this will return None instead.

A weak dictionary maps arbitrary keys to values, but does not own a
reference to the values. When the values are finalized, the (key, value)
pairs for which it is a value are removed from all the mappings
containing such pairs. Like dictionaries, weak dictionaries are not
hashable.

Proxy objects are weak references that attempt to behave like the object
they proxy, as much as they can. Regardless of the underlying type,
proxies are not hashable since their ability to act as a weak reference
relies on a fundamental mutability that will cause failures when used as
dictionary keys -- even if the proper hash value is computed before the
referent dies, the resulting proxy cannot be used as a dictionary key
since it cannot be compared once the referent has expired, and
comparability is necessary for dictionary keys. Operations on proxy
objects after the referent dies cause weakref.ReferenceError to be
raised in most cases. "is" comparisons, type(), and id() will continue
to work, but always refer to the proxy and not the referent.

The callbacks registered with weak references must accept a single
parameter, which will be the weak reference or proxy object itself. The
object cannot be accessed or resurrected in the callback.

Implementation Strategy

The implementation of weak references will include a list of reference
containers that must be cleared for each weakly-referencable object. If
the reference is from a weak dictionary, the dictionary entry is cleared
first. Then, any associated callback is called with the object passed as
a parameter. Once all callbacks have been called, the object is
finalized and deallocated.

Many built-in types will participate in the weak-reference management,
and any extension type can elect to do so. The type structure will
contain an additional field which provides an offset into the instance
structure which contains a list of weak reference structures. If the
value of the field is <= 0, the object does not participate. In this
case, weakref.ref(), <weakdict>.__setitem__() and .setdefault(), and
item assignment will raise TypeError. If the value of the field is > 0,
a new weak reference can be generated and added to the list.

This approach is taken to allow arbitrary extension types to
participate, without taking a memory hit for numbers or other small
types.

Standard types which support weak references include instances,
functions, and bound & unbound methods. With the addition of class types
("new-style classes") in Python 2.2, types grew support for weak
references. Instances of class types are weakly referencable if they
have a base type which is weakly referencable, the class not specify
__slots__, or a slot is named __weakref__. Generators also support weak
references.

Possible Applications

PyGTK+ bindings?

Tkinter -- could avoid circular references by using weak references from
widgets to their parents. Objects won't be discarded any sooner in the
typical case, but there won't be so much dependence on the programmer
calling .destroy() before releasing a reference. This would mostly
benefit long-running applications.

DOM trees.

Previous Weak Reference Work in Python

Dianne Hackborn has proposed something called "virtual references".
'vref' objects are very similar to java.lang.ref.WeakReference objects,
except there is no equivalent to the invalidation queues. Implementing a
"weak dictionary" would be just as difficult as using only weak
references (without the invalidation queue) in Java. Information on this
has disappeared from the Web, but is included below as an Appendix.

Marc-André Lemburg's mx.Proxy package:

  http://www.lemburg.com/files/python/mxProxy.html

The weakdict module by Dieter Maurer is implemented in C and Python. It
appears that the Web pages have not been updated since Python 1.5.2a, so
I'm not yet sure if the implementation is compatible with Python 2.0.

  http://www.handshake.de/~dieter/weakdict.html

PyWeakReference by Alex Shindich:

  http://sourceforge.net/projects/pyweakreference/

Eric Tiedemann has a weak dictionary implementation:

  http://www.hyperreal.org/~est/python/weak/

Weak References in Java

http://java.sun.com/j2se/1.3/docs/api/java/lang/ref/package-summary.html

Java provides three forms of weak references, and one interesting helper
class. The three forms are called "weak", "soft", and "phantom"
references. The relevant classes are defined in the java.lang.ref
package.

For each of the reference types, there is an option to add the reference
to a queue when it is invalidated by the memory allocator. The primary
purpose of this facility seems to be that it allows larger structures to
be composed to incorporate weak-reference semantics without having to
impose substantial additional locking requirements. For instance, it
would not be difficult to use this facility to create a "weak" hash
table which removes keys and referents when a reference is no longer
used elsewhere. Using weak references for the objects without some sort
of notification queue for invalidations leads to much more tedious
implementation of the various operations required on hash tables. This
can be a performance bottleneck if deallocations of the stored objects
are infrequent.

Java's "weak" references are most like Dianne Hackborn's old vref
proposal: a reference object refers to a single Python object, but does
not own a reference to that object. When that object is deallocated, the
reference object is invalidated. Users of the reference object can
easily determine that the reference has been invalidated, or a
NullObjectDereferenceError can be raised when an attempt is made to use
the referred-to object.

The "soft" references are similar, but are not invalidated as soon as
all other references to the referred-to object have been released. The
"soft" reference does own a reference, but allows the memory allocator
to free the referent if the memory is needed elsewhere. It is not clear
whether this means soft references are released before the malloc()
implementation calls sbrk() or its equivalent, or if soft references are
only cleared when malloc() returns NULL.

"Phantom" references are a little different; unlike weak and soft
references, the referent is not cleared when the reference is added to
its queue. When all phantom references for an object are dequeued, the
object is cleared. This can be used to keep an object alive until some
additional cleanup is performed which needs to happen before the objects
.finalize() method is called.

Unlike the other two reference types, "phantom" references must be
associated with an invalidation queue.

Appendix -- Dianne Hackborn's vref proposal (1995)

[This has been indented and paragraphs reflowed, but there have be no
content changes. --Fred]

Proposal: Virtual References

In an attempt to partly address the recurring discussion concerning
reference counting vs. garbage collection, I would like to propose an
extension to Python which should help in the creation of "well
structured" cyclic graphs. In particular, it should allow at least trees
with parent back-pointers and doubly-linked lists to be created without
worry about cycles.

The basic mechanism I'd like to propose is that of a "virtual
reference," or a "vref" from here on out. A vref is essentially a handle
on an object that does not increment the object's reference count. This
means that holding a vref on an object will not keep the object from
being destroyed. This would allow the Python programmer, for example, to
create the aforementioned tree structure, which is automatically
destroyed when it is no longer in use -- by making all of the parent
back-references into vrefs, they no longer create reference cycles which
keep the tree from being destroyed.

In order to implement this mechanism, the Python core must ensure that
no -real- pointers are ever left referencing objects that no longer
exist. The implementation I would like to propose involves two basic
additions to the current Python system:

1.  A new "vref" type, through which the Python programmer creates and
    manipulates virtual references. Internally, it is basically a
    C-level Python object with a pointer to the Python object it is a
    reference to. Unlike all other Python code, however, it does not
    change the reference count of this object. In addition, it includes
    two pointers to implement a doubly-linked list, which is used below.
2.  The addition of a new field to the basic Python object
    [PyObject_Head in object.h], which is either NULL, or points to the
    head of a list of all vref objects that reference it. When a vref
    object attaches itself to another object, it adds itself to this
    linked list. Then, if an object with any vrefs on it is deallocated,
    it may walk this list and ensure that all of the vrefs on it point
    to some safe value, e.g. Nothing.

This implementation should hopefully have a minimal impact on the
current Python core -- when no vrefs exist, it should only add one
pointer to all objects, and a check for a NULL pointer every time an
object is deallocated.

Back at the Python language level, I have considered two possible
semantics for the vref object --

Pointer semantics

In this model, a vref behaves essentially like a Python-level pointer;
the Python program must explicitly dereference the vref to manipulate
the actual object it references.

An example vref module using this model could include the function
"new"; When used as 'MyVref = vref.new(MyObject)', it returns a new vref
object such that MyVref.object == MyObject. MyVref.object would then
change to Nothing if MyObject is ever deallocated.

For a concrete example, we may introduce some new C-style syntax:

-   & -- unary operator, creates a vref on an object, same as
    vref.new().
-   * -- unary operator, dereference a vref, same as VrefObject.object.

We can then define:

    1.     type(&MyObject) == vref.VrefType
    2.        *(&MyObject) == MyObject
    3. (*(&MyObject)).attr == MyObject.attr
    4.          &&MyObject == Nothing
    5.           *MyObject -> exception

Rule #4 is subtle, but comes about because we have made a vref to (a
vref with no real references). Thus the outer vref is cleared to Nothing
when the inner one inevitably disappears.

Proxy semantics

In this model, the Python programmer manipulates vref objects just as if
she were manipulating the object it is a reference of. This is
accomplished by implementing the vref so that all operations on it are
redirected to its referenced object. With this model, the dereference
operator (*) no longer makes sense; instead, we have only the reference
operator (&), and define:

    1.  type(&MyObject) == type(MyObject)
    2.        &MyObject == MyObject
    3. (&MyObject).attr == MyObject.attr
    4.       &&MyObject == MyObject

Again, rule #4 is important -- here, the outer vref is in fact a
reference to the original object, and -not- the inner vref. This is
because all operations applied to a vref actually apply to its object,
so that creating a vref of a vref actually results in creating a vref of
the latter's object.

The first, pointer semantics, has the advantage that it would be very
easy to implement; the vref type is extremely simple, requiring at
minimum a single attribute, object, and a function to create a
reference.

However, I really like the proxy semantics. Not only does it put less of
a burden on the Python programmer, but it allows you to do nice things
like use a vref anywhere you would use the actual object. Unfortunately,
it would probably an extreme pain, if not practically impossible, to
implement in the current Python implementation. I do have some thoughts,
though, on how to do this, if it seems interesting; one possibility is
to introduce new type-checking functions which handle the vref. This
would hopefully older C modules which don't expect vrefs to simply
return a type error, until they can be fixed.

Finally, there are some other additional capabilities that this system
could provide. One that seems particularly interesting to me involves
allowing the Python programmer to add "destructor" function to a vref --
this Python function would be called immediately prior to the referenced
object being deallocated, allowing a Python program to invisibly attach
itself to another object and watch for it to disappear. This seems neat,
though I haven't actually come up with any practical uses for it, yet...
:)

-- Dianne

Copyright

This document has been placed in the public domain.

authors:
Fred L. Drake
Jr.


text:
PEP: 217 Title: Display Hook for Interactive Use Author: Moshe Zadka
<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 31-Jul-2000 Python-Version: 2.1
Post-History:

Abstract

Python's interactive mode is one of the implementation's great strengths
-- being able to write expressions on the command line and get back a
meaningful output. However, the output function cannot be all things to
all people, and the current output function too often falls short of
this goal. This PEP describes a way to provides alternatives to the
built-in display function in Python, so users will have control over the
output from the interactive interpreter.

Interface

The current Python solution has worked for many users, and this should
not break it. Therefore, in the default configuration, nothing will
change in the REPL loop. To change the way the interpreter prints
interactively entered expressions, users will have to rebind
sys.displayhook to a callable object. The result of calling this object
with the result of the interactively entered expression should be
print-able, and this is what will be printed on sys.stdout.

Solution

The bytecode PRINT_EXPR will call sys.displayhook(POP()). A
displayhook() will be added to the sys builtin module, which is
equivalent to:

    import __builtin__
    def displayhook(o):
        if o is None:
            return
        __builtin__._ = None
        print `o`
        __builtin__._ = o

Jython Issues

The method Py.printResult will be similarly changed.


text:
PEP: 3109 Title: Raising Exceptions in Python 3000 Version: $Revision$
Last-Modified: $Date$ Author: Collin Winter<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
19-Jan-2006 Python-Version: 3.0 Post-History:

Abstract

This PEP introduces changes to Python's mechanisms for raising
exceptions intended to reduce both line noise and the size of the
language.

Rationale

One of Python's guiding maxims is "there should be one -- and
preferably only one -- obvious way to do it" <20>. Python 2.x's raise
statement violates this principle, permitting multiple ways of
expressing the same thought. For example, these statements are
equivalent: :

    raise E, V

    raise E(V)

There is a third form of the raise statement, allowing arbitrary
tracebacks to be attached to an exception[1]: :

    raise E, V, T

where T is a traceback. As specified in PEP 344, exception objects in
Python 3.x will possess a __traceback__ attribute, admitting this
translation of the three-expression raise statement: :

    raise E, V, T

is translated to :

    e = E(V)
    e.__traceback__ = T
    raise e

Using these translations, we can reduce the raise statement from four
forms to two:

1.  raise (with no arguments) is used to re-raise the active exception
    in an except suite.

2.  raise EXCEPTION is used to raise a new exception. This form has two
    sub-variants: EXCEPTION may be an exception class or an instance of
    an exception class; valid exception classes are BaseException and
    its subclasses (PEP 352). If EXCEPTION is a subclass, it will be
    called with no arguments to obtain an exception instance.

    To raise anything else is an error.

There is a further, more tangible benefit to be obtained through this
consolidation, as noted by A.M. Kuchling[2]. :

    PEP 8 doesn't express any preference between the
    two forms of raise statements:
    raise ValueError, 'blah'
    raise ValueError("blah")

    I like the second form better, because if the exception arguments
    are long or include string formatting, you don't need to use line
    continuation characters because of the containing parens.

The BDFL has concurred[3] and endorsed the consolidation of the several
raise forms.

Grammar Changes

In Python 3, the grammar for raise statements will change from[4] :

    raise_stmt: 'raise' [test [',' test [',' test]]]

to :

    raise_stmt: 'raise' [test]

Changes to Builtin Types

Because of its relation to exception raising, the signature for the
throw() method on generator objects will change, dropping the optional
second and third parameters. The signature thus changes (PEP 342) from :

    generator.throw(E, [V, [T]])

to :

    generator.throw(EXCEPTION)

Where EXCEPTION is either a subclass of BaseException or an instance of
a subclass of BaseException.

Semantic Changes

In Python 2, the following raise statement is legal :

    raise ((E1, (E2, E3)), E4), V

The interpreter will take the tuple's first element as the exception
type (recursively), making the above fully equivalent to :

    raise E1, V

As of Python 3.0, support for raising tuples like this will be dropped.
This change will bring raise statements into line with the throw()
method on generator objects, which already disallows this.

Compatibility Issues

All two- and three-expression raise statements will require
modification, as will all two- and three-expression throw() calls on
generators. Fortunately, the translation from Python 2.x to Python 3.x
in this case is simple and can be handled mechanically by Guido van
Rossum's 2to3 utility[5] using the raise and throw fixers ([6],[7]).

The following translations will be performed:

1.  Zero- and one-expression raise statements will be left intact.

2.  Two-expression raise statements will be converted from :

        raise E, V

    to :

        raise E(V)

    Two-expression throw() calls will be converted from :

        generator.throw(E, V)

    to :

        generator.throw(E(V))

    See point #5 for a caveat to this transformation.

3.  Three-expression raise statements will be converted from :

        raise E, V, T

    to :

        e = E(V)
        e.__traceback__ = T
        raise e

    Three-expression throw() calls will be converted from :

        generator.throw(E, V, T)

    to :

        e = E(V)
        e.__traceback__ = T
        generator.throw(e)

    See point #5 for a caveat to this transformation.

4.  Two- and three-expression raise statements where E is a tuple
    literal can be converted automatically using 2to3's raise fixer.
    raise statements where E is a non-literal tuple, e.g., the result of
    a function call, will need to be converted manually.

5.  Two- and three-expression raise statements where E is an exception
    class and V is an exception instance will need special attention.
    These cases break down into two camps:

    1.  raise E, V as a long-hand version of the zero-argument raise
        statement. As an example, assuming F is a subclass of E :

            try:
                something()
            except F as V:
                raise F(V)
            except E as V:
                handle(V)

        This would be better expressed as :

            try:
                something()
            except F:
                raise
            except E as V:
                handle(V)

    2.  raise E, V as a way of "casting" an exception to another class.
        Taking an example from distutils.compiler.unixcompiler :

            try:
                self.spawn(pp_args)
            except DistutilsExecError as msg:
                raise CompileError(msg)

        This would be better expressed as :

            try:
                self.spawn(pp_args)
            except DistutilsExecError as msg:
                raise CompileError from msg

        Using the raise ... from ... syntax introduced in PEP 344.

Implementation

This PEP was implemented in revision 57783[8].

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] http://docs.python.org/reference/simple_stmts.html#raise

[2] https://mail.python.org/pipermail/python-dev/2005-August/055187.html

[3] https://mail.python.org/pipermail/python-dev/2005-August/055190.html

[4] http://docs.python.org/reference/simple_stmts.html#raise

[5] http://svn.python.org/view/sandbox/trunk/2to3/

[6] http://svn.python.org/view/sandbox/trunk/2to3/fixes/fix_raise.py

[7] http://svn.python.org/view/sandbox/trunk/2to3/fixes/fix_throw.py

[8] http://svn.python.org/view/python/branches/py3k/Include/?rev=57783&view=rev


text:
PEP: 514 Title: Python registration in the Windows registry Version:
$Revision$ Last-Modified: $Date$ Author: Steve Dower
<EMAIL_ADDRESS>BDFL-Delegate: Paul Moore<EMAIL_ADDRESS>Status: Active Type: Informational Content-Type: text/x-rst Created:
02-Feb-2016 Post-History: 02-Feb-2016, 01-Mar-2016, 18-Jul-2016
Resolution:
https://mail.python.org/pipermail/python-dev/2016-July/145697.html

Abstract

This PEP defines a schema for the Python registry key to allow
third-party installers to register their installation, and to allow
tools and applications to detect and correctly display all Python
environments on a user's machine. No implementation changes to Python
are proposed with this PEP.

Python environments are not required to be registered unless they want
to be automatically discoverable by external tools. As this relates to
Windows only, these tools are expected to be predominantly GUI
applications. However, console applications may also make use of the
registered information. This PEP covers the information that may be made
available, but the actual presentation and use of this information is
left to the tool designers.

The schema matches the registry values that have been used by the
official installer since at least Python 2.5, and the resolution
behaviour matches the behaviour of the official Python releases. Some
backwards compatibility rules are provided to ensure tools can correctly
detect versions of CPython that do not register full information.

Motivation

When installed on Windows, the official Python installer creates a
registry key for discovery and detection by other applications. This
allows tools such as installers or IDEs to automatically detect and
display a user's Python installations. For example, the PEP 397 py.exe
launcher and editors such as PyCharm and Visual Studio already make use
of this information.

Third-party installers, such as those used by distributions, typically
create identical keys for the same purpose. Most tools that use the
registry to detect Python installations only inspect the keys used by
the official installer. As a result, third-party installations that wish
to be discoverable will overwrite these values, often causing users to
"lose" their original Python installation.

By describing a layout for registry keys that allows third-party
installations to register themselves uniquely, as well as providing tool
developers guidance for discovering all available Python installations,
these collisions should be prevented. We also take the opportunity to
add some well-known metadata so that more information can be presented
to users.

Definitions

A "registry key" is the equivalent of a file-system path into the
registry. Each key may contain "subkeys" (keys nested within keys) and
"values" (named and typed attributes attached to a key). These are used
on Windows to store settings in much the same way that directories
containing configuration files would work.

HKEY_CURRENT_USER is the root of settings for the currently logged-in
user, and this user can generally read and write all settings under this
root.

HKEY_LOCAL_MACHINE is the root of settings for all users. Generally, any
user can read these settings but only administrators can modify them. It
is typical for values under HKEY_CURRENT_USER to take precedence over
those in HKEY_LOCAL_MACHINE.

On 64-bit Windows, HKEY_LOCAL_MACHINE\Software\Wow6432Node is a special
key that 32-bit processes transparently read and write to rather than
accessing the Software key directly.

Further documentation regarding registry redirection on Windows is
available from the MSDN Library[1].

Structure

We consider there to be a single collection of Python environments on a
machine, where the collection may be different for each user of the
machine. There are three potential registry locations where the
collection may be stored based on the installation options of each
environment:

    HKEY_CURRENT_USER\Software\Python\<Company>\<Tag>
    HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag>
    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag>

Official Python releases use PythonCore for Company, and the value of
sys.winver for Tag. The Company PyLauncher is reserved. Other registered
environments may use any values for Company and Tag. Recommendations are
made later in this document.

Company-Tag pairs are case-insensitive, and uniquely identify each
environment. Depending on the purpose and intended use of a tool, there
are two suggested approaches for resolving conflicts between Company-Tag
pairs.

Tools that list every installed environment may choose to include those
even where the Company-Tag pairs match. They should ensure users can
easily identify whether the registration was per-user or per-machine,
and which registration has the higher priority.

Tools that aim to select a single installed environment from all
registered environments based on the Company-Tag pair, such as the
py.exe launcher, should always select the environment registered in
HKEY_CURRENT_USER when than the matching one in HKEY_LOCAL_MACHINE.

Conflicts between HKEY_LOCAL_MACHINE\Software\Python and
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python should only occur when
both 64-bit and 32-bit versions of an interpreter have the same Tag. In
this case, the tool should select whichever is more appropriate for its
use.

If a tool is able to determine from the provided information (or lack
thereof) that it cannot use a registered environment, there is no
obligation to present it to users.

Except as discussed in the section on backwards compatibility, Company
and Tag values are considered opaque to tools, and no information about
the interpreter should be inferred from the text. However, some tools
may display the Company and Tag values to users, so ideally the Tag will
be able to help users identify the associated environment.

Python environments are not required to register themselves unless they
want to be automatically discoverable by external tools.

Backwards Compatibility

Python 3.4 and earlier did not distinguish between 32-bit and 64-bit
builds in sys.winver. As a result, it is not possible to have valid
side-by-side installations of both 32-bit and 64-bit interpreters under
this scheme since it would result in duplicate Tags.

To ensure backwards compatibility, applications should treat
environments listed under the following two registry keys as distinct,
even when the Tag matches:

    HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>

Environments listed under HKEY_CURRENT_USER may be treated as distinct
from both of the above keys, potentially resulting in three environments
discovered using the same Tag. Alternatively, a tool may determine
whether the per-user environment is 64-bit or 32-bit and give it
priority over the per-machine environment, resulting in a maximum of two
discovered environments.

It is not possible to detect side-by-side installations of both 64-bit
and 32-bit versions of Python prior to 3.5 when they have been installed
for the current user. Python 3.5 and later always uses different Tags
for 64-bit and 32-bit versions.

The following section describe user-visible information that may be
registered. For Python 3.5 and earlier, none of this information is
available, but alternative defaults are specified for the PythonCore
key.

Environments registered under other Company names have no backward
compatibility requirements and must use distinct Tags to support
side-by-side installations. Tools consuming these registrations are not
required to disambiguate tags other than by preferring the user's
setting.

Company

The Company part of the key is intended to group related environments
and to ensure that Tags are namespaced appropriately. The key name
should be alphanumeric without spaces and likely to be unique. For
example, a trademarked name (preferred), a hostname, or as a last
resort, a UUID would be appropriate:

    HKEY_CURRENT_USER\Software\Python\ExampleCorp
    HKEY_CURRENT_USER\Software\Python\www.example.com
    HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60

The company name PyLauncher is reserved for the PEP 397 launcher
(py.exe). It does not follow this convention and should be ignored by
tools.

If a string value named DisplayName exists, it should be used to
identify the environment manufacturer/developer/distributor to users.
Otherwise, the name of the key should be used. (For PythonCore, the
default display name is "Python Software Foundation".)

If a string value named SupportUrl exists, it may be displayed or
otherwise used to direct users to a web site related to the environment.
(For PythonCore, the default support URL is "http://www.python.org/".)

A complete example may look like:

    HKEY_CURRENT_USER\Software\Python\ExampleCorp
        (Default) = (value not set)
        DisplayName = "Example Corp"
        SupportUrl = "http://www.example.com"

Tag

The Tag part of the key is intended to uniquely identify an environment
within those provided by a single company. The key name should be
alphanumeric without spaces and stable across installations. For
example, the Python language version, a UUID or a partial/complete hash
would be appropriate, while a Tag based on the install directory or some
aspect of the current machine may not. For example:

    HKEY_CURRENT_USER\Software\Python\ExampleCorp\examplepy
    HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6
    HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66

It is expected that some tools will require users to type the Tag into a
command line, and that the Company may be optional provided the Tag is
unique across all Python installations. Short, human-readable and easy
to type Tags are recommended, and if possible, select a value likely to
be unique across all other Companies.

If a string value named DisplayName exists, it should be used to
identify the environment to users. Otherwise, the name of the key should
be used. (For PythonCore, the default is "Python " followed by the Tag.)

If a string value named SupportUrl exists, it may be displayed or
otherwise used to direct users to a web site related to the environment.
(For PythonCore, the default is "http://www.python.org/".)

If a string value named Version exists, it should be used to identify
the version of the environment. This is independent from the version of
Python implemented by the environment. (For PythonCore, the default is
the first three characters of the Tag.)

If a string value named SysVersion exists, it must be in x.y or x.y.z
format matching the version returned by sys.version_info in the
interpreter. If omitted, the Python version is unknown. (For PythonCore,
the default is the first three characters of the Tag.)

If a string value named SysArchitecture exists, it must match the first
element of the tuple returned by platform.architecture(). Typically,
this will be "32bit" or "64bit". If omitted, the architecture is
unknown. (For PythonCore, the architecture is "32bit" when registered
under HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python or anywhere on a
32-bit operating system, "64bit" when registered under
HKEY_LOCAL_MACHINE\Software\Python on a 64-bit machine, and unknown when
registered under HKEY_CURRENT_USER.)

Note that each of these values is recommended, but optional. Omitting
SysVersion or SysArchitecture may prevent some tools from correctly
supporting the environment. A complete example may look like this:

    HKEY_CURRENT_USER\Software\Python\ExampleCorp\examplepy
        (Default) = (value not set)
        DisplayName = "Example Py Distro 3"
        SupportUrl = "http://www.example.com/distro-3"
        Version = "3.0.12345.0"
        SysVersion = "3.6.0"
        SysArchitecture = "64bit"

InstallPath

Beneath the environment key, an InstallPath key must be created. This
key is always named InstallPath, and the default value must match
sys.prefix:

    HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath
        (Default) = "C:\ExampleCorpPy36"

If a string value named ExecutablePath exists, it must be the full path
to the python.exe (or equivalent) executable. If omitted, the
environment is not executable. (For PythonCore, the default is the
python.exe file in the directory referenced by the (Default) value.)

If a string value named ExecutableArguments exists, tools should use the
value as the first arguments when executing ExecutablePath. Tools may
add other arguments following these, and will reasonably expect standard
Python command line options to be available.

If a string value named WindowedExecutablePath exists, it must be a path
to the pythonw.exe (or equivalent) executable. If omitted, the default
is the value of ExecutablePath, and if that is omitted the environment
is not executable. (For PythonCore, the default is the pythonw.exe file
in the directory referenced by the (Default) value.)

If a string value named WindowedExecutableArguments exists, tools should
use the value as the first arguments when executing
WindowedExecutablePath. Tools may add other arguments following these,
and will reasonably expect standard Python command line options to be
available.

A complete example may look like:

    HKEY_CURRENT_USER\Software\Python\ExampleCorp\examplepy\InstallPath
        (Default) = "C:\ExampleDistro30"
        ExecutablePath = "C:\ExampleDistro30\ex_python.exe"
        ExecutableArguments = "--arg1"
        WindowedExecutablePath = "C:\ExampleDistro30\ex_pythonw.exe"
        WindowedExecutableArguments = "--arg1"

Help

Beneath the environment key, a Help key may be created. This key is
always named Help if present and has no default value.

Each subkey of Help specifies a documentation file, tool, or URL
associated with the environment. The subkey may have any name, and the
default value is a string appropriate for passing to os.startfile or
equivalent.

If a string value named DisplayName exists, it should be used to
identify the help file to users. Otherwise, the key name should be used.

A complete example may look like:

    HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\Help
        Python\
            (Default) = "C:\ExampleDistro30\python36.chm"
            DisplayName = "Python Documentation"
        Extras\
            (Default) = "http://www.example.com/tutorial"
            DisplayName = "Example Distro Online Tutorial"

Other Keys

All other subkeys under a Company-Tag pair are available for private
use.

Official CPython releases have traditionally used certain keys in this
space to determine the location of the Python standard library and other
installed modules. This behaviour is retained primarily for backward
compatibility. However, as the code that reads these values is embedded
into the interpreter, third-party distributions may be affected by
values written into PythonCore if using an unmodified interpreter.

Sample Code

This sample code enumerates the registry and displays the available
Company-Tag pairs that could be used to launch an environment and the
target executable. It only shows the most-preferred target for the tag.
Backwards-compatible handling of PythonCore is omitted but shown in a
later example:

    # Display most-preferred environments.
    # Assumes a 64-bit operating system
    # Does not correctly handle PythonCore compatibility

    import winreg

    def enum_keys(key):
        i = 0
        while True:
            try:
                yield winreg.EnumKey(key, i)
            except OSError:
                break
            i += 1

    def get_value(key, value_name):
        try:
            return winreg.QueryValue(key, value_name)
        except FileNotFoundError:
            return None

    seen = set()
    for hive, key, flags in [
        (winreg.HKEY_CURRENT_USER, r'Software\Python', 0),
        (winreg.HKEY_LOCAL_MACHINE, r'Software\Python', winreg.KEY_WOW64_64KEY),
        (winreg.HKEY_LOCAL_MACHINE, r'Software\Python', winreg.KEY_WOW64_32KEY),
    ]:
        with winreg.OpenKeyEx(hive, key, access=winreg.KEY_READ | flags) as root_key:
            for company in enum_keys(root_key):
                if company == 'PyLauncher':
                    continue

                with winreg.OpenKey(root_key, company) as company_key:
                    for tag in enum_keys(company_key):
                        if (company, tag) in seen:
                            if company == 'PythonCore':
                                # TODO: Backwards compatibility handling
                                pass
                            continue
                        seen.add((company, tag))

                        try:
                            with winreg.OpenKey(company_key, tag + r'\InstallPath') as ip_key:
                                exec_path = get_value(ip_key, 'ExecutablePath')
                                exec_args = get_value(ip_key, 'ExecutableArguments')
                                if company == 'PythonCore' and not exec_path:
                                    # TODO: Backwards compatibility handling
                                    pass
                        except OSError:
                            exec_path, exec_args = None, None

                        if exec_path:
                            print('{}\\{} - {} {}'.format(company, tag, exec_path, exec_args or ''))
                        else:
                            print('{}\\{} - (not executable)'.format(company, tag))

This example only scans PythonCore entries for the current user. Where
data is missing, the defaults as described earlier in the PEP are
substituted. Note that these defaults are only for use under PythonCore;
other registrations do not have any default values:

    # Only lists per-user PythonCore registrations
    # Uses fallback values as described in PEP 514

    import os
    import winreg

    def enum_keys(key):
        i = 0
        while True:
            try:
                yield winreg.EnumKey(key, i)
            except OSError:
                break
            i += 1

    def get_value(key, value_name):
        try:
            return winreg.QueryValue(key, value_name)
        except FileNotFoundError:
            return None

    with winreg.OpenKey(winreg.HKEY_CURRENT_USER, r"Software\Python\PythonCore") as company_key:
        print('Company:', get_value(company_key, 'DisplayName') or 'Python Software Foundation')
        print('Support:', get_value(company_key, 'SupportUrl') or 'http://www.python.org/')
        print()

        for tag in enum_keys(company_key):
            with winreg.OpenKey(company_key, tag) as tag_key:
                print('PythonCore\\' + tag)
                print('Name:', get_value(tag_key, 'DisplayName') or ('Python ' + tag))
                print('Support:', get_value(tag_key, 'SupportUrl') or 'http://www.python.org/')
                print('Version:', get_value(tag_key, 'Version') or tag[:3])
                print('SysVersion:', get_value(tag_key, 'SysVersion') or tag[:3])
                # Architecture is unknown because we are in HKCU
                # Tools may use alternate approaches to determine architecture when
                # the registration does not specify it.
                print('SysArchitecture:', get_value(tag_key, 'SysArchitecture') or '(unknown)')

            try:
                ip_key = winreg.OpenKey(company_key, tag + '\\InstallPath')
            except FileNotFoundError:
                pass
            else:
                with ip_key:
                    ip = get_value(ip_key, None)
                    exe = get_value(ip_key, 'ExecutablePath') or os.path.join(ip, 'python.exe')
                    exew = get_value(ip_key, 'WindowedExecutablePath') or os.path.join(ip, 'python.exe')
                    print('InstallPath:', ip)
                    print('ExecutablePath:', exe)
                    print('WindowedExecutablePath:', exew)
            print()

This example shows a subset of the registration that will be created by
a just-for-me install of 64-bit Python 3.6.0. Other keys may also be
created:

    HKEY_CURRENT_USER\Software\Python\PythonCore
        (Default) = (value not set)
        DisplayName = "Python Software Foundation"
        SupportUrl = "http://www.python.org/"

    HKEY_CURRENT_USER\Software\Python\PythonCore\3.6
        (Default) = (value not set)
        DisplayName = "Python 3.6 (64-bit)"
        SupportUrl = "http://www.python.org/"
        Version = "3.6.0"
        SysVersion = "3.6"
        SysArchitecture = "64bit"

    HKEY_CURRENT_USER\Software\Python\PythonCore\3.6\Help\Main Python Documentation
        (Default) = "C:\Users\Me\AppData\Local\Programs\Python\Python36\Doc\python360.chm"
        DisplayName = "Python 3.6.0 Documentation"

    HKEY_CURRENT_USER\Software\Python\PythonCore\3.6\InstallPath
        (Default) = "C:\Users\Me\AppData\Local\Programs\Python\Python36\"
        ExecutablePath = "C:\Users\Me\AppData\Local\Programs\Python\Python36\python.exe"
        WindowedExecutablePath = "C:\Users\Me\AppData\Local\Programs\Python\Python36\pythonw.exe"

References

Copyright

This document has been placed in the public domain.

[1] Registry Redirector (Windows)
(https://msdn.microsoft.com/en-us/library/windows/desktop/aa384232.aspx)


text:
PEP: 417 Title: Including mock in the Standard Library Version:
$Revision$ Last-Modified: $Date$ Author: Michael Foord
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 12-Mar-2012 Python-Version: 3.3 Post-History:
12-Mar-2012 Resolution:
https://mail.python.org/pipermail/python-dev/2012-March/117507.html

Abstract

This PEP proposes adding the mock[1] testing library to the Python
standard library as unittest.mock.

Rationale

Creating mock objects for testing is a common need in Python. Many
developers create ad-hoc mocks, as needed, in their test suites. This is
currently what we do in the Python test suite, where a standardised mock
object library would be helpful.

There are many mock object libraries available for Python[2]. Of these,
mock is overwhelmingly the most popular, with as many downloads on PyPI
as the other mocking libraries combined.

An advantage of mock is that it is a mocking library and not a
framework. It provides a configurable and flexible mock object, without
being opinionated about how you write your tests. The mock api is now
well battle-tested and stable.

mock also handles safely monkeypatching and unmonkeypatching objects
during the scope of a test. This is hard to do safely and many
developers / projects mimic this functionality (often incorrectly). A
standardised way to do this, handling the complexity of patching in the
presence of the descriptor protocol (etc) is useful. People are asking
for a "patch"[3] feature to unittest. Doing this via mock.patch is
preferable to re-implementing part of this functionality in unittest.

Background

Addition of mock to the Python standard library was discussed and agreed
to at the Python Language Summit 2012.

Open Issues

As of release 0.8, which is current at the time of writing, mock is
compatible with Python 2.4-3.2. Moving into the Python standard library
will allow for the removal of some Python 2 specific "compatibility
hacks".

mock 0.8 introduced a new feature, "auto-speccing", obsoletes an older
mock feature called "mocksignature". The "mocksignature" functionality
can be removed from mock altogether prior to inclusion.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] mock library on PyPI

[2] http://pypi.python.org/pypi?%3Aaction=search&term=mock&submit=search

[3] http://bugs.python.org/issue11664