text:
PEP: 705 Title: TypedDict: Read-only items Author: Alice Purcell
<EMAIL_ADDRESS>Sponsor: Pablo Galindo<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867
Status: Final Type: Standards Track Topic: Typing Created: 07-Nov-2022
Python-Version: 3.13 Post-History: 30-Sep-2022, 02-Nov-2022,
14-Mar-2023, 17-Oct-2023, 04-Nov-2023, Resolution: 29-Feb-2024

typing:readonly and typing.ReadOnly

Abstract

PEP 589 defines the structural type ~typing.TypedDict for dictionaries
with a fixed set of keys. As TypedDict is a mutable type, it is
difficult to correctly annotate methods which accept read-only
parameters in a way that doesn't prevent valid inputs.

This PEP proposes a new type qualifier, typing.ReadOnly, to support
these usages. It makes no Python grammar changes. Correct usage of
read-only keys of TypedDicts is intended to be enforced only by static
type checkers, and will not be enforced by Python itself at runtime.

Motivation

Representing structured data using (potentially nested) dictionaries
with string keys is a common pattern in Python programs. PEP 589 allows
these values to be type checked when the exact type is known up-front,
but it is hard to write read-only code that accepts more specific
variants: for instance, where values may be subtypes or restrict a union
of possible types. This is an especially common issue when writing APIs
for services, which may support a wide range of input structures, and
typically do not need to modify their input.

Pure functions

Consider trying to add type hints to a function movie_string:

    def movie_string(movie: Movie) -> str:
        if movie.get("year") is None:
            return movie["name"]
        else:
            return f'{movie["name"]} ({movie["year"]})'

We could define this Movie type using a TypedDict:

    from typing import NotRequired, TypedDict

    class Movie(TypedDict):
        name: str
        year: NotRequired[int | None]

But suppose we have another type where year is required:

    class MovieRecord(TypedDict):
        name: str
        year: int

Attempting to pass a MovieRecord into movie_string results in the error
(using mypy):

    Argument 1 to "movie_string" has incompatible type "MovieRecord"; expected "Movie"

This particular use case should be type-safe, but the type checker
correctly stops the user from passing a MovieRecord into a Movie
parameter in the general case, because the Movie class has mutator
methods that could potentially allow the function to break the type
constraints in MovieRecord (e.g. with movie["year"] = None or
del movie["year"]). The problem disappears if we don't have mutator
methods in Movie. This could be achieved by defining an immutable
interface using a PEP 544 ~typing.Protocol:

    from typing import Literal, Protocol, overload

    class Movie(Protocol):
        @overload
        def get(self, key: Literal["name"]) -> str: ...

        @overload
        def get(self, key: Literal["year"]) -> int | None: ...

        @overload
        def __getitem__(self, key: Literal["name"]) -> str: ...

        @overload
        def __getitem__(self, key: Literal["year"]) -> int | None: ...

This is very repetitive, easy to get wrong, and is still missing
important method definitions like __contains__() and keys().

Updating nested dicts

The structural typing of TypedDict is supposed to permit writing update
functions that only constrain the types of items they modify:

    class HasTimestamp(TypedDict):
        timestamp: float

    class Logs(TypedDict):
        timestamp: float
        loglines: list[str]

    def update_timestamp(d: HasTimestamp) -> None:
        d["timestamp"] = now()

    def add_logline(logs: Logs, logline: str) -> None:
        logs["loglines"].append(logline)
        update_timestamp(logs)  # Accepted by type checker

However, this no longer works once you start nesting dictionaries:

    class HasTimestampedMetadata(TypedDict):
        metadata: HasTimestamp

    class UserAudit(TypedDict):
        name: str
        metadata: Logs

    def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
        d["metadata"]["timestamp"] = now()

    def rename_user(d: UserAudit, name: str) -> None:
        d["name"] = name
        update_metadata_timestamp(d)  # Type check error: "metadata" is not of type HasTimestamp

This looks like an error, but is simply due to the (unwanted) ability to
overwrite the metadata item held by the HasTimestampedMetadata instance
with a different HasTimestamp instance, that may no longer be a Logs
instance.

It is possible to work around this issue with generics (as of Python
3.11), but it is very complicated, requiring a type parameter for every
nested dict.

Rationale

These problems can be resolved by removing the ability to update one or
more of the items in a TypedDict. This does not mean the items are
immutable: a reference to the underlying dictionary could still exist
with a different but compatible type in which those items have mutator
operations. These items are "read-only", and we introduce a new
typing.ReadOnly type qualifier for this purpose.

The movie_string function in the first motivating example can then be
typed as follows:

    from typing import NotRequired, ReadOnly, TypedDict

    class Movie(TypedDict):
        name: ReadOnly[str]
        year: ReadOnly[NotRequired[int | None]]

    def movie_string(movie: Movie) -> str:
        if movie.get("year") is None:
            return movie["name"]
        else:
            return f'{movie["name"]} ({movie["year"]})'

A mixture of read-only and non-read-only items is permitted, allowing
the second motivating example to be correctly annotated:

    class HasTimestamp(TypedDict):
        timestamp: float

    class HasTimestampedMetadata(TypedDict):
        metadata: ReadOnly[HasTimestamp]

    def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
        d["metadata"]["timestamp"] = now()

    class Logs(HasTimestamp):
        loglines: list[str]

    class UserAudit(TypedDict):
        name: str
        metadata: Logs

    def rename_user(d: UserAudit, name: str) -> None:
        d["name"] = name
        update_metadata_timestamp(d)  # Now OK

In addition to these benefits, by flagging arguments of a function as
read-only (by using a TypedDict like Movie with read-only items), it
makes explicit not just to typecheckers but also to users that the
function is not going to modify its inputs, which is usually a desirable
property of a function interface.

This PEP proposes making ReadOnly valid only in a TypedDict. A possible
future extension would be to support it in additional contexts, such as
in protocols.

Specification

A new typing.ReadOnly type qualifier is added.

typing.ReadOnly type qualifier

The typing.ReadOnly type qualifier is used to indicate that an item
declared in a TypedDict definition may not be mutated (added, modified,
or removed):

    from typing import ReadOnly

    class Band(TypedDict):
        name: str
        members: ReadOnly[list[str]]

    blur: Band = {"name": "blur", "members": []}
    blur["name"] = "Blur"  # OK: "name" is not read-only
    blur["members"] = ["Damon Albarn"]  # Type check error: "members" is read-only
    blur["members"].append("Damon Albarn")  # OK: list is mutable

Alternative functional syntax

The alternative functional syntax <589#alternative-syntax> for TypedDict
also supports the new type qualifier:

    Band = TypedDict("Band", {"name": str, "members": ReadOnly[list[str]]})

Interaction with other special types

ReadOnly[] can be used with Required[], NotRequired[] and Annotated[],
in any nesting order:

    class Movie(TypedDict):
        title: ReadOnly[Required[str]]  # OK
        year: ReadOnly[NotRequired[Annotated[int, ValueRange(-9999, 9999)]]]  # OK

    class Movie(TypedDict):
        title: Required[ReadOnly[str]]  # OK
        year: Annotated[NotRequired[ReadOnly[int]], ValueRange(-9999, 9999)]  # OK

This is consistent with the behavior introduced in PEP 655.

Inheritance

Subclasses can redeclare read-only items as non-read-only, allowing them
to be mutated:

    class NamedDict(TypedDict):
        name: ReadOnly[str]

    class Album(NamedDict):
        name: str
        year: int

    album: Album = { "name": "Flood", "year": 1990 }
    album["year"] = 1973
    album["name"] = "Dark Side Of The Moon"  # OK: "name" is not read-only in Album

If a read-only item is not redeclared, it remains read-only:

    class Album(NamedDict):
        year: int

    album: Album = { "name": "Flood", "year": 1990 }
    album["name"] = "Dark Side Of The Moon"  # Type check error: "name" is read-only in Album

Subclasses can narrow value types of read-only items:

    class AlbumCollection(TypedDict):
        albums: ReadOnly[Collection[Album]]

    class RecordShop(AlbumCollection):
        name: str
        albums: ReadOnly[list[Album]]  # OK: "albums" is read-only in AlbumCollection

Subclasses can require items that are read-only but not required in the
superclass:

    class OptionalName(TypedDict):
        name: ReadOnly[NotRequired[str]]

    class RequiredName(OptionalName):
        name: ReadOnly[Required[str]]

    d: RequiredName = {}  # Type check error: "name" required

Subclasses can combine these rules:

    class OptionalIdent(TypedDict):
        ident: ReadOnly[NotRequired[str | int]]

    class User(OptionalIdent):
        ident: str  # Required, mutable, and not an int

Note that these are just consequences of structural typing, but they are
highlighted here as the behavior now differs from the rules specified in
PEP 589.

Type consistency

This section updates the type consistency rules introduced in PEP 589 to
cover the new feature in this PEP. In particular, any pair of types that
do not use the new feature will be consistent under these new rules if
(and only if) they were already consistent.

A TypedDict type A is consistent with TypedDict B if A is structurally
compatible with B. This is true if and only if all of the following are
satisfied:

-   For each item in B, A has the corresponding key, unless the item in
    B is read-only, not required, and of top value type
    (ReadOnly[NotRequired[object]]).
-   For each item in B, if A has the corresponding key, the
    corresponding value type in A is consistent with the value type in
    B.
-   For each non-read-only item in B, its value type is consistent with
    the corresponding value type in A.
-   For each required key in B, the corresponding key is required in A.
-   For each non-required key in B, if the item is not read-only in B,
    the corresponding key is not required in A.

Discussion:

-   All non-specified items in a TypedDict implicitly have value type
    ReadOnly[NotRequired[object]].

-   Read-only items behave covariantly, as they cannot be mutated. This
    is similar to container types such as Sequence, and different from
    non-read-only items, which behave invariantly. Example:

        class A(TypedDict):
            x: ReadOnly[int | None]

        class B(TypedDict):
            x: int

        def f(a: A) -> None:
            print(a["x"] or 0)

        b: B = {"x": 1}
        f(b)  # Accepted by type checker

-   A TypedDict type A with no explicit key 'x' is not consistent with a
    TypedDict type B with a non-required key 'x', since at runtime the
    key 'x' could be present and have an incompatible type (which may
    not be visible through A due to structural subtyping). The only
    exception to this rule is if the item in B is read-only, and the
    value type is of top type (object). For example:

        class A(TypedDict):
            x: int

        class B(TypedDict):
            x: int
            y: ReadOnly[NotRequired[object]]

        a: A = { "x": 1 }
        b: B = a  # Accepted by type checker

Update method

In addition to existing type checking rules, type checkers should error
if a TypedDict with a read-only item is updated with another TypedDict
that declares that key:

    class A(TypedDict):
        x: ReadOnly[int]
        y: int

    a1: A = { "x": 1, "y": 2 }
    a2: A = { "x": 3, "y": 4 }
    a1.update(a2)  # Type check error: "x" is read-only in A

Unless the declared value is of bottom type (~typing.Never):

    class B(TypedDict):
        x: NotRequired[typing.Never]
        y: ReadOnly[int]

    def update_a(a: A, b: B) -> None:
        a.update(b)  # Accepted by type checker: "x" cannot be set on b

Note: Nothing will ever match the Never type, so an item annotated with
it must be absent.

Keyword argument typing

PEP 692 introduced Unpack to annotate **kwargs with a TypedDict. Marking
one or more of the items of a TypedDict used in this way as read-only
will have no effect on the type signature of the method. However, it
will prevent the item from being modified in the body of the function:

    class Args(TypedDict):
        key1: int
        key2: str

    class ReadOnlyArgs(TypedDict):
        key1: ReadOnly[int]
        key2: ReadOnly[str]

    class Function(Protocol):
        def __call__(self, **kwargs: Unpack[Args]) -> None: ...

    def impl(**kwargs: Unpack[ReadOnlyArgs]) -> None:
        kwargs["key1"] = 3  # Type check error: key1 is readonly

    fn: Function = impl  # Accepted by type checker: function signatures are identical

Runtime behavior

TypedDict types will gain two new attributes, __readonly_keys__ and
__mutable_keys__, which will be frozensets containing all read-only and
non-read-only keys, respectively:

    class Example(TypedDict):
        a: int
        b: ReadOnly[int]
        c: int
        d: ReadOnly[int]

    assert Example.__readonly_keys__ == frozenset({'b', 'd'})
    assert Example.__mutable_keys__ == frozenset({'a', 'c'})

typing.get_type_hints will strip out any ReadOnly type qualifiers,
unless include_extras is True:

    assert get_type_hints(Example)['b'] == int
    assert get_type_hints(Example, include_extras=True)['b'] == ReadOnly[int]

typing.get_origin and typing.get_args will be updated to recognize
ReadOnly:

    assert get_origin(ReadOnly[int]) is ReadOnly
    assert get_args(ReadOnly[int]) == (int,)

Backwards compatibility

This PEP adds a new feature to TypedDict, so code that inspects
TypedDict types will have to change to support types using it. This is
expected to mainly affect type-checkers.

Security implications

There are no known security consequences arising from this PEP.

How to teach this

Suggested changes to the typing module documentation, in line with
current practice:

-   Add this PEP to the others listed.
-   Add typing.ReadOnly, linked to TypedDict and this PEP.
-   Add the following text to the TypedDict entry:

The ReadOnly type qualifier indicates that an item declared in a
TypedDict definition may be read but not mutated (added, modified or
removed). This is useful when the exact type of the value is not known
yet, and so modifying it would break structural subtypes. insert example

Reference implementation

pyright 1.1.333 fully implements this proposal.

Rejected alternatives

A TypedMapping protocol type

An earlier version of this PEP proposed a TypedMapping protocol type,
behaving much like a read-only TypedDict but without the constraint that
the runtime type be a dict. The behavior described in the current
version of this PEP could then be obtained by inheriting a TypedDict
from a TypedMapping. This has been set aside for now as more complex,
without a strong use-case motivating the additional complexity.

A higher-order ReadOnly type

A generalized higher-order type could be added that removes mutator
methods from its parameter, e.g. ReadOnly[MovieRecord]. For a TypedDict,
this would be like adding ReadOnly to every item, including those
declared in superclasses. This would naturally want to be defined for a
wider set of types than just TypedDict subclasses, and also raises
questions about whether and how it applies to nested types. We decided
to keep the scope of this PEP narrower.

Calling the type Readonly

Read-only is generally hyphenated, and it appears to be common
convention to put initial caps onto words separated by a dash when
converting to CamelCase. This appears consistent with the definition of
CamelCase on Wikipedia: CamelCase uppercases the first letter of each
word. That said, Python examples or counter-examples, ideally from the
core Python libraries, or better explicit guidance on the convention,
would be greatly appreciated.

Reusing the Final annotation

The ~typing.Final annotation prevents an attribute from being modified,
like the proposed ReadOnly qualifier does for TypedDict items. However,
it is also documented as preventing redefinition in subclasses too; from
PEP 591:

  The typing.Final type qualifier is used to indicate that a variable or
  attribute should not be reassigned, redefined, or overridden.

This does not fit with the intended use of ReadOnly. Rather than
introduce confusion by having Final behave differently in different
contexts, we chose to introduce a new qualifier.

A readonly flag

Earlier versions of this PEP introduced a boolean flag that would ensure
all items in a TypedDict were read-only:

    class Movie(TypedDict, readonly=True):
        name: str
        year: NotRequired[int | None]

    movie: Movie = { "name": "A Clockwork Orange" }
    movie["year"] = 1971  # Type check error: "year" is read-only

However, this led to confusion when inheritance was introduced:

    class A(TypedDict):
        key1: int

    class B(A, TypedDict, readonly=True):
        key2: int

    b: B = { "key1": 1, "key2": 2 }
    b["key1"] = 4  # Accepted by type checker: "key1" is not read-only

It would be reasonable for someone familiar with frozen (from
dataclasses), on seeing just the definition of B, to assume that the
whole type was read-only. On the other hand, it would be reasonable for
someone familiar with total to assume that read-only only applies to the
current type.

The original proposal attempted to eliminate this ambiguity by making it
both a type check and a runtime error to define B in this way. This was
still a source of surprise to people expecting it to work like total.

Given that no extra types could be expressed with the readonly flag, it
has been removed from the proposal to avoid ambiguity and surprise.

Supporting type-checked removal of read-only qualifier via copy and other methods

An earlier version of this PEP mandated that code like the following be
supported by type-checkers:

    class A(TypedDict):
        x: ReadOnly[int]

    class B(TypedDict):
        x: ReadOnly[str]

    class C(TypedDict):
        x: int | str

    def copy_and_modify(a: A) -> C:
        c: C = copy.copy(a)
        if not c['x']:
            c['x'] = "N/A"
        return c

    def merge_and_modify(a: A, b: B) -> C:
        c: C = a | b
        if not c['x']:
            c['x'] = "N/A"
        return c

However, there is currently no way to express this in the typeshed,
meaning type-checkers would be forced to special-case these functions.
There is already a way to code these operations that mypy and pyright do
support, though arguably this is less readable:

    copied: C = { **a }
    merged: C = { **a, **b }

While not as flexible as would be ideal, the current typeshed stubs are
sound, and remain so if this PEP is accepted. Updating the typeshed
would require new typing features, like a type constructor to express
the type resulting from merging two or more dicts, and a type qualifier
to indicate a returned value is not shared (so may have type constraints
like read-only and invariance of generics loosened in specific ways),
plus details of how type-checkers would be expected to interpret these
features. These could be valuable additions to the language, but are
outside the scope of this PEP.

Given this, we have deferred any update of the typeshed stubs.

Preventing unspecified keys in TypedDicts

Consider the following "type discrimination" code:

    class A(TypedDict):
      foo: int

    class B(TypedDict):
      bar: int

    def get_field(d: A | B) -> int:
      if "foo" in d:
        return d["foo"]  # !!!
      else:
        return d["bar"]

This is a common idiom, and other languages like Typescript allow it.
Technically, however, this code is unsound: B does not declare foo, but
instances of B may still have the key present, and the associated value
may be of any type:

    class C(TypedDict):
      foo: str
      bar: int

    c: C = { "foo": "hi", "bar" 3 }
    b: B = c  # OK: C is structurally compatible with B
    v = get_field(b)  # Returns a string at runtime, not an int!

mypy rejects the definition of get_field on the marked line with the
error TypedDict "B" has no key "foo", which is a rather confusing error
message, but is caused by this unsoundness.

One option for correcting this would be to explicitly prevent B from
holding a foo:

    class B(TypedDict):
      foo: NotRequired[Never]
      bar: int

    b: B = c  # Type check error: key "foo" not allowed in B

However, this requires every possible key that might be used to
discriminate on to be explicitly declared in every type, which is not
generally feasible. A better option would be to have a way of preventing
all unspecified keys from being included in B. mypy supports this using
the @final decorator from PEP 591:

    @final
    class B(TypedDict):
      bar: int

The reasoning here is that this prevents C or any other type from being
considered a "subclass" of B, so instances of B can now be relied on to
never hold the key foo, even though it is not explicitly declared to be
of bottom type.

With the introduction of read-only items, however, this reasoning would
imply type-checkers should ban the following:

    @final
    class D(TypedDict):
      field: ReadOnly[Collection[str]]

    @final
    class E(TypedDict):
      field: list[str]

    e: E = { "field": ["value1", "value2"] }
    d: D = e  # Error?

The conceptual problem here is that TypedDicts are structural types:
they cannot really be subclassed. As such, using @final on them is not
well-defined; it is certainly not mentioned in PEP 591.

An earlier version of this PEP proposed resolving this by adding a new
flag to TypedDict that would explicitly prevent other keys from being
used, but not other kinds of structural compatibility:

    class B(TypedDict, other_keys=Never):
      bar: int

    b: B = c  # Type check error: key "foo" not allowed in B

However, during the process of drafting, the situation changed:

-   pyright, which previously worked similarly to mypy in this type
    discrimination case, changed to allow the original example without
    error, despite the unsoundness, due to it being a common idiom
-   mypy has an open issue to follow the lead of pyright and Typescript
    and permit the idiom as well
-   a draft of PEP-728 was created that is a superset of the other_keys
    functionality

As such, there is less urgency to address this issue in this PEP, and it
has been deferred to PEP-728.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 231 Title: __findattr__() Author: Barry Warsaw<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
30-Nov-2000 Python-Version: 2.1 Post-History:

Introduction

This PEP describes an extension to instance attribute lookup and
modification machinery, which allows pure-Python implementations of many
interesting programming models. This PEP tracks the status and ownership
of this feature. It contains a description of the feature and outlines
changes necessary to support the feature. This PEP summarizes
discussions held in mailing list forums, and provides URLs for further
information, where appropriate. The CVS revision history of this file
contains the definitive historical record.

Background

The semantics for Python instances allow the programmer to customize
some aspects of attribute lookup and attribute modification, through the
special methods __getattr__() and __setattr__()[1].

However, because of certain restrictions imposed by these methods, there
are useful programming techniques that can not be written in Python
alone, e.g. strict Java Bean-like[2] interfaces and Zope style
acquisitions[3]. In the latter case, Zope solves this by including a C
extension called ExtensionClass[4] which modifies the standard class
semantics, and uses a metaclass hook in Python's class model called
alternatively the "Don Beaudry Hook" or "Don Beaudry Hack"[5].

While Zope's approach works, it has several disadvantages. First, it
requires a C extension. Second it employs a very arcane, but truck-sized
loophole in the Python machinery. Third, it can be difficult for other
programmers to use and understand (the metaclass has well-known brain
exploding properties). And fourth, because ExtensionClass instances
aren't "real" Python instances, some aspects of the Python runtime
system don't work with ExtensionClass instances.

Proposals for fixing this problem have often been lumped under the
rubric of fixing the "class/type dichotomy"; that is, eliminating the
difference between built-in types and classes[6]. While a laudable goal
itself, repairing this rift is not necessary in order to achieve the
types of programming constructs described above. This proposal provides
an 80% solution with a minimum of modification to Python's class and
instance objects. It does nothing to address the type/class dichotomy.

Proposal

This proposal adds a new special method called __findattr__() with the
following semantics:

-   If defined in a class, it will be called on all instance attribute
    resolutions instead of __getattr__() and __setattr__().
-   __findattr__() is never called recursively. That is, when a specific
    instance's __findattr__() is on the call stack, further attribute
    accesses for that instance will use the standard __getattr__() and
    __setattr__() methods.
-   __findattr__() is called for both attribute access ('getting') and
    attribute modification ('setting'). It is not called for attribute
    deletion.
-   When called for getting, it is passed a single argument (not
    counting 'self'): the name of the attribute being accessed.
-   When called for setting, it is called with third argument, which is
    the value to set the attribute to.
-   __findattr__() methods have the same caching semantics as
    __getattr__() and __setattr__(); i.e. if they are present in the
    class at class definition time, they are used, but if they are
    subsequently added to a class later they are not.

Key Differences with the Existing Protocol

__findattr__()'s semantics are different from the existing protocol in
key ways:

First, __getattr__() is never called if the attribute is found in the
instance's __dict__. This is done for efficiency reasons, and because
otherwise, __setattr__() would have no way to get to the instance's
attributes.

Second, __setattr__() cannot use "normal" syntax for setting instance
attributes, e.g. "self.name = foo" because that would cause recursive
calls to __setattr__().

__findattr__() is always called regardless of whether the attribute is
in __dict__ or not, and a flag in the instance object prevents recursive
calls to __findattr__(). This gives the class a chance to perform some
action for every attribute access. And because it is called for both
gets and sets, it is easy to write similar policy for all attribute
access. Further, efficiency is not a problem because it is only paid
when the extended mechanism is used.

Related Work

PEP 213 describes a different approach to hooking into attribute access
and modification. The semantics proposed in PEP 213 can be implemented
using the __findattr__() hook described here, with one caveat. The
current reference implementation of __findattr__() does not support
hooking on attribute deletion. This could be added if it's found
desirable. See example below.

Examples

One programming style that this proposal allows is a Java Bean-like
interface to objects, where unadorned attribute access and modification
is transparently mapped to a functional interface. E.g.

    class Bean:
        def __init__(self, x):
            self.__myfoo = x

        def __findattr__(self, name, *args):
            if name.startswith('_'):
                # Private names
                if args: setattr(self, name, args[0])
                else:    return getattr(self, name)
            else:
                # Public names
                if args: name = '_set_' + name
                else:    name = '_get_' + name
                return getattr(self, name)(*args)

        def _set_foo(self, x):
            self.__myfoo = x

        def _get_foo(self):
            return self.__myfoo

    b = Bean(3)
    print b.foo
    b.foo = 9
    print b.foo

A second, more elaborate example is the implementation of both implicit
and explicit acquisition in pure Python:

    import types

    class MethodWrapper:
        def __init__(self, container, method):
            self.__container = container
            self.__method = method

        def __call__(self, *args, **kws):
            return self.__method.im_func(self.__container, *args, **kws)

    class WrapperImplicit:
        def __init__(self, contained, container):
            self.__contained = contained
            self.__container = container

        def __repr__(self):
            return '<Wrapper: [%s | %s]>' % (self.__container,
                                             self.__contained)

        def __findattr__(self, name, *args):
            # Some things are our own
            if name.startswith('_WrapperImplicit__'):
                if args: return setattr(self, name, *args)
                else:    return getattr(self, name)
            # setattr stores the name on the contained object directly
            if args:
                return setattr(self.__contained, name, args[0])
            # Other special names
            if name == 'aq_parent':
                return self.__container
            elif name == 'aq_self':
                return self.__contained
            elif name == 'aq_base':
                base = self.__contained
                try:
                    while 1:
                        base = base.aq_self
                except AttributeError:
                    return base
            # no acquisition for _ names
            if name.startswith('_'):
                return getattr(self.__contained, name)
            # Everything else gets wrapped
            missing = []
            which = self.__contained
            obj = getattr(which, name, missing)
            if obj is missing:
                which = self.__container
                obj = getattr(which, name, missing)
                if obj is missing:
                    raise AttributeError, name
            of = getattr(obj, '__of__', missing)
            if of is not missing:
                return of(self)
            elif type(obj) == types.MethodType:
                return MethodWrapper(self, obj)
            return obj

    class WrapperExplicit:
        def __init__(self, contained, container):
            self.__contained = contained
            self.__container = container

        def __repr__(self):
            return '<Wrapper: [%s | %s]>' % (self.__container,
                                             self.__contained)

        def __findattr__(self, name, *args):
            # Some things are our own
            if name.startswith('_WrapperExplicit__'):
                if args: return setattr(self, name, *args)
                else:    return getattr(self, name)
            # setattr stores the name on the contained object directly
            if args:
                return setattr(self.__contained, name, args[0])
            # Other special names
            if name == 'aq_parent':
                return self.__container
            elif name == 'aq_self':
                return self.__contained
            elif name == 'aq_base':
                base = self.__contained
                try:
                    while 1:
                        base = base.aq_self
                except AttributeError:
                    return base
            elif name == 'aq_acquire':
                return self.aq_acquire
            # explicit acquisition only
            obj = getattr(self.__contained, name)
            if type(obj) == types.MethodType:
                return MethodWrapper(self, obj)
            return obj

        def aq_acquire(self, name):
            # Everything else gets wrapped
            missing = []
            which = self.__contained
            obj = getattr(which, name, missing)
            if obj is missing:
                which = self.__container
                obj = getattr(which, name, missing)
                if obj is missing:
                    raise AttributeError, name
            of = getattr(obj, '__of__', missing)
            if of is not missing:
                return of(self)
            elif type(obj) == types.MethodType:
                return MethodWrapper(self, obj)
            return obj

    class Implicit:
        def __of__(self, container):
            return WrapperImplicit(self, container)

        def __findattr__(self, name, *args):
            # ignore setattrs
            if args:
                return setattr(self, name, args[0])
            obj = getattr(self, name)
            missing = []
            of = getattr(obj, '__of__', missing)
            if of is not missing:
                return of(self)
            return obj

    class Explicit(Implicit):
        def __of__(self, container):
            return WrapperExplicit(self, container)

    # tests
    class C(Implicit):
        color = 'red'

    class A(Implicit):
        def report(self):
            return self.color

    # simple implicit acquisition
    c = C()
    a = A()
    c.a = a
    assert c.a.report() == 'red'

    d = C()
    d.color = 'green'
    d.a = a
    assert d.a.report() == 'green'

    try:
        a.report()
    except AttributeError:
        pass
    else:
        assert 0, 'AttributeError expected'

    # special names
    assert c.a.aq_parent is c
    assert c.a.aq_self is a

    c.a.d = d
    assert c.a.d.aq_base is d
    assert c.a is not a

    # no acquisition on _ names
    class E(Implicit):
        _color = 'purple'

    class F(Implicit):
        def report(self):
            return self._color

    e = E()
    f = F()
    e.f = f
    try:
        e.f.report()
    except AttributeError:
        pass
    else:
        assert 0, 'AttributeError expected'

    # explicit
    class G(Explicit):
        color = 'pink'

    class H(Explicit):
        def report(self):
            return self.aq_acquire('color')

        def barf(self):
            return self.color

    g = G()
    h = H()
    g.h = h
    assert g.h.report() == 'pink'

    i = G()
    i.color = 'cyan'
    i.h = h
    assert i.h.report() == 'cyan'

    try:
        g.i.barf()
    except AttributeError:
        pass
    else:
        assert 0, 'AttributeError expected'

C++-like access control can also be accomplished, although less cleanly
because of the difficulty of figuring out what method is being called
from the runtime call stack:

    import sys
    import types

    PUBLIC = 0
    PROTECTED = 1
    PRIVATE = 2

    try:
        getframe = sys._getframe
    except ImportError:
        def getframe(n):
            try: raise Exception
            except Exception:
                frame = sys.exc_info()[2].tb_frame
            while n > 0:
                frame = frame.f_back
                if frame is None:
                    raise ValueError, 'call stack is not deep enough'
            return frame

    class AccessViolation(Exception):
        pass

    class Access:
        def __findattr__(self, name, *args):
            methcache = self.__dict__.setdefault('__cache__', {})
            missing = []
            obj = getattr(self, name, missing)
            # if obj is missing we better be doing a setattr for
            # the first time
            if obj is not missing and type(obj) == types.MethodType:
                # Digusting hack because there's no way to
                # dynamically figure out what the method being
                # called is from the stack frame.
                methcache[obj.im_func.func_code] = obj.im_class
            #
            # What's the access permissions for this name?
            access, klass = getattr(self, '__access__', {}).get(
                name, (PUBLIC, 0))
            if access is not PUBLIC:
                # Now try to see which method is calling us
                frame = getframe(0).f_back
                if frame is None:
                    raise AccessViolation
                # Get the class of the method that's accessing
                # this attribute, by using the code object cache
                if frame.f_code.co_name == '__init__':
                    # There aren't entries in the cache for ctors,
                    # because the calling mechanism doesn't go
                    # through __findattr__().  Are there other
                    # methods that might have the same behavior?
                    # Since we can't know who's __init__ we're in,
                    # for now we'll assume that only protected and
                    # public attrs can be accessed.
                    if access is PRIVATE:
                        raise AccessViolation
                else:
                    methclass = self.__cache__.get(frame.f_code)
                    if not methclass:
                        raise AccessViolation
                    if access is PRIVATE and methclass is not klass:
                        raise AccessViolation
                    if access is PROTECTED and not issubclass(methclass,
                                                              klass):
                        raise AccessViolation
            # If we got here, it must be okay to access the attribute
            if args:
                return setattr(self, name, *args)
            return obj

    # tests
    class A(Access):
        def __init__(self, foo=0, name='A'):
            self._foo = foo
            # can't set private names in __init__
            self.__initprivate(name)

        def __initprivate(self, name):
            self._name = name

        def getfoo(self):
            return self._foo

        def setfoo(self, newfoo):
            self._foo = newfoo

        def getname(self):
            return self._name

    A.__access__ = {'_foo'      : (PROTECTED, A),
                    '_name'     : (PRIVATE, A),
                    '__dict__'  : (PRIVATE, A),
                    '__access__': (PRIVATE, A),
                    }

    class B(A):
        def setfoo(self, newfoo):
            self._foo = newfoo + 3

        def setname(self, name):
            self._name = name

    b = B(1)
    b.getfoo()

    a = A(1)
    assert a.getfoo() == 1
    a.setfoo(2)
    assert a.getfoo() == 2

    try:
        a._foo
    except AccessViolation:
        pass
    else:
        assert 0, 'AccessViolation expected'

    try:
        a._foo = 3
    except AccessViolation:
        pass
    else:
        assert 0, 'AccessViolation expected'

    try:
        a.__dict__['_foo']
    except AccessViolation:
        pass
    else:
        assert 0, 'AccessViolation expected'

    b = B()
    assert b.getfoo() == 0
    b.setfoo(2)
    assert b.getfoo() == 5
    try:
        b.setname('B')
    except AccessViolation:
        pass
    else:
        assert 0, 'AccessViolation expected'

    assert b.getname() == 'A'

Here's an implementation of the attribute hook described in PEP 213
(except that hooking on attribute deletion isn't supported by the
current reference implementation).

    class Pep213:
        def __findattr__(self, name, *args):
            hookname = '__attr_%s__' % name
            if args:
                op = 'set'
            else:
                op = 'get'
            # XXX: op = 'del' currently not supported
            missing = []
            meth = getattr(self, hookname, missing)
            if meth is missing:
                if op == 'set':
                    return setattr(self, name, *args)
                else:
                    return getattr(self, name)
            else:
                return meth(op, *args)

    def computation(i):
        print 'doing computation:', i
        return i + 3

    def rev_computation(i):
        print 'doing rev_computation:', i
        return i - 3

    class X(Pep213):
        def __init__(self, foo=0):
            self.__foo = foo

        def __attr_foo__(self, op, val=None):
            if op == 'get':
                return computation(self.__foo)
            elif op == 'set':
                self.__foo = rev_computation(val)
            # XXX: 'del' not yet supported

    x = X()
    fooval = x.foo
    print fooval
    x.foo = fooval + 5
    print x.foo
    # del x.foo

Reference Implementation

The reference implementation, as a patch to the Python core, can be
found at this URL:

http://sourceforge.net/patch/?func=detailpatch&patch_id=102613&group_id=5470

References

-   http://docs.python.org/howto/regex.html

Rejection

There are serious problems with the recursion-protection feature. As
described here it's not thread-safe, and a thread-safe solution has
other problems. In general, it's not clear how helpful the
recursion-protection feature is; it makes it hard to write code that
needs to be callable inside __findattr__ as well as outside it. But
without the recursion-protection, it's hard to implement __findattr__ at
all (since __findattr__ would invoke itself recursively for every
attribute it tries to access). There seems to be no good solution here.

It's also dubious how useful it is to support __findattr__ both for
getting and for setting attributes -- __setattr__ gets called in all
cases already.

The examples can all be implemented using __getattr__ if care is taken
not to store instance variables under their own names.

Copyright

This document has been placed in the Public Domain.

[1] http://docs.python.org/reference/datamodel.html#customizing-attribute-access

[2] http://www.javasoft.com/products/javabeans/

[3] http://www.digicool.com/releases/ExtensionClass/Acquisition.html

[4] http://www.digicool.com/releases/ExtensionClass

[5] http://www.python.org/doc/essays/metaclasses/

[6] http://www.foretec.com/python/workshops/1998-11/dd-ascher-sum.html

license: Public Domain


text:
PEP: 536 Title: Final Grammar for Literal String Interpolation Version:
$Revision$ Last-Modified: $Date$ Author: Philipp Angerer
<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 11-Dec-2016 Python-Version: 3.7
Post-History: 18-Aug-2016, 23-Dec-2016, 15-Mar-2019 Resolution:
https://discuss.python.org/t/pep-536-should-be-marked-as-rejected/35226/4

Abstract

PEP 498 introduced Literal String Interpolation (or “f-strings”). The
expression portions of those literals however are subject to certain
restrictions. This PEP proposes a formal grammar lifting those
restrictions, promoting “f-strings” to “f expressions” or f-literals.

This PEP expands upon the f-strings introduced by PEP 498, so this text
requires familiarity with PEP 498.

PEP Withdrawal

This PEP has been withdrawn in favour of PEP 701. PEP 701 addresses all
important points of this PEP.

Terminology

This text will refer to the existing grammar as “f-strings”, and the
proposed one as “f-literals”.

Furthermore, it will refer to the {}-delimited expressions in
f-literals/f-strings as “expression portions” and the static string
content around them as “string portions”.

Motivation

The current implementation of f-strings in CPython relies on the
existing string parsing machinery and a post processing of its tokens.
This results in several restrictions to the possible expressions usable
within f-strings:

1.  It is impossible to use the quote character delimiting the f-string
    within the expression portion:

        >>> f'Magic wand: { bag['wand'] }'
                                     ^
        SyntaxError: invalid syntax

2.  A previously considered way around it would lead to escape sequences
    in executed code and is prohibited in f-strings:

        >>> f'Magic wand { bag[\'wand\'] } string'
        SyntaxError: f-string expression portion cannot include a backslash

3.  Comments are forbidden even in multi-line f-strings:

        >>> f'''A complex trick: {
        ... bag['bag']  # recursive bags!
        ... }'''
        SyntaxError: f-string expression part cannot include '#'

4.  Expression portions need to wrap ':' and '!' in braces:

        >>> f'Useless use of lambdas: { lambda x: x*2 }'
        SyntaxError: unexpected EOF while parsing

These limitations serve no purpose from a language user perspective and
can be lifted by giving f-literals a regular grammar without exceptions
and implementing it using dedicated parse code.

Rationale

The restrictions mentioned in Motivation are non-obvious and
counter-intuitive unless the user is familiar with the f-literals’
implementation details.

As mentioned, a previous version of PEP 498 allowed escape sequences
anywhere in f-strings, including as ways to encode the braces delimiting
the expression portions and in their code. They would be expanded before
the code is parsed, which would have had several important
ramifications:

#. It would not be clear to human readers which portions are Expressions
and which are strings. Great material for an “obfuscated/underhanded
Python challenge” #. Syntax highlighters are good in parsing nested
grammar, but not in recognizing escape sequences. ECMAScript 2016
(JavaScript) allows escape sequences in its identifiers[1] and the
author knows of no syntax highlighter able to correctly highlight code
making use of this.

As a consequence, the expression portions would be harder to recognize
with and without the aid of syntax highlighting. With the new grammar,
it is easy to extend syntax highlighters to correctly parse and display
f-literals:

Highlighting expression portions with possible escape sequences would
mean to create a modified copy of all rules of the complete expression
grammar, accounting for the possibility of escape sequences in key
words, delimiters, and all other language syntax. One such duplication
would yield one level of escaping depth and have to be repeated for a
deeper escaping in a recursive f-literal. This is the case since no
highlighting engine known to the author supports expanding escape
sequences before applying rules to a certain context. Nesting contexts
however is a standard feature of all highlighting engines.

Familiarity also plays a role: Arbitrary nesting of expressions without
expansion of escape sequences is available in every single other
language employing a string interpolation method that uses expressions
instead of just variable names.[2]

Specification

PEP 498 specified f-strings as the following, but places restrictions on
it:

    f ' <text> { <expression> <optional !s, !r, or !a> <optional : format specifier> } <text> ... '

All restrictions mentioned in the PEP are lifted from f-literals, as
explained below:

1.  Expression portions may now contain strings delimited with the same
    kind of quote that is used to delimit the f-literal.
2.  Backslashes may now appear within expressions just like anywhere
    else in Python code. In case of strings nested within f-literals,
    escape sequences are expanded when the innermost string is
    evaluated.
3.  Comments, using the '#' character, are possible only in multi-line
    f-literals, since comments are terminated by the end of the line
    (which makes closing a single-line f-literal impossible).
4.  Expression portions may contain ':' or '!' wherever syntactically
    valid. The first ':' or '!' that is not part of an expression has to
    be followed a valid coercion or format specifier.

A remaining restriction not explicitly mentioned by PEP 498 is line
breaks in expression portions. Since strings delimited by single ' or "
characters are expected to be single line, line breaks remain illegal in
expression portions of single line strings.

Note

Is lifting of the restrictions sufficient, or should we specify a more
complete grammar?

Backwards Compatibility

f-literals are fully backwards compatible to f-strings, and expands the
syntax considered legal.

Reference Implementation

TBD

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] ECMAScript IdentifierName specification (
http://ecma-international.org/ecma-262/6.0/#sec-names-and-keywords )

Yes, const cthulhu = { H̹̙̦̮͉̩̗̗ͧ̇̏̊̾E̷̮̣̫̤̣ͨ͆͒̆ͮ̃C̨̛͔̦̟͈̻ͯ̂͐O̜͎͍͙͚̬̝̣̽ͮ͐͗̀ͤ̍̀͢M̴̡̲̭͍͇̼̟̯̦̉̒͠Ḛ̛̙̞̪̗ͥͤͩ̾͑̔͐ͅṮ̴̷̷̗̼͍̿̿̓̽͐H̙̙̔̄͜\u0042: 42 } is valid ECMAScript 2016

[2] Wikipedia article on string interpolation (
https://en.wikipedia.org/wiki/String_interpolation )


text:
PEP: 479 Title: Change StopIteration handling inside generators Version:
$Revision$ Last-Modified: $Date$ Author: Chris Angelico
<EMAIL_ADDRESS>Guido van Rossum<EMAIL_ADDRESS>Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014
Python-Version: 3.5 Post-History: 15-Nov-2014, 19-Nov-2014, 05-Dec-2014

Abstract

This PEP proposes a change to generators: when StopIteration is raised
inside a generator, it is replaced with RuntimeError. (More precisely,
this happens when the exception is about to bubble out of the
generator's stack frame.) Because the change is backwards incompatible,
the feature is initially introduced using a __future__ statement.

Acceptance

This PEP was accepted by the BDFL on November 22. Because of the
exceptionally short period from first draft to acceptance, the main
objections brought up after acceptance were carefully considered and
have been reflected in the "Alternate proposals" section below. However,
none of the discussion changed the BDFL's mind and the PEP's acceptance
is now final. (Suggestions for clarifying edits are still welcome --
unlike IETF RFCs, the text of a PEP is not cast in stone after its
acceptance, although the core design/plan/specification should not
change after acceptance.)

Rationale

The interaction of generators and StopIteration is currently somewhat
surprising, and can conceal obscure bugs. An unexpected exception should
not result in subtly altered behaviour, but should cause a noisy and
easily-debugged traceback. Currently, StopIteration raised accidentally
inside a generator function will be interpreted as the end of the
iteration by the loop construct driving the generator.

The main goal of the proposal is to ease debugging in the situation
where an unguarded next() call (perhaps several stack frames deep)
raises StopIteration and causes the iteration controlled by the
generator to terminate silently. (Whereas, when some other exception is
raised, a traceback is printed pinpointing the cause of the problem.)

This is particularly pernicious in combination with the yield from
construct of PEP 380, as it breaks the abstraction that a subgenerator
may be factored out of a generator. That PEP notes this limitation, but
notes that "use cases for these [are] rare to non-existent".
Unfortunately while intentional use is rare, it is easy to stumble on
these cases by accident:

    import contextlib

    @contextlib.contextmanager
    def transaction():
        print('begin')
        try:
            yield from do_it()
        except:
            print('rollback')
            raise
        else:
            print('commit')

    def do_it():
        print('Refactored initial setup')
        yield # Body of with-statement is executed here
        print('Refactored finalization of successful transaction')

    def gene():
        for i in range(2):
            with transaction():
                yield i
                # return
                raise StopIteration  # This is wrong
            print('Should not be reached')

    for i in gene():
        print('main: i =', i)

Here factoring out do_it into a subgenerator has introduced a subtle
bug: if the wrapped block raises StopIteration, under the current
behavior this exception will be swallowed by the context manager; and,
worse, the finalization is silently skipped! Similarly problematic
behavior occurs when an asyncio coroutine raises StopIteration, causing
it to terminate silently, or when next is used to take the first result
from an iterator that unexpectedly turns out to be empty, for example:

    # using the same context manager as above
    import pathlib

    with transaction():
        print('commit file {}'.format(
            # I can never remember what the README extension is
            next(pathlib.Path('/some/dir').glob('README*'))))

In both cases, the refactoring abstraction of yield from breaks in the
presence of bugs in client code.

Additionally, the proposal reduces the difference between list
comprehensions and generator expressions, preventing surprises such as
the one that started this discussion[1]. Henceforth, the following
statements will produce the same result if either produces a result at
all:

    a = list(F(x) for x in xs if P(x))
    a = [F(x) for x in xs if P(x)]

With the current state of affairs, it is possible to write a function
F(x) or a predicate P(x) that causes the first form to produce a
(truncated) result, while the second form raises an exception (namely,
StopIteration). With the proposed change, both forms will raise an
exception at this point (albeit RuntimeError in the first case and
StopIteration in the second).

Finally, the proposal also clears up the confusion about how to
terminate a generator: the proper way is return, not
raise StopIteration.

As an added bonus, the above changes bring generator functions much more
in line with regular functions. If you wish to take a piece of code
presented as a generator and turn it into something else, you can
usually do this fairly simply, by replacing every yield with a call to
print() or list.append(); however, if there are any bare next() calls in
the code, you have to be aware of them. If the code was originally
written without relying on StopIteration terminating the function, the
transformation would be that much easier.

Background information

When a generator frame is (re)started as a result of a __next__() (or
send() or throw()) call, one of three outcomes can occur:

-   A yield point is reached, and the yielded value is returned.
-   The frame is returned from; StopIteration is raised.
-   An exception is raised, which bubbles out.

In the latter two cases the frame is abandoned (and the generator
object's gi_frame attribute is set to None).

Proposal

If a StopIteration is about to bubble out of a generator frame, it is
replaced with RuntimeError, which causes the next() call (which invoked
the generator) to fail, passing that exception out. From then on it's
just like any old exception.[2]

This affects the third outcome listed above, without altering any other
effects. Furthermore, it only affects this outcome when the exception
raised is StopIteration (or a subclass thereof).

Note that the proposed replacement happens at the point where the
exception is about to bubble out of the frame, i.e. after any except or
finally blocks that could affect it have been exited. The StopIteration
raised by returning from the frame is not affected (the point being that
StopIteration means that the generator terminated "normally", i.e. it
did not raise an exception).

A subtle issue is what will happen if the caller, having caught the
RuntimeError, calls the generator object's __next__() method again. The
answer is that from this point on it will raise StopIteration -- the
behavior is the same as when any other exception was raised by the
generator.

Another logical consequence of the proposal: if someone uses
g.throw(StopIteration) to throw a StopIteration exception into a
generator, if the generator doesn't catch it (which it could do using a
try/except around the yield), it will be transformed into RuntimeError.

During the transition phase, the new feature must be enabled per-module
using:

    from __future__ import generator_stop

Any generator function constructed under the influence of this directive
will have the REPLACE_STOPITERATION flag set on its code object, and
generators with the flag set will behave according to this proposal.
Once the feature becomes standard, the flag may be dropped; code should
not inspect generators for it.

A proof-of-concept patch has been created to facilitate testing.[3]

Consequences for existing code

This change will affect existing code that depends on StopIteration
bubbling up. The pure Python reference implementation of groupby[4]
currently has comments "Exit on StopIteration" where it is expected that
the exception will propagate and then be handled. This will be unusual,
but not unknown, and such constructs will fail. Other examples abound,
e.g.[5],[6].

(Alyssa Coghlan comments: """If you wanted to factor out a helper
function that terminated the generator you'd have to do "return yield
from helper()" rather than just "helper()".""")

There are also examples of generator expressions floating around that
rely on a StopIteration raised by the expression, the target or the
predicate (rather than by the __next__() call implied in the for loop
proper).

Writing backwards and forwards compatible code

With the exception of hacks that raise StopIteration to exit a generator
expression, it is easy to write code that works equally well under older
Python versions as under the new semantics.

This is done by enclosing those places in the generator body where a
StopIteration is expected (e.g. bare next() calls or in some cases
helper functions that are expected to raise StopIteration) in a
try/except construct that returns when StopIteration is raised. The
try/except construct should appear directly in the generator function;
doing this in a helper function that is not itself a generator does not
work. If raise StopIteration occurs directly in a generator, simply
replace it with return.

Examples of breakage

Generators which explicitly raise StopIteration can generally be changed
to simply return instead. This will be compatible with all existing
Python versions, and will not be affected by __future__. Here are some
illustrations from the standard library.

Lib/ipaddress.py:

    if other == self:
        raise StopIteration

Becomes:

    if other == self:
        return

In some cases, this can be combined with yield from to simplify the
code, such as Lib/difflib.py:

    if context is None:
        while True:
            yield next(line_pair_iterator)

Becomes:

    if context is None:
        yield from line_pair_iterator
        return

(The return is necessary for a strictly-equivalent translation, though
in this particular file, there is no further code, and the return can be
omitted.) For compatibility with pre-3.3 versions of Python, this could
be written with an explicit for loop:

    if context is None:
        for line in line_pair_iterator:
            yield line
        return

More complicated iteration patterns will need explicit try/except
constructs. For example, a hypothetical parser like this:

    def parser(f):
        while True:
            data = next(f)
            while True:
                line = next(f)
                if line == "- end -": break
                data += line
            yield data

would need to be rewritten as:

    def parser(f):
        while True:
            try:
                data = next(f)
                while True:
                    line = next(f)
                    if line == "- end -": break
                    data += line
                yield data
            except StopIteration:
                return

or possibly:

    def parser(f):
        for data in f:
            while True:
                line = next(f)
                if line == "- end -": break
                data += line
            yield data

The latter form obscures the iteration by purporting to iterate over the
file with a for loop, but then also fetches more data from the same
iterator during the loop body. It does, however, clearly differentiate
between a "normal" termination (StopIteration instead of the initial
line) and an "abnormal" termination (failing to find the end marker in
the inner loop, which will now raise RuntimeError).

This effect of StopIteration has been used to cut a generator expression
short, creating a form of takewhile:

    def stop():
        raise StopIteration
    print(list(x for x in range(10) if x < 5 or stop()))
    # prints [0, 1, 2, 3, 4]

Under the current proposal, this form of non-local flow control is not
supported, and would have to be rewritten in statement form:

    def gen():
        for x in range(10):
            if x >= 5: return
            yield x
    print(list(gen()))
    # prints [0, 1, 2, 3, 4]

While this is a small loss of functionality, it is functionality that
often comes at the cost of readability, and just as lambda has
restrictions compared to def, so does a generator expression have
restrictions compared to a generator function. In many cases, the
transformation to full generator function will be trivially easy, and
may improve structural clarity.

Explanation of generators, iterators, and StopIteration

The proposal does not change the relationship between generators and
iterators: a generator object is still an iterator, and not all
iterators are generators. Generators have additional methods that
iterators don't have, like send and throw. All this is unchanged.
Nothing changes for generator users -- only authors of generator
functions may have to learn something new. (This includes authors of
generator expressions that depend on early termination of the iteration
by a StopIteration raised in a condition.)

An iterator is an object with a __next__ method. Like many other special
methods, it may either return a value, or raise a specific exception -
in this case, StopIteration - to signal that it has no value to return.
In this, it is similar to __getattr__ (can raise AttributeError),
__getitem__ (can raise KeyError), and so on. A helper function for an
iterator can be written to follow the same protocol; for example:

    def helper(x, y):
        if x > y: return 1 / (x - y)
        raise StopIteration

    def __next__(self):
        if self.a: return helper(self.b, self.c)
        return helper(self.d, self.e)

Both forms of signalling are carried through: a returned value is
returned, an exception bubbles up. The helper is written to match the
protocol of the calling function.

A generator function is one which contains a yield expression. Each time
it is (re)started, it may either yield a value, or return (including
"falling off the end"). A helper function for a generator can also be
written, but it must also follow generator protocol:

    def helper(x, y):
        if x > y: yield 1 / (x - y)

    def gen(self):
        if self.a: return (yield from helper(self.b, self.c))
        return (yield from helper(self.d, self.e))

In both cases, any unexpected exception will bubble up. Due to the
nature of generators and iterators, an unexpected StopIteration inside a
generator will be converted into RuntimeError, but beyond that, all
exceptions will propagate normally.

Transition plan

-   Python 3.5: Enable new semantics under __future__ import; silent
    deprecation warning if StopIteration bubbles out of a generator not
    under __future__ import.
-   Python 3.6: Non-silent deprecation warning.
-   Python 3.7: Enable new semantics everywhere.

Alternate proposals

Raising something other than RuntimeError

Rather than the generic RuntimeError, it might make sense to raise a new
exception type UnexpectedStopIteration. This has the downside of
implicitly encouraging that it be caught; the correct action is to catch
the original StopIteration, not the chained exception.

Supplying a specific exception to raise on return

Alyssa (Nick) Coghlan suggested a means of providing a specific
StopIteration instance to the generator; if any other instance of
StopIteration is raised, it is an error, but if that particular one is
raised, the generator has properly completed. This subproposal has been
withdrawn in favour of better options, but is retained for reference.

Making return-triggered StopIterations obvious

For certain situations, a simpler and fully backward-compatible solution
may be sufficient: when a generator returns, instead of raising
StopIteration, it raises a specific subclass of StopIteration
(GeneratorReturn) which can then be detected. If it is not that
subclass, it is an escaping exception rather than a return statement.

The inspiration for this alternative proposal was Alyssa's observation
[7] that if an asyncio coroutine[8] accidentally raises StopIteration,
it currently terminates silently, which may present a hard-to-debug
mystery to the developer. The main proposal turns such accidents into
clearly distinguishable RuntimeError exceptions, but if that is
rejected, this alternate proposal would enable asyncio to distinguish
between a return statement and an accidentally-raised StopIteration
exception.

Of the three outcomes listed above, two change:

-   If a yield point is reached, the value, obviously, would still be
    returned.
-   If the frame is returned from, GeneratorReturn (rather than
    StopIteration) is raised.
-   If an instance of GeneratorReturn would be raised, instead an
    instance of StopIteration would be raised. Any other exception
    bubbles up normally.

In the third case, the StopIteration would have the value of the
original GeneratorReturn, and would reference the original exception in
its __cause__. If uncaught, this would clearly show the chaining of
exceptions.

This alternative does not affect the discrepancy between generator
expressions and list comprehensions, but allows generator-aware code
(such as the contextlib and asyncio modules) to reliably differentiate
between the second and third outcomes listed above.

However, once code exists that depends on this distinction between
GeneratorReturn and StopIteration, a generator that invokes another
generator and relies on the latter's StopIteration to bubble out would
still be potentially wrong, depending on the use made of the distinction
between the two exception types.

Converting the exception inside next()

Mark Shannon suggested[9] that the problem could be solved in next()
rather than at the boundary of generator functions. By having next()
catch StopIteration and raise instead ValueError, all unexpected
StopIteration bubbling would be prevented; however, the
backward-incompatibility concerns are far more serious than for the
current proposal, as every next() call now needs to be rewritten to
guard against ValueError instead of StopIteration - not to mention that
there is no way to write one block of code which reliably works on
multiple versions of Python. (Using a dedicated exception type, perhaps
subclassing ValueError, would help this; however, all code would still
need to be rewritten.)

Note that calling next(it, default) catches StopIteration and
substitutes the given default value; this feature is often useful to
avoid a try/except block.

Sub-proposal: decorator to explicitly request current behaviour

Alyssa Coghlan suggested[10] that the situations where the current
behaviour is desired could be supported by means of a decorator:

    from itertools import allow_implicit_stop

    @allow_implicit_stop
    def my_generator():
        ...
        yield next(it)
        ...

Which would be semantically equivalent to:

    def my_generator():
        try:
            ...
            yield next(it)
            ...
        except StopIteration
            return

but be faster, as it could be implemented by simply permitting the
StopIteration to bubble up directly.

Single-source Python 2/3 code would also benefit in a 3.7+ world, since
libraries like six and python-future could just define their own version
of "allow_implicit_stop" that referred to the new builtin in 3.5+, and
was implemented as an identity function in other versions.

However, due to the implementation complexities required, the ongoing
compatibility issues created, the subtlety of the decorator's effect,
and the fact that it would encourage the "quick-fix" solution of just
slapping the decorator onto all generators instead of properly fixing
the code in question, this sub-proposal has been rejected.[11]

Criticism

Unofficial and apocryphal statistics suggest that this is seldom, if
ever, a problem.[12] Code does exist which relies on the current
behaviour (e.g.[13],[14],[15]), and there is the concern that this would
be unnecessary code churn to achieve little or no gain.

Steven D'Aprano started an informal survey on comp.lang.python[16]; at
the time of writing only two responses have been received: one was in
favor of changing list comprehensions to match generator expressions
(!), the other was in favor of this PEP's main proposal.

The existing model has been compared to the perfectly-acceptable issues
inherent to every other case where an exception has special meaning. For
instance, an unexpected KeyError inside a __getitem__ method will be
interpreted as failure, rather than permitted to bubble up. However,
there is a difference. Special methods use return to indicate normality,
and raise to signal abnormality; generators yield to indicate data, and
return to signal the abnormal state. This makes explicitly raising
StopIteration entirely redundant, and potentially surprising. If other
special methods had dedicated keywords to distinguish between their
return paths, they too could turn unexpected exceptions into
RuntimeError; the fact that they cannot should not preclude generators
from doing so.

Why not fix all __next__() methods?

When implementing a regular __next__() method, the only way to indicate
the end of the iteration is to raise StopIteration. So catching
StopIteration here and converting it to RuntimeError would defeat the
purpose. This is a reminder of the special status of generator
functions: in a generator function, raising StopIteration is redundant
since the iteration can be terminated by a simple return.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Initial mailing list comment
(https://mail.python.org/pipermail/python-ideas/2014-November/029906.html)

[2] Proposal by GvR
(https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)

[3] Tracker issue with Proof-of-Concept patch
(http://bugs.python.org/issue22906)

[4] Pure Python implementation of groupby
(https://docs.python.org/3/library/itertools.html#itertools.groupby)

[5] Split a sequence or generator using a predicate
(http://code.activestate.com/recipes/578416-split-a-sequence-or-generator-using-a-predicate/)

[6] wrap unbounded generator to restrict its output
(http://code.activestate.com/recipes/66427-wrap-unbounded-generator-to-restrict-its-output/)

[7] Post from Alyssa (Nick) Coghlan mentioning asyncio
(https://mail.python.org/pipermail/python-ideas/2014-November/029961.html)

[8] Coroutines in asyncio
(https://docs.python.org/3/library/asyncio-task.html#coroutines)

[9] Post from Mark Shannon with alternate proposal
(https://mail.python.org/pipermail/python-dev/2014-November/137129.html)

[10] Idea from Alyssa Coghlan
(https://mail.python.org/pipermail/python-dev/2014-November/137201.html)

[11] Rejection of above idea by GvR
(https://mail.python.org/pipermail/python-dev/2014-November/137243.html)

[12] Response by Steven D'Aprano
(https://mail.python.org/pipermail/python-ideas/2014-November/029994.html)

[13] Proposal by GvR
(https://mail.python.org/pipermail/python-ideas/2014-November/029953.html)

[14] Split a sequence or generator using a predicate
(http://code.activestate.com/recipes/578416-split-a-sequence-or-generator-using-a-predicate/)

[15] wrap unbounded generator to restrict its output
(http://code.activestate.com/recipes/66427-wrap-unbounded-generator-to-restrict-its-output/)

[16] Thread on comp.lang.python started by Steven D'Aprano
(https://mail.python.org/pipermail/python-list/2014-November/680757.html)

authors:
Chris Angelico
Guido van Rossum


text:
PEP: 441 Title: Improving Python ZIP Application Support Author: Daniel
Holth<EMAIL_ADDRESS>Paul Moore<EMAIL_ADDRESS>Discussions-To:
https://mail.python.org/pipermail/python-dev/2015-February/138277.html
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
30-Mar-2013 Python-Version: 3.5 Post-History: 30-Mar-2013, 01-Apr-2013,
16-Feb-2015 Resolution:
https://mail.python.org/pipermail/python-dev/2015-February/138578.html

Improving Python ZIP Application Support

Python has had the ability to execute directories or ZIP-format archives
as scripts since version 2.6[1]. When invoked with a zip file or
directory as its first argument the interpreter adds that directory to
sys.path and executes the __main__ module. These archives provide a
great way to publish software that needs to be distributed as a single
file script but is complex enough to need to be written as a collection
of modules.

This feature is not as popular as it should be mainly because it was not
promoted as part of Python 2.6[2], so that it is relatively unknown, but
also because the Windows installer does not register a file extension
(other than .py) for this format of file, to associate with the
launcher.

This PEP proposes to fix these problems by re-publicising the feature,
defining the .pyz and .pyzw extensions as "Python ZIP Applications" and
"Windowed Python ZIP Applications", and providing some simple tooling to
manage the format.

A New Python ZIP Application Extension

The terminology "Python Zip Application" will be the formal term used
for a zip-format archive that contains Python code in a form that can be
directly executed by Python (specifically, it must have a __main__.py
file in the root directory of the archive). The extension .pyz will be
formally associated with such files.

The Python 3.5 installer will associate .pyz and .pyzw "Python Zip
Applications" with the platform launcher so they can be executed. A .pyz
archive is a console application and a .pyzw archive is a windowed
application, indicating whether the console should appear when running
the app.

On Unix, it would be ideal if the .pyz extension and the name "Python
Zip Application" were registered (in the mime types database?). However,
such an association is out of scope for this PEP.

Python Zip applications can be prefixed with a #! line pointing to the
correct Python interpreter and an optional explanation:

    #!/usr/bin/env python3
    #  Python application packed with zipapp module
    (binary contents of archive)

On Unix, this allows the OS to run the file with the correct
interpreter, via the standard "shebang" support. On Windows, the Python
launcher implements shebang support.

However, it is always possible to execute a .pyz application by
supplying the filename to the Python interpreter directly.

As background, ZIP archives are defined with a footer containing
relative offsets from the end of the file. They remain valid when
concatenated to the end of any other file. This feature is completely
standard and is how self-extracting ZIP archives and the bdist_wininst
installer format work.

Minimal Tooling: The zipapp Module

This PEP also proposes including a module for working with these
archives. The module will contain functions for working with Python zip
application archives, and a command line interface (via
python -m zipapp) for their creation and manipulation.

More complete tools for managing Python Zip Applications are encouraged
as 3rd party applications on PyPI. Currently, pyzzer[3] and pex[4] are
two such tools.

Module Interface

The zipapp module will provide the following functions:

create_archive(source, target=None, interpreter=None, main=None)

Create an application archive from source. The source can be any of the
following:

-   The name of a directory, in which case a new application archive
    will be created from the content of that directory.
-   The name of an existing application archive file, in which case the
    file is copied to the target. The file name should include the .pyz
    or .pyzw extension, if required.
-   A file object open for reading in bytes mode. The content of the
    file should be an application archive, and the file object is
    assumed to be positioned at the start of the archive.

The target argument determines where the resulting archive will be
written:

-   If it is the name of a file, the archive will be written to that
    file.
-   If it is an open file object, the archive will be written to that
    file object, which must be open for writing in bytes mode.
-   If the target is omitted (or None), the source must be a directory
    and the target will be a file with the same name as the source, with
    a .pyz extension added.

The interpreter argument specifies the name of the Python interpreter
with which the archive will be executed. It is written as a "shebang"
line at the start of the archive. On Unix, this will be interpreted by
the OS, and on Windows it will be handled by the Python launcher.
Omitting the interpreter results in no shebang line being written. If an
interpreter is specified, and the target is a filename, the executable
bit of the target file will be set.

The main argument specifies the name of a callable which will be used as
the main program for the archive. It can only be specified if the source
is a directory, and the source does not already contain a __main__.py
file. The main argument should take the form "pkg.module:callable" and
the archive will be run by importing "pkg.module" and executing the
given callable with no arguments. It is an error to omit main if the
source is a directory and does not contain a __main__.py file, as
otherwise the resulting archive would not be executable.

If a file object is specified for source or target, it is the caller's
responsibility to close it after calling create_archive.

When copying an existing archive, file objects supplied only need read
and readline, or write methods. When creating an archive from a
directory, if the target is a file object it will be passed to the
zipfile.ZipFile class, and must supply the methods needed by that class.

get_interpreter(archive)

Returns the interpreter specified in the shebang line of the archive. If
there is no shebang, the function returns None. The archive argument can
be a filename or a file-like object open for reading in bytes mode.

Command Line Usage

The zipapp module can be run with the python -m flag. The command line
interface is as follows:

    python -m zipapp directory [options]

        Create an archive from the given directory.  An archive will
        be created from the contents of that directory.  The archive
        will have the same name as the source directory with a .pyz
        extension.

        The following options can be specified:

        -o archive / --output archive

            The destination archive will have the specified name.  The
            given name will be used as written, so should include the
            ".pyz" or ".pyzw" extension.

        -p interpreter / --python interpreter

            The given interpreter will be written to the shebang line
            of the archive.  If this option is not given, the archive
            will have no shebang line.

        -m pkg.mod:fn / --main pkg.mod:fn

            The source directory must not have a __main__.py file. The
            archiver will write a __main__.py file into the target
            which calls fn from the module pkg.mod.

The behaviour of the command line interface matches that of
zipapp.create_archive().

In addition, it is possible to use the command line interface to work
with an existing archive:

    python -m zipapp app.pyz --show

        Displays the shebang line of an archive.  Output is of the
        form

            Interpreter: /usr/bin/env
        or
            Interpreter: <none>

        and is intended for diagnostic use, not for scripts.

    python -m zipapp app.pyz -o newapp.pyz [-p interpreter]

        Copy app.pyz to newapp.pyz, modifying the shebang line based
        on the -p option (as for creating an archive, no -p option
        means remove the shebang line).  Specifying a destination is
        mandatory.

        In-place modification of an archive is *not* supported, as the
        risk of damaging archives is too great for a simple tool.

As noted, the archives are standard zip files, and so can be unpacked
using any standard ZIP utility or Python's zipfile module. For this
reason, no interfaces to list the contents of an archive, or unpack
them, are provided or needed.

FAQ

Are you sure a standard ZIP utility can handle #! at the beginning?

    Absolutely. The zipfile specification allows for arbitrary data to
    be prepended to a zipfile. This feature is commonly used by
    "self-extracting zip" programs. If your archive program can't handle
    this, it is a bug in your archive program.

Isn't zipapp just a very thin wrapper over the zipfile module?

    Yes. If you prefer to build your own Python zip application archives
    using other tools, they will work just as well. The zipapp module is
    a convenience, nothing more.

Why not use just use a .zip or .py extension?

    Users expect a .zip file to be opened with an archive tool, and
    expect a .py file to contain readable text. Both would be confusing
    for this use case.

How does this compete with existing package formats?

    The sdist, bdist and wheel formats are designed for packaging of
    modules to be installed into an existing Python installation. They
    are not intended to be used without installing. The executable zip
    format is specifically designed for standalone use, without needing
    to be installed. They are in effect a multi-file version of a
    standalone Python script.

Rejected Proposals

Convenience Values for Shebang Lines

Is it worth having "convenience" forms for any of the common interpreter
values? For example, -p 3 meaning the same as -p "/usr/bin/env python3".
It would save a lot of typing for the common cases, as well as giving
cross-platform options for people who don't want or need to understand
the intricacies of shebang handling on "other" platforms.

Downsides are that it's not obvious how to translate the abbreviations.
For example, should "3" mean "/usr/bin/env python3", "/usr/bin/python3",
"python3", or something else? Also, there is no obvious short form for
the key case of "/usr/bin/env python" (any available version of Python),
which could easily result in scripts being written with
overly-restrictive shebang lines.

Overall, this seems like there are more problems than benefits, and as a
result has been dropped from consideration.

Registering .pyz as a Media Type

It was suggested[5] that the .pyz extension should be registered in the
Unix database of extensions. While it makes sense to do this as an
equivalent of the Windows installer registering the extension, the .py
extension is not listed in the media types database[6]. It doesn't seem
reasonable to register .pyz without .py, so this idea has been omitted
from this PEP. An interested party could arrange for both .py and .pyz
to be registered at a future date.

Default Interpreter

The initial draft of this PEP proposed using /usr/bin/env python as the
default interpreter. Unix users have problems with this behaviour, as
the default for the python command on many distributions is Python 2,
and it is felt that this PEP should prefer Python 3 by default. However,
using a command of python3 can result in unexpected behaviour for
Windows users, where the default behaviour of the launcher for the
command python is commonly customised by users, but the behaviour of
python3 may not be modified to match.

As a result, the principle "in the face of ambiguity, refuse to guess"
has been invoked, and archives have no shebang line unless explicitly
requested. On Windows, the archives will still be run (with the default
Python) by the launcher, and on Unix, the archives can be run by
explicitly invoking the desired Python interpreter.

Command Line Tool to Manage Shebang Lines

It is conceivable that users would want to modify the shebang line for
an existing archive, or even just display the current shebang line. This
is tricky to do so with existing tools (zip programs typically ignore
prepended data totally, and text editors can have trouble editing files
containing binary data).

The zipapp module provides functions to handle the shebang line, but
does not include a command line interface to that functionality. This is
because it is not clear how to provide one without the resulting
interface being over-complex and potentially confusing. Changing the
shebang line is expected to be an uncommon requirement.

Reference Implementation

A reference implementation is at http://bugs.python.org/issue23491.

References

The discussion of this PEP took place on the python-dev mailing list, in
the thread starting at
https://mail.python.org/pipermail/python-dev/2015-February/138277.html

Copyright

This document has been placed into the public domain.

[1] Allow interpreter to execute a zip file
(http://bugs.python.org/issue1739468)

[2] Feature is not documented (http://bugs.python.org/issue17359)

[3] pyzzer - A tool for creating Python-executable archives
(https://pypi.python.org/pypi/pyzzer)

[4] pex - The PEX packaging toolchain (https://pypi.python.org/pypi/pex)

[5] Discussion of adding a .pyz mime type on python-dev
(https://mail.python.org/pipermail/python-dev/2015-February/138338.html)

[6] Register of media types
(http://www.iana.org/assignments/media-types/media-types.xhtml)


text:
PEP: 616 Title: String methods to remove prefixes and suffixes Author:
Dennis Sweeney<EMAIL_ADDRESS>Sponsor: Eric V. Smith
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 19-Mar-2020 Python-Version: 3.9 Post-History:
20-Mar-2020

Abstract

This is a proposal to add two new methods, removeprefix() and
removesuffix(), to the APIs of Python's various string objects. These
methods would remove a prefix or suffix (respectively) from a string, if
present, and would be added to Unicode str objects, binary bytes and
bytearray objects, and collections.UserString.

Rationale

There have been repeated issues on Python-Ideas[1][2],
Python-Dev[3][4][5][6], the Bug Tracker, and StackOverflow[7], related
to user confusion about the existing str.lstrip and str.rstrip methods.
These users are typically expecting the behavior of removeprefix and
removesuffix, but they are surprised that the parameter for lstrip is
interpreted as a set of characters, not a substring. This repeated issue
is evidence that these methods are useful. The new methods allow a
cleaner redirection of users to the desired behavior.

As another testimonial for the usefulness of these methods, several
users on Python-Ideas[8] reported frequently including similar functions
in their code for productivity. The implementation often contained
subtle mistakes regarding the handling of the empty string, so a
well-tested built-in method would be useful.

The existing solutions for creating the desired behavior are to either
implement the methods as in the Specification below, or to use regular
expressions as in the expression re.sub('^' + re.escape(prefix), '', s),
which is less discoverable, requires a module import, and results in
less readable code.

Specification

The builtin str class will gain two new methods which will behave as
follows when type(self) is type(prefix) is type(suffix) is str:

    def removeprefix(self: str, prefix: str, /) -> str:
        if self.startswith(prefix):
            return self[len(prefix):]
        else:
            return self[:]

    def removesuffix(self: str, suffix: str, /) -> str:
        # suffix='' should not call self[:-0].
        if suffix and self.endswith(suffix):
            return self[:-len(suffix)]
        else:
            return self[:]

When the arguments are instances of str subclasses, the methods should
behave as though those arguments were first coerced to base str objects,
and the return value should always be a base str.

Methods with the corresponding semantics will be added to the builtin
bytes and bytearray objects. If b is either a bytes or bytearray object,
then b.removeprefix() and b.removesuffix() will accept any bytes-like
object as an argument. The two methods will also be added to
collections.UserString, with similar behavior.

Motivating examples from the Python standard library

The examples below demonstrate how the proposed methods can make code
one or more of the following:

1.  Less fragile:

    The code will not depend on the user to count the length of a
    literal.

2.  More performant:

    The code does not require a call to the Python built-in len function
    nor to the more expensive str.replace() method.

3.  More descriptive:

    The methods give a higher-level API for code readability as opposed
    to the traditional method of string slicing.

find_recursionlimit.py

-   Current:

        if test_func_name.startswith("test_"):
            print(test_func_name[5:])
        else:
            print(test_func_name)

-   Improved:

        print(test_func_name.removeprefix("test_"))

deccheck.py

This is an interesting case because the author chose to use the
str.replace method in a situation where only a prefix was intended to be
removed.

-   Current:

        if funcname.startswith("context."):
            self.funcname = funcname.replace("context.", "")
            self.contextfunc = True
        else:
            self.funcname = funcname
            self.contextfunc = False

-   Improved:

        if funcname.startswith("context."):
            self.funcname = funcname.removeprefix("context.")
            self.contextfunc = True
        else:
            self.funcname = funcname
            self.contextfunc = False

-   Arguably further improved:

        self.contextfunc = funcname.startswith("context.")
        self.funcname = funcname.removeprefix("context.")

cookiejar.py

-   Current:

        def strip_quotes(text):
            if text.startswith('"'):
                text = text[1:]
            if text.endswith('"'):
                text = text[:-1]
            return text

-   Improved:

        def strip_quotes(text):
            return text.removeprefix('"').removesuffix('"')

test_i18n.py

-   Current:

        creationDate = header['POT-Creation-Date']

        # peel off the escaped newline at the end of string
        if creationDate.endswith('\\n'):
            creationDate = creationDate[:-len('\\n')]

-   Improved:

        creationDate = header['POT-Creation-Date'].removesuffix('\\n')

There were many other such examples in the stdlib.

Rejected Ideas

Expand the lstrip and rstrip APIs

Because lstrip takes a string as its argument, it could be viewed as
taking an iterable of length-1 strings. The API could, therefore, be
generalized to accept any iterable of strings, which would be
successively removed as prefixes. While this behavior would be
consistent, it would not be obvious for users to have to call
'foobar'.lstrip(('foo',)) for the common use case of a single prefix.

Remove multiple copies of a prefix

This is the behavior that would be consistent with the aforementioned
expansion of the lstrip/rstrip API -- repeatedly applying the function
until the argument is unchanged. This behavior is attainable from the
proposed behavior via by the following:

    >>> s = 'Foo' * 100 + 'Bar'
    >>> prefix = 'Foo'
    >>> while s.startswith(prefix): s = s.removeprefix(prefix)
    >>> s
    'Bar'

Raising an exception when not found

There was a suggestion that s.removeprefix(pre) should raise an
exception if not s.startswith(pre). However, this does not match with
the behavior and feel of other string methods. There could be
required=False keyword added, but this violates the KISS principle.

Accepting a tuple of affixes

It could be convenient to write the test_concurrent_futures.py example
above as name.removesuffix(('Mixin', 'Tests', 'Test')), so there was a
suggestion that the new methods be able to take a tuple of strings as an
argument, similar to the startswith() API. Within the tuple, only the
first matching affix would be removed. This was rejected on the
following grounds:

-   This behavior can be surprising or visually confusing, especially
    when one prefix is empty or is a substring of another prefix, as in
    'FooBar'.removeprefix(('', 'Foo')) == 'FooBar' or
    'FooBar text'.removeprefix(('Foo', 'FooBar ')) == 'Bar text'.
-   The API for str.replace() only accepts a single pair of replacement
    strings, but has stood the test of time by refusing the temptation
    to guess in the face of ambiguous multiple replacements.
-   There may be a compelling use case for such a feature in the future,
    but generalization before the basic feature sees real-world use
    would be easy to get permanently wrong.

Alternative Method Names

Several alternatives method names have been proposed. Some are listed
below, along with commentary for why they should be rejected in favor of
removeprefix (the same arguments hold for removesuffix).

-   ltrim, trimprefix, etc.:

    "Trim" does in other languages (e.g. JavaScript, Java, Go, PHP) what
    strip methods do in Python.

-   lstrip(string=...)

    This would avoid adding a new method, but for different behavior,
    it's better to have two different methods than one method with a
    keyword argument that selects the behavior.

-   remove_prefix:

    All of the other methods of the string API, e.g. str.startswith(),
    use lowercase rather than lower_case_with_underscores.

-   removeleft, leftremove, or lremove:

    The explicitness of "prefix" is preferred.

-   cutprefix, deleteprefix, withoutprefix, dropprefix, etc.:

    Many of these might have been acceptable, but "remove" is
    unambiguous and matches how one would describe the "remove the
    prefix" behavior in English.

-   stripprefix:

    Users may benefit from remembering that "strip" means working with
    sets of characters, while other methods work with substrings, so
    re-using "strip" here should be avoided.

How to Teach This

Among the uses for the partition(), startswith(), and split() string
methods or the enumerate() or zip() built-in functions, a common theme
is that if a beginner finds themselves manually indexing or slicing a
string, then they should consider whether there is a higher-level method
that better communicates what the code should do rather than merely how
the code should do it. The proposed removeprefix() and removesuffix()
methods expand the high-level string "toolbox" and further allow for
this sort of skepticism toward manual slicing.

The main opportunity for user confusion will be the conflation of
lstrip/rstrip with removeprefix/removesuffix. It may therefore be
helpful to emphasize (as the documentation will) the following
differences between the methods:

-   (l/r)strip:
    -   The argument is interpreted as a character set.
    -   The characters are repeatedly removed from the appropriate end
        of the string.
-   remove(prefix/suffix):
    -   The argument is interpreted as an unbroken substring.
    -   Only at most one copy of the prefix/suffix is removed.

Reference Implementation

See the pull request on GitHub[9].

History of Major revisions

-   Version 3: Remove tuple behavior.
-   Version 2: Changed name to removeprefix/removesuffix; added support
    for tuples as arguments
-   Version 1: Initial draft with cutprefix/cutsuffix

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] [Python-Ideas] "New explicit methods to trim strings"
<EMAIL_ADDRESS>
[2] "Re: [Python-ideas] adding a trim convenience function"
<EMAIL_ADDRESS>
[3] "Re: [Python-Dev] strip behavior provides inconsistent results with
certain strings"
<EMAIL_ADDRESS>
[4] [Python-Dev] "correction of a bug"
<EMAIL_ADDRESS>
[5] [Python-Dev] "str.lstrip bug?"
<EMAIL_ADDRESS>
[6] [Python-Dev] "strip behavior provides inconsistent results with
certain strings"
<EMAIL_ADDRESS>
[7] Comment listing Bug Tracker and StackOverflow issues
<EMAIL_ADDRESS>
[8] [Python-Ideas] "New explicit methods to trim strings"
<EMAIL_ADDRESS>
[9] GitHub pull request with implementation
(https://github.com/python/cpython/pull/18939)