text:
PEP: 340 Title: Anonymous Block Statements Version: $Revision$
Last-Modified: $Date$ Author: Guido van Rossum Status: Rejected Type:
Standards Track Content-Type: text/x-rst Created: 27-Apr-2005
Post-History:

Introduction

This PEP proposes a new type of compound statement which can be used for
resource management purposes. The new statement type is provisionally
called the block-statement because the keyword to be used has not yet
been chosen.

This PEP competes with several other PEPs: PEP 288 (Generators
Attributes and Exceptions; only the second part), PEP 310 (Reliable
Acquisition/Release Pairs), and PEP 325 (Resource-Release Support for
Generators).

I should clarify that using a generator to "drive" a block statement is
really a separable proposal; with just the definition of the block
statement from the PEP you could implement all the examples using a
class (similar to example 6, which is easily turned into a template).
But the key idea is using a generator to drive a block statement; the
rest is elaboration, so I'd like to keep these two parts together.

(PEP 342, Enhanced Iterators, was originally a part of this PEP; but the
two proposals are really independent and with Steven Bethard's help I
have moved it to a separate PEP.)

Rejection Notice

I am rejecting this PEP in favor of PEP 343. See the motivational
section in that PEP for the reasoning behind this rejection. GvR.

Motivation and Summary

(Thanks to Shane Hathaway -- Hi Shane!)

Good programmers move commonly used code into reusable functions.
Sometimes, however, patterns arise in the structure of the functions
rather than the actual sequence of statements. For example, many
functions acquire a lock, execute some code specific to that function,
and unconditionally release the lock. Repeating the locking code in
every function that uses it is error prone and makes refactoring
difficult.

Block statements provide a mechanism for encapsulating patterns of
structure. Code inside the block statement runs under the control of an
object called a block iterator. Simple block iterators execute code
before and after the code inside the block statement. Block iterators
also have the opportunity to execute the controlled code more than once
(or not at all), catch exceptions, or receive data from the body of the
block statement.

A convenient way to write block iterators is to write a generator (PEP
255). A generator looks a lot like a Python function, but instead of
returning a value immediately, generators pause their execution at
"yield" statements. When a generator is used as a block iterator, the
yield statement tells the Python interpreter to suspend the block
iterator, execute the block statement body, and resume the block
iterator when the body has executed.

The Python interpreter behaves as follows when it encounters a block
statement based on a generator. First, the interpreter instantiates the
generator and begins executing it. The generator does setup work
appropriate to the pattern it encapsulates, such as acquiring a lock,
opening a file, starting a database transaction, or starting a loop.
Then the generator yields execution to the body of the block statement
using a yield statement. When the block statement body completes, raises
an uncaught exception, or sends data back to the generator using a
continue statement, the generator resumes. At this point, the generator
can either clean up and stop or yield again, causing the block statement
body to execute again. When the generator finishes, the interpreter
leaves the block statement.

Use Cases

See the Examples section near the end.

Specification: the __exit__() Method

An optional new method for iterators is proposed, called __exit__(). It
takes up to three arguments which correspond to the three "arguments" to
the raise-statement: type, value, and traceback. If all three arguments
are None, sys.exc_info() may be consulted to provide suitable default
values.

Specification: the Anonymous Block Statement

A new statement is proposed with the syntax:

    block EXPR1 as VAR1:
        BLOCK1

Here, 'block' and 'as' are new keywords; EXPR1 is an arbitrary
expression (but not an expression-list) and VAR1 is an arbitrary
assignment target (which may be a comma-separated list).

The "as VAR1" part is optional; if omitted, the assignments to VAR1 in
the translation below are omitted (but the expressions assigned are
still evaluated!).

The choice of the 'block' keyword is contentious; many alternatives have
been proposed, including not to use a keyword at all (which I actually
like). PEP 310 uses 'with' for similar semantics, but I would like to
reserve that for a with-statement similar to the one found in Pascal and
VB. (Though I just found that the C# designers don't like 'with'[1], and
I have to agree with their reasoning.) To sidestep this issue
momentarily I'm using 'block' until we can agree on the right keyword,
if any.

Note that the 'as' keyword is not contentious (it will finally be
elevated to proper keyword status).

Note that it is up to the iterator to decide whether a block-statement
represents a loop with multiple iterations; in the most common use case
BLOCK1 is executed exactly once. To the parser, however, it is always a
loop; break and continue return transfer to the block's iterator (see
below for details).

The translation is subtly different from a for-loop: iter() is not
called, so EXPR1 should already be an iterator (not just an iterable);
and the iterator is guaranteed to be notified when the block-statement
is left, regardless if this is due to a break, return or exception:

    itr = EXPR1  # The iterator
    ret = False  # True if a return statement is active
    val = None   # Return value, if ret == True
    exc = None   # sys.exc_info() tuple if an exception is active
    while True:
        try:
            if exc:
                ext = getattr(itr, "__exit__", None)
                if ext is not None:
                    VAR1 = ext(*exc)   # May re-raise *exc
                else:
                    raise exc[0], exc[1], exc[2]
            else:
                VAR1 = itr.next()  # May raise StopIteration
        except StopIteration:
            if ret:
                return val
            break
        try:
            ret = False
            val = exc = None
            BLOCK1
        except:
            exc = sys.exc_info()

(However, the variables 'itr' etc. are not user-visible and the built-in
names used cannot be overridden by the user.)

Inside BLOCK1, the following special translations apply:

-   "break" is always legal; it is translated into:

        exc = (StopIteration, None, None)
        continue

-   "return EXPR3" is only legal when the block-statement is contained
    in a function definition; it is translated into:

        exc = (StopIteration, None, None)
        ret = True
        val = EXPR3
        continue

The net effect is that break and return behave much the same as if the
block-statement were a for-loop, except that the iterator gets a chance
at resource cleanup before the block-statement is left, through the
optional __exit__() method. The iterator also gets a chance if the
block-statement is left through raising an exception. If the iterator
doesn't have an __exit__() method, there is no difference with a
for-loop (except that a for-loop calls iter() on EXPR1).

Note that a yield-statement in a block-statement is not treated
differently. It suspends the function containing the block without
notifying the block's iterator. The block's iterator is entirely unaware
of this yield, since the local control flow doesn't actually leave the
block. In other words, it is not like a break or return statement. When
the loop that was resumed by the yield calls next(), the block is
resumed right after the yield. (See example 7 below.) The generator
finalization semantics described below guarantee (within the limitations
of all finalization semantics) that the block will be resumed
eventually.

Unlike the for-loop, the block-statement does not have an else-clause. I
think it would be confusing, and emphasize the "loopiness" of the
block-statement, while I want to emphasize its difference from a
for-loop. In addition, there are several possible semantics for an
else-clause, and only a very weak use case.

Specification: Generator Exit Handling

Generators will implement the new __exit__() method API.

Generators will be allowed to have a yield statement inside a
try-finally statement.

The expression argument to the yield-statement will become optional
(defaulting to None).

When __exit__() is called, the generator is resumed but at the point of
the yield-statement the exception represented by the __exit__
argument(s) is raised. The generator may re-raise this exception, raise
another exception, or yield another value, except that if the exception
passed in to __exit__() was StopIteration, it ought to raise
StopIteration (otherwise the effect would be that a break is turned into
continue, which is unexpected at least). When the initial call resuming
the generator is an __exit__() call instead of a next() call, the
generator's execution is aborted and the exception is re-raised without
passing control to the generator's body.

When a generator that has not yet terminated is garbage-collected
(either through reference counting or by the cyclical garbage
collector), its __exit__() method is called once with StopIteration as
its first argument. Together with the requirement that a generator ought
to raise StopIteration when __exit__() is called with StopIteration,
this guarantees the eventual activation of any finally-clauses that were
active when the generator was last suspended. Of course, under certain
circumstances the generator may never be garbage-collected. This is no
different than the guarantees that are made about finalizers (__del__()
methods) of other objects.

Alternatives Considered and Rejected

-   Many alternatives have been proposed for 'block'. I haven't seen a
    proposal for another keyword that I like better than 'block' yet.
    Alas, 'block' is also not a good choice; it is a rather popular name
    for variables, arguments and methods. Perhaps 'with' is the best
    choice after all?

-   Instead of trying to pick the ideal keyword, the block-statement
    could simply have the form:

        EXPR1 as VAR1:
            BLOCK1

    This is at first attractive because, together with a good choice of
    function names (like those in the Examples section below) used in
    EXPR1, it reads well, and feels like a "user-defined statement". And
    yet, it makes me (and many others) uncomfortable; without a keyword
    the syntax is very "bland", difficult to look up in a manual
    (remember that 'as' is optional), and it makes the meaning of break
    and continue in the block-statement even more confusing.

-   Phillip Eby has proposed to have the block-statement use an entirely
    different API than the for-loop, to differentiate between the two. A
    generator would have to be wrapped in a decorator to make it support
    the block API. IMO this adds more complexity with very little
    benefit; and we can't really deny that the block-statement is
    conceptually a loop -- it supports break and continue, after all.

-   This keeps getting proposed: "block VAR1 = EXPR1" instead of "block
    EXPR1 as VAR1". That would be very misleading, since VAR1 does not
    get assigned the value of EXPR1; EXPR1 results in a generator which
    is assigned to an internal variable, and VAR1 is the value returned
    by successive calls to the __next__() method of that iterator.

-   Why not change the translation to apply iter(EXPR1)? All the
    examples would continue to work. But this makes the block-statement
    more like a for-loop, while the emphasis ought to be on the
    difference between the two. Not calling iter() catches a bunch of
    misunderstandings, like using a sequence as EXPR1.

Comparison to Thunks

Alternative semantics proposed for the block-statement turn the block
into a thunk (an anonymous function that blends into the containing
scope).

The main advantage of thunks that I can see is that you can save the
thunk for later, like a callback for a button widget (the thunk then
becomes a closure). You can't use a yield-based block for that (except
in Ruby, which uses yield syntax with a thunk-based implementation). But
I have to say that I almost see this as an advantage: I think I'd be
slightly uncomfortable seeing a block and not knowing whether it will be
executed in the normal control flow or later. Defining an explicit
nested function for that purpose doesn't have this problem for me,
because I already know that the 'def' keyword means its body is executed
later.

The other problem with thunks is that once we think of them as the
anonymous functions they are, we're pretty much forced to say that a
return statement in a thunk returns from the thunk rather than from the
containing function. Doing it any other way would cause major weirdness
when the thunk were to survive its containing function as a closure
(perhaps continuations would help, but I'm not about to go there :-).

But then an IMO important use case for the resource cleanup template
pattern is lost. I routinely write code like this:

    def findSomething(self, key, default=None):
        self.lock.acquire()
        try:
             for item in self.elements:
                 if item.matches(key):
                     return item
             return default
        finally:
           self.lock.release()

and I'd be bummed if I couldn't write this as:

    def findSomething(self, key, default=None):
        block locking(self.lock):
             for item in self.elements:
                 if item.matches(key):
                     return item
             return default

This particular example can be rewritten using a break:

    def findSomething(self, key, default=None):
        block locking(self.lock):
             for item in self.elements:
                 if item.matches(key):
                     break
             else:
                 item = default
         return item

but it looks forced and the transformation isn't always that easy; you'd
be forced to rewrite your code in a single-return style which feels too
restrictive.

Also note the semantic conundrum of a yield in a thunk -- the only
reasonable interpretation is that this turns the thunk into a generator!

Greg Ewing believes that thunks "would be a lot simpler, doing just what
is required without any jiggery pokery with exceptions and
break/continue/return statements. It would be easy to explain what it
does and why it's useful."

But in order to obtain the required local variable sharing between the
thunk and the containing function, every local variable used or set in
the thunk would have to become a 'cell' (our mechanism for sharing
variables between nested scopes). Cells slow down access compared to
regular local variables: access involves an extra C function call
(PyCell_Get() or PyCell_Set()).

Perhaps not entirely coincidentally, the last example above
(findSomething() rewritten to avoid a return inside the block) shows
that, unlike for regular nested functions, we'll want variables assigned
to by the thunk also to be shared with the containing function, even if
they are not assigned to outside the thunk.

Greg Ewing again: "generators have turned out to be more powerful,
because you can have more than one of them on the go at once. Is there a
use for that capability here?"

I believe there are definitely uses for this; several people have
already shown how to do asynchronous light-weight threads using
generators (e.g. David Mertz quoted in PEP 288, and Fredrik Lundh[2]).

And finally, Greg says: "a thunk implementation has the potential to
easily handle multiple block arguments, if a suitable syntax could ever
be devised. It's hard to see how that could be done in a general way
with the generator implementation."

However, the use cases for multiple blocks seem elusive.

(Proposals have since been made to change the implementation of thunks
to remove most of these objections, but the resulting semantics are
fairly complex to explain and to implement, so IMO that defeats the
purpose of using thunks in the first place.)

Examples

(Several of these examples contain "yield None". If PEP 342 is accepted,
these can be changed to just "yield" of course.)

1.  A template for ensuring that a lock, acquired at the start of a
    block, is released when the block is left:

        def locking(lock):
            lock.acquire()
            try:
                yield None
            finally:
                lock.release()

    Used as follows:

        block locking(myLock):
            # Code here executes with myLock held.  The lock is
            # guaranteed to be released when the block is left (even
            # if via return or by an uncaught exception).

2.  A template for opening a file that ensures the file is closed when
    the block is left:

        def opening(filename, mode="r"):
            f = open(filename, mode)
            try:
                yield f
            finally:
                f.close()

    Used as follows:

        block opening("/etc/passwd") as f:
            for line in f:
                print line.rstrip()

3.  A template for committing or rolling back a database transaction:

        def transactional(db):
            try:
                yield None
            except:
                db.rollback()
                raise
            else:
                db.commit()

4.  A template that tries something up to n times:

        def auto_retry(n=3, exc=Exception):
            for i in range(n):
                try:
                    yield None
                    return
                except exc, err:
                    # perhaps log exception here
                    continue
            raise # re-raise the exception we caught earlier

    Used as follows:

        block auto_retry(3, IOError):
            f = urllib.urlopen("https://www.example.com/")
            print f.read()

5.  It is possible to nest blocks and combine templates:

        def locking_opening(lock, filename, mode="r"):
            block locking(lock):
                block opening(filename) as f:
                    yield f

    Used as follows:

        block locking_opening(myLock, "/etc/passwd") as f:
            for line in f:
                print line.rstrip()

    (If this example confuses you, consider that it is equivalent to
    using a for-loop with a yield in its body in a regular generator
    which is invoking another iterator or generator recursively; see for
    example the source code for os.walk().)

6.  It is possible to write a regular iterator with the semantics of
    example 1:

        class locking:
           def __init__(self, lock):
               self.lock = lock
               self.state = 0
           def __next__(self, arg=None):
               # ignores arg
               if self.state:
                   assert self.state == 1
                   self.lock.release()
                   self.state += 1
                   raise StopIteration
               else:
                   self.lock.acquire()
                   self.state += 1
                   return None
           def __exit__(self, type, value=None, traceback=None):
               assert self.state in (0, 1, 2)
               if self.state == 1:
                   self.lock.release()
               raise type, value, traceback

    (This example is easily modified to implement the other examples; it
    shows how much simpler generators are for the same purpose.)

7.  Redirect stdout temporarily:

        def redirecting_stdout(new_stdout):
            save_stdout = sys.stdout
            try:
                sys.stdout = new_stdout
                yield None
            finally:
                sys.stdout = save_stdout

    Used as follows:

        block opening(filename, "w") as f:
            block redirecting_stdout(f):
                print "Hello world"

8.  A variant on opening() that also returns an error condition:

        def opening_w_error(filename, mode="r"):
            try:
                f = open(filename, mode)
            except IOError, err:
                yield None, err
            else:
                try:
                    yield f, None
                finally:
                    f.close()

    Used as follows:

        block opening_w_error("/etc/passwd", "a") as f, err:
            if err:
                print "IOError:", err
            else:
                f.write("guido::0:0::/:/bin/sh\n")

Acknowledgements

In no useful order: Alex Martelli, Barry Warsaw, Bob Ippolito, Brett
Cannon, Brian Sabbey, Chris Ryland, Doug Landauer, Duncan Booth, Fredrik
Lundh, Greg Ewing, Holger Krekel, Jason Diamond, Jim Jewett, Josiah
Carlson, Ka-Ping Yee, Michael Chermside, Michael Hudson, Neil
Schemenauer, Alyssa Coghlan, Paul Moore, Phillip Eby, Raymond Hettinger,
Georg Brandl, Samuele Pedroni, Shannon Behrens, Skip Montanaro, Steven
Bethard, Terry Reedy, Tim Delaney, Aahz, and others. Thanks all for the
valuable contributions!

References

[1] https://mail.python.org/pipermail/python-dev/2005-April/052821.html

Copyright

This document has been placed in the public domain.

[1] https://web.archive.org/web/20060719195933/http://msdn.microsoft.com/vcsharp/programming/language/ask/withstatement/

[2] https://web.archive.org/web/20050204062901/http://effbot.org/zone/asyncore-generators.htm


text:
PEP: 303 Title: Extend divmod() for Multiple Divisors Version:
$Revision$ Last-Modified: $Date$ Author: Thomas Bellman
<EMAIL_ADDRESS>Status: Rejected Type: Standards
Track Content-Type: text/x-rst Created: 31-Dec-2002 Python-Version: 2.3
Post-History:

Abstract

This PEP describes an extension to the built-in divmod() function,
allowing it to take multiple divisors, chaining several calls to
divmod() into one.

Pronouncement

This PEP is rejected. Most uses for chained divmod() involve a constant
modulus (in radix conversions for example) and are more properly coded
as a loop. The example of splitting seconds into
days/hours/minutes/seconds does not generalize to months and years;
rather, the whole use case is handled more flexibly and robustly by date
and time modules. The other use cases mentioned in the PEP are somewhat
rare in real code. The proposal is also problematic in terms of clarity
and obviousness. In the examples, it is not immediately clear that the
argument order is correct or that the target tuple is of the right
length. Users from other languages are more likely to understand the
standard two argument form without having to re-read the documentation.
See python-dev discussion on 17 June 2005[1].

Specification

The built-in divmod() function would be changed to accept multiple
divisors, changing its signature from divmod(dividend, divisor) to
divmod(dividend, *divisors). The dividend is divided by the last
divisor, giving a quotient and a remainder. The quotient is then divided
by the second to last divisor, giving a new quotient and remainder. This
is repeated until all divisors have been used, and divmod() then returns
a tuple consisting of the quotient from the last step, and the
remainders from all the steps.

A Python implementation of the new divmod() behaviour could look like:

    def divmod(dividend, *divisors):
        modulos = ()
        q = dividend
        while divisors:
            q, r = q.__divmod__(divisors[-1])
            modulos = (r,) + modulos
            divisors = divisors[:-1]
        return (q,) + modulos

Motivation

Occasionally one wants to perform a chain of divmod() operations,
calling divmod() on the quotient from the previous step, with varying
divisors. The most common case is probably converting a number of
seconds into weeks, days, hours, minutes and seconds. This would today
be written as:

    def secs_to_wdhms(seconds):
        m, s = divmod(seconds, 60)
        h, m = divmod(m, 60)
        d, h = divmod(h, 24)
        w, d = divmod(d, 7)
        return (w, d, h, m, s)

This is tedious and easy to get wrong each time you need it.

If instead the divmod() built-in is changed according the proposal, the
code for converting seconds to weeks, days, hours, minutes and seconds
then become :

    def secs_to_wdhms(seconds):
        w, d, h, m, s = divmod(seconds, 7, 24, 60, 60)
        return (w, d, h, m, s)

which is easier to type, easier to type correctly, and easier to read.

Other applications are:

-   Astronomical angles (declination is measured in degrees, minutes and
    seconds, right ascension is measured in hours, minutes and seconds).
-   Old British currency (1 pound = 20 shilling, 1 shilling = 12 pence).
-   Anglo-Saxon length units: 1 mile = 1760 yards, 1 yard = 3 feet, 1
    foot = 12 inches.
-   Anglo-Saxon weight units: 1 long ton = 160 stone, 1 stone = 14
    pounds, 1 pound = 16 ounce, 1 ounce = 16 dram.
-   British volumes: 1 gallon = 4 quart, 1 quart = 2 pint, 1 pint = 20
    fluid ounces.

Rationale

The idea comes from APL, which has an operator that does this. (I don't
remember what the operator looks like, and it would probably be
impossible to render in ASCII anyway.)

The APL operator takes a list as its second operand, while this PEP
proposes that each divisor should be a separate argument to the divmod()
function. This is mainly because it is expected that the most common
uses will have the divisors as constants right in the call (as the 7,
24, 60, 60 above), and adding a set of parentheses or brackets would
just clutter the call.

Requiring an explicit sequence as the second argument to divmod() would
seriously break backwards compatibility. Making divmod() check its
second argument for being a sequence is deemed to be too ugly to
contemplate. And in the case where one does have a sequence that is
computed other-where, it is easy enough to write divmod(x, *divs)
instead.

Requiring at least one divisor, i.e rejecting divmod(x), has been
considered, but no good reason to do so has come to mind, and is thus
allowed in the name of generality.

Calling divmod() with no divisors should still return a tuple (of one
element). Code that calls divmod() with a varying number of divisors,
and thus gets a return value with an "unknown" number of elements, would
otherwise have to special case that case. Code that knows it is calling
divmod() with no divisors is considered to be too silly to warrant a
special case.

Processing the divisors in the other direction, i.e dividing with the
first divisor first, instead of dividing with the last divisor first,
has been considered. However, the result comes with the most significant
part first and the least significant part last (think of the chained
divmod as a way of splitting a number into "digits", with varying
weights), and it is reasonable to specify the divisors (weights) in the
same order as the result.

The inverse operation:

    def inverse_divmod(seq, *factors):
        product = seq[0]
        for x, y in zip(factors, seq[1:]):
            product = product * x + y
        return product

could also be useful. However, writing :

    seconds = (((((w * 7) + d) * 24 + h) * 60 + m) * 60 + s)

is less cumbersome both to write and to read than the chained divmods.
It is therefore deemed to be less important, and its introduction can be
deferred to its own PEP. Also, such a function needs a good name, and
the PEP author has not managed to come up with one yet.

Calling divmod("spam") does not raise an error, despite strings
supporting neither division nor modulo. However, unless we know the
other object too, we can't determine whether divmod() would work or not,
and thus it seems silly to forbid it.

Backwards Compatibility

Any module that replaces the divmod() function in the __builtin__
module, may cause other modules using the new syntax to break. It is
expected that this is very uncommon.

Code that expects a TypeError exception when calling divmod() with
anything but two arguments will break. This is also expected to be very
uncommon.

No other issues regarding backwards compatibility are known.

Reference Implementation

Not finished yet, but it seems a rather straightforward new
implementation of the function builtin_divmod() in Python/bltinmodule.c.

References

Copyright

This document has been placed in the public domain.

[1] Raymond Hettinger, "Propose rejection of PEP 303 -- Extend divmod()
for Multiple Divisors"
https://mail.python.org/pipermail/python-dev/2005-June/054283.html

license: Public Domain


text:
PEP: 718 Title: Subscriptable functions Author: James Hilton-Balfe
<EMAIL_ADDRESS>Sponsor: Guido van Rossum<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/28457/ Status: Draft Type:
Standards Track Topic: Typing Content-Type: text/x-rst Created:
23-Jun-2023 Python-Version: 3.13 Post-History: 24-Jun-2023

Abstract

This PEP proposes making function objects subscriptable for typing
purposes. Doing so gives developers explicit control over the types
produced by the type checker where bi-directional inference (which
allows for the types of parameters of anonymous functions to be
inferred) and other methods than specialisation are insufficient. It
also brings functions in line with regular classes in their ability to
be subscriptable.

Motivation

Unknown Types

Currently, it is not possible to infer the type parameters to generic
functions in certain situations:

    def make_list[T](*args: T) -> list[T]: ...
    reveal_type(make_list())  # type checker cannot infer a meaningful type for T

Making instances of FunctionType subscriptable would allow for this
constructor to be typed:

    reveal_type(make_list[int]())  # type is list[int]

Currently you have to use an assignment to provide a precise type:

    x: list[int] = make_list()
    reveal_type(x)  # type is list[int]

but this code is unnecessarily verbose taking up multiple lines for a
simple function call.

Similarly, T in this example cannot currently be meaningfully inferred,
so x is untyped without an extra assignment:

    def factory[T](func: Callable[[T], Any]) -> Foo[T]: ...

    reveal_type(factory(lambda x: "Hello World" * x))

If function objects were subscriptable, however, a more specific type
could be given:

    reveal_type(factory[int](lambda x: "Hello World" * x))  # type is Foo[int]

Undecidable Inference

There are even cases where subclass relations make type inference
impossible. However, if you can specialise the function type checkers
can infer a meaningful type.

    def foo[T](x: Sequence[T] | T) -> list[T]: ...

    reveal_type(foo[bytes](b"hello"))

Currently, type checkers do not consistently synthesise a type here.

Unsolvable Type Parameters

Currently, with unspecialised literals, it is not possible to determine
a type for situations similar to:

    def foo[T](x: list[T]) -> T: ...
    reveal_type(foo([]))  # type checker cannot infer T (yet again)

    reveal_type(foo[int]([]))  # type is int

It is also useful to be able to specify in cases in which a certain type
must be passed to a function beforehand:

    words = ["hello", "world"]
    foo[int](words)  # Invalid: list[str] is incompatible with list[int]

Allowing subscription makes functions and methods consistent with
generic classes where they weren't already. Whilst all of the proposed
changes can be implemented using callable generic classes, syntactic
sugar would be highly welcome.

Due to this, specialising the function and using it as a new factory is
fine

    make_int_list = make_list[int]
    reveal_type(make_int_list())  # type is list[int]

Monomorphisation and Reification

This proposal also opens the door to monomorphisation and reified types.

This would allow for a functionality which anecdotally has been
requested many times.

Please note this feature is not being proposed by the PEP, but may be
implemented in the future.

The syntax for such a feature may look something like:

    def foo[T]():
       return T.__value__

    assert foo[int]() is int

Rationale

Function objects in this PEP is used to refer to FunctionType,
MethodType, BuiltinFunctionType, BuiltinMethodType and
MethodWrapperType.

For MethodType you should be able to write:

    class Foo:
        def make_list[T](self, *args: T) -> list[T]: ...

    Foo().make_list[int]()

and have it work similarly to a FunctionType.

For BuiltinFunctionType, so builtin generic functions (e.g. max and min)
work like ones defined in Python. Built-in functions should behave as
much like functions implemented in Python as possible.

BuiltinMethodType is the same type as BuiltinFunctionType.

MethodWrapperType (e.g. the type of object().__str__) is useful for
generic magic methods.

Specification

Function objects should implement __getitem__ to allow for subscription
at runtime and return an instance of types.GenericAlias with __origin__
set as the callable and __args__ as the types passed.

Type checkers should support subscripting functions and understand that
the parameters passed to the function subscription should follow the
same rules as a generic callable class.

Setting __orig_class__

Currently, __orig_class__ is an attribute set in GenericAlias.__call__
to the instance of the GenericAlias that created the called class e.g.

    class Foo[T]: ...

    assert Foo[int]().__orig_class__ == Foo[int]

Currently, __orig_class__ is unconditionally set; however, to avoid
potential erasure on any created instances, this attribute should not be
set if __origin__ is an instance of any function object.

The following code snippet would fail at runtime without this change as
__orig_class__ would be bar[str] and not Foo[int].

    def bar[U]():
        return Foo[int]()

    assert bar[str]().__orig_class__  == Foo[int]

Interactions with @typing.overload

Overloaded functions should work much the same as already, since they
have no effect on the runtime type. The only change is that more
situations will be decidable and the behaviour/overload can be specified
by the developer rather than leaving it to ordering of overloads/unions.

Backwards Compatibility

Currently these classes are not subclassable and so there are no
backwards compatibility concerns with regards to classes already
implementing __getitem__.

Reference Implementation

The runtime changes proposed can be found here
https://github.com/Gobot1234/cpython/tree/function-subscript

Acknowledgements

Thank you to Alex Waygood and Jelle Zijlstra for their feedback on this
PEP and Guido for some motivating examples.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 599 Title: The manylinux2014 Platform Tag Version: $Revision$
Last-Modified: $Date$ Author: Dustin Ingram<EMAIL_ADDRESS>Sponsor:
Paul Moore<EMAIL_ADDRESS>BDFL-Delegate: Paul Moore
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/the-next-manylinux-specification/1043
Status: Superseded Type: Informational Topic: Packaging Content-Type:
text/x-rst Created: 29-Apr-2019 Post-History: 29-Apr-2019 Superseded-By:
600 Resolution:
https://discuss.python.org/t/the-next-manylinux-specification/1043/199

Abstract

This PEP proposes the creation of a manylinux2014 platform tag to
succeed the manylinux2010 tag introduced by PEP 513. It also proposes
that PyPI and pip both be updated to support uploading, downloading, and
installing manylinux2014 distributions on compatible platforms.

Rationale

CentOS 6 is now the oldest supported CentOS release, and will receive
maintenance updates through November 30th, 2020,[1] at which point it
will reach end-of-life, and no further updates such as security patches
will be made available. All wheels built under the manylinux2010 images
will remain at obsolete versions after that point.

Therefore, we propose the continuation of the existing manylinux
standard, and that a new PEP 425-style platform tag called manylinux2014
be derived from CentOS 7 and that the manylinux toolchain, PyPI, and pip
be updated to support it.

Similar to how PEP 571 and PEP 513 drew allowed shared libraries and
their symbol versions from CentOS 5.11 and CentOS 6, respectively, a
manylinux2014 platform tag will draw its libraries and symbol versions
from CentOS 7, which will reach end-of-life on June 30th, 2024.[2]

The manylinuxYYYY pattern has a number of advantages that motivate
continuing with the current status quo:

-   Well-defined Docker images with clearly specified compatible
    libraries;
-   No need to survey for compatibility issues across multiple releases;
-   A single build image and auditwheel profile per architecture.

There are also some disadvantages:

-   Requires drafting a new PEP for every new standard;
-   Requires adding the new platform tag to installers (e.g., pip);
-   Installers are unable to install a platform tag which predates a
    given release.

There are also challenges which would exist for any proposal, including
the time and effort it takes to define, prepare and release the Docker
images and corresponding auditwheel profiles. These challenges were
experienced in the long rollout period for manylinux2010, which took
approximately 1 year from PEP acceptance to compatible build environment
published.[3]

However, if this PEP can be an indicator, the process is now
well-defined and easily repeatable, which should increase the timeline
for rollout of a newer, updated platform tag.

The manylinux2014 policy

The following criteria determine a linux wheel's eligibility for the
manylinux2014 tag:

1.  The wheel may only contain binary executables and shared objects
    compiled for one of the following architectures supported by CentOS
    7, or a CentOS 7 compatible base image (such as ubi7):[4] :

        x86_64
        i686
        aarch64
        armv7l
        ppc64
        ppc64le
        s390x

    This list adds support for ARMv7 (armv7l), ARMv8 (aarch64) and
    PowerPC (ppc64, ppc64le) architectures supported by the CentOS
    Alternative Architecture Special Interest Group, as well as the IBM
    Z (s390x) architecture.[5]

2.  The wheel's binary executables or shared objects may not link
    against externally-provided libraries except those in the following
    list: :

        libgcc_s.so.1
        libstdc++.so.6
        libm.so.6
        libdl.so.2
        librt.so.1
        libc.so.6
        libnsl.so.1
        libutil.so.1
        libpthread.so.0
        libresolv.so.2
        libX11.so.6
        libXext.so.6
        libXrender.so.1
        libICE.so.6
        libSM.so.6
        libGL.so.1
        libgobject-2.0.so.0
        libgthread-2.0.so.0
        libglib-2.0.so.0

    This list is identical to the externally-provided libraries
    originally allowed for manylinux2010, with one exception:
    libcrypt.so.1 was removed due to being deprecated in Fedora 30.
    libpythonX.Y remains ineligible for inclusion for the same reasons
    outlined in PEP 513.

    On Debian-based systems, these libraries are provided by the
    packages:

      Package        Libraries
      -------------- ----------------------------------------------------------------------------------------------------------
      libc6          libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6, libpthread.so.0, libm.so.6, libutil.so.1, libnsl.so.1
      libgcc1        libgcc_s.so.1
      libgl1         libGL.so.1
      libglib2.0-0   libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
      libice6        libICE.so.6
      libsm6         libSM.so.6
      libstdc++6     libstdc++.so.6
      libx11-6       libX11.so.6
      libxext6       libXext.so.6
      libxrender1    libXrender.so.1

    On RPM-based systems, they are provided by these packages:

      Package      Libraries
      ------------ ----------------------------------------------------------------------------------------------------------
      glib2        libglib-2.0.so.0, libgthread-2.0.so.0, libgobject-2.0.so.0
      glibc        libresolv.so.2, libutil.so.1, libnsl.so.1, librt.so.1, libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6
      libICE       libICE.so.6
      libX11       libX11.so.6
      libXext:     libXext.so.6
      libXrender   libXrender.so.1
      libgcc:      libgcc_s.so.1
      libstdc++    libstdc++.so.6
      mesa         libGL.so.1

3.  If the wheel contains binary executables or shared objects linked
    against any allowed libraries that also export versioned symbols,
    they may only depend on the following maximum versions:

        GLIBC_2.17
        CXXABI_1.3.7, CXXABI_TM_1 is also allowed
        GLIBCXX_3.4.19
        GCC_4.8.0

    As an example, manylinux2014 wheels may include binary artifacts
    that require glibc symbols at version GLIBC_2.12, because this an
    earlier version than the maximum of GLIBC_2.17.

4.  If a wheel is built for any version of CPython 2 or CPython versions
    3.0 up to and including 3.2, it must include a CPython ABI tag
    indicating its Unicode ABI. A manylinux2014 wheel built against
    Python 2, then, must include either the cpy27mu tag indicating it
    was built against an interpreter with the UCS-4 ABI or the cpy27m
    tag indicating an interpreter with the UCS-2 ABI. (PEP 3149[6])

5.  A wheel must not require the PyFPE_jbuf symbol. This is achieved by
    building it against a Python compiled without the --with-fpectl
    configure flag.

Compilation of Compliant Wheels

Like manylinux1, the auditwheel tool adds manylinux2014 platform tags to
linux wheels built by pip wheel or bdist_wheel in a manylinux2014 Docker
container.

Docker Images

A manylinux2014 Docker image based on CentOS 7 x86_64 should be provided
for building binary linux wheels that can reliably be converted to
manylinux2014 wheels. This image will come with a full compiler suite
installed (gcc, g++, and gfortran 4.8.5) as well as the latest releases
of Python and pip.

Auditwheel

The auditwheel tool will also be updated to produce manylinux2014
wheels.[7] Its behavior and purpose will be otherwise unchanged from PEP
513.

Platform Detection for Installers

Platforms may define a manylinux2014_compatible boolean attribute on the
_manylinux module described in PEP 513. A platform is considered
incompatible with manylinux2014 if the attribute is False.

If the _manylinux module is not found, or it does not have the attribute
manylinux2014_compatible, tools may fall back to checking for glibc. If
the platform has glibc 2.17 or newer, it is assumed to be compatible
unless the _manylinux module says otherwise.

Specifically, the algorithm we propose is:

    def is_manylinux2014_compatible():
        # Only Linux, and only supported architectures
        from distutils.util import get_platform

        if get_platform() not in [
            "linux-x86_64",
            "linux-i686",
            "linux-aarch64",
            "linux-armv7l",
            "linux-ppc64",
            "linux-ppc64le",
            "linux-s390x",
        ]:
            return False

        # Check for presence of _manylinux module
        try:
            import _manylinux

            return bool(_manylinux.manylinux2014_compatible)
        except (ImportError, AttributeError):
            # Fall through to heuristic check below
            pass

        # Check glibc version. CentOS 7 uses glibc 2.17.
        # PEP 513 contains an implementation of this function.
        return have_compatible_glibc(2, 17)

Backwards compatibility with manylinux2010 wheels

As explained in PEP 513, the specified symbol versions for manylinux1
allowed libraries constitute an upper bound. The same is true for the
symbol versions defined for manylinux2014 in this PEP. As a result,
manylinux1 and manylinux2010 wheels are considered manylinux2014 wheels.
A pip that recognizes the manylinux2014 platform tag will thus install
manylinux2010 wheels for manylinux2014 platforms -- even when explicitly
set --when no manylinux2014 wheels are available.

PyPI Support

PyPI should permit wheels containing the manylinux2014 platform tag to
be uploaded in the same way that it permits manylinux2010.

If technically feasible, PyPI should attempt to verify the compatibility
of manylinux2014 wheels, but that capability is not a requirement for
adoption of this PEP.

Package authors should not upload non-compliant manylinux2014 wheels to
PyPI, and should be aware that PyPI may begin blocking non-compliant
wheels from being uploaded.

References

Acceptance

PEP 599 was accepted by Paul Moore on July 31, 2019.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] CentOS Product Specifications
(https://wiki.centos.org/About/Product)

[2] CentOS Product Specifications
(https://wiki.centos.org/About/Product)

[3] Tracking issue for manylinux2010 rollout
(https://github.com/pypa/manylinux/issues/179)

[4] Red Hat Universal Base Image 7
(https://access.redhat.com/containers/?tab=overview#/registry.access.redhat.com/ubi7)

[5] The CentOS Alternative Architecture Special Interest Group
(https://wiki.centos.org/SpecialInterestGroup/AltArch)

[6] SOABI support for Python 2.X and PyPy
(https://github.com/pypa/pip/pull/3075)

[7] auditwheel (https://github.com/pypa/auditwheel/)


text:
PEP: 674 Title: Disallow using macros as l-values Author: Victor Stinner
<EMAIL_ADDRESS>Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 30-Nov-2021 Python-Version: 3.12

Abstract

Disallow using macros as l-values. For example, Py_TYPE(obj) = new_type
now fails with a compiler error.

In practice, the majority of affected projects only have to make two
changes:

-   Replace Py_TYPE(obj) = new_type with Py_SET_TYPE(obj, new_type).
-   Replace Py_SIZE(obj) = new_size with Py_SET_SIZE(obj, new_size).

PEP Deferral

See SC reply to PEP 674 -- Disallow using macros as l-values (February
2022).

Rationale

Using a macro as a an l-value

In the Python C API, some functions are implemented as macro because
writing a macro is simpler than writing a regular function. If a macro
exposes directly a structure member, it is technically possible to use
this macro to not only get the structure member but also set it.

Example with the Python 3.10 Py_TYPE() macro:

    #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)

This macro can be used as a r-value to get an object type:

    type = Py_TYPE(object);

It can also be used as an l-value to set an object type:

    Py_TYPE(object) = new_type;

It is also possible to set an object reference count and an object size
using Py_REFCNT() and Py_SIZE() macros.

Setting directly an object attribute relies on the current exact CPython
implementation. Implementing this feature in other Python
implementations can make their C API implementation less efficient.

CPython nogil fork

Sam Gross forked Python 3.9 to remove the GIL: the nogil branch. This
fork has no PyObject.ob_refcnt member, but a more elaborated
implementation for reference counting, and so the
Py_REFCNT(obj) = new_refcnt; code fails with a compiler error.

Merging the nogil fork into the upstream CPython main branch requires
first to fix this C API compatibility issue. It is a concrete example of
a Python optimization blocked indirectly by the C API.

This issue was already fixed in Python 3.10: the Py_REFCNT() macro has
been already modified to disallow using it as an l-value.

These statements are endorsed by Sam Gross (nogil developer).

HPy project

The HPy project is a brand new C API for Python using only handles and
function calls: handles are opaque, structure members cannot be accessed
directly, and pointers cannot be dereferenced.

Searching and replacing Py_SET_SIZE() is easier and safer than searching
and replacing some strange macro uses of Py_SIZE(). Py_SIZE() can be
semi-mechanically replaced by HPy_Length(), whereas seeing Py_SET_SIZE()
would immediately make clear that the code needs bigger changes in order
to be ported to HPy (for example by using HPyTupleBuilder or
HPyListBuilder).

The fewer internal details exposed via macros, the easier it will be for
HPy to provide direct equivalents. Any macro that references
"non-public" interfaces effectively exposes those interfaces publicly.

These statements are endorsed by Antonio Cuni (HPy developer).

GraalVM Python

In GraalVM, when a Python object is accessed by the Python C API, the C
API emulation layer has to wrap the GraalVM objects into wrappers that
expose the internal structure of the CPython structures (PyObject,
PyLongObject, PyTypeObject, etc). This is because when the C code
accesses it directly or via macros, all GraalVM can intercept is a read
at the struct offset, which has to be mapped back to the representation
in GraalVM. The smaller the "effective" number of exposed struct members
(by replacing macros with functions), the simpler GraalVM wrappers can
be.

This PEP alone is not enough to get rid of the wrappers in GraalVM, but
it is a step towards this long term goal. GraalVM already supports HPy
which is a better solution in the long term.

These statements are endorsed by Tim Felgentreff (GraalVM Python
developer).

Specification

Disallow using macros as l-values

The following 65 macros are modified to disallow using them as l-values.

PyObject and PyVarObject macros

-   Py_TYPE(): Py_SET_TYPE() must be used instead
-   Py_SIZE(): Py_SET_SIZE() must be used instead

GET macros

-   PyByteArray_GET_SIZE()
-   PyBytes_GET_SIZE()
-   PyCFunction_GET_CLASS()
-   PyCFunction_GET_FLAGS()
-   PyCFunction_GET_FUNCTION()
-   PyCFunction_GET_SELF()
-   PyCell_GET()
-   PyCode_GetNumFree()
-   PyDict_GET_SIZE()
-   PyFunction_GET_ANNOTATIONS()
-   PyFunction_GET_CLOSURE()
-   PyFunction_GET_CODE()
-   PyFunction_GET_DEFAULTS()
-   PyFunction_GET_GLOBALS()
-   PyFunction_GET_KW_DEFAULTS()
-   PyFunction_GET_MODULE()
-   PyHeapType_GET_MEMBERS()
-   PyInstanceMethod_GET_FUNCTION()
-   PyList_GET_SIZE()
-   PyMemoryView_GET_BASE()
-   PyMemoryView_GET_BUFFER()
-   PyMethod_GET_FUNCTION()
-   PyMethod_GET_SELF()
-   PySet_GET_SIZE()
-   PyTuple_GET_SIZE()
-   PyUnicode_GET_DATA_SIZE()
-   PyUnicode_GET_LENGTH()
-   PyUnicode_GET_LENGTH()
-   PyUnicode_GET_SIZE()
-   PyWeakref_GET_OBJECT()

AS macros

-   PyByteArray_AS_STRING()
-   PyBytes_AS_STRING()
-   PyFloat_AS_DOUBLE()
-   PyUnicode_AS_DATA()
-   PyUnicode_AS_UNICODE()

PyUnicode macros

-   PyUnicode_1BYTE_DATA()
-   PyUnicode_2BYTE_DATA()
-   PyUnicode_4BYTE_DATA()
-   PyUnicode_DATA()
-   PyUnicode_IS_ASCII()
-   PyUnicode_IS_COMPACT()
-   PyUnicode_IS_READY()
-   PyUnicode_KIND()
-   PyUnicode_READ()
-   PyUnicode_READ_CHAR()

PyDateTime GET macros

-   PyDateTime_DATE_GET_FOLD()
-   PyDateTime_DATE_GET_HOUR()
-   PyDateTime_DATE_GET_MICROSECOND()
-   PyDateTime_DATE_GET_MINUTE()
-   PyDateTime_DATE_GET_SECOND()
-   PyDateTime_DATE_GET_TZINFO()
-   PyDateTime_DELTA_GET_DAYS()
-   PyDateTime_DELTA_GET_MICROSECONDS()
-   PyDateTime_DELTA_GET_SECONDS()
-   PyDateTime_GET_DAY()
-   PyDateTime_GET_MONTH()
-   PyDateTime_GET_YEAR()
-   PyDateTime_TIME_GET_FOLD()
-   PyDateTime_TIME_GET_HOUR()
-   PyDateTime_TIME_GET_MICROSECOND()
-   PyDateTime_TIME_GET_MINUTE()
-   PyDateTime_TIME_GET_SECOND()
-   PyDateTime_TIME_GET_TZINFO()

Port C extensions to Python 3.11

In practice, the majority of projects affected by these PEP only have to
make two changes:

-   Replace Py_TYPE(obj) = new_type with Py_SET_TYPE(obj, new_type).
-   Replace Py_SIZE(obj) = new_size with Py_SET_SIZE(obj, new_size).

The pythoncapi_compat project can be used to update automatically C
extensions: add Python 3.11 support without losing support with older
Python versions. The project provides a header file which provides
Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE() functions to Python 3.8
and older.

PyTuple_GET_ITEM() and PyList_GET_ITEM() are left unchanged

The PyTuple_GET_ITEM() and PyList_GET_ITEM() macros are left unchanged.

The code patterns &PyTuple_GET_ITEM(tuple, 0) and
&PyList_GET_ITEM(list, 0) are still commonly used to get access to the
inner PyObject** array.

Changing these macros is out of the scope of this PEP.

PyDescr_NAME() and PyDescr_TYPE() are left unchanged

The PyDescr_NAME() and PyDescr_TYPE() macros are left unchanged.

These macros give access to PyDescrObject.d_name and
PyDescrObject.d_type members. They can be used as l-values to set these
members.

The SWIG project uses these macros as l-values to set these members. It
would be possible to modify SWIG to prevent setting PyDescrObject
structure members directly, but it is not really worth it since the
PyDescrObject structure is not performance critical and is unlikely to
change soon.

See the bpo-46538 "[C API] Make the PyDescrObject structure opaque:
PyDescr_NAME() and PyDescr_TYPE()" issue for more details.

Implementation

The implementation is tracked by bpo-45476: [C API] PEP 674: Disallow
using macros as l-values.

Py_TYPE() and Py_SIZE() macros

In May 2020, the Py_TYPE() and Py_SIZE() macros have been modified to
disallow using them as l-values (Py_TYPE, Py_SIZE).

In November 2020, the change was reverted, since it broke too many third
party projects.

In June 2021, once most third party projects were updated, a second
attempt was done, but had to be reverted again , since it broke
test_exceptions on Windows.

In September 2021, once test_exceptions has been fixed, Py_TYPE() and
Py_SIZE() were finally changed.

In November 2021, this backward incompatible change got a Steering
Council exception.

In October 2022, Python 3.11 got released with Py_TYPE() and Py_SIZE()
incompatible changes.

Backwards Compatibility

The proposed C API changes are backward incompatible on purpose.

In practice, only Py_TYPE() and Py_SIZE() macros are used as l-values.

This change does not follow the PEP 387 deprecation process. There is no
known way to emit a deprecation warning only when a macro is used as an
l-value, but not when it's used differently (ex: as a r-value).

The following 4 macros are left unchanged to reduce the number of
affected projects: PyDescr_NAME(), PyDescr_TYPE(), PyList_GET_ITEM() and
PyTuple_GET_ITEM().

Statistics

In total (projects on PyPI and not on PyPI), 34 projects are known to be
affected by this PEP:

-   16 projects (47%) are already fixed
-   18 projects (53%) are not fixed yet (pending fix or have to
    regenerate their Cython code)

On September 1, 2022, the PEP affects 18 projects (0.4%) of the top 5000
PyPI projects:

-   15 projects (0.3%) have to regenerate their Cython code
-   3 projects (0.1%) have a pending fix

Top 5000 PyPI

Projects with a pending fix (3):

-   datatable (1.0.0): fixed
-   guppy3 (3.1.2): fixed
-   scipy (1.9.3): need to update boost python

Moreover, 15 projects have to regenerate their Cython code.

Projects released with a fix (12):

-   bitarray (1.6.2): commit
-   Cython (0.29.20): commit
-   immutables (0.15): commit
-   mercurial (5.7): commit, bug report
-   mypy (v0.930): commit
-   numpy (1.22.1): commit, commit 2
-   pycurl (7.44.1): commit
-   PyGObject (3.42.0)
-   pyside2 (5.15.1): bug report
-   python-snappy (0.6.1): fixed
-   recordclass (0.17.2): fixed
-   zstd (<IP_ADDRESS>): commit

There are also two backport projects which are affected by this PEP:

-   pickle5 (0.0.12): backport for Python <= 3.7
-   pysha3 (1.0.2): backport for Python <= 3.5

They must not be used and cannot be used on Python 3.11.

Other affected projects

Other projects released with a fix (4):

-   boost (1.78.0): commit
-   breezy (3.2.1): bug report
-   duplicity (0.8.18): commit
-   gobject-introspection (1.70.0): MR

Relationship with the HPy project

The HPy project

The hope with the HPy project is to provide a C API that is close to the
original API—to make porting easy—and have it perform as close to the
existing API as possible. At the same time, HPy is sufficiently removed
to be a good "C extension API" (as opposed to a stable subset of the
CPython implementation API) that does not leak implementation details.
To ensure this latter property, the HPy project tries to develop
everything in parallel for CPython, PyPy, and GraalVM Python.

HPy is still evolving very fast. Issues are still being solved while
migrating NumPy, and work has begun on adding support for HPy to Cython.
Work on pybind11 is starting soon. Tim Felgentreff believes by the time
HPy has these users of the existing C API working, HPy should be in a
state where it is generally useful and can be deemed stable enough that
further development can follow a more stable process.

In the long run the HPy project would like to become a promoted API to
write Python C extensions.

The HPy project is a good solution for the long term. It has the
advantage of being developed outside Python and it doesn't require any C
API change.

The C API is here is stay for a few more years

The first concern about HPy is that right now, HPy is not mature nor
widely used, and CPython still has to continue supporting a large amount
of C extensions which are not likely to be ported to HPy soon.

The second concern is the inability to evolve CPython internals to
implement new optimizations, and the inefficient implementation of the
current C API in PyPy, GraalPython, etc. Sadly, HPy will only solve
these problems when most C extensions will be fully ported to HPy: when
it will become reasonable to consider dropping the "legacy" Python C
API.

While porting a C extension to HPy can be done incrementally on CPython,
it requires to modify a lot of code and takes time. Porting most C
extensions to HPy is expected to take a few years.

This PEP proposes to make the C API "less bad" by fixing one problem
which is clearily identified as causing practical issues: macros used as
l-values. This PEP only requires updating a minority of C extensions,
and usually only a few lines need to be changed in impacted extensions.

For example, NumPy 1.22 is made of 307,300 lines of C code, and adapting
NumPy to the this PEP only modified 11 lines (use Py_SET_TYPE and
Py_SET_SIZE) and adding 4 lines (to define Py_SET_TYPE and Py_SET_SIZE
for Python 3.8 and older). The beginnings of the NumPy port to HPy
already required modifying more lines than that.

Right now, it's hard to bet which approach is the best: fixing the
current C API, or focusing on HPy. It would be risky to only focus on
HPy.

Rejected Idea: Leave the macros as they are

The documentation of each function can discourage developers to use
macros to modify Python objects.

If these is a need to make an assignment, a setter function can be added
and the macro documentation can require to use the setter function. For
example, a Py_SET_TYPE() function has been added to Python 3.9 and the
Py_TYPE() documentation now requires to use the Py_SET_TYPE() function
to set an object type.

If developers use macros as an l-value, it's their responsibility when
their code breaks, not Python's responsibility. We are operating under
the consenting adults principle: we expect users of the Python C API to
use it as documented and expect them to take care of the fallout, if
things break when they don't.

This idea was rejected because only few developers read the
documentation, and only a minority is tracking changes of the Python C
API documentation. The majority of developers are only using CPython and
so are not aware of compatibility issues with other Python
implementations.

Moreover, continuing to allow using macros as an l-value does not help
the HPy project, and leaves the burden of emulating them on GraalVM's
Python implementation.

Macros already modified

The following C API macros have already been modified to disallow using
them as l-value:

-   PyCell_SET()
-   PyList_SET_ITEM()
-   PyTuple_SET_ITEM()
-   Py_REFCNT() (Python 3.10): Py_SET_REFCNT() must be used
-   _PyGCHead_SET_FINALIZED()
-   _PyGCHead_SET_NEXT()
-   asdl_seq_GET()
-   asdl_seq_GET_UNTYPED()
-   asdl_seq_LEN()
-   asdl_seq_SET()
-   asdl_seq_SET_UNTYPED()

For example, PyList_SET_ITEM(list, 0, item) < 0 now fails with a
compiler error as expected.

Post History

-   PEP 674 "Disallow using macros as l-values" and Python 3.11 (August
    18, 2022)
-   SC reply to PEP 674 -- Disallow using macros as l-values (February
    22, 2022)
-   PEP 674: Disallow using macros as l-value (version 2) (Jan 18, 2022)
-   PEP 674: Disallow using macros as l-value (Nov 30, 2021)

References

-   Python C API: Add functions to access PyObject (October
    2021) article by Victor Stinner
-   [capi-sig] Py_TYPE() and Py_SIZE() become static inline functions
    (September 2021)
-   [C API] Avoid accessing PyObject and PyVarObject members directly:
    add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type
    (February 2020)
-   bpo-30459: PyList_SET_ITEM could be safer (May 2017)

Version History

-   Version 3: No longer change PyDescr_TYPE() and PyDescr_NAME() macros
-   Version 2: Add "Relationship with the HPy project" section, remove
    the PyPy section
-   Version 1: First public version

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 563 Title: Postponed Evaluation of Annotations Version: $Revision$
Last-Modified: $Date$ Author: Łukasz Langa<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Accepted Type: Standards
Track Topic: Typing Content-Type: text/x-rst Created: 08-Sep-2017
Python-Version: 3.7 Post-History: 01-Nov-2017, 21-Nov-2017
Superseded-By: 649 Resolution:
https://mail.python.org/pipermail/python-dev/2017-December/151042.html

Abstract

PEP 3107 introduced syntax for function annotations, but the semantics
were deliberately left undefined. PEP 484 introduced a standard meaning
to annotations: type hints. PEP 526 defined variable annotations,
explicitly tying them with the type hinting use case.

This PEP proposes changing function annotations and variable annotations
so that they are no longer evaluated at function definition time.
Instead, they are preserved in __annotations__ in string form.

This change is being introduced gradually, starting with a __future__
import in Python 3.7.

Rationale and Goals

PEP 3107 added support for arbitrary annotations on parts of a function
definition. Just like default values, annotations are evaluated at
function definition time. This creates a number of issues for the type
hinting use case:

-   forward references: when a type hint contains names that have not
    been defined yet, that definition needs to be expressed as a string
    literal;
-   type hints are executed at module import time, which is not
    computationally free.

Postponing the evaluation of annotations solves both problems. NOTE: PEP
649 proposes an alternative solution to the above issues, putting this
PEP in danger of being superseded.

Non-goals

Just like in PEP 484 and PEP 526, it should be emphasized that Python
will remain a dynamically typed language, and the authors have no desire
to ever make type hints mandatory, even by convention.

This PEP is meant to solve the problem of forward references in type
annotations. There are still cases outside of annotations where forward
references will require usage of string literals. Those are listed in a
later section of this document.

Annotations without forced evaluation enable opportunities to improve
the syntax of type hints. This idea will require its own separate PEP
and is not discussed further in this document.

Non-typing usage of annotations

While annotations are still available for arbitrary use besides type
checking, it is worth mentioning that the design of this PEP, as well as
its precursors (PEP 484 and PEP 526), is predominantly motivated by the
type hinting use case.

In Python 3.8 PEP 484 will graduate from provisional status. Other
enhancements to the Python programming language like PEP 544, PEP 557,
or PEP 560, are already being built on this basis as they depend on type
annotations and the typing module as defined by PEP 484. In fact, the
reason PEP 484 is staying provisional in Python 3.7 is to enable rapid
evolution for another release cycle that some of the aforementioned
enhancements require.

With this in mind, uses for annotations incompatible with the
aforementioned PEPs should be considered deprecated.

Implementation

With this PEP, function and variable annotations will no longer be
evaluated at definition time. Instead, a string form will be preserved
in the respective __annotations__ dictionary. Static type checkers will
see no difference in behavior, whereas tools using annotations at
runtime will have to perform postponed evaluation.

The string form is obtained from the AST during the compilation step,
which means that the string form might not preserve the exact formatting
of the source. Note: if an annotation was a string literal already, it
will still be wrapped in a string.

Annotations need to be syntactically valid Python expressions, also when
passed as literal strings (i.e. compile(literal, '', 'eval')).
Annotations can only use names present in the module scope as postponed
evaluation using local names is not reliable (with the sole exception of
class-level names resolved by typing.get_type_hints()).

Note that as per PEP 526, local variable annotations are not evaluated
at all since they are not accessible outside of the function's closure.

Enabling the future behavior in Python 3.7

The functionality described above can be enabled starting from Python
3.7 using the following special import:

    from __future__ import annotations

A reference implementation of this functionality is available on GitHub.

Resolving Type Hints at Runtime

To resolve an annotation at runtime from its string form to the result
of the enclosed expression, user code needs to evaluate the string.

For code that uses type hints, the
typing.get_type_hints(obj, globalns=None, localns=None) function
correctly evaluates expressions back from its string form. Note that all
valid code currently using __annotations__ should already be doing that
since a type annotation can be expressed as a string literal.

For code which uses annotations for other purposes, a regular
eval(ann, globals, locals) call is enough to resolve the annotation.

In both cases it's important to consider how globals and locals affect
the postponed evaluation. An annotation is no longer evaluated at the
time of definition and, more importantly, in the same scope where it was
defined. Consequently, using local state in annotations is no longer
possible in general. As for globals, the module where the annotation was
defined is the correct context for postponed evaluation.

The get_type_hints() function automatically resolves the correct value
of globalns for functions and classes. It also automatically provides
the correct localns for classes.

When running eval(), the value of globals can be gathered in the
following way:

-   function objects hold a reference to their respective globals in an
    attribute called __globals__;

-   classes hold the name of the module they were defined in, this can
    be used to retrieve the respective globals:

        cls_globals = vars(sys.modules[SomeClass.__module__])

    Note that this needs to be repeated for base classes to evaluate all
    __annotations__.

-   modules should use their own __dict__.

The value of localns cannot be reliably retrieved for functions because
in all likelihood the stack frame at the time of the call no longer
exists.

For classes, localns can be composed by chaining vars of the given class
and its base classes (in the method resolution order). Since slots can
only be filled after the class was defined, we don't need to consult
them for this purpose.

Runtime annotation resolution and class decorators

Metaclasses and class decorators that need to resolve annotations for
the current class will fail for annotations that use the name of the
current class. Example:

    def class_decorator(cls):
        annotations = get_type_hints(cls)  # raises NameError on 'C'
        print(f'Annotations for {cls}: {annotations}')
        return cls

    @class_decorator
    class C:
        singleton: 'C' = None

This was already true before this PEP. The class decorator acts on the
class before it's assigned a name in the current definition scope.

Runtime annotation resolution and TYPE_CHECKING

Sometimes there's code that must be seen by a type checker but should
not be executed. For such situations the typing module defines a
constant, TYPE_CHECKING, that is considered True during type checking
but False at runtime. Example:

    import typing

    if typing.TYPE_CHECKING:
        import expensive_mod

    def a_func(arg: expensive_mod.SomeClass) -> None:
        a_var: expensive_mod.SomeClass = arg
        ...

This approach is also useful when handling import cycles.

Trying to resolve annotations of a_func at runtime using
typing.get_type_hints() will fail since the name expensive_mod is not
defined (TYPE_CHECKING variable being False at runtime). This was
already true before this PEP.

Backwards Compatibility

This is a backwards incompatible change. Applications depending on
arbitrary objects to be directly present in annotations will break if
they are not using typing.get_type_hints() or eval().

Annotations that depend on locals at the time of the function definition
will not be resolvable later. Example:

    def generate():
        A = Optional[int]
        class C:
            field: A = 1
            def method(self, arg: A) -> None: ...
        return C
    X = generate()

Trying to resolve annotations of X later by using get_type_hints(X) will
fail because A and its enclosing scope no longer exists. Python will
make no attempt to disallow such annotations since they can often still
be successfully statically analyzed, which is the predominant use case
for annotations.

Annotations using nested classes and their respective state are still
valid. They can use local names or the fully qualified name. Example:

    class C:
        field = 'c_field'
        def method(self) -> C.field:  # this is OK
            ...

        def method(self) -> field:  # this is OK
            ...

        def method(self) -> C.D:  # this is OK
            ...

        def method(self) -> D:  # this is OK
            ...

        class D:
            field2 = 'd_field'
            def method(self) -> C.D.field2:  # this is OK
                ...

            def method(self) -> D.field2:  # this FAILS, class D is local to C
                ...                        # and is therefore only available
                                           # as C.D. This was already true
                                           # before the PEP.

            def method(self) -> field2:  # this is OK
                ...

            def method(self) -> field:  # this FAILS, field is local to C and
                                        # is therefore not visible to D unless
                                        # accessed as C.field. This was already
                                        # true before the PEP.

In the presence of an annotation that isn't a syntactically valid
expression, SyntaxError is raised at compile time. However, since names
aren't resolved at that time, no attempt is made to validate whether
used names are correct or not.

Deprecation policy

Starting with Python 3.7, a __future__ import is required to use the
described functionality. No warnings are raised.

NOTE: Whether this will eventually become the default behavior is
currently unclear pending decision on PEP 649. In any case, use of
annotations that depend upon their eager evaluation is incompatible with
both proposals and is no longer supported.

Forward References

Deliberately using a name before it was defined in the module is called
a forward reference. For the purpose of this section, we'll call any
name imported or defined within a if TYPE_CHECKING: block a forward
reference, too.

This PEP addresses the issue of forward references in type annotations.
The use of string literals will no longer be required in this case.
However, there are APIs in the typing module that use other syntactic
constructs of the language, and those will still require working around
forward references with string literals. The list includes:

-   type definitions:

        T = TypeVar('T', bound='<type>')
        UserId = NewType('UserId', '<type>')
        Employee = NamedTuple('Employee', [('name', '<type>'), ('id', '<type>')])

-   aliases:

        Alias = Optional['<type>']
        AnotherAlias = Union['<type>', '<type>']
        YetAnotherAlias = '<type>'

-   casting:

        cast('<type>', value)

-   base classes:

        class C(Tuple['<type>', '<type>']): ...

Depending on the specific case, some of the cases listed above might be
worked around by placing the usage in a if TYPE_CHECKING: block. This
will not work for any code that needs to be available at runtime,
notably for base classes and casting. For named tuples, using the new
class definition syntax introduced in Python 3.6 solves the issue.

In general, fixing the issue for all forward references requires
changing how module instantiation is performed in Python, from the
current single-pass top-down model. This would be a major change in the
language and is out of scope for this PEP.

Rejected Ideas

Keeping the ability to use function local state when defining annotations

With postponed evaluation, this would require keeping a reference to the
frame in which an annotation got created. This could be achieved for
example by storing all annotations as lambdas instead of strings.

This would be prohibitively expensive for highly annotated code as the
frames would keep all their objects alive. That includes predominantly
objects that won't ever be accessed again.

To be able to address class-level scope, the lambda approach would
require a new kind of cell in the interpreter. This would proliferate
the number of types that can appear in __annotations__, as well as
wouldn't be as introspectable as strings.

Note that in the case of nested classes, the functionality to get the
effective "globals" and "locals" at definition time is provided by
typing.get_type_hints().

If a function generates a class or a function with annotations that have
to use local variables, it can populate the given generated object's
__annotations__ dictionary directly, without relying on the compiler.

Disallowing local state usage for classes, too

This PEP originally proposed limiting names within annotations to only
allow names from the model-level scope, including for classes. The
author argued this makes name resolution unambiguous, including in cases
of conflicts between local names and module-level names.

This idea was ultimately rejected in case of classes. Instead,
typing.get_type_hints() got modified to populate the local namespace
correctly if class-level annotations are needed.

The reasons for rejecting the idea were that it goes against the
intuition of how scoping works in Python, and would break enough
existing type annotations to make the transition cumbersome. Finally,
local scope access is required for class decorators to be able to
evaluate type annotations. This is because class decorators are applied
before the class receives its name in the outer scope.

Introducing a new dictionary for the string literal form instead

Yury Selivanov shared the following idea:

1.  Add a new special attribute to functions: __annotations_text__.
2.  Make __annotations__ a lazy dynamic mapping, evaluating expressions
    from the corresponding key in __annotations_text__ just-in-time.

This idea is supposed to solve the backwards compatibility issue,
removing the need for a new __future__ import. Sadly, this is not
enough. Postponed evaluation changes which state the annotation has
access to. While postponed evaluation fixes the forward reference
problem, it also makes it impossible to access function-level locals
anymore. This alone is a source of backwards incompatibility which
justifies a deprecation period.

A __future__ import is an obvious and explicit indicator of opting in
for the new functionality. It also makes it trivial for external tools
to recognize the difference between a Python files using the old or the
new approach. In the former case, that tool would recognize that local
state access is allowed, whereas in the latter case it would recognize
that forward references are allowed.

Finally, just-in-time evaluation in __annotations__ is an unnecessary
step if get_type_hints() is used later.

Dropping annotations with -O

There are two reasons this is not satisfying for the purpose of this
PEP.

First, this only addresses runtime cost, not forward references, those
still cannot be safely used in source code. A library maintainer would
never be able to use forward references since that would force the
library users to use this new hypothetical -O switch.

Second, this throws the baby out with the bath water. Now no runtime
annotation use can be performed. PEP 557 is one example of a recent
development where evaluating type annotations at runtime is useful.

All that being said, a granular -O option to drop annotations is a
possibility in the future, as it's conceptually compatible with existing
-O behavior (dropping docstrings and assert statements). This PEP does
not invalidate the idea.

Passing string literals in annotations verbatim to __annotations__

This PEP originally suggested directly storing the contents of a string
literal under its respective key in __annotations__. This was meant to
simplify support for runtime type checkers.

Mark Shannon pointed out this idea was flawed since it wasn't handling
situations where strings are only part of a type annotation.

The inconsistency of it was always apparent but given that it doesn't
fully prevent cases of double-wrapping strings anyway, it is not worth
it.

Making the name of the future import more verbose

Instead of requiring the following import:

    from __future__ import annotations

the PEP could call the feature more explicitly, for example
string_annotations, stringify_annotations, annotation_strings,
annotations_as_strings, lazy_annotations, static_annotations, etc.

The problem with those names is that they are very verbose. Each of them
besides lazy_annotations would constitute the longest future feature
name in Python. They are long to type and harder to remember than the
single-word form.

There is precedence of a future import name that sounds overly generic
but in practice was obvious to users as to what it does:

    from __future__ import division

Prior discussion

In PEP 484

The forward reference problem was discussed when PEP 484 was originally
drafted, leading to the following statement in the document:

  A compromise is possible where a __future__ import could enable
  turning all annotations in a given module into string literals, as
  follows:

      from __future__ import annotations

      class ImSet:
          def add(self, a: ImSet) -> List[ImSet]: ...

      assert ImSet.add.__annotations__ == {
          'a': 'ImSet', 'return': 'List[ImSet]'
      }

  Such a __future__ import statement may be proposed in a separate PEP.

python/typing#400

The problem was discussed at length on the typing module's GitHub
project, under Issue 400. The problem statement there includes critique
of generic types requiring imports from typing. This tends to be
confusing to beginners:

  Why this:

      from typing import List, Set
      def dir(o: object = ...) -> List[str]: ...
      def add_friends(friends: Set[Friend]) -> None: ...

  But not this:

      def dir(o: object = ...) -> list[str]: ...
      def add_friends(friends: set[Friend]) -> None ...

  Why this:

      up_to_ten = list(range(10))
      friends = set()

  But not this:

      from typing import List, Set
      up_to_ten = List[int](range(10))
      friends = Set[Friend]()

While typing usability is an interesting problem, it is out of scope of
this PEP. Specifically, any extensions of the typing syntax standardized
in PEP 484 will require their own respective PEPs and approval.

Issue 400 ultimately suggests postponing evaluation of annotations and
keeping them as strings in __annotations__, just like this PEP
specifies. This idea was received well. Ivan Levkivskyi supported using
the __future__ import and suggested unparsing the AST in compile.c.
Jukka Lehtosalo pointed out that there are some cases of forward
references where types are used outside of annotations and postponed
evaluation will not help those. For those cases using the string literal
notation would still be required. Those cases are discussed briefly in
the "Forward References" section of this PEP.

The biggest controversy on the issue was Guido van Rossum's concern that
untokenizing annotation expressions back to their string form has no
precedent in the Python programming language and feels like a hacky
workaround. He said:

  One thing that comes to mind is that it's a very random change to the
  language. It might be useful to have a more compact way to indicate
  deferred execution of expressions (using less syntax than lambda:).
  But why would the use case of type annotations be so all-important to
  change the language to do it there first (rather than proposing a more
  general solution), given that there's already a solution for this
  particular use case that requires very minimal syntax?

Eventually, Ethan Smith and schollii voiced that feedback gathered
during PyCon US suggests that the state of forward references needs
fixing. Guido van Rossum suggested coming back to the __future__ idea,
pointing out that to prevent abuse, it's important for the annotations
to be kept both syntactically valid and evaluating correctly at runtime.

First draft discussion on python-ideas

Discussion happened largely in two threads, the original announcement
and a follow-up called PEP 563 and expensive backwards compatibility.

The PEP received rather warm feedback (4 strongly in favor, 2 in favor
with concerns, 2 against). The biggest voice of concern on the former
thread being Steven D'Aprano's review stating that the problem
definition of the PEP doesn't justify breaking backwards compatibility.
In this response Steven seemed mostly concerned about Python no longer
supporting evaluation of annotations that depended on local
function/class state.

A few people voiced concerns that there are libraries using annotations
for non-typing purposes. However, none of the named libraries would be
invalidated by this PEP. They do require adapting to the new requirement
to call eval() on the annotation with the correct globals and locals
set.

This detail about globals and locals having to be correct was picked up
by a number of commenters. Alyssa (Nick) Coghlan benchmarked turning
annotations into lambdas instead of strings, sadly this proved to be
much slower at runtime than the current situation.

The latter thread was started by Jim J. Jewett who stressed that the
ability to properly evaluate annotations is an important requirement and
backwards compatibility in that regard is valuable. After some
discussion he admitted that side effects in annotations are a code smell
and modal support to either perform or not perform evaluation is a messy
solution. His biggest concern remained loss of functionality stemming
from the evaluation restrictions on global and local scope.

Alyssa Coghlan pointed out that some of those evaluation restrictions
from the PEP could be lifted by a clever implementation of an evaluation
helper, which could solve self-referencing classes even in the form of a
class decorator. She suggested the PEP should provide this helper
function in the standard library.

Second draft discussion on python-dev

Discussion happened mainly in the announcement thread, followed by a
brief discussion under Mark Shannon's post.

Steven D'Aprano was concerned whether it's acceptable for typos to be
allowed in annotations after the change proposed by the PEP. Brett
Cannon responded that type checkers and other static analyzers (like
linters or programming text editors) will catch this type of error.
Jukka Lehtosalo added that this situation is analogous to how names in
function bodies are not resolved until the function is called.

A major topic of discussion was Alyssa Coghlan's suggestion to store
annotations in "thunk form", in other words as a specialized lambda
which would be able to access class-level scope (and allow for scope
customization at call time). He presented a possible design for it
(indirect attribute cells). This was later seen as equivalent to
"special forms" in Lisp. Guido van Rossum expressed worry that this sort
of feature cannot be safely implemented in twelve weeks (i.e. in time
before the Python 3.7 beta freeze).

After a while it became clear that the point of division between
supporters of the string form vs. supporters of the thunk form is
actually about whether annotations should be perceived as a general
syntactic element vs. something tied to the type checking use case.

Finally, Guido van Rossum declared he's rejecting the thunk idea based
on the fact that it would require a new building block in the
interpreter. This block would be exposed in annotations, multiplying
possible types of values stored in __annotations__ (arbitrary objects,
strings, and now thunks). Moreover, thunks aren't as introspectable as
strings. Most importantly, Guido van Rossum explicitly stated interest
in gradually restricting the use of annotations to static typing (with
an optional runtime component).

Alyssa Coghlan got convinced to PEP 563, too, promptly beginning the
mandatory bike shedding session on the name of the __future__ import.
Many debaters agreed that annotations seems like an overly broad name
for the feature name. Guido van Rossum briefly decided to call it
string_annotations but then changed his mind, arguing that division is a
precedent of a broad name with a clear meaning.

The final improvement to the PEP suggested in the discussion by Mark
Shannon was the rejection of the temptation to pass string literals
through to __annotations__ verbatim.

A side-thread of discussion started around the runtime penalty of static
typing, with topic like the import time of the typing module (which is
comparable to re without dependencies, and three times as heavy as re
when counting dependencies).

Acknowledgements

This document could not be completed without valuable input,
encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and
Ivan Levkivskyi.

The implementation was thoroughly reviewed by Serhiy Storchaka who found
all sorts of issues, including bugs, bad readability, and performance
problems.

Copyright

This document has been placed in the public domain.