text:
PEP: 526 Title: Syntax for Variable Annotations Version: $Revision$
Last-Modified: $Date$ Author: Ryan Gonzalez<EMAIL_ADDRESS>Philip
House<EMAIL_ADDRESS>Ivan Levkivskyi<EMAIL_ADDRESS>Lisa Roach<EMAIL_ADDRESS>Guido van Rossum<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Typing Content-Type:
text/x-rst Created: 09-Aug-2016 Python-Version: 3.6 Post-History:
30-Aug-2016, 02-Sep-2016 Resolution:
https://mail.python.org/pipermail/python-dev/2016-September/146282.html

python:annassign, typing:classvar and typing.ClassVar

Status

This PEP has been provisionally accepted by the BDFL. See the acceptance
message for more color:
https://mail.python.org/pipermail/python-dev/2016-September/146282.html

Notice for Reviewers

This PEP was drafted in a separate repo:
https://github.com/phouse512/peps/tree/pep-0526.

There was preliminary discussion on python-ideas and at
https://github.com/python/typing/issues/258.

Before you bring up an objection in a public forum please at least read
the summary of rejected ideas listed at the end of this PEP.

Abstract

PEP 484 introduced type hints, a.k.a. type annotations. While its main
focus was function annotations, it also introduced the notion of type
comments to annotate variables:

    # 'primes' is a list of integers
    primes = []  # type: List[int]

    # 'captain' is a string (Note: initial value is a problem)
    captain = ...  # type: str

    class Starship:
        # 'stats' is a class variable
        stats = {}  # type: Dict[str, int]

This PEP aims at adding syntax to Python for annotating the types of
variables (including class variables and instance variables), instead of
expressing them through comments:

    primes: List[int] = []

    captain: str  # Note: no initial value!

    class Starship:
        stats: ClassVar[Dict[str, int]] = {}

PEP 484 explicitly states that type comments are intended to help with
type inference in complex cases, and this PEP does not change this
intention. However, since in practice type comments have also been
adopted for class variables and instance variables, this PEP also
discusses the use of type annotations for those variables.

Rationale

Although type comments work well enough, the fact that they're expressed
through comments has some downsides:

-   Text editors often highlight comments differently from type
    annotations.

-   There's no way to annotate the type of an undefined variable; one
    needs to initialize it to None (e.g. a = None # type: int).

-   Variables annotated in a conditional branch are difficult to read:

        if some_value:
            my_var = function() # type: Logger
        else:
            my_var = another_function() # Why isn't there a type here?

-   Since type comments aren't actually part of the language, if a
    Python script wants to parse them, it requires a custom parser
    instead of just using ast.

-   Type comments are used a lot in typeshed. Migrating typeshed to use
    the variable annotation syntax instead of type comments would
    improve readability of stubs.

-   In situations where normal comments and type comments are used
    together, it is difficult to distinguish them:

        path = None  # type: Optional[str]  # Path to module source

-   It's impossible to retrieve the annotations at runtime outside of
    attempting to find the module's source code and parse it at runtime,
    which is inelegant, to say the least.

The majority of these issues can be alleviated by making the syntax a
core part of the language. Moreover, having a dedicated annotation
syntax for class and instance variables (in addition to method
annotations) will pave the way to static duck-typing as a complement to
nominal typing defined by PEP 484.

Non-goals

While the proposal is accompanied by an extension of the
typing.get_type_hints standard library function for runtime retrieval of
annotations, variable annotations are not designed for runtime type
checking. Third party packages will have to be developed to implement
such functionality.

It should also be emphasized that Python will remain a dynamically typed
language, and the authors have no desire to ever make type hints
mandatory, even by convention. Type annotations should not be confused
with variable declarations in statically typed languages. The goal of
annotation syntax is to provide an easy way to specify structured type
metadata for third party tools.

This PEP does not require type checkers to change their type checking
rules. It merely provides a more readable syntax to replace type
comments.

Specification

Type annotation can be added to an assignment statement or to a single
expression indicating the desired type of the annotation target to a
third party type checker:

    my_var: int
    my_var = 5  # Passes type check.
    other_var: int  = 'a'  # Flagged as error by type checker,
                           # but OK at runtime.

This syntax does not introduce any new semantics beyond PEP 484, so that
the following three statements are equivalent:

    var = value # type: annotation
    var: annotation; var = value
    var: annotation = value

Below we specify the syntax of type annotations in different contexts
and their runtime effects.

We also suggest how type checkers might interpret annotations, but
compliance to these suggestions is not mandatory. (This is in line with
the attitude towards compliance in PEP 484.)

Global and local variable annotations

The types of locals and globals can be annotated as follows:

    some_number: int           # variable without initial value
    some_list: List[int] = []  # variable with initial value

Being able to omit the initial value allows for easier typing of
variables assigned in conditional branches:

    sane_world: bool
    if 2+2 == 4:
        sane_world = True
    else:
        sane_world = False

Note that, although the syntax does allow tuple packing, it does not
allow one to annotate the types of variables when tuple unpacking is
used:

    # Tuple packing with variable annotation syntax
    t: Tuple[int, ...] = (1, 2, 3)
    # or
    t: Tuple[int, ...] = 1, 2, 3  # This only works in Python 3.8+

    # Tuple unpacking with variable annotation syntax
    header: str
    kind: int
    body: Optional[List[str]]
    header, kind, body = message

Omitting the initial value leaves the variable uninitialized:

    a: int
    print(a)  # raises NameError

However, annotating a local variable will cause the interpreter to
always make it a local:

    def f():
        a: int
        print(a)  # raises UnboundLocalError
        # Commenting out the a: int makes it a NameError.

as if the code were:

    def f():
        if False: a = 0
        print(a)  # raises UnboundLocalError

Duplicate type annotations will be ignored. However, static type
checkers may issue a warning for annotations of the same variable by a
different type:

    a: int
    a: str  # Static type checker may or may not warn about this.

Class and instance variable annotations

Type annotations can also be used to annotate class and instance
variables in class bodies and methods. In particular, the value-less
notation a: int allows one to annotate instance variables that should be
initialized in __init__ or __new__. The proposed syntax is as follows:

    class BasicStarship:
        captain: str = 'Picard'               # instance variable with default
        damage: int                           # instance variable without default
        stats: ClassVar[Dict[str, int]] = {}  # class variable

Here ClassVar is a special class defined by the typing module that
indicates to the static type checker that this variable should not be
set on instances.

Note that a ClassVar parameter cannot include any type variables,
regardless of the level of nesting: ClassVar[T] and
ClassVar[List[Set[T]]] are both invalid if T is a type variable.

This could be illustrated with a more detailed example. In this class:

    class Starship:
        captain = 'Picard'
        stats = {}

        def __init__(self, damage, captain=None):
            self.damage = damage
            if captain:
                self.captain = captain  # Else keep the default

        def hit(self):
            Starship.stats['hits'] = Starship.stats.get('hits', 0) + 1

stats is intended to be a class variable (keeping track of many
different per-game statistics), while captain is an instance variable
with a default value set in the class. This difference might not be seen
by a type checker: both get initialized in the class, but captain serves
only as a convenient default value for the instance variable, while
stats is truly a class variable -- it is intended to be shared by all
instances.

Since both variables happen to be initialized at the class level, it is
useful to distinguish them by marking class variables as annotated with
types wrapped in ClassVar[...]. In this way a type checker may flag
accidental assignments to attributes with the same name on instances.

For example, annotating the discussed class:

    class Starship:
        captain: str = 'Picard'
        damage: int
        stats: ClassVar[Dict[str, int]] = {}

        def __init__(self, damage: int, captain: str = None):
            self.damage = damage
            if captain:
                self.captain = captain  # Else keep the default

        def hit(self):
            Starship.stats['hits'] = Starship.stats.get('hits', 0) + 1

    enterprise_d = Starship(3000)
    enterprise_d.stats = {} # Flagged as error by a type checker
    Starship.stats = {} # This is OK

As a matter of convenience (and convention), instance variables can be
annotated in __init__ or other methods, rather than in the class:

    from typing import Generic, TypeVar
    T = TypeVar('T')

    class Box(Generic[T]):
        def __init__(self, content):
            self.content: T = content

Annotating expressions

The target of the annotation can be any valid single assignment target,
at least syntactically (it is up to the type checker what to do with
this):

    class Cls:
        pass

    c = Cls()
    c.x: int = 0  # Annotates c.x with int.
    c.y: int      # Annotates c.y with int.

    d = {}
    d['a']: int = 0  # Annotates d['a'] with int.
    d['b']: int      # Annotates d['b'] with int.

Note that even a parenthesized name is considered an expression, not a
simple name:

    (x): int      # Annotates x with int, (x) treated as expression by compiler.
    (y): int = 0  # Same situation here.

Where annotations aren't allowed

It is illegal to attempt to annotate variables subject to global or
nonlocal in the same function scope:

    def f():
        global x: int  # SyntaxError

    def g():
        x: int  # Also a SyntaxError
        global x

The reason is that global and nonlocal don't own variables; therefore,
the type annotations belong in the scope owning the variable.

Only single assignment targets and single right hand side values are
allowed. In addition, one cannot annotate variables used in a for or
with statement; they can be annotated ahead of time, in a similar manner
to tuple unpacking:

    a: int
    for a in my_iter:
        ...

    f: MyFile
    with myfunc() as f:
        ...

Variable annotations in stub files

As variable annotations are more readable than type comments, they are
preferred in stub files for all versions of Python, including Python
2.7. Note that stub files are not executed by Python interpreters, and
therefore using variable annotations will not lead to errors. Type
checkers should support variable annotations in stubs for all versions
of Python. For example:

    # file lib.pyi

    ADDRESS: unicode = ...

    class Error:
        cause: Union[str, unicode]

Preferred coding style for variable annotations

Annotations for module level variables, class and instance variables,
and local variables should have a single space after corresponding
colon. There should be no space before the colon. If an assignment has
right hand side, then the equality sign should have exactly one space on
both sides. Examples:

-   Yes:

        code: int

        class Point:
            coords: Tuple[int, int]
            label: str = '<unknown>'

-   No:

        code:int  # No space after colon
        code : int  # Space before colon

        class Test:
            result: int=0  # No spaces around equality sign

Changes to Standard Library and Documentation

-   A new covariant type ClassVar[T_co] is added to the typing module.
    It accepts only a single argument that should be a valid type, and
    is used to annotate class variables that should not be set on class
    instances. This restriction is ensured by static checkers, but not
    at runtime. See the classvar section for examples and explanations
    for the usage of ClassVar, and see the rejected section for more
    information on the reasoning behind ClassVar.
-   Function get_type_hints in the typing module will be extended, so
    that one can retrieve type annotations at runtime from modules and
    classes as well as functions. Annotations are returned as a
    dictionary mapping from variable or arguments to their type hints
    with forward references evaluated. For classes it returns a mapping
    (perhaps collections.ChainMap) constructed from annotations in
    method resolution order.
-   Recommended guidelines for using annotations will be added to the
    documentation, containing a pedagogical recapitulation of
    specifications described in this PEP and in PEP 484. In addition, a
    helper script for translating type comments into type annotations
    will be published separately from the standard library.

Runtime Effects of Type Annotations

Annotating a local variable will cause the interpreter to treat it as a
local, even if it was never assigned to. Annotations for local variables
will not be evaluated:

    def f():
        x: NonexistentName  # No error.

However, if it is at a module or class level, then the type will be
evaluated:

    x: NonexistentName  # Error!
    class X:
        var: NonexistentName  # Error!

In addition, at the module or class level, if the item being annotated
is a simple name, then it and the annotation will be stored in the
__annotations__ attribute of that module or class (mangled if private)
as an ordered mapping from names to evaluated annotations. Here is an
example:

    from typing import Dict
    class Player:
        ...
    players: Dict[str, Player]
    __points: int

    print(__annotations__)
    # prints: {'players': typing.Dict[str, __main__.Player],
    #          '_Player__points': <class 'int'>}

__annotations__ is writable, so this is permitted:

    __annotations__['s'] = str

But attempting to update __annotations__ to something other than an
ordered mapping may result in a TypeError:

    class C:
        __annotations__ = 42
        x: int = 5  # raises TypeError

(Note that the assignment to __annotations__, which is the culprit, is
accepted by the Python interpreter without questioning it -- but the
subsequent type annotation expects it to be a MutableMapping and will
fail.)

The recommended way of getting annotations at runtime is by using
typing.get_type_hints function; as with all dunder attributes, any
undocumented use of __annotations__ is subject to breakage without
warning:

    from typing import Dict, ClassVar, get_type_hints
    class Starship:
        hitpoints: int = 50
        stats: ClassVar[Dict[str, int]] = {}
        shield: int = 100
        captain: str
        def __init__(self, captain: str) -> None:
            ...

    assert get_type_hints(Starship) == {'hitpoints': int,
                                        'stats': ClassVar[Dict[str, int]],
                                        'shield': int,
                                        'captain': str}

    assert get_type_hints(Starship.__init__) == {'captain': str,
                                                 'return': None}

Note that if annotations are not found statically, then the
__annotations__ dictionary is not created at all. Also the value of
having annotations available locally does not offset the cost of having
to create and populate the annotations dictionary on every function
call. Therefore, annotations at function level are not evaluated and not
stored.

Other uses of annotations

While Python with this PEP will not object to:

    alice: 'well done' = 'A+'
    bob: 'what a shame' = 'F-'

since it will not care about the type annotation beyond "it evaluates
without raising", a type checker that encounters it will flag it, unless
disabled with # type: ignore or @no_type_check.

However, since Python won't care what the "type" is, if the above
snippet is at the global level or in a class, __annotations__ will
include {'alice': 'well done', 'bob': 'what a shame'}.

These stored annotations might be used for other purposes, but with this
PEP we explicitly recommend type hinting as the preferred use of
annotations.

Rejected/Postponed Proposals

-   Should we introduce variable annotations at all? Variable
    annotations have already been around for almost two years in the
    form of type comments, sanctioned by PEP 484. They are extensively
    used by third party type checkers (mypy, pytype, PyCharm, etc.) and
    by projects using the type checkers. However, the comment syntax has
    many downsides listed in Rationale. This PEP is not about the need
    for type annotations, it is about what should be the syntax for such
    annotations.

-   Introduce a new keyword: The choice of a good keyword is hard, e.g.
    it can't be var because that is way too common a variable name, and
    it can't be local if we want to use it for class variables or
    globals. Second, no matter what we choose, we'd still need a
    __future__ import.

-   Use def as a keyword: The proposal would be:

        def primes: List[int] = []
        def captain: str

    The problem with this is that def means "define a function" to
    generations of Python programmers (and tools!), and using it also to
    define variables does not increase clarity. (Though this is of
    course subjective.)

-   Use function based syntax: It was proposed to annotate types of
    variables using var = cast(annotation[, value]). Although this
    syntax alleviates some problems with type comments like absence of
    the annotation in AST, it does not solve other problems such as
    readability and it introduces possible runtime overhead.

-   Allow type annotations for tuple unpacking: This causes ambiguity:
    it's not clear what this statement means:

        x, y: T

    Are x and y both of type T, or do we expect T to be a tuple type of
    two items that are distributed over x and y, or perhaps x has type
    Any and y has type T? (The latter is what this would mean if this
    occurred in a function signature.) Rather than leave the (human)
    reader guessing, we forbid this, at least for now.

-   Parenthesized form (var: type) for annotations: It was brought up on
    python-ideas as a remedy for the above-mentioned ambiguity, but it
    was rejected since such syntax would be hairy, the benefits are
    slight, and the readability would be poor.

-   Allow annotations in chained assignments: This has problems of
    ambiguity and readability similar to tuple unpacking, for example
    in:

        x: int = y = 1
        z = w: int = 1

    it is ambiguous, what should the types of y and z be? Also the
    second line is difficult to parse.

-   Allow annotations in with and for statement: This was rejected
    because in for it would make it hard to spot the actual iterable,
    and in with it would confuse the CPython's LL(1) parser.

-   Evaluate local annotations at function definition time: This has
    been rejected by Guido because the placement of the annotation
    strongly suggests that it's in the same scope as the surrounding
    code.

-   Store variable annotations also in function scope: The value of
    having the annotations available locally is just not enough to
    significantly offset the cost of creating and populating the
    dictionary on each function call.

-   Initialize variables annotated without assignment: It was proposed
    on python-ideas to initialize x in x: int to None or to an
    additional special constant like Javascript's undefined. However,
    adding yet another singleton value to the language would needed to
    be checked for everywhere in the code. Therefore, Guido just said
    plain "No" to this.

-   Add also InstanceVar to the typing module: This is redundant because
    instance variables are way more common than class variables. The
    more common usage deserves to be the default.

-   Allow instance variable annotations only in methods: The problem is
    that many __init__ methods do a lot of things besides initializing
    instance variables, and it would be harder (for a human) to find all
    the instance variable annotations. And sometimes __init__ is
    factored into more helper methods so it's even harder to chase them
    down. Putting the instance variable annotations together in the
    class makes it easier to find them, and helps a first-time reader of
    the code.

-   Use syntax x: class t = v for class variables: This would require a
    more complicated parser and the class keyword would confuse
    simple-minded syntax highlighters. Anyway we need to have ClassVar
    store class variables to __annotations__, so a simpler syntax was
    chosen.

-   Forget about ClassVar altogether: This was proposed since mypy seems
    to be getting along fine without a way to distinguish between class
    and instance variables. But a type checker can do useful things with
    the extra information, for example flag accidental assignments to a
    class variable via the instance (which would create an instance
    variable shadowing the class variable). It could also flag instance
    variables with mutable defaults, a well-known hazard.

-   Use ClassAttr instead of ClassVar: The main reason why ClassVar is
    better is following: many things are class attributes, e.g. methods,
    descriptors, etc. But only specific attributes are conceptually
    class variables (or maybe constants).

-   Do not evaluate annotations, treat them as strings: This would be
    inconsistent with the behavior of function annotations that are
    always evaluated. Although this might be reconsidered in future, it
    was decided in PEP 484 that this would have to be a separate PEP.

-   Annotate variable types in class docstring: Many projects already
    use various docstring conventions, often without much consistency
    and generally without conforming to the PEP 484 annotation syntax
    yet. Also this would require a special sophisticated parser. This,
    in turn, would defeat the purpose of the PEP --collaborating with
    the third party type checking tools.

-   Implement __annotations__ as a descriptor: This was proposed to
    prohibit setting __annotations__ to something non-dictionary or
    non-None. Guido has rejected this idea as unnecessary; instead a
    TypeError will be raised if an attempt is made to update
    __annotations__ when it is anything other than a mapping.

-   Treating bare annotations the same as global or nonlocal: The
    rejected proposal would prefer that the presence of an annotation
    without assignment in a function body should not involve any
    evaluation. In contrast, the PEP implies that if the target is more
    complex than a single name, its "left-hand part" should be evaluated
    at the point where it occurs in the function body, just to enforce
    that it is defined. For example, in this example:

        def foo(self):
            slef.name: str

    the name slef should be evaluated, just so that if it is not defined
    (as is likely in this example :-), the error will be caught at
    runtime. This is more in line with what happens when there is an
    initial value, and thus is expected to lead to fewer surprises.
    (Also note that if the target was self.name (this time correctly
    spelled :-), an optimizing compiler has no obligation to evaluate
    self as long as it can prove that it will definitely be defined.)

Backwards Compatibility

This PEP is fully backwards compatible.

Implementation

An implementation for Python 3.6 can be found on GitHub.

Copyright

This document has been placed in the public domain.

authors:
Guido van Rossum
Ivan Levkivskyi
Lisa Roach
Philip House
Ryan Gonzalez


text:
PEP: 3111 Title: Simple input built-in in Python 3000 Version:
$Revision$ Last-Modified: $Date$ Author: Andre Roberge <andre.roberge at
gmail.com> Status: Final Type: Standards Track Content-Type: text/x-rst
Created: 13-Sep-2006 Python-Version: 3.0 Post-History: 22-Dec-2006

Abstract

Input and output are core features of computer programs. Currently,
Python provides a simple means of output through the print keyword and
two simple means of interactive input through the input() and
raw_input() built-in functions.

Python 3.0 will introduce various incompatible changes with previous
Python versions (PEP 3100). Among the proposed changes, print will
become a built-in function, print(), while input() and raw_input() would
be removed completely from the built-in namespace, requiring importing
some module to provide even the most basic input capability.

This PEP proposes that Python 3.0 retains some simple interactive user
input capability, equivalent to raw_input(), within the built-in
namespace.

It was accepted by the BDFL in December 2006[1].

Motivation

With its easy readability and its support for many programming styles
(e.g. procedural, object-oriented, etc.) among others, Python is perhaps
the best computer language to use in introductory programming classes.
Simple programs often need to provide information to the user (output)
and to obtain information from the user (interactive input). Any
computer language intended to be used in an educational setting should
provide straightforward methods for both output and interactive input.

The current proposals for Python 3.0 <3100> include a simple output
pathway via a built-in function named print(), but a more complicated
method for input [e.g. via sys.stdin.readline()], one that requires
importing an external module. Current versions of Python (pre-3.0)
include raw_input() as a built-in function. With the availability of
such a function, programs that require simple input/output can be
written from day one, without requiring discussions of importing
modules, streams, etc.

Rationale

Current built-in functions, like input() and raw_input(), are found to
be extremely useful in traditional teaching settings. (For more details,
see[2] and the discussion that followed.) While the BDFL has clearly
stated[3] that input() was not to be kept in Python 3000, he has also
stated that he was not against revising the decision of killing
raw_input().

raw_input() provides a simple mean to ask a question and obtain a
response from a user. The proposed plans for Python 3.0 would require
the replacement of the single statement:

    name = raw_input("What is your name?")

by the more complicated:

    import sys
    print("What is your name?")
    same = sys.stdin.readline()

However, from the point of view of many Python beginners and educators,
the use of sys.stdin.readline() presents the following problems:

1. Compared to the name "raw_input", the name "sys.stdin.readline()" is
clunky and inelegant.

2. The names "sys" and "stdin" have no meaning for most beginners, who
are mainly interested in what the function does, and not where in the
package structure it is located. The lack of meaning also makes it
difficult to remember: is it "sys.stdin.readline()", or "
stdin.sys.readline()"? To a programming novice, there is not any obvious
reason to prefer one over the other. In contrast, functions simple and
direct names like print, input, and raw_input, and open are easier to
remember.

3. The use of "." notation is unmotivated and confusing to many
beginners. For example, it may lead some beginners to think "." is a
standard character that could be used in any identifier.

4. There is an asymmetry with the print function: why is print not
called sys.stdout.print()?

Specification

The existing raw_input() function will be renamed to input().

The Python 2 to 3 conversion tool will replace calls to input() with
eval(input()) and raw_input() with input().

Naming Discussion

With input() effectively removed from the language, the name raw_input()
makes much less sense and alternatives should be considered. The various
possibilities mentioned in various forums include:

    ask()
    ask_user()
    get_string()
    input()  # initially rejected by BDFL, later accepted
    prompt()
    read()
    user_input()
    get_response()

While it was initially rejected by the BDFL, it has been suggested that
the most direct solution would be to rename "raw_input" to "input" in
Python 3000. The main objection is that Python 2.x already has a
function named "input", and, even though it is not going to be included
in Python 3000, having a built-in function with the same name but
different semantics may confuse programmers migrating from 2.x to 3000.
Certainly, this is no problem for beginners, and the scope of the
problem is unclear for more experienced programmers, since raw_input(),
while popular with many, is not in universal use. In this instance, the
good it does for beginners could be seen to outweigh the harm it does to
experienced programmers -although it could cause confusion for people
reading older books or tutorials.

The rationale for accepting the renaming can be found here[4].

References

Copyright

This document has been placed in the public domain.

[1] BDFL acceptance of the PEP
https://mail.python.org/pipermail/python-3000/2006-December/005257.html

[2] The fate of raw_input() in Python 3000
https://mail.python.org/pipermail/edu-sig/2006-September/006967.html

[3] Educational aspects of Python 3000
https://mail.python.org/pipermail/python-3000/2006-September/003589.html

[4] Rationale for going with the straight renaming
https://mail.python.org/pipermail/python-3000/2006-December/005249.html

license: Public Domain


text:
PEP: 3148 Title: futures - execute computations asynchronously Version:
$Revision$ Last-Modified: $Date$ Author: Brian Quinlan
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 16-Oct-2009 Python-Version: 3.2 Post-History:

Abstract

This PEP proposes a design for a package that facilitates the evaluation
of callables using threads and processes.

Motivation

Python currently has powerful primitives to construct multi-threaded and
multi-process applications but parallelizing simple operations requires
a lot of work i.e. explicitly launching processes/threads, constructing
a work/results queue, and waiting for completion or some other
termination condition (e.g. failure, timeout). It is also difficult to
design an application with a global process/thread limit when each
component invents its own parallel execution strategy.

Specification

Naming

The proposed package would be called "futures" and would live in a new
"concurrent" top-level package. The rationale behind pushing the futures
library into a "concurrent" namespace has multiple components. The
first, most simple one is to prevent any and all confusion with the
existing "from __future__ import x" idiom which has been in use for a
long time within Python. Additionally, it is felt that adding the
"concurrent" precursor to the name fully denotes what the library is
related to - namely concurrency - this should clear up any addition
ambiguity as it has been noted that not everyone in the community is
familiar with Java Futures, or the Futures term except as it relates to
the US stock market.

Finally; we are carving out a new namespace for the standard library
-obviously named "concurrent". We hope to either add, or move existing,
concurrency-related libraries to this in the future. A prime example is
the multiprocessing.Pool work, as well as other "addons" included in
that module, which work across thread and process boundaries.

Interface

The proposed package provides two core classes: Executor and Future. An
Executor receives asynchronous work requests (in terms of a callable and
its arguments) and returns a Future to represent the execution of that
work request.

Executor

Executor is an abstract class that provides methods to execute calls
asynchronously.

submit(fn, *args, **kwargs)

  Schedules the callable to be executed as fn(*args, **kwargs) and
  returns a Future instance representing the execution of the callable.

  This is an abstract method and must be implemented by Executor
  subclasses.

map(func, *iterables, timeout=None)

  Equivalent to map(func, *iterables) but func is executed
  asynchronously and several calls to func may be made concurrently. The
  returned iterator raises a TimeoutError if __next__() is called and
  the result isn't available after timeout seconds from the original
  call to map(). If timeout is not specified or None then there is no
  limit to the wait time. If a call raises an exception then that
  exception will be raised when its value is retrieved from the
  iterator.

shutdown(wait=True)

  Signal the executor that it should free any resources that it is using
  when the currently pending futures are done executing. Calls to
  Executor.submit and Executor.map and made after shutdown will raise
  RuntimeError.

  If wait is True then this method will not return until all the pending
  futures are done executing and the resources associated with the
  executor have been freed. If wait is False then this method will
  return immediately and the resources associated with the executor will
  be freed when all pending futures are done executing. Regardless of
  the value of wait, the entire Python program will not exit until all
  pending futures are done executing.

__enter__()
__exit__(exc_type, exc_val, exc_tb)

  When using an executor as a context manager, __exit__ will call
  Executor.shutdown(wait=True).

ProcessPoolExecutor

The ProcessPoolExecutor class is an Executor subclass that uses a pool
of processes to execute calls asynchronously. The callable objects and
arguments passed to ProcessPoolExecutor.submit must be pickleable
according to the same limitations as the multiprocessing module.

Calling Executor or Future methods from within a callable submitted to a
ProcessPoolExecutor will result in deadlock.

__init__(max_workers)

  Executes calls asynchronously using a pool of a most max_workers
  processes. If max_workers is None or not given then as many worker
  processes will be created as the machine has processors.

ThreadPoolExecutor

The ThreadPoolExecutor class is an Executor subclass that uses a pool of
threads to execute calls asynchronously.

Deadlock can occur when the callable associated with a Future waits on
the results of another Future. For example:

    import time
    def wait_on_b():
        time.sleep(5)
        print(b.result())  # b will never complete because it is waiting on a.
        return 5

    def wait_on_a():
        time.sleep(5)
        print(a.result())  # a will never complete because it is waiting on b.
        return 6

    executor = ThreadPoolExecutor(max_workers=2)
    a = executor.submit(wait_on_b)
    b = executor.submit(wait_on_a)

And:

    def wait_on_future():
        f = executor.submit(pow, 5, 2)
        # This will never complete because there is only one worker thread and
        # it is executing this function.
        print(f.result())

    executor = ThreadPoolExecutor(max_workers=1)
    executor.submit(wait_on_future)

__init__(max_workers)

  Executes calls asynchronously using a pool of at most max_workers
  threads.

Future Objects

The Future class encapsulates the asynchronous execution of a callable.
Future instances are returned by Executor.submit.

cancel()

  Attempt to cancel the call. If the call is currently being executed
  then it cannot be cancelled and the method will return False,
  otherwise the call will be cancelled and the method will return True.

cancelled()

  Return True if the call was successfully cancelled.

running()

  Return True if the call is currently being executed and cannot be
  cancelled.

done()

  Return True if the call was successfully cancelled or finished
  running.

result(timeout=None)

  Return the value returned by the call. If the call hasn't yet
  completed then this method will wait up to timeout seconds. If the
  call hasn't completed in timeout seconds then a TimeoutError will be
  raised. If timeout is not specified or None then there is no limit to
  the wait time.

  If the future is cancelled before completing then CancelledError will
  be raised.

  If the call raised then this method will raise the same exception.

exception(timeout=None)

  Return the exception raised by the call. If the call hasn't yet
  completed then this method will wait up to timeout seconds. If the
  call hasn't completed in timeout seconds then a TimeoutError will be
  raised. If timeout is not specified or None then there is no limit to
  the wait time.

  If the future is cancelled before completing then CancelledError will
  be raised.

  If the call completed without raising then None is returned.

add_done_callback(fn)

  Attaches a callable fn to the future that will be called when the
  future is cancelled or finishes running. fn will be called with the
  future as its only argument.

  Added callables are called in the order that they were added and are
  always called in a thread belonging to the process that added them. If
  the callable raises an Exception then it will be logged and ignored.
  If the callable raises another BaseException then behavior is not
  defined.

  If the future has already completed or been cancelled then fn will be
  called immediately.

Internal Future Methods

The following Future methods are meant for use in unit tests and
Executor implementations.

set_running_or_notify_cancel()

  Should be called by Executor implementations before executing the work
  associated with the Future.

  If the method returns False then the Future was cancelled, i.e.
  Future.cancel was called and returned True. Any threads waiting on the
  Future completing (i.e. through as_completed() or wait()) will be
  woken up.

  If the method returns True then the Future was not cancelled and has
  been put in the running state, i.e. calls to Future.running() will
  return True.

  This method can only be called once and cannot be called after
  Future.set_result() or Future.set_exception() have been called.

set_result(result)

  Sets the result of the work associated with the Future.

set_exception(exception)

  Sets the result of the work associated with the Future to the given
  Exception.

Module Functions

wait(fs, timeout=None, return_when=ALL_COMPLETED)

  Wait for the Future instances (possibly created by different Executor
  instances) given by fs to complete. Returns a named 2-tuple of sets.
  The first set, named "done", contains the futures that completed
  (finished or were cancelled) before the wait completed. The second
  set, named "not_done", contains uncompleted futures.

  timeout can be used to control the maximum number of seconds to wait
  before returning. If timeout is not specified or None then there is no
  limit to the wait time.

  return_when indicates when the method should return. It must be one of
  the following constants:

    Constant          Description
    ----------------- ----------------------------------------------------------------------------------------------------------------------------------------------------
    FIRST_COMPLETED   The method will return when any future finishes or is cancelled.
    FIRST_EXCEPTION   The method will return when any future finishes by raising an exception. If not future raises an exception then it is equivalent to ALL_COMPLETED.
    ALL_COMPLETED     The method will return when all calls finish.

as_completed(fs, timeout=None)

  Returns an iterator over the Future instances given by fs that yields
  futures as they complete (finished or were cancelled). Any futures
  that completed before as_completed() was called will be yielded first.
  The returned iterator raises a TimeoutError if __next__() is called
  and the result isn't available after timeout seconds from the original
  call to as_completed(). If timeout is not specified or None then there
  is no limit to the wait time.

  The Future instances can have been created by different Executor
  instances.

Check Prime Example

    from concurrent import futures
    import math

    PRIMES = [
<PHONE_NUMBER>95293,
<PHONE_NUMBER>42171,
<PHONE_NUMBER>95293,
<PHONE_NUMBER>90773,
<PHONE_NUMBER>77099,
<PHONE_NUMBER>285419]

    def is_prime(n):
        if n % 2 == 0:
            return False

        sqrt_n = int(math.floor(math.sqrt(n)))
        for i in range(3, sqrt_n + 1, 2):
            if n % i == 0:
                return False
        return True

    def main():
        with futures.ProcessPoolExecutor() as executor:
            for number, prime in zip(PRIMES, executor.map(is_prime,
                                                          PRIMES)):
                print('%d is prime: %s' % (number, prime))

    if __name__ == '__main__':
        main()

Web Crawl Example

    from concurrent import futures
    import urllib.request

    URLS = ['http://www.foxnews.com/',
            'http://www.cnn.com/',
            'http://europe.wsj.com/',
            'http://www.bbc.co.uk/',
            'http://some-made-up-domain.com/']

    def load_url(url, timeout):
        return urllib.request.urlopen(url, timeout=timeout).read()

    def main():
        with futures.ThreadPoolExecutor(max_workers=5) as executor:
            future_to_url = dict(
                (executor.submit(load_url, url, 60), url)
                 for url in URLS)

            for future in futures.as_completed(future_to_url):
                url = future_to_url[future]
                try:
                    print('%r page is %d bytes' % (
                              url, len(future.result())))
                except Exception as e:
                    print('%r generated an exception: %s' % (
                              url, e))

    if __name__ == '__main__':
        main()

Rationale

The proposed design of this module was heavily influenced by the Java
java.util.concurrent package[1]. The conceptual basis of the module, as
in Java, is the Future class, which represents the progress and result
of an asynchronous computation. The Future class makes little commitment
to the evaluation mode being used e.g. it can be used to represent lazy
or eager evaluation, for evaluation using threads, processes or remote
procedure call.

Futures are created by concrete implementations of the Executor class
(called ExecutorService in Java). The reference implementation provides
classes that use either a process or a thread pool to eagerly evaluate
computations.

Futures have already been seen in Python as part of a popular Python
cookbook recipe[2] and have discussed on the Python-3000 mailing
list[3].

The proposed design is explicit, i.e. it requires that clients be aware
that they are consuming Futures. It would be possible to design a module
that would return proxy objects (in the style of weakref) that could be
used transparently. It is possible to build a proxy implementation on
top of the proposed explicit mechanism.

The proposed design does not introduce any changes to Python language
syntax or semantics. Special syntax could be introduced[4] to mark
function and method calls as asynchronous. A proxy result would be
returned while the operation is eagerly evaluated asynchronously, and
execution would only block if the proxy object were used before the
operation completed.

Anh Hai Trinh proposed a simpler but more limited API concept[5] and the
API has been discussed in some detail on stdlib-sig[6].

The proposed design was discussed on the Python-Dev mailing list[7].
Following those discussions, the following changes were made:

-   The Executor class was made into an abstract base class
-   The Future.remove_done_callback method was removed due to a lack of
    convincing use cases
-   The Future.add_done_callback method was modified to allow the same
    callable to be added many times
-   The Future class's mutation methods were better documented to
    indicate that they are private to the Executor that created them

Reference Implementation

The reference implementation[8] contains a complete implementation of
the proposed design. It has been tested on Linux and Mac OS X.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] java.util.concurrent package documentation
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/package-summary.html

[2] Python Cookbook recipe 84317, "Easy threading with Futures"
http://code.activestate.com/recipes/84317/

[3] Python-3000 thread, "mechanism for handling asynchronous
concurrency"
https://mail.python.org/pipermail/python-3000/2006-April/000960.html

[4] Python 3000 thread, "Futures in Python 3000 (was Re: mechanism for
handling asynchronous concurrency)"
https://mail.python.org/pipermail/python-3000/2006-April/000970.html

[5] A discussion of stream, a similar concept proposed by Anh Hai Trinh
<EMAIL_ADDRESS>
[6] A discussion of the proposed API on stdlib-sig
https://mail.python.org/pipermail/stdlib-sig/2009-November/000731.html

[7] A discussion of the PEP on python-dev
https://mail.python.org/pipermail/python-dev/2010-March/098169.html

[8] Reference futures implementation
http://code.google.com/p/pythonfutures/source/browse/#svn/branches/feedback


text:
PEP: 511 Title: API for code transformers Version: $Revision$
Last-Modified: $Date$ Author: Victor Stinner<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
04-Jan-2016 Python-Version: 3.6

Rejection Notice

This PEP was rejected by its author.

This PEP was seen as blessing new Python-like programming languages
which are close but incompatible with the regular Python language. It
was decided to not promote syntaxes incompatible with Python.

This PEP was also seen as a nice tool to experiment new Python features,
but it is already possible to experiment them without the PEP, only with
importlib hooks. If a feature becomes useful, it should be directly part
of Python, instead of depending on an third party Python module.

Finally, this PEP was driven was the FAT Python optimization project
which was abandoned in 2016, since it was not possible to show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.

Abstract

Propose an API to register bytecode and AST transformers. Add also
-o OPTIM_TAG command line option to change .pyc filenames, -o noopt
disables the peephole optimizer. Raise an ImportError exception on
import if the .pyc file is missing and the code transformers required to
transform the code are missing. code transformers are not needed code
transformed ahead of time (loaded from .pyc files).

Rationale

Python does not provide a standard way to transform the code. Projects
transforming the code use various hooks. The MacroPy project uses an
import hook: it adds its own module finder in sys.meta_path to hook its
AST transformer. Another option is to monkey-patch the builtin compile()
function. There are even more options to hook a code transformer.

Python 3.4 added a compile_source() method to
importlib.abc.SourceLoader. But code transformation is wider than just
importing modules, see described use cases below.

Writing an optimizer or a preprocessor is out of the scope of this PEP.

Usage 1: AST optimizer

Transforming an Abstract Syntax Tree (AST) is a convenient way to
implement an optimizer. It's easier to work on the AST than working on
the bytecode, AST contains more information and is more high level.

Since the optimization can done ahead of time, complex but slow
optimizations can be implemented.

Example of optimizations which can be implemented with an AST optimizer:

-   Copy propagation: replace x=1; y=x with x=1; y=1
-   Constant folding: replace 1+1 with 2
-   Dead code elimination

Using guards (see PEP 510), it is possible to implement a much wider
choice of optimizations. Examples:

-   Simplify iterable: replace range(3) with (0, 1, 2) when used as
    iterable
-   Loop unrolling
-   Call pure builtins: replace len("abc") with 3
-   Copy used builtin symbols to constants
-   See also optimizations implemented in fatoptimizer, a static
    optimizer for Python 3.6.

The following issues can be implemented with an AST optimizer:

-   Issue #1346238: A constant folding optimization pass for the AST
-   Issue #2181: optimize out local variables at end of function
-   Issue #2499: Fold unary + and not on constants
-   Issue #4264: Patch: optimize code to use LIST_APPEND instead of
    calling list.append
-   Issue #7682: Optimisation of if with constant expression
-   Issue #10399: AST Optimization: inlining of function calls
-   Issue #11549: Build-out an AST optimizer, moving some functionality
    out of the peephole optimizer
-   Issue #17068: peephole optimization for constant strings
-   Issue #17430: missed peephole optimization

Usage 2: Preprocessor

A preprocessor can be easily implemented with an AST transformer. A
preprocessor has various and different usages.

Some examples:

-   Remove debug code like assertions and logs to make the code faster
    to run it for production.
-   Tail-call Optimization
-   Add profiling code
-   Lazy evaluation: see lazy_python (bytecode transformer) and lazy
    macro of MacroPy (AST transformer)
-   Change dictionary literals into collection.OrderedDict instances
-   Declare constants: see @asconstants of codetransformer
-   Domain Specific Language (DSL) like SQL queries. The Python language
    itself doesn't need to be modified. Previous attempts to implement
    DSL for SQL like PEP 335 - Overloadable Boolean
    Operators <335> was rejected.
-   Pattern Matching of functional languages
-   String Interpolation, but PEP 498 was merged into Python 3.6.

MacroPy has a long list of examples and use cases.

This PEP does not add any new code transformer. Using a code transformer
will require an external module and to register it manually.

See also PyXfuscator: Python obfuscator, deobfuscator, and user-assisted
decompiler.

Usage 3: Disable all optimization

Ned Batchelder asked to add an option to disable the peephole optimizer
because it makes code coverage more difficult to implement. See the
discussion on the python-ideas mailing list: Disable all peephole
optimizations.

This PEP adds a new -o noopt command line option to disable the peephole
optimizer. In Python, it's as easy as:

    sys.set_code_transformers([])

It will fix the Issue #2506: Add mechanism to disable optimizations.

Usage 4: Write new bytecode optimizers in Python

Python 3.6 optimizes the code using a peephole optimizer. By definition,
a peephole optimizer has a narrow view of the code and so can only
implement basic optimizations. The optimizer rewrites the bytecode. It
is difficult to enhance it, because it written in C.

With this PEP, it becomes possible to implement a new bytecode optimizer
in pure Python and experiment new optimizations.

Some optimizations are easier to implement on the AST like constant
folding, but optimizations on the bytecode are still useful. For
example, when the AST is compiled to bytecode, useless jumps can be
emitted because the compiler is naive and does not try to optimize
anything.

Use Cases

This section give examples of use cases explaining when and how code
transformers will be used.

Interactive interpreter

It will be possible to use code transformers with the interactive
interpreter which is popular in Python and commonly used to demonstrate
Python.

The code is transformed at runtime and so the interpreter can be slower
when expensive code transformers are used.

Build a transformed package

It will be possible to build a package of the transformed code.

A transformer can have a configuration. The configuration is not stored
in the package.

All .pyc files of the package must be transformed with the same code
transformers and the same transformers configuration.

It is possible to build different .pyc files using different optimizer
tags. Example: fat for the default configuration and fat_inline for a
different configuration with function inlining enabled.

A package can contain .pyc files with different optimizer tags.

Install a package containing transformed .pyc files

It will be possible to install a package which contains transformed .pyc
files.

All .pyc files with any optimizer tag contained in the package are
installed, not only for the current optimizer tag.

Build .pyc files when installing a package

If a package does not contain any .pyc files of the current optimizer
tag (or some .pyc files are missing), the .pyc are created during the
installation.

Code transformers of the optimizer tag are required. Otherwise, the
installation fails with an error.

Execute transformed code

It will be possible to execute transformed code.

Raise an ImportError exception on import if the .pyc file of the current
optimizer tag is missing and the code transformers required to transform
the code are missing.

The interesting point here is that code transformers are not needed to
execute the transformed code if all required .pyc files are already
available.

Code transformer API

A code transformer is a class with ast_transformer() and/or
code_transformer() methods (API described below) and a name attribute.

For efficiency, do not define a code_transformer() or ast_transformer()
method if it does nothing.

The name attribute (str) must be a short string used to identify an
optimizer. It is used to build a .pyc filename. The name must not
contain dots ('.'), dashes ('-') or directory separators: dots are used
to separated fields in a .pyc filename and dashes areused to join code
transformer names to build the optimizer tag.

Note

It would be nice to pass the fully qualified name of a module in the
context when an AST transformer is used to transform a module on import,
but it looks like the information is not available in
PyParser_ASTFromStringObject().

code_transformer() method

Prototype:

    def code_transformer(self, code, context):
        ...
        new_code = ...
        ...
        return new_code

Parameters:

-   code: code object
-   context: an object with an optimize attribute (int), the
    optimization level (0, 1 or 2). The value of the optimize attribute
    comes from the optimize parameter of the compile() function, it is
    equal to sys.flags.optimize by default.

Each implementation of Python can add extra attributes to context. For
example, on CPython, context will also have the following attribute:

-   interactive (bool): true if in interactive mode

XXX add more flags?

XXX replace flags int with a sub-namespace, or with specific attributes?

The method must return a code object.

The code transformer is run after the compilation to bytecode

ast_transformer() method

Prototype:

    def ast_transformer(self, tree, context):
        ...
        return tree

Parameters:

-   tree: an AST tree
-   context: an object with a filename attribute (str)

It must return an AST tree. It can modify the AST tree in place, or
create a new AST tree.

The AST transformer is called after the creation of the AST by the
parser and before the compilation to bytecode. New attributes may be
added to context in the future.

Changes

In short, add:

-   -o OPTIM_TAG command line option
-   sys.implementation.optim_tag
-   sys.get_code_transformers()
-   sys.set_code_transformers(transformers)
-   ast.PyCF_TRANSFORMED_AST

API to get/set code transformers

Add new functions to register code transformers:

-   sys.set_code_transformers(transformers): set the list of code
    transformers and update sys.implementation.optim_tag
-   sys.get_code_transformers(): get the list of code transformers.

The order of code transformers matter. Running transformer A and then
transformer B can give a different output than running transformer B an
then transformer A.

Example to prepend a new code transformer:

    transformers = sys.get_code_transformers()
    transformers.insert(0, new_cool_transformer)
    sys.set_code_transformers(transformers)

All AST transformers are run sequentially (ex: the second transformer
gets the input of the first transformer), and then all bytecode
transformers are run sequentially.

Optimizer tag

Changes:

-   Add sys.implementation.optim_tag (str): optimization tag. The
    default optimization tag is 'opt'.
-   Add a new -o OPTIM_TAG command line option to set
    sys.implementation.optim_tag.

Changes on importlib:

-   importlib uses sys.implementation.optim_tag to build the .pyc
    filename to importing modules, instead of always using opt. Remove
    also the special case for the optimizer level 0 with the default
    optimizer tag 'opt' to simplify the code.
-   When loading a module, if the .pyc file is missing but the .py is
    available, the .py is only used if code optimizers have the same
    optimizer tag than the current tag, otherwise an ImportError
    exception is raised.

Pseudo-code of a use_py() function to decide if a .py file can be
compiled to import a module:

    def transformers_tag():
        transformers = sys.get_code_transformers()
        if not transformers:
            return 'noopt'
        return '-'.join(transformer.name
                        for transformer in transformers)

    def use_py():
        return (transformers_tag() == sys.implementation.optim_tag)

The order of sys.get_code_transformers() matter. For example, the fat
transformer followed by the pythran transformer gives the optimizer tag
fat-pythran.

The behaviour of the importlib module is unchanged with the default
optimizer tag ('opt').

Peephole optimizer

By default, sys.implementation.optim_tag is opt and
sys.get_code_transformers() returns a list of one code transformer: the
peephole optimizer (optimize the bytecode).

Use -o noopt to disable the peephole optimizer. In this case, the
optimizer tag is noopt and no code transformer is registered.

Using the -o opt option has not effect.

AST enhancements

Enhancements to simplify the implementation of AST transformers:

-   Add a new compiler flag PyCF_TRANSFORMED_AST to get the transformed
    AST. PyCF_ONLY_AST returns the AST before the transformers.

Examples

.pyc filenames

Example of .pyc filenames of the os module.

With the default optimizer tag 'opt':

+-------------------------+--------------------+
| .pyc filename           | Optimization level |
+=========================+====================+
| os.cpython-36.opt-0.pyc |   0                |
+-------------------------+--------------------+
| os.cpython-36.opt-1.pyc |   1                |
+-------------------------+--------------------+
| os.cpython-36.opt-2.pyc |   2                |
+-------------------------+--------------------+

With the 'fat' optimizer tag:

+-------------------------+--------------------+
| .pyc filename           | Optimization level |
+=========================+====================+
| os.cpython-36.fat-0.pyc |   0                |
+-------------------------+--------------------+
| os.cpython-36.fat-1.pyc |   1                |
+-------------------------+--------------------+
| os.cpython-36.fat-2.pyc |   2                |
+-------------------------+--------------------+

Bytecode transformer

Scary bytecode transformer replacing all strings with "Ni! Ni! Ni!":

    import sys
    import types

    class BytecodeTransformer:
        name = "knights_who_say_ni"

        def code_transformer(self, code, context):
            consts = ['Ni! Ni! Ni!' if isinstance(const, str) else const
                      for const in code.co_consts]
            return types.CodeType(code.co_argcount,
                                  code.co_kwonlyargcount,
                                  code.co_nlocals,
                                  code.co_stacksize,
                                  code.co_flags,
                                  code.co_code,
                                  tuple(consts),
                                  code.co_names,
                                  code.co_varnames,
                                  code.co_filename,
                                  code.co_name,
                                  code.co_firstlineno,
                                  code.co_lnotab,
                                  code.co_freevars,
                                  code.co_cellvars)

    # replace existing code transformers with the new bytecode transformer
    sys.set_code_transformers([BytecodeTransformer()])

    # execute code which will be transformed by code_transformer()
    exec("print('Hello World!')")

Output:

    Ni! Ni! Ni!

AST transformer

Similarly to the bytecode transformer example, the AST transformer also
replaces all strings with "Ni! Ni! Ni!":

    import ast
    import sys

    class KnightsWhoSayNi(ast.NodeTransformer):
        def visit_Str(self, node):
            node.s = 'Ni! Ni! Ni!'
            return node

    class ASTTransformer:
        name = "knights_who_say_ni"

        def __init__(self):
            self.transformer = KnightsWhoSayNi()

        def ast_transformer(self, tree, context):
            self.transformer.visit(tree)
            return tree

    # replace existing code transformers with the new AST transformer
    sys.set_code_transformers([ASTTransformer()])

    # execute code which will be transformed by ast_transformer()
    exec("print('Hello World!')")

Output:

    Ni! Ni! Ni!

Other Python implementations

The PEP 511 should be implemented by all Python implementation, but the
bytecode and the AST are not standardized.

By the way, even between minor version of CPython, there are changes on
the AST API. There are differences, but only minor differences. It is
quite easy to write an AST transformer which works on Python 2.7 and
Python 3.5 for example.

Discussion

-   [Python-ideas] PEP 511: API for code transformers (January 2016)
-   [Python-Dev] AST optimizer implemented in Python (August 2012)

Prior Art

AST optimizers

The Issue #17515 "Add sys.setasthook() to allow to use a custom AST"
optimizer was a first attempt of API for code transformers, but specific
to AST.

In 2015, Victor Stinner wrote the fatoptimizer project, an AST optimizer
specializing functions using guards.

In 2014, Kevin Conway created the PyCC optimizer.

In 2012, Victor Stinner wrote the astoptimizer project, an AST optimizer
implementing various optimizations. Most interesting optimizations break
the Python semantics since no guard is used to disable optimization if
something changes.

In 2011, Eugene Toder proposed to rewrite some peephole optimizations in
a new AST optimizer: issue #11549, Build-out an AST optimizer, moving
some functionality out of the peephole optimizer. The patch adds ast.Lit
(it was proposed to rename it to ast.Literal).

Python Preprocessors

-   MacroPy: MacroPy is an implementation of Syntactic Macros in the
    Python Programming Language. MacroPy provides a mechanism for
    user-defined functions (macros) to perform transformations on the
    abstract syntax tree (AST) of a Python program at import time.
-   pypreprocessor: C-style preprocessor directives in Python, like
    #define and #ifdef

Bytecode transformers

-   codetransformer: Bytecode transformers for CPython inspired by the
    ast module’s NodeTransformer.
-   byteplay: Byteplay lets you convert Python code objects into
    equivalent objects which are easy to play with, and lets you convert
    those objects back into living Python code objects. It's useful for
    applying crazy transformations on Python functions, and is also
    useful in learning Python byte code intricacies. See byteplay
    documentation.

See also:

-   BytecodeAssembler

Copyright

This document has been placed in the public domain.


text:
PEP: 3125 Title: Remove Backslash Continuation Version: $Revision$
Last-Modified: $Date$ Author: Jim J. Jewett<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
29-Apr-2007 Post-History: 29-Apr-2007, 30-Apr-2007, 04-May-2007

Rejection Notice

This PEP is rejected. There wasn't enough support in favor, the feature
to be removed isn't all that harmful, and there are some use cases that
would become harder.

Abstract

Python initially inherited its parsing from C. While this has been
generally useful, there are some remnants which have been less useful
for Python, and should be eliminated.

This PEP proposes elimination of terminal \ as a marker for line
continuation.

Motivation

One goal for Python 3000 should be to simplify the language by removing
unnecessary or duplicated features. There are currently several ways to
indicate that a logical line is continued on the following physical
line.

The other continuation methods are easily explained as a logical
consequence of the semantics they provide; \ is simply an escape
character that needs to be memorized.

Existing Line Continuation Methods

Parenthetical Expression - ([{}])

Open a parenthetical expression. It doesn't matter whether people view
the "line" as continuing; they do immediately recognize that the
expression needs to be closed before the statement can end.

Examples using each of (), [], and {}:

    def fn(long_argname1,
           long_argname2):
        settings = {"background": "random noise",
                    "volume": "barely audible"}
        restrictions = ["Warrantee void if used",
                        "Notice must be received by yesterday",
                        "Not responsible for sales pitch"]

Note that it is always possible to parenthesize an expression, but it
can seem odd to parenthesize an expression that needs parentheses only
for the line break:

    assert val>4, (
        "val is too small")

Triple-Quoted Strings

Open a triple-quoted string; again, people recognize that the string
needs to finish before the next statement starts. :

    banner_message = """
        Satisfaction Guaranteed,
        or DOUBLE YOUR MONEY BACK!!!

                                        some minor restrictions apply"""

Terminal \ in the general case

A terminal \ indicates that the logical line is continued on the
following physical line (after whitespace). There are no particular
semantics associated with this. This form is never required, although it
may look better (particularly for people with a C language background)
in some cases:

    >>> assert val>4, \
            "val is too small"

Also note that the \ must be the final character in the line. If your
editor navigation can add whitespace to the end of a line, that
invisible change will alter the semantics of the program. Fortunately,
the typical result is only a syntax error, rather than a runtime bug:

    >>> assert val>4, \
            "val is too small"

    SyntaxError: unexpected character after line continuation character

This PEP proposes to eliminate this redundant and potentially confusing
alternative.

Terminal \ within a string

A terminal \ within a single-quoted string, at the end of the line. This
is arguably a special case of the terminal \, but it is a special case
that may be worth keeping. :

    >>> "abd\
     def"
    'abd def'

-   Pro: Many of the objections to removing \ termination were really
    just objections to removing it within literal strings; several
    people clarified that they want to keep this literal-string usage,
    but don't mind losing the general case.
-   Pro: The use of \ for an escape character within strings is well
    known.
-   Contra: But note that this particular usage is odd, because the
    escaped character (the newline) is invisible, and the special
    treatment is to delete the character. That said, the \ of \(newline)
    is still an escape which changes the meaning of the following
    character.

Alternate Proposals

Several people have suggested alternative ways of marking the line end.
Most of these were rejected for not actually simplifying things.

The one exception was to let any unfinished expression signify a line
continuation, possibly in conjunction with increased indentation.

This is attractive because it is a generalization of the rule for
parentheses.

The initial objections to this were:

-   The amount of whitespace may be contentious; expression continuation
    should not be confused with opening a new suite.

-   The "expression continuation" markers are not as clearly marked in
    Python as the grouping punctuation "(), [], {}" marks are:

        # Plus needs another operand, so the line continues
        "abc" +
            "def"

        # String ends an expression, so the line does not
        # not continue.  The next line is a syntax error because
        # unary plus does not apply to strings.
        "abc"
            + "def"

-   Guido objected for technical reasons.[1] The most obvious
    implementation would require allowing INDENT or DEDENT tokens
    anywhere, or at least in a widely expanded (and ill-defined) set of
    locations. While this is of concern only for the internal parsing
    mechanism (rather than for users), it would be a major new source of
    complexity.

Andrew Koenig then pointed out[2] a better implementation strategy, and
said that it had worked quite well in other languages.[3] The improved
suggestion boiled down to:

  The whitespace that follows an (operator or) open bracket or
  parenthesis can include newline characters.

  It would be implemented at a very low lexical level -- even before the
  decision is made to turn a newline followed by spaces into an INDENT
  or DEDENT token.

There is still some concern that it could mask bugs, as in this
example[4]:

    # Used to be y+1, the 1 got dropped.  Syntax Error (today)
    # would become nonsense.
    x = y+
    f(x)

Requiring that the continuation be indented more than the initial line
would add both safety and complexity.

Open Issues

-   Should \-continuation be removed even inside strings?
-   Should the continuation markers be expanded from just ([{}]) to
    include lines ending with an operator?
-   As a safety measure, should the continuation line be required to be
    more indented than the initial line?

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] (email subject) PEP 30XZ: Simplified Parsing, van Rossum
https://mail.python.org/pipermail/python-3000/2007-April/007063.html

[2] (email subject) PEP 3125 -- remove backslash continuation, Koenig
https://mail.python.org/pipermail/python-3000/2007-May/007237.html

[3] The Snocone Programming Language, Koenig
http://www.snobol4.com/report.htm

[4] (email subject) PEP 3125 -- remove backslash continuation, van
Rossum
https://mail.python.org/pipermail/python-3000/2007-May/007244.html


text:
PEP: 688 Title: Making the buffer protocol accessible in Python Author:
Jelle Zijlstra<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/19756 Status: Final Type: Standards Track
Topic: Typing Created: 23-Apr-2022 Python-Version: 3.12 Post-History:
23-Apr-2022, 25-Apr-2022, 06-Oct-2022, 26-Oct-2022 Resolution:
07-Mar-2023

python:python-buffer-protocol

Abstract

This PEP proposes a Python-level API for the buffer protocol, which is
currently accessible only to C code. This allows type checkers to
evaluate whether objects implement the protocol.

Motivation

The CPython C API provides a versatile mechanism for accessing the
underlying memory of an object—the buffer protocol introduced in PEP
3118. Functions that accept binary data are usually written to handle
any object implementing the buffer protocol. For example, at the time of
writing, there are around 130 functions in CPython using the Argument
Clinic Py_buffer type, which accepts the buffer protocol.

Currently, there is no way for Python code to inspect whether an object
supports the buffer protocol. Moreover, the static type system does not
provide a type annotation to represent the protocol. This is a common
problem when writing type annotations for code that accepts generic
buffers.

Similarly, it is impossible for a class written in Python to support the
buffer protocol. A buffer class in Python would give users the ability
to easily wrap a C buffer object, or to test the behavior of an API that
consumes the buffer protocol. Granted, this is not a particularly common
need. However, there has been a CPython feature request for supporting
buffer classes written in Python that has been open since 2012.

Rationale

Current options

There are two known workarounds for annotating buffer types in the type
system, but neither is adequate.

First, the current workaround for buffer types in typeshed is a type
alias that lists well-known buffer types in the standard library, such
as bytes, bytearray, memoryview, and array.array. This approach works
for the standard library, but it does not extend to third-party buffer
types.

Second, the documentation for typing.ByteString currently states:

  This type represents the types bytes, bytearray, and memoryview of
  byte sequences.

  As a shorthand for this type, bytes can be used to annotate arguments
  of any of the types mentioned above.

Although this sentence has been in the documentation since 2015, the use
of bytes to include these other types is not specified in any of the
typing PEPs. Furthermore, this mechanism has a number of problems. It
does not include all possible buffer types, and it makes the bytes type
ambiguous in type annotations. After all, there are many operations that
are valid on bytes objects, but not on memoryview objects, and it is
perfectly possible for a function to accept bytes but not memoryview
objects. A mypy user reports that this shortcut has caused significant
problems for the psycopg project.

Kinds of buffers

The C buffer protocol supports many options, affecting strides,
contiguity, and support for writing to the buffer. Some of these options
would be useful in the type system. For example, typeshed currently
provides separate type aliases for writable and read-only buffers.

However, in the C buffer protocol, most of these options cannot be
queried directly on the type object. The only way to figure out whether
an object supports a particular flag is to actually ask for the buffer.
For some types, such as memoryview, the supported flags depend on the
instance. As a result, it would be difficult to represent support for
these flags in the type system.

Specification

Python-level buffer protocol

We propose to add two Python-level special methods, __buffer__ and
__release_buffer__. Python classes that implement these methods are
usable as buffers from C code. Conversely, classes implemented in C that
support the buffer protocol acquire synthesized methods accessible from
Python code.

The __buffer__ method is called to create a buffer from a Python object,
for example by the memoryview() constructor. It corresponds to the
bf_getbuffer C slot. The Python signature for this method is
def __buffer__(self, flags: int, /) -> memoryview: .... The method must
return a memoryview object. If the bf_getbuffer slot is invoked on a
Python class with a __buffer__ method, the interpreter extracts the
underlying Py_buffer from the memoryview returned by the method and
returns it to the C caller. Similarly, if Python code calls the
__buffer__ method on an instance of a C class that implements
bf_getbuffer, the returned buffer is wrapped in a memoryview for
consumption by Python code.

The __release_buffer__ method should be called when a caller no longer
needs the buffer returned by __buffer__. It corresponds to the
bf_releasebuffer C slot. This is an optional part of the buffer
protocol. The Python signature for this method is
def __release_buffer__(self, buffer: memoryview, /) -> None: .... The
buffer to be released is wrapped in a memoryview. When this method is
invoked through CPython's buffer API (for example, through calling
memoryview.release on a memoryview returned by __buffer__), the passed
memoryview is the same object as was returned by __buffer__. It is also
possible to call __release_buffer__ on a C class that implements
bf_releasebuffer.

If __release_buffer__ exists on an object, Python code that calls
__buffer__ directly on the object must call __release_buffer__ on the
same object when it is done with the buffer. Otherwise, resources used
by the object may not be reclaimed. Similarly, it is a programming error
to call __release_buffer__ without a previous call to __buffer__, or to
call it multiple times for a single call to __buffer__. For objects that
implement the C buffer protocol, calls to __release_buffer__ where the
argument is not a memoryview wrapping the same object will raise an
exception. After a valid call to __release_buffer__, the memoryview is
invalidated (as if its release() method had been called), and any
subsequent calls to __release_buffer__ with the same memoryview will
raise an exception. The interpreter will ensure that misuse of the
Python API will not break invariants at the C level -- for example, it
will not cause memory safety violations.

inspect.BufferFlags

To help implementations of __buffer__, we add inspect.BufferFlags, a
subclass of enum.IntFlag. This enum contains all flags defined in the C
buffer protocol. For example, inspect.BufferFlags.SIMPLE has the same
value as the PyBUF_SIMPLE constant.

collections.abc.Buffer

We add a new abstract base classes, collections.abc.Buffer, which
requires the __buffer__ method. This class is intended primarily for use
in type annotations:

    def need_buffer(b: Buffer) -> memoryview:
        return memoryview(b)

    need_buffer(b"xy")  # ok
    need_buffer("xy")  # rejected by static type checkers

It can also be used in isinstance and issubclass checks:

    >>> from collections.abc import Buffer
    >>> isinstance(b"xy", Buffer)
    True
    >>> issubclass(bytes, Buffer)
    True
    >>> issubclass(memoryview, Buffer)
    True
    >>> isinstance("xy", Buffer)
    False
    >>> issubclass(str, Buffer)
    False

In the typeshed stub files, the class should be defined as a Protocol,
following the precedent of other simple ABCs in collections.abc such as
collections.abc.Iterable or collections.abc.Sized.

Example

The following is an example of a Python class that implements the buffer
protocol:

    import contextlib
    import inspect

    class MyBuffer:
        def __init__(self, data: bytes):
            self.data = bytearray(data)
            self.view = None

        def __buffer__(self, flags: int) -> memoryview:
            if flags != inspect.BufferFlags.FULL_RO:
                raise TypeError("Only BufferFlags.FULL_RO supported")
            if self.view is not None:
                raise RuntimeError("Buffer already held")
            self.view = memoryview(self.data)
            return self.view

        def __release_buffer__(self, view: memoryview) -> None:
            assert self.view is view  # guaranteed to be true
            self.view.release()
            self.view = None

        def extend(self, b: bytes) -> None:
            if self.view is not None:
                raise RuntimeError("Cannot extend held buffer")
            self.data.extend(b)

    buffer = MyBuffer(b"capybara")
    with memoryview(buffer) as view:
        view[0] = ord("C")

        with contextlib.suppress(RuntimeError):
            buffer.extend(b"!")  # raises RuntimeError

    buffer.extend(b"!")  # ok, buffer is no longer held

    with memoryview(buffer) as view:
        assert view.tobytes() == b"Capybara!"

Equivalent for older Python versions

New typing features are usually backported to older Python versions in
the typing_extensions package. Because the buffer protocol is currently
accessible only in C, this PEP cannot be fully implemented in a
pure-Python package like typing_extensions. As a temporary workaround,
an abstract base class typing_extensions.Buffer will be provided for
Python versions that do not have collections.abc.Buffer available.

After this PEP is implemented, inheriting from collections.abc.Buffer
will not be necessary to indicate that an object supports the buffer
protocol. However, in older Python versions, it will be necessary to
explicitly inherit from typing_extensions.Buffer to indicate to type
checkers that a class supports the buffer protocol, since objects
supporting the buffer protocol will not have a __buffer__ method. It is
expected that this will happen primarily in stub files, because buffer
classes are necessarily implemented in C code, which cannot have types
defined inline. For runtime uses, the ABC.register API can be used to
register buffer classes with typing_extensions.Buffer.

No special meaning for bytes

The special case stating that bytes may be used as a shorthand for other
ByteString types will be removed from the typing documentation. With
collections.abc.Buffer available as an alternative, there will be no
good reason to allow bytes as a shorthand. Type checkers currently
implementing this behavior should deprecate and eventually remove it.

Backwards Compatibility

__buffer__ and __release_buffer__ attributes

As the runtime changes in this PEP only add new functionality, there are
few backwards compatibility concerns.

However, code that uses a __buffer__ or __release_buffer__ attribute for
other purposes may be affected. While all dunders are technically
reserved for the language, it is still good practice to ensure that a
new dunder does not interfere with too much existing code, especially
widely used packages. A survey of publicly accessible code found:

-   PyPy supports a __buffer__ method with compatible semantics to those
    proposed in this PEP. A PyPy core developer expressed his support
    for this PEP.
-   pyzmq implements a PyPy-compatible __buffer__ method.
-   mpi4py defines a SupportsBuffer protocol that would be equivalent to
    this PEP's collections.abc.Buffer.
-   NumPy used to have an undocumented behavior where it would access a
    __buffer__ attribute (not method) to get an object's buffer. This
    was removed in 2019 for NumPy 1.17. The behavior would have last
    worked in NumPy 1.16, which only supported Python 3.7 and older.
    Python 3.7 will have reached its end of life by the time this PEP is
    expected to be implemented.

Thus, this PEP's use of the __buffer__ method will improve
interoperability with PyPy and not interfere with the current versions
of any major Python packages.

No publicly accessible code uses the name __release_buffer__.

Removal of the bytes special case

Separately, the recommendation to remove the special behavior for bytes
in type checkers does have a backwards compatibility impact on their
users. An experiment with mypy shows that several major open source
projects that use it for type checking will see new errors if the bytes
promotion is removed. Many of these errors can be fixed by improving the
stubs in typeshed, as has already been done for the builtins, binascii,
pickle, and re modules. A review of all usage of bytes types in typeshed
is in progress. Overall, the change improves type safety and makes the
type system more consistent, so we believe the migration cost is worth
it.

How to Teach This

We will add notes pointing to collections.abc.Buffer in appropriate
places in the documentation, such as typing.readthedocs.io and the mypy
cheat sheet. Type checkers may provide additional pointers in their
error messages. For example, when they encounter a buffer object being
passed to a function that is annotated to only accept bytes, the error
message could include a note suggesting the use of
collections.abc.Buffer instead.

Reference Implementation

An implementation of this PEP is available in the author's fork.

Rejected Ideas

types.Buffer

An earlier version of this PEP proposed adding a new types.Buffer type
with an __instancecheck__ implemented in C so that isinstance() checks
can be used to check whether a type implements the buffer protocol. This
avoids the complexity of exposing the full buffer protocol to Python
code, while still allowing the type system to check for the buffer
protocol.

However, that approach does not compose well with the rest of the type
system, because types.Buffer would be a nominal type, not a structural
one. For example, there would be no way to represent "an object that
supports both the buffer protocol and __len__". With the current
proposal, __buffer__ is like any other special method, so a Protocol can
be defined combining it with another method.

More generally, no other part of Python works like the proposed
types.Buffer. The current proposal is more consistent with the rest of
the language, where C-level slots usually have corresponding
Python-level special methods.

Keep bytearray compatible with bytes

It has been suggested to remove the special case where memoryview is
always compatible with bytes, but keep it for bytearray, because the two
types have very similar interfaces. However, several standard library
functions (e.g., re.compile, socket.getaddrinfo, and most functions
accepting path-like arguments) accept bytes but not bytearray. In most
codebases, bytearray is also not a very common type. We prefer to have
users spell out accepted types explicitly (or use Protocol from PEP 544
if only a specific set of methods is required). This aspect of the
proposal was specifically discussed on the typing-sig mailing list,
without any strong disagreement from the typing community.

Distinguish between mutable and immutable buffers

The most frequently used distinction within buffer types is whether or
not the buffer is mutable. Some functions accept only mutable buffers
(e.g., bytearray, some memoryview objects), others accept all buffers.

An earlier version of this PEP proposed using the presence of the
bf_releasebuffer slot to determine whether a buffer type is mutable.
This rule holds for most standard library buffer types, but the
relationship between mutability and the presence of this slot is not
absolute. For example, numpy arrays are mutable but do not have this
slot.

The current buffer protocol does not provide any way to reliably
determine whether a buffer type represents a mutable or immutable
buffer. Therefore, this PEP does not add type system support for this
distinction. The question can be revisited in the future if the buffer
protocol is enhanced to provide static introspection support. A sketch
for such a mechanism exists.

Acknowledgments

Many people have provided useful feedback on drafts of this PEP. Petr
Viktorin has been particularly helpful in improving my understanding of
the subtleties of the buffer protocol.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 460 Title: Add binary interpolation and formatting Version:
$Revision$ Last-Modified: $Date$ Author: Antoine Pitrou
<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 06-Jan-2014 Python-Version: 3.5

Abstract

This PEP proposes to add minimal formatting operations to bytes and
bytearray objects. The proposed additions are:

-   bytes % ... and bytearray % ... for percent-formatting, similar in
    syntax to percent-formatting on str objects (accepting a single
    object, a tuple or a dict).
-   bytes.format(...) and bytearray.format(...) for a formatting similar
    in syntax to str.format() (accepting positional as well as keyword
    arguments).
-   bytes.format_map(...) and bytearray.format_map(...) for an API
    similar to str.format_map(...), with the same formatting syntax and
    semantics as bytes.format() and bytearray.format().

Rationale

In Python 2, str % args and str.format(args) allow the formatting and
interpolation of bytestrings. This feature has commonly been used for
the assembling of protocol messages when protocols are known to use a
fixed encoding.

Python 3 generally mandates that text be stored and manipulated as
unicode (i.e. str objects, not bytes). In some cases, though, it makes
sense to manipulate bytes objects directly. Typical usage is binary
network protocols, where you can want to interpolate and assemble
several bytes object (some of them literals, some of them compute) to
produce complete protocol messages. For example, protocols such as HTTP
or SIP have headers with ASCII names and opaque "textual" values using a
varying and/or sometimes ill-defined encoding. Moreover, those headers
can be followed by a binary body... which can be chunked and decorated
with ASCII headers and trailers!

While there are reasonably efficient ways to accumulate binary data
(such as using a bytearray object, the bytes.join method or even
io.BytesIO), none of them leads to the kind of readable and intuitive
code that is produced by a %-formatted or {}-formatted template and a
formatting operation.

Binary formatting features

Supported features

In this proposal, percent-formatting for bytes and bytearray supports
the following features:

-   Looking up formatting arguments by position as well as by name
    (i.e., %s as well as %(name)s).
-   %s will try to get a Py_buffer on the given value, and fallback on
    calling __bytes__. The resulting binary data is inserted at the
    given point in the string. This is expected to work with bytes,
    bytearray and memoryview objects (as well as a couple others such as
    pathlib's path objects).
-   %c will accept an integer between 0 and 255, and insert a byte of
    the given value.

Braces-formatting for bytes and bytearray supports the following
features:

-   All the kinds of argument lookup supported by str.format() (explicit
    positional lookup, auto-incremented positional lookup, keyword
    lookup, attribute lookup, etc.)
-   Insertion of binary data when no modifier or layout is specified
    (e.g. {}, {0}, {name}). This has the same semantics as %s for
    percent-formatting (see above).
-   The c modifier will accept an integer between 0 and 255, and insert
    a byte of the given value (same as %c above).

Unsupported features

All other features present in formatting of str objects (either through
the percent operator or the str.format() method) are unsupported. Those
features imply treating the recipient of the operator or method as text,
which goes counter to the text / bytes separation (for example,
accepting %d as a format code would imply that the bytes object really
is an ASCII-compatible text string).

Amongst those unsupported features are not only most type-specific
format codes, but also the various layout specifiers such as padding or
alignment. Besides, str objects are not acceptable as arguments to the
formatting operations, even when using e.g. the %s format code.

__format__ isn't called.

Criticisms

-   The development cost and maintenance cost.
-   In 3.3 encoding to ASCII or latin-1 is as fast as memcpy (but it
    still creates a separate object).
-   Developers will have to work around the lack of binary formatting
    anyway, if they want to support Python 3.4 and earlier.
-   bytes.join() is consistently faster than format to join bytes
    strings (XXX is it?).
-   Formatting functions could be implemented in a third party module,
    rather than added to builtin types.

Other proposals

A new type datatype

It was proposed to create a new datatype specialized for "network
programming". The authors of this PEP believe this is
counter-productive. Python 3 already has several major types dedicated
to manipulation of binary data: bytes, bytearray, memoryview,
io.BytesIO.

Adding yet another type would make things more confusing for users, and
interoperability between libraries more painful (also potentially
sub-optimal, due to the necessary conversions).

Moreover, not one type would be needed, but two: one immutable type (to
allow for hashing), and one mutable type (as efficient accumulation is
often necessary when working with network messages).

Resolution

This PEP is made obsolete by the acceptance of PEP 461, which introduces
a more extended formatting language for bytes objects in conjunction
with the modulo operator.

References

-   Issue #3982: support .format for bytes
-   Mercurial project
-   Twisted project
-   Documentation of Python 2 formatting (str % args)
-   Documentation of Python 2 formatting (str.format)

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: