text:
PEP: 734 Title: Multiple Interpreters in the Stdlib Author: Eric Snow
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-734-multiple-interpreters-in-the-stdlib/41147
Status: Deferred Type: Standards Track Created: 06-Nov-2023
Python-Version: 3.13 Post-History: 14-Dec-2023, Replaces: 554
Resolution:
https://discuss.python.org/t/pep-734-multiple-interpreters-in-the-stdlib/41147/24

Note

This PEP is essentially a continuation of PEP 554. That document had
grown a lot of ancillary information across 7 years of discussion. This
PEP is a reduction back to the essential information. Much of that extra
information is still valid and useful, just not in the immediate context
of the specific proposal here.

Abstract

This PEP proposes to add a new module, interpreters, to support
inspecting, creating, and running code in multiple interpreters in the
current process. This includes Interpreter objects that represent the
underlying interpreters. The module will also provide a basic Queue
class for communication between interpreters. Finally, we will add a new
concurrent.futures.InterpreterPoolExecutor based on the interpreters
module.

Introduction

Fundamentally, an "interpreter" is the collection of (essentially) all
runtime state which Python threads must share. So, let's first look at
threads. Then we'll circle back to interpreters.

Threads and Thread States

A Python process will have one or more OS threads running Python code
(or otherwise interacting with the C API). Each of these threads
interacts with the CPython runtime using its own thread state
(PyThreadState), which holds all the runtime state unique to that
thread. There is also some runtime state that is shared between multiple
OS threads.

Any OS thread may switch which thread state it is currently using, as
long as it isn't one that another OS thread is already using (or has
been using). This "current" thread state is stored by the runtime in a
thread-local variable, and may be looked up explicitly with
PyThreadState_Get(). It gets set automatically for the initial ("main")
OS thread and for threading.Thread objects. From the C API it is set
(and cleared) by PyThreadState_Swap() and may be set by
PyGILState_Ensure(). Most of the C API requires that there be a current
thread state, either looked up implicitly or passed in as an argument.

The relationship between OS threads and thread states is one-to-many.
Each thread state is associated with at most a single OS thread and
records its thread ID. A thread state is never used for more than one OS
thread. In the other direction, however, an OS thread may have more than
one thread state associated with it, though, again, only one may be
current.

When there's more than one thread state for an OS thread,
PyThreadState_Swap() is used in that OS thread to switch between them,
with the requested thread state becoming the current one. Whatever was
running in the thread using the old thread state is effectively paused
until that thread state is swapped back in.

Interpreter States

As noted earlier, there is some runtime state that multiple OS threads
share. Some of it is exposed by the sys module, though much is used
internally and not exposed explicitly or only through the C API.

This shared state is called the interpreter state (PyInterpreterState).
We'll sometimes refer to it here as just "interpreter", though that is
also sometimes used to refer to the python executable, to the Python
implementation, and to the bytecode interpreter (i.e. exec()/eval()).

CPython has supported multiple interpreters in the same process (AKA
"subinterpreters") since version 1.5 (1997). The feature has been
available via the C API <python:sub-interpreter-support>.

Interpreters and Threads

Thread states are related to interpreter states in much the same way
that OS threads and processes are related (at a high level). To begin
with, the relationship is one-to-many. A thread state belongs to a
single interpreter (and stores a pointer to it). That thread state is
never used for a different interpreter. In the other direction, however,
an interpreter may have zero or more thread states associated with it.
The interpreter is only considered active in OS threads where one of its
thread states is current.

Interpreters are created via the C API using
Py_NewInterpreterFromConfig() (or Py_NewInterpreter(), which is a light
wrapper around Py_NewInterpreterFromConfig()). That function does the
following:

1.  create a new interpreter state
2.  create a new thread state
3.  set the thread state as current (a current tstate is needed for
    interpreter init)
4.  initialize the interpreter state using that thread state
5.  return the thread state (still current)

Note that the returned thread state may be immediately discarded. There
is no requirement that an interpreter have any thread states, except as
soon as the interpreter is meant to actually be used. At that point it
must be made active in the current OS thread.

To make an existing interpreter active in the current OS thread, the C
API user first makes sure that interpreter has a corresponding thread
state. Then PyThreadState_Swap() is called like normal using that thread
state. If the thread state for another interpreter was already current
then it gets swapped out like normal and execution of that interpreter
in the OS thread is thus effectively paused until it is swapped back in.

Once an interpreter is active in the current OS thread like that, the
thread can call any of the C API, such as PyEval_EvalCode() (i.e.
exec()). This works by using the current thread state as the runtime
context.

The "Main" Interpreter

When a Python process starts, it creates a single interpreter state (the
"main" interpreter) with a single thread state for the current OS
thread. The Python runtime is then initialized using them.

After initialization, the script or module or REPL is executed using
them. That execution happens in the interpreter's __main__ module.

When the process finishes running the requested Python code or REPL, in
the main OS thread, the Python runtime is finalized in that thread using
the main interpreter.

Runtime finalization has only a slight, indirect effect on still-running
Python threads, whether in the main interpreter or in subinterpreters.
That's because right away it waits indefinitely for all non-daemon
Python threads to finish.

While the C API may be queried, there is no mechanism by which any
Python thread is directly alerted that finalization has begun, other
than perhaps with "atexit" functions that may be been registered using
threading._register_atexit().

Any remaining subinterpreters are themselves finalized later, but at
that point they aren't current in any OS threads.

Interpreter Isolation

CPython's interpreters are intended to be strictly isolated from each
other. That means interpreters never share objects (except in very
specific cases with immortal, immutable builtin objects). Each
interpreter has its own modules (sys.modules), classes, functions, and
variables. Even where two interpreters define the same class, each will
have its own copy. The same applies to state in C, including in
extension modules. The CPython C API docs explain more.

Notably, there is some process-global state that interpreters will
always share, some mutable and some immutable. Sharing immutable state
presents few problems, while providing some benefits (mainly
performance). However, all shared mutable state requires special
management, particularly for thread-safety, some of which the OS takes
care of for us.

Mutable:

-   file descriptors
-   low-level env vars
-   process memory (though allocators are isolated)
-   the list of interpreters

Immutable:

-   builtin types (e.g. dict, bytes)
-   singletons (e.g. None)
-   underlying static module data (e.g. functions) for
    builtin/extension/frozen modules

Existing Execution Components

There are a number of existing parts of Python that may help with
understanding how code may be run in a subinterpreter.

In CPython, each component is built around one of the following C API
functions (or variants):

-   PyEval_EvalCode(): run the bytecode interpreter with the given code
    object
-   PyRun_String(): compile + PyEval_EvalCode()
-   PyRun_File(): read + compile + PyEval_EvalCode()
-   PyRun_InteractiveOneObject(): compile + PyEval_EvalCode()
-   PyObject_Call(): calls PyEval_EvalCode()

builtins.exec()

The builtin exec() may be used to execute Python code. It is essentially
a wrapper around the C API functions PyRun_String() and
PyEval_EvalCode().

Here are some relevant characteristics of the builtin exec():

-   It runs in the current OS thread and pauses whatever was running
    there, which resumes when exec() finishes. No other OS threads are
    affected. (To avoid pausing the current Python thread, run exec() in
    a threading.Thread.)
-   It may start additional threads, which don't interrupt it.
-   It executes against a "globals" namespace (and a "locals"
    namespace). At module-level, exec() defaults to using __dict__ of
    the current module (i.e. globals()). exec() uses that namespace
    as-is and does not clear it before or after.
-   It propagates any uncaught exception from the code it ran. The
    exception is raised from the exec() call in the Python thread that
    originally called exec().

Command-line

The python CLI provides several ways to run Python code. In each case it
maps to a corresponding C API call:

-   <no args>, -i - run the REPL (PyRun_InteractiveOneObject())
-   <filename> - run a script (PyRun_File())
-   -c <code> - run the given Python code (PyRun_String())
-   -m module - run the module as a script (PyEval_EvalCode() via
    runpy._run_module_as_main())

In each case it is essentially a variant of running exec() at the
top-level of the __main__ module of the main interpreter.

threading.Thread

When a Python thread is started, it runs the "target" function with
PyObject_Call() using a new thread state. The globals namespace come
from func.__globals__ and any uncaught exception is discarded.

Motivation

The interpreters module will provide a high-level interface to the
multiple interpreter functionality. The goal is to make the existing
multiple-interpreters feature of CPython more easily accessible to
Python code. This is particularly relevant now that CPython has a
per-interpreter GIL (PEP 684) and people are more interested in using
multiple interpreters.

Without a stdlib module, users are limited to the
C API <python:sub-interpreter-support>, which restricts how much they
can try out and take advantage of multiple interpreters.

The module will include a basic mechanism for communicating between
interpreters. Without one, multiple interpreters are a much less useful
feature.

Specification

The module will:

-   expose the existing multiple interpreter support
-   introduce a basic mechanism for communicating between interpreters

The module will wrap a new low-level _interpreters module (in the same
way as the threading module). However, that low-level API is not
intended for public use and thus not part of this proposal.

Using Interpreters

The module defines the following functions:

-

    get_current() -> Interpreter

        Returns the Interpreter object for the currently executing
        interpreter.

-

    list_all() -> list[Interpreter]

        Returns the Interpreter object for each existing interpreter,
        whether it is currently running in any OS threads or not.

-

    create() -> Interpreter

        Create a new interpreter and return the Interpreter object for
        it. The interpreter doesn't do anything on its own and is not
        inherently tied to any OS thread. That only happens when
        something is actually run in the interpreter (e.g.
        Interpreter.exec()), and only while running. The interpreter may
        or may not have thread states ready to use, but that is strictly
        an internal implementation detail.

Interpreter Objects

An interpreters.Interpreter object that represents the interpreter
(PyInterpreterState) with the corresponding unique ID. There will only
be one object for any given interpreter.

If the interpreter was created with interpreters.create() then it will
be destroyed as soon as all Interpreter objects with its ID (across all
interpreters) have been deleted.

Interpreter objects may represent other interpreters than those created
by interpreters.create(). Examples include the main interpreter (created
by Python's runtime initialization) and those created via the C-API,
using Py_NewInterpreter(). Such Interpreter objects will not be able to
interact with their corresponding interpreters, e.g. via
Interpreter.exec() (though we may relax this in the future).

Attributes and methods:

-

    id

        (read-only) A non-negative int that identifies the interpreter
        that this Interpreter instance represents. Conceptually, this is
        similar to a process ID.

-

    __hash__()

        Returns the hash of the interpreter's id. This is the same as
        the hash of the ID's integer value.

-

    is_running() -> bool

        Returns True if the interpreter is currently executing code in
        its __main__ module. This excludes sub-threads.

        It refers only to if there is an OS thread running a script
        (code) in the interpreter's __main__ module. That basically
        means whether or not Interpreter.exec() is running in some OS
        thread. Code running in sub-threads is ignored.

-

    prepare_main(**kwargs)

        Bind one or more objects in the interpreter's __main__ module.

        The keyword argument names will be used as the attribute names.
        The values will be bound as new objects, though exactly
        equivalent to the original. Only objects specifically supported
        for passing between interpreters are allowed. See Shareable
        Objects.

        prepare_main() is helpful for initializing the globals for an
        interpreter before running code in it.

-

    exec(code, /)

        Execute the given source code in the interpreter (in the current
        OS thread), using its __main__ module. It doesn't return
        anything.

        This is essentially equivalent to switching to this interpreter
        in the current OS thread and then calling the builtin exec()
        using this interpreter's __main__ module's __dict__ as the
        globals and locals.

        The code running in the current OS thread (a different
        interpreter) is effectively paused until Interpreter.exec()
        finishes. To avoid pausing it, create a new threading.Thread and
        call Interpreter.exec() in it (like Interpreter.call_in_thread()
        does).

        Interpreter.exec() does not reset the interpreter's state nor
        the __main__ module, neither before nor after, so each
        successive call picks up where the last one left off. This can
        be useful for running some code to initialize an interpreter
        (e.g. with imports) before later performing some repeated task.

        If there is an uncaught exception, it will be propagated into
        the calling interpreter as an ExecutionFailed. The full error
        display of the original exception, generated relative to the
        called interpreter, is preserved on the propagated
        ExecutionFailed. That includes the full traceback, with all the
        extra info like syntax error details and chained exceptions. If
        the ExecutionFailed is not caught then that full error display
        will be shown, much like it would be if the propagated exception
        had been raised in the main interpreter and uncaught. Having the
        full traceback is particularly useful when debugging.

        If exception propagation is not desired then an explicit
        try-except should be used around the code passed to
        Interpreter.exec(). Likewise any error handling that depends on
        specific information from the exception must use an explicit
        try-except around the given code, since ExecutionFailed will not
        preserve that information.

-

    call(callable, /)

        Call the callable object in the interpreter. The return value is
        discarded. If the callable raises an exception then it gets
        propagated as an ExecutionFailed exception, in the same way as
        Interpreter.exec().

        For now only plain functions are supported and only ones that
        take no arguments and have no cell vars. Free globals are
        resolved against the target interpreter's __main__ module.

        In the future, we can add support for arguments, closures, and a
        broader variety of callables, at least partly via pickle. We can
        also consider not discarding the return value. The initial
        restrictions are in place to allow us to get the basic
        functionality of the module out to users sooner.

-

    call_in_thread(callable, /) -> threading.Thread

        Essentially, apply Interpreter.call() in a new thread. Return
        values are discarded and exceptions are not propagated.

        call_in_thread() is roughly equivalent to:

            def task():
                interp.call(func)
            t = threading.Thread(target=task)
            t.start()

-

    close()

        Destroy the underlying interpreter.

Communicating Between Interpreters

The module introduces a basic communication mechanism through special
queues.

There are interpreters.Queue objects, but they only proxy the actual
data structure: an unbounded FIFO queue that exists outside any one
interpreter. These queues have special accommodations for safely passing
object data between interpreters, without violating interpreter
isolation. This includes thread-safety.

As with other queues in Python, for each "put" the object is added to
the back and each "get" pops the next one off the front. Every added
object will be popped off in the order it was pushed on.

Only objects that are specifically supported for passing between
interpreters may be sent through an interpreters.Queue. Note that the
actual objects aren't sent, but rather their underlying data. However,
the popped object will still be strictly equivalent to the original. See
Shareable Objects.

The module defines the following functions:

-

    create_queue(maxsize=0, *, syncobj=False) -> Queue

        Create a new queue. If the maxsize is zero or negative then the
        queue is unbounded.

        "syncobj" is used as the default for put() and put_nowait().

Queue Objects

interpreters.Queue objects act as proxies for the underlying
cross-interpreter-safe queues exposed by the interpreters module. Each
Queue object represents the queue with the corresponding unique ID.
There will only be one object for any given queue.

Queue implements all the methods of queue.Queue except for task_done()
and join(), hence it is similar to asyncio.Queue and
multiprocessing.Queue.

Attributes and methods:

-

    id

        (read-only) A non-negative int that identifies the corresponding
        cross-interpreter queue. Conceptually, this is similar to the
        file descriptor used for a pipe.

-

    maxsize

        (read-only) Number of items allowed in the queue. Zero means
        "unbounded".

-

    __hash__()

        Return the hash of the queue's id. This is the same as the hash
        of the ID's integer value.

-

    empty()

        Return True if the queue is empty, False otherwise.

        This is only a snapshot of the state at the time of the call.
        Other threads or interpreters may cause this to change.

-

    full()

        Return True if there are maxsize items in the queue.

        If the queue was initialized with maxsize=0 (the default), then
        full() never returns True.

        This is only a snapshot of the state at the time of the call.
        Other threads or interpreters may cause this to change.

-

    qsize()

        Return the number of items in the queue.

        This is only a snapshot of the state at the time of the call.
        Other threads or interpreters may cause this to change.

-

    put(obj, timeout=None, *, syncobj=None)

        Add the object to the queue.

        If maxsize > 0 and the queue is full then this blocks until a
        free slot is available. If timeout is a positive number then it
        only blocks at least that many seconds and then raises
        interpreters.QueueFull. Otherwise is blocks forever.

        If "syncobj" is true then the object must be shareable, which
        means the object's data is passed through rather than the object
        itself. If "syncobj" is false then all objects are supported.
        However, there are some performance penalties and all objects
        are copies (e.g. via pickle). Thus mutable objects will never be
        automatically synchronized between interpreters. If "syncobj" is
        None (the default) then the queue's default value is used.

        If an object is still in the queue, and the interpreter which
        put it in the queue (i.e. to which it belongs) is destroyed,
        then the object is immediately removed from the queue. (We may
        later add an option to replace the removed object in the queue
        with a sentinel or to raise an exception for the corresponding
        get() call.)

-

    put_nowait(obj, *, syncobj=None)

        Like put() but effectively with an immediate timeout. Thus if
        the queue is full, it immediately raises interpreters.QueueFull.

-

    get(timeout=None) -> object

        Pop the next object from the queue and return it. Block while
        the queue is empty. If a positive timeout is provided and an
        object hasn't been added to the queue in that many seconds then
        raise interpreters.QueueEmpty.

-

    get_nowait() -> object

        Like get(), but do not block. If the queue is not empty then
        return the next item. Otherwise, raise interpreters.QueueEmpty.

Shareable Objects

Interpreter.prepare_main() only works with "shareable" objects. The same
goes for interpreters.Queue (optionally).

A "shareable" object is one which may be passed from one interpreter to
another. The object is not necessarily actually directly shared by the
interpreters. However, even if it isn't, the shared object should be
treated as though it were shared directly. That's a strong equivalence
guarantee for all shareable objects. (See below.)

For some types (builtin singletons), the actual object is shared. For
some, the object's underlying data is actually shared but each
interpreter has a distinct object wrapping that data. For all other
shareable types, a strict copy or proxy is made such that the
corresponding objects continue to match exactly. In cases where the
underlying data is complex but must be copied (e.g. tuple), the data is
serialized as efficiently as possible.

Shareable objects must be specifically supported internally by the
Python runtime. However, there is no restriction against adding support
for more types later.

Here's the initial list of supported objects:

-   str
-   bytes
-   int
-   float
-   bool (True/False)
-   None
-   tuple (only with shareable items)
-   interpreters.Queue
-   memoryview (underlying buffer actually shared)

Note that the last two on the list, queues and memoryview, are
technically mutable data types, whereas the rest are not. When any
interpreters share mutable data there is always a risk of data races.
Cross-interpreter safety, including thread-safety, is a fundamental
feature of queues.

However, memoryview does not have any native accommodations. The user is
responsible for managing thread-safety, whether passing a token back and
forth through a queue to indicate safety (see Synchronization), or by
assigning sub-range exclusivity to individual interpreters.

Most objects will be shared through queues (interpreters.Queue), as
interpreters communicate information between each other. Less
frequently, objects will be shared through prepare_main() to set up an
interpreter prior to running code in it. However, prepare_main() is the
primary way that queues are shared, to provide another interpreter with
a means of further communication.

Finally, a reminder: for a few types the actual object is shared,
whereas for the rest only the underlying data is shared, whether as a
copy or through a proxy. Regardless, it always preserves the strong
equivalence guarantee of "shareable" objects.

The guarantee is that a shared object in one interpreter is strictly
equivalent to the corresponding object in the other interpreter. In
other words, the two objects will be indistinguishable from each other.
The shared object should be treated as though the original had been
shared directly, whether or not it actually was. That's a slightly
different and stronger promise than just equality.

The guarantee is especially important for mutable objects, like
Interpreters.Queue and memoryview. Mutating the object in one
interpreter will always be reflected immediately in every other
interpreter sharing the object.

Synchronization

There are situations where two interpreters should be synchronized. That
may involve sharing a resource, worker management, or preserving
sequential consistency.

In threaded programming the typical synchronization primitives are types
like mutexes. The threading module exposes several. However,
interpreters cannot share objects which means they cannot share
threading.Lock objects.

The interpreters module does not provide any such dedicated
synchronization primitives. Instead, interpreters.Queue objects provide
everything one might need.

For example, if there's a shared resource that needs managed access then
a queue may be used to manage it, where the interpreters pass an object
around to indicate who can use the resource:

    import interpreters
    from mymodule import load_big_data, check_data

    numworkers = 10
    control = interpreters.create_queue()
    data = memoryview(load_big_data())

    def worker():
        interp = interpreters.create()
        interp.prepare_main(control=control, data=data)
        interp.exec("""if True:
            from mymodule import edit_data
            while True:
                token = control.get()
                edit_data(data)
                control.put(token)
            """)
    threads = [threading.Thread(target=worker) for _ in range(numworkers)]
    for t in threads:
        t.start()

    token = 'football'
    control.put(token)
    while True:
        control.get()
        if not check_data(data):
            break
        control.put(token)

Exceptions

-

    InterpreterError

        Indicates that some interpreter-related failure occurred.

        This exception is a subclass of Exception.

-

    InterpreterNotFoundError

        Raised from Interpreter methods after the underlying interpreter
        has been destroyed, e.g. via the C-API.

        This exception is a subclass of InterpreterError.

-

    ExecutionFailed

        Raised from Interpreter.exec() and Interpreter.call() when
        there's an uncaught exception. The error display for this
        exception includes the traceback of the uncaught exception,
        which gets shown after the normal error display, much like
        happens for ExceptionGroup.

        Attributes:

        -   type - a representation of the original exception's class,
            with __name__, __module__, and __qualname__ attrs.
        -   msg - str(exc) of the original exception
        -   snapshot - a traceback.TracebackException object for the
            original exception

        This exception is a subclass of InterpreterError.

-

    QueueError

        Indicates that some queue-related failure occurred.

        This exception is a subclass of Exception.

-

    QueueNotFoundError

        Raised from interpreters.Queue methods after the underlying
        queue has been destroyed.

        This exception is a subclass of QueueError.

-

    QueueEmpty

        Raised from Queue.get() (or get_nowait() with no default) when
        the queue is empty.

        This exception is a subclass of both QueueError and the stdlib
        queue.Empty.

-

    QueueFull

        Raised from Queue.put() (with a timeout) or put_nowait() when
        the queue is already at its max size.

        This exception is a subclass of both QueueError and the stdlib
        queue.Empty.

InterpreterPoolExecutor

Along with the new interpreters module, there will be a new
concurrent.futures.InterpreterPoolExecutor. It will be a derivative of
ThreadPoolExecutor, where each worker executes in its own thread, but
each with its own subinterpreter.

Like the other executors, InterpreterPoolExecutor will support callables
for tasks, and for the initializer. Also like the other executors, the
arguments in both cases will be mostly unrestricted. The callables and
arguments will typically be serialized when sent to a worker's
interpreter, e.g. with pickle, like how the ProcessPoolExecutor works.
This contrasts with Interpreter.call(), which will (at least initially)
be much more restricted.

Communication between workers, or between the executor (or generally its
interpreter) and the workers, may still be done through
interpreters.Queue objects, set with the initializer.

sys.implementation.supports_isolated_interpreters

Python implementations are not required to support subinterpreters,
though most major ones do. If an implementation does support them then
sys.implementation.supports_isolated_interpreters will be set to True.
Otherwise it will be False. If the feature is not supported then
importing the interpreters module will raise an ImportError.

Examples

The following examples demonstrate practical cases where multiple
interpreters may be useful.

Example 1:

There's a stream of requests coming in that will be handled via workers
in sub-threads.

-   each worker thread has its own interpreter
-   there's one queue to send tasks to workers and another queue to
    return results
-   the results are handled in a dedicated thread
-   each worker keeps going until it receives a "stop" sentinel (None)
-   the results handler keeps going until all workers have stopped

    import interpreters
    from mymodule import iter_requests, handle_result

    tasks = interpreters.create_queue()
    results = interpreters.create_queue()

    numworkers = 20
    threads = []

    def results_handler():
        running = numworkers
        while running:
            try:
                res = results.get(timeout=0.1)
            except interpreters.QueueEmpty:
                # No workers have finished a request since last time.
                pass
            else:
                if res is None:
                    # A worker has stopped.
                    running -= 1
                else:
                    handle_result(res)
        empty = object()
        assert results.get_nowait(empty) is empty
    threads.append(threading.Thread(target=results_handler))

    def worker():
        interp = interpreters.create()
        interp.prepare_main(tasks=tasks, results=results)
        interp.exec("""if True:
            from mymodule import handle_request, capture_exception

            while True:
                req = tasks.get()
                if req is None:
                    # Stop!
                    break
                try:
                    res = handle_request(req)
                except Exception as exc:
                    res = capture_exception(exc)
                results.put(res)
            # Notify the results handler.
            results.put(None)
            """)
    threads.extend(threading.Thread(target=worker) for _ in range(numworkers))

    for t in threads:
        t.start()

    for req in iter_requests():
        tasks.put(req)
    # Send the "stop" signal.
    for _ in range(numworkers):
        tasks.put(None)

    for t in threads:
        t.join()

Example 2:

This case is similar to the last as there are a bunch of workers in
sub-threads. However, this time the code is chunking up a big array of
data, where each worker processes one chunk at a time. Copying that data
to each interpreter would be exceptionally inefficient, so the code
takes advantage of directly sharing memoryview buffers.

-   all the interpreters share the buffer of the source array
-   each one writes its results to a second shared buffer
-   there's use a queue to send tasks to workers
-   only one worker will ever read any given index in the source array
-   only one worker will ever write to any given index in the results
    (this is how it ensures thread-safety)

    import interpreters
    import queue
    from mymodule import read_large_data_set, use_results

    numworkers = 3
    data, chunksize = read_large_data_set()
    buf = memoryview(data)
    numchunks = (len(buf) + 1) / chunksize
    results = memoryview(b'\0' * numchunks)

    tasks = interpreters.create_queue()

    def worker(id):
        interp = interpreters.create()
        interp.prepare_main(data=buf, results=results, tasks=tasks)
        interp.exec("""if True:
            from mymodule import reduce_chunk

            while True:
                req = tasks.get()
                if res is None:
                    # Stop!
                    break
                resindex, start, end = req
                chunk = data[start: end]
                res = reduce_chunk(chunk)
                results[resindex] = res
            """)
    threads = [threading.Thread(target=worker) for _ in range(numworkers)]
    for t in threads:
        t.start()

    for i in range(numchunks):
        # Assume there's at least one worker running still.
        start = i * chunksize
        end = start + chunksize
        if end > len(buf):
            end = len(buf)
        tasks.put((start, end, i))
    # Send the "stop" signal.
    for _ in range(numworkers):
        tasks.put(None)

    for t in threads:
        t.join()

    use_results(results)

Rationale

A Minimal API

Since the core dev team has no real experience with how users will make
use of multiple interpreters in Python code, this proposal purposefully
keeps the initial API as lean and minimal as possible. The objective is
to provide a well-considered foundation on which further (more advanced)
functionality may be added later, as appropriate.

That said, the proposed design incorporates lessons learned from
existing use of subinterpreters by the community, from existing stdlib
modules, and from other programming languages. It also factors in
experience from using subinterpreters in the CPython test suite and
using them in concurrency benchmarks.

create(), create_queue()

Typically, users call a type to create instances of the type, at which
point the object's resources get provisioned. The interpreters module
takes a different approach, where users must call create() to get a new
interpreter or create_queue() for a new queue. Calling
interpreters.Interpreter() directly only returns a wrapper around an
existing interpreters (likewise for interpreters.Queue()).

This is because interpreters (and queues) are special resources. They
exist globally in the process and are not managed/owned by the current
interpreter. Thus the interpreters module makes creating an interpreter
(or queue) a visibly distinct operation from creating an instance of
interpreters.Interpreter (or interpreters.Queue).

Interpreter.prepare_main() Sets Multiple Variables

prepare_main() may be seen as a setter function of sorts. It supports
setting multiple names at once, e.g.
interp.prepare_main(spam=1, eggs=2), whereas most setters set one item
at a time. The main reason is for efficiency.

To set a value in the interpreter's __main__.__dict__, the
implementation must first switch the OS thread to the identified
interpreter, which involves some non-negligible overhead. After setting
the value it must switch back. Furthermore, there is some additional
overhead to the mechanism by which it passes objects between
interpreters, which can be reduced in aggregate if multiple values are
set at once.

Therefore, prepare_main() supports setting multiple values at once.

Propagating Exceptions

An uncaught exception from a subinterpreter, via Interpreter.exec(),
could either be (effectively) ignored, like threading.Thread() does, or
propagated, like the builtin exec() does. Since Interpreter.exec() is a
synchronous operation, like the builtin exec(), uncaught exceptions are
propagated.

However, such exceptions are not raised directly. That's because
interpreters are isolated from each other and must not share objects,
including exceptions. That could be addressed by raising a surrogate of
the exception, whether a summary, a copy, or a proxy that wraps it. Any
of those could preserve the traceback, which is useful for debugging.
The ExecutionFailed that gets raised is such a surrogate.

There's another concern to consider. If a propagated exception isn't
immediately caught, it will bubble up through the call stack until
caught (or not). In the case that code somewhere else may catch it, it
is helpful to identify that the exception came from a subinterpreter
(i.e. a "remote" source), rather than from the current interpreter.
That's why Interpreter.exec() raises ExecutionFailed and why it is a
plain Exception, rather than a copy or proxy with a class that matches
the original exception. For example, an uncaught ValueError from a
subinterpreter would never get caught in a later
try: ... except ValueError: .... Instead, ExecutionFailed must be
handled directly.

In contrast, exceptions propagated from Interpreter.call() do not
involve ExecutionFailed but are raised directly, as though originating
in the calling interpreter. This is because Interpreter.call() is a
higher level method that uses pickle to support objects that can't
normally be passed between interpreters.

Limited Object Sharing

As noted in Interpreter Isolation, only a small number of builtin
objects may be truly shared between interpreters. In all other cases
objects can only be shared indirectly, through copies or proxies.

The set of objects that are shareable as copies through queues (and
Interpreter.prepare_main()) is limited for the sake of efficiency.

Supporting sharing of all objects is possible (via pickle) but not part
of this proposal. For one thing, it's helpful to know in those cases
that only an efficient implementation is being used. Furthermore, in
those cases supporting mutable objects via pickling would violate the
guarantee that "shared" objects be equivalent (and stay that way).

Objects vs. ID Proxies

For both interpreters and queues, the low-level module makes use of
proxy objects that expose the underlying state by their corresponding
process-global IDs. In both cases the state is likewise process-global
and will be used by multiple interpreters. Thus they aren't suitable to
be implemented as PyObject, which is only really an option for
interpreter-specific data. That's why the interpreters module instead
provides objects that are weakly associated through the ID.

Rejected Ideas

See PEP 554 <554#rejected-ideas>.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 608 Title: Coordinated Python release Author: Miro Hrončok
<EMAIL_ADDRESS>Victor Stinner<EMAIL_ADDRESS>Status: Rejected
Type: Standards Track Content-Type: text/x-rst Created: 25-Oct-2019
Python-Version: 3.9

Abstract

Block a Python release until a compatible version of selected projects
is available.

The Python release manager can decide to release Python even if a
project is not compatible, if they decide that the project is going to
be fixed soon enough, or if the issue severity is low enough.

Rationale

The PEP involves maintainers of the selected projects in the Python
release cycle. There are multiple benefit:

-   Detect more bugs before a Python final release
-   Discuss and maybe revert incompatible changes before a Python final
    release
-   Increase the number of compatible projects when the new Python final
    version is released

Too few projects are involved in the Python beta phase

Currently, Python beta versions are available four months before the
final 3.x.0 release.

Bugs reported during the beta phase can be easily fixed and can block a
release if they are serious enough.

Incompatible changes are discussed during the beta phase: enhance
documentation explaining how to update code, or consider to revert these
changes.

Even if more and more projects are tested on the master branch of Python
in their CI, too many projects of the top 50 PyPI projects are only
compatible with the new Python a few weeks, or even months, after the
final Python release.

DeprecationWarning is being ignored

Python has well defined process to deprecate features. A
DeprecationWarning must be emitted during at least one Python release,
before a feature can be removed.

In practice, DeprecationWarning warnings are ignored for years in major
Python projects. Usually, maintainers explain that there are too many
warnings and so they simply ignore warnings. Moreover,
DeprecationWarning is silent by default (except in the __main__ module:
PEP 565).

Even if more and more projects are running their test suite with
warnings treated as errors (-Werror), Python core developers still have
no idea how many projects are broken when a feature is removed.

Need to coordinate

When issues and incompatible changes are discovered and discussed after
the final Python release, it becomes way more complicated and expensive
to fix Python. Once an API is part of an official final release, Python
should provide backward compatibility for the whole 3.x release
lifetime. Some operating systems can be shipped with the buggy final
release and can take several months before being updated.

Too many projects are only updated to the new Python after the final
Python release, which makes this new Python version barely usable to run
large applications when Python is released.

It is proposed to block a Python release until a compatible version of
all selected projects is available.

Shorter Python release schedule

The PEP 602: Annual Release Cycle for Python
<602> and the PEP 605: A
rolling feature release stream for CPython
<605> would like to release Python more often to ship new features more
quickly.

The problem is that each Python 3.x release breaks many projects.

Coordinated Python releases reduces the number of broken projects and
makes new Python release more usable.

Specification

By default, a Python release is blocked until a compatible version of
all selected projects is available.

Before releasing the final Python version, the Python release manager is
responsible to send a report of the compatibility status of each project
of the selected projects. It is recommended to send such report at each
beta release to see the evolution and detects issues as soon as
possible.

The Python release manager can decide to release Python even if a
project is not compatible, if they decide that the project is going to
be fixed soon enough, or if the issue severity is low enough.

After each Python release, the project list can be updated to remove
projects and add new ones. For example, to remove old unused
dependencies and add new ones. The list can grow if the whole process
doesn't block Python releases for too long.

Limit the delay

When a build or test issue with the next Python version is reported to a
project, maintainers have one month to answer. With no answer, the
project can be excluded from the list of projects blocking the Python
release.

Multiple projects are already tested on the master branch of Python in a
CI. Problems can be detected very early in a Python release which should
provide enough time to handle them. More CI can be added for projects
which are not tested on the next Python yet.

Once selected projects issues are known, exceptions can be discussed
between the Python release manager and involved project maintainers on a
case-by-case basis. Not all issues deserve to block a Python release.

Selected projects

List of projects blocking a Python release (total: 27):

-   Projects (13):
    -   aiohttp
    -   cryptography
    -   Cython
    -   Django
    -   numpy
    -   pandas
    -   pip
    -   requests
    -   scipy
    -   Sphinx (needed to build Python)
    -   sqlalchemy
    -   pytest
    -   tox
-   Direct and indirect dependencies (14):
    -   certifi (needed by urllib3)
    -   cffi (needed by cryptography)
    -   chardet (needed by Sphinx)
    -   colorama (needed by pip)
    -   docutils (needed by Sphinx)
    -   idna (needed by Sphinx and requests)
    -   jinja2 (needed by Sphinx)
    -   MarkupSafe (needed by Sphinx)
    -   psycopg2 (needed by Django)
    -   pycparser (needed by cffi)
    -   setuptools (needed by pip and tons of Python projects)
    -   six (needed by tons of Python projects)
    -   urllib3 (needed by requests)
    -   wheel (needed by pip)

How projects are selected

Projects used by to build Python should be in the list, like Sphinx.

Most popular projects are picked from the most downloaded PyPI projects.

Most of project dependencies are included in the list as well, since a
single incompatible dependency can block a whole project. Some
dependencies are excluded to reduce the list length.

Test dependencies as pytest and tox should be included as well. If a
project cannot be tested, a new version cannot be shipped neither.

The list should be long enough to have a good idea of the cost of
porting a project to the next Python, but small enough to not block a
Python release for too long.

Obviously, projects which are not part of the list also are encouraged
to report issues with the next Python and to have a CI running on the
next Python version.

Incompatible changes

The definition here is large: any Python change which cause an issue
when building or testing a project.

See also the PEP 606: Python Compatibility Version
<606> for more examples of incompatible changes.

Examples

There are different kinds of incompatible changes:

-   Change in the Python build. For example, Python 3.8 removed 'm'
    (which stands for pymalloc) from sys.abiflags which impacts Python
    vendors like Linux distributions.
-   Change in the C extensions build. For example, Python 3.8 no longer
    links C extensions to libpython, and Python 3.7 removed os.errno
    alias to the errno module.
-   Removed function. For example, collections aliases to ABC classes
    have been removed in Python 3.9.
-   Changed function signature:
    -   Reject a type which was previously accepted (ex: only accept
        int, reject float).
    -   Add a new mandatory parameter.
    -   Convert a positional-or-keyword parameter to positional-only.
-   Behavior change. For example, Python 3.8 now serializes XML
    attributes in their insertion order, rather than sorting them by
    name.
-   New warning. Since more and more projects are tested with all
    warnings treated as errors, any new warning can cause a project test
    to fail.
-   Function removed from the C API.
-   Structure made opaque in the C API. For example, PyInterpreterState
    became opaque in Python 3.8 which broke projects accessing
    interp->modules (PyImport_GetModuleDict() should be used instead).

Cleaning up Python and DeprecationWarning

One of the Zen of Python (PEP 20)
<20> motto is:

  There should be one-- and preferably only one --obvious way to do it.

When Python evolves, new ways emerge inevitably. DeprecationWarning are
emitted to suggest to use the new way, but many developers ignore these
warnings which are silent by default.

Sometimes, supporting both ways has a minor maintenance cost, but Python
core developers prefer to drop the old way to clean up the Python code
base and standard library. Such kind of change is backward incompatible.

More incompatible changes than usual should be expected with the end of
the Python 2 support which is a good opportunity to cleaning up old
Python code.

Distributed CI

Checking if selected projects are compatible with the master branch of
Python can be automated using a distributed CI.

Existing CIs can be reused.

New CIs can be added for projects which are not tested on the next
Python yet.

It is recommended to treat DeprecationWarning warnings as errors when
testing on the next Python.

A job testing a project on the next Python doesn't have to be
"mandatory" (block the whole CI). It is fine to have failures during the
beta phase of a Python release. The job only has to pass for the final
Python release.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

license: Public Domain


text:
PEP: 274 Title: Dict Comprehensions Version: $Revision$ Last-Modified:
$Date$ Author: Barry Warsaw<EMAIL_ADDRESS>Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 25-Oct-2001
Python-Version: 2.7, 3.0 Post-History: 29-Oct-2001

Abstract

PEP 202 introduces a syntactical extension to Python called the "list
comprehension". This PEP proposes a similar syntactical extension called
the "dictionary comprehension" or "dict comprehension" for short. You
can use dict comprehensions in ways very similar to list comprehensions,
except that they produce Python dictionary objects instead of list
objects.

Resolution

This PEP was originally written for inclusion in Python 2.3. It was
withdrawn after observation that substantially all of its benefits were
subsumed by generator expressions coupled with the dict() constructor.

However, Python 2.7 and 3.0 introduces this exact feature, as well as
the closely related set comprehensions. On 2012-04-09, the PEP was
changed to reflect this reality by updating its Status to Accepted, and
updating the Python-Version field. The Open Questions section was also
removed since these have been long resolved by the current
implementation.

Proposed Solution

Dict comprehensions are just like list comprehensions, except that you
group the expression using curly braces instead of square braces. Also,
the left part before the for keyword expresses both a key and a value,
separated by a colon. The notation is specifically designed to remind
you of list comprehensions as applied to dictionaries.

Rationale

There are times when you have some data arranged as a sequences of
length-2 sequences, and you want to turn that into a dictionary. In
Python 2.2, the dict() constructor accepts an argument that is a
sequence of length-2 sequences, used as (key, value) pairs to initialize
a new dictionary object.

However, the act of turning some data into a sequence of length-2
sequences can be inconvenient or inefficient from a memory or
performance standpoint. Also, for some common operations, such as
turning a list of things into a set of things for quick duplicate
removal or set inclusion tests, a better syntax can help code clarity.

As with list comprehensions, an explicit for loop can always be used
(and in fact was the only way to do it in earlier versions of Python).
But as with list comprehensions, dict comprehensions can provide a more
syntactically succinct idiom that the traditional for loop.

Semantics

The semantics of dict comprehensions can actually be demonstrated in
stock Python 2.2, by passing a list comprehension to the built-in
dictionary constructor:

    >>> dict([(i, chr(65+i)) for i in range(4)])

is semantically equivalent to:

    >>> {i : chr(65+i) for i in range(4)}

The dictionary constructor approach has two distinct disadvantages from
the proposed syntax though. First, it isn't as legible as a dict
comprehension. Second, it forces the programmer to create an in-core
list object first, which could be expensive.

Examples

    >>> print {i : chr(65+i) for i in range(4)}
    {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'}

    >>> print {k : v for k, v in someDict.iteritems()} == someDict.copy()
    1

    >>> print {x.lower() : 1 for x in list_of_email_addrs}
<EMAIL_ADDRESS> : 1<EMAIL_ADDRESS>: 1<EMAIL_ADDRESS>: 1}

    >>> def invert(d):
    ...     return {v : k for k, v in d.iteritems()}
    ...
    >>> d = {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'}
    >>> print invert(d)
    {'A' : 0, 'B' : 1, 'C' : 2, 'D' : 3}

    >>> {(k, v): k+v for k in range(4) for v in range(4)}
    ... {(3, 3): 6, (3, 2): 5, (3, 1): 4, (0, 1): 1, (2, 1): 3,
       (0, 2): 2, (3, 0): 3, (0, 3): 3, (1, 1): 2, (1, 0): 1,
       (0, 0): 0, (1, 2): 3, (2, 0): 2, (1, 3): 4, (2, 2): 4, (
       2, 3): 5}

Implementation

All implementation details were resolved in the Python 2.7 and 3.0
time-frame.

Copyright

This document has been placed in the public domain.


text:
PEP: 306 Title: How to Change Python's Grammar Version: $Revision$
Last-Modified: $Date$ Author: Michael Hudson<EMAIL_ADDRESS>Jack
Diederich<EMAIL_ADDRESS>Alyssa Coghlan<EMAIL_ADDRESS>Benjamin Peterson<EMAIL_ADDRESS>Status: Withdrawn Type:
Informational Content-Type: text/x-rst Created: 29-Jan-2003
Post-History: 30-Jan-2003

Note

This PEP has been moved to the Python dev guide[1].

Abstract

There's more to changing Python's grammar than editing Grammar/Grammar
and Python/compile.c. This PEP aims to be a checklist of places that
must also be fixed.

It is probably incomplete. If you see omissions, just add them if you
can -- you are not going to offend the author's sense of ownership.
Otherwise submit a bug or patch and assign it to mwh.

This PEP is not intended to be an instruction manual on Python grammar
hacking, for several reasons.

Rationale

People are getting this wrong all the time; it took well over a year
before someone noticed[2] that adding the floor division operator (//)
broke the parser module.

Checklist

-   Grammar/Grammar: OK, you'd probably worked this one out :)
-   Parser/Python.asdl may need changes to match the Grammar. Run make
    to regenerate Include/Python-ast.h and Python/Python-ast.c.
-   Python/ast.c will need changes to create the AST objects involved
    with the Grammar change. Lib/compiler/ast.py will need matching
    changes to the pure-python AST objects.
-   Parser/pgen needs to be rerun to regenerate Include/graminit.h and
    Python/graminit.c. (make should handle this for you.)
-   Python/symbtable.c: This handles the symbol collection pass that
    happens immediately before the compilation pass.
-   Python/compile.c: You will need to create or modify the compiler_*
    functions to generate opcodes for your productions.
-   You may need to regenerate Lib/symbol.py and/or Lib/token.py and/or
    Lib/keyword.py.
-   The parser module. Add some of your new syntax to test_parser, bang
    on Modules/parsermodule.c until it passes.
-   Add some usage of your new syntax to test_grammar.py.
-   The compiler package. A good test is to compile the standard library
    and test suite with the compiler package and then check it runs.
    Note that this only needs to be done in Python 2.x.
-   If you've gone so far as to change the token structure of Python,
    then the Lib/tokenizer.py library module will need to be changed.
-   Certain changes may require tweaks to the library module pyclbr.
-   Documentation must be written!
-   After everything's been checked in, you're likely to see a new
    change to Python/Python-ast.c. This is because this (generated) file
    contains the SVN version of the source from which it was generated.
    There's no way to avoid this; you just have to submit this file
    separately.

References

Copyright

This document has been placed in the public domain.

[1] CPython Developer's Guide: Changing CPython's Grammar
https://devguide.python.org/grammar/

[2] SF Bug #676521, parser module validation failure
https://bugs.python.org/issue676521

authors:
Alyssa Coghlan
Benjamin Peterson
Jack Diederich
Michael Hudson


text:
PEP: 308 Title: Conditional Expressions Version: $Revision$
Last-Modified: $Date$ Author: Guido van Rossum, Raymond Hettinger
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
07-Feb-2003 Python-Version: 2.5 Post-History: 07-Feb-2003, 11-Feb-2003

Adding a conditional expression

On 9/29/2005, Guido decided to add conditional expressions in the form
of "X if C else Y".[1]

The motivating use case was the prevalence of error-prone attempts to
achieve the same effect using "and" and "or".[2]

Previous community efforts to add a conditional expression were stymied
by a lack of consensus on the best syntax. That issue was resolved by
simply deferring to a BDFL best judgment call.

The decision was validated by reviewing how the syntax fared when
applied throughout the standard library (this review approximates a
sampling of real-world use cases, across a variety of applications,
written by a number of programmers with diverse backgrounds).[3]

The following change will be made to the grammar. (The or_test symbols
is new, the others are modified.)

    test: or_test ['if' or_test 'else' test] | lambdef
    or_test: and_test ('or' and_test)*
    ...
    testlist_safe: or_test [(',' or_test)+ [',']]
    ...
    gen_for: 'for' exprlist 'in' or_test [gen_iter]

The new syntax nearly introduced a minor syntactical backwards
incompatibility. In previous Python versions, the following is legal:

    [f for f in lambda x: x, lambda x: x**2 if f(1) == 1]

(I.e. a list comprehension where the sequence following 'in' is an
unparenthesized series of lambdas -- or just one lambda, even.)

In Python 3.0, the series of lambdas will have to be parenthesized,
e.g.:

    [f for f in (lambda x: x, lambda x: x**2) if f(1) == 1]

This is because lambda binds less tight than the if-else expression, but
in this context, the lambda could already be followed by an 'if' keyword
that binds less tightly still (for details, consider the grammar changes
shown above).

However, in Python 2.5, a slightly different grammar is used that is
more backwards compatible, but constrains the grammar of a lambda used
in this position by forbidding the lambda's body to contain an
unparenthesized condition expression. Examples:

    [f for f in (1, lambda x: x if x >= 0 else -1)]    # OK
    [f for f in 1, (lambda x: x if x >= 0 else -1)]    # OK
    [f for f in 1, lambda x: (x if x >= 0 else -1)]    # OK
    [f for f in 1, lambda x: x if x >= 0 else -1]      # INVALID

References

Introduction to earlier draft of the PEP (kept for historical purposes)

Requests for an if-then-else ("ternary") expression keep coming up on
comp.lang.python. This PEP contains a concrete proposal of a fairly
Pythonic syntax. This is the community's one chance: if this PEP is
approved with a clear majority, it will be implemented in Python 2.4. If
not, the PEP will be augmented with a summary of the reasons for
rejection and the subject better not come up again. While the BDFL is
co-author of this PEP, he is neither in favor nor against this proposal;
it is up to the community to decide. If the community can't decide, the
BDFL will reject the PEP.

After unprecedented community response (very good arguments were made
both pro and con) this PEP has been revised with the help of Raymond
Hettinger. Without going through a complete revision history, the main
changes are a different proposed syntax, an overview of proposed
alternatives, the state of the current discussion, and a discussion of
short-circuit behavior.

Following the discussion, a vote was held. While there was an overall
interest in having some form of if-then-else expressions, no one format
was able to draw majority support. Accordingly, the PEP was rejected due
to the lack of an overwhelming majority for change. Also, a Python
design principle has been to prefer the status quo whenever there are
doubts about which path to take.

Proposal

The proposed syntax is as follows:

    (if <condition>: <expression1> else: <expression2>)

Note that the enclosing parentheses are not optional.

The resulting expression is evaluated like this:

-   First, <condition> is evaluated.
-   If <condition> is true, <expression1> is evaluated and is the result
    of the whole thing.
-   If <condition> is false, <expression2> is evaluated and is the
    result of the whole thing.

A natural extension of this syntax is to allow one or more 'elif' parts:

    (if <cond1>: <expr1> elif <cond2>: <expr2> ... else: <exprN>)

This will be implemented if the proposal is accepted.

The downsides to the proposal are:

-   the required parentheses
-   confusability with statement syntax
-   additional semantic loading of colons

Note that at most one of <expression1> and <expression2> is evaluated.
This is called a "short-circuit expression"; it is similar to the way
the second operand of 'and' / 'or' is only evaluated if the first
operand is true / false.

A common way to emulate an if-then-else expression is:

    <condition> and <expression1> or <expression2>

However, this doesn't work the same way: it returns <expression2> when
<expression1> is false! See FAQ 4.16 for alternatives that work --
however, they are pretty ugly and require much more effort to
understand.

Alternatives

Holger Krekel proposed a new, minimally invasive variant:

    <condition> and <expression1> else <expression2>

The concept behind it is that a nearly complete ternary operator already
exists with and/or and this proposal is the least invasive change that
makes it complete. Many respondants on the newsgroup found this to be
the most pleasing alternative. However, a couple of respondants were
able to post examples that were mentally difficult to parse. Later it
was pointed out that this construct works by having the "else" change
the existing meaning of "and".

As a result, there is increasing support for Christian Tismer's proposed
variant of the same idea:

    <condition> then <expression1> else <expression2>

The advantages are simple visual parsing, no required parentheses, no
change in the semantics of existing keywords, not as likely as the
proposal to be confused with statement syntax, and does not further
overload the colon. The disadvantage is the implementation costs of
introducing a new keyword. However, unlike other new keywords, the word
"then" seems unlikely to have been used as a name in existing programs.

---

Many C-derived languages use this syntax:

    <condition> ? <expression1> : <expression2>

Eric Raymond even implemented this. The BDFL rejected this for several
reasons: the colon already has many uses in Python (even though it would
actually not be ambiguous, because the question mark requires a matching
colon); for people not used to C-derived language, it is hard to
understand.

---

The original version of this PEP proposed the following syntax:

    <expression1> if <condition> else <expression2>

The out-of-order arrangement was found to be too uncomfortable for many
of participants in the discussion; especially when <expression1> is
long, it's easy to miss the conditional while skimming.

---

Some have suggested adding a new builtin instead of extending the syntax
of the language. For example:

    cond(<condition>, <expression1>, <expression2>)

This won't work the way a syntax extension will because both expression1
and expression2 must be evaluated before the function is called. There's
no way to short-circuit the expression evaluation. It could work if
'cond' (or some other name) were made a keyword, but that has all the
disadvantages of adding a new keyword, plus confusing syntax: it looks
like a function call so a casual reader might expect both <expression1>
and <expression2> to be evaluated.

Summary of the Current State of the Discussion

Groups are falling into one of three camps:

1.  Adopt a ternary operator built using punctuation characters:

        <condition> ? <expression1> : <expression2>

2.  Adopt a ternary operator built using new or existing keywords. The
    leading examples are:

        <condition> then <expression1> else <expression2>
        (if <condition>: <expression1> else: <expression2>)

3.  Do nothing.

The first two positions are relatively similar.

Some find that any form of punctuation makes the language more cryptic.
Others find that punctuation style is appropriate for expressions rather
than statements and helps avoid a COBOL style: 3 plus 4 times 5.

Adapting existing keywords attempts to improve on punctuation through
explicit meaning and a more tidy appearance. The downside is some loss
of the economy-of-expression provided by punctuation operators. The
other downside is that it creates some degree of confusion between the
two meanings and two usages of the keywords.

Those difficulties are overcome by options which introduce new keywords
which take more effort to implement.

The last position is doing nothing. Arguments in favor include keeping
the language simple and concise; maintaining backwards compatibility;
and that any every use case can already be already expressed in terms of
"if" and "else". Lambda expressions are an exception as they require the
conditional to be factored out into a separate function definition.

The arguments against doing nothing are that the other choices allow
greater economy of expression and that current practices show a
propensity for erroneous uses of "and", "or", or one their more complex,
less visually unappealing workarounds.

Short-Circuit Behavior

The principal difference between the ternary operator and the cond()
function is that the latter provides an expression form but does not
provide short-circuit evaluation.

Short-circuit evaluation is desirable on three occasions:

1.  When an expression has side-effects
2.  When one or both of the expressions are resource intensive
3.  When the condition serves as a guard for the validity of the
    expression.

    #  Example where all three reasons apply
    data = isinstance(source, file)  ?  source.readlines()
                                     :  source.split()

1.  readlines() moves the file pointer
2.  for long sources, both alternatives take time
3.  split() is only valid for strings and readlines() is only valid for
    file objects.

Supporters of a cond() function point out that the need for
short-circuit evaluation is rare. Scanning through existing code
directories, they found that if/else did not occur often; and of those
only a few contained expressions that could be helped by cond() or a
ternary operator; and that most of those had no need for short-circuit
evaluation. Hence, cond() would suffice for most needs and would spare
efforts to alter the syntax of the language.

More supporting evidence comes from scans of C code bases which show
that its ternary operator used very rarely (as a percentage of lines of
code).

A counterpoint to that analysis is that the availability of a ternary
operator helped the programmer in every case because it spared the need
to search for side-effects. Further, it would preclude errors arising
from distant modifications which introduce side-effects. The latter case
has become more of a reality with the advent of properties where even
attribute access can be given side-effects.

The BDFL's position is that short-circuit behavior is essential for an
if-then-else construct to be added to the language.

Detailed Results of Voting

    Votes rejecting all options:  82
    Votes with rank ordering:     436
                                  ---
    Total votes received:         518

            ACCEPT                  REJECT                  TOTAL
            ---------------------   ---------------------   -----
            Rank1   Rank2   Rank3   Rank1   Rank2   Rank3
    Letter
    A       51      33      19      18      20      20      161
    B       45      46      21      9       24      23      168
    C       94      54      29      20      20      18      235
    D       71      40      31      5       28      31      206
    E       7       7       10              3       5       32
    F       14      19      10              7       17      67
    G       7       6       10      1       2       4       30
    H       20      22      17      4       10      25      98
    I       16      20      9       5       5       20      75
    J       6       17      5       1               10      39
    K       1               6               4       13      24
    L               1       2               3       3       9
    M       7       3       4       2       5       11      32
    N               2       3               4       2       11
    O       1       6       5       1       4       9       26
    P       5       3       6       1       5       7       27
    Q       18      7       15      6       5       11      62
    Z                                               1       1
            ---     ---     ---     ---     ---     ---     ----
    Total   363     286     202     73      149     230     1303
    RejectAll                       82      82      82      246
            ---     ---     ---     ---     ---     ---     ----
    Total   363     286     202     155     231     312     1549

CHOICE KEY

    A.  x if C else y
    B.  if C then x else y
    C.  (if C: x else: y)
    D.  C ? x : y
    E.  C ? x ! y
    F.  cond(C, x, y)
    G.  C ?? x || y
    H.  C then x else y
    I.  x when C else y
    J.  C ? x else y
    K.  C -> x else y
    L.  C -> (x, y)
    M.  [x if C else y]
    N.  ifelse C: x else y
    O.  <if C then x else y>
    P.  C and x else y
    Q.  any write-in vote

Detail for write-in votes and their ranking

    3:  Q reject y x C elsethenif
    2:  Q accept (C ? x ! y)
    3:  Q reject ...
    3:  Q accept  ? C : x : y
    3:  Q accept (x if C, y otherwise)
    3:  Q reject ...
    3:  Q reject NONE
    1:  Q accept   select : (<c1> : <val1>; [<cx> : <valx>; ]* elseval)
    2:  Q reject if C: t else: f
    3:  Q accept C selects x else y
    2:  Q accept iff(C, x, y)    # "if-function"
    1:  Q accept (y, x)[C]
    1:  Q accept          C true: x false: y
    3:  Q accept          C then: x else: y
    3:  Q reject
    3:  Q accept (if C: x elif C2: y else: z)
    3:  Q accept C -> x : y
    1:  Q accept  x (if C), y
    1:  Q accept if c: x else: y
    3:  Q accept (c).{True:1, False:2}
    2:  Q accept if c: x else: y
    3:  Q accept (c).{True:1, False:2}
    3:  Q accept if C: x else y
    1:  Q accept  (x if C else y)
    1:  Q accept ifelse(C, x, y)
    2:  Q reject x or y <- C
    1:  Q accept (C ? x : y) required parens
    1:  Q accept  iif(C, x, y)
    1:  Q accept ?(C, x, y)
    1:  Q accept switch-case
    2:  Q accept multi-line if/else
    1:  Q accept C: x else: y
    2:  Q accept (C): x else: y
    3:  Q accept if C: x else: y
    1:  Q accept     x if C, else y
    1:  Q reject choice: c1->a; c2->b; ...; z
    3:  Q accept [if C then x else y]
    3:  Q reject no other choice has x as the first element
    1:  Q accept (x,y) ? C
    3:  Q accept x if C else y (The "else y" being optional)
    1:  Q accept (C ? x , y)
    1:  Q accept  any outcome (i.e form or plain rejection) from a usability study
    1:  Q reject (x if C else y)
    1:  Q accept  (x if C else y)
    2:  Q reject   NONE
    3:  Q reject   NONE
    3:  Q accept  (C ? x else y)
    3:  Q accept  x when C else y
    2:  Q accept  (x if C else y)
    2:  Q accept cond(C1, x1, C2, x2, C3, x3,...)
    1:  Q accept  (if C1: x elif C2: y else: z)
    1:  Q reject cond(C, :x, :y)
    3:  Q accept  (C and [x] or [y])[0]
    2:  Q reject
    3:  Q reject
    3:  Q reject all else
    1:  Q reject no-change
    3:  Q reject deliberately omitted as I have no interest in any other proposal
    2:  Q reject (C then x else Y)
    1:  Q accept       if C: x else: y
    1:  Q reject (if C then x else y)
    3:  Q reject C?(x, y)

Copyright

This document has been placed in the public domain.

[1] Pronouncement
https://mail.python.org/pipermail/python-dev/2005-September/056846.html

[2] Motivating use case:
https://mail.python.org/pipermail/python-dev/2005-September/056546.html
https://mail.python.org/pipermail/python-dev/2005-September/056510.html

[3] Review in the context of real-world code fragments:
https://mail.python.org/pipermail/python-dev/2005-September/056803.html

authors:
Guido van Rossum
Raymond Hettinger


text:
PEP: 3104 Title: Access to Names in Outer Scopes Author: Ka-Ping Yee
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 12-Oct-2006 Python-Version: 3.0 Post-History:

Abstract

In most languages that support nested scopes, code can refer to or
rebind (assign to) any name in the nearest enclosing scope. Currently,
Python code can refer to a name in any enclosing scope, but it can only
rebind names in two scopes: the local scope (by simple assignment) or
the module-global scope (using a global declaration).

This limitation has been raised many times on the Python-Dev mailing
list and elsewhere, and has led to extended discussion and many
proposals for ways to remove this limitation. This PEP summarizes the
various alternatives that have been suggested, together with advantages
and disadvantages that have been mentioned for each.

Rationale

Before version 2.1, Python's treatment of scopes resembled that of
standard C: within a file there were only two levels of scope, global
and local. In C, this is a natural consequence of the fact that function
definitions cannot be nested. But in Python, though functions are
usually defined at the top level, a function definition can be executed
anywhere. This gave Python the syntactic appearance of nested scoping
without the semantics, and yielded inconsistencies that were surprising
to some programmers -- for example, a recursive function that worked at
the top level would cease to work when moved inside another function,
because the recursive function's own name would no longer be visible in
its body's scope. This violates the intuition that a function should
behave consistently when placed in different contexts. Here's an
example:

    def enclosing_function():
        def factorial(n):
            if n < 2:
                return 1
            return n * factorial(n - 1)  # fails with NameError
        print factorial(5)

Python 2.1 moved closer to static nested scoping by making visible the
names bound in all enclosing scopes (see PEP 227). This change makes the
above code example work as expected. However, because any assignment to
a name implicitly declares that name to be local, it is impossible to
rebind a name in an outer scope (except when a global declaration forces
the name to be global). Thus, the following code, intended to display a
number that can be incremented and decremented by clicking buttons,
doesn't work as someone familiar with lexical scoping might expect:

    def make_scoreboard(frame, score=0):
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                score = score + step  # fails with UnboundLocalError
                label['text'] = score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

Python syntax doesn't provide a way to indicate that the name score
mentioned in increment refers to the variable score bound in
make_scoreboard, not a local variable in increment. Users and developers
of Python have expressed an interest in removing this limitation so that
Python can have the full flexibility of the Algol-style scoping model
that is now standard in many programming languages, including
JavaScript, Perl, Ruby, Scheme, Smalltalk, C with GNU extensions, and C#
2.0.

It has been argued that such a feature isn't necessary, because a
rebindable outer variable can be simulated by wrapping it in a mutable
object:

    class Namespace:
        pass

    def make_scoreboard(frame, score=0):
        ns = Namespace()
        ns.score = 0
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                ns.score = ns.score + step
                label['text'] = ns.score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

However, this workaround only highlights the shortcomings of existing
scopes: the purpose of a function is to encapsulate code in its own
namespace, so it seems unfortunate that the programmer should have to
create additional namespaces to make up for missing functionality in the
existing local scopes, and then have to decide whether each name should
reside in the real scope or the simulated scope.

Another common objection is that the desired functionality can be
written as a class instead, albeit somewhat more verbosely. One rebuttal
to this objection is that the existence of a different implementation
style is not a reason to leave a supported programming construct (nested
scopes) functionally incomplete. Python is sometimes called a
"multi-paradigm language" because it derives so much strength, practical
flexibility, and pedagogical power from its support and graceful
integration of multiple programming paradigms.

A proposal for scoping syntax appeared on Python-Dev as far back as
1994[1], long before PEP 227's support for nested scopes was adopted. At
the time, Guido's response was:

  This is dangerously close to introducing CSNS [classic static nested
  scopes]. If you were to do so, your proposed semantics of scoped seem
  alright. I still think there is not enough need for CSNS to warrant
  this kind of construct ...

After PEP 227, the "outer name rebinding discussion" has reappeared on
Python-Dev enough times that it has become a familiar event, having
recurred in its present form since at least 2003[2]. Although none of
the language changes proposed in these discussions have yet been
adopted, Guido has acknowledged that a language change is worth
considering[3].

Other Languages

To provide some background, this section describes how some other
languages handle nested scopes and rebinding.

JavaScript, Perl, Scheme, Smalltalk, GNU C, C# 2.0

These languages use variable declarations to indicate scope. In
JavaScript, a lexically scoped variable is declared with the var
keyword; undeclared variable names are assumed to be global. In Perl, a
lexically scoped variable is declared with the my keyword; undeclared
variable names are assumed to be global. In Scheme, all variables must
be declared (with define or let, or as formal parameters). In Smalltalk,
any block can begin by declaring a list of local variable names between
vertical bars. C and C# require type declarations for all variables. For
all these cases, the variable belongs to the scope containing the
declaration.

Ruby (as of 1.8)

Ruby is an instructive example because it appears to be the only other
currently popular language that, like Python, tries to support
statically nested scopes without requiring variable declarations, and
thus has to come up with an unusual solution. Functions in Ruby can
contain other function definitions, and they can also contain code
blocks enclosed in curly braces. Blocks have access to outer variables,
but nested functions do not. Within a block, an assignment to a name
implies a declaration of a local variable only if it would not shadow a
name already bound in an outer scope; otherwise assignment is
interpreted as rebinding of the outer name. Ruby's scoping syntax and
rules have also been debated at great length, and changes seem likely in
Ruby 2.0[4].

Overview of Proposals

There have been many different proposals on Python-Dev for ways to
rebind names in outer scopes. They all fall into two categories: new
syntax in the scope where the name is bound, or new syntax in the scope
where the name is used.

New Syntax in the Binding (Outer) Scope

Scope Override Declaration

The proposals in this category all suggest a new kind of declaration
statement similar to JavaScript's var. A few possible keywords have been
proposed for this purpose:

-   scope x[5]
-   var x[6][7]
-   my x[8]

In all these proposals, a declaration such as var x in a particular
scope S would cause all references to x in scopes nested within S to
refer to the x bound in S.

The primary objection to this category of proposals is that the meaning
of a function definition would become context-sensitive. Moving a
function definition inside some other block could cause any of the local
name references in the function to become nonlocal, due to declarations
in the enclosing block. For blocks in Ruby 1.8, this is actually the
case; in the following example, the two setters have different effects
even though they look identical:

    setter1 = proc { | x | y = x }      # y is local here
    y = 13
    setter2 = proc { | x | y = x }      # y is nonlocal here
    setter1.call(99)
    puts y                              # prints 13
    setter2.call(77)
    puts y                              # prints 77

Note that although this proposal resembles declarations in JavaScript
and Perl, the effect on the language is different because in those
languages undeclared variables are global by default, whereas in Python
undeclared variables are local by default. Thus, moving a function
inside some other block in JavaScript or Perl can only reduce the scope
of a previously global name reference, whereas in Python with this
proposal, it could expand the scope of a previously local name
reference.

Required Variable Declaration

A more radical proposal[9] suggests removing Python's scope-guessing
convention altogether and requiring that all names be declared in the
scope where they are to be bound, much like Scheme. With this proposal,
var x = 3 would both declare x to belong to the local scope and bind it,
where as x = 3 would rebind the existing visible x. In a context without
an enclosing scope containing a var x declaration, the statement x = 3
would be statically determined to be illegal.

This proposal yields a simple and consistent model, but it would be
incompatible with all existing Python code.

New Syntax in the Referring (Inner) Scope

There are three kinds of proposals in this category.

Outer Reference Expression

This type of proposal suggests a new way of referring to a variable in
an outer scope when using the variable in an expression. One syntax that
has been suggested for this is .x[10], which would refer to x without
creating a local binding for it. A concern with this proposal is that in
many contexts x and .x could be used interchangeably, which would
confuse the reader[11]. A closely related idea is to use multiple dots
to specify the number of scope levels to ascend[12], but most consider
this too error-prone [13].

Rebinding Operator

This proposal suggests a new assignment-like operator that rebinds a
name without declaring the name to be local[14]. Whereas the statement
x = 3 both declares x a local variable and binds it to 3, the statement
x := 3 would change the existing binding of x without declaring it
local.

This is a simple solution, but according to PEP 3099 it has been
rejected (perhaps because it would be too easy to miss or to confuse
with =).

Scope Override Declaration

The proposals in this category suggest a new kind of declaration
statement in the inner scope that prevents a name from becoming local.
This statement would be similar in nature to the global statement, but
instead of making the name refer to a binding in the top module-level
scope, it would make the name refer to the binding in the nearest
enclosing scope.

This approach is attractive due to its parallel with a familiar Python
construct, and because it retains context-independence for function
definitions.

This approach also has advantages from a security and debugging
perspective. The resulting Python would not only match the functionality
of other nested-scope languages but would do so with a syntax that is
arguably even better for defensive programming. In most other languages,
a declaration contracts the scope of an existing name, so inadvertently
omitting the declaration could yield farther-reaching (i.e. more
dangerous) effects than expected. In Python with this proposal, the
extra effort of adding the declaration is aligned with the increased
risk of non-local effects (i.e. the path of least resistance is the
safer path).

Many spellings have been suggested for such a declaration:

-   scoped x[15]
-   global x in f[16] (explicitly specify which scope)
-   free x[17]
-   outer x[18]
-   use x[19]
-   global x[20] (change the meaning of global)
-   nonlocal x[21]
-   global x outer[22]
-   global in x[23]
-   not global x[24]
-   extern x[25]
-   ref x[26]
-   refer x[27]
-   share x[28]
-   sharing x[29]
-   common x[30]
-   using x[31]
-   borrow x[32]
-   reuse x[33]
-   scope f x[34] (explicitly specify which scope)

The most commonly discussed choices appear to be outer, global, and
nonlocal. outer is already used as both a variable name and an attribute
name in the standard library. The word global has a conflicting meaning,
because "global variable" is generally understood to mean a variable
with top-level scope[35]. In C, the keyword extern means that a name
refers to a variable in a different compilation unit. While nonlocal is
a bit long and less pleasant-sounding than some of the other options, it
does have precisely the correct meaning: it declares a name not local.

Proposed Solution

The solution proposed by this PEP is to add a scope override declaration
in the referring (inner) scope. Guido has expressed a preference for
this category of solution on Python-Dev[36] and has shown approval for
nonlocal as the keyword[37].

The proposed declaration:

    nonlocal x

prevents x from becoming a local name in the current scope. All
occurrences of x in the current scope will refer to the x bound in an
outer enclosing scope. As with global, multiple names are permitted:

    nonlocal x, y, z

If there is no pre-existing binding in an enclosing scope, the compiler
raises a SyntaxError. (It may be a bit of a stretch to call this a
syntax error, but so far SyntaxError is used for all compile-time
errors, including, for example, __future__ import with an unknown
feature name.) Guido has said that this kind of declaration in the
absence of an outer binding should be considered an error[38].

If a nonlocal declaration collides with the name of a formal parameter
in the local scope, the compiler raises a SyntaxError.

A shorthand form is also permitted, in which nonlocal is prepended to an
assignment or augmented assignment:

    nonlocal x = 3

The above has exactly the same meaning as nonlocal x; x = 3. (Guido
supports a similar form of the global statement[39].)

On the left side of the shorthand form, only identifiers are allowed,
not target expressions like x[0]. Otherwise, all forms of assignment are
allowed. The proposed grammar of the nonlocal statement is:

    nonlocal_stmt ::=
        "nonlocal" identifier ("," identifier)*
                   ["=" (target_list "=")+ expression_list]
      | "nonlocal" identifier augop expression_list

The rationale for allowing all these forms of assignment is that it
simplifies understanding of the nonlocal statement. Separating the
shorthand form into a declaration and an assignment is sufficient to
understand what it means and whether it is valid.

Note

The shorthand syntax was not added in the original implementation of the
PEP. Later discussions[40][41] concluded this syntax should not be
implemented.

Backward Compatibility

This PEP targets Python 3000, as suggested by Guido[42]. However, others
have noted that some options considered in this PEP may be small enough
changes to be feasible in Python 2.x[43], in which case this PEP could
possibly be moved to be a 2.x series PEP.

As a (very rough) measure of the impact of introducing a new keyword,
here is the number of times that some of the proposed keywords appear as
identifiers in the standard library, according to a scan of the Python
SVN repository on November 5, 2006:

    nonlocal    0
    use         2
    using       3
    reuse       4
    free        8
    outer     147

global appears 214 times as an existing keyword. As a measure of the
impact of using global as the outer-scope keyword, there are 18 files in
the standard library that would break as a result of such a change
(because a function declares a variable global before that variable has
been introduced in the global scope):

    cgi.py
    dummy_thread.py
    mhlib.py
    mimetypes.py
    idlelib/PyShell.py
    idlelib/run.py
    msilib/__init__.py
    test/inspect_fodder.py
    test/test_compiler.py
    test/test_decimal.py
    test/test_descr.py
    test/test_dummy_threading.py
    test/test_fileinput.py
    test/test_global.py (not counted: this tests the keyword itself)
    test/test_grammar.py (not counted: this tests the keyword itself)
    test/test_itertools.py
    test/test_multifile.py
    test/test_scope.py (not counted: this tests the keyword itself)
    test/test_threaded_import.py
    test/test_threadsignals.py
    test/test_warnings.py

References

[15] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066995.html

Acknowledgements

The ideas and proposals mentioned in this PEP are gleaned from countless
Python-Dev postings. Thanks to Jim Jewett, Mike Orr, Jason Orendorff,
and Christian Tanzer for suggesting specific edits to this PEP.

Copyright

This document has been placed in the public domain.

[1] Scoping (was Re: Lambda binding solved?) (Rafael Bracho)
https://legacy.python.org/search/hypermail/python-1994q1/0301.html

[2] Extended Function syntax (Just van Rossum)
https://mail.python.org/pipermail/python-dev/2003-February/032764.html

[3] Lexical scoping in Python 3k (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066950.html

[4] Ruby 2.0 block local variable
https://web.archive.org/web/20070105131417/http://redhanded.hobix.com/inspect/ruby20BlockLocalVariable.html

[5] Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061568.html

[6] Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061568.html

[7] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066862.html

[8] Explicit Lexical Scoping (pre-PEP?) (Talin)
https://mail.python.org/pipermail/python-dev/2006-July/066978.html

[9] Sky pie: a "var" keyword (a thread started by Neil Toronto)
https://mail.python.org/pipermail/python-3000/2006-October/003968.html

[10] Using and binding relative names (Phillip J. Eby)
https://mail.python.org/pipermail/python-dev/2006-February/061636.html

[11] Using and binding relative names (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061761.html

[12] Using and binding relative names (Steven Bethard)
https://mail.python.org/pipermail/python-dev/2006-February/061749.html

[13] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067004.html

[14] Extended Function syntax (Just van Rossum)
https://mail.python.org/pipermail/python-dev/2003-February/032764.html

[15] Scoping (was Re: Lambda binding solved?) (Rafael Bracho)
https://legacy.python.org/search/hypermail/python-1994q1/0301.html

[16] Closure semantics (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2003-October/039214.html

[17] PEP for Better Control of Nested Lexical Scopes (Jeremy Hylton)
https://mail.python.org/pipermail/python-dev/2006-February/061602.html

[18] PEP for Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061603.html

[19] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066862.html

[20] Lexical scoping in Python 3k (Greg Ewing)
https://mail.python.org/pipermail/python-dev/2006-July/066889.html

[21] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066942.html

[22] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
https://mail.python.org/pipermail/python-dev/2006-July/067007.html

[23] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
https://mail.python.org/pipermail/python-dev/2006-July/067007.html

[24] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
https://mail.python.org/pipermail/python-dev/2006-July/067007.html

[25] Explicit Lexical Scoping (pre-PEP?) (Matthew Barnes)
https://mail.python.org/pipermail/python-dev/2006-July/067221.html

[26] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[27] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[28] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[29] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[30] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[31] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[32] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[33] Alternatives to 'outer' (Jim Jewett)
https://mail.python.org/pipermail/python-3000/2006-November/004153.html

[34] Draft PEP for outer scopes (Talin)
https://mail.python.org/pipermail/python-3000/2006-November/004190.html

[35] Global variable (version 2006-11-01T01:23:16)
https://en.wikipedia.org/w/index.php?title=Global_variable&oldid=85001451

[36] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066991.html

[37] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067067.html

[38] Lexical scoping in Python 3k (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066968.html

[39] Draft PEP for outer scopes (Guido van Rossum)
https://mail.python.org/pipermail/python-3000/2006-November/004166.html

[40] Issue 4199: combining assignment with global & nonlocal (Guido van
Rossum)
https://mail.python.org/pipermail/python-dev/2013-June/127142.html

[41] Whatever happened to 'nonlocal x = y'? (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2018-January/151627.html

[42] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067067.html

[43] Draft PEP for outer scopes (Alyssa Coghlan)
https://mail.python.org/pipermail/python-3000/2006-November/004237.html