text:
PEP: 558 Title: Defined semantics for locals() Author: Alyssa Coghlan
<EMAIL_ADDRESS>BDFL-Delegate: Nathaniel J. Smith Discussions-To:
<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 08-Sep-2017 Python-Version: 3.13
Post-History: 08-Sep-2017, 22-May-2019, 30-May-2019, 30-Dec-2019,
18-Jul-2021, 26-Aug-2021

PEP Withdrawal

In December 2021, this PEP and PEP 667 converged on a common definition
of the proposed changes to the Python level semantics of the locals()
builtin (as documented in the PEP text below), with the only remaining
differences being in the proposed C API changes and various internal
implementation details.

Of those remaining differences, the most significant one was that PEP
667 at the time still proposed an immediate backwards compatibility
break for the PyEval_GetLocals() API as soon as the PEP was accepted and
implemented.

PEP 667 has since been changed to propose a generous deprecation period
for the PyEval_GetLocals() API, continuing to support it in parallel
with the improved semantics offered by the new PyEval_GetFrameLocals()
API.

Any remaining C API design concerns relate to new informational APIs
that can be added at a later date if they are deemed necessary, and any
potential concerns about the exact performance characteristics of the
frame locals view implementation are outweighed by the availability of a
viable reference implementation.

Accordingly, this PEP has been withdrawn in favour of proceeding with
PEP 667.

Note: while implementing PEP 667 it became apparent that the rationale
for and impact of locals() being updated to return independent snapshots
in optimized scopes <py3.13:optimized scope> was not entirely clear in
either PEP. The Motivation and Rationale sections in this PEP have been
updated accordingly (since those aspects are equally applicable to the
accepted PEP 667).

Abstract

The semantics of the locals() builtin have historically been
underspecified and hence implementation dependent.

This PEP proposes formally standardising on the behaviour of the CPython
3.10 reference implementation for most execution scopes, with some
adjustments to the behaviour at function scope to make it more
predictable and independent of the presence or absence of tracing
functions.

In addition, it proposes that the following functions be added to the
stable Python C API/ABI:

    typedef enum {
      PyLocals_UNDEFINED = -1,
      PyLocals_DIRECT_REFERENCE = 0,
      PyLocals_SHALLOW_COPY = 1,
      _PyLocals_ENSURE_32BIT_ENUM =<PHONE_NUMBER>
    } PyLocals_Kind;

    PyLocals_Kind PyLocals_GetKind();
    PyObject * PyLocals_Get();
    PyObject * PyLocals_GetCopy();

It also proposes the addition of several supporting functions and type
definitions to the CPython C API.

Motivation

While the precise semantics of the locals() builtin are nominally
undefined, in practice, many Python programs depend on it behaving
exactly as it behaves in CPython (at least when no tracing functions are
installed).

Other implementations such as PyPy are currently replicating that
behaviour, up to and including replication of local variable mutation
bugs that can arise when a trace hook is installed[1].

While this PEP considers CPython's current behaviour when no trace hooks
are installed to be largely acceptable, it considers the current
behaviour when trace hooks are installed to be problematic, as it causes
bugs like[2] without even reliably enabling the desired functionality of
allowing debuggers like pdb to mutate local variables[3].

Review of the initial PEP and the draft implementation then identified
an opportunity for simplification of both the documentation and
implementation of the function level locals() behaviour by updating it
to return an independent snapshot of the function locals and closure
variables on each call, rather than continuing to return the
semi-dynamic intermittently updated shared copy that it has historically
returned in CPython.

Specifically, the proposal in this PEP eliminates the historical
behaviour where adding a new local variable can change the behaviour of
code executed with exec() in function scopes, even if that code runs
before the local variable is defined.

For example:

    def f():
        exec("x = 1")
        print(locals().get("x"))
    f()

prints 1, but:

    def f():
        exec("x = 1")
        print(locals().get("x"))
        x = 0
    f()

prints None (the default value from the .get() call).

With this PEP both examples would print None, as the call to exec() and
the subsequent call to locals() would use independent dictionary
snapshots of the local variables rather than using the same shared
dictionary cached on the frame object.

Proposal

The expected semantics of the locals() builtin change based on the
current execution scope. For this purpose, the defined scopes of
execution are:

-   module scope: top-level module code, as well as any other code
    executed using exec() or eval() with a single namespace
-   class scope: code in the body of a class statement, as well as any
    other code executed using exec() or eval() with separate local and
    global namespaces
-   function scope: code in the body of a def or async def statement, or
    any other construct that creates an optimized code block in CPython
    (e.g. comprehensions, lambda functions)

This PEP proposes elevating most of the current behaviour of the CPython
reference implementation to become part of the language specification,
except that each call to locals() at function scope will create a new
dictionary object, rather than caching a common dict instance in the
frame object that each invocation will update and return.

This PEP also proposes to largely eliminate the concept of a separate
"tracing" mode from the CPython reference implementation. In releases up
to and including Python 3.10, the CPython interpreter behaves
differently when a trace hook has been registered in one or more threads
via an implementation dependent mechanism like sys.settrace ([4]) in
CPython's sys module or PyEval_SetTrace ([5]) in CPython's C API. If
this PEP is accepted, then the only remaining behavioural difference
when a trace hook is installed is that some optimisations in the
interpreter eval loop are disabled when the tracing logic needs to run
after each opcode.

This PEP proposes changes to CPython's behaviour at function scope that
make the locals() builtin semantics when a trace hook is registered
identical to those used when no trace hook is registered, while also
making the related frame API semantics clearer and easier for
interactive debuggers to rely on.

The proposed elimination of tracing mode affects the semantics of frame
object references obtained through other means, such as via a traceback,
or via the sys._getframe() API, as the write-through semantics needed
for trace hook support are always provided by the f_locals attribute on
frame objects, rather than being runtime state dependent.

New locals() documentation

The heart of this proposal is to revise the documentation for the
locals() builtin to read as follows:

  Return a mapping object representing the current local symbol table,
  with variable names as the keys, and their currently bound references
  as the values.

  At module scope, as well as when using exec() or eval() with a single
  namespace, this function returns the same namespace as globals().

  At class scope, it returns the namespace that will be passed to the
  metaclass constructor.

  When using exec() or eval() with separate local and global namespaces,
  it returns the local namespace passed in to the function call.

  In all of the above cases, each call to locals() in a given frame of
  execution will return the same mapping object. Changes made through
  the mapping object returned from locals() will be visible as bound,
  rebound, or deleted local variables, and binding, rebinding, or
  deleting local variables will immediately affect the contents of the
  returned mapping object.

  At function scope (including for generators and coroutines), each call
  to locals() instead returns a fresh dictionary containing the current
  bindings of the function's local variables and any nonlocal cell
  references. In this case, name binding changes made via the returned
  dict are not written back to the corresponding local variables or
  nonlocal cell references, and binding, rebinding, or deleting local
  variables and nonlocal cell references does not affect the contents of
  previously returned dictionaries.

There would also be a versionchanged note for the release making this
change:

  In prior versions, the semantics of mutating the mapping object
  returned from locals() were formally undefined. In CPython
  specifically, the mapping returned at function scope could be
  implicitly refreshed by other operations, such as calling locals()
  again, or the interpreter implicitly invoking a Python level trace
  function. Obtaining the legacy CPython behaviour now requires explicit
  calls to update the initially returned dictionary with the results of
  subsequent calls to locals().

For reference, the current documentation of this builtin reads as
follows:

  Update and return a dictionary representing the current local symbol
  table. Free variables are returned by locals() when it is called in
  function blocks, but not in class blocks.

  Note: The contents of this dictionary should not be modified; changes
  may not affect the values of local and free variables used by the
  interpreter.

(In other words: the status quo is that the semantics and behaviour of
locals() are formally implementation defined, whereas the proposed state
after this PEP is that the only implementation defined behaviour will be
that associated with whether or not the implementation emulates the
CPython frame API, with the behaviour in all other cases being defined
by the language and library references)

Module scope

At module scope, as well as when using exec() or eval() with a single
namespace, locals() must return the same object as globals(), which must
be the actual execution namespace (available as
inspect.currentframe().f_locals in implementations that provide access
to frame objects).

Variable assignments during subsequent code execution in the same scope
must dynamically change the contents of the returned mapping, and
changes to the returned mapping must change the values bound to local
variable names in the execution environment.

To capture this expectation as part of the language specification, the
following paragraph will be added to the documentation for locals():

  At module scope, as well as when using exec() or eval() with a single
  namespace, this function returns the same namespace as globals().

This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.

Class scope

At class scope, as well as when using exec() or eval() with separate
global and local namespaces, locals() must return the specified local
namespace (which may be supplied by the metaclass __prepare__ method in
the case of classes). As for module scope, this must be a direct
reference to the actual execution namespace (available as
inspect.currentframe().f_locals in implementations that provide access
to frame objects).

Variable assignments during subsequent code execution in the same scope
must change the contents of the returned mapping, and changes to the
returned mapping must change the values bound to local variable names in
the execution environment.

The mapping returned by locals() will not be used as the actual class
namespace underlying the defined class (the class creation process will
copy the contents to a fresh dictionary that is only accessible by going
through the class machinery).

For nested classes defined inside a function, any nonlocal cells
referenced from the class scope are not included in the locals()
mapping.

To capture this expectation as part of the language specification, the
following two paragraphs will be added to the documentation for
locals():

  When using exec() or eval() with separate local and global namespaces,
  [this function] returns the given local namespace.

  At class scope, it returns the namespace that will be passed to the
  metaclass constructor.

This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.

Function scope

At function scope, interpreter implementations are granted significant
freedom to optimise local variable access, and hence are NOT required to
permit arbitrary modification of local and nonlocal variable bindings
through the mapping returned from locals().

Historically, this leniency has been described in the language
specification with the words "The contents of this dictionary should not
be modified; changes may not affect the values of local and free
variables used by the interpreter."

This PEP proposes to change that text to instead say:

  At function scope (including for generators and coroutines), each call
  to locals() instead returns a fresh dictionary containing the current
  bindings of the function's local variables and any nonlocal cell
  references. In this case, name binding changes made via the returned
  dict are not written back to the corresponding local variables or
  nonlocal cell references, and binding, rebinding, or deleting local
  variables and nonlocal cell references does not affect the contents of
  previously returned dictionaries.

This part of the proposal does require changes to the CPython reference
implementation, as CPython currently returns a shared mapping object
that may be implicitly refreshed by additional calls to locals(), and
the "write back" strategy currently used to support namespace changes
from trace functions also doesn't comply with it (and causes the quirky
behavioural problems mentioned in the Motivation above).

CPython Implementation Changes

Summary of proposed implementation-specific changes

-   Changes are made as necessary to provide the updated Python level
    semantics
-   Two new functions are added to the stable ABI to replicate the
    updated behaviour of the Python locals() builtin:

    PyObject * PyLocals_Get();
    PyLocals_Kind PyLocals_GetKind();

-   One new function is added to the stable ABI to efficiently get a
    snapshot of the local namespace in the running frame:

    PyObject * PyLocals_GetCopy();

-   Corresponding frame accessor functions for these new public APIs are
    added to the CPython frame C API
-   On optimised frames, the Python level f_locals API will return
    dynamically created read/write proxy objects that directly access
    the frame's local and closure variable storage. To provide
    interoperability with the existing PyEval_GetLocals() API, the proxy
    objects will continue to use the C level frame locals data storage
    field to hold a value cache that also allows for storage of
    arbitrary additional keys. Additional details on the expected
    behaviour of these fast locals proxy objects are covered below.
-   No C API function is added to get access to a mutable mapping for
    the local namespace. Instead,
    PyObject_GetAttrString(frame, "f_locals") is used, the same API as
    is used in Python code.
-   PyEval_GetLocals() remains supported and does not emit a
    programmatic warning, but will be deprecated in the documentation in
    favour of the new APIs that don't rely on returning a borrowed
    reference
-   PyFrame_FastToLocals() and PyFrame_FastToLocalsWithError() remain
    supported and do not emit a programmatic warning, but will be
    deprecated in the documentation in favour of the new APIs that don't
    require direct access to the internal data storage layout of frame
    objects
-   PyFrame_LocalsToFast() always raises RuntimeError(), indicating that
    PyObject_GetAttrString(frame, "f_locals") should be used to obtain a
    mutable read/write mapping for the local variables.
-   The trace hook implementation will no longer call
    PyFrame_FastToLocals() implicitly. The version porting guide will
    recommend migrating to PyFrame_GetLocals() for read-only access and
    PyObject_GetAttrString(frame, "f_locals") for read/write access.

Providing the updated Python level semantics

The implementation of the locals() builtin is modified to return a
distinct copy of the local namespace for optimised frames, rather than a
direct reference to the internal frame value cache updated by the
PyFrame_FastToLocals() C API and returned by the PyEval_GetLocals() C
API.

Resolving the issues with tracing mode behaviour

The current cause of CPython's tracing mode quirks (both the side
effects from simply installing a tracing function and the fact that
writing values back to function locals only works for the specific
function being traced) is the way that locals mutation support for trace
hooks is currently implemented: the PyFrame_LocalsToFast function.

When a trace function is installed, CPython currently does the following
for function frames (those where the code object uses "fast locals"
semantics):

1.  Calls PyFrame_FastToLocals to update the frame value cache
2.  Calls the trace hook (with tracing of the hook itself disabled)
3.  Calls PyFrame_LocalsToFast to capture any changes made to the frame
    value cache

This approach is problematic for a few different reasons:

-   Even if the trace function doesn't mutate the value cache, the final
    step resets any cell references back to the state they were in
    before the trace function was called (this is the root cause of the
    bug report in[6])
-   If the trace function does mutate the value cache, but then does
    something that causes the value cache to be refreshed from the
    frame, those changes are lost (this is one aspect of the bug report
    in[7])
-   If the trace function attempts to mutate the local variables of a
    frame other than the one being traced (e.g. frame.f_back.f_locals),
    those changes will almost certainly be lost (this is another aspect
    of the bug report in [8])
-   If a reference to the frame value cache (e.g. retrieved via
    locals()) is passed to another function, and that function mutates
    the value cache, then those changes may be written back to the
    execution frame if a trace hook is installed

The proposed resolution to this problem is to take advantage of the fact
that whereas functions typically access their own namespace using the
language defined locals() builtin, trace functions necessarily use the
implementation dependent frame.f_locals interface, as a frame reference
is what gets passed to hook implementations.

Instead of being a direct reference to the internal frame value cache
historically returned by the locals() builtin, the Python level
frame.f_locals will be updated to instead return instances of a
dedicated fast locals proxy type that writes and reads values directly
to and from the fast locals array on the underlying frame. Each access
of the attribute produces a new instance of the proxy (so creating proxy
instances is intentionally a cheap operation).

Despite the new proxy type becoming the preferred way to access local
variables on optimised frames, the internal value cache stored on the
frame is still retained for two key purposes:

-   maintaining backwards compatibility for and interoperability with
    the PyEval_GetLocals() C API
-   providing storage space for additional keys that don't have slots in
    the fast locals array (e.g. the __return__ and __exception__ keys
    set by pdb when tracing code execution for debugging purposes)

With the changes in this PEP, this internal frame value cache is no
longer directly accessible from Python code (whereas historically it was
both returned by the locals() builtin and available as the
frame.f_locals attribute). Instead, the value cache is only accessible
via the PyEval_GetLocals() C API and by directly accessing the internal
storage of a frame object.

Fast locals proxy objects and the internal frame value cache returned by
PyEval_GetLocals() offer the following behavioural guarantees:

-   changes made via a fast locals proxy will be immediately visible to
    the frame itself, to other fast locals proxy objects for the same
    frame, and in the internal value cache stored on the frame (it is
    this last point that provides PyEval_GetLocals() interoperability)
-   changes made directly to the internal frame value cache will never
    be visible to the frame itself, and will only be reliably visible
    via fast locals proxies for the same frame if the change relates to
    extra variables that don't have slots in the frame's fast locals
    array
-   changes made by executing code in the frame will be immediately
    visible to all fast locals proxy objects for that frame (both
    existing proxies and newly created ones). Visibility in the internal
    frame value cache cache returned by PyEval_GetLocals() is subject to
    the cache update guidelines discussed in the next section

As a result of these points, only code using PyEval_GetLocals(),
PyLocals_Get(), or PyLocals_GetCopy() will need to be concerned about
the frame value cache potentially becoming stale. Code using the new
frame fast locals proxy API (whether from Python or from C) will always
see the live state of the frame.

Fast locals proxy implementation details

Each fast locals proxy instance has a single internal attribute that is
not exposed as part of the Python runtime API:

-   frame: the underlying optimised frame that the proxy provides access
    to

In addition, proxy instances use and update the following attributes
stored on the underlying frame or code object:

-   _name_to_offset_mapping: a hidden mapping from variable names to
    fast local storage offsets. This mapping is lazily initialized on
    the first frame read or write access through a fast locals proxy,
    rather than being eagerly populated as soon as the first fast locals
    proxy is created. Since the mapping is identical for all frames
    running a given code object, a single copy is stored on the code
    object, rather than each frame object populating its own mapping
-   locals: the internal frame value cache returned by the
    PyEval_GetLocals() C API and updated by the PyFrame_FastToLocals() C
    API. This is the mapping that the locals() builtin returns in Python
    3.10 and earlier.

__getitem__ operations on the proxy will populate the
_name_to_offset_mapping on the code object (if it is not already
populated), and then either return the relevant value (if the key is
found in either the _name_to_offset_mapping mapping or the internal
frame value cache), or else raise KeyError. Variables that are defined
on the frame but not currently bound also raise KeyError (just as
they're omitted from the result of locals()).

As the frame storage is always accessed directly, the proxy will
automatically pick up name binding and unbinding operations that take
place as the function executes. The internal value cache is implicitly
updated when individual variables are read from the frame state
(including for containment checks, which need to check if the name is
currently bound or unbound).

Similarly, __setitem__ and __delitem__ operations on the proxy will
directly affect the corresponding fast local or cell reference on the
underlying frame, ensuring that changes are immediately visible to the
running Python code, rather than needing to be written back to the
runtime storage at some later time. Such changes are also immediately
written to the internal frame value cache to make them visible to users
of the PyEval_GetLocals() C API.

Keys that are not defined as local or closure variables on the
underlying frame are still written to the internal value cache on
optimised frames. This allows utilities like pdb (which writes
__return__ and __exception__ values into the frame's f_locals mapping)
to continue working as they always have. These additional keys that do
not correspond to a local or closure variable on the frame will be left
alone by future cache sync operations. Using the frame value cache to
store these extra keys (rather than defining a new mapping that holds
only the extra keys) provides full interoperability with the existing
PyEval_GetLocals() API (since users of either API will see extra keys
added by users of either API, rather than users of the new fast locals
proxy API only seeing keys added via that API).

An additional benefit of storing only the variable value cache on the
frame (rather than storing an instance of the proxy type), is that it
avoids creating a reference cycle from the frame back to itself, so the
frame will only be kept alive if another object retains a reference to a
proxy instance.

Note: calling the proxy.clear() method has a similarly broad impact as
calling PyFrame_LocalsToFast() on an empty frame value cache in earlier
versions. Not only will the frame local variables be cleared, but also
any cell variables accessible from the frame (whether those cells are
owned by the frame itself or by an outer frame). This can clear a
class's __class__ cell if called on the frame of a method that uses the
zero-arg super() construct (or otherwise references __class__). This
exceeds the scope of calling frame.clear(), as that only drop's the
frame's references to cell variables, it doesn't clear the cells
themselves. This PEP could be a potential opportunity to narrow the
scope of attempts to clear the frame variables directly by leaving cells
belonging to outer frames alone, and only clearing local variables and
cells belonging directly to the frame underlying the proxy (this issue
affects PEP 667 as well, as the question relates to the handling of cell
variables, and is entirely independent of the internal frame value
cache).

Changes to the stable C API/ABI

Unlike Python code, extension module functions that call in to the
Python C API can be called from any kind of Python scope. This means it
isn't obvious from the context whether locals() will return a snapshot
or not, as it depends on the scope of the calling Python code, not the C
code itself.

This means it is desirable to offer C APIs that give predictable, scope
independent, behaviour. However, it is also desirable to allow C code to
exactly mimic the behaviour of Python code at the same scope.

To enable mimicking the behaviour of Python code, the stable C ABI would
gain the following new functions:

    PyObject * PyLocals_Get();
    PyLocals_Kind PyLocals_GetKind();

PyLocals_Get() is directly equivalent to the Python locals() builtin. It
returns a new reference to the local namespace mapping for the active
Python frame at module and class scope, and when using exec() or eval().
It returns a shallow copy of the active namespace at
function/coroutine/generator scope.

PyLocals_GetKind() returns a value from the newly defined PyLocals_Kind
enum, with the following options being available:

-   PyLocals_DIRECT_REFERENCE: PyLocals_Get() returns a direct reference
    to the local namespace for the running frame.
-   PyLocals_SHALLOW_COPY: PyLocals_Get() returns a shallow copy of the
    local namespace for the running frame.
-   PyLocals_UNDEFINED: an error occurred (e.g. no active Python thread
    state). A Python exception will be set if this value is returned.

Since the enum is used in the stable ABI, an additional 31-bit value is
set to ensure that it is safe to cast arbitrary signed 32-bit signed
integers to PyLocals_Kind values.

This query API allows extension module code to determine the potential
impact of mutating the mapping returned by PyLocals_Get() without
needing access to the details of the running frame object. Python code
gets equivalent information visually through lexical scoping (as covered
in the new locals() builtin documentation).

To allow extension module code to behave consistently regardless of the
active Python scope, the stable C ABI would gain the following new
function:

    PyObject * PyLocals_GetCopy();

PyLocals_GetCopy() returns a new dict instance populated from the
current locals namespace. Roughly equivalent to dict(locals()) in Python
code, but avoids the double-copy in the case where locals() already
returns a shallow copy. Akin to the following code, but doesn't assume
there will only ever be two kinds of locals result:

    locals = PyLocals_Get();
    if (PyLocals_GetKind() == PyLocals_DIRECT_REFERENCE) {
        locals = PyDict_Copy(locals);
    }

The existing PyEval_GetLocals() API will retain its existing behaviour
in CPython (mutable locals at class and module scope, shared dynamic
snapshot otherwise). However, its documentation will be updated to note
that the conditions under which the shared dynamic snapshot get updated
have changed.

The PyEval_GetLocals() documentation will also be updated to recommend
replacing usage of this API with whichever of the new APIs is most
appropriate for the use case:

-   Use PyLocals_Get() (optionally combined with PyDictProxy_New()) for
    read-only access to the current locals namespace. This form of usage
    will need to be aware that the copy may go stale in optimised
    frames.
-   Use PyLocals_GetCopy() for a regular mutable dict that contains a
    copy of the current locals namespace, but has no ongoing connection
    to the active frame.
-   Use PyLocals_Get() to exactly match the semantics of the Python
    level locals() builtin.
-   Query PyLocals_GetKind() explicitly to implement custom handling
    (e.g. raising a meaningful exception) for scopes where
    PyLocals_Get() would return a shallow copy rather than granting
    read/write access to the locals namespace.
-   Use implementation specific APIs (e.g.
    PyObject_GetAttrString(frame, "f_locals")) if read/write access to
    the frame is required and PyLocals_GetKind() returns something other
    than PyLocals_DIRECT_REFERENCE.

Changes to the public CPython C API

The existing PyEval_GetLocals() API returns a borrowed reference, which
means it cannot be updated to return the new shallow copies at function
scope. Instead, it will continue to return a borrowed reference to an
internal dynamic snapshot stored on the frame object. This shared
mapping will behave similarly to the existing shared mapping in Python
3.10 and earlier, but the exact conditions under which it gets refreshed
will be different. Specifically, it will be updated only in the
following circumstance:

-   any call to PyEval_GetLocals(), PyLocals_Get(), PyLocals_GetCopy(),
    or the Python locals() builtin while the frame is running
-   any call to PyFrame_GetLocals(), PyFrame_GetLocalsCopy(),
    _PyFrame_BorrowLocals(), PyFrame_FastToLocals(), or
    PyFrame_FastToLocalsWithError() for the frame
-   any operation on a fast locals proxy object that updates the shared
    mapping as part of its implementation. In the initial reference
    implementation, those operations are those that are intrinsically
    O(n) operations (len(flp), mapping comparison, flp.copy() and
    rendering as a string), as well as those that refresh the cache
    entries for individual keys.

Requesting a fast locals proxy will not implicitly update the shared
dynamic snapshot, and the CPython trace hook handling will no longer
implicitly update it either.

(Note: even though PyEval_GetLocals() is part of the stable C API/ABI,
the specifics of when the namespace it returns gets refreshed are still
an interpreter implementation detail)

The additions to the public CPython C API are the frame level
enhancements needed to support the stable C API/ABI updates:

    PyLocals_Kind PyFrame_GetLocalsKind(frame);
    PyObject * PyFrame_GetLocals(frame);
    PyObject * PyFrame_GetLocalsCopy(frame);
    PyObject * _PyFrame_BorrowLocals(frame);

PyFrame_GetLocalsKind(frame) is the underlying API for
PyLocals_GetKind().

PyFrame_GetLocals(frame) is the underlying API for PyLocals_Get().

PyFrame_GetLocalsCopy(frame) is the underlying API for
PyLocals_GetCopy().

_PyFrame_BorrowLocals(frame) is the underlying API for
PyEval_GetLocals(). The underscore prefix is intended to discourage use
and to indicate that code using it is unlikely to be portable across
implementations. However, it is documented and visible to the linker in
order to avoid having to access the internals of the frame struct from
the PyEval_GetLocals() implementation.

The PyFrame_LocalsToFast() function will be changed to always emit
RuntimeError, explaining that it is no longer a supported operation, and
affected code should be updated to use
PyObject_GetAttrString(frame, "f_locals") to obtain a read/write proxy
instead.

In addition to the above documented interfaces, the draft reference
implementation also exposes the following undocumented interfaces:

    PyTypeObject _PyFastLocalsProxy_Type;
    #define _PyFastLocalsProxy_CheckExact(self) Py_IS_TYPE(op, &_PyFastLocalsProxy_Type)

This type is what the reference implementation actually returns from
PyObject_GetAttrString(frame, "f_locals") for optimized frames (i.e.
when PyFrame_GetLocalsKind() returns PyLocals_SHALLOW_COPY).

Reducing the runtime overhead of trace hooks

As noted in[9], the implicit call to PyFrame_FastToLocals() in the
Python trace hook support isn't free, and could be rendered unnecessary
if the frame proxy read values directly from the frame instead of
getting them from the mapping.

As the new frame locals proxy type doesn't require separate data refresh
steps, this PEP incorporates Victor Stinner's proposal to no longer
implicitly call PyFrame_FastToLocalsWithError() before calling trace
hooks implemented in Python.

Code using the new fast locals proxy objects will have the dynamic
locals snapshot implicitly refreshed when accessing methods that need
it, while code using the PyEval_GetLocals() API will implicitly refresh
it when making that call.

The PEP necessarily also drops the implicit call to
PyFrame_LocalsToFast() when returning from a trace hook, as that API now
always raises an exception.

Rationale and Design Discussion

Changing locals() to return independent snapshots at function scope

The locals() builtin is a required part of the language, and in the
reference implementation it has historically returned a mutable mapping
with the following characteristics:

-   each call to locals() returns the same mapping object
-   for namespaces where locals() returns a reference to something other
    than the actual local execution namespace, each call to locals()
    updates the mapping object with the current state of the local
    variables and any referenced nonlocal cells
-   changes to the returned mapping usually aren't written back to the
    local variable bindings or the nonlocal cell references, but write
    backs can be triggered by doing one of the following:
    -   installing a Python level trace hook (write backs then happen
        whenever the trace hook is called)
    -   running a function level wildcard import (requires bytecode
        injection in Py3)
    -   running an exec statement in the function's scope (Py2 only,
        since exec became an ordinary builtin in Python 3)

Originally this PEP proposed to retain the first two of these
properties, while changing the third in order to address the outright
behaviour bugs that it can cause.

In[10] Nathaniel Smith made a persuasive case that we could make the
behaviour of locals() at function scope substantially less confusing by
retaining only the second property and having each call to locals() at
function scope return an independent snapshot of the local variables and
closure references rather than updating an implicitly shared snapshot.

As this revised design also made the implementation markedly easier to
follow, the PEP was updated to propose this change in behaviour, rather
than retaining the historical shared snapshot.

Keeping locals() as a snapshot at function scope

As discussed in[11], it would theoretically be possible to change the
semantics of the locals() builtin to return the write-through proxy at
function scope, rather than switching it to return independent
snapshots.

This PEP doesn't (and won't) propose this as it's a backwards
incompatible change in practice, even though code that relies on the
current behaviour is technically operating in an undefined area of the
language specification.

Consider the following code snippet:

    def example():
        x = 1
        locals()["x"] = 2
        print(x)

Even with a trace hook installed, that function will consistently print
1 on the current reference interpreter implementation:

    >>> example()
    1
    >>> import sys
    >>> def basic_hook(*args):
    ...     return basic_hook
    ...
    >>> sys.settrace(basic_hook)
    >>> example()
    1

Similarly, locals() can be passed to the exec() and eval() builtins at
function scope (either explicitly or implicitly) without risking
unexpected rebinding of local variables or closure references.

Provoking the reference interpreter into incorrectly mutating the local
variable state requires a more complex setup where a nested function
closes over a variable being rebound in the outer function, and due to
the use of either threads, generators, or coroutines, it's possible for
a trace function to start running for the nested function before the
rebinding operation in the outer function, but finish running after the
rebinding operation has taken place (in which case the rebinding will be
reverted, which is the bug reported in[12]).

In addition to preserving the de facto semantics which have been in
place since PEP 227 introduced nested scopes in Python 2.1, the other
benefit of restricting the write-through proxy support to the
implementation-defined frame object API is that it means that only
interpreter implementations which emulate the full frame API need to
offer the write-through capability at all, and that JIT-compiled
implementations only need to enable it when a frame introspection API is
invoked, or a trace hook is installed, not whenever locals() is accessed
at function scope.

Returning snapshots from locals() at function scope also means that
static analysis for function level code will be more reliable, as only
access to the frame machinery will allow rebinding of local and nonlocal
variable references in a way that is hidden from static analysis.

What happens with the default args for eval() and exec()?

These are formally defined as inheriting globals() and locals() from the
calling scope by default.

There isn't any need for the PEP to change these defaults, so it
doesn't, and exec() and eval() will start running in a shallow copy of
the local namespace when that is what locals() returns.

This behaviour will have potential performance implications, especially
for functions with large numbers of local variables (e.g. if these
functions are called in a loop, calling globals() and locals() once
before the loop and then passing the namespace into the function
explicitly will give the same semantics and performance characteristics
as the status quo, whereas relying on the implicit default would create
a new shallow copy of the local namespace on each iteration).

(Note: the reference implementation draft PR has updated the locals()
and vars(), eval(), and exec() builtins to use PyLocals_Get(). The dir()
builtin still uses PyEval_GetLocals(), since it's only using it to make
a list from the keys).

Additional considerations for eval() and exec() in optimized scopes

Note: while implementing PEP 667, it was noted that neither that PEP nor
this one clearly explained the impact the locals() changes would have on
code execution APIs like exec() and eval(). This section was added to
this PEP's rationale to better describe the impact and explain the
intended benefits of the change.

When exec() was converted from a statement to a builtin function in
Python 3.0 (part of the core language changes in PEP 3100), the
associated implicit call to PyFrame_LocalsToFast() was removed, so it
typically appears as if attempts to write to local variables with exec()
in optimized frames are ignored:

    >>> def f():
    ...     x = 0
    ...     exec("x = 1")
    ...     print(x)
    ...     print(locals()["x"])
    ...
    >>> f()
    0
    0

In truth, the writes aren't being ignored, they just aren't being copied
from the dictionary cache back to the optimized local variable array.
The changes to the dictionary are then overwritten the next time the
dictionary cache is refreshed from the array:

    >>> def f():
    ...     x = 0
    ...     locals_cache = locals()
    ...     exec("x = 1")
    ...     print(x)
    ...     print(locals_cache["x"])
    ...     print(locals()["x"])
    ...
    >>> f()
    0
    1
    0

The behaviour becomes even stranger if a tracing function or another
piece of code invokes PyFrame_LocalsToFast() before the cache is next
refreshed. In those cases the change is written back to the optimized
local variable array:

    >>> from sys import _getframe
    >>> from ctypes import pythonapi, py_object, c_int
    >>> _locals_to_fast = pythonapi.PyFrame_LocalsToFast
    >>> _locals_to_fast.argtypes = [py_object, c_int]
    >>> def f():
    ...     _frame = _getframe()
    ...     _f_locals = _frame.f_locals
    ...     x = 0
    ...     exec("x = 1")
    ...     _locals_to_fast(_frame, 0)
    ...     print(x)
    ...     print(locals()["x"])
    ...     print(_f_locals["x"])
    ...
    >>> f()
    1
    1
    1

This situation was more common in Python 3.10 and earlier versions, as
merely installing a tracing function was enough to trigger implicit
calls to PyFrame_LocalsToFast() after every line of Python code.
However, it can still happen in Python 3.11+ depending on exactly which
tracing functions are active (e.g. interactive debuggers intentionally
do this so that changes made at the debugging prompt are visible when
code execution resumes).

All of the above comments in relation to exec() apply to any attempt to
mutate the result of locals() in optimized scopes, and are the main
reason that the locals() builtin docs contain this caveat:

  Note: The contents of this dictionary should not be modified; changes
  may not affect the values of local and free variables used by the
  interpreter.

While the exact wording in the library reference is not entirely
explicit, both exec() and eval() have long used the results of calling
globals() and locals() in the calling Python frame as their default
execution namespace.

This was historically also equivalent to using the calling frame's
frame.f_globals and frame.f_locals attributes, but this PEP maps the
default namespace arguments for exec() and eval() to globals() and
locals() in the calling frame in order to preserve the property of
defaulting to ignoring attempted writes to the local namespace in
optimized scopes.

This poses a potential compatibility issue for some code, as with the
previous implementation that returns the same dict when locals() is
called multiple times in function scope, the following code usually
worked due to the implicitly shared local variable namespace:

    def f():
        exec('a = 0')  # equivalent to exec('a = 0', globals(), locals())
        exec('print(a)')  # equivalent to exec('print(a)', globals(), locals())
        print(locals())  # {'a': 0}
        # However, print(a) will not work here
    f()

With locals() in an optimised scope returning the same shared dict for
each call, it was possible to store extra "fake locals" in that dict.
While these aren't real locals known by the compiler (so they can't be
printed with code like print(a)), they can still be accessed via
locals() and shared between multiple exec() calls in the same function
scope. Furthermore, because they're not real locals, they don't get
implicitly updated or removed when the shared cache is refreshed from
the local variable storage array.

When the code in exec() tries to write to an existing local variable,
the runtime behaviour gets harder to predict:

    def f():
        a = None
        exec('a = 0')  # equivalent to exec('a = 0', globals(), locals())
        exec('print(a)')  # equivalent to exec('print(a)', globals(), locals())
        print(locals())  # {'a': None}
    f()

print(a) will print None because the implicit locals() call in exec()
refreshes the cached dict with the actual values on the frame. This
means that, unlike the "fake" locals created by writing back to locals()
(including via previous calls to exec()), the real locals known by the
compiler can't easily be modified by exec() (it can be done, but it
requires both retrieving the frame.f_locals attribute to enable writes
back to the frame, and then invoking PyFrame_LocalsToFast(), as
shown <pep-558-ctypes-example> using ctypes above).

As noted in the pep-558-motivation section, this confusing side effect
happens even if the local variable is only defined after the exec()
calls:

    >>> def f():
    ...     exec("a = 0")
    ...     exec("print('a' in locals())") # Printing 'a' directly won't work
    ...     print(locals())
    ...     a = None
    ...     print(locals())
    ...
    >>> f()
    False
    {}
    {'a': None}

Because a is a real local variable that is not currently bound to a
value, it gets explicitly removed from the dictionary returned by
locals() whenever locals() is called prior to the a = None line. This
removal is intentional, as it allows the contents of locals() to be
updated correctly in optimized scopes when del statements are used to
delete previously bound local variables.

As noted in the ctypes example <pep-558-ctypes-example>, the above
behavioural description may be invalidated if the CPython
PyFrame_LocalsToFast() API gets invoked while the frame is still
running. In that case, the changes to a might become visible to the
running code, depending on exactly when that API is called (and whether
the frame has been primed for locals modification by accessing the
frame.f_locals attribute).

As described above, two options were considered to replace this
confusing behaviour:

-   make locals() return write-through proxy instances (similar to
    frame.f_locals)
-   make locals() return genuinely independent snapshots so that
    attempts to change the values of local variables via exec() would be
    consistently ignored without any of the caveats noted above.

The PEP chooses the second option for the following reasons:

-   returning independent snapshots in optimized scopes preserves the
    Python 3.0 change to exec() that resulted in attempts to mutate
    local variables via exec() being ignored in most cases
-   the distinction between "locals() gives an instantaneous snapshot of
    the local variables in optimized scopes, and read/write access in
    other scopes" and "frame.f_locals gives read/write access to the
    local variables in all scopes, including optimized scopes" allows
    the intent of a piece of code to be clearer than it would be if both
    APIs granted full read/write access in optimized scopes, even when
    write access wasn't needed or desired
-   in addition to improving clarity for human readers, ensuring that
    name rebinding in optimized scopes remains lexically visible in the
    code (as long as the frame introspection APIs are not accessed)
    allows compilers and interpreters to apply related performance
    optimizations more consistently
-   only Python implementations that support the optional frame
    introspection APIs will need to provide the new write-through proxy
    support for optimized frames

With the semantic changes to locals() in this PEP, it becomes much
easier to explain the behavior of exec() and eval(): in optimized
scopes, they will never implicitly affect local variables; in other
scopes, they will always implicitly affect local variables. In optimized
scopes, any implicit assignment to the local variables will be discarded
when the code execution API returns, since a fresh copy of the local
variables is used on each invocation.

Retaining the internal frame value cache

Retaining the internal frame value cache results in some visible quirks
when frame proxy instances are kept around and re-used after name
binding and unbinding operations have been executed on the frame.

The primary reason for retaining the frame value cache is to maintain
backwards compatibility with the PyEval_GetLocals() API. That API
returns a borrowed reference, so it must refer to persistent state
stored on the frame object. Storing a fast locals proxy object on the
frame creates a problematic reference cycle, so the cleanest option is
to instead continue to return a frame value cache, just as this function
has done since optimised frames were first introduced.

With the frame value cache being kept around anyway, it then further
made sense to rely on it to simplify the fast locals proxy mapping
implementation.

Note: the fact PEP 667 doesn't use the internal frame value cache as
part of the write-through proxy implementation is the key Python level
difference between the two PEPs.

Changing the frame API semantics in regular operation

Note: when this PEP was first written, it predated the Python 3.11
change to drop the implicit writeback of the frame local variables
whenever a tracing function was installed, so making that change was
included as part of the proposal.

Earlier versions of this PEP proposed having the semantics of the frame
f_locals attribute depend on whether or not a tracing hook was currently
installed - only providing the write-through proxy behaviour when a
tracing hook was active, and otherwise behaving the same as the
historical locals() builtin.

That was adopted as the original design proposal for a couple of key
reasons, one pragmatic and one more philosophical:

-   Object allocations and method wrappers aren't free, and tracing
    functions aren't the only operations that access frame locals from
    outside the function. Restricting the changes to tracing mode meant
    that the additional memory and execution time overhead of these
    changes would be as close to zero in regular operation as we can
    possibly make them.
-   "Don't change what isn't broken": the current tracing mode problems
    are caused by a requirement that's specific to tracing mode (support
    for external rebinding of function local variable references), so it
    made sense to also restrict any related fixes to tracing mode

However, actually attempting to implement and document that dynamic
approach highlighted the fact that it makes for a really subtle runtime
state dependent behaviour distinction in how frame.f_locals works, and
creates several new edge cases around how f_locals behaves as trace
functions are added and removed.

Accordingly, the design was switched to the current one, where
frame.f_locals is always a write-through proxy, and locals() is always a
snapshot, which is both simpler to implement and easier to explain.

Regardless of how the CPython reference implementation chooses to handle
this, optimising compilers and interpreters also remain free to impose
additional restrictions on debuggers, such as making local variable
mutation through frame objects an opt-in behaviour that may disable some
optimisations (just as the emulation of CPython's frame API is already
an opt-in flag in some Python implementations).

Continuing to support storing additional data on optimised frames

One of the draft iterations of this PEP proposed removing the ability to
store additional data on optimised frames by writing to frame.f_locals
keys that didn't correspond to local or closure variable names on the
underlying frame.

While this idea offered some attractive simplification of the fast
locals proxy implementation, pdb stores __return__ and __exception__
values on arbitrary frames, so the standard library test suite fails if
that functionality no longer works.

Accordingly, the ability to store arbitrary keys was retained, at the
expense of certain operations on proxy objects being slower than could
otherwise be (since they can't assume that only names defined on the
code object will be accessible through the proxy).

It is expected that the exact details of the interaction between the
fast locals proxy and the f_locals value cache on the underlying frame
will evolve over time as opportunities for improvement are identified.

Historical semantics at function scope

The current semantics of mutating locals() and frame.f_locals in CPython
are rather quirky due to historical implementation details:

-   actual execution uses the fast locals array for local variable
    bindings and cell references for nonlocal variables
-   there's a PyFrame_FastToLocals operation that populates the frame's
    f_locals attribute based on the current state of the fast locals
    array and any referenced cells. This exists for three reasons:
    -   allowing trace functions to read the state of local variables
    -   allowing traceback processors to read the state of local
        variables
    -   allowing locals() to read the state of local variables
-   a direct reference to frame.f_locals is returned from locals(), so
    if you hand out multiple concurrent references, then all those
    references will be to the exact same dictionary
-   the two common calls to the reverse operation, PyFrame_LocalsToFast,
    were removed in the migration to Python 3: exec is no longer a
    statement (and hence can no longer affect function local
    namespaces), and the compiler now disallows the use of
    from module import * operations at function scope
-   however, two obscure calling paths remain: PyFrame_LocalsToFast is
    called as part of returning from a trace function (which allows
    debuggers to make changes to the local variable state), and you can
    also still inject the IMPORT_STAR opcode when creating a function
    directly from a code object rather than via the compiler

This proposal deliberately doesn't formalise these semantics as is,
since they only make sense in terms of the historical evolution of the
language and the reference implementation, rather than being
deliberately designed.

Proposing several additions to the stable C API/ABI

Historically, the CPython C API (and subsequently, the stable ABI) has
exposed only a single API function related to the Python locals builtin:
PyEval_GetLocals(). However, as it returns a borrowed reference, it is
not possible to adapt that interface directly to supporting the new
locals() semantics proposed in this PEP.

An earlier iteration of this PEP proposed a minimalist adaptation to the
new semantics: one C API function that behaved like the Python locals()
builtin, and another that behaved like the frame.f_locals descriptor
(creating and returning the write-through proxy if necessary).

The feedback[13] on that version of the C API was that it was too
heavily based on how the Python level semantics were implemented, and
didn't account for the behaviours that authors of C extensions were
likely to need.

The broader API now being proposed came from grouping the potential
reasons for wanting to access the Python locals() namespace from an
extension module into the following cases:

-   needing to exactly replicate the semantics of the Python level
    locals() operation. This is the PyLocals_Get() API.
-   needing to behave differently depending on whether writes to the
    result of PyLocals_Get() will be visible to Python code or not. This
    is handled by the PyLocals_GetKind() query API.
-   always wanting a mutable namespace that has been pre-populated from
    the current Python locals() namespace, but not wanting any changes
    to be visible to Python code. This is the PyLocals_GetCopy() API.
-   always wanting a read-only view of the current locals namespace,
    without incurring the runtime overhead of making a full copy each
    time. This isn't readily offered for optimised frames due to the
    need to check whether names are currently bound or not, so no
    specific API is being added to cover it.

Historically, these kinds of checks and operations would only have been
possible if a Python implementation emulated the full CPython frame API.
With the proposed API, extension modules can instead ask more clearly
for the semantics that they actually need, giving Python implementations
more flexibility in how they provide those capabilities.

Comparison with PEP 667

NOTE: the comparison below is against PEP 667 as it was in December
2021. It does not reflect the state of PEP 667 as of April 2024 (when
this PEP was withdrawn in favour of proceeding with PEP 667).

PEP 667 offers a partially competing proposal for this PEP that suggests
it would be reasonable to eliminate the internal frame value cache on
optimised frames entirely.

These changes were originally offered as amendments to PEP 558, and the
PEP author rejected them for three main reasons:

-   the initial claim that PyEval_GetLocals() was unfixable because it
    returns a borrowed reference was simply false, as it is still
    working in the PEP 558 reference implementation. All that is
    required to keep it working is to retain the internal frame value
    cache and design the fast locals proxy in such a way that it is
    reasonably straightforward to keep the cache up to date with changes
    in the frame state without incurring significant runtime overhead
    when the cache isn't needed. Given that this claim is false, the
    proposal to require that all code using the PyEval_GetLocals() API
    be rewritten to use a new API with different refcounting semantics
    fails PEP 387's requirement that API compatibility breaks should
    have a large benefit to breakage ratio (since there's no significant
    benefit gained from dropping the cache, no code breakage can be
    justified). The only genuinely unfixable public API is
    PyFrame_LocalsToFast() (which is why both PEPs propose breaking
    that).
-   without some form of internal value cache, the API performance
    characteristics of the fast locals proxy mapping become quite
    unintuitive. len(proxy), for example, becomes consistently O(n) in
    the number of variables defined on the frame, as the proxy has to
    iterate over the entire fast locals array to see which names are
    currently bound to values before it can determine the answer. By
    contrast, maintaining an internal frame value cache potentially
    allows proxies to largely be treated as normal dictionaries from an
    algorithmic complexity point of view, with allowances only needing
    to be made for the initial implicit O(n) cache refresh that runs the
    first time an operation that relies on the cache being up to date is
    executed.
-   the claim that a cache-free implementation would be simpler is
    highly suspect, as PEP 667 includes only a pure Python sketch of a
    subset of a mutable mapping implementation, rather than a
    full-fledged C implementation of a new mapping type integrated with
    the underlying data storage for optimised frames. PEP 558's fast
    locals proxy implementation delegates heavily to the frame value
    cache for the operations needed to fully implement the mutable
    mapping API, allowing it to re-use the existing dict implementations
    of the following operations:
    -   __len__
    -   __str__
    -   __or__ (dict union)
    -   __iter__ (allowing the dict_keyiterator type to be reused)
    -   __reversed__ (allowing the dict_reversekeyiterator type to be
        reused)
    -   keys() (allowing the dict_keys type to be reused)
    -   values() (allowing the dict_values type to be reused)
    -   items() (allowing the dict_items type to be reused)
    -   copy()
    -   popitem()
    -   value comparison operations

Of the three reasons, the first is the most important (since we need
compelling reasons to break API backwards compatibility, and we don't
have them).

However, after reviewing PEP 667's proposed Python level semantics, the
author of this PEP eventually agreed that they would be simpler for
users of the Python locals() API, so this distinction between the two
PEPs has been eliminated: regardless of which PEP and implementation is
accepted, the fast locals proxy object always provides a consistent view
of the current state of the local variables, even if this results in
some operations becoming O(n) that would be O(1) on a regular dictionary
(specifically, len(proxy) becomes O(n), since it needs to check which
names are currently bound, and proxy mapping comparisons avoid relying
on the length check optimisation that allows differences in the number
of stored keys to be detected quickly for regular mappings).

Due to the adoption of these non-standard performance characteristics in
the proxy implementation, the PyLocals_GetView() and
PyFrame_GetLocalsView() C APIs were also removed from the proposal in
this PEP.

This leaves the only remaining points of distinction between the two
PEPs as specifically related to the C API:

-   PEP 667 still proposes completely unnecessary C API breakage (the
    programmatic deprecation and eventual removal of PyEval_GetLocals(),
    PyFrame_FastToLocalsWithError(), and PyFrame_FastToLocals()) without
    justification, when it is entirely possible to keep these working
    indefinitely (and interoperably) given a suitably designed fast
    locals proxy implementation
-   the fast locals proxy handling of additional variables is defined in
    this PEP in a way that is fully interoperable with the existing
    PyEval_GetLocals() API. In the proxy implementation proposed in PEP
    667, users of the new frame API will not see changes made to
    additional variables by users of the old API, and changes made to
    additional variables via the old API will be overwritten on
    subsequent calls to PyEval_GetLocals().
-   the PyLocals_Get() API in this PEP is called PyEval_Locals() in
    PEP 667. This function name is a bit strange as it lacks a verb,
    making it look more like a type name than a data access API.
-   this PEP adds PyLocals_GetCopy() and PyFrame_GetLocalsCopy() APIs to
    allow extension modules to easily avoid incurring a double copy
    operation in frames where PyLocals_Get() already makes a copy
-   this PEP adds PyLocals_Kind, PyLocals_GetKind(), and
    PyFrame_GetLocalsKind() to allow extension modules to identify when
    code is running at function scope without having to inspect
    non-portable frame and code object APIs (without the proposed query
    API, the existing equivalent to the new
    PyLocals_GetKind() == PyLocals_SHALLOW_COPY check is to include the
    CPython internal frame API headers and check if
    _PyFrame_GetCode(PyEval_GetFrame())->co_flags & CO_OPTIMIZED is set)

The Python pseudo-code below is based on the implementation sketch
presented in PEP 667 as of the time of writing (2021-10-24). The
differences that provide the improved interoperability between the new
fast locals proxy API and the existing PyEval_GetLocals() API are noted
in comments.

As in PEP 667, all attributes that start with an underscore are
invisible and cannot be accessed directly. They serve only to illustrate
the proposed design.

For simplicity (and as in PEP 667), the handling of module and class
level frames is omitted (they're much simpler, as _locals is the
execution namespace, so no translation is required).

    NULL: Object # NULL is a singleton representing the absence of a value.

    class CodeType:

        _name_to_offset_mapping_impl: dict | NULL
        ...

        def __init__(self, ...):
            self._name_to_offset_mapping_impl = NULL
            self._variable_names = deduplicate(
                self.co_varnames + self.co_cellvars + self.co_freevars
            )
            ...

        def _is_cell(self, offset):
            ... # How the interpreter identifies cells is an implementation detail

        @property
        def _name_to_offset_mapping(self):
            "Mapping of names to offsets in local variable array."
            if self._name_to_offset_mapping_impl is NULL:

                self._name_to_offset_mapping_impl = {
                    name: index for (index, name) in enumerate(self._variable_names)
                }
            return self._name_to_offset_mapping_impl

    class FrameType:

        _fast_locals : array[Object] # The values of the local variables, items may be NULL.
        _locals: dict | NULL # Dictionary returned by PyEval_GetLocals()

        def __init__(self, ...):
            self._locals = NULL
            ...

        @property
        def f_locals(self):
            return FastLocalsProxy(self)

    class FastLocalsProxy:

        __slots__ "_frame"

        def __init__(self, frame:FrameType):
            self._frame = frame

        def _set_locals_entry(self, name, val):
            f = self._frame
            if f._locals is NULL:
                f._locals = {}
            f._locals[name] = val

        def __getitem__(self, name):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                val = f._fast_locals[index]
                if val is NULL:
                    raise KeyError(name)
                if co._is_cell(offset)
                    val = val.cell_contents
                    if val is NULL:
                        raise KeyError(name)
                # PyEval_GetLocals() interop: implicit frame cache refresh
                self._set_locals_entry(name, val)
                return val
            # PyEval_GetLocals() interop: frame cache may contain additional names
            if f._locals is NULL:
                raise KeyError(name)
            return f._locals[name]

        def __setitem__(self, name, value):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                kind = co._local_kinds[index]
                if co._is_cell(offset)
                    cell = f._locals[index]
                    cell.cell_contents = val
                else:
                    f._fast_locals[index] = val
            # PyEval_GetLocals() interop: implicit frame cache update
            # even for names that are part of the fast locals array
            self._set_locals_entry(name, val)

        def __delitem__(self, name):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                kind = co._local_kinds[index]
                if co._is_cell(offset)
                    cell = f._locals[index]
                    cell.cell_contents = NULL
                else:
                    f._fast_locals[index] = NULL
            # PyEval_GetLocals() interop: implicit frame cache update
            # even for names that are part of the fast locals array
            if f._locals is not NULL:
                del f._locals[name]

        def __iter__(self):
            f = self._frame
            co = f.f_code
            for index, name in enumerate(co._variable_names):
                val = f._fast_locals[index]
                if val is NULL:
                    continue
                if co._is_cell(offset):
                    val = val.cell_contents
                    if val is NULL:
                        continue
                yield name
            for name in f._locals:
                # Yield any extra names not defined on the frame
                if name in co._name_to_offset_mapping:
                    continue
                yield name

        def popitem(self):
            f = self._frame
            co = f.f_code
            for name in self:
                val = self[name]
            # PyEval_GetLocals() interop: implicit frame cache update
            # even for names that are part of the fast locals array
            del name
            return name, val

        def _sync_frame_cache(self):
            # This method underpins PyEval_GetLocals, PyFrame_FastToLocals
            # PyFrame_GetLocals, PyLocals_Get, mapping comparison, etc
            f = self._frame
            co = f.f_code
            res = 0
            if f._locals is NULL:
                f._locals = {}
            for index, name in enumerate(co._variable_names):
                val = f._fast_locals[index]
                if val is NULL:
                    f._locals.pop(name, None)
                    continue
                if co._is_cell(offset):
                    if val.cell_contents is NULL:
                        f._locals.pop(name, None)
                        continue
                f._locals[name] = val

        def __len__(self):
            self._sync_frame_cache()
            return len(self._locals)

Note: the simplest way to convert the earlier iterations of the PEP 558
reference implementation into a preliminary implementation of the now
proposed semantics is to remove the frame_cache_updated checks in
affected operations, and instead always sync the frame cache in those
methods. Adopting that approach changes the algorithmic complexity of
the following operations as shown (where n is the number of local and
cell variables defined on the frame):

  -   __len__: O(1) -> O(n)
  -   value comparison operations: no longer benefit from O(1) length
      check shortcut
  -   __iter__: O(1) -> O(n)
  -   __reversed__: O(1) -> O(n)
  -   keys(): O(1) -> O(n)
  -   values(): O(1) -> O(n)
  -   items(): O(1) -> O(n)
  -   popitem(): O(1) -> O(n)

The length check and value comparison operations have relatively limited
opportunities for improvement: without allowing usage of a potentially
stale cache, the only way to know how many variables are currently bound
is to iterate over all of them and check, and if the implementation is
going to be spending that many cycles on an operation anyway, it may as
well spend it updating the frame value cache and then consuming the
result. These operations are O(n) in both this PEP and in PEP 667.
Customised implementations could be provided that are faster than
updating the frame cache, but it's far from clear that the extra code
complexity needed to speed these operations up would be worthwhile when
it only offers a linear performance improvement rather than an
algorithmic complexity improvement.

The O(1) nature of the other operations can be restored by adding
implementation code that doesn't rely on the value cache being up to
date.

Keeping the iterator/iterable retrieval methods as O(1) will involve
writing custom replacements for the corresponding builtin dict helper
types, just as proposed in PEP 667. As illustrated above, the
implementations would be similar to the pseudo-code presented in PEP
667, but not identical (due to the improved PyEval_GetLocals()
interoperability offered by this PEP affecting the way it stores extra
variables).

popitem() can be improved from "always O(n)" to "O(n) worst case" by
creating a custom implementation that relies on the improved iteration
APIs.

To ensure stale frame information is never presented in the Python fast
locals proxy API, these changes in the reference implementation will
need to be implemented before merging.

The current implementation at time of writing (2021-10-24) also still
stores a copy of the fast refs mapping on each frame rather than storing
a single instance on the underlying code object (as it still stores cell
references directly, rather than check for cells on each fast locals
array access). Fixing this would also be required before merging.

Implementation

The reference implementation update is in development as a draft pull
request on GitHub ([14]).

Acknowledgements

Thanks to Nathaniel J. Smith for proposing the write-through proxy idea
in [15] and pointing out some critical design flaws in earlier
iterations of the PEP that attempted to avoid introducing such a proxy.

Thanks to Steve Dower and Petr Viktorin for asking that more attention
be paid to the developer experience of the proposed C API
additions[16][17].

Thanks to Larry Hastings for the suggestion on how to use enums in the
stable ABI while ensuring that they safely support typecasting from
arbitrary integers.

Thanks to Mark Shannon for pushing for further simplification of the C
level API and semantics, as well as significant clarification of the PEP
text (and for restarting discussion on the PEP in early 2021 after a
further year of inactivity)[18][19][20]. Mark's comments that were
ultimately published as PEP 667 also directly resulted in several
implementation efficiency improvements that avoid incurring the cost of
redundant O(n) mapping refresh operations when the relevant mappings
aren't used, as well as the change to ensure that the state reported
through the Python level f_locals API is never stale.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] Broken local variable assignment given threads + trace hook +
closure

[2] Broken local variable assignment given threads + trace hook +
closure

[3] Updating function local variables from pdb is unreliable

[4] CPython's Python API for installing trace hooks

[5] CPython's C API for installing trace hooks

[6] Broken local variable assignment given threads + trace hook +
closure

[7] Updating function local variables from pdb is unreliable

[8] Updating function local variables from pdb is unreliable

[9] Disable automatic update of frame locals during tracing

[10] Nathaniel's review of possible function level semantics for
locals()

[11] Nathaniel's review of possible function level semantics for
locals()

[12] Broken local variable assignment given threads + trace hook +
closure

[13] Discussion of more intentionally designed C API enhancements

[14] PEP 558 reference implementation

[15] Broken local variable assignment given threads + trace hook +
closure

[16] Discussion of more intentionally designed C API enhancements

[17] Petr Viktorin's suggestion to use an enum for PyLocals_Get's
behaviour

[18] python-dev thread: Resurrecting PEP 558 (Defined semantics for
locals())

[19] python-dev thread: Comments on PEP 558

[20] python-dev thread: More comments on PEP 558


text:
PEP: 556 Title: Threaded garbage collection Author: Antoine Pitrou
<EMAIL_ADDRESS>Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 08-Sep-2017 Python-Version: 3.7
Post-History: 08-Sep-2017

Deferral Notice

This PEP is currently not being actively worked on. It may be revived in
the future. The main missing steps are:

-   polish the implementation, adapting the test suite where necessary;
-   ensure setting threaded garbage collection does not disrupt existing
    code in unexpected ways (expected impact includes lengthening the
    lifetime of objects in reference cycles).

Abstract

This PEP proposes a new optional mode of operation for CPython's cyclic
garbage collector (GC) where implicit (i.e. opportunistic) collections
happen in a dedicated thread rather than synchronously.

Terminology

An "implicit" GC run (or "implicit" collection) is one that is triggered
opportunistically based on a certain heuristic computed over allocation
statistics, whenever a new allocation is requested. Details of the
heuristic are not relevant to this PEP, as it does not propose to change
it.

An "explicit" GC run (or "explicit" collection) is one that is requested
programmatically by an API call such as gc.collect.

"Threaded" refers to the fact that GC runs happen in a dedicated thread
separate from sequential execution of application code. It does not mean
"concurrent" (the Global Interpreter Lock, or GIL, still serializes
execution among Python threads including the dedicated GC thread) nor
"parallel" (the GC is not able to distribute its work onto several
threads at once to lower wall-clock latencies of GC runs).

Rationale

The mode of operation for the GC has always been to perform implicit
collections synchronously. That is, whenever the aforementioned
heuristic is activated, execution of application code in the current
thread is suspended and the GC is launched in order to reclaim dead
reference cycles.

There is a catch, though. Over the course of reclaiming dead reference
cycles (and any ancillary objects hanging at those cycles), the GC can
execute arbitrary finalization code in the form of __del__ methods and
weakref callbacks. Over the years, Python has been used for more and
more sophisticated purposes, and it is increasingly common for
finalization code to perform complex tasks, for example in distributed
systems where loss of an object may require notifying other (logical or
physical) nodes.

Interrupting application code at arbitrary points to execute
finalization code that may rely on a consistent internal state and/or on
acquiring synchronization primitives gives rise to reentrancy issues
that even the most seasoned experts have trouble fixing properly[1].

This PEP bases itself on the observation that, despite the apparent
similarities, same-thread reentrancy is a fundamentally harder problem
than multi-thread synchronization. Instead of letting each developer or
library author struggle with extremely hard reentrancy issues, one by
one, this PEP proposes to allow the GC to run in a separate thread where
well-known multi-thread synchronization practices are sufficient.

Proposal

Under this PEP, the GC has two modes of operation:

-   "serial", which is the default and legacy mode, where an implicit GC
    run is performed immediately in the thread that detects such an
    implicit run is desired (based on the aforementioned allocation
    heuristic).
-   "threaded", which can be explicitly enabled at runtime on a
    per-process basis, where implicit GC runs are scheduled whenever the
    allocation heuristic is triggered, but run in a dedicated background
    thread.

Hard reentrancy problems which plague sophisticated uses of finalization
callbacks in the "serial" mode become relatively easy multi-thread
synchronization problems in the "threaded" mode of operation.

The GC also traditionally allows for explicit GC runs, using the Python
API gc.collect and the C API PyGC_Collect. The visible semantics of
these two APIs are left unchanged: they perform a GC run immediately
when called, and only return when the GC run is finished.

New public APIs

Two new Python APIs are added to the gc module:

-   gc.set_mode(mode) sets the current mode of operation (either
    "serial" or "threaded"). If setting to "serial" and the current mode
    is "threaded", then the function also waits for the GC thread to
    end.
-   gc.get_mode() returns the current mode of operation.

It is allowed to switch back and forth between modes of operation.

Intended use

Given the per-process nature of the switch and its repercussions on
semantics of all finalization callbacks, it is recommended that it is
set at the beginning of an application's code (and/or in initializers
for child processes e.g. when using multiprocessing). Library functions
should probably not mess with this setting, just as they shouldn't call
gc.enable or gc.disable, but there's nothing to prevent them from doing
so.

Non-goals

This PEP does not address reentrancy issues with other kinds of
asynchronous code execution (for example signal handlers registered with
the signal module). The author believes that the overwhelming majority
of painful reentrancy issues occur with finalizers. Most of the time,
signal handlers are able to set a single flag and/or wake up a file
descriptor for the main program to notice. As for those signal handlers
which raise an exception, they have to execute in-thread.

This PEP also does not change the execution of finalization callbacks
when they are called as part of regular reference counting, i.e. when
releasing a visible reference drops an object's reference count to zero.
Since such execution happens at deterministic points in code, it is
usually not a problem.

Internal details

TODO: Update this section to conform to the current implementation.

gc module

An internal flag gc_is_threaded is added, telling whether GC is serial
or threaded.

An internal structure gc_mutex is added to avoid two GC runs at once:

    static struct {
        PyThread_type_lock lock;  /* taken when collecting */
        PyThreadState *owner;  /* whichever thread is currently collecting
                                  (NULL if no collection is taking place) */
    } gc_mutex;

An internal structure gc_thread is added to handle synchronization with
the GC thread:

    static struct {
       PyThread_type_lock wakeup; /* acts as an event
                                     to wake up the GC thread */
       int collection_requested; /* non-zero if collection requested */
       PyThread_type_lock done; /* acts as an event signaling
                                   the GC thread has exited */
    } gc_thread;

threading module

Two private functions are added to the threading module:

-   threading._ensure_dummy_thread(name) creates and registers a Thread
    instance for the current thread with the given name, and returns it.
-   threading._remove_dummy_thread(thread) removes the given thread (as
    returned by _ensure_dummy_thread) from the threading module's
    internal state.

The purpose of these two functions is to improve debugging and
introspection by letting threading.current_thread() return a more
meaningfully-named object when called inside a finalization callback in
the GC thread.

Pseudo-code

Here is a proposed pseudo-code for the main primitives, public and
internal, required for implementing this PEP. All of them will be
implemented in C and live inside the gc module, unless otherwise noted:

    def collect_with_callback(generation):
        """
        Collect up to the given *generation*.
        """
        # Same code as currently (see collect_with_callback() in gcmodule.c)

    def collect_generations():
        """
        Collect as many generations as desired by the heuristic.
        """
        # Same code as currently (see collect_generations() in gcmodule.c)

    def lock_and_collect(generation=-1):
        """
        Perform a collection with thread safety.
        """
        me = PyThreadState_GET()
        if gc_mutex.owner == me:
            # reentrant GC collection request, bail out
            return
        Py_BEGIN_ALLOW_THREADS
        gc_mutex.lock.acquire()
        Py_END_ALLOW_THREADS
        gc_mutex.owner = me
        try:
            if generation >= 0:
                return collect_with_callback(generation)
            else:
                return collect_generations()
        finally:
            gc_mutex.owner = NULL
            gc_mutex.lock.release()

    def schedule_gc_request():
        """
        Ask the GC thread to run an implicit collection.
        """
        assert gc_is_threaded == True
        # Note this is extremely fast if a collection is already requested
        if gc_thread.collection_requested == False:
            gc_thread.collection_requested = True
            gc_thread.wakeup.release()

    def is_implicit_gc_desired():
        """
        Whether an implicit GC run is currently desired based on allocation
        stats.  Return a generation number, or -1 if none desired.
        """
        # Same heuristic as currently (see _PyObject_GC_Alloc in gcmodule.c)

    def PyGC_Malloc():
        """
        Allocate a GC-enabled object.
        """
        # Update allocation statistics (same code as currently, omitted for brevity)
        if is_implicit_gc_desired():
            if gc_is_threaded:
                schedule_gc_request()
            else:
                lock_and_collect()
        # Go ahead with allocation (same code as currently, omitted for brevity)

    def gc_thread(interp_state):
        """
        Dedicated loop for threaded GC.
        """
        # Init Python thread state (omitted, see t_bootstrap in _threadmodule.c)
        # Optional: init thread in Python threading module, for better introspection
        me = threading._ensure_dummy_thread(name="GC thread")

        while gc_is_threaded == True:
            Py_BEGIN_ALLOW_THREADS
            gc_thread.wakeup.acquire()
            Py_END_ALLOW_THREADS
            if gc_thread.collection_requested != 0:
                gc_thread.collection_requested = 0
                lock_and_collect(generation=-1)

        threading._remove_dummy_thread(me)
        # Signal we're exiting
        gc_thread.done.release()
        # Free Python thread state (omitted)

    def gc.set_mode(mode):
        """
        Set current GC mode.  This is a process-global setting.
        """
        if mode == "threaded":
            if not gc_is_threaded == False:
                # Launch thread
                gc_thread.done.acquire(block=False)  # should not fail
                gc_is_threaded = True
                PyThread_start_new_thread(gc_thread)
        elif mode == "serial":
            if gc_is_threaded == True:
                # Wake up thread, asking it to end
                gc_is_threaded = False
                gc_thread.wakeup.release()
                # Wait for thread exit
                Py_BEGIN_ALLOW_THREADS
                gc_thread.done.acquire()
                Py_END_ALLOW_THREADS
                gc_thread.done.release()
        else:
            raise ValueError("unsupported mode %r" % (mode,))

    def gc.get_mode(mode):
        """
        Get current GC mode.
        """
        return "threaded" if gc_is_threaded else "serial"

    def gc.collect(generation=2):
        """
        Schedule collection of the given generation and wait for it to
        finish.
        """
        return lock_and_collect(generation)

Discussion

Default mode

One may wonder whether the default mode should simply be changed to
"threaded". For multi-threaded applications, it would probably not be a
problem: those applications must already be prepared for finalization
handlers to be run in arbitrary threads. In single-thread applications,
however, it is currently guaranteed that finalizers will always be
called in the main thread. Breaking this property may induce subtle
behaviour changes or bugs, for example if finalizers rely on some
thread-local values.

Another problem is when a program uses fork() for concurrency. Calling
fork() from a single-threaded program is safe, but it's fragile (to say
the least) if the program is multi-threaded.

Explicit collections

One may ask whether explicit collections should also be delegated to the
background thread. The answer is it doesn't really matter: since
gc.collect and PyGC_Collect actually wait for the collection to end
(breaking this property would break compatibility), delegating the
actual work to a background thread wouldn't ease synchronization with
the thread requesting an explicit collection.

In the end, this PEP choses the behaviour that seems simpler to
implement based on the pseudo-code above.

Impact on memory use

The "threaded" mode incurs a slight delay in implicit collections
compared to the default "serial" mode. This obviously may change the
memory profile of certain applications. By how much remains to be
measured in real-world use, but we expect the impact to remain minor and
bearable. First because implicit collections are based on a heuristic
whose effect does not result in deterministic visible behaviour anyway.
Second because the GC deals with reference cycles while many objects are
reclaimed immediately when their last visible reference disappears.

Impact on CPU consumption

The pseudo-code above adds two lock operations for each implicit
collection request in "threaded" mode: one in the thread making the
request (a release call) and one in the GC thread (an acquire call). It
also adds two other lock operations, regardless of the current mode,
around each actual collection.

We expect the cost of those lock operations to be very small, on modern
systems, compared to the actual cost of crawling through the chains of
pointers during the collection itself ("pointer chasing" being one of
the hardest workloads on modern CPUs, as it lends itself poorly to
speculation and superscalar execution).

Actual measurements on worst-case mini-benchmarks may help provide
reassuring upper bounds.

Impact on GC pauses

While this PEP does not concern itself with GC pauses, there is a
practical chance that releasing the GIL at some point during an implicit
collection (for example by virtue of executing a pure Python finalizer)
will allow application code to run in-between, lowering the visible GC
pause time for some applications.

If this PEP is accepted, future work may try to better realize this
potential by speculatively releasing the GIL during collections, though
it is unclear how doable that is.

Open issues

-   gc.set_mode should probably be protected against multiple concurrent
    invocations. Also, it should raise when called from inside a GC run
    (i.e. from a finalizer).
-   What happens at shutdown? Does the GC thread run until _PyGC_Fini()
    is called?

Implementation

A draft implementation is available in the threaded_gc branch [2] of the
author's Github fork[3].

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] https://bugs.python.org/issue14976

[2] https://github.com/pitrou/cpython/tree/threaded_gc

[3] https://github.com/pitrou/cpython/

license: Public Domain