text:
PEP: 670 Title: Convert macros to functions in the Python C API Author:
Erlend Egeberg Aasland<EMAIL_ADDRESS>Victor Stinner
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 19-Oct-2021 Python-Version: 3.11 Post-History:
20-Oct-2021, 08-Feb-2022, 22-Feb-2022 Resolution:
<EMAIL_ADDRESS>
Abstract

Macros in the C API will be converted to static inline functions or
regular functions. This will help avoid macro pitfalls in C/C++, and
make the functions usable from other programming languages.

To avoid compiler warnings, function arguments of pointer types will be
cast to appropriate types using additional macros. The cast will not be
done in the limited C API version 3.11: users who opt in to the new
limited API may need to add casts to the exact expected type.

To avoid introducing incompatible changes, macros which can be used as
l-value in an assignment will not be converted.

Rationale

The use of macros may have unintended adverse effects that are hard to
avoid, even for experienced C developers. Some issues have been known
for years, while others have been discovered recently in Python. Working
around macro pitfalls makes the macro code harder to read and to
maintain.

Converting macros to functions has multiple advantages:

-   Functions don't suffer from macro pitfalls, for example the
    following ones described in GCC documentation:

    -   Misnesting
    -   Operator precedence problems
    -   Swallowing the semicolon
    -   Duplication of side effects
    -   Self-referential macros
    -   Argument prescan
    -   Newlines in arguments

    Functions don't need the following workarounds for macro pitfalls,
    making them usually easier to read and to maintain than similar
    macro code:

    -   Adding parentheses around arguments.
    -   Using line continuation characters if the function is written on
        multiple lines.
    -   Adding commas to execute multiple expressions.
    -   Using do { ... } while (0) to write multiple statements.

-   Argument types and the return type of functions are well defined.

-   Debuggers and profilers can retrieve the name of inlined functions.

-   Debuggers can put breakpoints on inlined functions.

-   Variables have a well-defined scope.

Converting macros and static inline functions to regular functions makes
these regular functions accessible to projects which use Python but
cannot use macros and static inline functions.

Specification

Convert macros to static inline functions

Most macros will be converted to static inline functions.

The following macros will not be converted:

-   Object-like macros (i.e. those which don't need parentheses and
    arguments). For example:
    -   Empty macros. Example: #define Py_HAVE_CONDVAR.
    -   Macros only defining a value, even if a constant with a well
        defined type would be better. Example:
        #define METH_VARARGS 0x0001.
-   Compatibility layer for different C compilers, C language
    extensions, or recent C features. Example: Py_GCC_ATTRIBUTE(),
    Py_ALWAYS_INLINE, Py_MEMCPY().
-   Macros used for definitions rather than behavior. Example:
    PyAPI_FUNC, Py_DEPRECATED, Py_PYTHON_H.
-   Macros that need C preprocessor features, like stringification and
    concatenation. Example: Py_STRINGIFY().
-   Macros which cannot be converted to functions. Examples:
    Py_BEGIN_ALLOW_THREADS (contains an unpaired }), Py_VISIT (relies on
    specific variable names), Py_RETURN_RICHCOMPARE (returns from the
    calling function).
-   Macros which can be used as an l-value in assignments. This would be
    an incompatible change and is out of the scope of this PEP. Example:
    PyBytes_AS_STRING().
-   Macros which have different return types depending on the code path
    or arguments.

Convert static inline functions to regular functions

Static inline functions in the public C API may be converted to regular
functions, but only if there is no measurable performance impact of
changing the function. The performance impact should be measured with
benchmarks.

Cast pointer arguments

Currently, most macros accepting pointers cast pointer arguments to
their expected types. For example, in Python 3.6, the Py_TYPE() macro
casts its argument to PyObject*:

    #define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)

The Py_TYPE() macro accepts the PyObject* type, but also any pointer
types, such as PyLongObject* and PyDictObject*.

Functions are strongly typed, and can only accept one type of argument.

To avoid compiler errors and warnings in existing code, when a macro is
converted to a function and the macro casts at least one of its
arguments a new macro will be added to keep the cast. The new macro and
the function will have the same name.

Example with the Py_TYPE() macro converted to a static inline function:

    static inline PyTypeObject* Py_TYPE(PyObject *ob) {
        return ob->ob_type;
    }
    #define Py_TYPE(ob) Py_TYPE((PyObject*)(ob))

The cast is kept for all pointer types, not only PyObject*. This
includes casts to void*: removing a cast to void* would emit a new
warning if the function is called with a const void* variable. For
example, the PyUnicode_WRITE() macro casts its data argument to void*,
and so it currently accepts const void* type, even though it writes into
data. This PEP will not change this.

Avoid the cast in the limited C API version 3.11

The casts will be excluded from the limited C API version 3.11 and
newer. When an API user opts into the new limited API, they must pass
the expected type or perform the cast.

As an example, Py_TYPE() will be defined like this:

    static inline PyTypeObject* Py_TYPE(PyObject *ob) {
        return ob->ob_type;
    }
    #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000
    #  define Py_TYPE(ob) Py_TYPE((PyObject*)(ob))
    #endif

Return type is not changed

When a macro is converted to a function, its return type must not change
to prevent emitting new compiler warnings.

For example, Python 3.7 changed the return type of PyUnicode_AsUTF8()
from char* to const char* (commit). The change emitted new compiler
warnings when building C extensions expecting char*. This PEP doesn't
change the return type to prevent this issue.

Backwards Compatibility

The PEP is designed to avoid C API incompatible changes.

Only C extensions explicitly targeting the limited C API version 3.11
must now pass the expected types to functions: pointer arguments are no
longer cast to the expected types.

Function arguments of pointer types are still cast and return types are
not changed to prevent emitting new compiler warnings.

Macros which can be used as l-value in an assignment are not modified by
this PEP to avoid incompatible changes.

Examples of Macro Pitfalls

Duplication of side effects

Macros:

    #define PySet_Check(ob) \
        (Py_IS_TYPE(ob, &PySet_Type) \
         || PyType_IsSubtype(Py_TYPE(ob), &PySet_Type))

    #define Py_IS_NAN(X) ((X) != (X))

If the op or the X argument has a side effect, the side effect is
duplicated: it executed twice by PySet_Check() and Py_IS_NAN().

For example, the pos++ argument in the
PyUnicode_WRITE(kind, data, pos++, ch) code has a side effect. This code
is safe because the PyUnicode_WRITE() macro only uses its 3rd argument
once and so does not duplicate pos++ side effect.

Misnesting

Example of the bpo-43181: Python macros don't shield arguments. The
PyObject_TypeCheck() macro before it has been fixed:

    #define PyObject_TypeCheck(ob, tp) \
        (Py_IS_TYPE(ob, tp) || PyType_IsSubtype(Py_TYPE(ob), (tp)))

C++ usage example:

    PyObject_TypeCheck(ob, U(f<a,b>(c)))

The preprocessor first expands it:

    (Py_IS_TYPE(ob, f<a,b>(c)) || ...)

C++ "<" and ">" characters are not treated as brackets by the
preprocessor, so the Py_IS_TYPE() macro is invoked with 3 arguments:

-   ob
-   f<a
-   b>(c)

The compilation fails with an error on Py_IS_TYPE() which only takes 2
arguments.

The bug is that the op and tp arguments of PyObject_TypeCheck() must be
put between parentheses: replace Py_IS_TYPE(ob, tp) with
Py_IS_TYPE((ob), (tp)). In regular C code, these parentheses are
redundant, can be seen as a bug, and so are often forgotten when writing
macros.

To avoid Macro Pitfalls, the PyObject_TypeCheck() macro has been
converted to a static inline function: commit.

Examples of hard to read macros

PyObject_INIT()

Example showing the usage of commas in a macro which has a return value.

Python 3.7 macro:

    #define PyObject_INIT(op, typeobj) \
        ( Py_TYPE(op) = (typeobj), _Py_NewReference((PyObject *)(op)), (op) )

Python 3.8 function (simplified code):

    static inline PyObject*
    _PyObject_INIT(PyObject *op, PyTypeObject *typeobj)
    {
        Py_TYPE(op) = typeobj;
        _Py_NewReference(op);
        return op;
    }

    #define PyObject_INIT(op, typeobj) \
        _PyObject_INIT(_PyObject_CAST(op), (typeobj))

-   The function doesn't need the line continuation character "\".
-   It has an explicit "return op;" rather than the surprising ", (op)"
    syntax at the end of the macro.
-   It uses short statements on multiple lines, rather than being
    written as a single long line.
-   Inside the function, the op argument has the well defined type
    PyObject* and so doesn't need casts like (PyObject *)(op).
-   Arguments don't need to be put inside parentheses: use typeobj,
    rather than (typeobj).

_Py_NewReference()

Example showing the usage of an #ifdef inside a macro.

Python 3.7 macro (simplified code):

    #ifdef COUNT_ALLOCS
    #  define _Py_INC_TPALLOCS(OP) inc_count(Py_TYPE(OP))
    #  define _Py_COUNT_ALLOCS_COMMA  ,
    #else
    #  define _Py_INC_TPALLOCS(OP)
    #  define _Py_COUNT_ALLOCS_COMMA
    #endif /* COUNT_ALLOCS */

    #define _Py_NewReference(op) (                   \
        _Py_INC_TPALLOCS(op) _Py_COUNT_ALLOCS_COMMA  \
        Py_REFCNT(op) = 1)

Python 3.8 function (simplified code):

    static inline void _Py_NewReference(PyObject *op)
    {
        _Py_INC_TPALLOCS(op);
        Py_REFCNT(op) = 1;
    }

PyUnicode_READ_CHAR()

This macro reuses arguments, and possibly calls PyUnicode_KIND multiple
times:

    #define PyUnicode_READ_CHAR(unicode, index) \
    (assert(PyUnicode_Check(unicode)),          \
     assert(PyUnicode_IS_READY(unicode)),       \
     (Py_UCS4)                                  \
        (PyUnicode_KIND((unicode)) == PyUnicode_1BYTE_KIND ? \
            ((const Py_UCS1 *)(PyUnicode_DATA((unicode))))[(index)] : \
            (PyUnicode_KIND((unicode)) == PyUnicode_2BYTE_KIND ? \
                ((const Py_UCS2 *)(PyUnicode_DATA((unicode))))[(index)] : \
                ((const Py_UCS4 *)(PyUnicode_DATA((unicode))))[(index)] \
            ) \
        ))

Possible implementation as a static inlined function:

    static inline Py_UCS4
    PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index)
    {
        assert(PyUnicode_Check(unicode));
        assert(PyUnicode_IS_READY(unicode));

        switch (PyUnicode_KIND(unicode)) {
        case PyUnicode_1BYTE_KIND:
            return (Py_UCS4)((const Py_UCS1 *)(PyUnicode_DATA(unicode)))[index];
        case PyUnicode_2BYTE_KIND:
            return (Py_UCS4)((const Py_UCS2 *)(PyUnicode_DATA(unicode)))[index];
        case PyUnicode_4BYTE_KIND:
        default:
            return (Py_UCS4)((const Py_UCS4 *)(PyUnicode_DATA(unicode)))[index];
        }
    }

Macros converted to functions since Python 3.8

This is a list of macros already converted to functions between Python
3.8 and Python 3.11. Even though some converted macros (like
Py_INCREF()) are very commonly used by C extensions, these conversions
did not significantly impact Python performance and most of them didn't
break backward compatibility.

Macros converted to static inline functions

Python 3.8:

-   Py_DECREF()
-   Py_INCREF()
-   Py_XDECREF()
-   Py_XINCREF()
-   PyObject_INIT()
-   PyObject_INIT_VAR()
-   _PyObject_GC_UNTRACK()
-   _Py_Dealloc()

Macros converted to regular functions

Python 3.9:

-   PyIndex_Check()
-   PyObject_CheckBuffer()
-   PyObject_GET_WEAKREFS_LISTPTR()
-   PyObject_IS_GC()
-   PyObject_NEW(): alias to PyObject_New()
-   PyObject_NEW_VAR(): alias to PyObjectVar_New()

To avoid performance slowdown on Python built without LTO, private
static inline functions have been added to the internal C API:

-   _PyIndex_Check()
-   _PyObject_IS_GC()
-   _PyType_HasFeature()
-   _PyType_IS_GC()

Static inline functions converted to regular functions

Python 3.11:

-   PyObject_CallOneArg()
-   PyObject_Vectorcall()
-   PyVectorcall_Function()
-   _PyObject_FastCall()

To avoid performance slowdown on Python built without LTO, a private
static inline function has been added to the internal C API:

-   _PyVectorcall_FunctionInline()

Incompatible changes

While other converted macros didn't break the backward compatibility,
there is an exception.

The 3 macros Py_REFCNT(), Py_TYPE() and Py_SIZE() have been converted to
static inline functions in Python 3.10 and 3.11 to disallow using them
as l-value in assignment. It is an incompatible change made on purpose:
see bpo-39573 for the rationale.

This PEP does not propose converting macros which can be used as l-value
to avoid introducing new incompatible changes.

Performance concerns and benchmarks

There have been concerns that converting macros to functions can degrade
performance.

This section explains performance concerns and shows benchmark results
using PR 29728, which replaces the following static inline functions
with macros:

-   PyObject_TypeCheck()
-   PyType_Check(), PyType_CheckExact()
-   PyType_HasFeature()
-   PyVectorcall_NARGS()
-   Py_DECREF(), Py_XDECREF()
-   Py_INCREF(), Py_XINCREF()
-   Py_IS_TYPE()
-   Py_NewRef()
-   Py_REFCNT(), Py_TYPE(), Py_SIZE()

The benchmarks were run on Fedora 35 (Linux) with GCC 11 on a laptop
with 8 logical CPUs (4 physical CPU cores).

Static inline functions

First of all, converting macros to static inline functions has
negligible impact on performance: the measured differences are
consistent with noise due to unrelated factors.

Static inline functions are a new feature in the C99 standard. Modern C
compilers have efficient heuristics to decide if a function should be
inlined or not.

When a C compiler decides to not inline, there is likely a good reason.
For example, inlining would reuse a register which requires to
save/restore the register value on the stack and so increases the stack
memory usage, or be less efficient.

Benchmark of the ./python -m test -j5 command on Python built in release
mode with gcc -O3, LTO and PGO:

-   Macros (PR 29728): 361 sec +- 1 sec
-   Static inline functions (reference): 361 sec +- 1 sec

There is no significant performance difference between macros and static
inline functions when static inline functions are inlined.

Debug build

Performance in debug builds can suffer when macros are converted to
functions. This is compensated by better debuggability: debuggers can
retrieve function names, set breakpoints inside functions, etc.

On Windows, when Python is built in debug mode by Visual Studio, static
inline functions are not inlined.

On other platforms, ./configure --with-pydebug uses the -Og compiler
option on compilers that support it (including GCC and LLVM Clang). -Og
means “optimize debugging experience”. Otherwise, the -O0 compiler
option is used. -O0 means “disable most optimizations”.

With GCC 11, gcc -Og can inline static inline functions, whereas gcc -O0
does not inline static inline functions.

Benchmark of the ./python -m test -j10 command on Python built in debug
mode with gcc -O0 (that is, compiler optimizations, including inlining,
are explicitly disabled):

-   Macros (PR 29728): 345 sec ± 5 sec
-   Static inline functions (reference): 360 sec ± 6 sec

Replacing macros with static inline functions makes Python 1.04x slower
when the compiler does not inline static inline functions.

Note that benchmarks should not be run on a Python debug build.
Moreover, using link-time optimization (LTO) and profile-guided
optimization (PGO) is recommended for best performance and reliable
benchmarks. PGO helps the compiler to decide if functions should be
inlined or not.

Force inlining

The Py_ALWAYS_INLINE macro can be used to force inlining. This macro
uses __attribute__((always_inline)) with GCC and Clang, and
__forceinline with MSC.

Previous attempts to use Py_ALWAYS_INLINE didn't show any benefit, and
were abandoned. See for example bpo-45094 "Consider using __forceinline
and __attribute__((always_inline)) on static inline functions
(Py_INCREF, Py_TYPE) for debug build".

When the Py_INCREF() macro was converted to a static inline function in
2018 (commit), it was decided not to force inlining. The machine code
was analyzed with multiple C compilers and compiler options, and
Py_INCREF() was always inlined without having to force inlining. The
only case where it was not inlined was the debug build. See discussion
in bpo-35059 "Convert Py_INCREF() and PyObject_INIT() to inlined
functions".

Disabling inlining

On the other side, the Py_NO_INLINE macro can be used to disable
inlining. It can be used to reduce the stack memory usage, or to prevent
inlining on LTO+PGO builds, which generally inline code more
aggressively: see bpo-33720. The Py_NO_INLINE macro uses
__attribute__ ((noinline)) with GCC and Clang, and __declspec(noinline)
with MSC.

This technique is available, though we currently don't know a concrete
function for which it would be useful. Note that with macros, it is not
possible to disable inlining at all.

Rejected Ideas

Keep macros, but fix some macro issues

Macros are always "inlined" with any C compiler.

The duplication of side effects can be worked around in the caller of
the macro.

People using macros should be considered "consenting adults". People who
feel unsafe with macros should simply not use them.

These ideas are rejected because macros are error prone, and it is too
easy to miss a macro pitfall when writing and reviewing macro code.
Moreover, macros are harder to read and maintain than functions.

Post History

python-dev mailing list threads:

-   Version 2 of PEP 670 - Convert macros to functions in the Python C
    API (February 2022)
-   Steering Council reply to PEP 670 -- Convert macros to functions in
    the Python C API (February 2022)
-   PEP 670: Convert macros to functions in the Python C API (October
    2021)

References

-   bpo-45490: [C API] PEP 670: Convert macros to functions in the
    Python C API (October 2021).
-   What to do with unsafe macros (March 2021).
-   bpo-43502: [C-API] Convert obvious unsafe macros to static inline
    functions (March 2021).

Version History

-   Version 2:
    -   Stricter policy on not changing argument types and return type.
    -   Better explain why pointer arguments require a cast to not emit
        new compiler warnings.
    -   Macros which can be used as l-values are no longer modified by
        the PEP.
    -   Macros having multiple return types are no longer modified by
        the PEP.
    -   Limited C API version 3.11 no longer casts pointer arguments.
    -   No longer remove return values of macros "which should not have
        a return value".
    -   Add "Macros converted to functions since Python 3.8" section.
    -   Add "Benchmark comparing macros and static inline functions"
        section.
-   Version 1: First public version

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 510 Title: Specialize functions with guards Version: $Revision$
Last-Modified: $Date$ Author: Victor Stinner<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
04-Jan-2016 Python-Version: 3.6

Rejection Notice

This PEP was rejected by its author since the design didn't show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.

Abstract

Add functions to the Python C API to specialize pure Python functions:
add specialized codes with guards. It allows to implement static
optimizers respecting the Python semantics.

Rationale

Python semantics

Python is hard to optimize because almost everything is mutable: builtin
functions, function code, global variables, local variables, ... can be
modified at runtime. Implement optimizations respecting the Python
semantics requires to detect when "something changes", we will call
these checks "guards".

This PEP proposes to add a public API to the Python C API to add
specialized codes with guards to a function. When the function is
called, a specialized code is used if nothing changed, otherwise use the
original bytecode.

Even if guards help to respect most parts of the Python semantics, it's
hard to optimize Python without making subtle changes on the exact
behaviour. CPython has a long history and many applications rely on
implementation details. A compromise must be found between "everything
is mutable" and performance.

Writing an optimizer is out of the scope of this PEP.

Why not a JIT compiler?

There are multiple JIT compilers for Python actively developed:

-   PyPy
-   Pyston
-   Numba
-   Pyjion

Numba is specific to numerical computation. Pyston and Pyjion are still
young. PyPy is the most complete Python interpreter, it is generally
faster than CPython in micro- and many macro-benchmarks and has a very
good compatibility with CPython (it respects the Python semantics).
There are still issues with Python JIT compilers which avoid them to be
widely used instead of CPython.

Many popular libraries like numpy, PyGTK, PyQt, PySide and wxPython are
implemented in C or C++ and use the Python C API. To have a small memory
footprint and better performances, Python JIT compilers do not use
reference counting to use a faster garbage collector, do not use C
structures of CPython objects and manage memory allocations differently.
PyPy has a cpyext module which emulates the Python C API but it has
worse performances than CPython and does not support the full Python C
API.

New features are first developed in CPython. In January 2016, the latest
CPython stable version is 3.5, whereas PyPy only supports Python 2.7 and
3.2, and Pyston only supports Python 2.7.

Even if PyPy has a very good compatibility with Python, some modules are
still not compatible with PyPy: see PyPy Compatibility Wiki. The
incomplete support of the Python C API is part of this problem. There
are also subtle differences between PyPy and CPython like reference
counting: object destructors are always called in PyPy, but can be
called "later" than in CPython. Using context managers helps to control
when resources are released.

Even if PyPy is much faster than CPython in a wide range of benchmarks,
some users still report worse performances than CPython on some specific
use cases or unstable performances.

When Python is used as a scripting program for programs running less
than 1 minute, JIT compilers can be slower because their startup time is
higher and the JIT compiler takes time to optimize the code. For
example, most Mercurial commands take a few seconds.

Numba now supports ahead of time compilation, but it requires decorator
to specify arguments types and it only supports numerical types.

CPython 3.5 has almost no optimization: the peephole optimizer only
implements basic optimizations. A static compiler is a compromise
between CPython 3.5 and PyPy.

Note

There was also the Unladen Swallow project, but it was abandoned in
2011.

Examples

Following examples are not written to show powerful optimizations
promising important speedup, but to be short and easy to understand,
just to explain the principle.

Hypothetical myoptimizer module

Examples in this PEP uses a hypothetical myoptimizer module which
provides the following functions and types:

-   specialize(func, code, guards): add the specialized code code with
    guards guards to the function func
-   get_specialized(func): get the list of specialized codes as a list
    of (code, guards) tuples where code is a callable or code object and
    guards is a list of a guards
-   GuardBuiltins(name): guard watching for builtins.__dict__[name] and
    globals()[name]. The guard fails if builtins.__dict__[name] is
    replaced, or if globals()[name] is set.

Using bytecode

Add specialized bytecode where the call to the pure builtin function
chr(65) is replaced with its result "A":

    import myoptimizer

    def func():
        return chr(65)

    def fast_func():
        return "A"

    myoptimizer.specialize(func, fast_func.__code__,
                           [myoptimizer.GuardBuiltins("chr")])
    del fast_func

Example showing the behaviour of the guard:

    print("func(): %s" % func())
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))
    print()

    import builtins
    builtins.chr = lambda obj: "mock"

    print("func(): %s" % func())
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))

Output:

    func(): A
    #specialized: 1

    func(): mock
    #specialized: 0

The first call uses the specialized bytecode which returns the string
"A". The second call removes the specialized code because the builtin
chr() function was replaced, and executes the original bytecode calling
chr(65).

On a microbenchmark, calling the specialized bytecode takes 88 ns,
whereas the original function takes 145 ns (+57 ns): 1.6 times as fast.

Using builtin function

Add the C builtin chr() function as the specialized code instead of a
bytecode calling chr(obj):

    import myoptimizer

    def func(arg):
        return chr(arg)

    myoptimizer.specialize(func, chr,
                           [myoptimizer.GuardBuiltins("chr")])

Example showing the behaviour of the guard:

    print("func(65): %s" % func(65))
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))
    print()

    import builtins
    builtins.chr = lambda obj: "mock"

    print("func(65): %s" % func(65))
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))

Output:

    func(): A
    #specialized: 1

    func(): mock
    #specialized: 0

The first call calls the C builtin chr() function (without creating a
Python frame). The second call removes the specialized code because the
builtin chr() function was replaced, and executes the original bytecode.

On a microbenchmark, calling the C builtin takes 95 ns, whereas the
original bytecode takes 155 ns (+60 ns): 1.6 times as fast. Calling
directly chr(65) takes 76 ns.

Choose the specialized code

Pseudo-code to choose the specialized code to call a pure Python
function:

    def call_func(func, args, kwargs):
        specialized = myoptimizer.get_specialized(func)
        nspecialized = len(specialized)
        index = 0
        while index < nspecialized:
            specialized_code, guards = specialized[index]

            for guard in guards:
                check = guard(args, kwargs)
                if check:
                    break

            if not check:
                # all guards succeeded:
                # use the specialized code
                return specialized_code
            elif check == 1:
                # a guard failed temporarily:
                # try the next specialized code
                index += 1
            else:
                assert check == 2
                # a guard will always fail:
                # remove the specialized code
                del specialized[index]

        # if a guard of each specialized code failed, or if the function
        # has no specialized code, use original bytecode
        code = func.__code__

Changes

Changes to the Python C API:

-   Add a PyFuncGuardObject object and a PyFuncGuard_Type type

-   Add a PySpecializedCode structure

-   Add the following fields to the PyFunctionObject structure:

        Py_ssize_t nb_specialized;
        PySpecializedCode *specialized;

-   Add function methods:

    -   PyFunction_Specialize()
    -   PyFunction_GetSpecializedCodes()
    -   PyFunction_GetSpecializedCode()
    -   PyFunction_RemoveSpecialized()
    -   PyFunction_RemoveAllSpecialized()

None of these function and types are exposed at the Python level.

All these additions are explicitly excluded of the stable ABI.

When a function code is replaced (func.__code__ = new_code), all
specialized codes and guards are removed.

Function guard

Add a function guard object:

    typedef struct {
        PyObject ob_base;
        int (*init) (PyObject *guard, PyObject *func);
        int (*check) (PyObject *guard, PyObject **stack, int na, int nk);
    } PyFuncGuardObject;

The init() function initializes a guard:

-   Return 0 on success
-   Return 1 if the guard will always fail: PyFunction_Specialize() must
    ignore the specialized code
-   Raise an exception and return -1 on error

The check() function checks a guard:

-   Return 0 on success
-   Return 1 if the guard failed temporarily
-   Return 2 if the guard will always fail: the specialized code must be
    removed
-   Raise an exception and return -1 on error

stack is an array of arguments: indexed arguments followed by (key,
value) pairs of keyword arguments. na is the number of indexed
arguments. nk is the number of keyword arguments: the number of (key,
value) pairs. stack contains na + nk * 2 objects.

Specialized code

Add a specialized code structure:

    typedef struct {
        PyObject *code;        /* callable or code object */
        Py_ssize_t nb_guard;
        PyObject **guards;     /* PyFuncGuardObject objects */
    } PySpecializedCode;

Function methods

PyFunction_Specialize

Add a function method to specialize the function, add a specialized code
with guards:

    int PyFunction_Specialize(PyObject *func,
                              PyObject *code, PyObject *guards)

If code is a Python function, the code object of the code function is
used as the specialized code. The specialized Python function must have
the same parameter defaults, the same keyword parameter defaults, and
must not have specialized code.

If code is a Python function or a code object, a new code object is
created and the code name and first line number of the code object of
func are copied. The specialized code must have the same cell variables
and the same free variables.

Result:

-   Return 0 on success
-   Return 1 if the specialization has been ignored
-   Raise an exception and return -1 on error

PyFunction_GetSpecializedCodes

Add a function method to get the list of specialized codes:

    PyObject* PyFunction_GetSpecializedCodes(PyObject *func)

Return a list of (code, guards) tuples where code is a callable or code
object and guards is a list of PyFuncGuard objects. Raise an exception
and return NULL on error.

PyFunction_GetSpecializedCode

Add a function method checking guards to choose a specialized code:

    PyObject* PyFunction_GetSpecializedCode(PyObject *func,
                                            PyObject **stack,
                                            int na, int nk)

See check() function of guards for stack, na and nk arguments. Return a
callable or a code object on success. Raise an exception and return NULL
on error.

PyFunction_RemoveSpecialized

Add a function method to remove a specialized code with its guards by
its index:

    int PyFunction_RemoveSpecialized(PyObject *func, Py_ssize_t index)

Return 0 on success or if the index does not exist. Raise an exception
and return -1 on error.

PyFunction_RemoveAllSpecialized

Add a function method to remove all specialized codes and guards of a
function:

    int PyFunction_RemoveAllSpecialized(PyObject *func)

Return 0 on success. Raise an exception and return -1 if func is not a
function.

Benchmark

Microbenchmark on python3.6 -m timeit -s 'def f(): pass' 'f()' (best of
3 runs):

-   Original Python: 79 ns
-   Patched Python: 79 ns

According to this microbenchmark, the changes has no overhead on calling
a Python function without specialization.

Implementation

The issue #26098: PEP 510: Specialize functions with guards contains a
patch which implements this PEP.

Other implementations of Python

This PEP only contains changes to the Python C API, the Python API is
unchanged. Other implementations of Python are free to not implement new
additions, or implement added functions as no-op:

-   PyFunction_Specialize(): always return 1 (the specialization has
    been ignored)
-   PyFunction_GetSpecializedCodes(): always return an empty list
-   PyFunction_GetSpecializedCode(): return the function code object, as
    the existing PyFunction_GET_CODE() macro

Discussion

Thread on the python-ideas mailing list: RFC: PEP: Specialized functions
with guards.

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 638 Title: Syntactic Macros Author: Mark Shannon<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Draft Type: Standards Track Content-Type: text/x-rst Created:
24-Sep-2020 Post-History: 26-Sep-2020

Abstract

This PEP adds support for syntactic macros to Python. A macro is a
compile-time function that transforms a part of the program to allow
functionality that cannot be expressed cleanly in normal library code.

The term "syntactic" means that this sort of macro operates on the
program's syntax tree. This reduces the chance of mistranslation that
can happen with text-based substitution macros, and allows the
implementation of hygienic macros.

Syntactic macros allow libraries to modify the abstract syntax tree
during compilation, providing the ability to extend the language for
specific domains without adding to complexity to the language as a
whole.

Motivation

New language features can be controversial, disruptive and sometimes
divisive. Python is now sufficiently powerful and complex, that many
proposed additions are a net loss for the language due to the additional
complexity.

Although a language change may make certain patterns easy to express, it
will have a cost. Each new feature makes the language larger, harder to
learn and harder to understand. Python was once described as Python Fits
Your Brain, but that becomes less and less true as more and more
features are added.

Because of the high cost of adding a new feature, it is very difficult
or impossible to add a feature that would benefit only some users,
regardless of how many users, or how beneficial that feature would be to
them.

The use of Python in data science and machine learning has grown very
rapidly over the last few years. However, most of the core developers of
Python do not have a background in data science or machine learning.
This makes it extremely difficult for the core developers to determine
whether a language extension for machine learning is worthwhile.

By allowing language extensions to be modular and distributable, like
libraries, domain-specific extensions can be implemented without
negatively impacting users outside of that domain. A web developer is
likely to want a very different set of extensions from a data scientist.
We need to let the community develop their own extensions.

Without some form of user-defined language extensions, there will be a
constant battle between those wanting to keep the language compact and
fitting their brains, and those wanting a new feature that suits their
domain or programming style.

Improving the expressiveness of libraries for specific domains

Many domains see repeated patterns that are difficult or impossible to
express as a library. Macros can express those patterns in a more
concise and less error-prone way.

Trialing new language features

It is possible to demonstrate potential language extensions using
macros. For example, macros would have enabled the with statement and
yield from expression to have been trialed. Doing so might well have
lead to a higher quality implementation at first release, by allowing
more testing before those features were included in the language.

It is nearly impossible to make sure that a new feature is completely
reliable before it is released; bugs relating to the with and yield from
features were still being fixed many years after they were released.

Long term stability for the bytecode interpreter

Historically, new language features have been implemented by naive
compilation of the AST into new, complex bytecode instructions. Those
bytecodes have often had their own internal flow-control, performing
operations that could, and should, have been done in the compiler.

For example, until recently flow control within the try-finally and with
statements was managed by complicated bytecodes with context-dependent
semantics. The control flow within those statements is now implemented
in the compiler, making the interpreter simpler and faster.

By implementing new features as AST transformations, the existing
compiler can generate the bytecode for a feature without having to
modify the interpreter.

A stable interpreter is necessary if we are to improve the performance
and portability of the CPython VM.

Rationale

Python is both expressive and easy to learn; it is widely recognized as
the easiest-to-learn, widely used programming language. However, it is
not the most flexible. That title belongs to lisp.

Because lisp is homoiconic, meaning that lisp programs are lisp data
structures, lisp programs can be manipulated by lisp programs. Thus much
of the language can be defined in itself.

We would like that ability in Python, without the many parentheses that
characterize lisp. Fortunately, homoiconicity is not needed for a
language to be able to manipulate itself, all that is needed is the
ability to manipulate programs after parsing, but before translation to
an executable form.

Python already has the components needed. The syntax tree of Python is
available through the ast module. All that is needed is a marker to tell
the compiler that a macro is present, and the ability for the compiler
to callback into user code to manipulate the AST.

Specification

Syntax

Lexical analysis

Any sequence of identifier characters followed by an exclamation point
(exclamation mark, UK English) will be tokenized as a MACRO_NAME.

Statement form

    macro_stmt = MACRO_NAME testlist [ "import" NAME ] [ "as"  NAME ] [ ":" NEWLINE suite ]

Expression form

    macro_expr = MACRO_NAME "(" testlist ")"

Resolving ambiguity

The statement form of a macro takes precedence, so that the code
macro_name!(x) will be parsed as a macro statement, not as an expression
statement containing a macro expression.

Semantics

Compilation

Upon encountering a macro during translation to bytecode, the code
generator will look up the macro processor registered for the macro, and
pass the AST rooted at the macro to the processor function. The returned
AST will then be substituted for the original tree.

For macros with multiple names, several trees will be passed to the
macro processor, but only one will be returned and substituted, shorting
the enclosing block of statements.

This process can be repeated, to enable macros to return AST nodes
including other macros.

The compiler will not look up a macro processor until that macro is
reached, so that inner macros do not need to have processors registered.
For example, in a switch macro, the case and default macros wouldn't
need processors registered as they would be eliminated by the switch
processor.

To enable definition of macros to be imported, the macros import! and
from! are predefined. They support the following syntax:

    "import!" dotted_name "as" name

    "from!" dotted_name "import" name [ "as" name ]

The import! macro performs a compile-time import of dotted_name to find
the macro processor, then registers it under name for the scope
currently being compiled.

The from! macro performs a compile-time import of dotted_name.name to
find the macro processor, then registers it under name (using the name
following "as", if present) for the scope currently being compiled.

Note that, since import! and from! only define the macro for the scope
in which the import is present, all uses of a macro must be preceded by
an explicit import! or from! to improve clarity.

For example, to import the macro "compile" from "my.compiler":

    from! my.compiler import compile

Defining macro processors

A macro processor is defined by a four-tuple, consisting of
(func, kind, version, additional_names):

-   func must be a callable that takes len(additional_names)+1
    arguments, all of which are abstract syntax trees, and returns a
    single abstract syntax tree.
-   kind must be one of the following:
    -   macros.STMT_MACRO: A statement macro where the body of the macro
        is indented. This is the only form allowed to have additional
        names.
    -   macros.SIBLING_MACRO: A statement macro where the body of the
        macro is the next statement in the same block. The following
        statement is moved into the macro as its body.
    -   macros.EXPR_MACRO: An expression macro.
-   version is used to track versions of macros, so that generated
    bytecodes can be correctly cached. It must be an integer.
-   additional_names are the names of the additional parts of the macro,
    and must be a tuple of strings.

    # (func, _ast.STMT_MACRO, VERSION, ())
    stmt_macro!:
        multi_statement_body

    # (func, _ast.SIBLING_MACRO, VERSION, ())
    sibling_macro!
    single_statement_body

    # (func, _ast.EXPR_MACRO, VERSION, ())
    x = expr_macro!(...)

    # (func, _ast.STMT_MACRO, VERSION, ("subsequent_macro_part",))
    multi_part_macro!:
        multi_statement_body
    subsequent_macro_part!:
        multi_statement_body

The compiler will check that the syntax used matches the declared kind.

For convenience, the decorator macro_processor is provided in the macros
module to mark a function as a macro processor:

    def macro_processor(kind, version, *additional_names):
        def deco(func):
            return func, kind, version, additional_names
        return deco

Which can be used to help declare macro processors, for example:

    @macros.macro_processor(macros.STMT_MACRO, 1_08)
    def switch(astnode):
        ...

AST extensions

Two new AST nodes will be needed to express macros, macro_stmt and
macro_expr.

    class macro_stmt(_ast.stmt):
        _fields = "name", "args", "importname", "asname", "body"

    class macro_expr(_ast.expr):
        _fields = "name", "args"

In addition, macro processors will need a means to express control flow
or side-effecting code, that produces a value. A new AST node called
stmt_expr will be added, combining a statement and an expression. This
new ast node will be a subtype of expr, but include a statement to allow
side effects. It will be compiled to bytecode by compiling the
statement, then compiling the value.

    class stmt_expr(_ast.expr):
        _fields = "stmt", "value"

Hygiene and debugging

Macro processors will often need to create new variables. Those
variables need to named in such as way as to avoid contaminating the
original code and other macros. No rules for naming will be enforced,
but to ensure hygiene and help debugging, the following naming scheme is
recommended:

-   All generated variable names should start with a $
-   Purely artificial variable names should start $$mname where mname is
    the name of the macro.
-   Variables derived from real variables should start $vname where
    vname is the name of the variable.
-   All variable names should include the line number and the column
    offset, separated by an underscore.

Examples:

-   Purely generated name: $$macro_17_0
-   Name derived from a variable for an expression macro: $var_12_5

Examples

Compile-time-checked data structures

It is common to encode tables of data in Python as large dictionaries.
However, these can be hard to maintain and error prone. Macros allow
such data to be written in a more readable format. Then, at compile
time, the data can be verified and converted to an efficient format.

For example, suppose we have a two dictionary literals mapping codes to
names, and vice versa. This is error prone, as the dictionaries may have
duplicate keys, or one table may not be the inverse of the other. A
macro could generate the two mappings from a single table and, at the
same time, verify that no duplicates are present.

    color_to_code = {
        "red": 1,
        "blue": 2,
        "green": 3,
    }

    code_to_color = {
        1: "red",
        2: "blue",
        3: "yellow", # error
    }

would become: :

    bijection! color_to_code, code_to_color:
        "red" = 1
        "blue" = 2
        "green" = 3

Domain-specific extensions

Where I see macros having real value is in specific domains, not in
general-purpose language features.

For example, parsers. Here's part of a parser definition for Python,
using macros:

    choice! single_input:
        NEWLINE
        simple_stmt
        sequence!:
            compound_stmt
            NEWLINE

Compilers

Runtime compilers, such as numba have to reconstitute the Python source,
or attempt to analyze the bytecode. It would be simpler and more
reliable for them to get the AST directly:

    from! my.jit.library import jit

    jit!
    def func():
        ...

Matching symbolic expressions

When matching something representing syntax, such a Python ast node, or
a sympy expression, it is convenient to match against the actual syntax,
not the data structure representing it. For example, a calculator could
be implemented using a domain-specific macro for matching syntax:

    from! ast_matcher import match

    def calculate(node):
        if isinstance(node, Num):
            return node.n
        match! node:
            case! a + b:
                return calculate(a) + calculate(b)
            case! a - b:
                return calculate(a) - calculate(b)
            case! a * b:
                return calculate(a) * calculate(b)
            case! a / b:
                return calculate(a) / calculate(b)

Which could be converted to:

    def calculate(node):
        if isinstance(node, Num):
            return node.n
        $$match_4_0 = node
        if isinstance($$match_4_0, _ast.Add):
            a, b = $$match_4_0.left, $$match_4_0.right
            return calculate(a) + calculate(b)
        elif isinstance($$match_4_0, _ast.Sub):
            a, b = $$match_4_0.left, $$match_4_0.right
            return calculate(a) - calculate(b)
        elif isinstance($$match_4_0, _ast.Mul):
            a, b = $$match_4_0.left, $$match_4_0.right
            return calculate(a) * calculate(b)
        elif isinstance($$match_4_0, _ast.Div):
            a, b = $$match_4_0.left, $$match_4_0.right
            return calculate(a) / calculate(b)

Zero-cost markers and annotations

Annotations, either decorators or PEP 3107 function annotations, have a
runtime cost even if they serve only as markers for checkers or as
documentation.

    @do_nothing_marker
    def foo(...):
        ...

can be replaced with the zero-cost macro:

    do_nothing_marker!:
    def foo(...):
        ...

Prototyping language extensions

Although macros would be most valuable for domain-specific extensions,
it is possible to demonstrate possible language extensions using macros.

f-strings:

The f-string f"..." could be implemented as macro as f!("..."). Not
quite as nice to read, but would still be useful for experimenting with.

Try finally statement:

    try_!:
        body
    finally!:
        closing

Would be translated roughly as:

    try:
        body
    except:
        closing
    else:
        closing

Note:

    Care must be taken to handle returns, breaks and continues
    correctly. The above code is merely illustrative.

With statement:

    with! open(filename) as fd:
        return fd.read()

The above would require handling open specially. An alternative that
would be more explicit, would be:

    with! open!(filename) as fd:
        return fd.read()

Macro definition macros

Languages that have syntactic macros usually provide a macro for
defining macros. This PEP intentionally does not do that, as it is not
yet clear what a good design would be, and we want to allow the
community to define their own macros.

One possible form could be:

    macro_def! name:
        input:
            ... # input pattern, defining meta-variables
        output:
            ... # output pattern, using meta-variables

Backwards Compatibility

This PEP is fully backwards compatible.

Performance Implications

For code that doesn't use macros, there will be no effect on
performance.

For code that does use macros and has already been compiled to bytecode,
there will be some slight overhead to check that the version of macros
used to compile the code match the imported macro processors.

For code that has not been compiled, or compiled with different versions
of the macro processors, then there would be the usual overhead of
bytecode compilation, plus any additional overhead of macro processing.

It is worth noting that the speed of source to bytecode compilation is
largely irrelevant for Python performance.

Implementation

In order to allow transformation of the AST at compile time by Python
code, all AST nodes in the compiler will have to be Python objects.

To do that efficiently, will mean making all the nodes in the _ast
module immutable, so as not degrade performance by much. They will need
to be immutable to guarantee that the AST remains a tree to avoid having
to support cyclic GC. Making them immutable means they will not have a
__dict__ attribute, making them compact.

AST nodes in the ast module will remain mutable.

Currently, all AST nodes are allocated using an arena allocator.
Changing to use the standard allocator might slow compilation down a
little, but has advantages in terms of maintenance, as much code can be
deleted.

Reference Implementation

None as yet.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 486 Title: Make the Python Launcher aware of virtual environments
Version: $Revision$ Last-Modified: $Date$ Author: Paul Moore
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 12-Feb-2015 Python-Version: 3.5 Post-History:
12-Feb-2015 Resolution:
https://mail.python.org/pipermail/python-dev/2015-February/138579.html

Abstract

The Windows installers for Python include a launcher that locates the
correct Python interpreter to run (see PEP 397). However, the launcher
is not aware of virtual environments (virtualenv[1] or PEP 405 based),
and so cannot be used to run commands from the active virtualenv.

This PEP proposes making the launcher "virtualenv aware". This means
that when run without specifying an explicit Python interpreter to use,
the launcher will use the currently active virtualenv, if any, before
falling back to the configured default Python.

Rationale

Windows users with multiple copies of Python installed need a means of
selecting which one to use. The Python launcher provides this facility
by means of a py command that can be used to run either a configured
"default" Python or a specific interpreter, by means of command line
arguments. So typical usage would be:

    # Run the Python interactive interpreter
    py

    # Execute an installed module
    py -m pip install pytest
    py -m pytest

When using virtual environments, the py launcher is unaware that a
virtualenv is active, and will continue to use the system Python. So
different command invocations are needed to run the same commands in a
virtualenv:

    # Run the Python interactive interpreter
    python

    # Execute an installed module (these could use python -m,
    # which is longer to type but is a little more similar to the
    # launcher approach)
    pip install pytest
    py.test

Having to use different commands is error-prone, and in many cases the
error is difficult to spot immediately. The PEP proposes making the py
command usable with virtual environments, so that the first form of
command can be used in all cases.

Implementation

Both virtualenv and the core venv module set an environment variable
VIRTUAL_ENV when activating a virtualenv. This PEP proposes that the
launcher checks for the VIRTUAL_ENV environment variable whenever it
would run the "default" Python interpreter for the system (i.e., when no
specific version flags such as py -2.7 are used) and if present, run the
Python interpreter for the virtualenv rather than the default system
Python.

The "default" Python interpreter referred to above is (as per PEP 397)
either the latest version of Python installed on the system, or a
version configured via the py.ini configuration file. When the user
specifies an explicit Python version on the command line, this will
always be used (as at present).

Impact on Script Launching

As well as interactive use, the launcher is used as the Windows file
association for Python scripts. In that case, a "shebang" (#!) line at
the start of the script is used to identify the interpreter to run. A
fully-qualified path can be used, or a version-specific Python (python3
or python2, or even python3.5), or the generic python, which means to
use the default interpreter.

The launcher also looks for the specific shebang line
#!/usr/bin/env python. On Unix, the env program searches for a command
on $PATH and runs the command so located. Similarly, with this shebang
line, the launcher will look for a copy of python.exe on the user's
current %PATH% and will run that copy.

As activating a virtualenv means that it is added to PATH, no special
handling is needed to run scripts with the active virtualenv -they just
need to use the #!/usr/bin/env python shebang line, exactly as on Unix.
(If there is no activated virtualenv, and no python.exe on PATH, the
launcher will look for a default Python exactly as if the shebang line
had said #!python).

Exclusions

The PEP makes no attempt to promote the use of the launcher for running
Python on Windows. Most existing documentation assumes the user of
python as the command to run Python, and (for example) pip to run an
installed Python command. This documentation is not expected to change,
and users who choose to manage their PATH environment variable can
continue to use this form. The focus of this PEP is purely on allowing
users who prefer to use the launcher when dealing with their system
Python installations, to be able to continue to do so when using virtual
environments.

Reference Implementation

A patch implementing the proposed behaviour is available at
http://bugs.python.org/issue23465

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] https://virtualenv.pypa.io/


text:
PEP: 8012 Title: The Community Governance Model Author: Łukasz Langa
<EMAIL_ADDRESS>Status: Rejected Type: Informational Topic:
Governance Content-Type: text/x-rst Created: 03-Oct-2018

PEP Rejection

PEP 8012 was rejected by a core developer vote described in PEP 8001 on
Monday, December 17, 2018.

PEP 8016 and the governance model it describes were chosen instead.

Abstract

This PEP proposes a new model of Python governance based on consensus
and voting by the Python community. This model relies on workgroups to
carry out the governance of the Python language. This governance model
works without the role of a centralized singular leader or a governing
council.

It describes how, when, and why votes are conducted for decisions
affecting the Python language. It also describes the criteria for voting
eligibility.

Should this model be adopted, it will be codified in PEP 13.

This model can be affectionately called "The Least Worst Governance
Model" by its property that while far from ideal, it's still the most
robust one compared to the others. Since avoiding issues inherent to the
other models is a paramount feature of the Community Governance Model,
we start the discussion a bit unusually: by rejecting the other models.

Rejected Models

Let's have another BDFL

This seems like a very attractive idea because it's a model we know. One
Dictator to rule us all.

Challenge: There is no other Guido

There is no other single person with the unique skillset of Guido van
Rossum. Such a person would need to have the technical, communication,
and organizational experience to lead the project successfully.
Specifically, the person would need to:

-   set and articulate a cohesive long-term vision for the project;
-   possess deep technical understanding of the runtime, the standard
    library, and the wider third-party library context;
-   negotiate and resolve contentious issues in ways acceptable to all
    parties involved;
-   have free time and possess the energy to sustain continuous
    involvement over periods of years.

Risk: Malevolent Dictator For Life

What if we got somebody who is not as well suited for the position as
our first Dictator? There are possible scenarios in which this could
lead to severe consequences.

The Dictator could gather insufficient trust due to missing technical
depth, a "close" election, inconsistent vision, poor ability to deal
with conflict or burnout, and so on. Given a controversial decision
decided by the Dictator in a specific way, a Dictator with insufficient
trust may cause a split within the project.

The Dictator setup invites lobbying concentrated on a single person.
Unless that person is immune to leverage due to wealth, health, and a
stable life situation, this poses risk of malicious actors steering the
project from behind the curtain.

Finally, the Dictator coming from a particular part of the community may
put more weight on the needs and interests of that particular part of
the user base, alienating others.

Observation: We don't actually need a Dictator

The irony of the Dictator model is that it requires an election. Better
yet, we need an election to even decide on which governance model to
use.

If we are already able solve two problems of this gravity via the
community process, why not keep using it for all subsequent decisions?

Risk: The warm and fuzzy feeling of a vague proposal

One last thing worth mentioning is that when a BDFL model is suggested,
it's easy to bypass the criticism above by not mentioning who the BDFL
should be. That way the hopeful reader can project their best
expectations and wants onto the abstract BDFL, making the idea appear
more attractive. This is a mistake.

Without naming the BDFL in the model proposal we are not talking about a
concrete model. We can avoid asking and answering the hard questions. We
can imagine our best-case scenario, a candidate we'd like to serve the
role.

Omitting a name for the BDFL also puts the Community Model at an unfair
disadvantage. We already know the good, the bad, and the ugly of our
core developer group. It's no platonic ideal, no perfect sphere with no
friction. In fact, we expect there to be a fair amount of friction and
imperfections.

Thus, to fairly assess the BDFL model proposal, dear reader, you should
imagine the worst possible person within our team as that BDFL. A
concrete human being. Imagine it's me.

Conclusion While this has been our history, without Guido, this model
does not serve the best interests of the language into the future.

Let's have a Council

This group of people roughly shares the responsibilities of a Dictator.
The group can also be called a Triumvirate, a Quorum, Elders, Steering
Committee, and so on.

Risk: Dilution and confusion

This model favors a small group, between three and five people. That way
it shares most of the criticism with the Dictator model, amplified.
Having not one but, say, three people in position of power dilutes
responsibility while still providing high risk of lobbying, insufficient
trust, or alienating parts of the community.

Risk: Internal Conflict

Additionally, having multiple people share the responsibility of
governance creates ample opportunity for internal conflict, inconsistent
long-term vision of the project, and multiplies the required continuous
time involvement by its members (it's no Quorum if they can't "reach
quorum" due to other time commitments).

Just like with a frictionless spherical BDFL, reject ideas of Councils
without considering how would it work for you if that Council consisted
of three people you find inadequate for the role. Imagine if I had two
friends.

Most importantly, just like with a Dictator, we don't need a Council. By
the time we had one, we would have already had two successful elections.
Why not keep voting?

Conclusion This model has similar risks like a Dictator, only worse.

Motivation

Now that we rejected the basics of other governance models, let's talk
why we even need a governance model on top of a loosely defined group of
committers.

Stability and Reliability We want to prevent single committers from
making wide-reaching changes that impact the future of the language or
its usability. Coherent vision and backwards compatibility are important
in any programming language, but they are doubly important for Python
which is very dynamic (e.g. has very complex backwards compatibility
implications).

Diverse Uses of Python Moreover, Python is used by a diverse group of
users, from school children through scientists to corporations with
multi-million line codebases. We want to include all our varied
audiences.

Vitality We want to avoid stagnation. Python is a mature project but it
needs to keep evolving to stay relevant, both the runtime and the
programming language. To do that, people interested in improving a
particular part of the project should be able to do so without needless
friction. But for substantial changes, we want some discourse and
reflection to ensure the changes are wise.

Rationale

Inclusive The Community Model is the most inclusive model. No single
person or a small group of people is in a distinguished position of
power over others. Contributors and any workgroups in this model are
self-selecting.

Pragmatic This model ensures no user group is put at a disadvantage due
to the interests of a single person or a small group of people.

Proven This model works. There is a number of large open-source projects
run this way (two of which, Rust and Django, are described in PEP 8002).
ECMAScript and C++ are similarly developed.

Specification

Key people and their functions

The core team

The Python project is developed by a team of core developers. While
membership is determined by presence in the "Python core" team in the
"python" organization on GitHub, contribution takes many forms:

-   committing changes to the repository;
-   reviewing pull requests by others;
-   triaging bug reports on the issue tracker;
-   discussing topics on official Python communication channels.

Some contributors are may be considered dormant, in other words they did
not contribute to the last two releases of CPython. Any dormant
contributor can at any time resume contribution.

Experts

The Python Developer's Guide lists a number of interest areas along with
names of core developers who are recognized as experts in the given
area. An expert or a sub-team of experts has the following
responsibilities:

-   responding to issues on the bug tracker triaged to the given
    interest area on a timely basis;
-   reviewing pull requests identified as belonging to the given
    interest area on a timely basis;
-   overviewing cohesive design in the evolution of the given interest
    area.

A core developer can assign and unassign themselves at will to a given
interest area. Existing experts listed for the given interest area must
be made aware of this change and have to unanimously agree to it.

If a given interest area lists multiple experts, they form a sub-team
within the core team. They are responsible for the given interest area
together.

A core developer should avoid membership as an expert in too many
interest areas at the same time. This document deliberately doesn't
specify a maximum number, it simply signals that overexertion leads to
burnout and is a risk to the project's ability to function without a
given contributor.

Moderators

There is a group of people, some of which are not core developers,
responsible for ensuring that discussions on official communication
channels adhere to the Code of Conduct. They take action in view of
violations.

Regular decision process

Primary work happens through bug tracker issues and pull requests. Core
developers should avoid pushing their changes directly to the cpython
repository, instead relying on pull requests. Approving a pull request
by a core developer allows it to be merged without further process.

Notifying relevant experts about a bug tracker issue or a pull request
is important. Reviews from experts in the given interest area are
strongly preferred, especially on pull request approvals. Failure to do
so might end up with the change being reverted by the relevant expert.

Experts are not required to listen to the firehose of GitHub and bug
tracker activity at all times. Notifying an expert explicitly during
triage or bug/pull request creation may be necessary to get their
attention.

Controversial decision process

Substantial changes in a given interest area require a PEP. This
includes:

-   Any semantic or syntactic change to the language.
-   Backwards-incompatible changes to the standard library or the C API.
-   Additions to the standard library, including substantial new
    functionality within an existing library.
-   Removing language, standard library, or C API features.

Failure to get a substantial change through the PEP process might result
with the change being reverted.

Changes that are bug fixes can be exempt from the PEP requirement. Use
your best judgement.

PEP, Enhanced

The PEP process is augmented with the following changes and
clarifications over information already present in PEP 1:

-   PEPs are not merged until the final decision is made on them; they
    are open pull requests on GitHub until that moment;
    -   to make review easier, all changes to the PEP under review
        should be made as separate commits, allowing for granular
        comparison;
-   a submitted PEP needs to identify the area of interest and relevant
    experts as the body that makes the final decision on it;
-   if the PEP author is one of the experts of the relevant area of
    interest, they must name another person from outside of that
    interest area to contribute to the final decision in their place;
-   the PEP author is responsible for gathering and integrating feedback
    on the PEP using the official communication channels, with the goal
    of building consensus;
-   all community members must be enabled to give feedback;
-   at some point, one of the named experts posts a "summary comment"
    that lays out the current state of discussion, especially major
    points of disagreement and tradeoffs; at the same time the expert
    proposes a "motion for final comment period" (FCP), along with a
    proposed disposition to either:
    -   accept;
    -   accept provisionally;
    -   reject; or
    -   defer the PEP.
-   to enter the FCP, the PEP must be signed off by all experts of the
    relevant area of interest;
-   the FCP lasts for fourteen calendar days to allow stakeholders to
    file any final objections before a decision is reached.

Very controversial PEPs

If a core contributor feels strongly against a particular PEP, during
its FCP they may raise a motion to reject it by vote. Voting details are
described below in "Voting Mechanics".

This should be a last resort and thus a rare occurrence. It splits the
core team and is a stressful event for all involved. However, the
experts filing for a FCP for a PEP should have a good sense whether a
motion to reject it by vote is likely. In such a case, care should be
taken to avoid prematurely filing for a FCP.

There is no recourse for the opposite situation, i.e. when the experts
want to reject a PEP but others would like it accepted. This ensures
that the relevant experts have the last say on what goes in. If you
really want that change, find a way to convince them.

Moderators on official communication channels enforce the Code of
Conduct first and foremost, to ensure healthy interaction between all
interested parties. Enforcement can result in a given participant being
excluded from further discussion and thus the decision process.

Revisiting deferred and rejected PEPs

If a PEP is deferred or rejected, the relevant experts should be
contacted first before another attempt at the same idea is made. If the
experts agree there is substantial evidence to justify revisiting the
idea, a pull request editing the deferred or rejected PEP can be opened.

Failure to get proper expert buy-in beforehand will likely result in
immediate rejection of a pull request on a deferred or rejected PEP.

Other Voting Situations

Nominating a new core developer

A champion nominates a person to become a new core developer by posting
on official communication channels. A vote is opened.

If any existing core developer does not feel comfortable with the
nominee receiving the commit bit, they should preferably address this
concern in the nomination thread. If there is no satisfactory
resolution, they can cast a negative vote.

In practice, nominating a person for a core developer should often meet
with surprise by others that this person is not a core developer yet. In
other words, it should be done when the candidate is already known and
trusted well enough by others. We should avoid nominations based on
potential.

Votes of no confidence

-   Removing a core developer from the core team;
-   Disbanding the experts team for a given area of interest.

Those describe a situation where a core developer is forcefully removed
from the core team or an experts team is forcefully disbanded. Hopefully
those will never have to be exercised but they are explicitly mentioned
to demonstrate how a dysfunctional area of interest can be healed.

If a core developer is removed by vote from the core team, they lose the
ability to interact with the project. It's up to the Moderators'
discretion to remove their ability to post on the bug tracker and GitHub
or just moderate their future behavior on a case-by-case basis.

If the experts team for an area of interest is disbanded, other core
developers can step up to fill the void at will. Members of the
disbanded experts team cannot self-nominate to return.

Voting Mechanics

All votes described in this document are +1/-1/0 ("Yea"/"Nay"/"Present")
recorded votes. There are no other vote values, in particular values out
of range or fractions (like +0.5) are invalid.

Votes take fourteen calendar days. The starting date is taken looking at
the timezone of the person who filed for the motion to vote. The end
date is fourteen days later Anywhere-On-Earth.

Dormant core developers as defined in "Key people and their functions"
above are not counted towards the totals if they abstain. However, they
can vote if they choose to do so and that way they count as active.
Voting is a form of contribution.

Voting is done by a commit to a private repository in the "python"
organization on GitHub. The repository is archived and publicized after
the voting period is over. The repository's name should start with
"vote-".

Changes to one's vote during the voting period is allowed. Peeking at
other developers' cast votes during the time of the vote is possible.

Every situation requires a different vote percentage:

-   PEP rejection by vote requires over 1/3rd of the non-dormant core
    developer population to explicitly vote to reject. Note that if more
    than 1/3rd of core developers decide against a PEP, this means there
    exists no super-majority of core developers who are in favor of the
    change. This strongly suggests the change should not be made in the
    shape described by the PEP.
-   New core developer nomination requires there to be no votes cast
    against it.
-   Votes of no confidence require a super-majority of at least 2/3rds
    of the non-dormant core developer population to explicitly vote in
    favor of the motion.

Omissions

This document deliberately omits listing possible areas of interest
within the project. It also does not address election and management of
Moderators which are done by the Python Software Foundation and its Code
of Conduct Working Group which can be contacted by mailing
<EMAIL_ADDRESS>
Acknowledgements

Thank you to the authors of PEP 8002 which was a helpful resource in
shaping this document.

Thank you to Alex Crichton and the Rust team for a governance model that
was a major inspiration for this document.

Copyright

This document has been placed in the public domain.


text:
PEP: 3126 Title: Remove Implicit String Concatenation Version:
$Revision$ Last-Modified: $Date$ Author: Jim J. Jewett
<EMAIL_ADDRESS>Raymond Hettinger<EMAIL_ADDRESS>Status:
Rejected Type: Standards Track Content-Type: text/x-rst Created:
29-Apr-2007 Post-History: 29-Apr-2007, 30-Apr-2007, 07-May-2007

Rejection Notice

This PEP is rejected. There wasn't enough support in favor, the feature
to be removed isn't all that harmful, and there are some use cases that
would become harder.

Abstract

Python inherited many of its parsing rules from C. While this has been
generally useful, there are some individual rules which are less useful
for python, and should be eliminated.

This PEP proposes to eliminate implicit string concatenation based only
on the adjacency of literals.

Instead of:

    "abc" "def" == "abcdef"

authors will need to be explicit, and either add the strings:

    "abc" + "def" == "abcdef"

or join them:

    "".join(["abc", "def"]) == "abcdef"

Motivation

One goal for Python 3000 should be to simplify the language by removing
unnecessary features. Implicit string concatenation should be dropped in
favor of existing techniques. This will simplify the grammar and
simplify a user's mental picture of Python. The latter is important for
letting the language "fit in your head". A large group of current users
do not even know about implicit concatenation. Of those who do know
about it, a large portion never use it or habitually avoid it. Of those
who both know about it and use it, very few could state with confidence
the implicit operator precedence and under what circumstances it is
computed when the definition is compiled versus when it is run.

History or Future

Many Python parsing rules are intentionally compatible with C. This is a
useful default, but Special Cases need to be justified based on their
utility in Python. We should no longer assume that python programmers
will also be familiar with C, so compatibility between languages should
be treated as a tie-breaker, rather than a justification.

In C, implicit concatenation is the only way to join strings without
using a (run-time) function call to store into a variable. In Python,
the strings can be joined (and still recognized as immutable) using more
standard Python idioms, such + or "".join.

Problem

Implicit String concatenation leads to tuples and lists which are
shorter than they appear; this is turn can lead to confusing, or even
silent, errors. For example, given a function which accepts several
parameters, but offers a default value for some of them:

    def f(fmt, *args):
        print fmt % args

This looks like a valid call, but isn't:

    >>> f("User %s got a message %s",
          "Bob"
          "Time for dinner")

    Traceback (most recent call last):
      File "<pyshell#8>", line 2, in <module>
        "Bob"
      File "<pyshell#3>", line 2, in f
        print fmt % args
    TypeError: not enough arguments for format string

Calls to this function can silently do the wrong thing:

    def g(arg1, arg2=None):
        ...

    # silently transformed into the possibly very different
    # g("arg1 on this linearg2 on this line", None)
    g("arg1 on this line"
      "arg2 on this line")

To quote Jason Orendorff [#Orendorff]

  Oh. I just realized this happens a lot out here. Where I work, we use
  scons, and each SConscript has a long list of filenames:

      sourceFiles = [
          'foo.c'
          'bar.c',
          #...many lines omitted...
          'q1000x.c']

  It's a common mistake to leave off a comma, and then scons complains
  that it can't find 'foo.cbar.c'. This is pretty bewildering behavior
  even if you are a Python programmer, and not everyone here is.

Solution

In Python, strings are objects and they support the __add__ operator, so
it is possible to write:

    "abc" + "def"

Because these are literals, this addition can still be optimized away by
the compiler; the CPython compiler already does so. [1]

Other existing alternatives include multiline (triple-quoted) strings,
and the join method:

    """This string
       extends across
       multiple lines, but you may want to use something like
       Textwrap.dedent
       to clear out the leading spaces
       and/or reformat.
    """

    >>> "".join(["empty", "string", "joiner"]) == "emptystringjoiner"
    True

    >>> " ".join(["space", "string", "joiner"]) == "space string joiner"
    True

    >>> "\n".join(["multiple", "lines"]) == "multiple\nlines" == (
    """multiple
    lines""")
    True

Concerns

Operator Precedence

Guido indicated[2] that this change should be handled by PEP, because
there were a few edge cases with other string operators, such as the %.
(Assuming that str % stays -- it may be eliminated in favor of PEP 3101
-- Advanced String Formatting. [3])

The resolution is to use parentheses to enforce precedence -- the same
solution that can be used today:

    # Clearest, works today, continues to work, optimization is
    # already possible.
    ("abc %s def" + "ghi") % var

    # Already works today; precedence makes the optimization more
    # difficult to recognize, but does not change the semantics.
    "abc" + "def %s ghi" % var

as opposed to:

    # Already fails because modulus (%) is higher precedence than
    # addition (+)
    ("abc %s def" + "ghi" % var)

    # Works today only because adjacency is higher precedence than
    # modulus.  This will no longer be available.
    "abc %s" "def" % var

    # So the 2-to-3 translator can automatically replace it with the
    # (already valid):
    ("abc %s" + "def") % var

Long Commands

  ... build up (what I consider to be) readable SQL queries[4]:

      rows = self.executesql("select cities.city, state, country"
                             "    from cities, venues, events, addresses"
                             "    where cities.city like %s"
                             "      and events.active = 1"
                             "      and venues.address = addresses.id"
                             "      and addresses.city = cities.id"
                             "      and events.venue = venues.id",
                             (city,))

Alternatives again include triple-quoted strings, +, and .join:

    query="""select cities.city, state, country
                 from cities, venues, events, addresses
                 where cities.city like %s
                   and events.active = 1"
                   and venues.address = addresses.id
                   and addresses.city = cities.id
                   and events.venue = venues.id"""

    query=( "select cities.city, state, country"
          + "    from cities, venues, events, addresses"
          + "    where cities.city like %s"
          + "      and events.active = 1"
          + "      and venues.address = addresses.id"
          + "      and addresses.city = cities.id"
          + "      and events.venue = venues.id"
          )

    query="\n".join(["select cities.city, state, country",
                     "    from cities, venues, events, addresses",
                     "    where cities.city like %s",
                     "      and events.active = 1",
                     "      and venues.address = addresses.id",
                     "      and addresses.city = cities.id",
                     "      and events.venue = venues.id"])

    # And yes, you *could* inline any of the above querystrings
    # the same way the original was inlined.
    rows = self.executesql(query, (city,))

Regular Expressions

Complex regular expressions are sometimes stated in terms of several
implicitly concatenated strings with each regex component on a different
line and followed by a comment. The plus operator can be inserted here
but it does make the regex harder to read. One alternative is to use the
re.VERBOSE option. Another alternative is to build-up the regex with a
series of += lines:

    # Existing idiom which relies on implicit concatenation
    r = ('a{20}'  # Twenty A's
         'b{5}'   # Followed by Five B's
         )

    # Mechanical replacement
    r = ('a{20}'  +# Twenty A's
         'b{5}'   # Followed by Five B's
         )

    # already works today
    r = '''a{20}  # Twenty A's
           b{5}   # Followed by Five B's
        '''                 # Compiled with the re.VERBOSE flag

    # already works today
    r = 'a{20}'   # Twenty A's
    r += 'b{5}'   # Followed by Five B's

Internationalization

Some internationalization tools -- notably xgettext -- have already been
special-cased for implicit concatenation, but not for Python's explicit
concatenation.[5]

These tools will fail to extract the (already legal):

    _("some string" +
      " and more of it")

but often have a special case for:

    _("some string"
      " and more of it")

It should also be possible to just use an overly long line (xgettext
limits messages to 2048 characters[6], which is less than Python's
enforced limit) or triple-quoted strings, but these solutions sacrifice
some readability in the code:

    # Lines over a certain length are unpleasant.
    _("some string and more of it")

    # Changing whitespace is not ideal.
    _("""Some string
         and more of it""")
    _("""Some string
    and more of it""")
    _("Some string \
    and more of it")

I do not see a good short-term resolution for this.

Transition

The proposed new constructs are already legal in current Python, and can
be used immediately.

The 2 to 3 translator can be made to mechanically change:

    "str1" "str2"
    ("line1"  #comment
     "line2")

into:

    ("str1" + "str2")
    ("line1"   +#comments
     "line2")

If users want to use one of the other idioms, they can; as these idioms
are all already legal in python 2, the edits can be made to the original
source, rather than patching up the translator.

Open Issues

Is there a better way to support external text extraction tools, or at
least xgettext[7] in particular?

References

Copyright

  This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Reminder: Py3k PEPs due by April, Hettinger, van Rossum
https://mail.python.org/pipermail/python-3000/2007-April/006563.html

[2] Reminder: Py3k PEPs due by April, Hettinger, van Rossum
https://mail.python.org/pipermail/python-3000/2007-April/006563.html

[3] ps to question Re: Need help completing ABC pep, van Rossum
https://mail.python.org/pipermail/python-3000/2007-April/006737.html

[4] (email Subject) PEP 30XZ: Simplified Parsing, Skip,
https://mail.python.org/pipermail/python-3000/2007-May/007261.html

[5] (email Subject) PEP 30XZ: Simplified Parsing
https://mail.python.org/pipermail/python-3000/2007-May/007305.html

[6] Unix man page for xgettext -- Notes section
http://www.scit.wlv.ac.uk/cgi-bin/mansec?1+xgettext

[7] GNU gettext manual http://www.gnu.org/software/gettext/


text:
PEP: 704 Title: Require virtual environments by default for package
installers Author: Pradyun Gedam<EMAIL_ADDRESS>Sponsor: Brett
Cannon<EMAIL_ADDRESS>PEP-Delegate: Paul Moore<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/22846 Status: Withdrawn
Type: Standards Track Topic: Packaging Content-Type: text/x-rst Created:
16-Jan-2023 Post-History: 16-Jan-2023

Abstract

This PEP recommends that package installers like pip require a virtual
environment by default on Python 3.13+.

PEP Withdrawal

During discussion of this PEP, it became clear that changes to pip's UX
are not controlled by PEPs as proposed. It also became clear that a
significant number of users rely on being able to mix managed-by-pip
dependencies with managed-by-some-other-tool dependencies (most
prominently, when using Conda).

Further, a significant subset of the benefits of the proposed change are
achievable via PEP 668 (accepted and implemented at the time of
writing). It provides redistributors of the Python interpreter control
over whether users should be required to use a virtual environment, what
the messaging presented to the user is and how the rollout of that
change should happen for their users.

Since enforcement of virtual environments with pip was the primary focus
of this PEP, it felt more appropriate to withdraw this PEP than to
refocus it on a different topic.

A future PEP to resolve the virtual environment naming convention
question/issues would still be appropriate, but it's worth starting any
such effort as a fresh PEP focused on the benefits of such a convention,
rather than on the enforcement of it.

Motivation

Python virtual environments are an essential part of the development
workflow for Python. However, they require extra effort since they are
an opt-in feature, and requires users to either:

-   take explicit steps to activate/deactivate a virtual environment
-   use {<path-to-venv>}/{<bin-path>}/{<executable>} to run files

For new users, things will seemingly work correctly when virtual
environments are not used -— until they don't. Further, activating a
virtual environment uses slightly different syntax and mechanisms on
different platforms. This complicates the introduction to virtual
environments, since now information and context about how/why they are
useful needs to explained to justify adding additional steps to the
workflow.

It also creates a scope for mistakes, since users need to remember to
activate the virtual environment before running an installer like pip or
configure those installers to error out. On certain Linux distributions,
forgetting to do so can result in the installer modifying files that are
owned by the operating system (which is partially mitigated by PEP 668
for distributions that opt-in to marking their environments
accordingly).

Rationale

Changing the default behaviour of installers like pip to require a
virtual environment to be active would:

-   make it easier for new users to get started with Python (since
    there's a consistent experience and virtual environments are
    understood as a thing you're required to use)
-   reduce the scope for accidental installation issues for all users by
    default (by explicitly flagging when you're not using a virtual
    environment).

Setting up a convention of placing the virtual environment in-tree in a
directory named .venv removes a decision point for common workflows and
creates a clear convention within the ecosystem.

Specification

Requiring a virtual environment by default

When a user runs an installer without an active virtual environment, the
installer SHOULD print an error message and exit with a non-zero exit
code.

The error message SHOULD inform the user that a virtual environment is
required, SHOULD provide shell-specific instructions on how to create
and activate a virtual environment named .venv, and SHOULD provide a
link to a documentation page that explains how to create and activate a
virtual environment.

See Implementation Notes for more details.

Opting out of virtual environments

The installer SHOULD also provide a explicit opt-in to disable this
requirement, allowing an end user to use it outside of a virtual
environment. If the installer does not provide this functionality, it
SHOULD mention this in the error message and documentation page.

Consistent timeline for the change

Installers MAY choose to implement this default behaviour on any Python
versions, but SHOULD implement it on Python 3.13 or newer.

Backwards Compatibility

This PEP is backwards incompatible with workflows where users are using
installers outside of virtual environments. Such users will be prompted
with an error message and will need to either:

-   explicitly opt-in to running the installer outside of a virtual
    environment, or
-   create and use a virtual environment

Users who are already using virtual environments will not be affected by
this change.

Workflow tools (which manage virtual environments for the user, under
the hood) should be unaffected, since they should already be using a
virtual environment for running the installer.

Security Implications

This PEP does not introduce any new security implications.

How to Teach This

This PEP requires that new users create and use a virtual environment to
get started with using Python packages. This is, however, a best
practice, as demonstrated by the section on "basics of how to install
Python packages" in the Python Packaging User Guide, which explains
how/what virtual environments are before discussing using pip.

Reference Implementation

There is no reference implementation for this PEP. However, the proposed
behaviour is largely already implemented in pip and can be activated by
setting the PIP_REQUIRE_VENV environment variable to 1. (Leaving it
unset results in the proposed opt-in behaviour of not requiring a
virtual environment for installation.)

Implementation Notes

Detecting an active virtual environment

As discussed in PEP 668 <668#backwards-compatibility>, the logic for
robustly detecting a virtual environment is something like:

    def is_virtual_environment():
        return sys.base_prefix != sys.prefix or hasattr(sys, "real_prefix")

Documentation on using a virtual environment

Package installers are expected to provide a link to a documentation
page in the error message.

Ideally, such a documentation page would explain what virtual
environments are, why they are required, and how to create and activate
a virtual environment using venv. It should include instructions for the
most common shells and platforms.

Such a documentation page should be made available in the Python
Packaging User Guide to reduce duplicated effort across installers for
covering this topic.

Rejected Ideas

Do not specify a name for the virtual environment directory

Using a consistent name for the virtual environment directory is
important for a few reasons:

1.  It makes it easier for users to find the virtual environment
    directory, and to activate it.
2.  It removes a decision point for new users, since they do not need to
    decide on a name for the virtual environment directory.
3.  It creates a clear convention within the ecosystem, which makes it
    easier for users to find documentation.
4.  It ensures consistency across different tools, so that differences
    in the error messages do not confuse users.

Use a different name for the virtual environment directory

Functionally, the directory name does not matter much as long as there
is a single consistent suggestion.

The name .venv was picked since it:

1.  does not conflict with any valid Python import name
2.  does not conflict venv module in the standard library
3.  has pre-existing usage in the Python community
4.  has support for auto-detection in common text editors
5.  can be typed without modifier keys on common keyboard layouts

Do not couple tooling behaviour with a Python version

This PEP creates a coupling between the behaviour of installers and the
Python version.

This is already a rollout mechanism being used for behaviour changes in
the installation tooling. For example, pip on Python 3.11 will use
importlib.metadata instead of pkg_resources for parsing/fetching package
metadata, and sysconfig instead of distutils.sysconfig for getting the
paths to unpack wheels into.

The difference with those cases is that they're supposed to be largely
transparent to end users. This PEP is proposing a behaviour change that
is not transparent to end users, and requires them to take action.

The primary benefit of this is that it allows for redistributors to
adapt their tooling in time for the new Python version and provides a
clear and consistent point for change across the ecosystem. It also puts
a clear deadline on when the default behaviour will consistently require
a virtual environment by default (once Python 3.12 goes end-of-life).

The primary issue with this approach is that it enforces a behaviour
change on users when they upgrade to a new Python version, which can
hamper the adoption of a new Python version. However, this is a
migration/upgrade for existing users and it is a common expectation that
some changes will be needed for migration/upgrades.

The author of this PEP believes that the benefits of applying this
consistently throughout the ecosystem with a deadline outweigh the
drawbacks of enforcing a best-practice on users when they upgrade.

Open Issues

None.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.