text:
PEP: 587 Title: Python Initialization Configuration Author: Victor
Stinner<EMAIL_ADDRESS>Alyssa Coghlan<EMAIL_ADDRESS>BDFL-Delegate: Thomas Wouters<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 27-Mar-2019 Python-Version: 3.8

Abstract

Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.

It becomes possible to read the configuration and then override some
computed parameters before it is applied. It also becomes possible to
completely override how Python computes the module search paths
(sys.path).

The new Isolated Configuration provides sane default values to isolate
Python from the system. For example, to embed Python into an
application. Using the environment are now opt-in options, rather than
an opt-out options. For example, environment variables, command line
arguments and global configuration variables are ignored by default.

Building a customized Python which behaves as the regular Python becomes
easier using the new Py_RunMain() function. Moreover, using the Python
Configuration, PyConfig.argv arguments are now parsed the same way the
regular Python parses command line arguments, and PyConfig.xoptions are
handled as -X opt command line options.

This extracts a subset of the API design from the PEP 432 development
and refactoring work that is now considered sufficiently stable to make
public (allowing 3rd party embedding applications access to the same
configuration APIs that the native CPython CLI is now using).

Rationale

Python is highly configurable but its configuration evolved organically.
The initialization configuration is scattered all around the code using
different ways to set them: global configuration variables (ex:
Py_IsolatedFlag), environment variables (ex: PYTHONPATH), command line
arguments (ex: -b), configuration files (ex: pyvenv.cfg), function calls
(ex: Py_SetProgramName()). A straightforward and reliable way to
configure Python is needed.

Some configuration parameters are not accessible from the C API, or not
easily. For example, there is no API to override the default values of
sys.executable.

Some options like PYTHONPATH can only be set using an environment
variable which has a side effect on Python child processes if not unset
properly.

Some options also depends on other options: see Priority and Rules.
Python 3.7 API does not provide a consistent view of the overall
configuration.

The C API of Python 3.7 Initialization takes wchar_t* strings as input
whereas the Python filesystem encoding is set during the initialization
which can lead to mojibake.

Python 3.7 APIs like Py_Initialize() aborts the process on memory
allocation failure which is not convenient when Python is embedded.
Moreover, Py_Main() could exit directly the process rather than
returning an exit code. Proposed new API reports the error or exit code
to the caller which can decide how to handle it.

Implementing the PEP 540 (UTF-8 Mode) and the new -X dev correctly was
almost impossible in Python 3.6. The code base has been deeply reworked
in Python 3.7 and then in Python 3.8 to read the configuration into a
structure with no side effect. It becomes possible to clear the
configuration (release memory) and read again the configuration if the
encoding changed . It is required to implement properly the UTF-8 which
changes the encoding using -X utf8 command line option. Internally,
bytes argv strings are decoded from the filesystem encoding. The -X dev
changes the memory allocator (behaves as PYTHONMALLOC=debug), whereas it
was not possible to change the memory allocation while parsing the
command line arguments. The new design of the internal implementation
not only allowed to implement properly -X utf8 and -X dev, it also
allows to change the Python behavior way more easily, especially for
corner cases like that, and ensure that the configuration remains
consistent: see Priority and Rules.

This PEP is a partial implementation of PEP 432 which is the overall
design. New fields can be added later to PyConfig structure to finish
the implementation of the PEP 432 (e.g. by adding a new partial
initialization API which allows to configure Python using Python objects
to finish the full initialization). However, those features are omitted
from this PEP as even the native CPython CLI doesn't work that way - the
public API proposal in this PEP is limited to features which have
already been implemented and adopted as private APIs for us in the
native CPython CLI.

Python Initialization C API

This PEP proposes to add the following new structures and functions.

New structures:

-   PyConfig
-   PyPreConfig
-   PyStatus
-   PyWideStringList

New functions:

-   PyConfig_Clear(config)
-   PyConfig_InitIsolatedConfig()
-   PyConfig_InitPythonConfig()
-   PyConfig_Read(config)
-   PyConfig_SetArgv(config, argc, argv)
-   PyConfig_SetBytesArgv(config, argc, argv)
-   PyConfig_SetBytesString(config, config_str, str)
-   PyConfig_SetString(config, config_str, str)
-   PyPreConfig_InitIsolatedConfig(preconfig)
-   PyPreConfig_InitPythonConfig(preconfig)
-   PyStatus_Error(err_msg)
-   PyStatus_Exception(status)
-   PyStatus_Exit(exitcode)
-   PyStatus_IsError(status)
-   PyStatus_IsExit(status)
-   PyStatus_NoMemory()
-   PyStatus_Ok()
-   PyWideStringList_Append(list, item)
-   PyWideStringList_Insert(list, index, item)
-   Py_BytesMain(argc, argv)
-   Py_ExitStatusException(status)
-   Py_InitializeFromConfig(config)
-   Py_PreInitialize(preconfig)
-   Py_PreInitializeFromArgs(preconfig, argc, argv)
-   Py_PreInitializeFromBytesArgs(preconfig, argc, argv)
-   Py_RunMain()

This PEP also adds _PyRuntimeState.preconfig (PyPreConfig type) and
PyInterpreterState.config (PyConfig type) fields to these internal
structures. PyInterpreterState.config becomes the new reference
configuration, replacing global configuration variables and other
private variables.

PyWideStringList

PyWideStringList is a list of wchar_t* strings.

PyWideStringList structure fields:

-   length (Py_ssize_t)
-   items (wchar_t**)

Methods:

-   PyStatus PyWideStringList_Append(PyWideStringList *list, const wchar_t *item):
    Append item to list.
-   PyStatus PyWideStringList_Insert(PyWideStringList *list, Py_ssize_t index, const wchar_t *item):
    Insert item into list at index. If index is greater than list
    length, just append item to list.

If length is non-zero, items must be non-NULL and all strings must be
non-NULL.

PyStatus

PyStatus is a structure to store the status of an initialization
function: success, error or exit. For an error, it can store the C
function name which created the error.

Example:

    PyStatus alloc(void **ptr, size_t size)
    {
        *ptr = PyMem_RawMalloc(size);
        if (*ptr == NULL) {
            return PyStatus_NoMemory();
        }
        return PyStatus_Ok();
    }

    int main(int argc, char **argv)
    {
        void *ptr;
        PyStatus status = alloc(&ptr, 16);
        if (PyStatus_Exception(status)) {
            Py_ExitStatusException(status);
        }
        PyMem_Free(ptr);
        return 0;
    }

PyStatus fields:

-   exitcode (int): Argument passed to exit().
-   err_msg (const char*): Error message.
-   func (const char *): Name of the function which created an error,
    can be NULL.
-   private _type field: for internal usage only.

Functions to create a status:

-   PyStatus_Ok(): Success.
-   PyStatus_Error(err_msg): Initialization error with a message.
-   PyStatus_NoMemory(): Memory allocation failure (out of memory).
-   PyStatus_Exit(exitcode): Exit Python with the specified exit code.

Functions to handle a status:

-   PyStatus_Exception(status): Is the result an error or an exit? If
    true, the exception must be handled; by calling
    Py_ExitStatusException(status) for example.
-   PyStatus_IsError(status): Is the result an error?
-   PyStatus_IsExit(status): Is the result an exit?
-   Py_ExitStatusException(status): Call exit(exitcode) if status is an
    exit. Print the error messageand exit with a non-zero exit code if
    status is an error. Must only be called if
    PyStatus_Exception(status) is true.

Note

Internally, Python uses macros which set PyStatus.func, whereas
functions to create a status set func to NULL.

Preinitialization with PyPreConfig

The PyPreConfig structure is used to preinitialize Python:

-   Set the Python memory allocator
-   Configure the LC_CTYPE locale
-   Set the UTF-8 mode

Example using the preinitialization to enable the UTF-8 Mode:

    PyStatus status;
    PyPreConfig preconfig;

    PyPreConfig_InitPythonConfig(&preconfig);

    preconfig.utf8_mode = 1;

    status = Py_PreInitialize(&preconfig);
    if (PyStatus_Exception(status)) {
        Py_ExitStatusException(status);
    }

    /* at this point, Python will speak UTF-8 */

    Py_Initialize();
    /* ... use Python API here ... */
    Py_Finalize();

Function to initialize a preconfiguration:

-   PyStatus PyPreConfig_InitIsolatedConfig(PyPreConfig *preconfig)
-   PyStatus PyPreConfig_InitPythonConfig(PyPreConfig *preconfig)

Functions to preinitialize Python:

-   PyStatus Py_PreInitialize(const PyPreConfig *preconfig)
-   PyStatus Py_PreInitializeFromBytesArgs(const PyPreConfig *preconfig, int argc, char * const *argv)
-   PyStatus Py_PreInitializeFromArgs(const PyPreConfig *preconfig, int argc, wchar_t * const * argv)

The caller is responsible to handle exceptions (error or exit) using
PyStatus_Exception() and Py_ExitStatusException().

For Python Configuration (PyPreConfig_InitPythonConfig()), if Python is
initialized with command line arguments, the command line arguments must
also be passed to preinitialize Python, since they have an effect on the
pre-configuration like encodings. For example, the -X utf8 command line
option enables the UTF-8 Mode.

PyPreConfig fields:

-   allocator (int): Name of the memory allocator (ex:
    PYMEM_ALLOCATOR_MALLOC). Valid values:
    -   PYMEM_ALLOCATOR_NOT_SET (0): don't change memory allocators (use
        defaults)
    -   PYMEM_ALLOCATOR_DEFAULT (1): default memory allocators
    -   PYMEM_ALLOCATOR_DEBUG (2): default memory allocators with debug
        hooks
    -   PYMEM_ALLOCATOR_MALLOC (3): force usage of malloc()
    -   PYMEM_ALLOCATOR_MALLOC_DEBUG (4): force usage of malloc() with
        debug hooks
    -   PYMEM_ALLOCATOR_PYMALLOC (5): Python "pymalloc" allocator
    -   PYMEM_ALLOCATOR_PYMALLOC_DEBUG (6): Python "pymalloc" allocator
        with debug hooks
    -   Note: PYMEM_ALLOCATOR_PYMALLOC and
        PYMEM_ALLOCATOR_PYMALLOC_DEBUG are not supported if Python is
        configured using --without-pymalloc
-   configure_locale (int): Set the LC_CTYPE locale to the user
    preferred locale? If equals to 0, set coerce_c_locale and
    coerce_c_locale_warn to 0.
-   coerce_c_locale (int): If equals to 2, coerce the C locale; if
    equals to 1, read the LC_CTYPE locale to decide if it should be
    coerced.
-   coerce_c_locale_warn (int): If non-zero, emit a warning if the C
    locale is coerced.
-   dev_mode (int): See PyConfig.dev_mode.
-   isolated (int): See PyConfig.isolated.
-   legacy_windows_fs_encoding (int, Windows only): If non-zero, disable
    UTF-8 Mode, set the Python filesystem encoding to mbcs, set the
    filesystem error handler to replace.
-   parse_argv (int): If non-zero, Py_PreInitializeFromArgs() and
    Py_PreInitializeFromBytesArgs() parse their argv argument the same
    way the regular Python parses command line arguments: see Command
    Line Arguments.
-   use_environment (int): See PyConfig.use_environment.
-   utf8_mode (int): If non-zero, enable the UTF-8 mode.

The legacy_windows_fs_encoding field is only available on Windows.
#ifdef MS_WINDOWS macro can be used for Windows specific code.

PyPreConfig private fields, for internal use only:

-   _config_init (int): Function used to initialize PyConfig, used for
    preinitialization.

PyMem_SetAllocator() can be called after Py_PreInitialize() and before
Py_InitializeFromConfig() to install a custom memory allocator. It can
be called before Py_PreInitialize() if allocator is set to
PYMEM_ALLOCATOR_NOT_SET (default value).

Python memory allocation functions like PyMem_RawMalloc() must not be
used before Python preinitialization, whereas calling directly malloc()
and free() is always safe. Py_DecodeLocale() must not be called before
the preinitialization.

Initialization with PyConfig

The PyConfig structure contains most parameters to configure Python.

Example setting the program name:

    void init_python(void)
    {
        PyStatus status;

        PyConfig config;
        PyConfig_InitPythonConfig(&config);

        /* Set the program name. Implicitly preinitialize Python. */
        status = PyConfig_SetString(&config, &config.program_name,
                                    L"/path/to/my_program");
        if (PyStatus_Exception(status)) {
            goto fail;
        }

        status = Py_InitializeFromConfig(&config);
        if (PyStatus_Exception(status)) {
            goto fail;
        }
        PyConfig_Clear(&config);
        return;

    fail:
        PyConfig_Clear(&config);
        Py_ExitStatusException(status);
    }

PyConfig methods:

-   void PyConfig_InitPythonConfig(PyConfig *config) Initialize
    configuration with Python Configuration.
-   void PyConfig_InitIsolatedConfig(PyConfig *config): Initialize
    configuration with Isolated Configuration.
-   PyStatus PyConfig_SetString(PyConfig *config, wchar_t * const *config_str, const wchar_t *str):
    Copy the wide character string str into *config_str. Preinitialize
    Python if needed.
-   PyStatus PyConfig_SetBytesString(PyConfig *config, wchar_t * const *config_str, const char *str):
    Decode str using Py_DecodeLocale() and set the result into
    *config_str. Preinitialize Python if needed.
-   PyStatus PyConfig_SetArgv(PyConfig *config, int argc, wchar_t * const *argv):
    Set command line arguments from wide character strings.
    Preinitialize Python if needed.
-   PyStatus PyConfig_SetBytesArgv(PyConfig *config, int argc, char * const *argv):
    Set command line arguments: decode bytes using Py_DecodeLocale().
    Preinitialize Python if needed.
-   PyStatus PyConfig_Read(PyConfig *config): Read all Python
    configuration. Fields which are already initialized are left
    unchanged. Preinitialize Python if needed.
-   void PyConfig_Clear(PyConfig *config): Release configuration memory.

Most PyConfig methods preinitialize Python if needed. In that case, the
Python preinitialization configuration in based on the PyConfig. If
configuration fields which are in common with PyPreConfig are tuned,
they must be set before calling a PyConfig method:

-   dev_mode
-   isolated
-   parse_argv
-   use_environment

Moreover, if PyConfig_SetArgv() or PyConfig_SetBytesArgv() is used, this
method must be called first, before other methods, since the
preinitialization configuration depends on command line arguments (if
parse_argv is non-zero).

Functions to initialize Python:

-   PyStatus Py_InitializeFromConfig(const PyConfig *config): Initialize
    Python from config configuration.

The caller of these methods and functions is responsible to handle
exceptions (error or exit) using PyStatus_Exception() and
Py_ExitStatusException().

PyConfig fields:

-   argv (PyWideStringList): Command line arguments, sys.argv. See
    parse_argv to parse argv the same way the regular Python parses
    Python command line arguments. If argv is empty, an empty string is
    added to ensure that sys.argv always exists and is never empty.
-   base_exec_prefix (wchar_t*): sys.base_exec_prefix.
-   base_prefix (wchar_t*): sys.base_prefix.
-   buffered_stdio (int): If equals to 0, enable unbuffered mode, making
    the stdout and stderr streams unbuffered.
-   bytes_warning (int): If equals to 1, issue a warning when comparing
    bytes or bytearray with str, or comparing bytes with int. If equal
    or greater to 2, raise a BytesWarning exception.
-   check_hash_pycs_mode (wchar_t*): --check-hash-based-pycs command
    line option value (see PEP 552). Valid values: always, never and
    default. The default value is default.
-   configure_c_stdio (int): If non-zero, configure C standard streams
    (stdio, stdout, stdout). For example, set their mode to O_BINARY on
    Windows.
-   dev_mode (int): Development mode
-   dump_refs (int): If non-zero, dump all objects which are still alive
    at exit. Require a special Python build with Py_REF_DEBUG macro
    defined.
-   exec_prefix (wchar_t*): sys.exec_prefix.
-   executable (wchar_t*): sys.executable.
-   faulthandler (int): If non-zero, call faulthandler.enable().
-   filesystem_encoding (wchar_t*): Filesystem encoding,
    sys.getfilesystemencoding().
-   filesystem_errors (wchar_t*): Filesystem encoding errors,
    sys.getfilesystemencodeerrors().
-   use_hash_seed (int), hash_seed (unsigned long): Randomized hash
    function seed.
-   home (wchar_t*): Python home directory.
-   import_time (int): If non-zero, profile import time.
-   inspect (int): Enter interactive mode after executing a script or a
    command.
-   install_signal_handlers (int): Install signal handlers?
-   interactive (int): Interactive mode.
-   isolated (int): If greater than 0, enable isolated mode:
    -   sys.path contains neither the script's directory (computed from
        argv[0] or the current directory) nor the user's site-packages
        directory.
    -   Python REPL doesn't import readline nor enable default readline
        configuration on interactive prompts.
    -   Set use_environment and user_site_directory to 0.
-   legacy_windows_stdio (int, Windows only): If non-zero, use io.FileIO
    instead of WindowsConsoleIO for sys.stdin, sys.stdout and
    sys.stderr.
-   malloc_stats (int): If non-zero, dump statistics on pymalloc memory
    allocator at exit. The option is ignored if Python is built using
    --without-pymalloc.
-   pythonpath_env (wchar_t*): Module search paths as a string separated
    by DELIM (usually : character). Initialized from PYTHONPATH
    environment variable value by default.
-   module_search_paths_set (int), module_search_paths
    (PyWideStringList): sys.path. If module_search_paths_set is equal to
    0, the module_search_paths is overridden by the function computing
    the Path Configuration.
-   optimization_level (int): Compilation optimization level:
    -   0: Peephole optimizer (and __debug__ is set to True)
    -   1: Remove assertions, set __debug__ to False
    -   2: Strip docstrings
-   parse_argv (int): If non-zero, parse argv the same way the regular
    Python command line arguments, and strip Python arguments from argv:
    see Command Line Arguments.
-   parser_debug (int): If non-zero, turn on parser debugging output
    (for expert only, depending on compilation options).
-   pathconfig_warnings (int): If equal to 0, suppress warnings when
    computing the path configuration (Unix only, Windows does not log
    any warning). Otherwise, warnings are written into stderr.
-   prefix (wchar_t*): sys.prefix.
-   program_name (wchar_t*): Program name.
-   pycache_prefix (wchar_t*): .pyc cache prefix.
-   quiet (int): Quiet mode. For example, don't display the copyright
    and version messages even in interactive mode.
-   run_command (wchar_t*): python3 -c COMMAND argument.
-   run_filename (wchar_t*): python3 FILENAME argument.
-   run_module (wchar_t*): python3 -m MODULE argument.
-   show_alloc_count (int): Show allocation counts at exit? Need a
    special Python build with COUNT_ALLOCS macro defined.
-   show_ref_count (int): Show total reference count at exit? Need a
    debug build of Python (Py_REF_DEBUG macro should be defined).
-   site_import (int): Import the site module at startup?
-   skip_source_first_line (int): Skip the first line of the source?
-   stdio_encoding (wchar_t*), stdio_errors (wchar_t*): Encoding and
    encoding errors of sys.stdin, sys.stdout and sys.stderr.
-   tracemalloc (int): If non-zero, call tracemalloc.start(value).
-   user_site_directory (int): If non-zero, add user site directory to
    sys.path.
-   verbose (int): If non-zero, enable verbose mode.
-   warnoptions (PyWideStringList): Options of the warnings module to
    build warnings filters.
-   write_bytecode (int): If non-zero, write .pyc files.
-   xoptions (PyWideStringList): sys._xoptions.

The legacy_windows_stdio field is only available on Windows.
#ifdef MS_WINDOWS macro can be used for Windows specific code.

If parse_argv is non-zero, argv arguments are parsed the same way the
regular Python parses command line arguments, and Python arguments are
stripped from argv: see Command Line Arguments.

The xoptions options are parsed to set other options: see -X Options.

PyConfig private fields, for internal use only:

-   _config_init (int): Function used to initialize PyConfig, used for
    preinitialization.
-   _install_importlib (int): Install importlib?
-   _init_main (int): If equal to 0, stop Python initialization before
    the "main" phase (see PEP 432).

More complete example modifying the default configuration, read the
configuration, and then override some parameters:

    PyStatus init_python(const char *program_name)
    {
        PyStatus status;

        PyConfig config;
        PyConfig_InitPythonConfig(&config);

        /* Set the program name before reading the configuration
           (decode byte string from the locale encoding).

           Implicitly preinitialize Python. */
        status = PyConfig_SetBytesString(&config, &config.program_name,
                                      program_name);
        if (PyStatus_Exception(status)) {
            goto done;
        }

        /* Read all configuration at once */
        status = PyConfig_Read(&config);
        if (PyStatus_Exception(status)) {
            goto done;
        }

        /* Append our custom search path to sys.path */
        status = PyWideStringList_Append(&config.module_search_paths,
                                      L"/path/to/more/modules");
        if (PyStatus_Exception(status)) {
            goto done;
        }

        /* Override executable computed by PyConfig_Read() */
        status = PyConfig_SetString(&config, &config.executable,
                                    L"/path/to/my_executable");
        if (PyStatus_Exception(status)) {
            goto done;
        }

        status = Py_InitializeFromConfig(&config);

    done:
        PyConfig_Clear(&config);
        return status;
    }

Note

PyImport_FrozenModules, PyImport_AppendInittab() and
PyImport_ExtendInittab() functions are still relevant and continue to
work as previously. They should be set or called after Python
preinitialization and before the Python initialization.

Isolated Configuration

PyPreConfig_InitIsolatedConfig() and PyConfig_InitIsolatedConfig()
functions create a configuration to isolate Python from the system. For
example, to embed Python into an application.

This configuration ignores global configuration variables, environments
variables and command line arguments (argv is not parsed). The C
standard streams (ex: stdout) and the LC_CTYPE locale are left unchanged
by default.

Configuration files are still used with this configuration. Set the Path
Configuration ("output fields") to ignore these configuration files and
avoid the function computing the default path configuration.

Python Configuration

PyPreConfig_InitPythonConfig() and PyConfig_InitPythonConfig() functions
create a configuration to build a customized Python which behaves as the
regular Python.

Environments variables and command line arguments are used to configure
Python, whereas global configuration variables are ignored.

This function enables C locale coercion (PEP 538) and UTF-8 Mode (PEP
540) depending on the LC_CTYPE locale, PYTHONUTF8 and
PYTHONCOERCECLOCALE environment variables.

Example of customized Python always running in isolated mode:

    int main(int argc, char **argv)
    {
        PyStatus status;

        PyConfig config;
        PyConfig_InitPythonConfig(&config);

        config.isolated = 1;

        /* Decode command line arguments.
           Implicitly preinitialize Python (in isolated mode). */
        status = PyConfig_SetBytesArgv(&config, argc, argv);
        if (PyStatus_Exception(status)) {
            goto fail;
        }

        status = Py_InitializeFromConfig(&config);
        if (PyStatus_Exception(status)) {
            goto fail;
        }
        PyConfig_Clear(&config);

        return Py_RunMain();

    fail:
        PyConfig_Clear(&config);
        if (PyStatus_IsExit(status)) {
            return status.exitcode;
        }
        /* Display the error message and exit the process with
           non-zero exit code */
        Py_ExitStatusException(status);
    }

This example is a basic implementation of the "System Python Executable"
discussed in PEP 432.

Path Configuration

PyConfig contains multiple fields for the path configuration:

-   Path configuration input fields:
    -   home
    -   pythonpath_env
    -   pathconfig_warnings
-   Path configuration output fields:
    -   exec_prefix
    -   executable
    -   prefix
    -   module_search_paths_set, module_search_paths

If at least one "output field" is not set, Python computes the path
configuration to fill unset fields. If module_search_paths_set is equal
to 0, module_search_paths is overridden and module_search_paths_set is
set to 1.

It is possible to completely ignore the function computing the default
path configuration by setting explicitly all path configuration output
fields listed above. A string is considered as set even if it is
non-empty. module_search_paths is considered as set if
module_search_paths_set is set to 1. In this case, path configuration
input fields are ignored as well.

Set pathconfig_warnings to 0 to suppress warnings when computing the
path configuration (Unix only, Windows does not log any warning).

If base_prefix or base_exec_prefix fields are not set, they inherit
their value from prefix and exec_prefix respectively.

Py_RunMain() and Py_Main() modify sys.path:

-   If run_filename is set and is a directory which contains a
    __main__.py script, prepend run_filename to sys.path.
-   If isolated is zero:
    -   If run_module is set, prepend the current directory to sys.path.
        Do nothing if the current directory cannot be read.
    -   If run_filename is set, prepends the directory of the filename
        to sys.path.
    -   Otherwise, prepends an empty string to sys.path.

If site_import is non-zero, sys.path can be modified by the site module.
If user_site_directory is non-zero and the user's site-package directory
exists, the site module appends the user's site-package directory to
sys.path.

See also Configuration Files used by the path configuration.

Py_BytesMain()

Python 3.7 provides a high-level Py_Main() function which requires to
pass command line arguments as wchar_t* strings. It is non-trivial to
use the correct encoding to decode bytes. Python has its own set of
issues with C locale coercion and UTF-8 Mode.

This PEP adds a new Py_BytesMain() function which takes command line
arguments as bytes:

    int Py_BytesMain(int argc, char **argv)

Py_RunMain()

The new Py_RunMain() function executes the command
(PyConfig.run_command), the script (PyConfig.run_filename) or the module
(PyConfig.run_module) specified on the command line or in the
configuration, and then finalizes Python. It returns an exit status that
can be passed to the exit() function. :

    int Py_RunMain(void);

See Python Configuration for an example of customized Python always
running in isolated mode using Py_RunMain().

Multi-Phase Initialization Private Provisional API

This section is a private provisional API introducing multi-phase
initialization, the core feature of the PEP 432:

-   "Core" initialization phase, "bare minimum Python":
    -   Builtin types;
    -   Builtin exceptions;
    -   Builtin and frozen modules;
    -   The sys module is only partially initialized (ex: sys.path
        doesn't exist yet);
-   "Main" initialization phase, Python is fully initialized:
    -   Install and configure importlib;
    -   Apply the Path Configuration;
    -   Install signal handlers;
    -   Finish sys module initialization (ex: create sys.stdout and
        sys.path);
    -   Enable optional features like faulthandler and tracemalloc;
    -   Import the site module;
    -   etc.

Private provisional API:

-   PyConfig._init_main: if set to 0, Py_InitializeFromConfig() stops at
    the "Core" initialization phase.
-   PyStatus _Py_InitializeMain(void): move to the "Main" initialization
    phase, finish the Python initialization.

No module is imported during the "Core" phase and the importlib module
is not configured: the Path Configuration is only applied during the
"Main" phase. It may allow to customize Python in Python to override or
tune the Path Configuration, maybe install a custom sys.meta_path
importer or an import hook, etc.

It may become possible to compute the Path Configuration in Python,
after the Core phase and before the Main phase, which is one of the PEP
432 motivation.

The "Core" phase is not properly defined: what should be and what should
not be available at this phase is not specified yet. The API is marked
as private and provisional: the API can be modified or even be removed
anytime until a proper public API is designed.

Example running Python code between "Core" and "Main" initialization
phases:

    void init_python(void)
    {
        PyStatus status;

        PyConfig config;
        PyConfig_InitPythonConfig(&config);

        config._init_main = 0;

        /* ... customize 'config' configuration ... */

        status = Py_InitializeFromConfig(&config);
        PyConfig_Clear(&config);
        if (PyStatus_Exception(status)) {
            Py_ExitStatusException(status);
        }

        /* Use sys.stderr because sys.stdout is only created
           by _Py_InitializeMain() */
        int res = PyRun_SimpleString(
            "import sys; "
            "print('Run Python code before _Py_InitializeMain', "
                   "file=sys.stderr)");
        if (res < 0) {
            exit(1);
        }

        /* ... put more configuration code here ... */

        status = _Py_InitializeMain();
        if (PyStatus_Exception(status)) {
            Py_ExitStatusException(status);
        }
    }

Backwards Compatibility

This PEP only adds a new API: it leaves the existing API unchanged and
has no impact on the backwards compatibility.

The Python 3.7 Py_Initialize() function now disable the C locale
coercion (PEP 538) and the UTF-8 Mode (PEP 540) by default to prevent
mojibake. The new API using the Python Configuration is needed to enable
them automatically.

Annexes

Comparison of Python and Isolated Configurations

Differences between PyPreConfig_InitPythonConfig() and
PyPreConfig_InitIsolatedConfig():

+----------------------------+--------+----------+
| PyPreConfig                | Python | Isolated |
+============================+========+==========+
| coerce_c_locale_warn       |   -1   |   0      |
+----------------------------+--------+----------+
| coerce_c_locale            |   -1   |   0      |
+----------------------------+--------+----------+
| configure_locale           |   1    |   0      |
+----------------------------+--------+----------+
| dev_mode                   |   -1   |   0      |
+----------------------------+--------+----------+
| isolated                   |   0    |   1      |
+----------------------------+--------+----------+
| legacy_windows_fs_encoding |   -1   |   0      |
+----------------------------+--------+----------+
| use_environment            |   0    |   0      |
+----------------------------+--------+----------+
| parse_argv                 |   1    |   0      |
+----------------------------+--------+----------+
| utf8_mode                  |   -1   |   0      |
+----------------------------+--------+----------+

Differences between PyConfig_InitPythonConfig() and
PyConfig_InitIsolatedConfig():

+-------------------------+--------+----------+
| PyConfig                | Python | Isolated |
+=========================+========+==========+
| configure_c_stdio       |   1    |   0      |
+-------------------------+--------+----------+
| install_signal_handlers |   1    |   0      |
+-------------------------+--------+----------+
| isolated                |   0    |   1      |
+-------------------------+--------+----------+
| parse_argv              |   1    |   0      |
+-------------------------+--------+----------+
| pathconfig_warnings     |   1    |   0      |
+-------------------------+--------+----------+
| use_environment         |   1    |   0      |
+-------------------------+--------+----------+
| user_site_directory     |   1    |   0      |
+-------------------------+--------+----------+

Priority and Rules

Priority of configuration parameters, highest to lowest:

-   PyConfig
-   PyPreConfig
-   Configuration files
-   Command line options
-   Environment variables
-   Global configuration variables

Priority of warning options, highest to lowest:

-   PyConfig.warnoptions
-   PySys_AddWarnOption()
-   PyConfig.bytes_warning (add "error::BytesWarning" filter if greater
    than 1, add "default::BytesWarning filter if equals to 1)
-   -W opt command line argument
-   PYTHONWARNINGS environment variable
-   PyConfig.dev_mode (add "default" filter)

Rules on PyConfig parameters:

-   If isolated is non-zero, use_environment and user_site_directory are
    set to 0.
-   If dev_mode is non-zero, allocator is set to "debug", faulthandler
    is set to 1, and "default" filter is added to warnoptions. But the
    PYTHONMALLOC environment variable has the priority over dev_mode to
    set the memory allocator.
-   If base_prefix is not set, it inherits prefix value.
-   If base_exec_prefix is not set, it inherits exec_prefix value.
-   If the python._pth configuration file is present, isolated is set to
    1 and site_import is set to 0; but site_import is set to 1 if
    python._pth contains import site.

Rules on PyConfig and PyPreConfig parameters:

-   If PyPreConfig.legacy_windows_fs_encoding is non-zero, set
    PyPreConfig.utf8_mode to 0, set PyConfig.filesystem_encoding to
    mbcs, and set PyConfig.filesystem_errors to replace.

Configuration Files

Python configuration files used by the Path Configuration:

-   pyvenv.cfg
-   python._pth (Windows only)
-   pybuilddir.txt (Unix only)

Global Configuration Variables

Global configuration variables mapped to PyPreConfig fields:

  Variable                         Field
  -------------------------------- ----------------------------
  Py_IgnoreEnvironmentFlag         use_environment (NOT)
  Py_IsolatedFlag                  isolated
  Py_LegacyWindowsFSEncodingFlag   legacy_windows_fs_encoding
  Py_UTF8Mode                      utf8_mode

(NOT) means that the PyPreConfig value is the opposite of the global
configuration variable value. Py_LegacyWindowsFSEncodingFlag is only
available on Windows.

Global configuration variables mapped to PyConfig fields:

  Variable                               Field
  -------------------------------------- ---------------------------
  Py_BytesWarningFlag                    bytes_warning
  Py_DebugFlag                           parser_debug
  Py_DontWriteBytecodeFlag               write_bytecode (NOT)
  Py_FileSystemDefaultEncodeErrors       filesystem_errors
  Py_FileSystemDefaultEncoding           filesystem_encoding
  Py_FrozenFlag                          pathconfig_warnings (NOT)
  Py_HasFileSystemDefaultEncoding        filesystem_encoding
  Py_HashRandomizationFlag               use_hash_seed, hash_seed
  Py_IgnoreEnvironmentFlag               use_environment (NOT)
  Py_InspectFlag                         inspect
  Py_InteractiveFlag                     interactive
  Py_IsolatedFlag                        isolated
  Py_LegacyWindowsStdioFlag              legacy_windows_stdio
  Py_NoSiteFlag                          site_import (NOT)
  Py_NoUserSiteDirectory                 user_site_directory (NOT)
  Py_OptimizeFlag                        optimization_level
  Py_QuietFlag                           quiet
  Py_UnbufferedStdioFlag                 buffered_stdio (NOT)
  Py_VerboseFlag                         verbose
  _Py_HasFileSystemDefaultEncodeErrors   filesystem_errors

(NOT) means that the PyConfig value is the opposite of the global
configuration variable value. Py_LegacyWindowsStdioFlag is only
available on Windows.

Command Line Arguments

Usage:

    python3 [options]
    python3 [options] -c COMMAND
    python3 [options] -m MODULE
    python3 [options] SCRIPT

Command line options mapped to pseudo-action on PyPreConfig fields:

  Option          PyConfig field
  --------------- ---------------------
  -E              use_environment = 0
  -I              isolated = 1
  -X dev          dev_mode = 1
  -X utf8         utf8_mode = 1
  -X utf8=VALUE   utf8_mode = VALUE

Command line options mapped to pseudo-action on PyConfig fields:

  Option                         PyConfig field
  ------------------------------ --------------------------------------------
  -b                             bytes_warning++
  -B                             write_bytecode = 0
  -c COMMAND                     run_command = COMMAND
  --check-hash-based-pycs=MODE   check_hash_pycs_mode = MODE
  -d                             parser_debug++
  -E                             use_environment = 0
  -i                             inspect++ and interactive++
  -I                             isolated = 1
  -m MODULE                      run_module = MODULE
  -O                             optimization_level++
  -q                             quiet++
  -R                             use_hash_seed = 0
  -s                             user_site_directory = 0
  -S                             site_import
  -t                             ignored (kept for backwards compatibility)
  -u                             buffered_stdio = 0
  -v                             verbose++
  -W WARNING                     add WARNING to warnoptions
  -x                             skip_source_first_line = 1
  -X OPTION                      add OPTION to xoptions

-h, -? and -V options are handled without PyConfig.

-X Options

-X options mapped to pseudo-action on PyConfig fields:

  Option                     PyConfig field
  -------------------------- -------------------------
  -X dev                     dev_mode = 1
  -X faulthandler            faulthandler = 1
  -X importtime              import_time = 1
  -X pycache_prefix=PREFIX   pycache_prefix = PREFIX
  -X showalloccount          show_alloc_count = 1
  -X showrefcount            show_ref_count = 1
  -X tracemalloc=N           tracemalloc = N

Environment Variables

Environment variables mapped to PyPreConfig fields:

  Variable                        PyPreConfig field
  ------------------------------- ---------------------------------------
  PYTHONCOERCECLOCALE             coerce_c_locale, coerce_c_locale_warn
  PYTHONDEVMODE                   dev_mode
  PYTHONLEGACYWINDOWSFSENCODING   legacy_windows_fs_encoding
  PYTHONMALLOC                    allocator
  PYTHONUTF8                      utf8_mode

Environment variables mapped to PyConfig fields:

  Variable                   PyConfig field
  -------------------------- ------------------------------
  PYTHONDEBUG                parser_debug
  PYTHONDEVMODE              dev_mode
  PYTHONDONTWRITEBYTECODE    write_bytecode
  PYTHONDUMPREFS             dump_refs
  PYTHONEXECUTABLE           program_name
  PYTHONFAULTHANDLER         faulthandler
  PYTHONHASHSEED             use_hash_seed, hash_seed
  PYTHONHOME                 home
  PYTHONINSPECT              inspect
  PYTHONIOENCODING           stdio_encoding, stdio_errors
  PYTHONLEGACYWINDOWSSTDIO   legacy_windows_stdio
  PYTHONMALLOCSTATS          malloc_stats
  PYTHONNOUSERSITE           user_site_directory
  PYTHONOPTIMIZE             optimization_level
  PYTHONPATH                 pythonpath_env
  PYTHONPROFILEIMPORTTIME    import_time
  PYTHONPYCACHEPREFIX,       pycache_prefix
  PYTHONTRACEMALLOC          tracemalloc
  PYTHONUNBUFFERED           buffered_stdio
  PYTHONVERBOSE              verbose
  PYTHONWARNINGS             warnoptions

PYTHONLEGACYWINDOWSFSENCODING and PYTHONLEGACYWINDOWSSTDIO are specific
to Windows.

Default Python Configuration

PyPreConfig_InitPythonConfig():

-   allocator = PYMEM_ALLOCATOR_NOT_SET
-   coerce_c_locale_warn = -1
-   coerce_c_locale = -1
-   configure_locale = 1
-   dev_mode = -1
-   isolated = 0
-   legacy_windows_fs_encoding = -1
-   use_environment = 1
-   utf8_mode = -1

PyConfig_InitPythonConfig():

-   argv = []
-   base_exec_prefix = NULL
-   base_prefix = NULL
-   buffered_stdio = 1
-   bytes_warning = 0
-   check_hash_pycs_mode = NULL
-   configure_c_stdio = 1
-   dev_mode = 0
-   dump_refs = 0
-   exec_prefix = NULL
-   executable = NULL
-   faulthandler = 0
-   filesystem_encoding = NULL
-   filesystem_errors = NULL
-   hash_seed = 0
-   home = NULL
-   import_time = 0
-   inspect = 0
-   install_signal_handlers = 1
-   interactive = 0
-   isolated = 0
-   malloc_stats = 0
-   module_search_path_env = NULL
-   module_search_paths = []
-   optimization_level = 0
-   parse_argv = 1
-   parser_debug = 0
-   pathconfig_warnings = 1
-   prefix = NULL
-   program_name = NULL
-   pycache_prefix = NULL
-   quiet = 0
-   run_command = NULL
-   run_filename = NULL
-   run_module = NULL
-   show_alloc_count = 0
-   show_ref_count = 0
-   site_import = 1
-   skip_source_first_line = 0
-   stdio_encoding = NULL
-   stdio_errors = NULL
-   tracemalloc = 0
-   use_environment = 1
-   use_hash_seed = 0
-   user_site_directory = 1
-   verbose = 0
-   warnoptions = []
-   write_bytecode = 1
-   xoptions = []
-   _init_main = 1
-   _install_importlib = 1

Default Isolated Configuration

PyPreConfig_InitIsolatedConfig():

-   allocator = PYMEM_ALLOCATOR_NOT_SET
-   coerce_c_locale_warn = 0
-   coerce_c_locale = 0
-   configure_locale = 0
-   dev_mode = 0
-   isolated = 1
-   legacy_windows_fs_encoding = 0
-   use_environment = 0
-   utf8_mode = 0

PyConfig_InitIsolatedConfig():

-   argv = []
-   base_exec_prefix = NULL
-   base_prefix = NULL
-   buffered_stdio = 1
-   bytes_warning = 0
-   check_hash_pycs_mode = NULL
-   configure_c_stdio = 0
-   dev_mode = 0
-   dump_refs = 0
-   exec_prefix = NULL
-   executable = NULL
-   faulthandler = 0
-   filesystem_encoding = NULL
-   filesystem_errors = NULL
-   hash_seed = 0
-   home = NULL
-   import_time = 0
-   inspect = 0
-   install_signal_handlers = 0
-   interactive = 0
-   isolated = 1
-   malloc_stats = 0
-   module_search_path_env = NULL
-   module_search_paths = []
-   optimization_level = 0
-   parse_argv = 0
-   parser_debug = 0
-   pathconfig_warnings = 0
-   prefix = NULL
-   program_name = NULL
-   pycache_prefix = NULL
-   quiet = 0
-   run_command = NULL
-   run_filename = NULL
-   run_module = NULL
-   show_alloc_count = 0
-   show_ref_count = 0
-   site_import = 1
-   skip_source_first_line = 0
-   stdio_encoding = NULL
-   stdio_errors = NULL
-   tracemalloc = 0
-   use_environment = 0
-   use_hash_seed = 0
-   user_site_directory = 0
-   verbose = 0
-   warnoptions = []
-   write_bytecode = 1
-   xoptions = []
-   _init_main = 1
-   _install_importlib = 1

Python 3.7 API

Python 3.7 has 4 functions in its C API to initialize and finalize
Python:

-   Py_Initialize(), Py_InitializeEx(): initialize Python
-   Py_Finalize(), Py_FinalizeEx(): finalize Python

Python 3.7 can be configured using Global Configuration Variables,
Environment Variables, and the following functions:

-   PyImport_AppendInittab()
-   PyImport_ExtendInittab()
-   PyMem_SetAllocator()
-   PyMem_SetupDebugHooks()
-   PyObject_SetArenaAllocator()
-   Py_SetPath()
-   Py_SetProgramName()
-   Py_SetPythonHome()
-   Py_SetStandardStreamEncoding()
-   PySys_AddWarnOption()
-   PySys_AddXOption()
-   PySys_ResetWarnOptions()

There is also a high-level Py_Main() function and PyImport_FrozenModules
variable which can be overridden.

See Initialization, Finalization, and Threads documentation.

Python Issues

Issues that will be fixed by this PEP, directly or indirectly:

-   bpo-1195571: "simple callback system for Py_FatalError"
-   bpo-11320: "Usage of API method Py_SetPath causes errors in
    Py_Initialize() (Posix only)"
-   bpo-13533: "Would like Py_Initialize to play friendly with host app"
-   bpo-14956: "custom PYTHONPATH may break apps embedding Python"
-   bpo-19983: "When interrupted during startup, Python should not call
    abort() but exit()"
-   bpo-22213: "Make pyvenv style virtual environments easier to
    configure when embedding Python".
-   bpo-29778: "_Py_CheckPython3 uses uninitialized dllpath when
    embedder sets module path with Py_SetPath"
-   bpo-30560: "Add Py_SetFatalErrorAbortFunc: Allow embedding program
    to handle fatal errors".
-   bpo-31745: "Overloading "Py_GetPath" does not work"
-   bpo-32573: "All sys attributes (.argv, ...) should exist in embedded
    environments".
-   bpo-33135: "Define field prefixes for the various config structs".
    The PEP now defines well how warnings options are handled.
-   bpo-34725: "Py_GetProgramFullPath() odd behaviour in Windows"
-   bpo-36204: "Deprecate calling Py_Main() after Py_Initialize()? Add
    Py_InitializeFromArgv()?"

Issues of the PEP implementation:

-   bpo-16961: "No regression tests for -E and individual environment
    vars"
-   bpo-20361: "-W command line options and PYTHONWARNINGS environmental
    variable should not override -b / -bb command line options"
-   bpo-26122: "Isolated mode doesn't ignore PYTHONHASHSEED"
-   bpo-29818: "Py_SetStandardStreamEncoding leads to a memory error in
    debug mode"
-   bpo-31845: "PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no
    effect"
-   bpo-32030: "PEP 432: Rewrite Py_Main()"
-   bpo-32124: "Document functions safe to be called before
    Py_Initialize()"
-   bpo-33042: "New 3.7 startup sequence crashes PyInstaller"
-   bpo-33932: "Calling Py_Initialize() twice now triggers a fatal error
    (Python 3.7)"
-   bpo-34008: "Do we support calling Py_Main() after Py_Initialize()?"
-   bpo-34170: "Py_Initialize(): computing path configuration must not
    have side effect (PEP 432)"
-   bpo-34589: "Py_Initialize() and Py_Main() should not enable C locale
    coercion"
-   bpo-34639: "PYTHONCOERCECLOCALE is ignored when using -E or -I
    option"
-   bpo-36142: "Add a new _PyPreConfig step to Python initialization to
    setup memory allocator and encodings"
-   bpo-36202: "Calling Py_DecodeLocale() before _PyPreConfig_Write()
    can produce mojibake"
-   bpo-36301: "Add _Py_PreInitialize() function"
-   bpo-36443: "Disable coerce_c_locale and utf8_mode by default in
    _PyPreConfig?"
-   bpo-36444: "Python initialization: remove _PyMainInterpreterConfig"
-   bpo-36471: "PEP 432, PEP 587: Add _Py_RunMain()"
-   bpo-36763: "PEP 587: Rework initialization API to prepare second
    version of the PEP"
-   bpo-36775: "Rework filesystem codec implementation"
-   bpo-36900: "Use _PyCoreConfig rather than global configuration
    variables"

Issues related to this PEP:

-   bpo-12598: "Move sys variable initialization from import.c to
    sysmodule.c"
-   bpo-15577: "Real argc and argv in embedded interpreter"
-   bpo-16202: "sys.path[0] security issues"
-   bpo-18309: "Make python slightly more relocatable"
-   bpo-22257: "PEP 432: Redesign the interpreter startup sequence"
-   bpo-25631: "Segmentation fault with invalid Unicode command-line
    arguments in embedded Python"
-   bpo-26007: "Support embedding the standard library in an executable"
-   bpo-31210: "Can not import modules if sys.prefix contains DELIM".
-   bpo-31349: "Embedded initialization ignores Py_SetProgramName()"
-   bpo-33919: "Expose _PyCoreConfig structure to Python"
-   bpo-35173: "Re-use already existing functionality to allow Python
    2.7.x (both embedded and standalone) to locate the module path
    according to the shared library"

Discussions

-   May 2019:
    -   [Python-Dev] PEP 587 "Python Initialization Configuration"
        version 4
    -   [Python-Dev] RFC: PEP 587 "Python Initialization Configuration":
        3rd version
    -   Study on applications embedding Python
    -   [Python-Dev] RFC: PEP 587 "Python Initialization Configuration":
        2nd version
-   March 2019:
    -   [Python-Dev] PEP 587: Python Initialization Configuration
    -   [Python-Dev] New Python Initialization API
-   February 2019:
    -   Adding char* based APIs for Unix
-   July-August 2018:
    -   July: [Python-Dev] New _Py_InitializeFromConfig() function (PEP
        432)
    -   August: [Python-Dev] New _Py_InitializeFromConfig() function
        (PEP 432)

Version History

-   Version 5:
    -   Rename PyInitError to PyStatus
    -   Rename PyInitError_Failed() to PyStatus_Exception()
    -   Rename Py_ExitInitError() to Py_ExitStatusException()
    -   Add PyPreConfig._config_init private field.
    -   Fix Python Configuration default values: isolated=0 and
        use_environment=1, instead of -1.
    -   Add "Multi-Phase Initialization Private Provisional API" and
        "Discussions" sections
-   Version 4:
    -   Introduce "Python Configuration" and "Isolated Configuration"
        which are well better defined. Replace all macros with
        functions.
    -   Replace PyPreConfig_INIT and PyConfig_INIT macros with
        functions:
        -   PyPreConfig_InitIsolatedConfig(),
            PyConfig_InitIsolatedConfig()
        -   PyPreConfig_InitPythonConfig(), PyConfig_InitPythonConfig()
    -   PyPreConfig no longer uses dynamic memory, the allocator field
        type becomes an int, add configure_locale and parse_argv field.
    -   PyConfig: rename module_search_path_env to pythonpath_env,
        rename use_module_search_paths to module_search_paths_set,
        remove program and dll_path.
    -   Replace Py_INIT_xxx() macros with PyInitError_xxx() functions.
    -   Remove the "Constant PyConfig" section. Remove
        Py_InitializeFromArgs() and Py_InitializeFromBytesArgs()
        functions.
-   Version 3:
    -   PyConfig: Add configure_c_stdio and parse_argv; rename _frozen
        to pathconfig_warnings.
    -   Rename functions using bytes strings and wide character strings.
        For example, Py_PreInitializeFromWideArgs() becomes
        Py_PreInitializeFromArgs(), and PyConfig_SetArgv() becomes
        PyConfig_SetBytesArgv().
    -   Add PyWideStringList_Insert() function.
    -   New "Path configuration", "Isolate Python", "Python Issues" and
        "Version History" sections.
    -   PyConfig_SetString() and PyConfig_SetBytesString() now requires
        the configuration as the first argument.
    -   Rename Py_UnixMain() to Py_BytesMain()
-   Version 2: Add PyConfig methods (ex: PyConfig_Read()), add
    PyWideStringList_Append(), rename PyWideCharList to
    PyWideStringList.
-   Version 1: Initial version.

Acceptance

PEP 587 was accepted by Thomas Wouters on May 26, 2019.

Copyright

This document has been placed in the public domain.

authors:
Alyssa Coghlan
Victor Stinner


text:
PEP: 671 Title: Syntax for late-bound function argument defaults Author:
Chris Angelico<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Draft Type: Standards Track Content-Type: text/x-rst Created:
24-Oct-2021 Python-Version: 3.12 Post-History: 24-Oct-2021, 01-Dec-2021

Abstract

Function parameters can have default values which are calculated during
function definition and saved. This proposal introduces a new form of
argument default, defined by an expression to be evaluated at function
call time.

Motivation

Optional function arguments, if omitted, often have some sort of logical
default value. When this value depends on other arguments, or needs to
be reevaluated each function call, there is currently no clean way to
state this in the function header.

Currently-legal idioms for this include:

    # Very common: Use None and replace it in the function
    def bisect_right(a, x, lo=0, hi=None, *, key=None):
        if hi is None:
            hi = len(a)

    # Also well known: Use a unique custom sentinel object
    _USE_GLOBAL_DEFAULT = object()
    def connect(timeout=_USE_GLOBAL_DEFAULT):
        if timeout is _USE_GLOBAL_DEFAULT:
            timeout = default_timeout

    # Unusual: Accept star-args and then validate
    def add_item(item, *optional_target):
        if not optional_target:
            target = []
        else:
            target = optional_target[0]

In each form, help(function) fails to show the true default value. Each
one has additional problems, too; using None is only valid if None is
not itself a plausible function parameter, the custom sentinel requires
a global constant; and use of star-args implies that more than one
argument could be given.

Specification

Function default arguments can be defined using the new => notation:

    def bisect_right(a, x, lo=0, hi=>len(a), *, key=None):
    def connect(timeout=>default_timeout):
    def add_item(item, target=>[]):
    def format_time(fmt, time_t=>time.time()):

The expression is saved in its source code form for the purpose of
inspection, and bytecode to evaluate it is prepended to the function's
body.

Notably, the expression is evaluated in the function's run-time scope,
NOT the scope in which the function was defined (as are early-bound
defaults). This allows the expression to refer to other arguments.

Multiple late-bound arguments are evaluated from left to right, and can
refer to previously-defined values. Order is defined by the function,
regardless of the order in which keyword arguments may be passed.

  def prevref(word="foo", a=>len(word), b=>a//2): # Valid def
  selfref(spam=>spam): # UnboundLocalError def spaminate(sausage=>eggs +
  1, eggs=>sausage - 1): # Confusing, don't do this def
  frob(n=>len(items), items=[]): # See below

Evaluation order is left-to-right; however, implementations MAY choose
to do so in two separate passes, first for all passed arguments and
early-bound defaults, and then a second pass for late-bound defaults.
Otherwise, all arguments will be assigned strictly left-to-right.

Rejected choices of spelling

While this document specifies a single syntax name=>expression,
alternate spellings are similarly plausible. The following spellings
were considered:

    def bisect(a, hi=>len(a)):
    def bisect(a, hi:=len(a)):
    def bisect(a, hi?=len(a)):
    def bisect(a, @hi=len(a)):

Since default arguments behave largely the same whether they're early or
late bound, the chosen syntax hi=>len(a) is deliberately similar to the
existing early-bind syntax.

One reason for rejection of the := syntax is its behaviour with
annotations. Annotations go before the default, so in all syntax
options, it must be unambiguous (both to the human and the parser)
whether this is an annotation, a default, or both. The alternate syntax
target:=expr runs the risk of being misinterpreted as target:int=expr
with the annotation omitted in error, and may thus mask bugs. The chosen
syntax target=>expr does not have this problem.

How to Teach This

Early-bound default arguments should always be taught first, as they are
the simpler and more efficient way to evaluate arguments. Building on
them, late bound arguments are broadly equivalent to code at the top of
the function:

    def add_item(item, target=>[]):

    # Equivalent pseudocode:
    def add_item(item, target=<OPTIONAL>):
        if target was omitted: target = []

A simple rule of thumb is: "target=expression" is evaluated when the
function is defined, and "target=>expression" is evaluated when the
function is called. Either way, if the argument is provided at call
time, the default is ignored. While this does not completely explain all
the subtleties, it is sufficient to cover the important distinction here
(and the fact that they are similar).

Interaction with other proposals

PEP 661 attempts to solve one of the same problems as this does. It
seeks to improve the documentation of sentinel values in default
arguments, where this proposal seeks to remove the need for sentinels in
many common cases. PEP 661 is able to improve documentation in
arbitrarily complicated functions (it cites traceback.print_exception as
its primary motivation, which has two arguments which must
both-or-neither be specified); on the other hand, many of the common
cases would no longer need sentinels if the true default could be
defined by the function. Additionally, dedicated sentinel objects can be
used as dictionary lookup keys, where PEP 671 does not apply.

A generic system for deferred evaluation has been proposed at times (not
to be confused with PEP 563 and PEP 649 which are specific to
annotations). While it may seem, on the surface, that late-bound
argument defaults are of a similar nature, they are in fact unrelated
and orthogonal ideas, and both could be of value to the language. The
acceptance or rejection of this proposal would not affect the viability
of a deferred evaluation proposal, and vice versa. (A key difference
between generalized deferred evaluation and argument defaults is that
argument defaults will always and only be evaluated as the function
begins executing, whereas deferred expressions would only be realized
upon reference.)

Implementation details

The following relates to the reference implementation, and is not
necessarily part of the specification.

Argument defaults (positional or keyword) have both their values, as
already retained, and an extra piece of information. For positional
arguments, the extras are stored in a tuple in __defaults_extra__, and
for keyword-only, a dict in __kwdefaults_extra__. If this attribute is
None, it is equivalent to having None for every argument default.

For each parameter with a late-bound default, the special value Ellipsis
is stored as the value placeholder, and the corresponding extra
information needs to be queried. If it is None, then the default is
indeed the value Ellipsis; otherwise, it is a descriptive string and the
true value is calculated as the function begins.

When a parameter with a late-bound default is omitted, the function will
begin with the parameter unbound. The function begins by testing for
each parameter with a late-bound default using a new opcode
QUERY_FAST/QUERY_DEREF, and if unbound, evaluates the original
expression. This opcode (available only for fast locals and closure
variables) pushes True onto the stack if the given local has a value,
and False if not - meaning that it pushes False if LOAD_FAST or
LOAD_DEREF would raise UnboundLocalError, and True if it would succeed.

Out-of-order variable references are permitted as long as the referent
has a value from an argument or early-bound default.

Costs

When no late-bound argument defaults are used, the following costs
should be all that are incurred:

-   Function objects require two additional pointers, which will be NULL
-   Compiling code and constructing functions have additional flag
    checks
-   Using Ellipsis as a default value will require run-time verification
    to see if late-bound defaults exist.

These costs are expected to be minimal (on 64-bit Linux, this increases
all function objects from 152 bytes to 168), with virtually no run-time
cost when late-bound defaults are not used.

Backward incompatibility

Where late-bound defaults are not used, behaviour should be identical.
Care should be taken if Ellipsis is found, as it may not represent
itself, but beyond that, tools should see existing code unchanged.

References

https://github.com/rosuav/cpython/tree/pep-671

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

license: Public Domain


text:
PEP: 6 Title: Bug Fix Releases Author: Aahz<EMAIL_ADDRESS>Anthony Baxter<EMAIL_ADDRESS>Status: Superseded Type:
Process Content-Type: text/x-rst Created: 15-Mar-2001 Post-History:
15-Mar-2001, 18-Apr-2001, 19-Aug-2004

Note

This PEP is obsolete. The current release policy is documented in the
devguide. See also PEP 101 for mechanics of the release process.

Abstract

Python has historically had only a single fork of development, with
releases having the combined purpose of adding new features and
delivering bug fixes (these kinds of releases will be referred to as
"major releases"). This PEP describes how to fork off maintenance, or
bug fix, releases of old versions for the primary purpose of fixing
bugs.

This PEP is not, repeat NOT, a guarantee of the existence of bug fix
releases; it only specifies a procedure to be followed if bug fix
releases are desired by enough of the Python community willing to do the
work.

Motivation

With the move to SourceForge, Python development has accelerated. There
is a sentiment among part of the community that there was too much
acceleration, and many people are uncomfortable with upgrading to new
versions to get bug fixes when so many features have been added,
sometimes late in the development cycle.

One solution for this issue is to maintain the previous major release,
providing bug fixes until the next major release. This should make
Python more attractive for enterprise development, where Python may need
to be installed on hundreds or thousands of machines.

Prohibitions

Bug fix releases are required to adhere to the following restrictions:

1.  There must be zero syntax changes. All .pyc and .pyo files must work
    (no regeneration needed) with all bugfix releases forked off from a
    major release.
2.  There must be zero pickle changes.
3.  There must be no incompatible C API changes. All extensions must
    continue to work without recompiling in all bugfix releases in the
    same fork as a major release.

Breaking any of these prohibitions requires a BDFL proclamation (and a
prominent warning in the release notes).

Not-Quite-Prohibitions

Where possible, bug fix releases should also:

1.  Have no new features. The purpose of a bug fix release is to fix
    bugs, not add the latest and greatest whizzo feature from the HEAD
    of the CVS root.
2.  Be a painless upgrade. Users should feel confident that an upgrade
    from 2.x.y to 2.x.(y+1) will not break their running systems. This
    means that, unless it is necessary to fix a bug, the standard
    library should not change behavior, or worse yet, APIs.

Applicability of Prohibitions

The above prohibitions and not-quite-prohibitions apply both for a final
release to a bugfix release (for instance, 2.4 to 2.4.1) and for one
bugfix release to the next in a series (for instance 2.4.1 to 2.4.2).

Following the prohibitions listed in this PEP should help keep the
community happy that a bug fix release is a painless and safe upgrade.

Helping the Bug Fix Releases Happen

Here's a few pointers on helping the bug fix release process along.

1.  Backport bug fixes. If you fix a bug, and it seems appropriate, port
    it to the CVS branch for the current bug fix release. If you're
    unwilling or unable to backport it yourself, make a note in the
    commit message, with words like 'Bugfix candidate' or 'Backport
    candidate'.
2.  If you're not sure, ask. Ask the person managing the current bug fix
    releases if they think a particular fix is appropriate.
3.  If there's a particular bug you'd particularly like fixed in a bug
    fix release, jump up and down and try to get it done. Do not wait
    until 48 hours before a bug fix release is due, and then start
    asking for bug fixes to be included.

Version Numbers

Starting with Python 2.0, all major releases are required to have a
version number of the form X.Y; bugfix releases will always be of the
form X.Y.Z.

The current major release under development is referred to as release N;
the just-released major version is referred to as N-1.

In CVS, the bug fix releases happen on a branch. For release 2.x, the
branch is named 'release2x-maint'. For example, the branch for the 2.3
maintenance releases is release23-maint

Procedure

The process for managing bugfix releases is modeled in part on the Tcl
system[1].

The Patch Czar is the counterpart to the BDFL for bugfix releases.
However, the BDFL and designated appointees retain veto power over
individual patches. A Patch Czar might only be looking after a single
branch of development - it's quite possible that a different person
might be maintaining the 2.3.x and the 2.4.x releases.

As individual patches get contributed to the current trunk of CVS, each
patch committer is requested to consider whether the patch is a bug fix
suitable for inclusion in a bugfix release. If the patch is considered
suitable, the committer can either commit the release to the maintenance
branch, or else mark the patch in the commit message.

In addition, anyone from the Python community is free to suggest patches
for inclusion. Patches may be submitted specifically for bugfix
releases; they should follow the guidelines in PEP 3. In general,
though, it's probably better that a bug in a specific release also be
fixed on the HEAD as well as the branch.

The Patch Czar decides when there are a sufficient number of patches to
warrant a release. The release gets packaged up, including a Windows
installer, and made public. If any new bugs are found, they must be
fixed immediately and a new bugfix release publicized (with an
incremented version number). For the 2.3.x cycle, the Patch Czar
(Anthony) has been trying for a release approximately every six months,
but this should not be considered binding in any way on any future
releases.

Bug fix releases are expected to occur at an interval of roughly six
months. This is only a guideline, however - obviously, if a major bug is
found, a bugfix release may be appropriate sooner. In general, only the
N-1 release will be under active maintenance at any time. That is,
during Python 2.4's development, Python 2.3 gets bugfix releases. If,
however, someone qualified wishes to continue the work to maintain an
older release, they should be encouraged.

Patch Czar History

Anthony Baxter is the Patch Czar for 2.3.1 through 2.3.4.

Barry Warsaw is the Patch Czar for 2.2.3.

Guido van Rossum is the Patch Czar for 2.2.2.

Michael Hudson is the Patch Czar for 2.2.1.

Anthony Baxter is the Patch Czar for 2.1.2 and 2.1.3.

Thomas Wouters is the Patch Czar for 2.1.1.

Moshe Zadka is the Patch Czar for 2.0.1.

History

This PEP started life as a proposal on comp.lang.python. The original
version suggested a single patch for the N-1 release to be released
concurrently with the N release. The original version also argued for
sticking with a strict bug fix policy.

Following feedback from the BDFL and others, the draft PEP was written
containing an expanded bugfix release cycle that permitted any previous
major release to obtain patches and also relaxed the strict bug fix
requirement (mainly due to the example of PEP 235, which could be argued
as either a bug fix or a feature).

Discussion then mostly moved to python-dev, where BDFL finally issued a
proclamation basing the Python bugfix release process on Tcl's, which
essentially returned to the original proposal in terms of being only the
N-1 release and only bug fixes, but allowing multiple bugfix releases
until release N is published.

Anthony Baxter then took this PEP and revised it, based on lessons from
the 2.3 release cycle.

References

Copyright

This document has been placed in the public domain.

[1] http://www.tcl.tk/cgi-bin/tct/tip/28.html

authors:
Aahz
Anthony Baxter


text:
PEP: 347 Title: Migrating the Python CVS to Subversion Version:
$Revision$ Last-Modified: $Date$ Author: Martin von Löwis
<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final
Type: Process Content-Type: text/x-rst Created: 14-Jul-2004
Post-History: 14-Jul-2004

Abstract

The Python source code is currently managed in a CVS repository on
sourceforge.net. This PEP proposes to move it to a Subversion repository
on svn.python.org.

Rationale

This change has two aspects: moving from CVS to Subversion, and moving
from SourceForge to python.org. For each, a rationale will be given.

Moving to Subversion

CVS has a number of limitations that have been eliminated by Subversion.
For the development of Python, the most notable improvements are:

-   the ability to rename files and directories, and to remove
    directories, while keeping the history of these files.
-   support for change sets (sets of correlated changes to multiple
    files) through global revision numbers. Change sets are
    transactional.
-   atomic, fast tagging: a cvs tag might take many minutes; a
    Subversion tag (svn cp) will complete quickly, and atomically.
    Likewise, branches are very efficient.
-   support for offline diffs, which is useful when creating patches.

Moving to python.org

SourceForge has kindly provided an important infrastructure for the past
years. Unfortunately, the attention that SF received has also caused
repeated overload situations in the past, to which the SF operators
could not always respond in a timely manner. In particular, for CVS,
they had to reduce the load on the primary CVS server by introducing a
second, read-only CVS server for anonymous access. This server is
regularly synchronized, but lags behind the read-write CVS repository
between synchronizations. As a result, users without commit access can
see recent changes to the repository only after a delay.

On python.org, it would be possible to make the repository accessible
for anonymous access.

Migration Procedure

To move the Python CVS repository, the following steps need to be
executed. The steps are elaborated upon in the following sections.

1.  Collect SSH keys for all current committers, along with usernames to
    appear in commit messages.
2.  At the beginning of the migration, announce that the repository on
    SourceForge closed.
3.  24 hours after the last commit, download the CVS repository.
4.  Convert the CVS repository into a Subversion repository.
5.  Publish the repository with write access for committers, and
    read-only anonymous access.
6.  Disable CVS access on SF.

Collect SSH keys

After some discussion, svn+ssh was selected as the best method for write
access to the repository. Developers can continue to use their SSH keys,
but they must be installed on python.org.

In order to avoid having to create a new Unix user for each developer, a
single account should be used, with command= attributes in the
authorized_keys files.

The lines in the authorized_keys file should read like this (wrapped for
better readability):

    command="/usr/bin/svnserve --root=/svnroot -t
    --tunnel-user='<username>'",no-port-forwarding,
    no-X11-forwarding,no-agent-forwarding,no-pty
    ssh-dss <key> <comment>

As the usernames, the real names should be used instead of the SF
account names, so that people can be better identified in log messages.

Administrator Access

Administrator access to the pythondev account should be granted to all
current admins of the Python SF project. To distinguish between shell
login and svnserve login, admins need to maintain two keys. Using
OpenSSH, the following procedure can be used to create a second key:

    cd .ssh
    ssh-keygen -t DSA -f pythondev -C <user>@pythondev
    vi config

In the config file, the following lines need to be added:

    Host pythondev
      Hostname dinsdale.python.org
      User pythondev
      IdentityFile ~/.ssh/pythondev

Then, shell login becomes possible through "ssh pythondev".

Downloading the CVS Repository

The CVS repository can be downloaded from

  http://cvs.sourceforge.net/cvstarballs/python-cvsroot.tar.bz2

Since this tarball is generated only once a day, some time must pass
after the repository freeze before the tarball can be picked up. It
should be verified that the last commit, as recorded on the
python-commits mailing list, is indeed included in the tarball.

After the conversion, the converted CVS tarball should be kept forever
on www.python.org/archive/python-cvsroot-<date>.tar.bz2

Converting the CVS Repository

The Python CVS repository contains two modules: distutils and python.
The python module is further structured into dist and nondist, where
dist only contains src (the python code proper). nondist contains
various subdirectories.

These should be reorganized in the Subversion repository to get shorter
URLs, following the <project>/{trunk,tags,branches} structure. A project
will be created for each nondist directory, plus for src (called
python), plus distutils. Reorganizing the repository is best done in the
CVS tree, as shown below.

The fsfs backend should be used as the repository format (which requires
Subversion 1.1). The fsfs backend has the advantage of being more
backup-friendly, as it allows incremental repository backups, without
requiring any dump commands to be run.

The conversion should be done using the cvs2svn utility, available e.g.
in the cvs2svn Debian package. As cvs2svn does not currently support the
project/trunk structure, each project needs to be converted separately.
To get each conversion result into a separate directory in the target
repository, svnadmin load must be used.

Subversion has a different view on binary-vs-text files than CVS. To
correctly carry the CVS semantics forward, svn:eol-style should be set
to native on all files that are not marked binary in the CVS.

In summary, the conversion script is:

    #!/bin/sh
    rm cvs2svn-*
    rm -rf python py.new
    tar xjf python-cvsroot.tar.bz2
    rm -rf python/CVSROOT
    svnadmin create --fs-type fsfs py.new
    mv python/python python/orig
    mv python/orig/dist/src python/python
    mv python/orig/nondist/* python
    # nondist/nondist is empty
    rmdir python/nondist
    rm -rf python/orig
    for a in python/*
    do
      b=`basename $a`
      cvs2svn -q --dump-only --encoding=latin1 --force-branch=cnri-16-start \
      --force-branch=descr-branch --force-branch=release152p1-patches \
      --force-tag=r16b1 $a
      svn mkdir -m"Conversion to SVN" file:///`pwd`/py.new/$b
      svnadmin load -q --parent-dir $b py.new < cvs2svn-dump
      rm cvs2svn-dump
    done

Sample results of this conversion are available at

  http://www.dcl.hpi.uni-potsdam.de/pysvn/

Publish the Repository

The repository should be published at http://svn.python.org/projects.
Read-write access should be granted to all current SF committers through
<EMAIL_ADDRESS>read-only anonymous access through
WebDAV should also be granted.

As an option, websvn (available e.g. from the Debian websvn package)
could be provided. Unfortunately, in the test installation, websvn
breaks because it runs out of memory.

The current SF project admins should get write access to the
authorized_keys2 file of the pythondev account.

Disable CVS

It appears that CVS cannot be disabled entirely. Only the user interface
can be removed from the project page; the repository itself remains
available. If desired, write access to the python and distutils modules
can be disabled through a CVS commitinfo entry.

Discussion

Several alternatives had been suggested to the procedure above. The
rejected alternatives are shortly discussed here:

-   create multiple repositories, one for python and one for distutils.
    This would have allowed even shorter URLs, but was rejected because
    a single repository supports moving code across projects.
-   Several people suggested to create the project/trunk structure
    through standard cvs2svn, followed by renames. This would have the
    disadvantage that old revisions use different path names than recent
    revisions; the suggested approach through dump files works without
    renames.
-   Several people also expressed concern about the administrative
    overhead that hosting the repository on python.org would cause to
    pydotorg admins. As a specific alternative, BerliOS has been
    suggested. The pydotorg admins themselves haven't objected to the
    additional workload; migrating the repository again if they get
    overworked is an option.
-   Different authentication strategies were discussed. As alternatives
    to svn+ssh were suggested
    -   Subversion over WebDAV, using SSL and basic authentication, with
        pydotorg-generated passwords mailed to the user. People did not
        like that approach, since they would need to store the password
        on disk (because they can't remember it); this is a security
        risk.
    -   Subversion over WebDAV, using SSL client certificates. This
        would work, but would require us to administer a certificate
        authority.
-   Instead of hosting this on python.org, people suggested hosting it
    elsewhere. One issue is whether this alternative should be free or
    commercial; several people suggested it should better be commercial,
    to reduce the load on the volunteers. In particular:
    -   Greg Stein suggested http://www.wush.net/subversion.php. They
        offer 5 GB for $90/month, with 200 GB download/month. The data
        is on a RAID drive and fully backed up. Anonymous access and
        email commit notifications are supported. wush.net elaborated
        the following details:

        -   The machine would be a Virtuozzo Virtual Private Server
            (VPS), hosted at PowerVPS.
        -   The default repository URL would be
            http://python.wush.net/svn/projectname/, but anything else
            could be arranged
        -   we would get SSH login to the machine, with sudo
            capabilities.
        -   They have a Web interface for management of the various SVN
            repositories that we want to host, and to manage user
            accounts. While svn+ssh would be supported, the user
            interface does not yet support it.
        -   For offsite mirroring/backup, they suggest to use rsync
            instead of download of repository tarballs.

        Bob Ippolito reported that they had used wush.net for a
        commercial project for about 6 months, after which time they
        left wush.net, because the service was down for three days, with
        nobody reachable, and no explanation when it came back.

Copyright

This document has been placed in the public domain.