text:
PEP: 432 Title: Restructuring the CPython startup sequence Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Victor Stinner<EMAIL_ADDRESS>Eric Snow
<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track Content-Type: text/x-rst
Requires: 587 Created: 28-Dec-2012 Post-History: 28-Dec-2012,
02-Jan-2013, 30-Mar-2019, 28-Jun-2020

PEP Withdrawal

From late 2012 to mid 2020, this PEP provided general background and
specific concrete proposals for making the CPython startup sequence
easier to maintain and the CPython runtime easier to embed as part of a
larger application.

For most of that time, the changes were maintained either in a separate
feature branch, or else as underscore-prefixed private APIs in the main
CPython repo.

In 2019, PEP 587 migrated a subset of those API changes to the public
CPython API for Python 3.8+ (specifically, the PEP updated the
interpreter runtime to offer an explicitly multi-stage struct-based
configuration interface).

In June 2020, in response to a query from the Steering Council, the PEP
authors decided that it made sense to withdraw the original PEP, as
enough has changed since PEP 432 was first written that we think any
further changes to the startup sequence and embedding API would be best
formulated as a new PEP (or PEPs) that take into account not only the
not-yet-implemented ideas from PEP 432 that weren't considered
sufficiently well validated to make their way into PEP 587, but also any
feedback on the public PEP 587 API, and any other lessons that have been
learned while adjusting the CPython implementation to be more embedding
and subinterpreter friendly.

In particular, PEPs proposing the following changes, and any further
infrastructure changes needed to enable them, would likely still be
worth exploring:

-   shipping an alternate Python executable that ignores all user level
    settings and runs in isolated mode by default, and would hence be
    more suitable for execution of system level Python applications than
    the default interpreter
-   enhancing the zipapp module to support the creation of single-file
    executables from pure Python scripts (and potentially even Python
    extension modules, given the introduction of multi-phase extension
    module initialisation)
-   migrating the complex sys.path initialisation logic from C to Python
    in order to improve test suite coverage and the general
    maintainability of that code

Abstract

This PEP proposes a mechanism for restructuring the startup sequence for
CPython, making it easier to modify the initialization behaviour of the
reference interpreter executable, as well as making it easier to control
CPython's startup behaviour when creating an alternate executable or
embedding it as a Python execution engine inside a larger application.

When implementation of this proposal is completed, interpreter startup
will consist of three clearly distinct and independently configurable
phases:

-   Python core runtime preinitialization
    -   setting up memory management
    -   determining the encodings used for system interfaces (including
        settings passed in for later configuration phase)
-   Python core runtime initialization
    -   ensuring C API is ready for use
    -   ensuring builtin and frozen modules are accessible
-   Main interpreter configuration
    -   ensuring external modules are accessible
    -   (Note: the name of this phase is quite likely to change)

Changes are also proposed that impact main module execution and
subinterpreter initialization.

Note: TBC = To Be Confirmed, TBD = To Be Determined. The appropriate
resolution for most of these should become clearer as the reference
implementation is developed.

Proposal

This PEP proposes that initialization of the CPython runtime be split
into three clearly distinct phases:

-   core runtime preinitialization
-   core runtime initialization
-   main interpreter configuration

(Earlier versions proposed only two phases, but experience with
attempting to implement the PEP as an internal CPython refactoring
showed that at least 3 phases are needed to get clear separation of
concerns)

The proposed design also has significant implications for:

-   main module execution
-   subinterpreter initialization

In the new design, the interpreter will move through the following
well-defined phases during the initialization sequence:

-   Uninitialized - haven't even started the pre-initialization phase
    yet
-   Pre-Initialization - no interpreter available
-   Runtime Initialized - main interpreter partially available,
    subinterpreter creation not yet available
-   Initialized - main interpreter fully available, subinterpreter
    creation available

PEP 587 is a more detailed proposal that covers separating out the
Pre-Initialization phase from the last two phases, but doesn't allow
embedding applications to run arbitrary code while in the "Runtime
Initialized" state (instead, initializing the core runtime will also
always fully initialize the main interpreter, as that's the way the
native CPython CLI still works in Python 3.8).

As a concrete use case to help guide any design changes, and to solve a
known problem where the appropriate defaults for system utilities differ
from those for running user scripts, this PEP proposes the creation and
distribution of a separate system Python (system-python) executable
which, by default, operates in "isolated mode" (as selected by the
CPython -I switch), as well as the creation of an example stub binary
that just runs an appended zip archive (permitting single-file pure
Python executables) rather than going through the normal CPython startup
sequence.

To keep the implementation complexity under control, this PEP does not
propose wholesale changes to the way the interpreter state is accessed
at runtime. Changing the order in which the existing initialization
steps occur in order to make the startup sequence easier to maintain is
already a substantial change, and attempting to make those other changes
at the same time will make the change significantly more invasive and
much harder to review. However, such proposals may be suitable topics
for follow-on PEPs or patches - one key benefit of this PEP and its
related subproposals is decreasing the coupling between the internal
storage model and the configuration interface, so such changes should be
easier once this PEP has been implemented.

Background

Over time, CPython's initialization sequence has become progressively
more complicated, offering more options, as well as performing more
complex tasks (such as configuring the Unicode settings for OS
interfaces in Python 3[1], bootstrapping a pure Python implementation of
the import system, and implementing an isolated mode more suitable for
system applications that run with elevated privileges[2]).

Much of this complexity is formally accessible only through the Py_Main
and Py_Initialize APIs, offering embedding applications little
opportunity for customisation. This creeping complexity also makes life
difficult for maintainers, as much of the configuration needs to take
place prior to the Py_Initialize call, meaning much of the Python C API
cannot be used safely.

A number of proposals are on the table for even more sophisticated
startup behaviour, such as better control over sys.path initialization
(e.g. easily adding additional directories on the command line in a
cross-platform fashion[3], controlling the configuration of
sys.path[0][4]), easier configuration of utilities like coverage tracing
when launching Python subprocesses[5]).

Rather than continuing to bolt such behaviour onto an already
complicated system indefinitely, this PEP proposes to start simplifying
the status quo by introducing a more structured startup sequence, with
the aim of making these further feature requests easier to implement.

Originally the entire proposal was maintained in this one PEP, but that
proved impractical, so as parts of the proposed design stabilised, they
are now split out into their own PEPs, allowing progress to be made,
even while the details of the overall design are still evolving.

Key Concerns

There are a few key concerns that any change to the startup sequence
needs to take into account.

Maintainability

The CPython startup sequence as of Python 3.6 was difficult to
understand, and even more difficult to modify. It was not clear what
state the interpreter was in while much of the initialization code
executed, leading to behaviour such as lists, dictionaries and Unicode
values being created prior to the call to Py_Initialize when the -X or
-W options are used[6].

By moving to an explicitly multi-phase startup sequence, developers
should only need to understand:

-   which APIs and features are available prior to pre-configuration
    (essentially none, except for the pre-configuration API itself)
-   which APIs and features are available prior to core runtime
    configuration, and will implicitly run the pre-configuration with
    default settings that match the behaviour of Python 3.6 if the
    pre-configuration hasn't been run explicitly
-   which APIs and features are only available after the main
    interpreter has been fully configured (which will hopefully be a
    relatively small subset of the full C API)

The first two aspects of that are covered by PEP 587, while the details
of the latter distinction are still being considered.

By basing the new design on a combination of C structures and Python
data types, it should also be easier to modify the system in the future
to add new configuration options.

Testability

One of the problems with the complexity of the CPython startup sequence
is the combinatorial explosion of possible interactions between
different configuration settings.

This concern impacts both the design of the new initialisation system,
and the proposed approach for getting there.

Performance

CPython is used heavily to run short scripts where the runtime is
dominated by the interpreter initialization time. Any changes to the
startup sequence should minimise their impact on the startup overhead.

Experience with the importlib migration suggests that the startup time
is dominated by IO operations. However, to monitor the impact of any
changes, a simple benchmark can be used to check how long it takes to
start and then tear down the interpreter:

    python3 -m timeit -s "from subprocess import call" "call(['./python', '-Sc', 'pass'])"

Current numbers on my system for Python 3.7 (as built by the Fedora
project):

    $ python3 -m timeit -s "from subprocess import call" "call(['python3', '-Sc', 'pass'])"
    50 loops, best of 5: 6.48 msec per loop

(TODO: run this microbenchmark with perf rather than the stdlib timeit)

This PEP is not expected to have any significant effect on the startup
time, as it is aimed primarily at reordering the existing initialization
sequence, without making substantial changes to the individual steps.

However, if this simple check suggests that the proposed changes to the
initialization sequence may pose a performance problem, then a more
sophisticated microbenchmark will be developed to assist in
investigation.

Required Configuration Settings

See PEP 587 for a detailed listing of CPython interpreter configuration
settings and the various means available for setting them.

Implementation Strategy

An initial attempt was made at implementing an earlier version of this
PEP for Python 3.4[7], with one of the significant problems encountered
being merge conflicts after the initial structural changes were put in
place to start the refactoring process. Unlike some other previous major
changes, such as the switch to an AST-based compiler in Python 2.5, or
the switch to the importlib implementation of the import system in
Python 3.3, there is no clear way to structure a draft implementation
that won't be prone to the kinds of merge conflicts that afflicted the
original attempt.

Accordingly, the implementation strategy was revised to instead first
implement this refactoring as a private API for CPython 3.7, and then
review the viability of exposing the new functions and structures as
public API elements in CPython 3.8.

After the initial merge, Victor Stinner then proceeded to actually
migrate settings to the new structure in order to successfully implement
the PEP 540 UTF-8 mode changes (which required the ability to track all
settings that had previously been decoded with the locale encoding, and
decode them again using UTF-8 instead). Eric Snow also migrated a number
of internal subsystems over as part of making the subinterpreter feature
more robust.

That work showed that the detailed design originally proposed in this
PEP had a range of practical issues, so Victor designed and implemented
an improved private API (inspired by an earlier iteration of this PEP),
which PEP 587 proposes to promote to a public API in Python 3.8.

Design Details

Note

The API details here are still very much in flux. The header files that
show the current state of the private API are mainly:

-   https://github.com/python/cpython/blob/master/Include/cpython/coreconfig.h
-   https://github.com/python/cpython/blob/master/Include/cpython/pystate.h
-   https://github.com/python/cpython/blob/master/Include/cpython/pylifecycle.h

PEP 587 covers the aspects of the API that are considered potentially
stable enough to make public. Where a proposed API is covered by that
PEP, "(see PEP 587)" is added to the text below.

The main theme of this proposal is to initialize the core language
runtime and create a partially initialized interpreter state for the
main interpreter much earlier in the startup process. This will allow
most of the CPython API to be used during the remainder of the
initialization process, potentially simplifying a number of operations
that currently need to rely on basic C functionality rather than being
able to use the richer data structures provided by the CPython C API.

PEP 587 covers a subset of that task, which is splitting out the
components that even the existing "May be called before Py_Initialize"
interfaces need (like memory allocators and operating system interface
encoding details) into a separate pre-configuration step.

In the following, the term "embedding application" also covers the
standard CPython command line application.

Interpreter Initialization Phases

The following distinct interpreter initialisation phases are proposed:

-   Uninitialized:
    -   Not really a phase, but the absence of a phase
    -   Py_IsInitializing() returns 0
    -   Py_IsRuntimeInitialized() returns 0
    -   Py_IsInitialized() returns 0
    -   The embedding application determines which memory allocator to
        use, and which encoding to use to access operating system
        interfaces (or chooses to delegate those decisions to the Python
        runtime)
    -   Application starts the initialization process by calling one of
        the Py_PreInitialize APIs (see PEP 587)
-   Runtime Pre-Initialization:
    -   no interpreter is available
    -   Py_IsInitializing() returns 1
    -   Py_IsRuntimeInitialized() returns 0
    -   Py_IsInitialized() returns 0
    -   The embedding application determines the settings required to
        initialize the core CPython runtime and create the main
        interpreter and moves to the next phase by calling
        Py_InitializeRuntime
    -   Note: as of PEP 587, the embedding application instead calls
        Py_Main(), Py_UnixMain, or one of the Py_Initialize APIs, and
        hence jumps directly to the Initialized state.
-   Main Interpreter Initialization:
    -   the builtin data types and other core runtime services are
        available
    -   the main interpreter is available, but only partially configured
    -   Py_IsInitializing() returns 1
    -   Py_IsRuntimeInitialized() returns 1
    -   Py_IsInitialized() returns 0
    -   The embedding application determines and applies the settings
        required to complete the initialization process by calling
        Py_InitializeMainInterpreter
    -   Note: as of PEP 587, this state is not reachable via any public
        API, it only exists as an implicit internal state while one of
        the Py_Initialize functions is running
-   Initialized:
    -   the main interpreter is available and fully operational, but
        __main__ related metadata is incomplete
    -   Py_IsInitializing() returns 0
    -   Py_IsRuntimeInitialized() returns 1
    -   Py_IsInitialized() returns 1

Invocation of Phases

All listed phases will be used by the standard CPython interpreter and
the proposed System Python interpreter.

An embedding application may still continue to leave initialization
almost entirely under CPython's control by using the existing
Py_Initialize or Py_Main() APIs - backwards compatibility will be
preserved.

Alternatively, if an embedding application wants greater control over
CPython's initial state, it will be able to use the new, finer grained
API, which allows the embedding application greater control over the
initialization process.

PEP 587 covers an initial iteration of that API, separating out the
pre-initialization phase without attempting to separate core runtime
initialization from main interpreter initialization.

Uninitialized State

The uninitialized state is where an embedding application determines the
settings which are required in order to be able to correctly pass
configurations settings to the embedded Python runtime.

This covers telling Python which memory allocator to use, as well as
which text encoding to use when processing provided settings.

PEP 587 defines the settings needed to exit this state in its
PyPreConfig struct.

A new query API will allow code to determine if the interpreter hasn't
even started the initialization process:

    int Py_IsInitializing();

The query for a completely uninitialized environment would then be
!(Py_Initialized() || Py_Initializing()).

Runtime Pre-Initialization Phase

Note

In PEP 587, the settings for this phase are not yet separated out, and
are instead only available through the combined PyConfig struct

The pre-initialization phase is where an embedding application
determines the settings which are absolutely required before the CPython
runtime can be initialized at all. Currently, the primary configuration
settings in this category are those related to the randomised hash
algorithm - the hash algorithms must be consistent for the lifetime of
the process, and so they must be in place before the core interpreter is
created.

The essential settings needed are a flag indicating whether or not to
use a specific seed value for the randomised hashes, and if so, the
specific value for the seed (a seed value of zero disables randomised
hashing). In addition, due to the possible use of PYTHONHASHSEED in
configuring the hash randomisation, the question of whether or not to
consider environment variables must also be addressed early. Finally, to
support the CPython build process, an option is offered to completely
disable the import system.

The proposed APIs for this step in the startup sequence are:

    PyInitError Py_InitializeRuntime(
        const PyRuntimeConfig *config
    );

    PyInitError Py_InitializeRuntimeFromArgs(
        const PyRuntimeConfig *config, int argc, char **argv
    );

    PyInitError Py_InitializeRuntimeFromWideArgs(
        const PyRuntimeConfig *config, int argc, wchar_t **argv
    );

If Py_IsInitializing() is false, the Py_InitializeRuntime functions will
implicitly call the corresponding Py_PreInitialize function. The
use_environment setting will be passed down, while other settings will
be processed according to their defaults, as described in PEP 587.

The PyInitError return type is defined in PEP 587, and allows an
embedding application to gracefully handle Python runtime initialization
failures, rather than having the entire process abruptly terminated by
Py_FatalError.

The new PyRuntimeConfig struct holds the settings required for
preliminary configuration of the core runtime and creation of the main
interpreter:

    /* Note: if changing anything in PyRuntimeConfig, also update
     * PyRuntimeConfig_INIT */
    typedef struct {
        bool use_environment;     /* as in PyPreConfig, PyConfig from PEP 587 */
        int use_hash_seed;        /* PYTHONHASHSEED, as in PyConfig from PEP 587 */
        unsigned long hash_seed;  /* PYTHONHASHSEED, as in PyConfig from PEP 587 */
        bool _install_importlib;  /* Needed by freeze_importlib */
    } PyRuntimeConfig;

    /* Rely on the "designated initializer" feature of C99 */
    #define PyRuntimeConfig_INIT {.use_hash_seed=-1}

The core configuration settings pointer may be NULL, in which case the
default values are as specified in PyRuntimeConfig_INIT.

The PyRuntimeConfig_INIT macro is designed to allow easy initialization
of a struct instance with sensible defaults:

    PyRuntimeConfig runtime_config = PyRuntimeConfig_INIT;

use_environment controls the processing of all Python related
environment variables. If the flag is true, then PYTHONHASHSEED is
processed normally. Otherwise, all Python-specific environment variables
are considered undefined (exceptions may be made for some OS specific
environment variables, such as those used on Mac OS X to communicate
between the App bundle and the main Python binary).

use_hash_seed controls the configuration of the randomised hash
algorithm. If it is zero, then randomised hashes with a random seed will
be used. It is positive, then the value in hash_seed will be used to
seed the random number generator. If the hash_seed is zero in this case,
then the randomised hashing is disabled completely.

If use_hash_seed is negative (and use_environment is true), then CPython
will inspect the PYTHONHASHSEED environment variable. If the environment
variable is not set, is set to the empty string, or to the value
"random", then randomised hashes with a random seed will be used. If the
environment variable is set to the string "0" the randomised hashing
will be disabled. Otherwise, the hash seed is expected to be a string
representation of an integer in the range [0;<PHONE_NUMBER>].

To make it easier for embedding applications to use the PYTHONHASHSEED
processing with a different data source, the following helper function
will be added to the C API:

    int Py_ReadHashSeed(char *seed_text,
                        int *use_hash_seed,
                        unsigned long *hash_seed);

This function accepts a seed string in seed_text and converts it to the
appropriate flag and seed values. If seed_text is NULL, the empty string
or the value "random", both use_hash_seed and hash_seed will be set to
zero. Otherwise, use_hash_seed will be set to 1 and the seed text will
be interpreted as an integer and reported as hash_seed. On success the
function will return zero. A non-zero return value indicates an error
(most likely in the conversion to an integer).

The _install_importlib setting is used as part of the CPython build
process to create an interpreter with no import capability at all. It is
considered private to the CPython development team (hence the leading
underscore), as the only currently supported use case is to permit
compiler changes that invalidate the previously frozen bytecode for
importlib._bootstrap without breaking the build process.

The aim is to keep this initial level of configuration as small as
possible in order to keep the bootstrapping environment consistent
across different embedding applications. If we can create a valid
interpreter state without the setting, then the setting should appear
solely in the comprehensive PyConfig struct rather than in the core
runtime configuration.

A new query API will allow code to determine if the interpreter is in
the bootstrapping state between the core runtime initialization and the
creation of the main interpreter state and the completion of the bulk of
the main interpreter initialization process:

    int Py_IsRuntimeInitialized();

Attempting to call Py_InitializeRuntime() again when
Py_IsRuntimeInitialized() is already true is reported as a user
configuration error. (TBC, as existing public initialisation APIs
support being called multiple times without error, and simply ignore
changes to any write-once settings. It may make sense to keep that
behaviour rather than trying to make the new API stricter than the old
one)

As frozen bytecode may now be legitimately run in an interpreter which
is not yet fully initialized, sys.flags will gain a new initialized
flag.

With the core runtime initialised, the main interpreter and most of the
CPython C API should be fully functional except that:

-   compilation is not allowed (as the parser and compiler are not yet
    configured properly)
-   creation of subinterpreters is not allowed
-   creation of additional thread states is not allowed
-   The following attributes in the sys module are all either missing or
    None:
    -   sys.path
    -   sys.argv
    -   sys.executable
    -   sys.base_exec_prefix
    -   sys.base_prefix
    -   sys.exec_prefix
    -   sys.prefix
    -   sys.warnoptions
    -   sys.dont_write_bytecode
    -   sys.stdin
    -   sys.stdout
-   The filesystem encoding is not yet defined
-   The IO encoding is not yet defined
-   CPython signal handlers are not yet installed
-   Only builtin and frozen modules may be imported (due to above
    limitations)
-   sys.stderr is set to a temporary IO object using unbuffered binary
    mode
-   The sys.flags attribute exists, but the individual flags may not yet
    have their final values.
-   The sys.flags.initialized attribute is set to 0
-   The warnings module is not yet initialized
-   The __main__ module does not yet exist

<TBD: identify any other notable missing functionality>

The main things made available by this step will be the core Python data
types, in particular dictionaries, lists and strings. This allows them
to be used safely for all of the remaining configuration steps (unlike
the status quo).

In addition, the current thread will possess a valid Python thread
state, allowing any further configuration data to be stored on the main
interpreter object rather than in C process globals.

Any call to Py_InitializeRuntime() must have a matching call to
Py_Finalize(). It is acceptable to skip calling
Py_InitializeMainInterpreter() in between (e.g. if attempting to build
the main interpreter configuration settings fails).

Determining the remaining configuration settings

The next step in the initialization sequence is to determine the
remaining settings needed to complete the process. No changes are made
to the interpreter state at this point. The core APIs for this step are:

    int Py_BuildPythonConfig(
        PyConfigAsObjects *py_config, const PyConfig *c_config
    );

    int Py_BuildPythonConfigFromArgs(
        PyConfigAsObjects *py_config, const PyConfig *c_config, int argc, char **argv
    );

    int Py_BuildPythonConfigFromWideArgs(
        PyConfigAsObjects *py_config, const PyConfig *c_config, int argc, wchar_t **argv
    );

The py_config argument should be a pointer to a PyConfigAsObjects struct
(which may be a temporary one stored on the C stack). For any already
configured value (i.e. any non-NULL pointer), CPython will sanity check
the supplied value, but otherwise accept it as correct.

A struct is used rather than a Python dictionary as the struct is easier
to work with from C, the list of supported fields is fixed for a given
CPython version and only a read-only view needs to be exposed to Python
code (which is relatively straightforward, thanks to the infrastructure
already put in place to expose sys.implementation).

Unlike Py_InitializeRuntime, this call will raise a Python exception and
report an error return rather than returning a Python initialization
specific C struct if a problem is found with the config data.

Any supported configuration setting which is not already set will be
populated appropriately in the supplied configuration struct. The
default configuration can be overridden entirely by setting the value
before calling Py_BuildPythonConfig. The provided value will then also
be used in calculating any other settings derived from that value.

Alternatively, settings may be overridden after the Py_BuildPythonConfig
call (this can be useful if an embedding application wants to adjust a
setting rather than replace it completely, such as removing
sys.path[0]).

The c_config argument is an optional pointer to a PyConfig structure, as
defined in PEP 587. If provided, it is used in preference to reading
settings directly from the environment or process global state.

Merely reading the configuration has no effect on the interpreter state:
it only modifies the passed in configuration struct. The settings are
not applied to the running interpreter until the
Py_InitializeMainInterpreter call (see below).

Supported configuration settings

The interpreter configuration is split into two parts: settings which
are either relevant only to the main interpreter or must be identical
across the main interpreter and all subinterpreters, and settings which
may vary across subinterpreters.

NOTE: For initial implementation purposes, only the flag indicating
whether or not the interpreter is the main interpreter will be
configured on a per interpreter basis. Other fields will be reviewed for
whether or not they can feasibly be made interpreter specific over the
course of the implementation.

Note

The list of config fields below is currently out of sync with PEP 587.
Where they differ, PEP 587 takes precedence.

The PyConfigAsObjects struct mirrors the PyConfig struct from PEP 587,
but uses full Python objects to store values, rather than C level data
types. It adds raw_argv and argv list fields, so later initialisation
steps don't need to accept those separately.

Fields are always pointers to Python data types, with unset values
indicated by NULL:

    typedef struct {
        /* Argument processing */
        PyListObject *raw_argv;
        PyListObject *argv;
        PyListObject *warnoptions; /* -W switch, PYTHONWARNINGS */
        PyDictObject *xoptions;    /* -X switch */

        /* Filesystem locations */
        PyUnicodeObject *program_name;
        PyUnicodeObject *executable;
        PyUnicodeObject *prefix;           /* PYTHONHOME */
        PyUnicodeObject *exec_prefix;      /* PYTHONHOME */
        PyUnicodeObject *base_prefix;      /* pyvenv.cfg */
        PyUnicodeObject *base_exec_prefix; /* pyvenv.cfg */

        /* Site module */
        PyBoolObject *enable_site_config;  /* -S switch (inverted) */
        PyBoolObject *no_user_site;        /* -s switch, PYTHONNOUSERSITE */

        /* Import configuration */
        PyBoolObject *dont_write_bytecode; /* -B switch, PYTHONDONTWRITEBYTECODE */
        PyBoolObject *ignore_module_case;  /* PYTHONCASEOK */
        PyListObject *import_path;        /* PYTHONPATH (etc) */

        /* Standard streams */
        PyBoolObject    *use_unbuffered_io; /* -u switch, PYTHONUNBUFFEREDIO */
        PyUnicodeObject *stdin_encoding;    /* PYTHONIOENCODING */
        PyUnicodeObject *stdin_errors;      /* PYTHONIOENCODING */
        PyUnicodeObject *stdout_encoding;   /* PYTHONIOENCODING */
        PyUnicodeObject *stdout_errors;     /* PYTHONIOENCODING */
        PyUnicodeObject *stderr_encoding;   /* PYTHONIOENCODING */
        PyUnicodeObject *stderr_errors;     /* PYTHONIOENCODING */

        /* Filesystem access */
        PyUnicodeObject *fs_encoding;

        /* Debugging output */
        PyBoolObject *debug_parser;    /* -d switch, PYTHONDEBUG */
        PyLongObject *verbosity;       /* -v switch */

        /* Code generation */
        PyLongObject *bytes_warnings;  /* -b switch */
        PyLongObject *optimize;        /* -O switch */

        /* Signal handling */
        PyBoolObject *install_signal_handlers;

        /* Implicit execution */
        PyUnicodeObject *startup_file;  /* PYTHONSTARTUP */

        /* Main module
         *
         * If prepare_main is set, at most one of the main_* settings should
         * be set before calling PyRun_PrepareMain (Py_ReadMainInterpreterConfig
         * will set one of them based on the command line arguments if
         * prepare_main is non-zero when that API is called).
        PyBoolObject    *prepare_main;
        PyUnicodeObject *main_source; /* -c switch */
        PyUnicodeObject *main_path;   /* filesystem path */
        PyUnicodeObject *main_module; /* -m switch */
        PyCodeObject    *main_code;   /* Run directly from a code object */
        PyObject        *main_stream; /* Run from stream */
        PyBoolObject    *run_implicit_code; /* Run implicit code during prep */

        /* Interactive main
         *
         * Note: Settings related to interactive mode are very much in flux.
         */
        PyObject *prompt_stream;      /* Output interactive prompt */
        PyBoolObject *show_banner;    /* -q switch (inverted) */
        PyBoolObject *inspect_main;   /* -i switch, PYTHONINSPECT */

    } PyConfigAsObjects;

The PyInterpreterConfig struct holds the settings that may vary between
the main interpreter and subinterpreters. For the main interpreter,
these settings are automatically populated by
Py_InitializeMainInterpreter().

    typedef struct {
        PyBoolObject *is_main_interpreter;    /* Easily check for subinterpreters */
    } PyInterpreterConfig;

As these structs consist solely of object pointers, no explicit
initializer definitions are needed - C99's default initialization of
struct memory to zero is sufficient.

Completing the main interpreter initialization

The final step in the initialization process is to actually put the
configuration settings into effect and finish bootstrapping the main
interpreter up to full operation:

    int Py_InitializeMainInterpreter(const PyConfigAsObjects *config);

Like Py_BuildPythonConfig, this call will raise an exception and report
an error return rather than exhibiting fatal errors if a problem is
found with the config data. (TBC, as existing public initialisation APIs
support being called multiple times without error, and simply ignore
changes to any write-once settings. It may make sense to keep that
behaviour rather than trying to make the new API stricter than the old
one)

All configuration settings are required - the configuration struct
should always be passed through Py_BuildPythonConfig to ensure it is
fully populated.

After a successful call Py_IsInitialized() will become true and
Py_IsInitializing() will become false. The caveats described above for
the interpreter during the phase where only the core runtime is
initialized will no longer hold.

Attempting to call Py_InitializeMainInterpreter() again when
Py_IsInitialized() is true is an error.

However, some metadata related to the __main__ module may still be
incomplete:

-   sys.argv[0] may not yet have its final value
    -   it will be -m when executing a module or package with CPython
    -   it will be the same as sys.path[0] rather than the location of
        the __main__ module when executing a valid sys.path entry
        (typically a zipfile or directory)
    -   otherwise, it will be accurate:
        -   the script name if running an ordinary script
        -   -c if executing a supplied string
        -   - or the empty string if running from stdin
-   the metadata in the __main__ module will still indicate it is a
    builtin module

This function will normally implicitly import site as its final
operation (after Py_IsInitialized() is already set). Setting the
"enable_site_config" flag to Py_False in the configuration settings will
disable this behaviour, as well as eliminating any side effects on
global state if import site is later explicitly executed in the process.

Preparing the main module

Note

In PEP 587, PyRun_PrepareMain and PyRun_ExecMain are not exposed
separately, and are instead accessed through a Py_RunMain API that both
prepares and executes main, and then finalizes the Python interpreter.

This subphase completes the population of the __main__ module related
metadata, without actually starting execution of the __main__ module
code.

It is handled by calling the following API:

    int PyRun_PrepareMain();

This operation is only permitted for the main interpreter, and will
raise RuntimeError when invoked from a thread where the current thread
state belongs to a subinterpreter.

The actual processing is driven by the main related settings stored in
the interpreter state as part of the configuration struct.

If prepare_main is zero, this call does nothing.

If all of main_source, main_path, main_module, main_stream and main_code
are NULL, this call does nothing.

If more than one of main_source, main_path, main_module, main_stream or
main_code are set, RuntimeError will be reported.

If main_code is already set, then this call does nothing.

If main_stream is set, and run_implicit_code is also set, then the file
identified in startup_file will be read, compiled and executed in the
__main__ namespace.

If main_source, main_path or main_module are set, then this call will
take whatever steps are needed to populate main_code:

-   For main_source, the supplied string will be compiled and saved to
    main_code.
-   For main_path:
    -   if the supplied path is recognised as a valid sys.path entry, it
        is inserted as sys.path[0], main_module is set to __main__ and
        processing continues as for main_module below.
    -   otherwise, path is read as a CPython bytecode file
    -   if that fails, it is read as a Python source file and compiled
    -   in the latter two cases, the code object is saved to main_code
        and __main__.__file__ is set appropriately
-   For main_module:
    -   any parent package is imported
    -   the loader for the module is determined
    -   if the loader indicates the module is a package, add .__main__
        to the end of main_module and try again (if the final name
        segment is already .__main__ then fail immediately)
    -   once the module source code is located, save the compiled module
        code as main_code and populate the following attributes in
        __main__ appropriately: __name__, __loader__, __file__,
        __cached__, __package__.

(Note: the behaviour described in this section isn't new, it's a
write-up of the current behaviour of the CPython interpreter adjusted
for the new configuration system)

Executing the main module

Note

In PEP 587, PyRun_PrepareMain and PyRun_ExecMain are not exposed
separately, and are instead accessed through a Py_RunMain API that both
prepares and executes main, and then finalizes the Python interpreter.

This subphase covers the execution of the actual __main__ module code.

It is handled by calling the following API:

    int PyRun_ExecMain();

This operation is only permitted for the main interpreter, and will
raise RuntimeError when invoked from a thread where the current thread
state belongs to a subinterpreter.

The actual processing is driven by the main related settings stored in
the interpreter state as part of the configuration struct.

If both main_stream and main_code are NULL, this call does nothing.

If both main_stream and main_code are set, RuntimeError will be
reported.

If main_stream and prompt_stream are both set, main execution will be
delegated to a new internal API:

    int _PyRun_InteractiveMain(PyObject *input, PyObject* output);

If main_stream is set and prompt_stream is NULL, main execution will be
delegated to a new internal API:

    int _PyRun_StreamInMain(PyObject *input);

If main_code is set, main execution will be delegated to a new internal
API:

    int _PyRun_CodeInMain(PyCodeObject *code);

After execution of main completes, if inspect_main is set, or the
PYTHONINSPECT environment variable has been set, then PyRun_ExecMain
will invoke _PyRun_InteractiveMain(sys.__stdin__, sys.__stdout__).

Internal Storage of Configuration Data

The interpreter state will be updated to include details of the
configuration settings supplied during initialization by extending the
interpreter state object with at least an embedded copy of the
PyConfigAsObjects and PyInterpreterConfig structs.

For debugging purposes, the configuration settings will be exposed as a
sys._configuration simple namespace (similar to sys.flags and
sys.implementation. The attributes will be themselves by simple
namespaces corresponding to the two levels of configuration setting:

-   all_interpreters
-   active_interpreter

Field names will match those in the configuration structs, except for
hash_seed, which will be deliberately excluded.

An underscored attribute is chosen deliberately, as these configuration
settings are part of the CPython implementation, rather than part of the
Python language definition. If new settings are needed to support
cross-implementation compatibility in the standard library, then those
should be agreed with the other implementations and exposed as new
required attributes on sys.implementation, as described in PEP 421.

These are snapshots of the initial configuration settings. They are not
modified by the interpreter during runtime (except as noted above).

Creating and Configuring Subinterpreters

As the new configuration settings are stored in the interpreter state,
they need to be initialised when a new subinterpreter is created. This
turns out to be trickier than one might expect due to
PyThreadState_Swap(NULL); (which is fortunately exercised by CPython's
own embedding tests, allowing this problem to be detected during
development).

To provide a straightforward solution for this case, the PEP proposes to
add a new API:

    Py_InterpreterState *Py_InterpreterState_Main();

This will be a counterpart to Py_InterpreterState_Head(), only reporting
the oldest currently existing interpreter rather than the newest. If
Py_NewInterpreter() is called from a thread with an existing thread
state, then the interpreter configuration for that thread will be used
when initialising the new subinterpreter. If there is no current thread
state, the configuration from Py_InterpreterState_Main() will be used.

While the existing Py_InterpreterState_Head() API could be used instead,
that reference changes as subinterpreters are created and destroyed,
while PyInterpreterState_Main() will always refer to the initial
interpreter state created in Py_InitializeRuntime().

A new constraint is also added to the embedding API: attempting to
delete the main interpreter while subinterpreters still exist will now
be a fatal error.

Stable ABI

Most of the APIs proposed in this PEP are excluded from the stable ABI,
as embedding a Python interpreter involves a much higher degree of
coupling than merely writing an extension module.

The only newly exposed APIs that will be part of the stable ABI are the
Py_IsInitializing() and Py_IsRuntimeInitialized() queries.

Build time configuration

This PEP makes no changes to the handling of build time configuration
settings, and thus has no effect on the contents of sys.implementation
or the result of sysconfig.get_config_vars().

Backwards Compatibility

Backwards compatibility will be preserved primarily by ensuring that
Py_BuildPythonConfig() interrogates all the previously defined
configuration settings stored in global variables and environment
variables, and that Py_InitializeMainInterpreter() writes affected
settings back to the relevant locations.

One acknowledged incompatibility is that some environment variables
which are currently read lazily may instead be read once during
interpreter initialization. As the reference implementation matures,
these will be discussed in more detail on a case-by-case basis. The
environment variables which are currently known to be looked up
dynamically are:

-   PYTHONCASEOK: writing to os.environ['PYTHONCASEOK'] will no longer
    dynamically alter the interpreter's handling of filename case
    differences on import (TBC)
-   PYTHONINSPECT: os.environ['PYTHONINSPECT'] will still be checked
    after execution of the __main__ module terminates

The Py_Initialize() style of initialization will continue to be
supported. It will use (at least some elements of) the new API
internally, but will continue to exhibit the same behaviour as it does
today, ensuring that sys.argv is not populated until a subsequent
PySys_SetArgv call (TBC). All APIs that currently support being called
prior to Py_Initialize() will continue to do so, and will also support
being called prior to Py_InitializeRuntime().

A System Python Executable

When executing system utilities with administrative access to a system,
many of the default behaviours of CPython are undesirable, as they may
allow untrusted code to execute with elevated privileges. The most
problematic aspects are the fact that user site directories are enabled,
environment variables are trusted and that the directory containing the
executed file is placed at the beginning of the import path.

Issue 16499[8] added a -I option to change the behaviour of the normal
CPython executable, but this is a hard to discover solution (and adds
yet another option to an already complex CLI). This PEP proposes to
instead add a separate system-python executable

Currently, providing a separate executable with different default
behaviour would be prohibitively hard to maintain. One of the goals of
this PEP is to make it possible to replace much of the hard to maintain
bootstrapping code with more normal CPython code, as well as making it
easier for a separate application to make use of key components of
Py_Main. Including this change in the PEP is designed to help avoid
acceptance of a design that sounds good in theory but proves to be
problematic in practice.

Cleanly supporting this kind of "alternate CLI" is the main reason for
the proposed changes to better expose the core logic for deciding
between the different execution modes supported by CPython:

-   script execution
-   directory/zipfile execution
-   command execution ("-c" switch)
-   module or package execution ("-m" switch)
-   execution from stdin (non-interactive)
-   interactive stdin

Actually implementing this may also reveal the need for some better
argument parsing infrastructure for use during the initializing phase.

Open Questions

-   Error details for Py_BuildPythonConfig and
    Py_InitializeMainInterpreter (these should become clearer as the
    implementation progresses)

Implementation

The reference implementation is being developed as a private API
refactoring within the CPython reference interpreter (as attempting to
maintain it as an independent project proved impractical).

PEP 587 extracts a subset of the proposal that is considered
sufficiently stable to be worth proposing as a public API for Python
3.8.

The Status Quo (as of Python 3.6)

The current mechanisms for configuring the interpreter have accumulated
in a fairly ad hoc fashion over the past 20+ years, leading to a rather
inconsistent interface with varying levels of documentation.

Also see PEP 587 for further discussion of the existing settings and
their handling.

(Note: some of the info below could probably be cleaned up and added to
the C API documentation for 3.x - it's all CPython specific, so it
doesn't belong in the language reference)

Ignoring Environment Variables

The -E command line option allows all environment variables to be
ignored when initializing the Python interpreter. An embedding
application can enable this behaviour by setting
Py_IgnoreEnvironmentFlag before calling Py_Initialize().

In the CPython source code, the Py_GETENV macro implicitly checks this
flag, and always produces NULL if it is set.

<TBD: I believe PYTHONCASEOK is checked regardless of this setting >
<TBD: Does -E also ignore Windows registry keys? >

Randomised Hashing

The randomised hashing is controlled via the -R command line option (in
releases prior to 3.3), as well as the PYTHONHASHSEED environment
variable.

In Python 3.3, only the environment variable remains relevant. It can be
used to disable randomised hashing (by using a seed value of 0) or else
to force a specific hash value (e.g. for repeatability of testing, or to
share hash values between processes)

However, embedding applications must use the Py_HashRandomizationFlag to
explicitly request hash randomisation (CPython sets it in Py_Main()
rather than in Py_Initialize()).

The new configuration API should make it straightforward for an
embedding application to reuse the PYTHONHASHSEED processing with a text
based configuration setting provided by other means (e.g. a config file
or separate environment variable).

Locating Python and the standard library

The location of the Python binary and the standard library is influenced
by several elements. The algorithm used to perform the calculation is
not documented anywhere other than in the source code[9],[10]. Even that
description is incomplete, as it failed to be updated for the virtual
environment support added in Python 3.3 (detailed in PEP 405).

These calculations are affected by the following function calls (made
prior to calling Py_Initialize()) and environment variables:

-   Py_SetProgramName()
-   Py_SetPythonHome()
-   PYTHONHOME

The filesystem is also inspected for pyvenv.cfg files (see PEP 405) or,
failing that, a lib/os.py (Windows) or lib/python$VERSION/os.py file.

The build time settings for PREFIX and EXEC_PREFIX are also relevant, as
are some registry settings on Windows. The hardcoded fallbacks are based
on the layout of the CPython source tree and build output when working
in a source checkout.

Configuring sys.path

An embedding application may call Py_SetPath() prior to Py_Initialize()
to completely override the calculation of sys.path. It is not
straightforward to only allow some of the calculations, as modifying
sys.path after initialization is already complete means those
modifications will not be in effect when standard library modules are
imported during the startup sequence.

If Py_SetPath() is not used prior to the first call to Py_GetPath()
(implicit in Py_Initialize()), then it builds on the location data
calculations above to calculate suitable path entries, along with the
PYTHONPATH environment variable.

<TBD: On Windows, there's also a bunch of stuff to do with the registry>

The site module, which is implicitly imported at startup (unless
disabled via the -S option) adds additional paths to this initial set of
paths, as described in its documentation[11].

The -s command line option can be used to exclude the user site
directory from the list of directories added. Embedding applications can
control this by setting the Py_NoUserSiteDirectory global variable.

The following commands can be used to check the default path
configurations for a given Python executable on a given system:

-   ./python -c "import sys, pprint; pprint.pprint(sys.path)"
    -   standard configuration
-   ./python -s -c "import sys, pprint; pprint.pprint(sys.path)"
    -   user site directory disabled
-   ./python -S -c "import sys, pprint; pprint.pprint(sys.path)"
    -   all site path modifications disabled

(Note: you can see similar information using -m site instead of -c, but
this is slightly misleading as it calls os.abspath on all of the path
entries, making relative path entries look absolute. Using the site
module also causes problems in the last case, as on Python versions
prior to 3.3, explicitly importing site will carry out the path
modifications -S avoids, while on 3.3+ combining -m site with -S
currently fails)

The calculation of sys.path[0] is comparatively straightforward:

-   For an ordinary script (Python source or compiled bytecode),
    sys.path[0] will be the directory containing the script.
-   For a valid sys.path entry (typically a zipfile or directory),
    sys.path[0] will be that path
-   For an interactive session, running from stdin or when using the -c
    or -m switches, sys.path[0] will be the empty string, which the
    import system interprets as allowing imports from the current
    directory

Configuring sys.argv

Unlike most other settings discussed in this PEP, sys.argv is not set
implicitly by Py_Initialize(). Instead, it must be set via an explicitly
call to Py_SetArgv().

CPython calls this in Py_Main() after calling Py_Initialize(). The
calculation of sys.argv[1:] is straightforward: they're the command line
arguments passed after the script name or the argument to the -c or -m
options.

The calculation of sys.argv[0] is a little more complicated:

-   For an ordinary script (source or bytecode), it will be the script
    name
-   For a sys.path entry (typically a zipfile or directory) it will
    initially be the zipfile or directory name, but will later be
    changed by the runpy module to the full path to the imported
    __main__ module.
-   For a module specified with the -m switch, it will initially be the
    string "-m", but will later be changed by the runpy module to the
    full path to the executed module.
-   For a package specified with the -m switch, it will initially be the
    string "-m", but will later be changed by the runpy module to the
    full path to the executed __main__ submodule of the package.
-   For a command executed with -c, it will be the string "-c"
-   For explicitly requested input from stdin, it will be the string "-"
-   Otherwise, it will be the empty string

Embedding applications must call Py_SetArgv themselves. The CPython
logic for doing so is part of Py_Main() and is not exposed separately.
However, the runpy module does provide roughly equivalent logic in
runpy.run_module and runpy.run_path.

Other configuration settings

TBD: Cover the initialization of the following in more detail:

-   Completely disabling the import system
-   The initial warning system state:
    -   sys.warnoptions
    -   (-W option, PYTHONWARNINGS)
-   Arbitrary extended options (e.g. to automatically enable
    faulthandler):
    -   sys._xoptions
    -   (-X option)
-   The filesystem encoding used by:
    -   sys.getfsencoding
    -   os.fsencode
    -   os.fsdecode
-   The IO encoding and buffering used by:
    -   sys.stdin
    -   sys.stdout
    -   sys.stderr
    -   (-u option, PYTHONIOENCODING, PYTHONUNBUFFEREDIO)
-   Whether or not to implicitly cache bytecode files:
    -   sys.dont_write_bytecode
    -   (-B option, PYTHONDONTWRITEBYTECODE)
-   Whether or not to enforce correct case in filenames on
    case-insensitive platforms
    -   os.environ["PYTHONCASEOK"]
-   The other settings exposed to Python code in sys.flags:
    -   debug (Enable debugging output in the pgen parser)
    -   inspect (Enter interactive interpreter after __main__
        terminates)
    -   interactive (Treat stdin as a tty)
    -   optimize (__debug__ status, write .pyc or .pyo, strip doc
        strings)
    -   no_user_site (don't add the user site directory to sys.path)
    -   no_site (don't implicitly import site during startup)
    -   ignore_environment (whether environment vars are used during
        config)
    -   verbose (enable all sorts of random output)
    -   bytes_warning (warnings/errors for implicit str/bytes
        interaction)
    -   quiet (disable banner output even if verbose is also enabled or
        stdin is a tty and the interpreter is launched in interactive
        mode)
-   Whether or not CPython's signal handlers should be installed

Much of the configuration of CPython is currently handled through C
level global variables:

    Py_BytesWarningFlag (-b)
    Py_DebugFlag (-d option)
    Py_InspectFlag (-i option, PYTHONINSPECT)
    Py_InteractiveFlag (property of stdin, cannot be overridden)
    Py_OptimizeFlag (-O option, PYTHONOPTIMIZE)
    Py_DontWriteBytecodeFlag (-B option, PYTHONDONTWRITEBYTECODE)
    Py_NoUserSiteDirectory (-s option, PYTHONNOUSERSITE)
    Py_NoSiteFlag (-S option)
    Py_UnbufferedStdioFlag (-u, PYTHONUNBUFFEREDIO)
    Py_VerboseFlag (-v option, PYTHONVERBOSE)

For the above variables, the conversion of command line options and
environment variables to C global variables is handled by Py_Main, so
each embedding application must set those appropriately in order to
change them from their defaults.

Some configuration can only be provided as OS level environment
variables:

    PYTHONSTARTUP
    PYTHONCASEOK
    PYTHONIOENCODING

The Py_InitializeEx() API also accepts a boolean flag to indicate
whether or not CPython's signal handlers should be installed.

Finally, some interactive behaviour (such as printing the introductory
banner) is triggered only when standard input is reported as a terminal
connection by the operating system.

TBD: Document how the "-x" option is handled (skips processing of the
first comment line in the main script)

Also see detailed sequence of operations notes at[12].

References

Copyright

This document has been placed in the public domain.

[1] Problems with PYTHONIOENCODING in Blender
(http://bugs.python.org/issue16129)

[2] Proposed CLI option for isolated mode
(http://bugs.python.org/issue16499)

[3] Adding to sys.path on the command line
(https://mail.python.org/pipermail/python-ideas/2010-October/008299.html)
(https://mail.python.org/pipermail/python-ideas/2012-September/016128.html)

[4] Control sys.path[0] initialisation
(http://bugs.python.org/issue13475)

[5] Enabling code coverage in subprocesses when testing
(http://bugs.python.org/issue14803)

[6] CPython interpreter initialization notes
(http://wiki.python.org/moin/CPythonInterpreterInitialization)

[7] BitBucket Sandbox
(https://bitbucket.org/ncoghlan/cpython_sandbox/compare/pep432_modular_bootstrap..default#commits)

[8] Proposed CLI option for isolated mode
(http://bugs.python.org/issue16499)

[9] *nix getpath implementation
(http://hg.python.org/cpython/file/default/Modules/getpath.c)

[10] Windows getpath implementation
(http://hg.python.org/cpython/file/default/PC/getpathp.c)

[11] Site module documentation
(http://docs.python.org/3/library/site.html)

[12] CPython interpreter initialization notes
(http://wiki.python.org/moin/CPythonInterpreterInitialization)


text:
PEP: 220 Title: Coroutines, Generators, Continuations Author: Gordon
McMillan<EMAIL_ADDRESS>Status: Rejected Type: Informational
Created: 14-Aug-2000 Post-History:

Abstract

Demonstrates why the changes described in the stackless PEP are
desirable. A low-level continuations module exists. With it, coroutines
and generators and "green" threads can be written. A higher level module
that makes coroutines and generators easy to create is desirable (and
being worked on). The focus of this PEP is on showing how coroutines,
generators, and green threads can simplify common programming problems.

license: Public Domain


text:
PEP: 243 Title: Module Repository Upload Mechanism Author: Sean
Reifschneider<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track Topic:
Packaging Content-Type: text/x-rst Created: 18-Mar-2001 Python-Version:
2.1 Post-History: 20-Mar-2001, 24-Mar-2001

Abstract

For a module repository system (such as Perl's CPAN) to be successful,
it must be as easy as possible for module authors to submit their work.
An obvious place for this submit to happen is in the Distutils tools
after the distribution archive has been successfully created. For
example, after a module author has tested their software (verifying the
results of setup.py sdist), they might type setup.py sdist --submit.
This would flag Distutils to submit the source distribution to the
archive server for inclusion and distribution to the mirrors.

This PEP only deals with the mechanism for submitting the software
distributions to the archive, and does not deal with the actual
archive/catalog server.

Upload Process

The upload will include the Distutils PKG-INFO meta-data information (as
specified in PEP 241), the actual software distribution, and other
optional information. This information will be uploaded as a multi-part
form encoded the same as a regular HTML file upload request. This form
is posted using ENCTYPE="multipart/form-data" encoding (1867).

The upload will be made to the host "www.python.org" on port 80/tcp
(POST http://www.python.org:80/pypi). The form will consist of the
following fields:

-   distribution -- The file containing the module software (for
    example, a .tar.gz or .zip file).
-   distmd5sum -- The MD5 hash of the uploaded distribution, encoded in
    ASCII representing the hexadecimal representation of the digest
    (for byte in digest: s = s + ('%02x' % ord(byte))).
-   pkginfo (optional) -- The file containing the distribution meta-data
    (as specified in PEP 241). Note that if this is not included, the
    distribution file is expected to be in .tar format (gzipped and
    bzipped compressed are allowed) or .zip format, with a PKG-INFO file
    in the top-level directory it extracts (package-1.00/PKG-INFO).
-   infomd5sum (required if pkginfo field is present) -- The MD5 hash of
    the uploaded meta-data, encoded in ASCII representing the
    hexadecimal representation of the digest
    (for byte in digest: s = s + ('%02x' % ord(byte))).
-   platform (optional) -- A string representing the target platform for
    this distribution. This is only for binary distributions. It is
    encoded as
    <os_name>-<os_version>-<platform architecture>-<python version>.
-   signature (optional) -- A OpenPGP-compatible <2440> signature of the
    uploaded distribution as signed by the author. This may be used by
    the cataloging system to automate acceptance of uploads.
-   protocol_version -- A string indicating the protocol version that
    the client supports. This document describes protocol version "1".

Return Data

The status of the upload will be reported using HTTP non-standard (X-*)
headers. The X-Swalow-Status header may have the following values:

-   SUCCESS -- Indicates that the upload has succeeded.
-   FAILURE -- The upload is, for some reason, unable to be processed.
-   TRYAGAIN -- The server is unable to accept the upload at this time,
    but the client should try again at a later time. Potential causes of
    this are resource shortages on the server, administrative down-time,
    etc...

Optionally, there may be a X-Swalow-Reason header which includes a
human-readable string which provides more detailed information about the
X-Swalow-Status.

If there is no X-Swalow-Status header, or it does not contain one of the
three strings above, it should be treated as a temporary failure.

Example:

    >>> f = urllib.urlopen('http://www.python.org:80/pypi')
    >>> s = f.headers['x-swalow-status']
    >>> s = s + ': ' + f.headers.get('x-swalow-reason', '<None>')
    >>> print s
    FAILURE: Required field "distribution" missing.

Sample Form

The upload client must submit the page in the same form as Netscape
Navigator version 4.76 for Linux produces when presented with the
following form:

Upload file

    <FORM NAME="fileupload" METHOD="POST" ACTION="pypi"
        ENCTYPE="multipart/form-data">
    <INPUT TYPE="file" NAME="distribution">

    <INPUT TYPE="text" NAME="distmd5sum">

    <INPUT TYPE="file" NAME="pkginfo">

    <INPUT TYPE="text" NAME="infomd5sum">

    <INPUT TYPE="text" NAME="platform">

    <INPUT TYPE="text" NAME="signature">

    <INPUT TYPE="hidden" NAME="protocol_version" VALUE="1">

    <INPUT TYPE="SUBMIT" VALUE="Upload">
    </FORM>

Platforms

The following are valid os names:

    aix beos debian dos freebsd hpux mac macos mandrake netbsd
    openbsd qnx redhat solaris suse windows yellowdog

The above include a number of different types of distributions of Linux.
Because of versioning issues these must be split out, and it is expected
that when it makes sense for one system to use distributions made on
other similar systems, the download client will make the distinction.

Version is the official version string specified by the vendor for the
particular release. For example, "2000" and "nt" (Windows), "9.04"
(HP-UX), "7.0" (RedHat, Mandrake).

The following are valid architectures:

    alpha hppa ix86 powerpc sparc ultrasparc

Status

I currently have a proof-of-concept client and server implemented. I
plan to have the Distutils patches ready for the 2.1 release. Combined
with Andrew's PEP 241 for specifying distribution meta-data, I hope to
have a platform which will allow us to gather real-world data for
finalizing the catalog system for the 2.2 release.

Copyright

This document has been placed in the public domain.


text:
PEP: 546 Title: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7
Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner
<EMAIL_ADDRESS>Cory Benfield<EMAIL_ADDRESS>BDFL-Delegate:
Benjamin Peterson<EMAIL_ADDRESS>Status: Rejected Type: Standards
Track Content-Type: text/x-rst Created: 30-May-2017 Python-Version: 2.7
Post-History: 23-May-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-June/148301.html

Abstract

Backport the ssl.MemoryBIO and ssl.SSLObject classes from Python 3 to
Python 2.7 to enhance the overall security of Python 2.7.

Rejection Notice

This PEP is rejected, see Withdraw PEP 546? Backport ssl.MemoryBIO and
ssl.SSLObject to Python 2.7 discussion for the rationale.

Rationale

While Python 2.7 is getting closer to its end-of-support date (scheduled
for 2020), it is still used on production systems and the Python
community is still responsible for its security. This PEP will help
facilitate the future adoption of PEP 543 across all supported Python
versions, which will improve security for both Python 2 and Python 3
users.

This PEP does NOT propose a general exception for backporting new
features to Python 2.7 - every new feature proposed for backporting will
still need to be justified independently. In particular, it will need to
be explained why relying on an independently updated backport on the
Python Package Index instead is not an acceptable solution.

PEP 543

PEP 543 defines a new TLS API for Python which would enhance Python
security by giving Python applications access to the native TLS
implementations on Windows and macOS, instead of using OpenSSL. A side
effect is that it gives access to the system trust store and
certificates installed locally by system administrators, enabling Python
applications to use "company certificates" without having to modify each
application and so to correctly validate TLS certificates (instead of
having to ignore or bypass TLS certificate validation).

For practical reasons, Cory Benfield would like to first implement an
I/O-less class similar to ssl.MemoryBIO and ssl.SSLObject for PEP 543,
and to provide a second class based on the first one to use sockets or
file descriptors. This design would help to structure the code to
support more backends and simplify testing and auditing, as well as
implementation. Later, optimized classes using directly sockets or file
descriptors may be added for performance.

While PEP 543 defines an API, the PEP would only make sense if it comes
with at least one complete and good implementation. The first
implementation would ideally be based on the ssl module of the Python
standard library, as this is shipped to all users by default and can be
used as a fallback implementation in the absence of anything more
targeted.

If this backport is not performed, the only baseline implementation that
could be used would be pyOpenSSL. This is problematic, however, because
of the interaction with pip, which is shipped with CPython on all
supported versions.

requests, pip and ensurepip

There are plans afoot to look at moving Requests to a more event-loop-y
model. The Requests team does not feel at this time it is possible to
abandon support for Python 2.7, so doing so would require using either
Twisted or Tornado, or writing their own asynchronous abstraction.

For asynchronous code, a MemoryBIO provides substantial advantages over
using a wrapped socket. It reduces the amount of buffering that must be
done, works on IOCP-based reactors as well as select/poll based ones,
and also greatly simplifies the reactor and implementation code. For
this reason, Requests is disinclined to use a wrapped-socket-based
implementation. In the absence of a backport to Python 2.7, Requests is
required to use the same solution that Twisted does: namely, a mandatory
dependency on pyOpenSSL.

The pip program has to embed all its dependencies for practical reasons:
namely, that it cannot rely on any other installation method being
present. Since pip depends on requests, it means that it would have to
embed a copy of pyOpenSSL. That would imply substantial usability pain
to install pip. Currently, pip doesn't support embedding C extensions
which must be compiled on each platform and so require a C compiler.

Since Python 2.7.9, Python embeds a copy of pip both for default
installation and for use in virtual environments via the new ensurepip
module. If pip ends up bundling PyOpenSSL, then CPython will end up
bundling PyOpenSSL. Only backporting ssl.MemoryBIO and ssl.SSLObject
would avoid the need to embed pyOpenSSL, and would fix the bootstrap
issue (python -> ensurepip -> pip -> requests -> MemoryBIO).

This situation is less problematic than the barrier to adoption of PEP
543, as naturally Requests does not have to move to an event loop model
before it drops support for Python 2.7. However, it does make it painful
for Requests (and pip) to embrace both asyncio and the async and await
keywords for as long as it continues to support Python 2.

Other Benefits

Adopting this PEP would have other smaller ecosystem benefits. For
example, Twisted would be able to reduce its dependency on third-party C
extensions. Additionally, the PyOpenSSL development team would like to
sunset the module, and this backport would free them up to do so in a
graceful manner without leaving their users in the lurch.

Each of these fringe benefits, while small, also provides value to the
wider Python ecosystem.

Concerns

There are some concerns that people have about this backport.

What About Old Python 2?

A number of the Python 2 users in the world are not keeping pace with
Python 2 releases. This is most usually because they are using LTS
releases that are not keeping pace with the minor releases of Python 2.
These users would not be able to use the MemoryBIO, and so projects
concerned with Python 2 compatibility may be unable to rely on the
MemoryBIO being present on most of their user's systems.

This concern is reasonable. How critical it is depends on the likelihood
of current users of Python 2 migrating to Python 3, or just trying to
use the most recent Python 2 release. Put another way, at some point
libraries will want to drop Python 2 support: the question is only
whether a significant majority of their Python 2 users have moved to
whatever Python 2 release contains this backport before they do so.

Ultimately, the authors of this PEP believe that the burden of this
backport is sufficiently minimal to justify backporting despite this
concern. If it turns out that migration to newer 2.7 releases is too
slow, then the value of the work will be minimal, but if the migration
to newer 2.7 releases is anything like reasonable then there will be
substantial value gained.

Changes

Add MemoryBIO and SSLObject classes to the ssl module of Python 2.7.

The code will be backported and adapted from the master branch (Python
3).

The backport also significantly reduced the size of the Python 2/Python
3 difference of the _ssl module, which make maintenance easier.

Links

-   PEP 543
-   [backport] ssl.MemoryBIO: Implementation of this PEP written by Alex
    Gaynor (first version written at October 2014)
-   PEP 466

Discussions

-   [Python-Dev] Backport ssl.MemoryBIO on Python 2.7? (May 2017)

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 712 Title: Adding a "converter" parameter to dataclasses.field
Author: Joshua Cannon<EMAIL_ADDRESS>Sponsor: Eric V. Smith
<eric at trueblade.com> Discussions-To:
https://discuss.python.org/t/pep-712-adding-a-converter-parameter-to-dataclasses-field/26126
Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
01-Jan-2023 Python-Version: 3.13 Post-History: 27-Dec-2022, 19-Jan-2023,
23-Apr-2023, Resolution:
https://discuss.python.org/t/pep-712-adding-a-converter-parameter-to-dataclasses-field/26126/98

Rejection Notice

The reasons for the 2024 Steering Council rejection include:

-   We did not find evidence of a strong consensus that this feature was
    needed in the standard library, despite some proponents arguing in
    favor in order to reduce their dependence on third party packages.
    For those who need such functionality, we think those existing third
    party libraries such as attrs and Pydantic (which the PEP
    references) are acceptable alternatives.
-   This feature seems to us like an accumulation of what could be
    considered more cruft in the standard library, leading us ever
    farther away from the “simple” use cases that dataclasses are ideal
    for.
-   Reading the “How to Teach This” section of the PEP gives us pause
    that the pitfalls and gotchas are significant, with a heightened
    confusion and complexity outweighing any potential benefits.
-   The PEP seems more focused toward helping type checkers than people
    using the library.

Abstract

PEP 557 added dataclasses to the Python stdlib. PEP 681 added
~py3.11:typing.dataclass_transform to help type checkers understand
several common dataclass-like libraries, such as attrs, Pydantic, and
object relational mapper (ORM) packages such as SQLAlchemy and Django.

A common feature other libraries provide over the standard library
implementation is the ability for the library to convert arguments given
at initialization time into the types expected for each field using a
user-provided conversion function.

Therefore, this PEP adds a converter parameter to dataclasses.field
(along with the requisite changes to dataclasses.Field and
~py3.11:typing.dataclass_transform) to specify the function to use to
convert the input value for each field to the representation to be
stored in the dataclass.

Motivation

There is no existing, standard way for dataclasses or third-party
dataclass-like libraries to support argument conversion in a
type-checkable way. To work around this limitation, library
authors/users are forced to choose to:

-   Opt-in to a custom Mypy plugin. These plugins help Mypy understand
    the conversion semantics, but not other tools.
-   Shift conversion responsibility onto the caller of the dataclass
    constructor. This can make constructing certain dataclasses
    unnecessarily verbose and repetitive.
-   Provide a custom __init__ which declares "wider" parameter types and
    converts them when setting the appropriate attribute. This not only
    duplicates the typing annotations between the converter and
    __init__, but also opts the user out of many of the features
    dataclasses provides.
-   Provide a custom __init__ but without meaningful type annotations
    for the parameter types requiring conversion.

None of these choices are ideal.

Rationale

Adding argument conversion semantics is useful and beneficial enough
that most dataclass-like libraries provide support. Adding this feature
to the standard library means more users are able to opt-in to these
benefits without requiring third-party libraries. Additionally
third-party libraries are able to clue type-checkers into their own
conversion semantics through added support in
~py3.11:typing.dataclass_transform, meaning users of those libraries
benefit as well.

Specification

New converter parameter

This specification introduces a new parameter named converter to the
dataclasses.field function. If provided, it represents a single-argument
callable used to convert all values when assigning to the associated
attribute.

For frozen dataclasses, the converter is only used inside a
dataclass-synthesized __init__ when setting the attribute. For
non-frozen dataclasses, the converter is used for all attribute
assignment (E.g. obj.attr = value), which includes assignment of default
values.

The converter is not used when reading attributes, as the attributes
should already have been converted.

Adding this parameter also implies the following changes:

-   A converter attribute will be added to dataclasses.Field.
-   converter will be added to ~py3.11:typing.dataclass_transform's list
    of supported field specifier parameters.

Example

    def str_or_none(x: Any) -> str | None:
      return str(x) if x is not None else None

    @dataclasses.dataclass
    class InventoryItem:
        # `converter` as a type (including a GenericAlias).
        id: int = dataclasses.field(converter=int)
        skus: tuple[int, ...] = dataclasses.field(converter=tuple[int, ...])
        # `converter` as a callable.
        vendor: str | None = dataclasses.field(converter=str_or_none))
        names: tuple[str, ...] = dataclasses.field(
          converter=lambda names: tuple(map(str.lower, names))
        )  # Note that lambdas are supported, but discouraged as they are untyped.

        # The default value is also converted; therefore the following is not a
        # type error.
        stock_image_path: pathlib.PurePosixPath = dataclasses.field(
          converter=pathlib.PurePosixPath, default="assets/unknown.png"
        )

        # Default value conversion extends to `default_factory`;
        # therefore the following is also not a type error.
        shelves: tuple = dataclasses.field(
          converter=tuple, default_factory=list
        )

    item1 = InventoryItem(
      "1",
      [234, 765],
      None,
      ["PYTHON PLUSHIE", "FLUFFY SNAKE"]
    )
    # item1's repr would be (with added newlines for readability):
    #   InventoryItem(
    #     id=1,
    #     skus=(234, 765),
    #     vendor=None,
    #     names=('PYTHON PLUSHIE', 'FLUFFY SNAKE'),
    #     stock_image_path=PurePosixPath('assets/unknown.png'),
    #     shelves=()
    #   )

    # Attribute assignment also participates in conversion.
    item1.skus = [555]
    # item1's skus attribute is now (555,).

Impact on typing

A converter must be a callable that accepts a single positional
argument, and the parameter type corresponding to this positional
argument provides the type of the the synthesized __init__ parameter
associated with the field.

In other words, the argument provided for the converter parameter must
be compatible with Callable[[T], X] where T is the input type for the
converter and X is the output type of the converter.

Type-checking default and default_factory

Because default values are unconditionally converted using converter, if
an argument for converter is provided alongside either default or
default_factory, the type of the default (the default argument if
provided, otherwise the return value of default_factory) should be
checked using the type of the single argument to the converter callable.

Converter return type

The return type of the callable must be a type that's compatible with
the field's declared type. This includes the field's type exactly, but
can also be a type that's more specialized (such as a converter
returning a list[int] for a field annotated as list, or a converter
returning an int for a field annotated as int | str).

Indirection of allowable argument types

One downside introduced by this PEP is that knowing what argument types
are allowed in the dataclass' __init__ and during attribute assignment
is not immediately obvious from reading the dataclass. The allowable
types are defined by the converter.

This is true when reading code from source, however typing-related aides
such as typing.reveal_type and "IntelliSense" in an IDE should make it
easy to know exactly what types are allowed without having to read any
source code.

Backward Compatibility

These changes don't introduce any compatibility problems since they only
introduce opt-in new features.

Security Implications

There are no direct security concerns with these changes.

How to Teach This

Documentation and examples explaining the new parameter and behavior
will be added to the relevant sections of the docs site (primarily on
dataclasses) and linked from the What's New document.

The added documentation/examples will also cover the "common pitfalls"
that users of converters are likely to encounter. Such pitfalls include:

-   Needing to handle None/sentinel values.
-   Needing to handle values that are already of the correct type.
-   Avoiding lambdas for converters, as the synthesized __init__
    parameter's type will become Any.
-   Forgetting to convert values in the bodies of user-defined __init__
    in frozen dataclasses.
-   Forgetting to convert values in the bodies of user-defined
    __setattr__ in non-frozen dataclasses.

Additionally, potentially confusing pattern matching semantics should be
covered:

    @dataclass
    class Point:
        x: int = field(converter=int)
        y: int

    match Point(x="0", y=0):
        case Point(x="0", y=0):  # Won't be matched
            ...
        case Point():  # Will be matched
            ...
        case _:
            ...

However it's worth noting this behavior is true of any type that does
conversion in its initializer, and type-checkers should be able to catch
this pitfall:

    match int("0"):
      case int("0"):  # Won't be matched
          ...
      case _:  # Will be matched
          ...

Reference Implementation

The attrs library already includes a converter parameter exhibiting the
same converter semantics (converting in the initializer and on attribute
setting) when using the @define class decorator.

CPython support is implemented on a branch in the author's fork.

Rejected Ideas

Just adding "converter" to typing.dataclass_transform's field_specifiers

The idea of isolating this addition to
~py3.11:typing.dataclass_transform was briefly discussed on Typing-SIG
where it was suggested to broaden this to dataclasses more generally.

Additionally, adding this to dataclasses ensures anyone can reap the
benefits without requiring additional libraries.

Not converting default values

There are pros and cons with both converting and not converting default
values. Leaving default values as-is allows type-checkers and dataclass
authors to expect that the type of the default matches the type of the
field. However, converting default values has three large advantages:

1.  Consistency. Unconditionally converting all values that are assigned
    to the attribute, involves fewer "special rules" that users must
    remember.
2.  Simpler defaults. Allowing the default value to have the same type
    as user-provided values means dataclass authors get the same
    conveniences as their callers.
3.  Compatibility with attrs. Attrs unconditionally uses the converter
    to convert default values.

Automatic conversion using the field's type

One idea could be to allow the type of the field specified (e.g. str or
int) to be used as a converter for each argument provided. Pydantic's
data conversion has semantics which appear to be similar to this
approach.

This works well for fairly simple types, but leads to ambiguity in
expected behavior for complex types such as generics. E.g. For
tuple[int, ...] it is ambiguous if the converter is supposed to simply
convert an iterable to a tuple, or if it is additionally supposed to
convert each element type to int. Or for int | None, which isn't
callable.

Deducing the attribute type from the return type of the converter

Another idea would be to allow the user to omit the attribute's type
annotation if providing a field with a converter argument. Although this
would reduce the common repetition this PEP introduces (e.g.
x: str = field(converter=str)), it isn't clear how to best support this
while maintaining the current dataclass semantics (namely, that the
attribute order is preserved for things like the synthesized __init__,
or dataclasses.fields). This is because there isn't an easy way in
Python (today) to get the annotation-only attributes interspersed with
un-annotated attributes in the order they were defined.

A sentinel annotation could be applied (e.g. x: FromConverter = ...),
however this breaks a fundamental assumption of type annotations.

Lastly, this is feasible if all fields (including those without a
converter) were assigned to dataclasses.field, which means the class'
own namespace holds the order, however this trades repetition of
type+converter with repetition of field assignment. The end result is no
gain or loss of repetition, but with the added complexity of dataclasses
semantics.

This PEP doesn't suggest it can't or shouldn't be done. Just that it
isn't included in this PEP.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 370 Title: Per user site-packages directory Author: Christian
Heimes<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 11-Jan-2008 Python-Version: 2.6, 3.0
Post-History:

Abstract

This PEP proposes a new a per user site-packages directory to allow
users the local installation of Python packages in their home directory.

Rationale

Current Python versions don't have a unified way to install packages
into the home directory of a user (except for Mac Framework builds).
Users are either forced to ask the system administrator to install or
update a package for them or to use one of the many workarounds like
Virtual Python[1], Working Env[2] or Virtual Env[3].

It's not the goal of the PEP to replace the tools or to implement
isolated installations of Python. It only implements the most common use
case of an additional site-packages directory for each user.

The feature can't be implemented using the environment variable
PYTHONPATH. The env var just inserts a new directory to the beginning of
sys.path but it doesn't parse the pth files in the directory. A full
blown site-packages path is required for several applications and Python
eggs.

Specification

site directory (site-packages)

  A directory in sys.path. In contrast to ordinary directories the pth
  files in the directory are processed, too.

user site directory

  A site directory inside the users' home directory. A user site
  directory is specific to a Python version. The path contains the
  version number (major and minor only).

  Unix (including Mac OS X)

      ~/.local/lib/python2.6/site-packages

  Windows

      %APPDATA%/Python/Python26/site-packages

user data directory

  Usually the parent directory of the user site directory. It's meant
  for Python version specific data like config files, docs, images and
  translations.

  Unix (including Mac)

      ~/.local/lib/python2.6

  Windows

      %APPDATA%/Python/Python26

user base directory

  It's located inside the user's home directory. The user site and use
  config directory are inside the base directory. On some systems the
  directory may be shared with 3rd party apps.

  Unix (including Mac)

      ~/.local

  Windows

      %APPDATA%/Python

user script directory

  A directory for binaries and scripts.[4] It's shared across Python
  versions and the destination directory for scripts.

  Unix (including Mac)

      ~/.local/bin

  Windows

      %APPDATA%/Python/Scripts

Windows Notes

On Windows the Application Data directory (aka APPDATA) was chosen
because it is the most designated place for application data. Microsoft
recommends that software doesn't write to USERPROFILE[5] and
My Documents is not suited for application data, either.[6] The code
doesn't query the Win32 API, instead it uses the environment variable
%APPDATA%.

The application data directory is part of the roaming profile. In
networks with domain logins the application data may be copied from and
to the a central server. This can slow down log-in and log-off. Users
can keep the data on the server by e.g. setting PYTHONUSERBASE to the
value "%HOMEDRIVE%%HOMEPATH%Applicata Data". Users should consult their
local administrator for more information.[7]

Unix Notes

On Unix ~/.local was chosen in favor over ~/.python because the
directory is already used by several other programs in analogy to
/usr/local.[8][9]

Mac OS X Notes

On Mac OS X Python uses ~/.local directory as well.[10] Framework builds
of Python include ~/Library/Python/2.6/site-packages as an additional
search path.

Implementation

The site module gets a new method adduserpackage() which adds the
appropriate directory to the search path. The directory is not added if
it doesn't exist when Python is started. However the location of the
user site directory and user base directory is stored in an internal
variable for distutils.

The user site directory is added before the system site directories but
after Python's search paths and PYTHONPATH. This setup allows the user
to install a different version of a package than the system
administrator but it prevents the user from accidentally overwriting a
stdlib module. Stdlib modules can still be overwritten with PYTHONPATH.

For security reasons the user site directory is not added to sys.path
when the effective user id or group id is not equal to the process uid /
gid[11]. It's an additional barrier against code injection into suid
apps. However Python suid scripts must always use the -E and -s option
or users can sneak in their own code.

The user site directory can be suppressed with a new option -s or the
environment variable PYTHONNOUSERSITE. The feature can be disabled
globally by setting site.ENABLE_USER_SITE to the value False. It must be
set by editing site.py. It can't be altered in sitecustomize.py or
later.

The path to the user base directory can be overwritten with the
environment variable PYTHONUSERBASE. The default location is used when
PYTHONUSERBASE is not set or empty.

distutils.command.install (setup.py install) gets a new argument --user
to install packages in the user site directory. The required directories
are created on demand.

distutils.command.build_ext (setup.py build_ext) gets a new argument
--user which adds the include/ and lib/ directories in the user base
directory to the search paths for header files and libraries. It also
adds the lib/ directory to rpath.

The site module gets two arguments --user-base and --user-site to print
the path to the user base or user site directory to the standard output.
The feature is intended for scripting, e.g.
./configure --prefix $(python2.5 -m site --user-base)

distutils.sysconfig will get methods to access the private variables of
site. (not yet implemented)

The Windows updater needs to be updated, too. It should create a menu
item which opens the user site directory in a new explorer windows.

Backwards Compatibility

TBD

Reference Implementation

A reference implementation is available in the bug tracker.[12]

Copyright

This document has been placed in the public domain.

References

[6] Initial suggestion for a per user site-packages directory
<EMAIL_ADDRESS>
[1] Virtual Python
http://peak.telecommunity.com/DevCenter/EasyInstall#creating-a-virtual-python

[2] Working Env https://pypi.org/project/workingenv.py/
https://ianbicking.org/archive/workingenv-revisited.html

[3] Virtual Env https://pypi.org/project/virtualenv/

[4] Discussion about the bin directory
https://mail.python.org/pipermail/python-dev/2008-January/076162.html

[5] MSDN: CSIDL https://learn.microsoft.com/en/windows/win32/shell/csidl

[6] APPDATA discussion
https://mail.python.org/pipermail/python-dev/2008-January/075993.html

[7] Roaming profile on Windows
https://mail.python.org/pipermail/python-dev/2008-January/076256.html

[8] Suggestion of ~/.local/
https://mail.python.org/pipermail/python-dev/2008-January/075985.html

[9] freedesktop.org XGD basedir specs mentions ~/.local
https://www.freedesktop.org/wiki/Specifications/basedir-spec/

[10] ~/.local for Mac and usercustomize file
https://mail.python.org/pipermail/python-dev/2008-January/076236.html

[11] Security concerns and -s option
https://mail.python.org/pipermail/python-dev/2008-January/076130.html

[12] reference implementation
https://github.com/python/cpython/issues/46132
http://svn.python.org/view/sandbox/trunk/pep370


text:
PEP: 3120 Title: Using UTF-8 as the default source encoding Version:
$Revision$ Last-Modified: $Date$ Author: Martin von Löwis
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 15-Apr-2007 Python-Version: 3.0 Post-History:

Specification

This PEP proposes to change the default source encoding from ASCII to
UTF-8. Support for alternative source encodings (PEP 263) continues to
exist; an explicit encoding declaration takes precedence over the
default.

A Bit of History

In Python 1, the source encoding was unspecified, except that the source
encoding had to be a superset of the system's basic execution character
set (i.e. an ASCII superset, on most systems). The source encoding was
only relevant for the lexis itself (bytes representing letters for
keywords, identifiers, punctuation, line breaks, etc). The contents of a
string literal was copied literally from the file on source.

In Python 2.0, the source encoding changed to Latin-1 as a side effect
of introducing Unicode. For Unicode string literals, the characters were
still copied literally from the source file, but widened on a
character-by-character basis. As Unicode gives a fixed interpretation to
code points, this algorithm effectively fixed a source encoding, at
least for files containing non-ASCII characters in Unicode literals.

PEP 263 identified the problem that you can use only those Unicode
characters in a Unicode literal which are also in Latin-1, and
introduced a syntax for declaring the source encoding. If no source
encoding was given, the default should be ASCII. For compatibility with
Python 2.0 and 2.1, files were interpreted as Latin-1 for a transitional
period. This transition ended with Python 2.5, which gives an error if
non-ASCII characters are encountered and no source encoding is declared.

Rationale

With PEP 263, using arbitrary non-ASCII characters in a Python file is
possible, but tedious. One has to explicitly add an encoding
declaration. Even though some editors (like IDLE and Emacs) support the
declarations of PEP 263, many editors still do not (and never will);
users have to explicitly adjust the encoding which the editor assumes on
a file-by-file basis.

When the default encoding is changed to UTF-8, adding non-ASCII text to
Python files becomes easier and more portable: On some systems, editors
will automatically choose UTF-8 when saving text (e.g. on Unix systems
where the locale uses UTF-8). On other systems, editors will guess the
encoding when reading the file, and UTF-8 is easy to guess. Yet other
editors support associating a default encoding with a file extension,
allowing users to associate .py with UTF-8.

For Python 2, an important reason for using non-UTF-8 encodings was that
byte string literals would be in the source encoding at run-time,
allowing then to output them to a file or render them to the user as-is.
With Python 3, all strings will be Unicode strings, so the original
encoding of the source will have no impact at run-time.

Implementation

The parser needs to be changed to accept bytes > 127 if no source
encoding is specified; instead of giving an error, it needs to check
that the bytes are well-formed UTF-8 (decoding is not necessary, as the
parser converts all source code to UTF-8, anyway).

IDLE needs to be changed to use UTF-8 as the default encoding.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: