text:
PEP: 573 Title: Module State Access from C Extension Methods Version:
$Revision$ Last-Modified: $Date$ Author: Petr Viktorin
<EMAIL_ADDRESS>Alyssa Coghlan<EMAIL_ADDRESS>Eric Snow
<EMAIL_ADDRESS>Marcel Plch<EMAIL_ADDRESS>BDFL-Delegate: Stefan Behnel Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
02-Jun-2016 Python-Version: 3.9 Post-History:

Abstract

This PEP proposes to add a way for CPython extension methods to access
context, such as the state of the modules they are defined in.

This will allow extension methods to use direct pointer dereferences
rather than PyState_FindModule for looking up module state, reducing or
eliminating the performance cost of using module-scoped state over
process global state.

This fixes one of the remaining roadblocks for adoption of PEP 3121
(Extension module initialization and finalization) and PEP 489
(Multi-phase extension module initialization).

While this PEP takes an additional step towards fully solving the
problems that PEP 3121 and PEP 489 started tackling, it does not attempt
to resolve all remaining concerns. In particular, access to the module
state from slot methods (nb_add, etc) is not solved.

Terminology

Process-Global State

C-level static variables. Since this is very low-level memory storage,
it must be managed carefully.

Per-module State

State local to a module object, allocated dynamically as part of a
module object's initialization. This isolates the state from other
instances of the module (including those in other subinterpreters).

Accessed by PyModule_GetState().

Static Type

A type object defined as a C-level static variable, i.e. a compiled-in
type object.

A static type needs to be shared between module instances and has no
information of what module it belongs to. Static types do not have
__dict__ (although their instances might).

Heap Type

A type object created at run time.

Defining Class

The defining class of a method (either bound or unbound) is the class on
which the method was defined. A class that merely inherits the method
from its base is not the defining class.

For example, int is the defining class of True.to_bytes, True.__floor__
and int.__repr__.

In C, the defining class is the one defined with the corresponding
tp_methods or "tp slots"[1] entry. For methods defined in Python, the
defining class is saved in the __class__ closure cell.

C-API

The "Python/C API" as described in Python documentation. CPython
implements the C-API, but other implementations exist.

Rationale

PEP 489 introduced a new way to initialize extension modules, which
brings several advantages to extensions that implement it:

-   The extension modules behave more like their Python counterparts.
-   The extension modules can easily support loading into pre-existing
    module objects, which paves the way for extension module support for
    runpy or for systems that enable extension module reloading.
-   Loading multiple modules from the same extension is possible, which
    makes it possible to test module isolation (a key feature for proper
    sub-interpreter support) from a single interpreter.

The biggest hurdle for adoption of PEP 489 is allowing access to module
state from methods of extension types. Currently, the way to access this
state from extension methods is by looking up the module via
PyState_FindModule (in contrast to module level functions in extension
modules, which receive a module reference as an argument). However,
PyState_FindModule queries the thread-local state, making it relatively
costly compared to C level process global access and consequently
deterring module authors from using it.

Also, PyState_FindModule relies on the assumption that in each
subinterpreter, there is at most one module corresponding to a given
PyModuleDef. This assumption does not hold for modules that use PEP
489's multi-phase initialization, so PyState_FindModule is unavailable
for these modules.

A faster, safer way of accessing module-level state from extension
methods is needed.

Background

The implementation of a Python method may need access to one or more of
the following pieces of information:

-   The instance it is called on (self)
-   The underlying function
-   The defining class, i. e. the class the method was defined in
-   The corresponding module
-   The module state

In Python code, the Python-level equivalents may be retrieved as:

    import sys

    class Foo:
        def meth(self):
            instance = self
            module_globals = globals()
            module_object = sys.modules[__name__]  # (1)
            underlying_function = Foo.meth         # (1)
            defining_class = Foo                   # (1)
            defining_class = __class__             # (2)

Note

The defining class is not type(self), since type(self) might be a
subclass of Foo.

The statements marked (1) implicitly rely on name-based lookup via the
function's __globals__: either the Foo attribute to access the defining
class and Python function object, or __name__ to find the module object
in sys.modules.

In Python code, this is feasible, as __globals__ is set appropriately
when the function definition is executed, and even if the namespace has
been manipulated to return a different object, at worst an exception
will be raised.

The __class__ closure, (2), is a safer way to get the defining class,
but it still relies on __closure__ being set appropriately.

By contrast, extension methods are typically implemented as normal C
functions. This means that they only have access to their arguments and
C level thread-local and process-global states. Traditionally, many
extension modules have stored their shared state in C-level process
globals, causing problems when:

-   running multiple initialize/finalize cycles in the same process
-   reloading modules (e.g. to test conditional imports)
-   loading extension modules in subinterpreters

PEP 3121 attempted to resolve this by offering the PyState_FindModule
API, but this still has significant problems when it comes to extension
methods (rather than module level functions):

-   it is markedly slower than directly accessing C-level process-global
    state
-   there is still some inherent reliance on process global state that
    means it still doesn't reliably handle module reloading

It's also the case that when looking up a C-level struct such as module
state, supplying an unexpected object layout can crash the interpreter,
so it's significantly more important to ensure that extension methods
receive the kind of object they expect.

Proposal

Currently, a bound extension method (PyCFunction or
PyCFunctionWithKeywords) receives only self, and (if applicable) the
supplied positional and keyword arguments.

While module-level extension functions already receive access to the
defining module object via their self argument, methods of extension
types don't have that luxury: they receive the bound instance via self,
and hence have no direct access to the defining class or the module
level state.

The additional module level context described above can be made
available with two changes. Both additions are optional; extension
authors need to opt in to start using them:

-   Add a pointer to the module to heap type objects.

-   Pass the defining class to the underlying C function.

    In CPython, the defining class is readily available at the time the
    built-in method object (PyCFunctionObject) is created, so it can be
    stored in a new struct that extends PyCFunctionObject.

The module state can then be retrieved from the module object via
PyModule_GetState.

Note that this proposal implies that any type whose methods need to
access per-module state must be a heap type, rather than a static type.
This is necessary to support loading multiple module objects from a
single extension: a static type, as a C-level global, has no information
about which module object it belongs to.

Slot methods

The above changes don't cover slot methods, such as tp_iter or nb_add.

The problem with slot methods is that their C API is fixed, so we can't
simply add a new argument to pass in the defining class. Two possible
solutions have been proposed to this problem:

-   Look up the class through walking the MRO. This is potentially
    expensive, but will be usable if performance is not a problem (such
    as when raising a module-level exception).
-   Storing a pointer to the defining class of each slot in a separate
    table, __typeslots__[2]. This is technically feasible and fast, but
    quite invasive.

Modules affected by this concern also have the option of using
thread-local state or PEP 567 context variables as a caching mechanism,
or else defining their own reload-friendly lookup caching scheme.

Solving the issue generally is deferred to a future PEP.

Specification

Adding module references to heap types

A new factory method will be added to the C-API for creating modules:

    PyObject* PyType_FromModuleAndSpec(PyObject *module,
                                       PyType_Spec *spec,
                                       PyObject *bases)

This acts the same as PyType_FromSpecWithBases, and additionally
associates the provided module object with the new type. (In CPython,
this will set ht_module described below.)

Additionally, an accessor, PyObject * PyType_GetModule(PyTypeObject *)
will be provided. It will return the type's associated module if one is
set, otherwise it will set TypeError and return NULL. When given a
static type, it will always set TypeError and return NULL.

To implement this in CPython, the PyHeapTypeObject struct will get a new
member, PyObject *ht_module, that will store a pointer to the associated
module. It will be NULL by default and should not be modified after the
type object is created.

The ht_module member will not be inherited by subclasses; it needs to be
set using PyType_FromSpecWithBases for each individual type that needs
it.

Usually, creating a class with ht_module set will create a reference
cycle involving the class and the module. This is not a problem, as
tearing down modules is not a performance-sensitive operation, and
module-level functions typically also create reference cycles. The
existing "set all module globals to None" code that breaks function
cycles through f_globals will also break the new cycles through
ht_module.

Passing the defining class to extension methods

A new signature flag, METH_METHOD, will be added for use in
PyMethodDef.ml_flags. Conceptually, it adds defining_class to the
function signature. To make the initial implementation easier, the flag
can only be used as (METH_FASTCALL | METH_KEYWORDS | METH_METHOD). (It
can't be used with other flags like METH_O or bare METH_FASTCALL, though
it may be combined with METH_CLASS or METH_STATIC).

C functions for methods defined using this flag combination will be
called using a new C signature called PyCMethod:

    PyObject *PyCMethod(PyObject *self,
                        PyTypeObject *defining_class,
                        PyObject *const *args,
                        size_t nargsf,
                        PyObject *kwnames)

Additional combinations like (METH_VARARGS | METH_METHOD) may be added
in the future (or even in the initial implementation of this PEP).
However, METH_METHOD should always be an additional flag, i.e., the
defining class should only be passed in if needed.

In CPython, a new structure extending PyCFunctionObject will be added to
hold the extra information:

    typedef struct {
        PyCFunctionObject func;
        PyTypeObject *mm_class; /* Passed as 'defining_class' arg to the C func */
    } PyCMethodObject;

The PyCFunction implementation will pass mm_class into a PyCMethod C
function when it finds the METH_METHOD flag being set. A new macro
PyCFunction_GET_CLASS(cls) will be added for easier access to mm_class.

C methods may continue to use the other METH_* signatures if they do not
require access to their defining class/module. If METH_METHOD is not
set, casting to PyCMethodObject is invalid.

Argument Clinic

To support passing the defining class to methods using Argument Clinic,
a new converter called defining_class will be added to CPython's
Argument Clinic tool.

Each method may only have one argument using this converter, and it must
appear after self, or, if self is not used, as the first argument. The
argument will be of type PyTypeObject *.

When used, Argument Clinic will select
METH_FASTCALL | METH_KEYWORDS | METH_METHOD as the calling convention.
The argument will not appear in __text_signature__.

The new converter will initially not be compatible with __init__ and
__new__ methods, which cannot use the METH_METHOD convention.

Helpers

Getting to per-module state from a heap type is a very common task. To
make this easier, a helper will be added:

    void *PyType_GetModuleState(PyObject *type)

This function takes a heap type and on success, it returns pointer to
the state of the module that the heap type belongs to.

On failure, two scenarios may occur. When a non-type object, or a type
without a module is passed in, TypeError is set and NULL returned. If
the module is found, the pointer to the state, which may be NULL, is
returned without setting any exception.

Modules Converted in the Initial Implementation

To validate the approach, the _elementtree module will be modified
during the initial implementation.

Summary of API Changes and Additions

The following will be added to Python C-API:

  -   PyType_FromModuleAndSpec function
  -   PyType_GetModule function
  -   PyType_GetModuleState function
  -   METH_METHOD call flag
  -   PyCMethod function signature

The following additions will be added as CPython implementation details,
and won't be documented:

  -   PyCFunction_GET_CLASS macro
  -   PyCMethodObject struct
  -   ht_module member of _heaptypeobject
  -   defining_class converter in Argument Clinic

Backwards Compatibility

One new pointer is added to all heap types. All other changes are adding
new functions and structures, or changes to private implementation
details.

Implementation

An initial implementation is available in a Github repository[3]; a
patchset is at[4].

Possible Future Extensions

Slot methods

A way of passing defining class (or module state) to slot methods may be
added in the future.

A previous version of this PEP proposed a helper function that would
determine a defining class by searching the MRO for a class that defines
a slot to a particular function. However, this approach would fail if a
class is mutated (which is, for heap types, possible from Python code).
Solving this problem is left to future discussions.

Easy creation of types with module references

It would be possible to add a PEP 489 execution slot type to make
creating heap types significantly easier than calling
PyType_FromModuleAndSpec. This is left to a future PEP.

It may be good to add a good way to create static exception types from
the limited API. Such exception types could be shared between
subinterpreters, but instantiated without needing specific module state.
This is also left to possible future discussions.

Optimization

As proposed here, methods defined with the METH_METHOD flag only support
one specific signature.

If it turns out that other signatures are needed for performance
reasons, they may be added.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] https://docs.python.org/3/c-api/typeobj.html#tp-slots

[2] [Import-SIG] On singleton modules, heap types, and subinterpreters
(https://mail.python.org/pipermail/import-sig/2015-July/001035.html)

[3] https://github.com/Dormouse759/cpython/tree/pep-c-rebase_newer

[4] https://github.com/Dormouse759/cpython/compare/master...Dormouse759:pep-c-rebase_newer


text:
PEP: 453 Title: Explicit bootstrapping of pip in Python installations
Author: Donald Stufft<EMAIL_ADDRESS>Alyssa Coghlan
<EMAIL_ADDRESS>BDFL-Delegate: Martin von Löwis Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013,
23-Sep-2013, 29-Sep-2013, 13-Oct-2013, 20-Oct-2013 Resolution:
https://mail.python.org/pipermail/python-dev/2013-October/129810.html

Abstract

This PEP proposes that the Installing Python Modules guide in Python
2.7, 3.3 and 3.4 be updated to officially recommend the use of pip as
the default installer for Python packages, and that appropriate
technical changes be made in Python 3.4 to provide pip by default in
support of that recommendation.

PEP Acceptance

This PEP was accepted for inclusion in Python 3.4 by Martin von Löwis on
Tuesday 22nd October, 2013.

Issue 19347 has been created to track the implementation of this PEP.

Rationale

There are two related, but distinct rationales for the proposal in this
PEP. The first relates to the experience of new users, while the second
relates to better enabling the evolution of the broader Python packaging
ecosystem.

Improving the new user experience

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that do have a platform package manager, it is unlikely
to include every package that is available on the Python Package Index,
and even when a desired third-party package is available, the correct
name in the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to
get it, and how to install it. The effect of this is that third-party
Python projects are currently required to choose from a variety of
undesirable alternatives:

-   Assume the user already has a suitable cross-platform package
    manager installed.
-   Duplicate the instructions and tell their users how to install the
    package manager.
-   Completely forgo the use of dependencies to ease installation
    concerns for their users.

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then
beginning users may get a confusing error message when the installation
command doesn't work. Some operating systems may ease this pain by
providing a global hook that looks for commands that don't exist and
suggest an OS package they can install to make the command work, but
that only works on systems with platform package managers that include a
package that provides the relevant cross-platform installer command
(such as many major Linux distributions). No such assistance is
available for Windows and Mac OS X users, or more conservative Linux
distributions. The challenges of dealing with this problem for beginners
(who are often also completely new to programming, the use of command
line tools and editing system environment variables) are a regular
feature of feedback the core Python developers receive from professional
educators and others introducing new users to Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how
to install their own project then whenever these instructions need
updates they need updating by every project that has duplicated them.
This is particular problematic when there are multiple competing
installation tools available, and different projects recommend different
tools.

This specific problem can be partially alleviated by strongly promoting
pip as the default installer and recommending that other projects
reference pip's own bootstrapping instructions rather than duplicating
them. However the user experience created by this approach still isn't
particularly good (although there is an effort under way to create a
combined Windows installer for pip and its dependencies that should
improve matters on that platform, and Mac OS X and *nix platforms
generally have wget and hence the ability to easily download and run the
bootstrap scripts from the command line).

The projects that have decided to forgo dependencies altogether are
forced to either duplicate the efforts of other projects by inventing
their own solutions to problems or are required to simply include the
other projects in their own source trees. Both of these options present
their own problems either in duplicating maintenance work across the
ecosystem or potentially leaving users vulnerable to security issues
because the included code or duplicated efforts are not automatically
updated when upstream releases a new version.

By officially recommending and providing by default a specific
cross-platform package manager it will be easier for users trying to
install these third-party packages as well as easier for the people
distributing them as they should now be able to safely assume that most
users will have the appropriate installation tools available (or access
to clear instructions on how to obtain them). This is expected to become
more important in the future as the Wheel <427> package format
(deliberately) does not have a built in "installer" in the form of
setup.py so users wishing to install from a wheel file will want an
installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24
month feature release cycle, instead of using the general difficulty of
installing third-party packages as justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as zc.buildout, hashdist and
conda. So long as pip install <tool> works, then a standard
Python-specific installer provides a reasonably secure, cross platform
mechanism to get access to these utilities.

Enabling the evolution of the broader Python packaging ecosystem

As no new packaging standard can achieve widespread adoption without a
transition strategy that covers the versions of Python that are in
widespread current use (rather than merely future versions, like most
language features), the change proposed in this PEP is considered a
necessary step in the evolution of the Python packaging ecosystem

The broader community has embraced the Python Package Index as a
mechanism for distributing and installing Python software, but the
different concerns of language evolution and secure software
distribution mean that a faster feature release cycle that encompasses
older versions is needed to properly support the latter.

In addition, the core CPython development team have the luxury of
dropping support for earlier Python versions well before the rest of the
community, as downstream commercial redistributors pick up the task of
providing support for those versions to users that still need it, while
many third party libraries maintain compatibility with those versions as
long as they remain in widespread use.

This means that the current setup.py install based model for package
installation poses serious difficulties for the development and adoption
of new packaging standards, as, depending on how a project writes their
setup.py file, the installation command (along with other operations)
may end up invoking the standard library's distutils package.

As an indicator of how this may cause problems for the broader
ecosystem, consider that the feature set of distutils in Python 2.6 was
frozen in June 2008 (with the release of Python 2.6b1), while the
feature set of distutils in Python 2.7 was frozen in April 2010 (with
the release of Python 2.7b1).

By contrast, using a separate installer application like pip (which
ensures that even setup.py files that invoke distutils directly still
support the new packaging standards) makes it possible to support new
packaging standards in older versions of Python, just by upgrading pip
(which receives new feature releases roughly every 6 months). The
situation on older versions of Python is further improved by making it
easier for end users to install and upgrade newer build systems like
setuptools or improved PyPI upload utilities like twine.

It is not coincidental that this proposed model of using a separate
installer program with more metadata heavy and less active distribution
formats matches that used by most operating systems (including Windows
since the introduction of the installer service and the MSI file
format), as well as many other language specific installers.

For Python 2.6, this compatibility issue is largely limited to various
enterprise Linux distributions (and their downstream derivatives). These
distributions often have even slower update cycles than CPython, so they
offer full support for versions of Python that are considered "security
fix only" versions upstream (and sometimes may even be to the point
where the core development team no longer support them at all - you can
still get commercial support for Python 2.3 if you really need it!).

In practice, the fact that tools like wget and curl are readily
available on Linux systems, that most users of Python on Linux are
already familiar with the command line, and that most Linux
distributions ship with a default configuration that makes running
Python scripts easy, means that the existing pip bootstrapping
instructions for any *nix system are already quite straightforward. Even
if pip isn't provided by the system package manager, then using wget or
curl to retrieve the bootstrap script from www.pip-installer.org and
then running it is just a couple of shell commands that can easily be
copied and pasted as necessary.

Accordingly, for any version of Python on any *nix system, the need to
bootstrap pip in older versions isn't considered a major barrier to
adoption of new packaging standards, since it's just one more small
speedbump encountered by users of these long term stable releases. For
*nix systems, this PEP's formal endorsement of pip as the preferred
default packaging tool is seen as more important than the underlying
technical details involved in making pip available by default, since it
shifts the nature of the conversation between the developers of pip and
downstream repackagers of both pip and CPython.

For Python 2.7, on the other hand, the compatibility issue for adopting
new metadata standards is far more widespread, as it affects the
python.org binary installers for Windows and Mac OS X, as well as even
relatively fast moving *nix platforms.

Firstly, and unlike Python 2.6, Python 2.7 is still a fully supported
upstream version, and will remain so until the release of Python 2.7.9
(currently scheduled for May 2015), at which time it is expected to
enter the usual "security fix only" mode. That means there are at least
another 19 months where Python 2.7 is a deployment target for Python
applications that enjoys full upstream support. Even after the core
development team switches 2.7 to security release only mode in 2015,
Python 2.7 will likely remain a commercially supported legacy target out
beyond 2020.

While Python 3 already presents a compelling alternative over Python 2
for new Python applications and deployments without an existing
investment in Python 2 and without a dependency on specific Python 2
only third party modules (a set which is getting ever smaller over
time), it is going to take longer to create compelling business cases to
update existing Python 2.7 based infrastructure to Python 3, especially
in situations where the culture of automated testing is weak (or
nonexistent), making it difficult to effectively use the available
migration utilities.

While this PEP only proposes documentation changes for Python 2.7, once
pip has a Windows installer available, a separate PEP will be created
and submitted proposing the creation and distribution of aggregate
installers for future CPython 2.7 maintenance releases that combine the
CPython, pip and Python Launcher for Windows installers into a single
download (the separate downloads would still remain available - the
aggregate installers would be provided as a convenience, and as a clear
indication of the recommended operating environment for Python in
Windows systems).

Why pip?

pip has been chosen as the preferred default installer, as it is an
already popular tool that addresses several design and user experience
issues with its predecessor easy_install (these issues can't readily be
fixed in easy_install itself due to backwards compatibility concerns).
pip is also well suited to working within the bounds of a single Python
runtime installation (including associated virtual environments), which
is a desirable feature for a tool bundled with CPython.

Other tools like zc.buildout and conda are more ambitious in their aims
(and hence substantially better than pip at handling external binary
dependencies), so it makes sense for the Python ecosystem to treat them
more like platform package managers to interoperate with rather than as
the default cross-platform installation tool. This relationship is
similar to that between pip and platform package management systems like
apt and yum (which are also designed to handle arbitrary binary
dependencies).

Proposal Overview

This PEP proposes that the Installing Python Modules guide be updated to
officially recommend the use of pip as the default installer for Python
packages, rather than the current approach of recommending the direct
invocation of the setup.py install command.

However, to avoid recommending a tool that CPython does not provide, it
is further proposed that the pip package manager be made available by
default when installing CPython 3.4 or later and when creating virtual
environments using the standard library's venv module via the pyvenv
command line utility.

To support that end, this PEP proposes the inclusion of an ensurepip
bootstrapping module in Python 3.4, as well as automatic invocation of
that module from pyvenv and changes to the way Python installed scripts
are handled on Windows. Using a bootstrap module rather than providing
pip directly helps to clearly demarcate development responsibilities,
and to avoid inadvertently downgrading pip when updating CPython.

To provide clear guidance for new users of Python that may not be
starting with the latest release, this PEP also proposes that the
"Installing Python Modules" guides in Python 2.7 and 3.3 be updated to
recommend installing and using pip, rather than invoking distutils
directly. It does not propose backporting any of the code changes that
are being proposed for Python 3.4.

Finally, the PEP also strongly recommends that CPython redistributors
and other Python implementations ensure that pip is available by
default, or at the very least, explicitly document the fact that it is
not included.

This PEP does not propose making pip (or any dependencies) directly
available as part of the standard library. Instead, pip will be a
bundled application provided along with CPython for the convenience of
Python users, but subject to its own development life cycle and able to
be upgraded independently of the core interpreter and standard library.

Explicit bootstrapping mechanism

An additional module called ensurepip will be added to the standard
library whose purpose is to install pip and any of its dependencies into
the appropriate location (most commonly site-packages). It will expose a
callable named bootstrap() as well as offer direct execution via
python -m ensurepip.

The bootstrap will not contact PyPI, but instead rely on a private copy
of pip stored inside the standard library. Accordingly, only options
related to the installation location will be supported (--user, --root,
etc).

It is considered desirable that users be strongly encouraged to use the
latest available version of pip, in order to take advantage of the
ongoing efforts to improve the security of the PyPI based ecosystem, as
well as benefiting from the efforts to improve the speed, reliability
and flexibility of that ecosystem.

In order to satisfy this goal of providing the most recent version of
pip by default, the private copy of pip will be updated in CPython
maintenance releases, which should align well with the 6-month cycle
used for new pip releases.

Security considerations

The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of CPython for end users that do
not subsequently run the command pip install --upgrade pip.

The installers will contain all the components of a fully functioning
version of Python, including the pip installer. The installation process
will not require network access, and will not rely on trusting the
security of the network connection established between pip and the
Python package index.

Only users that choose to use pip to communicate with PyPI will need to
pay attention to the additional security considerations that come with
doing so.

However, the core CPython team will still assist with reviewing and
resolving at least the certificate update management issue currently
affecting the requests project (and hence pip), and may also be able to
offer assistance in resolving other identified security concerns [1].

Reliability considerations

By including the bootstrap as part of the standard library (rather than
solely as a feature of the binary installers), the correct operation of
the bootstrap command can be easily tested using the existing CPython
buildbot infrastructure rather than adding significantly to the testing
burden for the installers themselves.

Implementation strategy

To ensure there is no need for network access when installing Python or
creating virtual environments, the ensurepip module will, as an
implementation detail, include a complete private copy of pip and its
dependencies which will be used to extract pip and install it into the
target environment. It is important to stress that this private copy of
pip is only an implementation detail and it should not be relied on or
assumed to exist beyond the public capabilities exposed through the
ensurepip module (and indirectly through venv).

There is not yet a reference ensurepip implementation. The existing
get-pip.py bootstrap script demonstrates an earlier variation of the
general concept, but the standard library version would take advantage
of the improved distribution capabilities offered by the CPython
installers to include private copies of pip and setuptools as wheel
files (rather than as embedded base64 encoded data), and would not try
to contact PyPI (instead installing directly from the private wheel
files).

Rather than including separate code to handle the bootstrapping, the
ensurepip module will manipulate sys.path appropriately to allow the
wheel files to be used to install themselves, either into the current
Python installation or into a virtual environment (as determined by the
options passed to the bootstrap command).

It is proposed that the implementation be carried out in five separate
steps (all steps after the first two are independent of each other and
can be carried out in any order):

-   the first step would update the "Installing Python Modules"
    documentation to recommend the use of pip and reference the pip
    team's instructions for downloading and installing it. This change
    would be applied to Python 2.7, 3.3, and 3.4.
-   the ensurepip module and the private copies of the most recently
    released versions of pip and setuptools would be added to Python 3.4
    and the 3.4 "Installing Python Modules" documentation updated
    accordingly.
-   the CPython Windows installer would be updated to offer the new pip
    installation option for Python 3.4.
-   the CPython Mac OS X installer would be updated to offer the new pip
    installation option for Python 3.4.
-   the venv module and pyvenv command would be updated to make use of
    ensurepip in Python 3.4
-   the PATH handling on Windows would be updated for Python 3.4+

Integration timeline

If this PEP is accepted, the proposed time frame for integration of pip
into the CPython release is as follows:

-   as soon as possible after the release of 3.4.0 alpha 4
    -   Documentation updated and ensurepip implemented based on a
        pre-release version of pip 1.5.
    -   All other proposed functional changes for Python 3.4
        implemented, including the installer updates to invoke
        ensurepip.
-   by November 20th (3 days prior to the scheduled date of 3.4.0 beta
    1)
    -   ensurepip updated to use a pip 1.5 release candidate.
    -   PEP 101 updated to cover ensuring the bundled version of pip is
        up to date.
-   by November 24th (scheduled date of 3.4.0 beta 1)
    -   As with any other new feature, all proposed functional changes
        for Python 3.4 must be implemented prior to the beta feature
        freeze.
-   by December 29th (1 week prior to the scheduled date of 3.4.0 beta
    2)
    -   requests certificate management issue resolved
    -   ensurepip updated to the final release of pip 1.5, or a
        subsequent maintenance release (including a suitably updated
        vendored copy of requests)

(See PEP 429 for the current official scheduled dates of each release.
Dates listed above are accurate as of October 20th, 2013.)

If there is no final or maintenance release of pip 1.5 with a suitable
updated version of requests available by one week before the scheduled
Python 3.4 beta 2 release, then implementation of this PEP will be
deferred to Python 3.5. Note that this scenario is considered unlikely
-the tentative date for the pip 1.5 release is currently December 1st.

In future CPython releases, this kind of coordinated scheduling
shouldn't be needed: the CPython release manager will be able to just
update to the latest released version of pip. However, in this case,
some fixes are needed in pip in order to allow the bundling to work
correctly, and the certificate update mechanism for requests needs to be
improved, so the pip 1.5 release cycle needs to be properly aligned with
the CPython 3.4 beta releases.

Proposed CLI

The proposed CLI is based on a subset of the existing pip install
options:

    Usage:
      python -m ensurepip [options]

    General Options:
      -h, --help          Show help.
      -v, --verbose       Give more output. Option is additive, and can be used up to 3 times.
      -V, --version       Show the pip version that would be extracted and exit.
      -q, --quiet         Give less output.

    Installation Options:
      -U, --upgrade       Upgrade pip and dependencies, even if already installed
      --user              Install using the user scheme.
      --root <dir>        Install everything relative to this alternate root directory.

In most cases, end users won't need to use this CLI directly, as pip
should have been installed automatically when installing Python or when
creating a virtual environment. However, it is formally documented as a
public interface to support at least these known use cases:

-   Windows and Mac OS X installations where the "Install pip" option
    was not chosen during installation
-   any installation where the user previously ran "pip uninstall pip"

Users that want to retrieve the latest version from PyPI, or otherwise
need more flexibility, can then invoke the extracted pip appropriately.

Proposed module API

The proposed ensurepip module API consists of the following two
functions:

    def version():
        """
        Returns a string specifying the bundled version of pip.
        """

    def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
        """
        Bootstrap pip into the current Python installation (or the given root
        directory).
        """

Invocation from the CPython installers

The CPython Windows and Mac OS X installers will each gain a new option:

-   Install pip (the default Python package management utility)?

This option will be checked by default.

If the option is checked, then the installer will invoke the following
command with the just installed Python:

    python -m ensurepip --upgrade

This ensures that, by default, installing or updating CPython will
ensure that the installed version of pip is at least as recent as the
one included with that version of CPython. If a newer version of pip has
already been installed then python -m ensurepip --upgrade will simply
return without doing anything.

Installing from source

Just as the prebuilt binary installers will be updated to run
python -m ensurepip by default, a similar change will be made to the
make install and make altinstall commands of the source distribution.
The directory settings in the sysconfig module should ensure the pip
components are automatically installed to the expected locations.

ensurepip itself (including the private copy of pip and its
dependencies) will always be installed normally (as it is a regular part
of the standard library), but an option will be provided to skip the
invocation of ensurepip.

This means that even installing from source will provide pip by default,
but redistributors provide pip by other means (or not providing it at
all) will still be able to opt out of installing it using ensurepip.

Changes to virtual environments

Python 3.3 included a standard library approach to virtual Python
environments through the venv module. Since its release it has become
clear that very few users have been willing to use this feature
directly, in part due to the lack of an installer present by default
inside of the virtual environment. They have instead opted to continue
using the virtualenv package which does include pip installed by
default.

To make the venv more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating
it. This will allow people the same convenience inside of the virtual
environment as this PEP provides outside of it as well as bringing the
venv module closer to feature parity with the external virtualenv
package, making it a more suitable replacement.

To handle cases where a user does not wish to have pip bootstrapped into
their virtual environment a --without-pip option will be added.

The venv.EnvBuilder and venv.create APIs will be updated to accept one
new parameter: with_pip (defaulting to False).

The new default for the module API is chosen for backwards compatibility
with the current behaviour (as it is assumed that most invocation of the
venv module happens through third part tools that likely will not want
pip installed without explicitly requesting it), while the default for
the command line interface is chosen to try to ensure pip is available
in most virtual environments without additional action on the part of
the end user.

As this change will only benefit Python 3.4 and later versions, the
third-party virtualenv project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.

Documentation

The "Installing Python Modules" section of the standard library
documentation in Python 2.7, 3.3 and 3.4 will be updated to recommend
the use of the pip installer, either provided by default in Python 3.4
or retrieved and installed by the user in Python 2.7 or 3.3. It will
give a brief description of the most common commands and options, but
delegate to the externally maintained pip documentation for the full
details.

In Python 3.4, the pyvenv and venv documentation will also be updated to
reference the revised module installation guide.

The existing content of the module installation guide will be retained
in all versions, but under a new "Invoking distutils directly"
subsection.

Bundling CA certificates with CPython

The ensurepip implementation will include the pip CA bundle along with
the rest of pip. This means CPython effectively includes a CA bundle
that is used solely by pip after it has been extracted.

This is considered preferable to relying solely on the system
certificate stores, as it ensures that pip will behave the same across
all supported versions of Python, even those prior to Python 3.4 that
cannot access the system certificate store on Windows.

Automatic installation of setuptools

pip currently depends on setuptools to handle metadata generation during
the build process, along with some other features. While work is ongoing
to reduce or eliminate this dependency, it is not clear if that work
will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
ensurepip will include a private copy of setuptools (in addition to the
private copy of ensurepip). python -m ensurepip will then install the
private copy in addition to installing pip itself.

However, this behavior is officially considered an implementation
detail. Other projects which explicitly require setuptools must still
provide an appropriate dependency declaration, rather than assuming
setuptools will always be installed alongside pip.

The private copy of setuptools will be removed from ensurepip once it is
no longer needed. This is likely to be at the point when get-pip.py
stops installing setuptools by default. As long as setuptools is needed,
it will be a completely unmodified copy of the latest upstream
setuptools release, including the easy_install script if the upstream
setuptools continues to include it. The installation of easy_install
along with pip isn't considered desirable, but installing a broken
setuptools would be worse. This problem will naturally resolve itself
once the pip developers have managed to eliminate their dependency on
setuptools and the private copy of setuptools can be removed entirely
from CPython.

Updating the private copy of pip

In order to keep up with evolutions in packaging as well as providing
users with as recent version a possible the ensurepip module will be
regularly updated to the latest versions of everything it bootstraps.

After each new pip release, and again during the preparation for any
release of Python (including feature releases), a script, provided as
part of the implementation for this PEP, will be run to ensure the
private copies stored in the CPython source repository have been updated
to the latest versions.

Updating the ensurepip module API and CLI

Like venv and pyvenv, the ensurepip module API and CLI will be governed
by the normal rules for the standard library: no new features are
permitted in maintenance releases.

However, the embedded components may be updated as noted above, so the
extracted pip may offer additional functionality in maintenance
releases.

Uninstallation

No changes are proposed to the CPython uninstallation process by this
PEP. The bootstrapped pip will be installed the same way as any other
pip installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be left
behind after uninstallation, since those files won't be associated with
the Python MSI installer.

While the case can be made for the CPython installers clearing out these
directories automatically, changing that behaviour is considered outside
the scope of this PEP.

Script Execution on Windows

While the Windows installer was updated in Python 3.3 to optionally make
python available on the PATH, no such change was made to include the
script installation directory returned by sysconfig.get_path("scripts").

Accordingly, in addition to adding the option to extract and install pip
during installation, this PEP proposes that the Windows installer in
Python 3.4 and later be updated to also add the path returned by
sysconfig.get_path("scripts") to the Windows PATH when the PATH
modification option is enabled during installation

Note that this change will only be available in Python 3.4 and later.

This means that, for Python 3.3, the most reliable way to invoke pip
globally on Windows (without tinkering manually with PATH) will still
remain py -m pip (or py -3 -m pip to select the Python 3 version if both
Python 2 and 3 are installed) rather than simply calling pip. This works
because Python 3.3 provides the Python Launcher for Windows (and the
associated py command) by default.

For Python 2.7 and 3.2, the most reliable mechanism will be to install
the Python Launcher for Windows using the standalone installer and then
use py -m pip as noted above.

Adding the scripts directory to the system PATH will mean that pip works
reliably in the "only one Python installation on the system PATH" case,
with py -m pip, pipX, or pipX.Y needed only to select a non-default
version in the parallel installation case (and outside a virtual
environment). This change should also make the pyvenv command
substantially easier to invoke on Windows, along with all scripts
installed by pip, easy_install and similar tools.

While the script invocations on recent versions of Python will run
through the Python launcher for Windows, this shouldn't cause any
issues, as long as the Python files in the Scripts directory correctly
specify a Python version in their shebang line or have an adjacent
Windows executable (as easy_install and pip do).

Recommendations for Downstream Distributors

A common source of Python installations are through downstream
distributors such as the various Linux Distributions[2][3][4], OSX
package managers[5][6][7], and commercial Python
redistributors[8][9][10]. In order to provide a consistent,
user-friendly experience to all users of Python regardless of how they
obtained Python this PEP recommends and asks that downstream
distributors:

-   Ensure that whenever Python is installed pip is either installed or
    is otherwise made readily available to end users.
    -   For redistributors using binary installers, this may take the
        form of optionally executing the ensurepip bootstrap during
        installation, similar to the CPython installers.
    -   For redistributors using package management systems, it may take
        the form of separate packages with dependencies on each other so
        that installing the Python package installs the pip package and
        installing the pip package installs the Python package.
    -   Another reasonable way to implement this is to package pip
        separately but ensure that there is some sort of global hook
        that will recommend installing the separate pip package when a
        user executes pip without it being installed. Systems that
        choose this option should ensure that the ensurepip module still
        installs pip directly when invoked inside a virtual environment,
        but may modify the module in the system Python installation to
        redirect to the platform provided mechanism when installing pip
        globally.
-   Even if pip is made available globally by other means, do not remove
    the ensurepip module in Python 3.4 or later.
    -   ensurepip will be required for automatic installation of pip
        into virtual environments by the venv module.
    -   This is similar to the existing virtualenv package for which
        many downstream distributors have already made exception to the
        common "debundling" policy.
    -   This does mean that if pip needs to be updated due to a security
        issue, so does the private copy in the ensurepip bootstrap
        module
    -   However, altering the private copy of pip to remove the embedded
        CA certificate bundle and rely on the system CA bundle instead
        is a reasonable change.
-   Ensure that all features of this PEP continue to work with any
    modifications made to the redistributed version of Python.
    -   Checking the version of pip that will be bootstrapped using
        python -m ensurepip --version or ensurepip.version().
    -   Installation of pip into a global or virtual python environment
        using python -m ensurepip or ensurepip.bootstrap().
    -   pip install --upgrade pip in a global installation should not
        affect any already created virtual environments (but is
        permitted to affect future virtual environments, even though it
        will not do so when using the standard implementation of
        ensurepip).
    -   pip install --upgrade pip in a virtual environment should not
        affect the global installation.
-   Migrate build systems to utilize pip and Wheel <427> wherever
    feasible and avoid directly invoking setup.py.
    -   This will help ensure a smoother and more timely migration to
        improved metadata formats as the Python packaging ecosystem
        continues to evolve.

In the event that a Python redistributor chooses not to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream pip
based installation instructions into something appropriate for the
platform.

Other Python implementations are also encouraged to follow these
guidelines where applicable.

Policies & Governance

The maintainers of the bootstrapped software and the CPython core team
will work together in order to address the needs of both. The
bootstrapped software will still remain external to CPython and this PEP
does not include CPython subsuming the development responsibilities or
design decisions of the bootstrapped software. This PEP aims to decrease
the burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of pip, setuptools, PyPI, virtualenv and
other related projects.

Backwards Compatibility

The public API and CLI of the ensurepip module itself will fall under
the typical backwards compatibility policy of Python for its standard
library. The externally developed software that this PEP bundles does
not.

Most importantly, this means that the bootstrapped version of pip may
gain new features in CPython maintenance releases, and pip continues to
operate on its own 6 month release cycle rather than CPython's 18-24
month cycle.

Security Releases

Any security update that affects the ensurepip module will be shared
prior to release with the Python Security Response Team
(security@python.org). The PSRT will then decide if the reported issue
warrants a security release of CPython with an updated private copy of
pip.

Licensing

pip is currently licensed as 1 Clause BSD, and it contains code taken
from other projects. Additionally this PEP will include setuptools until
such time as pip no longer requires it. The licenses for these appear in
the table below.

+-------------------+--------------+
| Project           | License      |
+===================+==============+
| requests          |   Apache 2.0 |
+-------------------+--------------+
| six               | 1 Clause BSD |
+-------------------+--------------+
| html5lib          | 1 Clause BSD |
+-------------------+--------------+
| distlib           |   PSF        |
+-------------------+--------------+
| colorama          | 3 Clause BSD |
+-------------------+--------------+
| Mozilla CA Bundle |   LGPL       |
+-------------------+--------------+
| setuptools        |   PSF        |
+-------------------+--------------+

All of these licenses should be compatible with the PSF license.
Additionally it is unclear if a CA Bundle is copyrightable material and
thus if it needs or can be licensed at all.

Appendix: Rejected Proposals

Changing the name of the scripts directory on Windows

Earlier versions of this PEP proposed changing the name of the script
installation directory on Windows from "Scripts" to "bin" in order to
improve the cross-platform consistency of the virtual environments
created by pyvenv.

However, Paul Moore determined that this change was likely backwards
incompatible with cross-version Windows installers created with previous
versions of Python, so the change has been removed from this PEP [11].

Including ensurepip in Python 2.7, and 3.3

Earlier versions of this PEP made the case that the challenges of
getting pip bootstrapped for new users posed a significant enough
barrier to Python's future growth that it justified adding ensurepip as
a new feature in the upcoming Python 2.7 and 3.3 maintenance releases.

While the proposal to provide pip with Python 3.4 was universally
popular, this part of the proposal was highly controversial and
ultimately rejected by MvL as BDFL-Delegate.

Accordingly, the proposal to backport ensurepip to Python 2.7 and 3.3
has been removed from this PEP in favour of creating a Windows installer
for pip and a possible future PEP suggesting creation of an aggregate
installer for Python 2.7 that combines CPython 2.7, pip and the Python
Launcher for Windows.

Automatically contacting PyPI when bootstrapping pip

Earlier versions of this PEP called the bootstrapping module getpip and
defaulted to downloading and installing pip from PyPI, with the private
copy used only as a fallback option or when explicitly requested.

This resulted in several complex edge cases, along with difficulties in
defining a clean API and CLI for the bootstrap module. It also
significantly altered the default trust model for the binary installers
published on python.org, as end users would need to explicitly opt-out
of trusting the security of the PyPI ecosystem (rather than opting in to
it by explicitly invoking pip following installation).

As a result, the PEP was simplified to the current design, where the
bootstrapping always uses the private copy of pip. Contacting PyPI is
now always an explicit separate step, with direct access to the full pip
interface.

Removing the implicit attempt to access PyPI also made it feasible to
invoke ensurepip by default when installing from a custom source build.

Implicit bootstrap

PEP 439, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake pip command that when executed would
implicitly bootstrap and install pip if it does not already exist. This
has been rejected because it is too "magical". It hides from the end
user when exactly the pip command will be installed or that it is being
installed at all. It also does not provide any recommendations or
considerations towards downstream packagers who wish to manage the
globally installed pip through the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions
issues, if a user inadvertently attempted to bootstrap pip without write
access to the appropriate installation directories.

Including pip directly in the standard library

Similar to this PEP is the proposal of just including pip in the
standard library. This would ensure that Python always includes pip and
fixes all of the end user facing problems with not having pip present by
default. This has been rejected because we've learned, through the
inclusion and history of distutils in the standard library, that losing
the ability to update the packaging tools independently can leave the
tooling in a state of constant limbo. Making it unable to ever
reasonably evolve in a time frame that actually affects users as any new
features will not be available to the general population for years.

Allowing the packaging tools to progress separately from the Python
release and adoption schedules allows the improvements to be used by all
members of the Python community and not just those able to live on the
bleeding edge of Python releases.

There have also been issues in the past with the "dual maintenance"
problem if a project continues to be maintained externally while also
having a fork maintained in the standard library. Since external
maintenance of pip will always be needed to support earlier Python
versions, the proposed bootstrapping mechanism will becoming the
explicit responsibility of the CPython core developers (assisted by the
pip developers), while pip issues reported to the CPython tracker will
be migrated to the pip issue tracker. There will no doubt still be some
user confusion over which tracker to use, but hopefully less than has
been seen historically when including complete public copies of
third-party projects in the standard library.

The approach described in this PEP also avoids some technical issues
related to handling CPython maintenance updates when pip has been
independently updated to a more recent version. The proposed pip-based
bootstrapping mechanism handles that automatically, since pip and the
system installer never get into a fight about who owns the pip
installation (it is always managed through pip, either directly, or
indirectly via the ensurepip bootstrap module).

Finally, the separate bootstrapping step means it is also easy to avoid
installing pip at all if end users so desire. This is often the case if
integrators are using system packages to handle installation of
components written in multiple languages using a common set of tools.

Defaulting to --user installation

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behavior would be
surprising (as it differs from the default behavior of pip itself) and
is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).

References

-   Discussion thread 1 (distutils-sig)
-   Discussion thread 2 (distutils-sig)
-   Discussion thread 3 (python-dev)
-   Discussion thread 4 (python-dev)
-   Discussion thread 5 (python-dev)

Copyright

This document has been placed in the public domain.

[1] pip/requests certificate management concerns

[2] Ubuntu

[3] Debian

[4] Fedora

[5] Homebrew

[6] MacPorts

[7] Fink

[8] Anaconda

[9] ActivePython

[10] Enthought Canopy

[11] Windows installer compatibility concerns

license: Public Domain


text:
PEP: 256 Title: Docstring Processing System Framework Version:
$Revision$ Last-Modified: $Date$ Author: David Goodger
<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Rejected
Type: Standards Track Content-Type: text/x-rst Created: 01-Jun-2001
Post-History: 13-Jun-2001

Rejection Notice

This proposal seems to have run out of steam.

Abstract

Python lends itself to inline documentation. With its built-in docstring
syntax, a limited form of Literate Programming is easy to do in Python.
However, there are no satisfactory standard tools for extracting and
processing Python docstrings. The lack of a standard toolset is a
significant gap in Python's infrastructure; this PEP aims to fill the
gap.

The issues surrounding docstring processing have been contentious and
difficult to resolve. This PEP proposes a generic Docstring Processing
System (DPS) framework, which separates out the components (program and
conceptual), enabling the resolution of individual issues either through
consensus (one solution) or through divergence (many). It promotes
standard interfaces which will allow a variety of plug-in components
(input context readers, markup parsers, and output format writers) to be
used.

The concepts of a DPS framework are presented independently of
implementation details.

Road Map to the Docstring PEPs

There are many aspects to docstring processing. The "Docstring PEPs"
have broken up the issues in order to deal with each of them in
isolation, or as close as possible. The individual aspects and
associated PEPs are as follows:

-   Docstring syntax. PEP 287, "reStructuredText Docstring Format",
    proposes a syntax for Python docstrings, PEPs, and other uses.
-   Docstring semantics consist of at least two aspects:
    -   Conventions: the high-level structure of docstrings. Dealt with
        in PEP 257, "Docstring Conventions".
    -   Methodology: rules for the informational content of docstrings.
        Not addressed.
-   Processing mechanisms. This PEP (PEP 256) outlines the high-level
    issues and specification of an abstract docstring processing system
    (DPS). PEP 258, "Docutils Design Specification", is an overview of
    the design and implementation of one DPS under development.
-   Output styles: developers want the documentation generated from
    their source code to look good, and there are many different ideas
    about what that means. PEP 258 touches on "Stylist Transforms". This
    aspect of docstring processing has yet to be fully explored.

By separating out the issues, we can form consensus more easily (smaller
fights ;-), and accept divergence more readily.

Rationale

There are standard inline documentation systems for some other
languages. For example, Perl has POD ("Plain Old Documentation") and
Java has Javadoc, but neither of these mesh with the Pythonic way. POD
syntax is very explicit, but takes after Perl in terms of readability.
Javadoc is HTML-centric; except for "@field" tags, raw HTML is used for
markup. There are also general tools such as Autoduck and Web (Tangle &
Weave), useful for multiple languages.

There have been many attempts to write auto-documentation systems for
Python (not an exhaustive list):

-   Marc-Andre Lemburg's doc.py
-   Daniel Larsson's pythondoc & gendoc
-   Doug Hellmann's HappyDoc
-   Laurence Tratt's Crystal (no longer available on the web)
-   Ka-Ping Yee's pydoc (pydoc.py is now part of the Python standard
    library; see below)
-   Tony Ibbs' docutils (Tony has donated this name to the Docutils
    project)
-   Edward Loper's STminus formalization and related efforts

These systems, each with different goals, have had varying degrees of
success. A problem with many of the above systems was over-ambition
combined with inflexibility. They provided a self-contained set of
components: a docstring extraction system, a markup parser, an internal
processing system and one or more output format writers with a fixed
style. Inevitably, one or more aspects of each system had serious
shortcomings, and they were not easily extended or modified, preventing
them from being adopted as standard tools.

It has become clear (to this author, at least) that the "all or nothing"
approach cannot succeed, since no monolithic self-contained system could
possibly be agreed upon by all interested parties. A modular component
approach designed for extension, where components may be multiply
implemented, may be the only chance for success. Standard
inter-component APIs will make the DPS components comprehensible without
requiring detailed knowledge of the whole, lowering the barrier for
contributions, and ultimately resulting in a rich and varied system.

Each of the components of a docstring processing system should be
developed independently. A "best of breed" system should be chosen,
either merged from existing systems, and/or developed anew. This system
should be included in Python's standard library.

PyDoc & Other Existing Systems

PyDoc became part of the Python standard library as of release 2.1. It
extracts and displays docstrings from within the Python interactive
interpreter, from the shell command line, and from a GUI window into a
web browser (HTML). Although a very useful tool, PyDoc has several
deficiencies, including:

-   In the case of the GUI/HTML, except for some heuristic hyperlinking
    of identifier names, no formatting of the docstrings is done. They
    are presented within <p><small><tt> tags to avoid unwanted line
    wrapping. Unfortunately, the result is not attractive.
-   PyDoc extracts docstrings and structural information (class
    identifiers, method signatures, etc.) from imported module objects.
    There are security issues involved with importing untrusted code.
    Also, information from the source is lost when importing, such as
    comments, "additional docstrings" (string literals in non-docstring
    contexts; see PEP 258), and the order of definitions.

The functionality proposed in this PEP could be added to or used by
PyDoc when serving HTML pages. The proposed docstring processing
system's functionality is much more than PyDoc needs in its current
form. Either an independent tool will be developed (which PyDoc may or
may not use), or PyDoc could be expanded to encompass this functionality
and become the docstring processing system (or one such system). That
decision is beyond the scope of this PEP.

Similarly for other existing docstring processing systems, their authors
may or may not choose compatibility with this framework. However, if
this framework is accepted and adopted as the Python standard,
compatibility will become an important consideration in these systems'
future.

Specification

The docstring processing system framework is broken up as follows:

1.  Docstring conventions. Documents issues such as:

    -   What should be documented where.
    -   First line is a one-line synopsis.

    PEP 257 documents some of these issues.

2.  Docstring processing system design specification. Documents issues
    such as:

    -   High-level spec: what a DPS does.
    -   Command-line interface for executable script.
    -   System Python API.
    -   Docstring extraction rules.
    -   Readers, which encapsulate the input context.
    -   Parsers.
    -   Document tree: the intermediate internal data structure. The
        output of the Parser and Reader, and the input to the Writer all
        share the same data structure.
    -   Transforms, which modify the document tree.
    -   Writers for output formats.
    -   Distributors, which handle output management (one file, many
        files, or objects in memory).

    These issues are applicable to any docstring processing system
    implementation. PEP 258 documents these issues.

3.  Docstring processing system implementation.

4.  Input markup specifications: docstring syntax. PEP 287 proposes a
    standard syntax.

5.  Input parser implementations.

6.  Input context readers ("modes": Python source code, PEP, standalone
    text file, email, etc.) and implementations.

7.  Stylists: certain input context readers may have associated stylists
    which allow for a variety of output document styles.

8.  Output formats (HTML, XML, TeX, DocBook, info, etc.) and writer
    implementations.

Components 1, 2/3/5, and 4 are the subject of individual companion PEPs.
If there is another implementation of the framework or syntax/parser,
additional PEPs may be required. Multiple implementations of each of
components 6 and 7 will be required; the PEP mechanism may be overkill
for these components.

Project Web Site

A SourceForge project has been set up for this work at
http://docutils.sourceforge.net/.

References and Footnotes

Copyright

This document has been placed in the public domain.

Acknowledgements

This document borrows ideas from the archives of the Python Doc-SIG.
Thanks to all members past & present.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:


text:
PEP: 547 Title: Running extension modules using the -m option Version:
$Revision$ Last-Modified: $Date$ Author: Marcel Plch
<EMAIL_ADDRESS>Petr Viktorin<EMAIL_ADDRESS>Status:
Deferred Type: Standards Track Content-Type: text/x-rst Created:
25-May-2017 Python-Version: 3.7 Post-History:

Deferral Notice

Cython -- the most important use case for this PEP and the only explicit
one -- is not ready for multi-phase initialization yet. It keeps global
state in C-level static variables. See discussion at Cython issue 1923.

The PEP is deferred until the situation changes.

Abstract

This PEP proposes implementation that allows built-in and extension
modules to be executed in the __main__ namespace using the PEP 489
multi-phase initialization.

With this, a multi-phase initialization enabled module can be run using
following command:

    $ python3 -m _testmultiphase
    This is a test module named __main__.

Motivation

Currently, extension modules do not support all functionality of Python
source modules. Specifically, it is not possible to run extension
modules as scripts using Python's -m option.

The technical groundwork to make this possible has been done for PEP
489, and enabling the -m option is listed in that PEP's “Possible Future
Extensions” section. Technically, the additional changes proposed here
are relatively small.

Rationale

Extension modules' lack of support for the -m option has traditionally
been worked around by providing a Python wrapper. For example, the
_pickle module's command line interface is in the pure-Python pickle
module (along with a pure-Python reimplementation).

This works well for standard library modules, as building command line
interfaces using the C API is cumbersome. However, other users may want
to create executable extension modules directly.

An important use case is Cython, a Python-like language that compiles to
C extension modules. Cython is a (near) superset of Python, meaning that
compiling a Python module with Cython will typically not change the
module's functionality, allowing Cython-specific features to be added
gradually. This PEP will allow Cython extension modules to behave the
same as their Python counterparts when run using the -m option. Cython
developers consider the feature worth implementing (see Cython issue
1715).

Background

Python's -m option is handled by the function runpy._run_module_as_main.

The module specified by -m is not imported normally. Instead, it is
executed in the namespace of the __main__ module, which is created quite
early in interpreter initialization.

For Python source modules, running in another module's namespace is not
a problem: the code is executed with locals and globals set to the
existing module's __dict__. This is not the case for extension modules,
whose PyInit_* entry point traditionally both created a new module
object (using PyModule_Create), and initialized it.

Since Python 3.5, extension modules can use PEP 489 multi-phase
initialization. In this scenario, the PyInit_* entry point returns a
PyModuleDef structure: a description of how the module should be created
and initialized. The extension can choose to customize creation of the
module object using the Py_mod_create callback, or opt to use a normal
module object by not specifying Py_mod_create. Another callback,
Py_mod_exec, is then called to initialize the module object, e.g. by
populating it with methods and classes.

Proposal

Multi-phase initialization makes it possible to execute an extension
module in another module's namespace: if a Py_mod_create callback is not
specified, the __main__ module can be passed to the Py_mod_exec callback
to be initialized, as if __main__ was a freshly constructed module
object.

One complication in this scheme is C-level module state. Each module has
a md_state pointer that points to a region of memory allocated when an
extension module is created. The PyModuleDef specifies how much memory
is to be allocated.

The implementation must take care that md_state memory is allocated at
most once. Also, the Py_mod_exec callback should only be called once per
module. The implications of multiply-initialized modules are too subtle
to require expecting extension authors to reason about them. The
md_state pointer itself will serve as a guard: allocating the memory and
calling Py_mod_exec will always be done together, and initializing an
extension module will fail if md_state is already non-NULL.

Since the __main__ module is not created as an extension module, its
md_state is normally NULL. Before initializing an extension module in
__main__'s context, its module state will be allocated according to the
PyModuleDef of that module.

While PEP 489 was designed to make these changes generally possible,
it's necessary to decouple module discovery, creation, and
initialization steps for extension modules, so that another module can
be used instead of a newly initialized one, and the functionality needs
to be added to runpy and importlib.

Specification

A new optional method for importlib loaders will be added. This method
will be called exec_in_module and will take two positional arguments:
module spec and an already existing module. Any import-related
attributes, such as __spec__ or __name__, already set on the module will
be ignored.

The runpy._run_module_as_main function will look for this new loader
method. If it is present, runpy will execute it instead of trying to
load and run the module's Python code. Otherwise, runpy will act as
before.

ExtensionFileLoader Changes

importlib's ExtensionFileLoader will get an implementation of
exec_in_module that will call a new function, _imp.exec_in_module.

_imp.exec_in_module will use existing machinery to find and call an
extension module's PyInit_* function.

The PyInit_* function can return either a fully initialized module
(single-phase initialization) or a PyModuleDef (for PEP 489 multi-phase
initialization).

In the single-phase initialization case, _imp.exec_in_module will raise
ImportError.

In the multi-phase initialization case, the PyModuleDef and the module
to be initialized will be passed to a new function,
PyModule_ExecInModule.

This function raises ImportError if the PyModuleDef specifies a
Py_mod_create slot, or if the module has already been initialized (i.e.
its md_state pointer is not NULL). Otherwise, the function will
initialize the module according to the PyModuleDef.

Backwards Compatibility

This PEP maintains backwards compatibility. It only adds new functions,
and a new loader method that is added for a loader that previously did
not support running modules as __main__.

Reference Implementation

The reference implementation of this PEP is available at GitHub.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: