text:
PEP: 732 Title: The Python Documentation Editorial Board Author: Joanna
Jablonski Sponsor: Mariatta Wijaya Discussions-To:
https://discuss.python.org/t/pep-732-the-python-documentation-editorial-board/36710
Status: Active Type: Process Topic: Governance Created: 14-Oct-2023
Post-History: 20-Oct-2023 Resolution:
https://discuss.python.org/t/pep-732-the-python-documentation-editorial-board/36710/9

Abstract

This PEP:

-   Establishes the Python Documentation Editorial Board
-   Proposes how the editorial board will work

Motivation

The Steering Council approved the creation of a Documentation Working
Group in March 2021 to set direction for the docs. This group is now
called the Editorial Board to differentiate it from the Documentation
Working Group that was created since then to focus on more tactical
work.

The purpose of the Python documentation is to serve the present and
future end users of Python. As such, the core development community and
the greater Python documentation contributors work together to achieve
this:

[Three concentric circles. At the centre: Documentation Editorial
Board, trusted group. Around this: Documentation Working Group,
volunteers who contribute to the docs. Finally, the outer circle
is the world, includes readers of the documentation.
:class: invert-in-dark-mode
:width: 75%]

Specification

Mandate

The editorial board will:

-   Ensure processes are in place to maintain and improve the quality of
    Python's documentation
-   Foster Python documentation as a community resource to serve the
    current and future users
-   Act in alignment with the Python Software Foundation mission, which
    is to advance the Python programming language, and to support and
    facilitate the growth of a diverse and international community of
    Python programmers
-   Ensure that contributing to documentation is accessible, inclusive,
    and sustainable
-   Establish appropriate decision-making processes for documentation
    content
-   Seek to achieve consensus among contributors prior to making
    decisions
-   Be the final arbiter for documentation content decisions

Responsibilities

The board has authority to make decisions about Python’s documentation,
as scoped below. For example, it can:

-   Set big-picture strategy for Python’s documentation
-   Set the intended structure for documentation
-   Make style and editorial decisions for both writing and design
-   Handle documentation governance (for example, delegation of
    decision-making to subject-matter experts, resolution of
    disagreements, decisions.)

Scope

The Editorial board oversees the content and strategy for the following:

  -----------------------------------------------------------------------
  In scope                            Not in scope
  ----------------------------------- -----------------------------------
  CPython documentation               Code comments in CPython codebase
  (docs.python.org)

  CPython devguide                    CPython docstrings
  (devguide.python.org)

  Translations of CPython docs        PEPs (peps.python.org)

                                      PyPA documentation

                                      www.python.org

                                      The Python Wiki (wiki.python.org)
  -----------------------------------------------------------------------

Composition

The Python Documentation Editorial Board is composed of five members.

Editorial Board Members

The initial Editorial Board members are:

-   Mariatta Wijaya
-   Ned Batchelder
-   Joanna Jablonski
-   Guido van Rossum
-   Carol Willing

Editorial Board Member Qualifications

Editorial board members should have:

-   A good grasp of the philosophy of the Python project
-   A background in Python education and developer-facing documentation
-   A solid track record of being constructive and helpful
-   A history of making significant contributions to Python
-   A willingness to dedicate time to improving Python's docs

Members of the Editorial Board should have experience in education,
communication, technical writing, Python’s documentation, accessibility,
translation, or community management.

Term

Editorial Board members serve for an indefinite term, though it is
generally expected that there will be changes in Editorial Board
composition each year. Editorial Board members will confirm annually
whether they wish to continue as a board member. Members may resign at
any time.

If a board member drops out of touch and cannot be contacted for a month
or longer, then the rest of the board may vote to replace them.

Changes to the Editorial Board's Size

Annually after each major Python release, the Editorial Board will
review whether the board's size should change. This provides flexibility
if the needs of the documentation community change over time. A simple
majority is needed to make a decision to increase the board's size where
quorum is 80% of the current board.

As the sponsoring organization of the Documentation Editorial Board, the
Steering Council may change the number of members of the Board at any
time, including appointing new members or dismissing existing members.

Vacancies

If a vacancy exists on the board for any reason, the Documentation
Editorial Board will publicly announce a call for prospective board
members. Prospective board members would submit a brief document stating
qualifications and their motivation to serve. The sitting members of the
Editorial Board will select new board members by a simple majority where
quorum is 80% of the current board.

Amendments

This PEP serves as a charter for the Docs Editorial Board. Changes to
its operation can be made either through a new PEP or through a change
to this PEP. In either case, the change would be decided upon by the
Steering Council after discussion in the community.

PEP Acceptance

PEP 732 was accepted by the Python Steering Council on December 11,
2023.

The Steering Council commented that, while they don't disagree with the
scoping set out in the PEP, it would probably make sense for the
Editorial Board to consider expanding the scope to include docstrings in
the standard library, once the Board is sufficiently established and the
higher priorities have been taken care of.

Contact

To ask the Editorial Board for a decision, community members may open an
issue in the python/editorial-board repository.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

license: Public Domain


text:
PEP: 3123 Title: Making PyObject_HEAD conform to standard C Version:
$Revision$ Last-Modified: $Date$ Author: Martin von Löwis
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 27-Apr-2007 Python-Version: 3.0 Post-History:

Abstract

Python currently relies on undefined C behavior, with its usage of
PyObject_HEAD. This PEP proposes to change that into standard C.

Rationale

Standard C defines that an object must be accessed only through a
pointer of its type, and that all other accesses are undefined behavior,
with a few exceptions. In particular, the following code has undefined
behavior:

    struct FooObject{
      PyObject_HEAD
      int data;
    };

    PyObject *foo(struct FooObject*f){
     return (PyObject*)f;
    }

    int bar(){
     struct FooObject *f = malloc(sizeof(struct FooObject));
     struct PyObject *o = foo(f);
     f->ob_refcnt = 0;
     o->ob_refcnt = 1;
     return f->ob_refcnt;
    }

The problem here is that the storage is both accessed as if it where
struct PyObject, and as struct FooObject.

Historically, compilers did not have any problems with this code.
However, modern compilers use that clause as an optimization
opportunity, finding that f->ob_refcnt and o->ob_refcnt cannot possibly
refer to the same memory, and that therefore the function should return
0, without having to fetch the value of ob_refcnt at all in the return
statement. For GCC, Python now uses -fno-strict-aliasing to work around
that problem; with other compilers, it may just see undefined behavior.
Even with GCC, using -fno-strict-aliasing may pessimize the generated
code unnecessarily.

Specification

Standard C has one specific exception to its aliasing rules precisely
designed to support the case of Python: a value of a struct type may
also be accessed through a pointer to the first field. E.g. if a struct
starts with an int, the struct * may also be cast to an int *, allowing
to write int values into the first field.

For Python, PyObject_HEAD and PyObject_VAR_HEAD will be changed to not
list all fields anymore, but list a single field of type
PyObject/PyVarObject:

    typedef struct _object {
      _PyObject_HEAD_EXTRA
      Py_ssize_t ob_refcnt;
      struct _typeobject *ob_type;
    } PyObject;

    typedef struct {
      PyObject ob_base;
      Py_ssize_t ob_size;
    } PyVarObject;

    #define PyObject_HEAD        PyObject ob_base;
    #define PyObject_VAR_HEAD    PyVarObject ob_base;

Types defined as fixed-size structure will then include PyObject as its
first field, PyVarObject for variable-sized objects. E.g.:

    typedef struct {
      PyObject ob_base;
      PyObject *start, *stop, *step;
    } PySliceObject;

    typedef struct {
      PyVarObject ob_base;
      PyObject **ob_item;
      Py_ssize_t allocated;
    } PyListObject;

The above definitions of PyObject_HEAD are normative, so extension
authors MAY either use the macro, or put the ob_base field explicitly
into their structs.

As a convention, the base field SHOULD be called ob_base. However, all
accesses to ob_refcnt and ob_type MUST cast the object pointer to
PyObject* (unless the pointer is already known to have that type), and
SHOULD use the respective accessor macros. To simplify access to
ob_type, ob_refcnt, and ob_size, macros:

    #define Py_TYPE(o)    (((PyObject*)(o))->ob_type)
    #define Py_REFCNT(o)  (((PyObject*)(o))->ob_refcnt)
    #define Py_SIZE(o)    (((PyVarObject*)(o))->ob_size)

are added. E.g. the code blocks :

    #define PyList_CheckExact(op) ((op)->ob_type == &PyList_Type)

    return func->ob_type->tp_name;

needs to be changed to:

    #define PyList_CheckExact(op) (Py_TYPE(op) == &PyList_Type)

    return Py_TYPE(func)->tp_name;

For initialization of type objects, the current sequence :

    PyObject_HEAD_INIT(NULL)
    0, /* ob_size */

becomes incorrect, and must be replaced with :

    PyVarObject_HEAD_INIT(NULL, 0)

Compatibility with Python 2.6

To support modules that compile with both Python 2.6 and Python 3.0, the
Py_* macros are added to Python 2.6. The macros Py_INCREF and Py_DECREF
will be changed to cast their argument to PyObject *, so that module
authors can also explicitly declare the ob_base field in modules
designed for Python 2.6.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 302 Title: New Import Hooks Version: $Revision$ Last-Modified:
$Date$ Author: Just van Rossum<EMAIL_ADDRESS>Paul Moore
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 19-Dec-2002 Python-Version: 2.3 Post-History:
19-Dec-2002

Warning

The language reference for import[1] and importlib documentation [2] now
supersede this PEP. This document is no longer updated and provided for
historical purposes only.

Abstract

This PEP proposes to add a new set of import hooks that offer better
customization of the Python import mechanism. Contrary to the current
__import__ hook, a new-style hook can be injected into the existing
scheme, allowing for a finer grained control of how modules are found
and how they are loaded.

Motivation

The only way to customize the import mechanism is currently to override
the built-in __import__ function. However, overriding __import__ has
many problems. To begin with:

-   An __import__ replacement needs to fully reimplement the entire
    import mechanism, or call the original __import__ before or after
    the custom code.
-   It has very complex semantics and responsibilities.
-   __import__ gets called even for modules that are already in
    sys.modules, which is almost never what you want, unless you're
    writing some sort of monitoring tool.

The situation gets worse when you need to extend the import mechanism
from C: it's currently impossible, apart from hacking Python's import.c
or reimplementing much of import.c from scratch.

There is a fairly long history of tools written in Python that allow
extending the import mechanism in various way, based on the __import__
hook. The Standard Library includes two such tools: ihooks.py (by GvR)
and imputil.py[3] (Greg Stein), but perhaps the most famous is iu.py by
Gordon McMillan, available as part of his Installer package. Their
usefulness is somewhat limited because they are written in Python;
bootstrapping issues need to worked around as you can't load the module
containing the hook with the hook itself. So if you want the entire
Standard Library to be loadable from an import hook, the hook must be
written in C.

Use cases

This section lists several existing applications that depend on import
hooks. Among these, a lot of duplicate work was done that could have
been saved if there had been a more flexible import hook at the time.
This PEP should make life a lot easier for similar projects in the
future.

Extending the import mechanism is needed when you want to load modules
that are stored in a non-standard way. Examples include modules that are
bundled together in an archive; byte code that is not stored in a pyc
formatted file; modules that are loaded from a database over a network.

The work on this PEP was partly triggered by the implementation of PEP
273, which adds imports from Zip archives as a built-in feature to
Python. While the PEP itself was widely accepted as a must-have feature,
the implementation left a few things to desire. For one thing it went
through great lengths to integrate itself with import.c, adding lots of
code that was either specific for Zip file imports or not specific to
Zip imports, yet was not generally useful (or even desirable) either.
Yet the PEP 273 implementation can hardly be blamed for this: it is
simply extremely hard to do, given the current state of import.c.

Packaging applications for end users is a typical use case for import
hooks, if not the typical use case. Distributing lots of source or pyc
files around is not always appropriate (let alone a separate Python
installation), so there is a frequent desire to package all needed
modules in a single file. So frequent in fact that multiple solutions
have been implemented over the years.

The oldest one is included with the Python source code: Freeze[4]. It
puts marshalled byte code into static objects in C source code. Freeze's
"import hook" is hard wired into import.c, and has a couple of issues.
Later solutions include Fredrik Lundh's Squeeze, Gordon McMillan's
Installer, and Thomas Heller's py2exe[5]. MacPython ships with a tool
called BuildApplication.

Squeeze, Installer and py2exe use an __import__ based scheme (py2exe
currently uses Installer's iu.py, Squeeze used ihooks.py), MacPython has
two Mac-specific import hooks hard wired into import.c, that are similar
to the Freeze hook. The hooks proposed in this PEP enables us (at least
in theory; it's not a short-term goal) to get rid of the hard coded
hooks in import.c, and would allow the __import__-based tools to get rid
of most of their import.c emulation code.

Before work on the design and implementation of this PEP was started, a
new BuildApplication-like tool for Mac OS X prompted one of the authors
of this PEP (JvR) to expose the table of frozen modules to Python, in
the imp module. The main reason was to be able to use the freeze import
hook (avoiding fancy __import__ support), yet to also be able to supply
a set of modules at runtime. This resulted in issue #642578[6], which
was mysteriously accepted (mostly because nobody seemed to care either
way ;-). Yet it is completely superfluous when this PEP gets accepted,
as it offers a much nicer and general way to do the same thing.

Rationale

While experimenting with alternative implementation ideas to get
built-in Zip import, it was discovered that achieving this is possible
with only a fairly small amount of changes to import.c. This allowed to
factor out the Zip-specific stuff into a new source file, while at the
same time creating a general new import hook scheme: the one you're
reading about now.

An earlier design allowed non-string objects on sys.path. Such an object
would have the necessary methods to handle an import. This has two
disadvantages: 1) it breaks code that assumes all items on sys.path are
strings; 2) it is not compatible with the PYTHONPATH environment
variable. The latter is directly needed for Zip imports. A compromise
came from Jython: allow string subclasses on sys.path, which would then
act as importer objects. This avoids some breakage, and seems to work
well for Jython (where it is used to load modules from .jar files), but
it was perceived as an "ugly hack".

This led to a more elaborate scheme, (mostly copied from McMillan's
iu.py) in which each in a list of candidates is asked whether it can
handle the sys.path item, until one is found that can. This list of
candidates is a new object in the sys module: sys.path_hooks.

Traversing sys.path_hooks for each path item for each new import can be
expensive, so the results are cached in another new object in the sys
module: sys.path_importer_cache. It maps sys.path entries to importer
objects.

To minimize the impact on import.c as well as to avoid adding extra
overhead, it was chosen to not add an explicit hook and importer object
for the existing file system import logic (as iu.py has), but to simply
fall back to the built-in logic if no hook on sys.path_hooks could
handle the path item. If this is the case, a None value is stored in
sys.path_importer_cache, again to avoid repeated lookups. (Later we can
go further and add a real importer object for the built-in mechanism,
for now, the None fallback scheme should suffice.)

A question was raised: what about importers that don't need any entry on
sys.path? (Built-in and frozen modules fall into that category.) Again,
Gordon McMillan to the rescue: iu.py contains a thing he calls the
metapath. In this PEP's implementation, it's a list of importer objects
that is traversed before sys.path. This list is yet another new object
in the sys module: sys.meta_path. Currently, this list is empty by
default, and frozen and built-in module imports are done after
traversing sys.meta_path, but still before sys.path.

Specification part 1: The Importer Protocol

This PEP introduces a new protocol: the "Importer Protocol". It is
important to understand the context in which the protocol operates, so
here is a brief overview of the outer shells of the import mechanism.

When an import statement is encountered, the interpreter looks up the
__import__ function in the built-in name space. __import__ is then
called with four arguments, amongst which are the name of the module
being imported (may be a dotted name) and a reference to the current
global namespace.

The built-in __import__ function (known as PyImport_ImportModuleEx() in
import.c) will then check to see whether the module doing the import is
a package or a submodule of a package. If it is indeed a (submodule of
a) package, it first tries to do the import relative to the package (the
parent package for a submodule). For example, if a package named "spam"
does "import eggs", it will first look for a module named "spam.eggs".
If that fails, the import continues as an absolute import: it will look
for a module named "eggs". Dotted name imports work pretty much the
same: if package "spam" does "import eggs.bacon" (and "spam.eggs" exists
and is itself a package), "spam.eggs.bacon" is tried. If that fails
"eggs.bacon" is tried. (There are more subtleties that are not described
here, but these are not relevant for implementers of the Importer
Protocol.)

Deeper down in the mechanism, a dotted name import is split up by its
components. For "import spam.ham", first an "import spam" is done, and
only when that succeeds is "ham" imported as a submodule of "spam".

The Importer Protocol operates at this level of individual imports. By
the time an importer gets a request for "spam.ham", module "spam" has
already been imported.

The protocol involves two objects: a finder and a loader. A finder
object has a single method:

    finder.find_module(fullname, path=None)

This method will be called with the fully qualified name of the module.
If the finder is installed on sys.meta_path, it will receive a second
argument, which is None for a top-level module, or package.__path__ for
submodules or subpackages[7]. It should return a loader object if the
module was found, or None if it wasn't. If find_module() raises an
exception, it will be propagated to the caller, aborting the import.

A loader object also has one method:

    loader.load_module(fullname)

This method returns the loaded module or raises an exception, preferably
ImportError if an existing exception is not being propagated. If
load_module() is asked to load a module that it cannot, ImportError is
to be raised.

In many cases the finder and loader can be one and the same object:
finder.find_module() would just return self.

The fullname argument of both methods is the fully qualified module
name, for example "spam.eggs.ham". As explained above, when
finder.find_module("spam.eggs.ham") is called, "spam.eggs" has already
been imported and added to sys.modules. However, the find_module()
method isn't necessarily always called during an actual import: meta
tools that analyze import dependencies (such as freeze, Installer or
py2exe) don't actually load modules, so a finder shouldn't depend on the
parent package being available in sys.modules.

The load_module() method has a few responsibilities that it must fulfill
before it runs any code:

-   If there is an existing module object named 'fullname' in
    sys.modules, the loader must use that existing module. (Otherwise,
    the reload() builtin will not work correctly.) If a module named
    'fullname' does not exist in sys.modules, the loader must create a
    new module object and add it to sys.modules.

    Note that the module object must be in sys.modules before the loader
    executes the module code. This is crucial because the module code
    may (directly or indirectly) import itself; adding it to sys.modules
    beforehand prevents unbounded recursion in the worst case and
    multiple loading in the best.

    If the load fails, the loader needs to remove any module it may have
    inserted into sys.modules. If the module was already in sys.modules
    then the loader should leave it alone.

-   The __file__ attribute must be set. This must be a string, but it
    may be a dummy value, for example "<frozen>". The privilege of not
    having a __file__ attribute at all is reserved for built-in modules.

-   The __name__ attribute must be set. If one uses imp.new_module()
    then the attribute is set automatically.

-   If it's a package, the __path__ variable must be set. This must be a
    list, but may be empty if __path__ has no further significance to
    the importer (more on this later).

-   The __loader__ attribute must be set to the loader object. This is
    mostly for introspection and reloading, but can be used for
    importer-specific extras, for example getting data associated with
    an importer.

-   The __package__ attribute must be set (PEP 366).

    If the module is a Python module (as opposed to a built-in module or
    a dynamically loaded extension), it should execute the module's code
    in the module's global name space (module.__dict__).

    Here is a minimal pattern for a load_module() method:

        # Consider using importlib.util.module_for_loader() to handle
        # most of these details for you.
        def load_module(self, fullname):
            code = self.get_code(fullname)
            ispkg = self.is_package(fullname)
            mod = sys.modules.setdefault(fullname, imp.new_module(fullname))
            mod.__file__ = "<%s>" % self.__class__.__name__
            mod.__loader__ = self
            if ispkg:
                mod.__path__ = []
                mod.__package__ = fullname
            else:
                mod.__package__ = fullname.rpartition('.')[0]
            exec(code, mod.__dict__)
            return mod

Specification part 2: Registering Hooks

There are two types of import hooks: Meta hooks and Path hooks. Meta
hooks are called at the start of import processing, before any other
import processing (so that meta hooks can override sys.path processing,
frozen modules, or even built-in modules). To register a meta hook,
simply add the finder object to sys.meta_path (the list of registered
meta hooks).

Path hooks are called as part of sys.path (or package.__path__)
processing, at the point where their associated path item is
encountered. A path hook is registered by adding an importer factory to
sys.path_hooks.

sys.path_hooks is a list of callables, which will be checked in sequence
to determine if they can handle a given path item. The callable is
called with one argument, the path item. The callable must raise
ImportError if it is unable to handle the path item, and return an
importer object if it can handle the path item. Note that if the
callable returns an importer object for a specific sys.path entry, the
builtin import machinery will not be invoked to handle that entry any
longer, even if the importer object later fails to find a specific
module. The callable is typically the class of the import hook, and
hence the class __init__() method is called. (This is also the reason
why it should raise ImportError: an __init__() method can't return
anything. This would be possible with a __new__() method in a new style
class, but we don't want to require anything about how a hook is
implemented.)

The results of path hook checks are cached in sys.path_importer_cache,
which is a dictionary mapping path entries to importer objects. The
cache is checked before sys.path_hooks is scanned. If it is necessary to
force a rescan of sys.path_hooks, it is possible to manually clear all
or part of sys.path_importer_cache.

Just like sys.path itself, the new sys variables must have specific
types:

-   sys.meta_path and sys.path_hooks must be Python lists.
-   sys.path_importer_cache must be a Python dict.

Modifying these variables in place is allowed, as is replacing them with
new objects.

Packages and the role of __path__

If a module has a __path__ attribute, the import mechanism will treat it
as a package. The __path__ variable is used instead of sys.path when
importing submodules of the package. The rules for sys.path therefore
also apply to pkg.__path__. So sys.path_hooks is also consulted when
pkg.__path__ is traversed. Meta importers don't necessarily use sys.path
at all to do their work and may therefore ignore the value of
pkg.__path__. In this case it is still advised to set it to list, which
can be empty.

Optional Extensions to the Importer Protocol

The Importer Protocol defines three optional extensions. One is to
retrieve data files, the second is to support module packaging tools
and/or tools that analyze module dependencies (for example Freeze),
while the last is to support execution of modules as scripts. The latter
two categories of tools usually don't actually load modules, they only
need to know if and where they are available. All three extensions are
highly recommended for general purpose importers, but may safely be left
out if those features aren't needed.

To retrieve the data for arbitrary "files" from the underlying storage
backend, loader objects may supply a method named get_data():

    loader.get_data(path)

This method returns the data as a string, or raise IOError if the "file"
wasn't found. The data is always returned as if "binary" mode was used
-there is no CRLF translation of text files, for example. It is meant
for importers that have some file-system-like properties. The 'path'
argument is a path that can be constructed by munging module.__file__
(or pkg.__path__ items) with the os.path.* functions, for example:

    d = os.path.dirname(__file__)
    data = __loader__.get_data(os.path.join(d, "logo.gif"))

The following set of methods may be implemented if support for (for
example) Freeze-like tools is desirable. It consists of three additional
methods which, to make it easier for the caller, each of which should be
implemented, or none at all:

    loader.is_package(fullname)
    loader.get_code(fullname)
    loader.get_source(fullname)

All three methods should raise ImportError if the module wasn't found.

The loader.is_package(fullname) method should return True if the module
specified by 'fullname' is a package and False if it isn't.

The loader.get_code(fullname) method should return the code object
associated with the module, or None if it's a built-in or extension
module. If the loader doesn't have the code object but it does have the
source code, it should return the compiled source code. (This is so that
our caller doesn't also need to check get_source() if all it needs is
the code object.)

The loader.get_source(fullname) method should return the source code for
the module as a string (using newline characters for line endings) or
None if the source is not available (yet it should still raise
ImportError if the module can't be found by the importer at all).

To support execution of modules as scripts (PEP 338), the above three
methods for finding the code associated with a module must be
implemented. In addition to those methods, the following method may be
provided in order to allow the runpy module to correctly set the
__file__ attribute:

    loader.get_filename(fullname)

This method should return the value that __file__ would be set to if the
named module was loaded. If the module is not found, then ImportError
should be raised.

Integration with the 'imp' module

The new import hooks are not easily integrated in the existing
imp.find_module() and imp.load_module() calls. It's questionable whether
it's possible at all without breaking code; it is better to simply add a
new function to the imp module. The meaning of the existing
imp.find_module() and imp.load_module() calls changes from: "they expose
the built-in import mechanism" to "they expose the basic unhooked
built-in import mechanism". They simply won't invoke any import hooks. A
new imp module function is proposed (but not yet implemented) under the
name get_loader(), which is used as in the following pattern:

    loader = imp.get_loader(fullname, path)
    if loader is not None:
        loader.load_module(fullname)

In the case of a "basic" import, one the imp.find_module() function
would handle, the loader object would be a wrapper for the current
output of imp.find_module(), and loader.load_module() would call
imp.load_module() with that output.

Note that this wrapper is currently not yet implemented, although a
Python prototype exists in the test_importhooks.py script (the
ImpWrapper class) included with the patch.

Forward Compatibility

Existing __import__ hooks will not invoke new-style hooks by magic,
unless they call the original __import__ function as a fallback. For
example, ihooks.py, iu.py and imputil.py are in this sense not forward
compatible with this PEP.

Open Issues

Modules often need supporting data files to do their job, particularly
in the case of complex packages or full applications. Current practice
is generally to locate such files via sys.path (or a package.__path__
attribute). This approach will not work, in general, for modules loaded
via an import hook.

There are a number of possible ways to address this problem:

-   "Don't do that". If a package needs to locate data files via its
    __path__, it is not suitable for loading via an import hook. The
    package can still be located on a directory in sys.path, as at
    present, so this should not be seen as a major issue.
-   Locate data files from a standard location, rather than relative to
    the module file. A relatively simple approach (which is supported by
    distutils) would be to locate data files based on sys.prefix (or
    sys.exec_prefix). For example, looking in
    os.path.join(sys.prefix, "data", package_name).
-   Import hooks could offer a standard way of getting at data files
    relative to the module file. The standard zipimport object provides
    a method get_data(name) which returns the content of the "file"
    called name, as a string. To allow modules to get at the importer
    object, zipimport also adds an attribute __loader__ to the module,
    containing the zipimport object used to load the module. If such an
    approach is used, it is important that client code takes care not to
    break if the get_data() method is not available, so it is not clear
    that this approach offers a general answer to the problem.

It was suggested on python-dev that it would be useful to be able to
receive a list of available modules from an importer and/or a list of
available data files for use with the get_data() method. The protocol
could grow two additional extensions, say list_modules() and
list_files(). The latter makes sense on loader objects with a get_data()
method. However, it's a bit unclear which object should implement
list_modules(): the importer or the loader or both?

This PEP is biased towards loading modules from alternative places: it
currently doesn't offer dedicated solutions for loading modules from
alternative file formats or with alternative compilers. In contrast, the
ihooks module from the standard library does have a fairly
straightforward way to do this. The Quixote project[8] uses this
technique to import PTL files as if they are ordinary Python modules. To
do the same with the new hooks would either mean to add a new module
implementing a subset of ihooks as a new-style importer, or add a
hookable built-in path importer object.

There is no specific support within this PEP for "stacking" hooks. For
example, it is not obvious how to write a hook to load modules from
tar.gz files by combining separate hooks to load modules from .tar and
.gz files. However, there is no support for such stacking in the
existing hook mechanisms (either the basic "replace __import__" method,
or any of the existing import hook modules) and so this functionality is
not an obvious requirement of the new mechanism. It may be worth
considering as a future enhancement, however.

It is possible (via sys.meta_path) to add hooks which run before
sys.path is processed. However, there is no equivalent way of adding
hooks to run after sys.path is processed. For now, if a hook is required
after sys.path has been processed, it can be simulated by adding an
arbitrary "cookie" string at the end of sys.path, and having the
required hook associated with this cookie, via the normal sys.path_hooks
processing. In the longer term, the path handling code will become a
"real" hook on sys.meta_path, and at that stage it will be possible to
insert user-defined hooks either before or after it.

Implementation

The PEP 302 implementation has been integrated with Python as of 2.3a1.
An earlier version is available as patch #652586[9], but more
interestingly, the issue contains a fairly detailed history of the
development and design.

PEP 273 has been implemented using PEP 302's import hooks.

References and Footnotes

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Language reference for imports
http://docs.python.org/3/reference/import.html

[2] importlib documentation
http://docs.python.org/3/library/importlib.html#module-importlib

[3] imputil module http://docs.python.org/library/imputil.html

[4] The Freeze tool. See also the Tools/freeze/ directory in a Python
source distribution

[5] py2exe by Thomas Heller http://www.py2exe.org/

[6] imp.set_frozenmodules() patch http://bugs.python.org/issue642578

[7] The path argument to finder.find_module() is there because the
pkg.__path__ variable may be needed at this point. It may either come
from the actual parent module or be supplied by imp.find_module() or the
proposed imp.get_loader() function.

[8] Quixote, a framework for developing Web applications
http://www.mems-exchange.org/software/quixote/

[9] New import hooks + Import from Zip files
http://bugs.python.org/issue652586


text:
PEP: 3144 Title: IP Address Manipulation Library for the Python Standard
Library Version: $Revision$ Last-Modified: $Date$ Author: Peter Moody
<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 06-Feb-2012 Python-Version: 3.3
Resolution:
https://mail.python.org/pipermail/python-dev/2012-May/119474.html

Abstract

This PEP proposes a design and for an IP address manipulation module for
python.

PEP Acceptance

This PEP was accepted by Alyssa Coghlan on the 15th of May, 2012.

Motivation

Several very good IP address modules for python already exist. The truth
is that all of them struggle with the balance between adherence to
Pythonic principals and the shorthand upon which network engineers and
administrators rely. ipaddress aims to strike the right balance.

Rationale

The existence of several Python IP address manipulation modules is
evidence of an outstanding need for the functionality this module seeks
to provide.

Background

PEP 3144 and ipaddr have been up for inclusion before. The version of
the library specified here is backwards incompatible with the version on
PyPI and the one which was discussed before. In order to avoid confusing
users of the current ipaddr, I've renamed this version of the library
ipaddress.

The main differences between ipaddr and ipaddress are:

-   ipaddress *Network classes are equivalent to the ipaddr *Network
    class counterparts with the strict flag set to True.
-   ipaddress *Interface classes are equivalent to the ipaddr *Network
    class counterparts with the strict flag set to False.
-   The factory functions in ipaddress were renamed to disambiguate them
    from classes.
-   A few attributes were renamed to disambiguate their purpose as well.
    (eg. network, network_address)
-   A number of methods and functions which returned containers in
    ipaddr now return iterators. This includes subnets, address_exclude,
    summarize_address_range and collapse_address_list.

Due to the backwards incompatible API changes between ipaddress and
ipaddr, the proposal is to add the module using the new provisional API
status:

-   http://docs.python.org/dev/glossary.html#term-provisional-package

Relevant messages on python-dev:

-   https://mail.python.org/pipermail/python-dev/2012-January/116016.html
-   https://mail.python.org/pipermail/python-dev/2012-February/116656.html
-   https://mail.python.org/pipermail/python-dev/2012-February/116688.html

Specification

The ipaddr module defines a total of 6 new public classes, 3 for
manipulating IPv4 objects and 3 for manipulating IPv6 objects. The
classes are as follows:

-   IPv4Address/IPv6Address - These define individual addresses, for
    example the IPv4 address returned by an A record query for
    www.google.com (<IP_ADDRESS>) or the IPv6 address returned by a
    AAAA record query for ipv6.google.com (2001:4860:4001:801::1011).
-   IPv4Network/IPv6Network - These define networks or groups of
    addresses, for example the IPv4 network reserved for multicast use
    (<IP_ADDRESS>/4) or the IPv6 network reserved for multicast (ff00::/8,
    wow, that's big).
-   IPv4Interface/IPv6Interface - These hybrid classes refer to an
    individual address on a given network. For example, the IPV4 address
    <IP_ADDRESS> on the network <IP_ADDRESS>/24 could be referred to as
    <IP_ADDRESS>/24. Likewise, the IPv6 address 2001:DB8::1 on the network
    2001:DB8::/96 could be referred to as 2001:DB8::1/96. It's very
    common to refer to addresses assigned to computer network interfaces
    like this, hence the Interface name.

All IPv4 classes share certain characteristics and methods; the number
of bits needed to represent them, whether or not they belong to certain
special IPv4 network ranges, etc. Similarly, all IPv6 classes share
characteristics and methods.

ipaddr makes extensive use of inheritance to avoid code duplication as
much as possible. The parent classes are private, but they are outlined
here:

-   _IPAddrBase - Provides methods common to all ipaddr objects.
-   _BaseAddress - Provides methods common to IPv4Address and
    IPv6Address.
-   _BaseInterface - Provides methods common to IPv4Interface and
    IPv6Interface, as well as IPv4Network and IPv6Network (ipaddr treats
    the Network classes as a special case of Interface).
-   _BaseV4 - Provides methods and variables (eg, _max_prefixlen) common
    to all IPv4 classes.
-   _BaseV6 - Provides methods and variables common to all IPv6 classes.

Comparisons between objects of differing IP versions results in a
TypeError[1]. Additionally, comparisons of objects with different _Base
parent classes results in a TypeError. The effect of the _Base parent
class limitation is that IPv4Interface's can be compared to
IPv4Network's and IPv6Interface's can be compared to IPv6Network's.

Reference Implementation

The current reference implementation can be found at:

http://code.google.com/p/ipaddress-py/source/browse/ipaddress.py

Or see the tarball to include the README and unittests.
http://code.google.com/p/ipaddress-py/downloads/detail?name=ipaddress-1.0.tar.gz

More information about using the reference implementation can be found
at: http://code.google.com/p/ipaddr-py/wiki/Using3144

References

Copyright

This document has been placed in the public domain.

[1] Appealing to authority is a logical fallacy, but Vint Cerf is an
authority who can't be ignored. Full text of the email follows:

  I have seen a substantial amount of traffic about IPv4 and IPv6
  comparisons and the general consensus is that these are not
  comparable.

  If we were to take a very simple minded view, we might treat these as
  pure integers in which case there is an ordering but not a useful one.

  In the IPv4 world, "length" is important because we take longest (most
  specific) address first for routing. Length is determine by the mask,
  as you know.

  Assuming that the same style of argument works in IPv6, we would have
  to conclude that treating an IPv6 value purely as an integer for
  comparison with IPv4 would lead to some really strange results.

  All of IPv4 space would lie in the host space of 0::0/96 prefix of
  IPv6. For any useful interpretation of IPv4, this is a non-starter.

  I think the only sensible conclusion is that IPv4 values and IPv6
  values should be treated as non-comparable.

  Vint


text:
PEP: 472 Title: Support for indexing with keyword arguments Version:
$Revision$ Last-Modified: $Date$ Author: Stefano Borini, Joseph
Martinot-Lagarde Discussions-To<EMAIL_ADDRESS>Status:
Rejected Type: Standards Track Content-Type: text/x-rst Created:
24-Jun-2014 Python-Version: 3.6 Post-History: 02-Jul-2014 Resolution:
https://mail.python.org/pipermail/python-dev/2019-March/156693.html

Abstract

This PEP proposes an extension of the indexing operation to support
keyword arguments. Notations in the form a[K=3,R=2] would become legal
syntax. For future-proofing considerations, a[1:2, K=3, R=4] are
considered and may be allowed as well, depending on the choice for
implementation. In addition to a change in the parser, the index
protocol (__getitem__, __setitem__ and __delitem__) will also
potentially require adaptation.

Motivation

The indexing syntax carries a strong semantic content, differentiating
it from a method call: it implies referring to a subset of data. We
believe this semantic association to be important, and wish to expand
the strategies allowed to refer to this data.

As a general observation, the number of indices needed by an indexing
operation depends on the dimensionality of the data: one-dimensional
data (e.g. a list) requires one index (e.g. a[3]), two-dimensional data
(e.g. a matrix) requires two indices (e.g. a[2,3]) and so on. Each index
is a selector along one of the axes of the dimensionality, and the
position in the index tuple is the metainformation needed to associate
each index to the corresponding axis.

The current python syntax focuses exclusively on position to express the
association to the axes, and also contains syntactic sugar to refer to
non-punctiform selection (slices)

    >>> a[3]       # returns the fourth element of a
    >>> a[1:10:2]  # slice notation (extract a non-trivial data subset)
    >>> a[3,2]     # multiple indexes (for multidimensional arrays)

The additional notation proposed in this PEP would allow notations
involving keyword arguments in the indexing operation, e.g.

    >>> a[K=3, R=2]

which would allow to refer to axes by conventional names.

One must additionally consider the extended form that allows both
positional and keyword specification

    >>> a[3,R=3,K=4]

This PEP will explore different strategies to enable the use of these
notations.

Use cases

The following practical use cases present two broad categories of usage
of a keyworded specification: Indexing and contextual option. For
indexing:

1.  To provide a more communicative meaning to the index, preventing
    e.g. accidental inversion of indexes

        >>> gridValues[x=3, y=5, z=8]
        >>> rain[time=0:12, location=location]

2.  In some domain, such as computational physics and chemistry, the use
    of a notation such as Basis[Z=5] is a Domain Specific Language
    notation to represent a level of accuracy

        >>> low_accuracy_energy = computeEnergy(molecule, BasisSet[Z=3])

    In this case, the index operation would return a basis set at the
    chosen level of accuracy (represented by the parameter Z). The
    reason behind an indexing is that the BasisSet object could be
    internally represented as a numeric table, where rows (the
    "coefficient" axis, hidden to the user in this example) are
    associated to individual elements (e.g. row 0:5 contains
    coefficients for element 1, row 5:8 coefficients for element 2) and
    each column is associated to a given degree of accuracy ("accuracy"
    or "Z" axis) so that first column is low accuracy, second column is
    medium accuracy and so on. With that indexing, the user would obtain
    another object representing the contents of the column of the
    internal table for accuracy level 3.

Additionally, the keyword specification can be used as an option
contextual to the indexing. Specifically:

1.  A "default" option allows to specify a default return value when the
    index is not present

        >>> lst = [1, 2, 3]
        >>> value = lst[5, default=0]  # value is 0

2.  For a sparse dataset, to specify an interpolation strategy to infer
    a missing point from e.g. its surrounding data.

        >>> value = array[1, 3, interpolate=spline_interpolator]

3.  A unit could be specified with the same mechanism

        >>> value = array[1, 3, unit="degrees"]

How the notation is interpreted is up to the implementing class.

Current implementation

Currently, the indexing operation is handled by methods __getitem__,
__setitem__ and __delitem__. These methods' signature accept one
argument for the index (with __setitem__ accepting an additional
argument for the set value). In the following, we will analyze
__getitem__(self, idx) exclusively, with the same considerations implied
for the remaining two methods.

When an indexing operation is performed, __getitem__(self, idx) is
called. Traditionally, the full content between square brackets is
turned into a single object passed to argument idx:

-   When a single element is passed, e.g. a[2], idx will be 2.
-   When multiple elements are passed, they must be separated by commas:
    a[2, 3]. In this case, idx will be a tuple (2, 3). With
    a[2, 3, "hello", {}] idx will be (2, 3, "hello", {}).
-   A slicing notation e.g. a[2:10] will produce a slice object, or a
    tuple containing slice objects if multiple values were passed.

Except for its unique ability to handle slice notation, the indexing
operation has similarities to a plain method call: it acts like one when
invoked with only one element; If the number of elements is greater than
one, the idx argument behaves like a *args. However, as stated in the
Motivation section, an indexing operation has the strong semantic
implication of extraction of a subset out of a larger set, which is not
automatically associated to a regular method call unless appropriate
naming is chosen. Moreover, its different visual style is important for
readability.

Specifications

The implementation should try to preserve the current signature for
__getitem__, or modify it in a backward-compatible way. We will present
different alternatives, taking into account the possible cases that need
to be addressed

    C0. a[1]; a[1,2]         # Traditional indexing
    C1. a[Z=3]
    C2. a[Z=3, R=4]
    C3. a[1, Z=3]
    C4. a[1, Z=3, R=4]
    C5. a[1, 2, Z=3]
    C6. a[1, 2, Z=3, R=4]
    C7. a[1, Z=3, 2, R=4]    # Interposed ordering

Strategy "Strict dictionary"

This strategy acknowledges that __getitem__ is special in accepting only
one object, and the nature of that object must be non-ambiguous in its
specification of the axes: it can be either by order, or by name. As a
result of this assumption, in presence of keyword arguments, the passed
entity is a dictionary and all labels must be specified.

    C0. a[1]; a[1,2]      -> idx = 1; idx = (1, 2)
    C1. a[Z=3]            -> idx = {"Z": 3}
    C2. a[Z=3, R=4]       -> idx = {"Z": 3, "R": 4}
    C3. a[1, Z=3]         -> raise SyntaxError
    C4. a[1, Z=3, R=4]    -> raise SyntaxError
    C5. a[1, 2, Z=3]      -> raise SyntaxError
    C6. a[1, 2, Z=3, R=4] -> raise SyntaxError
    C7. a[1, Z=3, 2, R=4] -> raise SyntaxError

Pros

-   Strong conceptual similarity between the tuple case and the
    dictionary case. In the first case, we are specifying a tuple, so we
    are naturally defining a plain set of values separated by commas. In
    the second, we are specifying a dictionary, so we are specifying a
    homogeneous set of key/value pairs, as in dict(Z=3, R=4);
-   Simple and easy to parse on the __getitem__ side: if it gets a
    tuple, determine the axes using positioning. If it gets a
    dictionary, use the keywords.
-   C interface does not need changes.

Neutral

-   Degeneracy of a[{"Z": 3, "R": 4}] with a[Z=3, R=4] means the
    notation is syntactic sugar.

Cons

-   Very strict.
-   Destroys ordering of the passed arguments. Preserving the order
    would be possible with an OrderedDict as drafted by PEP 468.
-   Does not allow use cases with mixed positional/keyword arguments
    such as a[1, 2, default=5].

Strategy "mixed dictionary"

This strategy relaxes the above constraint to return a dictionary
containing both numbers and strings as keys.

    C0. a[1]; a[1,2]      -> idx = 1; idx = (1, 2)
    C1. a[Z=3]            -> idx = {"Z": 3}
    C2. a[Z=3, R=4]       -> idx = {"Z": 3, "R": 4}
    C3. a[1, Z=3]         -> idx = { 0: 1, "Z": 3}
    C4. a[1, Z=3, R=4]    -> idx = { 0: 1, "Z": 3, "R": 4}
    C5. a[1, 2, Z=3]      -> idx = { 0: 1, 1: 2, "Z": 3}
    C6. a[1, 2, Z=3, R=4] -> idx = { 0: 1, 1: 2, "Z": 3, "R": 4}
    C7. a[1, Z=3, 2, R=4] -> idx = { 0: 1, "Z": 3, 2: 2, "R": 4}

Pros

-   Opens for mixed cases.

Cons

-   Destroys ordering information for string keys. We have no way of
    saying if "Z" in C7 was in position 1 or 3.
-   Implies switching from a tuple to a dict as soon as one specified
    index has a keyword argument. May be confusing to parse.

Strategy "named tuple"

Return a named tuple for idx instead of a tuple. Keyword arguments would
obviously have their stated name as key, and positional argument would
have an underscore followed by their order:

    C0. a[1]; a[1,2]      -> idx = 1; idx = (_0=1, _1=2)
    C1. a[Z=3]            -> idx = (Z=3)
    C2. a[Z=3, R=2]       -> idx = (Z=3, R=2)
    C3. a[1, Z=3]         -> idx = (_0=1, Z=3)
    C4. a[1, Z=3, R=2]    -> idx = (_0=1, Z=3, R=2)
    C5. a[1, 2, Z=3]      -> idx = (_0=1, _2=2, Z=3)
    C6. a[1, 2, Z=3, R=4] -> (_0=1, _1=2, Z=3, R=4)
    C7. a[1, Z=3, 2, R=4] -> (_0=1, Z=3, _1=2, R=4)
                          or (_0=1, Z=3, _2=2, R=4)
                          or raise SyntaxError

The required typename of the namedtuple could be Index or the name of
the argument in the function definition, it keeps the ordering and is
easy to analyse by using the _fields attribute. It is backward
compatible, provided that C0 with more than one entry now passes a
namedtuple instead of a plain tuple.

Pros

-   Looks nice. namedtuple transparently replaces tuple and gracefully
    degrades to the old behavior.
-   Does not require a change in the C interface

Cons

-   According to some sources[1] namedtuple is not well developed. To
    include it as such important object would probably require rework
    and improvement;
-   The namedtuple fields, and thus the type, will have to change
    according to the passed arguments. This can be a performance
    bottleneck, and makes it impossible to guarantee that two subsequent
    index accesses get the same Index class;
-   the _n "magic" fields are a bit unusual, but ipython already uses
    them for result history.
-   Python currently has no builtin namedtuple. The current one is
    available in the "collections" module in the standard library.
-   Differently from a function, the two notations
    gridValues[x=3, y=5, z=8] and gridValues[3,5,8] would not gracefully
    match if the order is modified at call time (e.g. we ask for
    gridValues[y=5, z=8, x=3]). In a function, we can pre-define
    argument names so that keyword arguments are properly matched. Not
    so in __getitem__, leaving the task for interpreting and matching to
    __getitem__ itself.

Strategy "New argument contents"

In the current implementation, when many arguments are passed to
__getitem__, they are grouped in a tuple and this tuple is passed to
__getitem__ as the single argument idx. This strategy keeps the current
signature, but expands the range of variability in type and contents of
idx to more complex representations.

We identify four possible ways to implement this strategy:

-   P1: uses a single dictionary for the keyword arguments.
-   P2: uses individual single-item dictionaries.
-   P3: similar to P2, but replaces single-item dictionaries with a
    (key, value) tuple.
-   P4: similar to P2, but uses a special and additional new object:
    keyword()

Some of these possibilities lead to degenerate notations, i.e.
indistinguishable from an already possible representation. Once again,
the proposed notation becomes syntactic sugar for these representations.

Under this strategy, the old behavior for C0 is unchanged.

    C0: a[1]        -> idx = 1                    # integer
        a[1,2]      -> idx = (1,2)                # tuple

In C1, we can use either a dictionary or a tuple to represent key and
value pair for the specific indexing entry. We need to have a tuple with
a tuple in C1 because otherwise we cannot differentiate a["Z", 3] from
a[Z=3].

    C1: a[Z=3]      -> idx = {"Z": 3}             # P1/P2 dictionary with single key
                    or idx = (("Z", 3),)          # P3 tuple of tuples
                    or idx = keyword("Z", 3)      # P4 keyword object

As you can see, notation P1/P2 implies that a[Z=3] and a[{"Z": 3}] will
call __getitem__ passing the exact same value, and is therefore
syntactic sugar for the latter. Same situation occurs, although with
different index, for P3. Using a keyword object as in P4 would remove
this degeneracy.

For the C2 case:

    C2. a[Z=3, R=4] -> idx = {"Z": 3, "R": 4}     # P1 dictionary/ordereddict
                    or idx = ({"Z": 3}, {"R": 4}) # P2 tuple of two single-key dict
                    or idx = (("Z", 3), ("R", 4)) # P3 tuple of tuples
                    or idx = (keyword("Z", 3),
                              keyword("R", 4) )   # P4 keyword objects

P1 naturally maps to the traditional **kwargs behavior, however it
breaks the convention that two or more entries for the index produce a
tuple. P2 preserves this behavior, and additionally preserves the order.
Preserving the order would also be possible with an OrderedDict as
drafted by PEP 468.

The remaining cases are here shown:

    C3. a[1, Z=3]   -> idx = (1, {"Z": 3})                     # P1/P2
                    or idx = (1, ("Z", 3))                     # P3
                    or idx = (1, keyword("Z", 3))              # P4

    C4. a[1, Z=3, R=4] -> idx = (1, {"Z": 3, "R": 4})          # P1
                       or idx = (1, {"Z": 3}, {"R": 4})        # P2
                       or idx = (1, ("Z", 3), ("R", 4))        # P3
                       or idx = (1, keyword("Z", 3),
                                    keyword("R", 4))           # P4

    C5. a[1, 2, Z=3]   -> idx = (1, 2, {"Z": 3})               # P1/P2
                       or idx = (1, 2, ("Z", 3))               # P3
                       or idx = (1, 2, keyword("Z", 3))        # P4

    C6. a[1, 2, Z=3, R=4] -> idx = (1, 2, {"Z":3, "R": 4})     # P1
                          or idx = (1, 2, {"Z": 3}, {"R": 4})  # P2
                          or idx = (1, 2, ("Z", 3), ("R", 4))  # P3
                          or idx = (1, 2, keyword("Z", 3),
                                          keyword("R", 4))     # P4

    C7. a[1, Z=3, 2, R=4] -> idx = (1, 2, {"Z": 3, "R": 4})    # P1. Pack the keyword arguments. Ugly.
                          or raise SyntaxError                 # P1. Same behavior as in function calls.
                          or idx = (1, {"Z": 3}, 2, {"R": 4})  # P2
                          or idx =  (1, ("Z", 3), 2, ("R", 4)) # P3
                          or idx =  (1, keyword("Z", 3),
                                     2, keyword("R", 4))       # P4

Pros

-   Signature is unchanged;
-   P2/P3 can preserve ordering of keyword arguments as specified at
    indexing,
-   P1 needs an OrderedDict, but would destroy interposed ordering if
    allowed: all keyword indexes would be dumped into the dictionary;
-   Stays within traditional types: tuples and dicts. Evt. OrderedDict;
-   Some proposed strategies are similar in behavior to a traditional
    function call;
-   The C interface for PyObject_GetItem and family would remain
    unchanged.

Cons

-   Apparently complex and wasteful;
-   Degeneracy in notation (e.g. a[Z=3] and a[{"Z":3}] are equivalent
    and indistinguishable notations at the __[get|set|del]item__ level).
    This behavior may or may not be acceptable.
-   for P4, an additional object similar in nature to slice() is needed,
    but only to disambiguate the above degeneracy.
-   idx type and layout seems to change depending on the whims of the
    caller;
-   May be complex to parse what is passed, especially in the case of
    tuple of tuples;
-   P2 Creates a lot of single keys dictionary as members of a tuple.
    Looks ugly. P3 would be lighter and easier to use than the tuple of
    dicts, and still preserves order (unlike the regular dict), but
    would result in clumsy extraction of keywords.

Strategy "kwargs argument"

__getitem__ accepts an optional **kwargs argument which should be
keyword only. idx also becomes optional to support a case where no
non-keyword arguments are allowed. The signature would then be either

    __getitem__(self, idx)
    __getitem__(self, idx, **kwargs)
    __getitem__(self, **kwargs)

Applied to our cases would produce:

    C0. a[1,2]            -> idx=(1,2);  kwargs={}
    C1. a[Z=3]            -> idx=None ;  kwargs={"Z":3}
    C2. a[Z=3, R=4]       -> idx=None ;  kwargs={"Z":3, "R":4}
    C3. a[1, Z=3]         -> idx=1    ;  kwargs={"Z":3}
    C4. a[1, Z=3, R=4]    -> idx=1    ;  kwargs={"Z":3, "R":4}
    C5. a[1, 2, Z=3]      -> idx=(1,2);  kwargs={"Z":3}
    C6. a[1, 2, Z=3, R=4] -> idx=(1,2);  kwargs={"Z":3, "R":4}
    C7. a[1, Z=3, 2, R=4] -> raise SyntaxError # in agreement to function behavior

Empty indexing a[] of course remains invalid syntax.

Pros

-   Similar to function call, evolves naturally from it;
-   Use of keyword indexing with an object whose __getitem__ doesn't
    have a kwargs will fail in an obvious way. That's not the case for
    the other strategies.

Cons

-   It doesn't preserve order, unless an OrderedDict is used;
-   Forbids C7, but is it really needed?
-   Requires a change in the C interface to pass an additional PyObject
    for the keyword arguments.

C interface

As briefly introduced in the previous analysis, the C interface would
potentially have to change to allow the new feature. Specifically,
PyObject_GetItem and related routines would have to accept an additional
PyObject *kw argument for Strategy "kwargs argument". The remaining
strategies would not require a change in the C function signatures, but
the different nature of the passed object would potentially require
adaptation.

Strategy "named tuple" would behave correctly without any change: the
class returned by the factory method in collections returns a subclass
of tuple, meaning that PyTuple_* functions can handle the resulting
object.

Alternative Solutions

In this section, we present alternative solutions that would workaround
the missing feature and make the proposed enhancement not worth of
implementation.

Use a method

One could keep the indexing as is, and use a traditional get() method
for those cases where basic indexing is not enough. This is a good
point, but as already reported in the introduction, methods have a
different semantic weight from indexing, and you can't use slices
directly in methods. Compare e.g. a[1:3, Z=2] with
a.get(slice(1,3), Z=2).

The authors however recognize this argument as compelling, and the
advantage in semantic expressivity of a keyword-based indexing may be
offset by a rarely used feature that does not bring enough benefit and
may have limited adoption.

Emulate requested behavior by abusing the slice object

This extremely creative method exploits the slice objects' behavior,
provided that one accepts to use strings (or instantiate properly named
placeholder objects for the keys), and accept to use ":" instead of "=".

    >>> a["K":3]
    slice('K', 3, None)
    >>> a["K":3, "R":4]
    (slice('K', 3, None), slice('R', 4, None))
    >>>

While clearly smart, this approach does not allow easy inquire of the
key/value pair, it's too clever and esotheric, and does not allow to
pass a slice as in a[K=1:10:2].

However, Tim Delaney comments

  "I really do think that a[b=c, d=e] should just be syntax sugar for
  a['b':c, 'd':e]. It's simple to explain, and gives the greatest
  backwards compatibility. In particular, libraries that already abused
  slices in this way will just continue to work with the new syntax."

We think this behavior would produce inconvenient results. The library
Pandas uses strings as labels, allowing notation such as

    >>> a[:, "A":"F"]

to extract data from column "A" to column "F". Under the above comment,
this notation would be equally obtained with

    >>> a[:, A="F"]

which is weird and collides with the intended meaning of keyword in
indexing, that is, specifying the axis through conventional names rather
than positioning.

Pass a dictionary as an additional index

    >>> a[1, 2, {"K": 3}]

this notation, although less elegant, can already be used and achieves
similar results. It's evident that the proposed Strategy "New argument
contents" can be interpreted as syntactic sugar for this notation.

Additional Comments

Commenters also expressed the following relevant points:

Relevance of ordering of keyword arguments

As part of the discussion of this PEP, it's important to decide if the
ordering information of the keyword arguments is important, and if
indexes and keys can be ordered in an arbitrary way (e.g.
a[1,Z=3,2,R=4]). PEP 468 tries to address the first point by proposing
the use of an ordereddict, however one would be inclined to accept that
keyword arguments in indexing are equivalent to kwargs in function
calls, and therefore as of today equally unordered, and with the same
restrictions.

Need for homogeneity of behavior

Relative to Strategy "New argument contents", a comment from Ian
Cordasco points out that

  "it would be unreasonable for just one method to behave totally
  differently from the standard behaviour in Python. It would be
  confusing for only __getitem__ (and ostensibly, __setitem__) to take
  keyword arguments but instead of turning them into a dictionary, turn
  them into individual single-item dictionaries." We agree with his
  point, however it must be pointed out that __getitem__ is already
  special in some regards when it comes to passed arguments.

Chris Angelico also states:

  "it seems very odd to start out by saying "here, let's give indexing
  the option to carry keyword args, just like with function calls", and
  then come back and say "oh, but unlike function calls, they're
  inherently ordered and carried very differently"." Again, we agree on
  this point. The most straightforward strategy to keep homogeneity
  would be Strategy "kwargs argument", opening to a **kwargs argument on
  __getitem__.

One of the authors (Stefano Borini) thinks that only the "strict
dictionary" strategy is worth of implementation. It is non-ambiguous,
simple, does not force complex parsing, and addresses the problem of
referring to axes either by position or by name. The "options" use case
is probably best handled with a different approach, and may be
irrelevant for this PEP. The alternative "named tuple" is another valid
choice.

Having .get() become obsolete for indexing with default fallback

Introducing a "default" keyword could make dict.get() obsolete, which
would be replaced by d["key", default=3]. Chris Angelico however states:

  "Currently, you need to write __getitem__ (which raises an exception
  on finding a problem) plus something else, e.g. get(), which returns a
  default instead. By your proposal, both branches would go inside
  __getitem__, which means they could share code; but there still need
  to be two branches."

Additionally, Chris continues:

  "There'll be an ad-hoc and fairly arbitrary puddle of names (some will
  go default=, others will say that's way too long and go def=, except
  that that's a keyword so they'll use dflt= or something...), unless
  there's a strong force pushing people to one consistent name.".

This argument is valid but it's equally valid for any function call, and
is generally fixed by established convention and documentation.

On degeneracy of notation

User Drekin commented: "The case of a[Z=3] and a[{"Z": 3}] is similar to
current a[1, 2] and a[(1, 2)]. Even though one may argue that the
parentheses are actually not part of tuple notation but are just needed
because of syntax, it may look as degeneracy of notation when compared
to function call: f(1, 2) is not the same thing as f((1, 2)).".

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] "namedtuple is not as good as it should be"
(https://mail.python.org/pipermail/python-ideas/2013-June/021257.html)

authors:
Joseph Martinot-Lagarde
Stefano Borini


text:
PEP: 266 Title: Optimizing Global Variable/Attribute Access Author: Skip
Montanaro<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 13-Aug-2001 Python-Version: 2.3
Post-History:

Abstract

The bindings for most global variables and attributes of other modules
typically never change during the execution of a Python program, but
because of Python's dynamic nature, code which accesses such global
objects must run through a full lookup each time the object is needed.
This PEP proposes a mechanism that allows code that accesses most global
objects to treat them as local objects and places the burden of updating
references on the code that changes the name bindings of such objects.

Introduction

Consider the workhorse function sre_compile._compile. It is the internal
compilation function for the sre module. It consists almost entirely of
a loop over the elements of the pattern being compiled, comparing
opcodes with known constant values and appending tokens to an output
list. Most of the comparisons are with constants imported from the
sre_constants module. This means there are lots of LOAD_GLOBAL bytecodes
in the compiled output of this module. Just by reading the code it's
apparent that the author intended LITERAL, NOT_LITERAL, OPCODES and many
other symbols to be constants. Still, each time they are involved in an
expression, they must be looked up anew.

Most global accesses are actually to objects that are "almost
constants". This includes global variables in the current module as well
as the attributes of other imported modules. Since they rarely change,
it seems reasonable to place the burden of updating references to such
objects on the code that changes the name bindings. If
sre_constants.LITERAL is changed to refer to another object, perhaps it
would be worthwhile for the code that modifies the sre_constants module
dict to correct any active references to that object. By doing so, in
many cases global variables and the attributes of many objects could be
cached as local variables. If the bindings between the names given to
the objects and the objects themselves changes rarely, the cost of
keeping track of such objects should be low and the potential payoff
fairly large.

In an attempt to gauge the effect of this proposal, I modified the
Pystone benchmark program included in the Python distribution to cache
global functions. Its main function, Proc0, makes calls to ten different
functions inside its for loop. In addition, Func2 calls Func1 repeatedly
inside a loop. If local copies of these 11 global identifiers are made
before the functions' loops are entered, performance on this particular
benchmark improves by about two percent (from 5561 pystones to 5685 on
my laptop). It gives some indication that performance would be improved
by caching most global variable access. Note also that the pystone
benchmark makes essentially no accesses of global module attributes, an
anticipated area of improvement for this PEP.

Proposed Change

I propose that the Python virtual machine be modified to include
TRACK_OBJECT and UNTRACK_OBJECT opcodes. TRACK_OBJECT would associate a
global name or attribute of a global name with a slot in the local
variable array and perform an initial lookup of the associated object to
fill in the slot with a valid value. The association it creates would be
noted by the code responsible for changing the name-to-object binding to
cause the associated local variable to be updated. The UNTRACK_OBJECT
opcode would delete any association between the name and the local
variable slot.

Threads

Operation of this code in threaded programs will be no different than in
unthreaded programs. If you need to lock an object to access it, you
would have had to do that before TRACK_OBJECT would have been executed
and retain that lock until after you stop using it.

FIXME: I suspect I need more here.

Rationale

Global variables and attributes rarely change. For example, once a
function imports the math module, the binding between the name math and
the module it refers to aren't likely to change. Similarly, if the
function that uses the math module refers to its sin attribute, it's
unlikely to change. Still, every time the module wants to call the
math.sin function, it must first execute a pair of instructions:

    LOAD_GLOBAL     math
    LOAD_ATTR       sin

If the client module always assumed that math.sin was a local constant
and it was the responsibility of "external forces" outside the function
to keep the reference correct, we might have code like this:

    TRACK_OBJECT       math.sin
    ...
    LOAD_FAST          math.sin
    ...
    UNTRACK_OBJECT     math.sin

If the LOAD_FAST was in a loop the payoff in reduced global loads and
attribute lookups could be significant.

This technique could, in theory, be applied to any global variable
access or attribute lookup. Consider this code:

    l = []
    for i in range(10):
        l.append(math.sin(i))
    return l

Even though l is a local variable, you still pay the cost of loading
l.append ten times in the loop. The compiler (or an optimizer) could
recognize that both math.sin and l.append are being called in the loop
and decide to generate the tracked local code, avoiding it for the
builtin range() function because it's only called once during loop
setup. Performance issues related to accessing local variables make
tracking l.append less attractive than tracking globals such as
math.sin.

According to a post to python-dev by Marc-Andre Lemburg[1], LOAD_GLOBAL
opcodes account for over 7% of all instructions executed by the Python
virtual machine. This can be a very expensive instruction, at least
relative to a LOAD_FAST instruction, which is a simple array index and
requires no extra function calls by the virtual machine. I believe many
LOAD_GLOBAL instructions and LOAD_GLOBAL/LOAD_ATTR pairs could be
converted to LOAD_FAST instructions.

Code that uses global variables heavily often resorts to various tricks
to avoid global variable and attribute lookup. The aforementioned
sre_compile._compile function caches the append method of the growing
output list. Many people commonly abuse functions' default argument
feature to cache global variable lookups. Both of these schemes are
hackish and rarely address all the available opportunities for
optimization. (For example, sre_compile._compile does not cache the two
globals that it uses most frequently: the builtin len function and the
global OPCODES array that it imports from sre_constants.py.

Questions

What about threads? What if math.sin changes while in cache?

I believe the global interpreter lock will protect values from being
corrupted. In any case, the situation would be no worse than it is
today. If one thread modified math.sin after another thread had already
executed LOAD_GLOBAL math, but before it executed LOAD_ATTR sin, the
client thread would see the old value of math.sin.

The idea is this. I use a multi-attribute load below as an example, not
because it would happen very often, but because by demonstrating the
recursive nature with an extra call hopefully it will become clearer
what I have in mind. Suppose a function defined in module foo wants to
access spam.eggs.ham and that spam is a module imported at the module
level in foo:

    import spam
    ...
    def somefunc():
    ...
    x = spam.eggs.ham

Upon entry to somefunc, a TRACK_GLOBAL instruction will be executed:

    TRACK_GLOBAL spam.eggs.ham n

spam.eggs.ham is a string literal stored in the function's constants
array. n is a fastlocals index. &fastlocals[n] is a reference to slot n
in the executing frame's fastlocals array, the location in which the
spam.eggs.ham reference will be stored. Here's what I envision
happening:

1.  The TRACK_GLOBAL instruction locates the object referred to by the
    name spam and finds it in its module scope. It then executes a C
    function like:

        _PyObject_TrackName(m, "spam.eggs.ham", &fastlocals[n])

    where m is the module object with an attribute spam.

2.  The module object strips the leading spam. and stores the necessary
    information (eggs.ham and &fastlocals[n]) in case its binding for
    the name eggs changes. It then locates the object referred to by the
    key eggs in its dict and recursively calls:

        _PyObject_TrackName(eggs, "eggs.ham", &fastlocals[n])

3.  The eggs object strips the leading eggs., stores the (ham,
    &fastlocals[n]) info, locates the object in its namespace called ham
    and calls _PyObject_TrackName once again:

        _PyObject_TrackName(ham, "ham", &fastlocals[n])

4.  The ham object strips the leading string (no "." this time, but
    that's a minor point), sees that the result is empty, then uses its
    own value (self, probably) to update the location it was handed:

        Py_XDECREF(&fastlocals[n]);
        &fastlocals[n] = self;
        Py_INCREF(&fastlocals[n]);

    At this point, each object involved in resolving spam.eggs.ham knows
    which entry in its namespace needs to be tracked and what location
    to update if that name changes. Furthermore, if the one name it is
    tracking in its local storage changes, it can call
    _PyObject_TrackName using the new object once the change has been
    made. At the bottom end of the food chain, the last object will
    always strip a name, see the empty string and know that its value
    should be stuffed into the location it's been passed.

    When the object referred to by the dotted expression spam.eggs.ham
    is going to go out of scope, an UNTRACK_GLOBAL spam.eggs.ham n
    instruction is executed. It has the effect of deleting all the
    tracking information that TRACK_GLOBAL established.

    The tracking operation may seem expensive, but recall that the
    objects being tracked are assumed to be "almost constant", so the
    setup cost will be traded off against hopefully multiple local
    instead of global loads. For globals with attributes the tracking
    setup cost grows but is offset by avoiding the extra LOAD_ATTR cost.
    The TRACK_GLOBAL instruction needs to perform a PyDict_GetItemString
    for the first name in the chain to determine where the top-level
    object resides. Each object in the chain has to store a string and
    an address somewhere, probably in a dict that uses storage locations
    as keys (e.g. the &fastlocals[n]) and strings as values. (This dict
    could possibly be a central dict of dicts whose keys are object
    addresses instead of a per-object dict.) It shouldn't be the other
    way around because multiple active frames may want to track
    spam.eggs.ham, but only one frame will want to associate that name
    with one of its fast locals slots.

Unresolved Issues

Threading

What about this (dumb) code?:

    l = []
    lock = threading.Lock()
    ...
    def fill_l()::
       for i in range(1000)::
          lock.acquire()
          l.append(math.sin(i))
          lock.release()
    ...
    def consume_l()::
       while 1::
          lock.acquire()
          if l::
             elt = l.pop()
          lock.release()
          fiddle(elt)

It's not clear from a static analysis of the code what the lock is
protecting. (You can't tell at compile-time that threads are even
involved can you?) Would or should it affect attempts to track l.append
or math.sin in the fill_l function?

If we annotate the code with mythical track_object and untrack_object
builtins (I'm not proposing this, just illustrating where stuff would
go!), we get:

    l = []
    lock = threading.Lock()
    ...
    def fill_l()::
       track_object("l.append", append)
       track_object("math.sin", sin)
       for i in range(1000)::
          lock.acquire()
          append(sin(i))
          lock.release()
       untrack_object("math.sin", sin)
       untrack_object("l.append", append)
    ...
    def consume_l()::
       while 1::
          lock.acquire()
          if l::
             elt = l.pop()
          lock.release()
          fiddle(elt)

Is that correct both with and without threads (or at least equally
incorrect with and without threads)?

Nested Scopes

The presence of nested scopes will affect where TRACK_GLOBAL finds a
global variable, but shouldn't affect anything after that. (I think.)

Missing Attributes

Suppose I am tracking the object referred to by spam.eggs.ham and
spam.eggs is rebound to an object that does not have a ham attribute.
It's clear this will be an AttributeError if the programmer attempts to
resolve spam.eggs.ham in the current Python virtual machine, but suppose
the programmer has anticipated this case:

    if hasattr(spam.eggs, "ham"):
        print spam.eggs.ham
    elif hasattr(spam.eggs, "bacon"):
        print spam.eggs.bacon
    else:
        print "what? no meat?"

You can't raise an AttributeError when the tracking information is
recalculated. If it does not raise AttributeError and instead lets the
tracking stand, it may be setting the programmer up for a very subtle
error.

One solution to this problem would be to track the shortest possible
root of each dotted expression the function refers to directly. In the
above example, spam.eggs would be tracked, but spam.eggs.ham and
spam.eggs.bacon would not.

Who does the dirty work?

In the Questions section I postulated the existence of a
_PyObject_TrackName function. While the API is fairly easy to specify,
the implementation behind-the-scenes is not so obvious. A central
dictionary could be used to track the name/location mappings, but it
appears that all setattr functions might need to be modified to
accommodate this new functionality.

If all types used the PyObject_GenericSetAttr function to set attributes
that would localize the update code somewhat. They don't however (which
is not too surprising), so it seems that all getattrfunc and
getattrofunc functions will have to be updated. In addition, this would
place an absolute requirement on C extension module authors to call some
function when an attribute changes value (PyObject_TrackUpdate?).

Finally, it's quite possible that some attributes will be set by side
effect and not by any direct call to a setattr method of some sort.
Consider a device interface module that has an interrupt routine that
copies the contents of a device register into a slot in the object's
struct whenever it changes. In these situations, more extensive
modifications would have to be made by the module author. To identify
such situations at compile time would be impossible. I think an extra
slot could be added to PyTypeObjects to indicate if an object's code is
safe for global tracking. It would have a default value of 0
(Py_TRACKING_NOT_SAFE). If an extension module author has implemented
the necessary tracking support, that field could be initialized to 1
(Py_TRACKING_SAFE). _PyObject_TrackName could check that field and issue
a warning if it is asked to track an object that the author has not
explicitly said was safe for tracking.

Discussion

Jeremy Hylton has an alternate proposal on the table[2]. His proposal
seeks to create a hybrid dictionary/list object for use in global name
lookups that would make global variable access look more like local
variable access. While there is no C code available to examine, the
Python implementation given in his proposal still appears to require
dictionary key lookup. It doesn't appear that his proposal could speed
local variable attribute lookup, which might be worthwhile in some
situations if potential performance burdens could be addressed.

Backwards Compatibility

I don't believe there will be any serious issues of backward
compatibility. Obviously, Python bytecode that contains TRACK_OBJECT
opcodes could not be executed by earlier versions of the interpreter,
but breakage at the bytecode level is often assumed between versions.

Implementation

TBD. This is where I need help. I believe there should be either a
central name/location registry or the code that modifies object
attributes should be modified, but I'm not sure the best way to go about
this. If you look at the code that implements the STORE_GLOBAL and
STORE_ATTR opcodes, it seems likely that some changes will be required
to PyDict_SetItem and PyObject_SetAttr or their String variants.
Ideally, there'd be a fairly central place to localize these changes. If
you begin considering tracking attributes of local variables you get
into issues of modifying STORE_FAST as well, which could be a problem,
since the name bindings for local variables are changed much more
frequently. (I think an optimizer could avoid inserting the tracking
code for the attributes for any local variables where the variable's
name binding changes.)

Performance

I believe (though I have no code to prove it at this point), that
implementing TRACK_OBJECT will generally not be much more expensive than
a single LOAD_GLOBAL instruction or a LOAD_GLOBAL/LOAD_ATTR pair. An
optimizer should be able to avoid converting LOAD_GLOBAL and
LOAD_GLOBAL/LOAD_ATTR to the new scheme unless the object access
occurred within a loop. Further down the line, a register-oriented
replacement for the current Python virtual machine[3] could conceivably
eliminate most of the LOAD_FAST instructions as well.

The number of tracked objects should be relatively small. All active
frames of all active threads could conceivably be tracking objects, but
this seems small compared to the number of functions defined in a given
application.

References

Copyright

This document has been placed in the public domain.

[1] https://mail.python.org/pipermail/python-dev/2000-July/007609.html

[2] http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/FastGlobalsPEP

[3] http://www.musi-cal.com/~skip/python/rattlesnake20010813.tar.gz


text:
PEP: 549 Title: Instance Descriptors Version: $Revision$ Last-Modified:
$Date$ Author: Larry Hastings<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 04-Sep-2017 Python-Version: 3.7
Post-History: 04-Sep-2017

Rejection Notice

https://mail.python.org/pipermail/python-dev/2017-November/150528.html

Abstract

Python's descriptor protocol requires that descriptors be members of the
type of an object. This PEP proposes an extension to the descriptor
protocol allowing use of the descriptor protocol for members of
instances. This would permit using properties in modules.

Rationale

Python's descriptor protocol guides programmers towards elegant API
design. If your class supports a data-like member, and you might someday
need to run code when changing the member's value, you're encouraged to
simply declare it as a simple data member of the class for now. If in
the future you do need to run code, you can change it to a "property",
and happily the API doesn't change.

But consider this second bit of best-practice Python API design: if
you're writing a singleton, don't write a class, just build your code
directly into a module. Don't make your users instantiate a singleton
class, don't make your users have to dereference through a singleton
object stored in a module, just have module-level functions and
module-level data.

Unfortunately these two best practices are in opposition. The problem is
that properties aren't supported on modules. Modules are instances of a
single generic module type, and it's not feasible to modify or subclass
this type to add a property to one's module. This means that programmers
facing this API design decision, where the data-like member is a
singleton stored in a module, must preemptively add ugly "getters" and
"setters" for the data.

Adding support for module properties in pure Python has recently become
possible; as of Python 3.5, Python permits assigning to the __class__
attribute of module objects, specifically for this purpose. Here's an
example of using this functionality to add a property to a module:

    import sys, types
    class _MyModuleType(types.ModuleType):
        @property
        def prop(self, instance, owner):
            ...

    sys.modules[__name__].__class__ = _MyModuleType

This works, and is supported behavior, but it's clumsy and obscure.

This PEP proposes a per-type opt-in extension to the descriptor protocol
specifically designed to enable properties in modules. The mechanism is
a way to honor the descriptor protocol for members of instances of a
class without the member being declared as a class variable.

Although this is being proposed as a general mechanism, the author
currently only foresees this as being useful for module objects.

Implementation

The basic idea is simple: modify the tp_descr_get and tp_descr_set
functions exposed by PyModule_Type to inspect the attribute interacted
with, and if it supports the descriptor protocol, call the relevant
exposed function.

Our implementation faces two challenges:

1.  Since this code will be run every time an attribute is looked up on
    a method, it needs to add very little overhead in the general case,
    where the object stored in the attribute is not a descriptor.
2.  Since functions are descriptors, we must take care to not honor the
    descriptor protocol for all objects. Otherwise, all module-level
    functions will suddenly become bound to the module instance as if
    they were method calls on the module object. The module handle would
    be passed in as a "self" argument to all module-level functions.

Both challenges can be solved with the same approach: we define a new
"fast subclass" flag that means "This object is a descriptor, and it
should be honored directly when this object is looked up as an attribute
of an instance". So far this flag is only set on two types: property and
collections.abc.InstanceDescriptor. The latter is an abstract base
class, whose only purpose is to allow user classes to inherit this "fast
subclass" flag.

Prototype

A prototype of this functionality is under development at GitHub
[github].

Acknowledgements

Armin Rigo essentially proposed this mechanism when presented with the
idea of "module properties", and educated the author both on the
complexities of the problem and the proper solution. Nathaniel J. Smith
pointed out the 3.5 extension about assigning to __class__ on module
objects, and provided the example.

References

Copyright

This document has been placed in the public domain.

github

    The branch is here:
    https://github.com/larryhastings/cpython/tree/module-properties A
    pull request against the main CPython repo is here:
    https://github.com/python/cpython/pull/3534