text:
PEP: 459 Title: Standard Metadata Extensions for Python Software
Packages Version: $Revision$ Last-Modified: $Date$ Author: Alyssa
Coghlan<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan
<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status:
Withdrawn Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Requires: 426 Created: 11-Nov-2013 Post-History: 21-Dec-2013

PEP Withdrawal

This PEP depends on PEP 426, which has itself been withdrawn. See the
PEP Withdrawal section in that PEP for details.

In the meantime, metadata extensions will continue to be handled as they
have been for past examples like entry_points.txt: as additional files
installed into metadata directories alongside the main METADATA file.

Abstract

This PEP describes several standard extensions to the Python metadata.

Like all metadata extensions, each standard extension format is
independently versioned. Changing any of the formats requires an update
to this PEP, but does not require an update to the core packaging
metadata.

Standard Extension Namespace

The python project on the Python Package Index refers to the CPython
reference interpreter. This namespace is used as the namespace for the
standard metadata extensions.

The currently defined standard extensions are:

-   python.details
-   python.project
-   python.integrator
-   python.exports
-   python.commands
-   python.constraints

All standard extensions are currently at version 1.0, and thus the
extension_metadata field may be omitted without losing access to any
functionality.

The python.details extension

The python.details extension allows for more information to be provided
regarding the software distribution.

The python.details extension contains four custom subfields:

-   license: the copyright license for the distribution
-   keywords: package index keywords for the distribution
-   classifiers: package index Trove classifiers for the distribution
-   document_names: the names of additional metadata files

All of these fields are optional. Automated tools MUST operate correctly
if a distribution does not provide them, including failing cleanly when
an operation depending on one of these fields is requested.

License

A short string summarising the license used for this distribution.

Note that distributions that provide this field should still specify any
applicable license Trove classifiers in the Classifiers field. Even when
an appropriate Trove classifier is available, the license summary can be
a good way to specify a particular version of that license, or to
indicate any variations or exception to the license.

This field SHOULD contain fewer than 512 characters and MUST contain
fewer than 2048.

This field SHOULD NOT contain any line breaks.

The full license text SHOULD be included as a separate file in the
source archive for the distribution. See Document names for details.

Example:

    "license": "GPL version 3, excluding DRM provisions"

Keywords

A list of additional keywords to be used to assist searching for the
distribution in a larger catalog.

Example:

    "keywords": ["comfy", "chair", "cushions", "too silly", "monty python"]

Classifiers

A list of strings, with each giving a single classification value for
the distribution. Classifiers are described in PEP 301 [2].

Example:

    "classifiers": [
      "Development Status :: 4 - Beta",
      "Environment :: Console (Text Based)",
      "License :: OSI Approved :: GNU General Public License v3 (GPLv3)"
    ]

Document names

Filenames for supporting documents included in the distribution's
dist-info metadata directory.

The following supporting documents can be named:

-   description: a file containing a long description of the
    distribution
-   license: a file with the full text of the distribution's license
-   changelog: a file describing changes made to the distribution

Supporting documents MUST be included directly in the dist-info
directory. Directory separators are NOT permitted in document names.

The markup format (if any) for the file is indicated by the file
extension. This allows index servers and other automated tools to render
included text documents correctly and provide feedback on rendering
errors, rather than having to guess the intended format.

If the filename has no extension, or the extension is not recognised,
the default rendering format MUST be plain text.

The following markup renderers SHOULD be used for the specified file
extensions:

-   Plain text: .txt, no extension, unknown extension
-   reStructured Text: .rst
-   Markdown: .md
-   AsciiDoc: .adoc, .asc, .asciidoc
-   HTML: .html, .htm

Automated tools MAY render one or more of the specified formats as plain
text and MAY render other markup formats beyond those listed.

Automated tools SHOULD NOT make any assumptions regarding the maximum
length of supporting document content, except as necessary to protect
the integrity of a service.

Example:

    "document_names": {
        "description": "README.rst",
        "license": "LICENSE.rst",
        "changelog": "NEWS"
    }

The python.project extension

The python.project extension allows for more information to be provided
regarding the creation and maintenance of the distribution.

The python.project extension contains three custom subfields:

-   contacts: key contact points for the distribution
-   contributors: other contributors to the distribution
-   project_urls: relevant URLs for the distribution

Contact information

Details on individuals and organisations are recorded as mappings with
the following subfields:

-   name: the name of an individual or group
-   email: an email address (this may be a mailing list)
-   url: a URL (such as a profile page on a source code hosting service)
-   role: one of "author", "maintainer" or "contributor"

The name subfield is required, the other subfields are optional.

If no specific role is stated, the default is contributor.

Email addresses must be in the form local-part@domain where the
local-part may be up to 64 characters long and the entire email address
contains no more than 254 characters. The formal specification of the
format is in 5322 (sections 3.2.3 and 3.4.1) and 5321, with a more
readable form given in the informational 3696 and the associated errata.

The defined contributor roles are as follows:

-   author: the original creator of a distribution
-   maintainer: the current lead contributor for a distribution, when
    they are not the original creator
-   contributor: any other individuals or organizations involved in the
    creation of the distribution

Contact and contributor metadata is optional. Automated tools MUST
operate correctly if a distribution does not provide it, including
failing cleanly when an operation depending on one of these fields is
requested.

Contacts

A list of contributor entries giving the recommended contact points for
getting more information about the project.

The example below would be suitable for a project that was in the
process of handing over from the original author to a new lead
maintainer, while operating as part of a larger development group.

Example:

    "contacts": [
      {
        "name": "Python Packaging Authority/Distutils-SIG",
        "email"<EMAIL_ADDRESS>        "url": "https://bitbucket.org/pypa/"
      },
      {
        "name": "Samantha C.",
        "role": "maintainer",
        "email"<EMAIL_ADDRESS>      },
      {
        "name": "Charlotte C.",
        "role": "author",
        "email"<EMAIL_ADDRESS>      }
    ]

Contributors

A list of contributor entries for other contributors not already listed
as current project points of contact. The subfields within the list
elements are the same as those for the main contact field.

Example:

    "contributors": [
      {"name": "John C."},
      {"name": "Erik I."},
      {"name": "Terry G."},
      {"name": "Mike P."},
      {"name": "Graeme C."},
      {"name": "Terry J."}
    ]

Project URLs

A mapping of arbitrary text labels to additional URLs relevant to the
project.

While projects are free to choose their own labels and specific URLs, it
is RECOMMENDED that home page, source control, issue tracker and
documentation links be provided using the labels in the example below.

URL labels MUST be treated as case insensitive by automated tools, but
they are not required to be valid Python identifiers. Any legal JSON
string is permitted as a URL label.

Example:

    "project_urls": {
      "Documentation": "https://distlib.readthedocs.org",
      "Home": "https://bitbucket.org/pypa/distlib",
      "Repository": "https://bitbucket.org/pypa/distlib/src",
      "Tracker": "https://bitbucket.org/pypa/distlib/issues"
    }

The python.integrator extension

Structurally, this extension is largely identical to the python.project
extension (the extension name is the only difference).

However, where the project metadata refers to the upstream creators of
the software, the integrator metadata refers to the downstream
redistributor of a modified version.

If the software is being redistributed unmodified, then typically this
extension will not be used. However, if the software has been patched
(for example, backporting compatible fixes from a later version, or
addressing a platform compatibility issue), then this extension SHOULD
be used, and a local version label added to the distribution's version
identifier.

If there are multiple redistributors in the chain, each one just
overwrites this extension with their particular metadata.

The python.exports extension

Most Python distributions expose packages and modules for import through
the Python module namespace. Distributions may also expose other
interfaces when installed.

The python.exports extension contains three custom subfields:

-   modules: modules exported by the distribution
-   namespaces: namespace packages that the distribution contributes to
-   exports: other Python interfaces exported by the distribution

Export specifiers

An export specifier is a string consisting of a fully qualified name, as
well as an optional extra name enclosed in square brackets. This gives
the following four possible forms for an export specifier:

    module
    module:name
    module[requires_extra]
    module:name[requires_extra]

Note

The jsonschema file currently restricts qualified names using the Python
2 ASCII identifier rules. This may need to be reconsidered given the
more relaxed identifier rules in Python 3.

The meaning of the subfields is as follows:

-   module: the module providing the export
-   name: if applicable, the qualified name of the export within the
    module
-   requires_extra: indicates the export will only work correctly if the
    additional dependencies named in the given extra are available in
    the installed environment

Note

I tried this as a mapping with subfields, and it made the examples below
unreadable. While this PEP is mostly for tool use, readability still
matters to some degree for debugging purposes, and because I expect
snippets of the format to be reused elsewhere.

Modules

A list of qualified names of modules and packages that the distribution
provides for import.

Note

The jsonschema file currently restricts qualified names using the Python
2 ASCII identifier rules. This may need to be reconsidered given the
more relaxed identifier rules in Python 3.

For names that contain dots, the portion of the name before the final
dot MUST appear either in the installed module list or in the namespace
package list.

To help avoid name conflicts, it is RECOMMENDED that distributions
provide a single top level module or package that matches the
distribution name (or a lower case equivalent). This requires that the
distribution name also meet the requirements of a Python identifier,
which are stricter than those for distribution names). This practice
will also make it easier to find authoritative sources for modules.

Index servers SHOULD allow multiple distributions to publish the same
modules, but MAY notify distribution authors of potential conflicts.

Installation tools SHOULD report an error when asked to install a
distribution that provides a module that is also provided by a
different, previously installed, distribution.

Note that attempting to import some declared modules may result in an
exception if the appropriate extras are not installed.

Example:

    "modules": ["chair", "chair.cushions", "python_sketches.nobody_expects"]

Note

Making this a list of export specifiers instead would allow a
distribution to declare when a particular module requires a particular
extra in order to run correctly. On the other hand, there's an argument
to be made that that is the point where it starts to become worthwhile
to split out a separate distribution rather than using extras.

Namespaces

A list of qualified names of namespace packages that the distribution
contributes modules to.

Note

The jsonschema file currently restricts qualified names using the Python
2 ASCII identifier rules. This may need to be reconsidered given the
more relaxed identifier rules in Python 3.

On versions of Python prior to Python 3.3 (which provides native
namespace package support), installation tools SHOULD emit a suitable
__init__.py file to properly initialise the namespace rather than using
a distribution provided file.

Installation tools SHOULD emit a warning and MAY emit an error if a
distribution declares a namespace package that conflicts with the name
of an already installed module or vice-versa.

Example:

    "namespaces": ["python_sketches"]

Exports

The exports field is a mapping containing prefixed names as keys. Each
key identifies an export group containing one or more exports published
by the distribution.

Export group names are defined by distributions that will then make use
of the published export information in some way. The primary use case is
for distributions that support a plugin model: defining an export group
allows other distributions to indicate which plugins they provide, how
they can be imported and accessed, and which additional dependencies (if
any) are needed for the plugin to work correctly.

To reduce the chance of name conflicts, export group names SHOULD use a
prefix that corresponds to a module name in the distribution that
defines the meaning of the export group. This practice will also make it
easier to find authoritative documentation for export groups.

Each individual export group is then a mapping of arbitrary non-empty
string keys to export specifiers. The meaning of export names within an
export group is up to the distribution that defines the export group.
Creating an appropriate definition for the export name format can allow
the importing distribution to determine whether or not an export is
relevant without needing to import every exporting module.

Example:

    "exports": {
      "nose.plugins.0.10": {
        "chairtest": "chair:NosePlugin"
      }
    }

The python.commands extension

The python.commands extension contains three custom subfields:

-   wrap_console: console wrapper scripts to be generated by the
    installer
-   wrap_gui: GUI wrapper scripts to be generated by the installer
-   prebuilt: scripts created by the distribution's build process and
    installed directly to the configured scripts directory

wrap_console and wrap_gui are both mappings of script names to export
specifiers. The script names must follow the same naming rules as
distribution names.

The export specifiers for wrapper scripts must refer to either a package
with a __main__ submodule (if no name subfield is given in the export
specifier) or else to a callable inside the named module.

Installation tools should generate appropriate wrappers as part of the
installation process.

Note

Still needs more detail on what "appropriate wrappers" means. For now,
refer to what setuptools and zc.buildout generate as wrapper scripts.

prebuilt is a list of script paths, relative to the scripts directory in
a wheel file or following installation. They are provided for
informational purpose only - installing them is handled through the
normal processes for files created when building a distribution.

Build tools SHOULD mark this extension as requiring handling by
installers.

Index servers SHOULD allow multiple distributions to publish the same
commands, but MAY notify distribution authors of potential conflicts.

Installation tools SHOULD report an error when asked to install a
distribution that provides a command that is also provided by a
different, previously installed, distribution.

Example:

    "python.commands": {
      "installer_must_handle": true,
      "wrap_console": [{"chair": "chair:run_cli"}],
      "wrap_gui": [{"chair-gui": "chair:run_gui"}],
      "prebuilt": ["reduniforms"]
    }

The python.constraints extension

The python.constraints extension contains two custom subfields:

-   environments: supported installation environments
-   extension_metadata: required exact matches in extension metadata
    fields published by other installed components

Build tools SHOULD mark this extension as requiring handling by
installers.

Index servers SHOULD allow distributions to be uploaded with constraints
that cannot be satisfied using that index, but MAY notify distribution
authors of any such potential compatibility issues.

Installation tools SHOULD report an error if constraints are specified
by the distribution and the target installation environment fails to
satisfy them, MUST at least emit a warning, and MAY allow the user to
force the installation to proceed regardless.

Example:

    "python.constraints": {
      "installer_must_handle": true,
      "environments": ["python_version >= 2.6"],
      "extension_metadata": {
        "fortranlib": {
          "fortranlib.compatibility": {
            "fortran_abi": "openblas-g77"
          }
        }
      }
    }

Supported Environments

The environments subfield is a list of strings specifying the
environments that the distribution explicitly supports. An environment
is considered supported if it matches at least one of the environment
markers given.

If this field is not given in the metadata, it is assumed that the
distribution supports any platform supported by Python.

Individual entries are environment markers, as described in PEP 426.

The two main uses of this field are to declare which versions of Python
and which underlying operating systems are supported.

Examples indicating supported Python versions:

    # Supports Python 2.6+
    "environments": ["python_version >= '2.6'"]

    # Supports Python 2.6+ (for 2.x) or 3.3+ (for 3.x)
    "environments": ["python_version >= '3.3'",
                     "'3.0' > python_version >= '2.6'"]

Examples indicating supported operating systems:

    # Windows only
    "environments": ["sys_platform == 'win32'"]

    # Anything except Windows
    "environments": ["sys_platform != 'win32'"]

    # Linux or BSD only
    "environments": ["'linux' in sys_platform",
                     "'bsd' in sys_platform"]

Example where the supported Python version varies by platform:

    # The standard library's os module has long supported atomic renaming
    # on POSIX systems, but only gained atomic renaming on Windows in Python
    # 3.3. A distribution that needs atomic renaming support for reliable
    # operation might declare the following supported environments.
    "environment": ["python_version >= '2.6' and sys_platform != 'win32'",
                    "python_version >= '3.3' and sys_platform == 'win32'"]

Extension metadata constraints

The extension_metadata subfield is a mapping from distribution names to
extension metadata snippets that are expected to exactly match the
metadata of the named distribution in the target installation
environment.

Each submapping then consists of a mapping from metadata extension names
to the exact expected values of a subset of fields.

For example, a distribution called fortranlib may publish a different
FORTRAN ABI depending on how it is built, and any related projects that
are installed into the same runtime environment should use matching
build options. This can be handled by having the base distribution
publish a custom extension that indicates the build option that was used
to create the binary extensions:

    "extensions": {
      "fortranlib.compatibility": {
        "fortran_abi": "openblas-g77"
      }
    }

Other distributions that contain binary extensions that need to be
compatible with the base distribution would then define a suitable
constraint in their own metadata:

    "python.constraints": {
      "installer_must_handle": true,
      "extension_metadata": {
        "fortranlib": {
          "fortranlib.compatibility": {
            "fortran_abi": "openblas-g77"
          }
        }
      }
    }

This constraint specifies that:

-   fortranlib must be installed (this should also be expressed as a
    normal dependency so that installers ensure it is satisfied)
-   The installed version of fortranlib must include the custom
    fortranlib.compatibility extension in its published metadata
-   The fortan_abi subfield of that extension must have the exact value
    openblas-g77.

If all of these conditions are met (the distribution is installed, the
specified extension is included in the metadata, the specified subfields
have the exact specified value), then the constraint is considered to be
satisfied.

Note

The primary intended use case here is allowing C extensions with
additional ABI compatibility requirements to declare those in a way that
any installation tool can enforce without needing to understand the
details. In particular, many NumPy based scientific libraries need to be
built using a consistent set of FORTRAN libraries, hence the
"fortranlib" example.

This is the reason there's no support for pattern matching or boolean
logic: even the "simple" version of this extension is relatively
complex, and there's currently no compelling rationale for making it
more complicated than it already is.

Copyright

This document has been placed in the public domain.


text:
PEP: 311 Title: Simplified Global Interpreter Lock Acquisition for
Extensions Version: $Revision$ Last-Modified: $Date$ Author: Mark
Hammond<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 05-Feb-2003 Python-Version: 2.3
Post-History: 05-Feb-2003, 14-Feb-2003, 19-Apr-2003

Abstract

This PEP proposes a simplified API for access to the Global Interpreter
Lock (GIL) for Python extension modules. Specifically, it provides a
solution for authors of complex multi-threaded extensions, where the
current state of Python (i.e., the state of the GIL is unknown.

This PEP proposes a new API, for platforms built with threading support,
to manage the Python thread state. An implementation strategy is
proposed, along with an initial, platform independent implementation.

Rationale

The current Python interpreter state API is suitable for simple,
single-threaded extensions, but quickly becomes incredibly complex for
non-trivial, multi-threaded extensions.

Currently Python provides two mechanisms for dealing with the GIL:

-   Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros. These macros
    are provided primarily to allow a simple Python extension that
    already owns the GIL to temporarily release it while making an
    "external" (ie, non-Python), generally expensive, call. Any existing
    Python threads that are blocked waiting for the GIL are then free to
    run. While this is fine for extensions making calls from Python into
    the outside world, it is no help for extensions that need to make
    calls into Python when the thread state is unknown.
-   PyThreadState and PyInterpreterState APIs. These API functions allow
    an extension/embedded application to acquire the GIL, but suffer
    from a serious boot-strapping problem - they require you to know the
    state of the Python interpreter and of the GIL before they can be
    used. One particular problem is for extension authors that need to
    deal with threads never before seen by Python, but need to call
    Python from this thread. It is very difficult, delicate and error
    prone to author an extension where these "new" threads always know
    the exact state of the GIL, and therefore can reliably interact with
    this API.

For these reasons, the question of how such extensions should interact
with Python is quickly becoming a FAQ. The main impetus for this PEP, a
thread on python-dev[1], immediately identified the following projects
with this exact issue:

-   The win32all extensions
-   Boost
-   ctypes
-   Python-GTK bindings
-   Uno
-   PyObjC
-   Mac toolbox
-   PyXPCOM

Currently, there is no reasonable, portable solution to this problem,
forcing each extension author to implement their own hand-rolled
version. Further, the problem is complex, meaning many implementations
are likely to be incorrect, leading to a variety of problems that will
often manifest simply as "Python has hung".

While the biggest problem in the existing thread-state API is the lack
of the ability to query the current state of the lock, it is felt that a
more complete, simplified solution should be offered to extension
authors. Such a solution should encourage authors to provide error-free,
complex extension modules that take full advantage of Python's threading
mechanisms.

Limitations and Exclusions

This proposal identifies a solution for extension authors with complex
multi-threaded requirements, but that only require a single
"PyInterpreterState". There is no attempt to cater for extensions that
require multiple interpreter states. At the time of writing, no
extension has been identified that requires multiple
PyInterpreterStates, and indeed it is not clear if that facility works
correctly in Python itself.

This API will not perform automatic initialization of Python, or
initialize Python for multi-threaded operation. Extension authors must
continue to call Py_Initialize(), and for multi-threaded applications,
PyEval_InitThreads(). The reason for this is that the first thread to
call PyEval_InitThreads() is nominated as the "main thread" by Python,
and so forcing the extension author to specify the main thread (by
requiring them to make this first call) removes ambiguity. As
Py_Initialize() must be called before PyEval_InitThreads(), and as both
of these functions currently support being called multiple times, the
burden this places on extension authors is considered reasonable.

It is intended that this API be all that is necessary to acquire the
Python GIL. Apart from the existing, standard Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS macros, it is assumed that no additional thread
state API functions will be used by the extension. Extensions with such
complicated requirements are free to continue to use the existing thread
state API.

Proposal

This proposal recommends a new API be added to Python to simplify the
management of the GIL. This API will be available on all platforms built
with WITH_THREAD defined.

The intent is that assuming Python has correctly been initialized, an
extension author be able to use a small, well-defined "prologue dance",
at any time and on any thread, which will ensure Python is ready to be
used on that thread. After the extension has finished with Python, it
must also perform an "epilogue dance" to release any resources
previously acquired. Ideally, these dances can be expressed in a single
line.

Specifically, the following new APIs are proposed:

    /* Ensure that the current thread is ready to call the Python
       C API, regardless of the current state of Python, or of its
       thread lock.  This may be called as many times as desired
       by a thread so long as each call is matched with a call to
       PyGILState_Release().  In general, other thread-state APIs may
       be used between _Ensure() and _Release() calls, so long as the
       thread-state is restored to its previous state before the Release().
       For example, normal use of the Py_BEGIN_ALLOW_THREADS/
       Py_END_ALLOW_THREADS macros are acceptable.

       The return value is an opaque "handle" to the thread state when
       PyGILState_Acquire() was called, and must be passed to
       PyGILState_Release() to ensure Python is left in the same state. Even
       though recursive calls are allowed, these handles can *not* be
       shared - each unique call to PyGILState_Ensure must save the handle
       for its call to PyGILState_Release.

       When the function returns, the current thread will hold the GIL.

       Failure is a fatal error.
    */
    PyAPI_FUNC(PyGILState_STATE) PyGILState_Ensure(void);

    /* Release any resources previously acquired.  After this call, Python's
       state will be the same as it was prior to the corresponding
       PyGILState_Acquire call (but generally this state will be unknown to
       the caller, hence the use of the GILState API.)

       Every call to PyGILState_Ensure must be matched by a call to
       PyGILState_Release on the same thread.
    */
    PyAPI_FUNC(void) PyGILState_Release(PyGILState_STATE);

Common usage will be:

    void SomeCFunction(void)
    {
        /* ensure we hold the lock */
        PyGILState_STATE state = PyGILState_Ensure();
        /* Use the Python API */
        ...
        /* Restore the state of Python */
        PyGILState_Release(state);
    }

Design and Implementation

The general operation of PyGILState_Ensure() will be:

-   assert Python is initialized.
-   Get a PyThreadState for the current thread, creating and saving if
    necessary.
-   remember the current state of the lock (owned/not owned)
-   If the current state does not own the GIL, acquire it.
-   Increment a counter for how many calls to PyGILState_Ensure have
    been made on the current thread.
-   return

The general operation of PyGILState_Release() will be:

-   assert our thread currently holds the lock.
-   If old state indicates lock was previously unlocked, release GIL.
-   Decrement the PyGILState_Ensure counter for the thread.
-   If counter == 0:
    -   release and delete the PyThreadState.
    -   forget the ThreadState as being owned by the thread.
-   return

It is assumed that it is an error if two discrete PyThreadStates are
used for a single thread. Comments in pystate.h ("State unique per
thread") support this view, although it is never directly stated. Thus,
this will require some implementation of Thread Local Storage.
Fortunately, a platform independent implementation of Thread Local
Storage already exists in the Python source tree, in the SGI threading
port. This code will be integrated into the platform independent Python
core, but in such a way that platforms can provide a more optimal
implementation if desired.

Implementation

An implementation of this proposal can be found at
https://bugs.python.org/issue684256

References

Copyright

This document has been placed in the public domain.

[1] David Abrahams, Extension modules, Threading, and the GIL
https://mail.python.org/pipermail/python-dev/2002-December/031424.html

license: Public Domain


text:
PEP: 3 Title: Guidelines for Handling Bug Reports Author: Jeremy Hylton
<EMAIL_ADDRESS>Status: Withdrawn Type: Process Created:
25-Sep-2000 Post-History:

This PEP contained guidelines for handling bug reports in the Python bug
tracker. It has been replaced by the Developer's Guide description of
issue triaging. Guidelines for people submitting Python bugs are in the
Python docs.

Original Guidelines

1.  Make sure the bug category and bug group are correct. If they are
    correct, it is easier for someone interested in helping to find out,
    say, what all the open Tkinter bugs are.

2.  If it's a minor feature request that you don't plan to address right
    away, add it to PEP 42 or ask the owner to add it for you. If you
    add the bug to PEP 42, mark the bug as "feature request", "later",
    and "closed"; and add a comment to the bug saying that this is the
    case (mentioning the PEP explicitly).

      XXX do we prefer the tracker or PEP 42?

3.  Assign the bug a reasonable priority. We don't yet have a clear
    sense of what each priority should mean. One rule, however, is that
    bugs with priority "urgent" or higher must be fixed before the next
    release.

4.  If a bug report doesn't have enough information to allow you to
    reproduce or diagnose it, ask the original submitter for more
    information. If the original report is really thin and your email
    doesn't get a response after a reasonable waiting period, you can
    close the bug.

5.  If you fix a bug, mark the status as "Fixed" and close it. In the
    comments, include the SVN revision numbers of the commit(s). In the
    SVN checkin message, include the issue number and a normal
    description of the change, mentioning the contributor if a patch was
    applied.

6.  If you are assigned a bug that you are unable to deal with, assign
    it to someone else if you think they will be able to deal with it,
    otherwise it's probably best to unassign it.

References

-   https://bugs.python.org


text:
PEP: 613 Title: Explicit Type Aliases Author: Shannon Zhu<EMAIL_ADDRESS>Sponsor: Guido van Rossum<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Typing Created: 21-Jan-2020
Python-Version: 3.10 Post-History: 21-Jan-2020

typing:type-aliases and typing.TypeAlias

Abstract

Type aliases are user-specified types which may be as complex as any
type hint, and are specified with a simple variable assignment on a
module top level.

This PEP formalizes a way to explicitly declare an assignment as a type
alias.

Motivation

Type aliases are declared as top level variable assignments. In
PEP 484 <484#type-aliases>, the distinction between a valid type alias
and a global variable was implicitly determined: if a top level
assignment is unannotated, and the assigned value is a valid type, then
the name being assigned to is a valid type alias. Otherwise, that name
is simply a global value that cannot be used as a type hint.

These implicit type alias declaration rules create confusion when type
aliases involve forward references, invalid types, or violate other
restrictions enforced on type alias declaration. Because the distinction
between an unannotated value and a type alias is implicit, ambiguous or
incorrect type alias declarations implicitly default to a valid value
assignment. This creates expressions that are impossible to express as
type aliases and punts error diagnosis of malformed type aliases
downstream.

The following examples each include an illustration of some of the
suboptimal or confusing behaviors resulting from existing implicit alias
declarations. We also introduce explicit aliases of the format
TypeName: TypeAlias = Expression here for the sake of comparison, but
the syntax is discussed in further detail in later sections.

Forward References:

    MyType = "ClassName"
    def foo() -> MyType: ...

This code snippet should not error so long as ClassName is defined later
on. However, a type checker is forced to assume that MyType is a value
assignment rather than a type alias, and therefore may throw spurious
errors that (1) MyType is an unannotated global string, and (2) MyType
cannot be used as a return annotation because it is not a valid type.

    MyType: TypeAlias = "ClassName"
    def foo() -> MyType: ...

Explicit aliases remove ambiguity so neither of the above errors will be
thrown. Additionally, if something is wrong with ClassName (i.e., it’s
not actually defined later), the type checker can throw an error.

Error Messaging:

    MyType1 = InvalidType
    MyType2 = MyGeneric(int)  # i.e., intention was MyGeneric[int]

A type checker should warn on this code snippet that InvalidType is not
a valid type, and therefore cannot be used to annotate an expression or
to construct a type alias. Instead, type checkers are forced to throw
spurious errors that (1) MyType is a global expression missing an
annotation, and (2) MyType is not a valid type in all usages of MyType
across the codebase.

    MyType1: TypeAlias = InvalidType
    MyType2: TypeAlias = MyGeneric(int)

With explicit aliases, the type checker has enough information to error
on the actual definition of the bad type alias, and explain why: that
MyGeneric(int) and InvalidType are not valid types. When the value
expression is no longer evaluated as a global value, unactionable type
errors on all usages of MyType across the codebase can be suppressed.

Scope Restrictions:

    class Foo:
      x = ClassName
      y: TypeAlias = ClassName
      z: Type[ClassName] = ClassName

Type aliases are valid within class scope, both implicitly (x) and
explicitly (y). If the line should be interpreted as a class variable,
it must be explicitly annotated (z).

    x = ClassName
    def foo() -> None:
      x = ClassName

The outer x is a valid type alias, but type checkers must error if the
inner x is ever used as a type because type aliases cannot be defined
inside of a function. This is confusing because the alias declaration
rule is not explicit, and because a type error will not be thrown on the
location of the inner type alias declaration but rather on every one of
its subsequent use cases.

    x: TypeAlias = ClassName
    def foo() -> None:
      x = ClassName
    def bar() -> None:
      x: TypeAlias = ClassName

With explicit aliases, the outer assignment is still a valid type
variable. Inside foo, the inner assignment should be interpreted as
x: Type[ClassName]. Inside bar, the type checker should raise a clear
error, communicating to the author that type aliases cannot be defined
inside a function.

Specification

The explicit alias declaration syntax clearly differentiates between the
three possible kinds of assignments: typed global expressions, untyped
global expressions, and type aliases. This avoids the existence of
assignments that break type checking when an annotation is added, and
avoids classifying the nature of the assignment based on the type of the
value.

Implicit syntax (pre-existing):

    x = 1  # untyped global expression
    x: int = 1  # typed global expression

    x = int  # type alias
    x: Type[int] = int  # typed global expression

Explicit syntax:

    x = 1  # untyped global expression
    x: int = 1  # typed global expression

    x = int  # untyped global expression (see note below)
    x: Type[int] = int  # typed global expression

    x: TypeAlias = int  # type alias
    x: TypeAlias = "MyClass"  # type alias

Note: The examples above illustrate implicit and explicit alias
declarations in isolation. For the sake of backwards compatibility, type
checkers should support both simultaneously, meaning an untyped global
expression x = int will still be considered a valid type alias.

Backwards Compatibility

Explicit aliases provide an alternative way to declare type aliases, but
all pre-existing code and old alias declarations will work as before.

Reference Implementation

The Pyre type checker supports explicit type alias declarations.

Rejected Ideas

Some alternative syntaxes were considered for explicit aliases:

    MyType: TypeAlias[int]

This looks a lot like an uninitialized variable.

    MyType = TypeAlias[int]

Along with the option above, this format potentially adds confusion
around what the runtime value of MyType is.

In comparison, the chosen syntax option MyType: TypeAlias = int is
appealing because it still sticks with the MyType = int assignment
syntax, and adds some information for the type checker purely as an
annotation.

Version History

-   2021-11-16
    -   Allow TypeAlias inside class scope

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 700 Title: Additional Fields for the Simple API for Package Indexes
Author: Paul Moore<EMAIL_ADDRESS>PEP-Delegate: Donald Stufft
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-700-additional-fields-for-the-simple-api-for-package-indexes/20177
Status: Final Type: Standards Track Topic: Packaging Created:
21-Oct-2022 Post-History: 21-Oct-2022 Resolution: 19-Dec-2022

packaging:simple-repository-api

Abstract

PEP 691 defined a JSON form for the "Simple Repository API". This
allowed clients to more easily query the data that was previously only
available in HTML, as defined in PEP 503.

This proposal adds three fields to the JSON form, which allow it to be
used in place of PyPI's JSON API in a number of situations.

-   A field to allow retrieval of a list of all the published versions
    of a project.
-   Fields containing the size and upload time for a project file.

The new fields are all part of the data returned from the "project
details" URL.

Rationale

With the introduction of the JSON form of the simple API in PEP 691, the
simple API offers functionality that is almost as complete as the PyPI
JSON API. This PEP adds a number of fields which were previously only
available through the JSON API, in order to allow more clients which
were previously Warehouse specific to support arbitrary
standards-compliant indexes.

Specification

This specification defines version 1.1 of the simple repository API. For
the HTML version of the API, there is no change from version 1.0. For
the JSON version of the API, the following changes are made:

-   The api-version must specify version 1.1 or later.
-   A new versions key is added at the top level.
-   Two new "file information" keys, size and upload-time, are added to
    the files data.
-   Keys (at any level) with a leading underscore are reserved as
    private for index server use. No future standard will assign a
    meaning to any such key.

The versions and size keys are mandatory. The upload-time key is
optional.

Versions

An additional key, versions MUST be present at the top level, in
addition to the keys name, files and meta defined in PEP 691. This key
MUST contain a list of version strings specifying all of the project
versions uploaded for this project. The value is logically a set, and as
such may not contain duplicates, and the order of the values is not
significant.

All of the files listed in the files key MUST be associated with one of
the versions in the versions key. The versions key MAY contain versions
with no associated files (to represent versions with no files uploaded,
if the server has such a concept).

Note that because servers may hold "legacy" data from before the
adoption of PEP 440, version strings currently cannot be required to be
valid PEP 440 versions, and therefore cannot be assumed to be orderable
using the PEP 440 rules. However, servers SHOULD use normalised PEP 440
versions where possible.

Additional file information

Two new keys are added to the files key.

-   size: This field is mandatory. It MUST contain an integer which is
    the file size in bytes.
-   upload-time: This field is optional. If present, it MUST contain a
    valid ISO 8601 date/time string, in the format
    yyyy-mm-ddThh:mm:ss.ffffffZ, which represents the time the file was
    uploaded to the index. As indicated by the Z suffix, the upload time
    MUST use the UTC timezone. The fractional seconds part of the
    timestamp (the .ffffff part) is optional, and if present may contain
    up to 6 digits of precision. If a server does not record upload time
    information for a file, it MAY omit the upload-time key.

FAQ

Why not add this data to the HTML API as well?

It would be possible to add the data to the HTML API, but the vast
majority of consumers for this data are likely to be currently getting
it from the PyPI JSON API, and so will already be expecting to parse
JSON. Traditional consumers of the HTML API have never needed this data
previously.

Does this imply that the HTML API is obsolete?

No. The FAQ of PEP 691 was clear that the HTML API is not being
deprecated, and this PEP does not change that position. However, clients
wishing to access the new data introduced by this PEP will need to use
the JSON API to get it. And indexes wanting to provide it will need to
serve the JSON format.

Is the simple API replacing the Warehouse JSON and XML-RPC APIs?

Where possible, clients should prefer the simple API over the JSON or
XML-RPC APIs, as the former is standardised and can be assumed to be
available from any index, whereas the latter are exclusive to the
Warehouse project.

However, while this PEP brings the simple API closer to being able to
replace the JSON API, there is no formal policy that the simple API will
replicate all of the functionality covered by the existing Warehouse
APIs. Proposed additions to the simple API will still be considered on
their individual merits, and the requirement that the API should be
simple and fast for the primary use case of locating files for a project
will remain the overriding consideration.

Why not allow other date formats?

The ISO 8601 standard is complex, and there seems little value in
requiring clients to deal with that. The standard library datetime
module provides methods to parse ISO 8601 strings, but it is possible
that users may want to access index data without using Python (for
example, piping the output of curl into jq). Having a single,
well-defined format makes this easier, and doesn't have any significant
disadvantages.

What if file sizes are too big for a JSON number?

The JSON standard does not specify how numbers are to be interpreted.
Python can read and write arbitrary-length integers in a JSON file, so
this should not be an issue for code written in Python. Non-Python
implementations may need to take care to handle large integers
correctly, but this is not expected to be a significant problem.

Why not require PEP 440 versions?

At the time this PEP was written, PyPI still contains (and serves)
projects and files with "legacy" versions. Requiring PEP 440 versions
would make it impossible for PyPI to follow this specification while
still serving the existing content.

Ideally, at some future time, the simple index API will be updated to
require PEP 440 versions, at which time this specification should be
updated to reflect that. However, that change will need to be
co-ordinated with existing index providers including PyPI, to desupport
and remove non-conforming projects and/or files.

Why not provide a "latest version" value?

For PEP 440 versions, this is easy enough for the client to do (using
the packaging library,
latest = max(Version(s) for s in proj["versions"])). For non-standard
versions, there is no well-defined ordering, and clients will need to
decide on what rule is appropriate for their needs. Requiring the server
to supply a latest version value takes the choice away from the client.

Servers with an explicit concept of which version is the "latest", which
cannot be calculated from data available to the client, can provide a
non-standard, underscore-prefixed key to convey that information to the
client if they wish.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 685 Title: Comparison of extra names for optional distribution
dependencies Author: Brett Cannon<EMAIL_ADDRESS>PEP-Delegate: Paul
Moore<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/14141 Status: Accepted Type: Standards
Track Topic: Packaging Content-Type: text/x-rst Created: 08-Mar-2022
Post-History: 08-Mar-2022 Resolution:
https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141/55

Abstract

This PEP specifies how to normalize distribution extra names when
performing comparisons. This prevents tools from either failing to find
an extra name, or accidentally matching against an unexpected name.

Motivation

The Provides-Extra core metadata specification states that an extra's
name "must be a valid Python identifier". PEP 508 specifies that the
value of an extra marker may contain a letter, digit, or any one of .,
-, or _ after the initial character. There is no other PyPA
specification which outlines how extra names should be written or
normalized for comparison. Due to the amount of packaging-related code
in existence, it is important to evaluate current practices by the
community and standardize on one that doesn't break most existing code,
while being something tool authors can agree to following.

The issue of there being no consistent standard was brought forward by
an initial discussion noting that the extra adhoc-ssl was not considered
equal to the name adhoc_ssl by pip 22.

Rationale

PEP 503 specifies how to normalize distribution names:

    re.sub(r"[-_.]+", "-", name).lower()

This collapses any run of the characters -, _ and . down to a single -.
For example, --- . and __ all get converted to just -. This does not
produce a valid Python identifier, per the core metadata 2.2
specification for extra names.

Setuptools 60 performs normalization via:

    re.sub(r'[^A-Za-z0-9-.]+', '_', name).lower()

The use of an underscore/_ differs from PEP 503's use of a hyphen/-, and
it also normalizes characters outside of those allowed by PEP 508. Runs
of . and -, unlike PEP 503, do not get normalized to one _, e.g. ..
stays the same. To note, this is inconsistent with this function's
docstring, which does specify that all non-alphanumeric characters
(which would include - and .) are normalized and collapsed.

For pip 22, its "extra normalisation behaviour is quite convoluted and
erratic" [pip-erratic] and so its use is not considered.

Specification

When comparing extra names, tools MUST normalize the names being
compared using the semantics outlined in PEP 503 <0503#normalized-names>
for names:

    re.sub(r"[-_.]+", "-", name).lower()

The core metadata specification will be updated such that the allowed
names for Provides-Extra matches what PEP 508 specifies for names. This
will bring extra naming in line with that of the Name field. Because
this changes what is considered valid, it will lead to a core metadata
version increase to 2.3.

For tools writing core metadata, they MUST write out extra names in
their normalized form. This applies to the Provides-Extra field and the
extra marker <0508#extras> when used in the Requires-Dist field.

Tools generating metadata MUST raise an error if a user specified two or
more extra names which would normalize to the same name. Tools
generating metadata MUST raise an error if an invalid extra name is
provided as appropriate for the specified core metadata version. If a
project's metadata specifies an older core metadata version and the name
would be invalid with newer core metadata versions, tools reading that
metadata SHOULD warn the user. Tools SHOULD warn users when an invalid
extra name is read and SHOULD ignore the name to avoid ambiguity. Tools
MAY raise an error instead of a warning when reading an invalid name, if
they so desire.

Backwards Compatibility

Moving to PEP 503 normalization and PEP 508 name acceptance allows for
all preexisting, valid names to continue to be valid.

Based on research looking at a collection of wheels on PyPI
[pypi-results], the risk of extra name clashes is limited to 73
instances when considering all extras names on PyPI, valid or not (not
just those within a single package) while only looking at valid names
leads to only 3 clashes:

-   dev-test: dev_test, dev-test, dev.test
-   dev-lint: dev-lint, dev.lint, dev_lint
-   apache-beam: apache-beam, apache.beam

By requiring tools writing core metadata to only record the normalized
name, the issue of preexisting, invalid extra names should diminish over
time.

Security Implications

It is possible that for a distribution that has conflicting extra names,
a tool ends up installing dependencies that somehow weaken the security
of the system. This is only hypothetical and if it were to occur, it
would probably be more of a security concern for the distributions
specifying such extras names rather than the distribution that pulled
them in together.

How to Teach This

This should be transparent to users on a day-to-day basis. It will be up
to tools to educate/stop users when they select extra names which
conflict.

Reference Implementation

No reference implementation is provided aside from the code above, but
the expectation is the packaging project will provide a function in its
packaging.utils module that will implement extra name normalization. It
will also implement extra name comparisons appropriately. Finally, if
the project ever gains the ability to write out metadata, it will also
implement this PEP.

Transition Plan

There is a risk that a build tool will produce core metadata conforming
to version 2.3 and thus this PEP but which is consumed by a tool that is
unaware of this PEP (if that tool chooses to attempt to read a core
metadata version it does not directly support). In such a case there is
a chance that a user may specify an extra using an non-normalized name
which worked previously but which fails now.

As such, consumers of this PEP should be prioritized more than producers
so that users can be notified that they are specifying extra names which
are not normalized (and thus may break in the future).

Rejected Ideas

Using setuptools 60's normalization

Initially, this PEP proposed using setuptools safe_extra() for
normalization to try to minimize backwards-compatibility issues.
However, after checking various wheels on PyPI, it became clear that
standardizing all naming on PEP 508 and PEP 503 semantics was easier and
better long-term, while causing minimal backwards compatibility issues.

Open Issues

N/A

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

pip-erratic

    Tzu-ping Chung on Python Discourse
    <https://discuss.python.org/t/7614/10

pypi-results

    Paul Moore on Python Discourse https://discuss.python.org/t/14141/17


text:
PEP: 260 Title: Simplify xrange() Author: Guido van Rossum
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 26-Jun-2001 Python-Version: 2.2 Post-History:
26-Jun-2001

Abstract

This PEP proposes to strip the xrange() object from some rarely used
behavior like x[i:j] and x*n.

Problem

The xrange() function has one idiomatic use:

    for i in xrange(...): ...

However, the xrange() object has a bunch of rarely used behaviors that
attempt to make it more sequence-like. These are so rarely used that
historically they have has serious bugs (e.g. off-by-one errors) that
went undetected for several releases.

I claim that it's better to drop these unused features. This will
simplify the implementation, testing, and documentation, and reduce
maintenance and code size.

Proposed Solution

I propose to strip the xrange() object to the bare minimum. The only
retained sequence behaviors are x[i], len(x), and repr(x). In
particular, these behaviors will be dropped:

-   x[i:j] (slicing)
-   x*n, n*x (sequence-repeat)
-   cmp(x1, x2) (comparisons)
-   i in x (containment test)
-   x.tolist() method
-   x.start, x.stop, x.step attributes

I also propose to change the signature of the PyRange_New() C API to
remove the 4th argument (the repetition count).

By implementing a custom iterator type, we could speed up the common
use, but this is optional (the default sequence iterator does just
fine).

Scope

This PEP affects the xrange() built-in function and the PyRange_New() C
API.

Risks

Somebody's code could be relying on the extended code, and this code
would break. However, given that historically bugs in the extended code
have gone undetected for so long, it's unlikely that much code is
affected.

Transition

For backwards compatibility, the existing functionality will still be
present in Python 2.2, but will trigger a warning. A year after Python
2.2 final is released (probably in 2.4) the functionality will be ripped
out.

Copyright

This document has been placed in the public domain.


text:
PEP: 651 Title: Robust Stack Overflow Handling Author: Mark Shannon
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type:
text/x-rst Created: 18-Jan-2021 Post-History: 19-Jan-2021

Rejection Notice

This PEP has been rejected by the Python Steering Council.

Abstract

This PEP proposes that Python should treat machine stack overflow
differently from runaway recursion.

This would allow programs to set the maximum recursion depth to fit
their needs and provide additional safety guarantees.

If this PEP is accepted, then the following program will run safely to
completion:

    sys.setrecursionlimit(1_000_000)

    def f(n):
        if n:
            f(n-1)

    f(500_000)

and the following program will raise a StackOverflow, without causing a
VM crash:

    sys.setrecursionlimit(1_000_000)

    class X:
        def __add__(self, other):
            return self + other

    X() + 1

Motivation

CPython uses a single recursion depth counter to prevent both runaway
recursion and C stack overflow. However, runaway recursion and machine
stack overflow are two different things. Allowing machine stack overflow
is a potential security vulnerability, but limiting recursion depth can
prevent the use of some algorithms in Python.

Currently, if a program needs to deeply recurse it must manage the
maximum recursion depth allowed, hopefully managing to set it in the
region between the minimum needed to run correctly and the maximum that
is safe to avoid a memory protection error.

By separating the checks for C stack overflow from checks for recursion
depth, pure Python programs can run safely, using whatever level of
recursion they require.

Rationale

CPython currently relies on a single limit to guard against potentially
dangerous stack overflow in the virtual machine and to guard against run
away recursion in the Python program.

This is a consequence of the implementation which couples the C and
Python call stacks. By breaking this coupling, we can improve both the
usability of CPython and its safety.

The recursion limit exists to protect against runaway recursion, the
integrity of the virtual machine should not depend on it. Similarly,
recursion should not be limited by implementation details.

Specification

Two new exception classes will be added, StackOverflow and
RecursionOverflow, both of which will be sub-classes of RecursionError

StackOverflow exception

A StackOverflow exception will be raised whenever the interpreter or
builtin module code determines that the C stack is at or nearing a limit
of safety. StackOverflow is a sub-class of RecursionError, so any code
that handles RecursionError will handle StackOverflow

RecursionOverflow exception

A RecursionOverflow exception will be raised when a call to a Python
function causes the recursion limit to be exceeded. This is a slight
change from current behavior which raises a RecursionError.
RecursionOverflow is a sub-class of RecursionError, so any code that
handles RecursionError will continue to work as before.

Decoupling the Python stack from the C stack

In order to provide the above guarantees and ensure that any program
that worked previously continues to do so, the Python and C stack will
need to be separated. That is, calls to Python functions from Python
functions, should not consume space on the C stack. Calls to and from
builtin functions will continue to consume space on the C stack.

The size of the C stack will be implementation defined, and may vary
from machine to machine. It may even differ between threads. However,
there is an expectation that any code that could run with the recursion
limit set to the previous default value, will continue to run.

Many operations in Python perform some sort of call at the C level. Most
of these will continue to consume C stack, and will result in a
StackOverflow exception if uncontrolled recursion occurs.

Other Implementations

Other implementations are required to fail safely regardless of what
value the recursion limit is set to.

If the implementation couples the Python stack to the underlying VM or
hardware stack, then it should raise a RecursionOverflow exception when
the recursion limit is exceeded, but the underlying stack does not
overflow. If the underlying stack overflows, or is near to overflow,
then a StackOverflow exception should be raised.

C-API

A new function, Py_CheckStackDepth() will be added, and the behavior of
Py_EnterRecursiveCall() will be modified slightly.

Py_CheckStackDepth()

int Py_CheckStackDepth(const char *where) will return 0 if there is no
immediate danger of C stack overflow. It will return -1 and set an
exception, if the C stack is near to overflowing. The where parameter is
used in the exception message, in the same fashion as the where
parameter of Py_EnterRecursiveCall().

Py_EnterRecursiveCall()

Py_EnterRecursiveCall() will be modified to call Py_CheckStackDepth()
before performing its current function.

PyLeaveRecursiveCall()

Py_LeaveRecursiveCall() will remain unchanged.

Backwards Compatibility

This feature is fully backwards compatible at the Python level. Some
low-level tools, such as machine-code debuggers, will need to be
modified. For example, the gdb scripts for Python will need to be aware
that there may be more than one Python frame per C frame.

C code that uses the Py_EnterRecursiveCall(), PyLeaveRecursiveCall()
pair of functions will continue to work correctly. In addition,
Py_EnterRecursiveCall() may raise a StackOverflow exception.

New code should use the Py_CheckStackDepth() function, unless the code
wants to count as a Python function call with regard to the recursion
limit.

We recommend that "python-like" code, such as Cython-generated
functions, use Py_EnterRecursiveCall(), but other code use
Py_CheckStackDepth().

Security Implications

It will no longer be possible to crash the CPython virtual machine
through recursion.

Performance Impact

It is unlikely that the performance impact will be significant.

The additional logic required will probably have a very small negative
impact on performance. The improved locality of reference from reduced C
stack use should have some small positive impact.

It is hard to predict whether the overall effect will be positive or
negative, but it is quite likely that the net effect will be too small
to be measured.

Implementation

Monitoring C stack consumption

Gauging whether a C stack overflow is imminent is difficult. So we need
to be conservative. We need to determine a safe bounds for the stack,
which is not something possible in portable C code.

For major platforms, the platform specific API will be used to provide
an accurate stack bounds. However, for minor platforms some amount of
guessing may be required. While this might sound bad, it is no worse
than the current situation, where we guess that the size of the C stack
is at least 1000 times the stack space required for the chain of calls
from _PyEval_EvalFrameDefault to _PyEval_EvalFrameDefault.

This means that in some cases the amount of recursion possible may be
reduced. In general, however, the amount of recursion possible should be
increased, as many calls will use no C stack.

Our general approach to determining a limit for the C stack is to get an
address within the current C frame, as early as possible in the call
chain. The limit can then be guessed by adding some constant to that.

Making Python-to-Python calls without consuming the C stack

Calls in the interpreter are handled by the CALL_FUNCTION,
CALL_FUNCTION_KW, CALL_FUNCTION_EX and CALL_METHOD instructions. The
code for those instructions will be modified so that when a Python
function or method is called, instead of making a call in C, the
interpreter will setup the callee's frame and continue interpretation as
normal.

The RETURN_VALUE instruction will perform the reverse operation, except
when the current frame is the entry frame of the interpreter when it
will return as normal.

Rejected Ideas

None, as yet.

Open Issues

None, as yet.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 739 Title: Static description file for build details of Python
installations Author: Filipe Laíns<EMAIL_ADDRESS>PEP-Delegate: Paul
Moore<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-739-static-description-file-for-build-details-of-python-installations/44572
Status: Draft Type: Standards Track Topic: Packaging Created:
19-Dec-2023 Python-Version: 3.14

Abstract

Introduce a standard format for a static description file with build
details of Python installations.

Rationale

When introspecting a Python installation, running code is often
undesirable or impossible. Having a static description file makes
various of Python build details available without having to run the
interpreter.

This is helpful for use-cases such as cross-compilation, Python
launchers, etc.

Scope

This PEP defines a format for the description file, and a standard
location for where to place it.

Location

When possible, Python installations should install the static
description file inside the standard library directory, with the name
build-details.json (Eg. /usr/lib/python3.14/build-details.json).

Important

Given that there may be technical challenges, Python implementations are
not required to provide the file if not feasable. In such scenarios,
they may choose to provide it in a different maner.

Attention

Notwithstanding the standard location specified here, it does not
prevent the file from additionally being provided in another location,
and with a different name. In fact, the PEP authors expect future PEPs
to define additional locations to install this file, for better
discoverability.

Format

The format specification is defined by the JSON Schema definition
provided below, which is rendered in an human-readable format here.

  ----------------- -----------------------------------------------------------------------------------------
  $schema           https://json-schema.org/draft/2020-12/schema

  $id               https://github.com/python/peps/blob/main/peps/pep-0739/python-build-info-v1.schema.json

  Title             Static description file for the build details of Python installations

  Type              object

  Additional        Not allowed
  properties
  ----------------- -----------------------------------------------------------------------------------------

schema_version

+-----------------+-----------------------------------------------------+
| Type            | string (constant — 1)                               |
+-----------------+-----------------------------------------------------+
| Description     | Schema version.                                     |
|                 |                                                     |
|                 | This is a constant value and MUST be 1, when using  |
|                 | the schema described here. Future iterations of     |
|                 | this schema MUST update this value.                 |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+

base_prefix

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | Base prefix of the Python installation.             |
|                 |                                                     |
|                 | Either an absolute path, or a relative path to      |
|                 | directory where this file is contained.             |
+-----------------+-----------------------------------------------------+
| Examples        | /usr, ../.., etc.                                   |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+

platform

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | System platform string.                             |
|                 |                                                     |
|                 | This field SHOULD be equivalent to                  |
|                 | sysconfig.get_platform().                           |
+-----------------+-----------------------------------------------------+
| Examples        | -   linux-x86_64                                    |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+

language

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Object containing details related to the Python     |
|                 | language specification.                             |
|                 |                                                     |
|                 | In addition to the required keys, implementations   |
|                 | may choose to include extra keys with               |
|                 | implementation-specific details.                    |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+
| Additional      | Not allowed                                         |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

language.version

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | String representation the Python language version — |
|                 | a version string consisting only of the major and   |
|                 | minor components.                                   |
|                 |                                                     |
|                 | This field SHOULD be equivalent to                  |
|                 | sysconfig.get_python_version().                     |
+-----------------+-----------------------------------------------------+
| Examples        | 3.14, etc.                                          |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+

implementation

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Object containing details related to Python         |
|                 | implementation.                                     |
|                 |                                                     |
|                 | This section SHOULD be equivalent to                |
|                 | sys.implementation. It follows specification        |
|                 | defined in PEP 421, meaning that on top of the      |
|                 | required keys, implementation-specific keys can     |
|                 | also exist, but must be prefixed with an            |
|                 | underscore.                                         |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+
| Additional      | Allowed                                             |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

implementation.name

  ----------------- -----------------------------------------------------
  Type              string

  Description       Lower-case name of the Python implementation.

  Examples          cpython, pypy, etc.

  Required          True
  ----------------- -----------------------------------------------------

implementation.version

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Object in the format of sys.version_info,           |
|                 | containing the implementation version.              |
+-----------------+-----------------------------------------------------+
| Examples        | -   {'major': 3, 'minor': 13                        |
|                 | , 'micro': 1, 'releaselevel': 'final', 'serial': 0} |
|                 | -   {'major': 7, 'minor': 3,                        |
|                 |  'micro': 16, 'releaselevel': 'final', 'serial': 0} |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+
| Additional      | Not allowed                                         |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

implementation.version.major

  ----------------- -----------------------------------------------------
  Type              number

  Required          True
  ----------------- -----------------------------------------------------

implementation.version.minor

  ----------------- -----------------------------------------------------
  Type              number

  Required          True
  ----------------- -----------------------------------------------------

implementation.version.micro

  ----------------- -----------------------------------------------------
  Type              number

  Required          True
  ----------------- -----------------------------------------------------

implementation.version.releaselevel

  ----------------- -----------------------------------------------------
  Type              string (enum — alpha, beta, candidate, final)

  Required          True
  ----------------- -----------------------------------------------------

implementation.version.serial

  ----------------- -----------------------------------------------------
  Type              number

  Required          True
  ----------------- -----------------------------------------------------

interpreter

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Object containing details Python interpreter.       |
|                 |                                                     |
|                 | This section MUST be present if the Python          |
|                 | installation provides an interpreter binary,        |
|                 | otherwise this section will be missing.             |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+
| Additional      | Not allowed                                         |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

interpreter.path

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | The path to the Python interprer. Either an         |
|                 | absolute path, or a relative path to the path       |
|                 | defined in the base_prefix key.                     |
+-----------------+-----------------------------------------------------+
| Examples        | -   /usr/bin/python                                 |
|                 | -   bin/python                                      |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+

abi

  ----------------- -----------------------------------------------------
  Type              object

  Description       Object containing details related to ABI.

  Required          False

  Additional        Not allowed
  properties
  ----------------- -----------------------------------------------------

abi.flags

+-----------------+-----------------------------------------------------+
| Type            | array                                               |
+-----------------+-----------------------------------------------------+
| Description     | Build configuration flags, used to calculate the    |
|                 | extension suffix.                                   |
|                 |                                                     |
|                 | The flags MUST be defined in the order they appear  |
|                 | on the extension suffix.                            |
+-----------------+-----------------------------------------------------+
| Examples        | ['t', 'd'], etc.                                    |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+

abi.extension_suffix

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | Suffix used for extensions built against the        |
|                 | current implementation version.                     |
|                 |                                                     |
|                 | This field MUST be present if the Python            |
|                 | implementation supports extensions, otherwise this  |
|                 | entry will be missing.                              |
+-----------------+-----------------------------------------------------+
| Examples        | -   .cpython-314-x86_64-linux-gnu.so                |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+

abi.stable_abi_suffix

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | Suffix used for extensions built against the stable |
|                 | ABI.                                                |
|                 |                                                     |
|                 | This field MUST be present if the Python            |
|                 | implementation has a stable ABI extension suffix,   |
|                 | otherwise this entry will be missing.               |
+-----------------+-----------------------------------------------------+
| Examples        | .abi3.so, etc.                                      |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+

suffixes

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Valid module suffixes grouped by type.              |
|                 |                                                     |
|                 | This section SHOULD be equivalent to the            |
|                 | importlib.machinery.*_SUFFIXES attributes, if the   |
|                 | implementation provides such suffixes. However, if  |
|                 | the Python implementation does not provide suffixes |
|                 | of the kind specified by any of the attributes, the |
|                 | equivalent sub-section is not required to be        |
|                 | present. Additionally, if a Python implementation   |
|                 | provides extension kinds other than the ones listed |
|                 | on importlib.machinery module, they MAY add a       |
|                 | sub-section for them.                               |
+-----------------+-----------------------------------------------------+
| Examples        | -   {'source': ['.py'],                             |
|                 |  'bytecode': ['.pyc'], 'optimized_bytecode': ['.pyc |
|                 | '], 'debug_bytecode': ['.pyc'], 'extensions': ['.cp |
|                 | ython-313-x86_64-linux-gnu.so', '.abi3.so', '.so']} |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+
| Additional      | Allowed                                             |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

libpython

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Object containing details related to the libpython  |
|                 | library.                                            |
|                 |                                                     |
|                 | This section MUST by present if Python installation |
|                 | provides a libpython library, otherwise this        |
|                 | section will be missing.                            |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+
| Additional      | Not allowed                                         |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

libpython.dynamic

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | The path to the dynamic libpython library. Either   |
|                 | an absolute path, or a relative path to the path    |
|                 | defined in the base_prefix key.                     |
|                 |                                                     |
|                 | This field MUST be present if the Python            |
|                 | installation provides a dynamic libpython library,  |
|                 | otherwise this entry will be missing.               |
+-----------------+-----------------------------------------------------+
| Examples        | -   /usr/lib/libpython3.14.so.1.0                   |
|                 | -   lib/libpython3.14.so.1.0                        |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+

libpython.static

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | The path to the static libpython library. Either an |
|                 | absolute path, or a relative path to the path       |
|                 | defined in the base_prefix key.                     |
|                 |                                                     |
|                 | This field MUST be present if the Python            |
|                 | installation provides a static libpython library,   |
|                 | otherwise this entry will be missing.               |
+-----------------+-----------------------------------------------------+
| Examples        | -   /usr/lib/pyth                                   |
|                 | on3.14/config-3.14-x86_64-linux-gnu/libpython3.14.a |
|                 | -   lib/pyth                                        |
|                 | on3.14/config-3.14-x86_64-linux-gnu/libpython3.14.a |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+

libpython.link_to_libpython

+-----------------+-----------------------------------------------------+
| Type            | boolean                                             |
+-----------------+-----------------------------------------------------+
| Description     | Should extensions built against a dynamic libpython |
|                 | link to it?                                         |
|                 |                                                     |
|                 | This field MUST be present if the Python            |
|                 | installation provides a dynamic libpython library,  |
|                 | otherwise this entry will be missing.               |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+

c_api

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Object containing details related to the Python C   |
|                 | API, if available.                                  |
|                 |                                                     |
|                 | This section MUST be present if the Python          |
|                 | implementation provides a C API, otherwise this     |
|                 | section will be missing.                            |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+
| Additional      | Not allowed                                         |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

c_api.headers

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | The path to the C API headers. Either an absolute   |
|                 | path, or a relative path to the path defined in the |
|                 | base_prefix key.                                    |
+-----------------+-----------------------------------------------------+
| Examples        | -   /usr/include/python3.14                         |
|                 | -   include/python3.14                              |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | True                                                |
+-----------------+-----------------------------------------------------+

c_api.pkgconfig_path

+-----------------+-----------------------------------------------------+
| Type            | string                                              |
+-----------------+-----------------------------------------------------+
| Description     | The path to the pkg-config definition files. Either |
|                 | an absolute path, or a relative path to the path    |
|                 | defined in the base_prefix key.                     |
|                 |                                                     |
|                 | This field MUST be present if the Python            |
|                 | implementation provides pkg-config definition files |
|                 | for the C API, otherwise this section will be       |
|                 | missing.                                            |
+-----------------+-----------------------------------------------------+
| Examples        | -   /usr/lib/pkgconfig                              |
|                 | -   lib/pkgconfig                                   |
|                 | -   etc.                                            |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+

arbitrary_data

+-----------------+-----------------------------------------------------+
| Type            | object                                              |
+-----------------+-----------------------------------------------------+
| Description     | Object containing extra arbitrary data.             |
|                 |                                                     |
|                 | This is meant to be used as an escape-hatch, to     |
|                 | include any relevant data that is not covered by    |
|                 | this specification. Implamentations may choose what |
|                 | data to provide in this section.                    |
+-----------------+-----------------------------------------------------+
| Required        | False                                               |
+-----------------+-----------------------------------------------------+
| Additional      | Allowed                                             |
| properties      |                                                     |
+-----------------+-----------------------------------------------------+

Example

pep-0739/example.json

JSON Schema

pep-0739/python-build-info-v1.schema.json

Rejected Ideas

Having a larger scope

One of the main requests in the discussion of this PEP was the inclusion
of other kind of information, such as the site-packages path. It is the
opinion of the PEP authors that information regarding the Python
environment should be provided by a separate file, creating the a clear
separation between the build details, which should be immutable across
any interpreter instance, and details that can change, such as
environment details.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 221 Title: Import As Author: Thomas Wouters<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
15-Aug-2000 Python-Version: 2.0 Post-History:

Introduction

This PEP describes the import as proposal for Python 2.0. This PEP
tracks the status and ownership of this feature. It contains a
description of the feature and outlines changes necessary to support the
feature. The CVS revision history of this file contains the definitive
historical record.

Rationale

This PEP proposes an extension of Python syntax regarding the import and
from <module> import statements. These statements load in a module, and
either bind that module to a local name, or binds objects from that
module to a local name. However, it is sometimes desirable to bind those
objects to a different name, for instance to avoid name clashes. This
can currently be achieved using the following idiom:

    import os
    real_os = os
    del os

And similarly for the from ... import statement:

    from os import fdopen, exit, stat
    os_fdopen = fdopen
    os_stat = stat
    del fdopen, stat

The proposed syntax change would add an optional as clause to both these
statements, as follows:

    import os as real_os
    from os import fdopen as os_fdopen, exit, stat as os_stat

The as name is not intended to be a keyword, and some trickery has to be
used to convince the CPython parser it isn't one. For more advanced
parsers/tokenizers, however, this should not be a problem.

A slightly special case exists for importing sub-modules. The statement
:

    import os.path

stores the module os locally as os, so that the imported submodule path
is accessible as os.path. As a result, :

    import os.path as p

stores os.path, not os, in p. This makes it effectively the same as :

    from os import path as p

Implementation details

This PEP has been accepted, and the suggested code change has been
checked in. The patch can still be found in the SourceForge patch
manager[1]. Currently, a NAME field is used in the grammar rather than a
bare string, to avoid the keyword issue. It introduces a new bytecode,
IMPORT_STAR, which performs the from module import * behaviour, and
changes the behaviour of the IMPORT_FROM bytecode so that it loads the
requested name (which is always a single name) onto the stack, to be
subsequently stored by a STORE opcode. As a result, all names explicitly
imported now follow the global directives.

The special case of from module import * remains a special case, in that
it cannot accommodate an as clause, and that no STORE opcodes are
generated; the objects imported are loaded directly into the local
namespace. This also means that names imported in this fashion are
always local, and do not follow the global directive.

An additional change to this syntax has also been suggested, to
generalize the expression given after the as clause. Rather than a
single name, it could be allowed to be any expression that yields a
valid l-value; anything that can be assigned to. The change to
accommodate this is minimal, as the patch[2] proves, and the resulting
generalization allows a number of new constructs that run completely
parallel with other Python assignment constructs. However, this idea has
been rejected by Guido, as "hypergeneralization".

Copyright

This document has been placed in the Public Domain.

References

[1] https://hg.python.org/cpython/rev/18385172fac0

[2] http://sourceforge.net/patch/?func=detailpatch&patch_id=101234&group_id=5470