text:
PEP: 504 Title: Using the System RNG by default Version: $Revision$
Last-Modified: $Date$ Author: Alyssa Coghlan<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track Content-Type: text/x-rst
Created: 15-Sep-2015 Python-Version: 3.6 Post-History: 15-Sep-2015

Abstract

Python currently defaults to using the deterministic Mersenne Twister
random number generator for the module level APIs in the random module,
requiring users to know that when they're performing "security
sensitive" work, they should instead switch to using the
cryptographically secure os.urandom or random.SystemRandom interfaces or
a third party library like cryptography.

Unfortunately, this approach has resulted in a situation where
developers that aren't aware that they're doing security sensitive work
use the default module level APIs, and thus expose their users to
unnecessary risks.

This isn't an acute problem, but it is a chronic one, and the often long
delays between the introduction of security flaws and their exploitation
means that it is difficult for developers to naturally learn from
experience.

In order to provide an eventually pervasive solution to the problem,
this PEP proposes that Python switch to using the system random number
generator by default in Python 3.6, and require developers to opt-in to
using the deterministic random number generator process wide either by
using a new random.ensure_repeatable() API, or by explicitly creating
their own random.Random() instance.

To minimise the impact on existing code, module level APIs that require
determinism will implicitly switch to the deterministic PRNG.

PEP Withdrawal

During discussion of this PEP, Steven D'Aprano proposed the simpler
alternative of offering a standardised secrets module that provides "one
obvious way" to handle security sensitive tasks like generating default
passwords and other tokens.

Steven's proposal has the desired effect of aligning the easy way to
generate such tokens and the right way to generate them, without
introducing any compatibility risks for the existing random module API,
so this PEP has been withdrawn in favour of further work on refining
Steven's proposal as PEP 506.

Proposal

Currently, it is never correct to use the module level functions in the
random module for security sensitive applications. This PEP proposes to
change that admonition in Python 3.6+ to instead be that it is not
correct to use the module level functions in the random module for
security sensitive applications if random.ensure_repeatable() is ever
called (directly or indirectly) in that process.

To achieve this, rather than being bound methods of a random.Random
instance as they are today, the module level callables in random would
change to be functions that delegate to the corresponding method of the
existing random._inst module attribute.

By default, this attribute will be bound to a random.SystemRandom
instance.

A new random.ensure_repeatable() API will then rebind the random._inst
attribute to a system.Random instance, restoring the same module level
API behaviour as existed in previous Python versions (aside from the
additional level of indirection):

    def ensure_repeatable():
        """Switch to using random.Random() for the module level APIs

        This switches the default RNG instance from the cryptographically
        secure random.SystemRandom() to the deterministic random.Random(),
        enabling the seed(), getstate() and setstate() operations. This means
        a particular random scenario can be replayed later by providing the
        same seed value or restoring a previously saved state.

        NOTE: Libraries implementing security sensitive operations should
        always explicitly use random.SystemRandom() or os.urandom in order to
        correctly handle applications that call this function.
        """
        if not isinstance(_inst, Random):
            _inst = random.Random()

To minimise the impact on existing code, calling any of the following
module level functions will implicitly call random.ensure_repeatable():

-   random.seed
-   random.getstate
-   random.setstate

There are no changes proposed to the random.Random or
random.SystemRandom class APIs - applications that explicitly
instantiate their own random number generators will be entirely
unaffected by this proposal.

Warning on implicit opt-in

In Python 3.6, implicitly opting in to the use of the deterministic PRNG
will emit a deprecation warning using the following check:

    if not isinstance(_inst, Random):
        warnings.warn(DeprecationWarning,
                      "Implicitly ensuring repeatability. "
                      "See help(random.ensure_repeatable) for details")
        ensure_repeatable()

The specific wording of the warning should have a suitable answer added
to Stack Overflow as was done for the custom error message that was
added for missing parentheses in a call to print[1].

In the first Python 3 release after Python 2.7 switches to security fix
only mode, the deprecation warning will be upgraded to a RuntimeWarning
so it is visible by default.

This PEP does not propose ever removing the ability to ensure the
default RNG used process wide is a deterministic PRNG that will produce
the same series of outputs given a specific seed. That capability is
widely used in modelling and simulation scenarios, and requiring that
ensure_repeatable() be called either directly or indirectly is a
sufficient enhancement to address the cases where the module level
random API is used for security sensitive tasks in web applications
without due consideration for the potential security implications of
using a deterministic PRNG.

Performance impact

Due to the large performance difference between random.Random and
random.SystemRandom, applications ported to Python 3.6 will encounter a
significant performance regression in cases where:

-   the application is using the module level random API
-   cryptographic quality randomness isn't needed
-   the application doesn't already implicitly opt back in to the
    deterministic PRNG by calling random.seed, random.getstate, or
    random.setstate
-   the application isn't updated to explicitly call
    random.ensure_repeatable

This would be noted in the Porting section of the Python 3.6 What's New
guide, with the recommendation to include the following code in the
__main__ module of affected applications:

    if hasattr(random, "ensure_repeatable"):
        random.ensure_repeatable()

Applications that do need cryptographic quality randomness should be
using the system random number generator regardless of speed
considerations, so in those cases the change proposed in this PEP will
fix a previously latent security defect.

Documentation changes

The random module documentation would be updated to move the
documentation of the seed, getstate and setstate interfaces later in the
module, along with the documentation of the new ensure_repeatable
function and the associated security warning.

That section of the module documentation would also gain a discussion of
the respective use cases for the deterministic PRNG enabled by
ensure_repeatable (games, modelling & simulation, software testing) and
the system RNG that is used by default (cryptography, security token
generation). This discussion will also recommend the use of third party
security libraries for the latter task.

Rationale

Writing secure software under deadline and budget pressures is a hard
problem. This is reflected in regular notifications of data breaches
involving personally identifiable information[2], as well as with
failures to take security considerations into account when new systems,
like motor vehicles [3], are connected to the internet. It's also the
case that a lot of the programming advice readily available on the
internet [#search] simply doesn't take the mathematical arcana of
computer security into account. Compounding these issues is the fact
that defenders have to cover all of their potential vulnerabilities, as
a single mistake can make it possible to subvert other defences[4].

One of the factors that contributes to making this last aspect
particularly difficult is APIs where using them inappropriately creates
a silent security failure - one where the only way to find out that what
you're doing is incorrect is for someone reviewing your code to say
"that's a potential security problem", or for a system you're
responsible for to be compromised through such an oversight (and you're
not only still responsible for that system when it is compromised, but
your intrusion detection and auditing mechanisms are good enough for you
to be able to figure out after the event how the compromise took place).

This kind of situation is a significant contributor to "security
fatigue", where developers (often rightly[5]) feel that security
engineers spend all their time saying "don't do that the easy way, it
creates a security vulnerability".

As the designers of one of the world's most popular languages[6], we can
help reduce that problem by making the easy way the right way (or at
least the "not wrong" way) in more circumstances, so developers and
security engineers can spend more time worrying about mitigating
actually interesting threats, and less time fighting with default
language behaviours.

Discussion

Why "ensure_repeatable" over "ensure_deterministic"?

This is a case where the meaning of a word as specialist jargon
conflicts with the typical meaning of the word, even though it's
technically the same.

From a technical perspective, a "deterministic RNG" means that given
knowledge of the algorithm and the current state, you can reliably
compute arbitrary future states.

The problem is that "deterministic" on its own doesn't convey those
qualifiers, so it's likely to instead be interpreted as "predictable" or
"not random" by folks that are familiar with the conventional meaning,
but aren't familiar with the additional qualifiers on the technical
meaning.

A second problem with "deterministic" as a description for the
traditional RNG is that it doesn't really tell you what you can do with
the traditional RNG that you can't do with the system one.

"ensure_repeatable" aims to address both of those problems, as its
common meaning accurately describes the main reason for preferring the
deterministic PRNG over the system RNG: ensuring you can repeat the same
series of outputs by providing the same seed value, or by restoring a
previously saved PRNG state.

Only changing the default for Python 3.6+

Some other recent security changes, such as upgrading the capabilities
of the ssl module and switching to properly verifying HTTPS certificates
by default, have been considered critical enough to justify backporting
the change to all currently supported versions of Python.

The difference in this case is one of degree - the additional benefits
from rolling out this particular change a couple of years earlier than
will otherwise be the case aren't sufficient to justify either the
additional effort or the stability risks involved in making such an
intrusive change in a maintenance release.

Keeping the module level functions

In additional to general backwards compatibility considerations, Python
is widely used for educational purposes, and we specifically don't want
to invalidate the wide array of educational material that assumes the
availability of the current random module API. Accordingly, this
proposal ensures that most of the public API can continue to be used not
only without modification, but without generating any new warnings.

Warning when implicitly opting in to the deterministic RNG

It's necessary to implicitly opt in to the deterministic PRNG as Python
is widely used for modelling and simulation purposes where this is the
right thing to do, and in many cases, these software models won't have a
dedicated maintenance team tasked with ensuring they keep working on the
latest versions of Python.

Unfortunately, explicitly calling random.seed with data from os.urandom
is also a mistake that appears in a number of the flawed "how to
generate a security token in Python" guides readily available online.

Using first DeprecationWarning, and then eventually a RuntimeWarning, to
advise against implicitly switching to the deterministic PRNG aims to
nudge future users that need a cryptographically secure RNG away from
calling random.seed() and those that genuinely need a deterministic
generator towards explicitly calling random.ensure_repeatable().

Avoiding the introduction of a userspace CSPRNG

The original discussion of this proposal on python-ideas[7] suggested
introducing a cryptographically secure pseudo-random number generator
and using that by default, rather than defaulting to the relatively slow
system random number generator.

The problem[8] with this approach is that it introduces an additional
point of failure in security sensitive situations, for the sake of
applications where the random number generation may not even be on a
critical performance path.

Applications that do need cryptographic quality randomness should be
using the system random number generator regardless of speed
considerations, so in those cases.

Isn't the deterministic PRNG "secure enough"?

In a word, "No" - that's why there's a warning in the module
documentation that says not to use it for security sensitive purposes.
While we're not currently aware of any studies of Python's random number
generator specifically, studies of PHP's random number generator[9] have
demonstrated the ability to use weaknesses in that subsystem to
facilitate a practical attack on password recovery tokens in popular PHP
web applications.

However, one of the rules of secure software development is that
"attacks only get better, never worse", so it may be that by the time
Python 3.6 is released we will actually see a practical attack on
Python's deterministic PRNG publicly documented.

Security fatigue in the Python ecosystem

Over the past few years, the computing industry as a whole has been
making a concerted effort to upgrade the shared network infrastructure
we all depend on to a "secure by default" stance. As one of the most
widely used programming languages for network service development
(including the OpenStack Infrastructure-as-a-Service platform) and for
systems administration on Linux systems in general, a fair share of that
burden has fallen on the Python ecosystem, which is understandably
frustrating for Pythonistas using Python in other contexts where these
issues aren't of as great a concern.

This consideration is one of the primary factors driving the substantial
backwards compatibility improvements in this proposal relative to the
initial draft concept posted to python-ideas[10].

Acknowledgements

-   Theo de Raadt, for making the suggestion to Guido van Rossum that we
    seriously consider defaulting to a cryptographically secure random
    number generator
-   Serhiy Storchaka, Terry Reedy, Petr Viktorin, and anyone else in the
    python-ideas threads that suggested the approach of transparently
    switching to the random.Random implementation when any of the
    functions that only make sense for a deterministic RNG are called
-   Nathaniel Smith for providing the reference on practical attacks
    against PHP's random number generator when used to generate password
    reset tokens
-   Donald Stufft for pursuing additional discussions with network
    security experts that suggested the introduction of a userspace
    CSPRNG would mean additional complexity for insufficient gain
    relative to just using the system RNG directly
-   Paul Moore for eloquently making the case for the current level of
    security fatigue in the Python ecosystem

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Stack Overflow answer for missing parentheses in call to print
(http://stackoverflow.com/questions/25445439/what-does-syntaxerror-missing-parentheses-in-call-to-print-mean-in-python/25445440#25445440)

[2] Visualization of data breaches involving more than 30k records
(each)
(http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/)

[3] Remote UConnect hack for Jeep Cherokee
(http://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/)

[4] Bypassing bcrypt through an insecure data cache
(http://arstechnica.com/security/2015/09/once-seen-as-bulletproof-11-million-ashley-madison-passwords-already-cracked/)

[5] OWASP Top Ten Web Security Issues for 2013
(https://www.owasp.org/index.php/OWASP_Top_Ten_Project#tab=OWASP_Top_10_for_2013)

[6] IEEE Spectrum 2015 Top Ten Programming Languages
(http://spectrum.ieee.org/computing/software/the-2015-top-ten-programming-languages)

[7] python-ideas thread discussing using a userspace CSPRNG
(https://mail.python.org/pipermail/python-ideas/2015-September/035886.html)

[8] Safely generating random numbers
(http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/)

[9] PRNG based attack against password reset tokens in PHP applications
(https://media.blackhat.com/bh-us-12/Briefings/Argyros/BH_US_12_Argyros_PRNG_WP.pdf)

[10] Initial draft concept that eventually became this PEP
(https://mail.python.org/pipermail/python-ideas/2015-September/036095.html)


text:
PEP: 527 Title: Removing Un(der)used file types/extensions on PyPI
Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft
<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards
Track Topic: Packaging Content-Type: text/x-rst Created: 23-Aug-2016
Post-History: 23-Aug-2016 Resolution:
https://mail.python.org/pipermail/distutils-sig/2016-September/029624.html

Abstract

This PEP recommends deprecating, and ultimately removing, support for
uploading certain unused or under used file types and extensions to
PyPI. In particular it recommends disallowing further uploads of any
files of the types bdist_dumb, bdist_rpm, bdist_dmg, bdist_msi, and
bdist_wininst, leaving PyPI to only accept new uploads of the sdist,
bdist_wheel, and bdist_egg file types.

In addition, this PEP proposes removing support for new uploads of
sdists using the .tar, .tar.bz2, .tar.xz, .tar.Z, .tgz, .tbz, and any
other extension besides .tar.gz and .zip.

Finally, this PEP also proposes limiting the number of allowed sdist
uploads for each individual release of a project on PyPI to one instead
of one for each allowed extension.

Rationale

File Formats

Currently PyPI supports the following file types:

-   sdist
-   bdist_wheel
-   bdist_egg
-   bdist_wininst
-   bdist_msi
-   bdist_dmg
-   bdist_rpm
-   bdist_dumb

However, these different types of files have varying amounts of
usefulness or general use in the ecosystem. Continuing to support them
adds a maintenance burden on PyPI as well as tool authors and incurs a
cost in both bandwidth and disk space not only on PyPI itself, but also
on any mirrors of PyPI.

Python packaging is a multi-level ecosystem where PyPI is primarily
suited and used to distribute virtual environment compatible packages
directly from their respective project owners. These packages are then
consumed either directly by end-users, or by downstream distributors
that take these packages and turn them into their respective system
level packages (such as RPM, deb, MSI, etc).

While PyPI itself only directly works with these Python specific but
platform agnostic packages, we encourage community-driven and commercial
conversions of these packages to downstream formats for particular
target environments, like:

-   The conda cross-platform data analysis ecosystem (conda-forge)
-   The deb based Linux ecosystem (Debian, Ubuntu, etc)
-   The RPM based Linux ecosystem (Fedora, openSuSE, Mageia, etc)
-   The homebrew, MacPorts and fink ecosystems for Mac OS X
-   The Windows Package Management ecosystem (NuGet, Chocolatey, etc)
-   3rd party creation of Windows MSIs and installers (e.g. Christoph
    Gohlke's work at http://www.lfd.uci.edu/~gohlke/pythonlibs/ )
-   other commercial redistribution formats (ActiveState's PyPM,
    Enthought Canopy, etc)
-   other open source community redistribution formats (Nix, Gentoo,
    Arch, *BSD, etc)

It is the belief of this PEP that the entire ecosystem is best supported
by keeping PyPI focused on the platform agnostic formats, where the
limited amount of time by volunteers can be best used instead of
spreading the available time out amongst several platforms. Further
more, this PEP believes that the people best positioned to provide well
integrated packages for a particular platform are people focused on that
platform, and not across all possible platforms.

bdist_dumb

As it's name implies, bdist_dumb is not a very complex format, however
it is so simple as to be worthless for actual usage.

For instance, if you're using something like pyenv on macOS and you're
building a library using Python 3.5, then bdist_dumb will produce a
.tar.gz file named something like
exampleproject-1.0.macosx-10.11-x86_64.tar.gz. Right off the bat this
file name is somewhat difficult to differentiate from an sdist since
they both use the same file extension (and with the legacy pre PEP 440
versions, 1.0-macosx-10.11-x86_64 is a valid, although quite silly,
version number). However, once you open up the created .tar.gz, you'd
find that there is no metadata inside that could be used for things like
dependency discovery and in fact, it is quite simply a tarball
containing hardcoded paths to wherever files would have been installed
on the computer creating the bdist_dumb. Going back to our pyenv on
macOS example, this means that if I created it, it would contain files
like:

Users/dstufft/.pyenv/versions/3.5.2/lib/python3.5/site-packages/example.py

bdist_rpm

The bdist_rpm format on PyPI allows people to upload .rpm files for end
users to manually download by hand and then manually install by hand.
However, the common usage of rpm is with a specially designed repository
that allows automatic installation of dependencies, upgrades, etc which
PyPI does not provide. Thus, it is a type of file that is barely being
used on PyPI with only ~460 files of this type having been uploaded to
PyPI (out a total of 662,544).

In addition, services like COPR provide a better supported mechanism for
publishing and using RPM files than we're ever likely to get on PyPI.

bdist_dmg, bdist_msi, and bdist_wininst

The bdist_dmg, bdist_msi, and bdist_winist formats are similar in that
they are an OS specific installer that will only install a library into
an environment and are not designed for real user facing installs of
applications (which would require things like bundling a Python
interpreter and the like).

Out of these three, the usage for bdist_dmg and bdist_msi is very low,
with only ~500 bdist_msi files and ~50 bdist_dmg files having been
uploaded to PyPI. The bdist_wininst format has more use, with ~14,000
files having ever been uploaded to PyPI.

It's quite easy to look at the low usage of bdist_dmg and bdist_msi and
conclude that removing them will be fairly low impact, however
bdist_wininst has several orders of magnitude more usage. This is
somewhat misleading though, because although it has more people
uploading those files the actual usage of those uploaded files is fairly
low. Taking a look at the previous 30 days, we can see that 90% of all
downloads of bdist_winist files from PyPI were generated by the
mirroring infrastructure and 7% of them were generated by setuptools
(which can currently be better covered by bdist_egg files).

Given the small number of files uploaded for bdist_dmg and bdist_msi and
that bdist_wininst is largely existing to either consume bandwidth and
disk space via the mirroring infrastructure or could be trivially
replaced with bdist_egg, this PEP proposes to include these three
formats in the list of those to be disallowed.

File Extensions

Currently sdist supports a wide variety of file extensions like .tar.gz,
.tar, .tar.bz2, .tar.xz, .zip, .tar.Z, .tgz, and .tbz. However, of those
the only extensions which get anything more than negligible usage is
.tar.gz with 444,338 sdists currently, .zip with 58,774 sdists
currently, and .tar.bz2 with 3,265 sdists currently.

Having multiple formats accepted requires tooling both within PyPI and
outside of PyPI to handle all of the various extensions that might be
used (even if nobody is currently using them). This doesn't only affect
PyPI, but ripples out throughout the ecosystem. In addition, the
different formats all have different requirements for what optional C
libraries Python was linked against and different requirements for what
versions of Python they support. In addition, multiple formats also
create a weird situation where there may be two sdist files for a
particular project/release with subtly different content.

It's easy to advocate that anything outside of .tar.gz, .zip, and
.tar.bz2 should be disallowed. Outside of a tiny handful, nobody has
actively been uploading these other types of files in the ~15 years of
PyPI's existence so they've obviously not been particularly useful. In
addition, while .tar.xz is theoretically a nicer format than the other
.tar.* formats due to the better compression ratio achieved by LZMA, it
is only available in Python 3.3+ and has an optional dependency on the
lzma C library.

Looking at the three extensions we do have in current use, it's also
fairly easy to conclude that .tar.bz2 can be disallowed as well. It has
a fairly small number of files ever uploaded with it and it requires an
additional optional C library to handle the bzip2 compression.

Finally we get down to .tar.gz and .zip. Looking at the pure numbers for
these two, we can see that .tar.gz is by far the most uploaded format,
with 444,338 total uploaded compared to .zip's 58,774 and on POSIX
operating systems .tar.gz is also the default produced by all currently
released versions of Python and setuptools. In addition, these two file
types both use the same C library (zlib) which is also required for
bdist_wheel and bdist_egg. The two wrinkles with deciding between
.tar.gz and .zip is that while on POSIX operating systems .tar.gz is the
default, on Windows .zip is the default and the bdist_wheel format also
uses zip.

Instead of trying to standardize on either .tar.gz or .zip, this PEP
proposes that we allow either .tar.gz or .zip for sdists.

Limiting number of sdists per release

A sdist on PyPI should be a single source of truth for a particular
release of software. However, currently PyPI allows you to upload one
sdist for each of the sdist file extensions it allows. Currently this
allows something like 10 different sdists for a project, but even with
this PEP it allows two different sources of truth for a single version.
Having multiple sdists oftentimes can account for strange bugs that only
expose themselves based on which sdist that the person used.

To resolve this, this PEP proposes to allow one, and only one, sdist per
release of a project.

Removal Process

This PEP does NOT propose removing any existing files from PyPI, only
disallowing new ones from being uploaded. This restriction will be
phased in on a per-project basis to allow projects to adjust to the new
restrictions where applicable.

First, any existing projects will be flagged to allow legacy file types
to be uploaded, and any project without that flag (i.e. new projects)
will not be able to upload anything but sdist with a .tar.gz or .zip
extension, bdist_wheel, and bdist_egg. Then, any existing projects that
have never uploaded a file that requires the legacy file type flag will
have that flag removed, also making them fall under the new
restrictions. Finally, an email will be generated to the maintainers of
all projects still given the legacy flag, which will inform them of the
upcoming new restrictions on uploads and tell them that these
restrictions will be applied to future uploads to their projects
starting in 1 month. Finally, after 1 month all projects will have the
legacy file type flag removed, and support for uploading these types of
files will cease to exist on PyPI.

This plan should provide minimal disruption since it does not remove any
existing files, and the types of files it does prevent from being
uploaded are either not particularly useful (or used) types of files or
they can continue to upload a similar type of file with a slight change
to their process.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

license: Public Domain


text:
PEP: 325 Title: Resource-Release Support for Generators Version:
$Revision$ Last-Modified: $Date$ Author: Samuele Pedroni
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 25-Aug-2003 Python-Version: 2.4
Post-History:

Abstract

Generators allow for natural coding and abstraction of traversal over
data. Currently if external resources needing proper timely release are
involved, generators are unfortunately not adequate. The typical idiom
for timely release is not supported, a yield statement is not allowed in
the try clause of a try-finally statement inside a generator. The
finally clause execution can be neither guaranteed nor enforced.

This PEP proposes that the built-in generator type implement a close
method and destruction semantics, such that the restriction on yield
placement can be lifted, expanding the applicability of generators.

Pronouncement

Rejected in favor of PEP 342 which includes substantially all of the
requested behavior in a more refined form.

Rationale

Python generators allow for natural coding of many data traversal
scenarios. Their instantiation produces iterators, i.e. first-class
objects abstracting traversal (with all the advantages of first-
classness). In this respect they match in power and offer some
advantages over the approach using iterator methods taking a
(smalltalkish) block. On the other hand, given current limitations (no
yield allowed in a try clause of a try-finally inside a generator) the
latter approach seems better suited to encapsulating not only traversal
but also exception handling and proper resource acquisition and release.

Let's consider an example (for simplicity, files in read-mode are used):

    def all_lines(index_path):
        for path in file(index_path, "r"):
            for line in file(path.strip(), "r"):
                yield line

this is short and to the point, but the try-finally for timely closing
of the files cannot be added. (While instead of a path, a file, whose
closing then would be responsibility of the caller, could be passed in
as argument, the same is not applicable for the files opened depending
on the contents of the index).

If we want timely release, we have to sacrifice the simplicity and
directness of the generator-only approach: (e.g.) :

    class AllLines:

         def __init__(self, index_path):
             self.index_path = index_path
             self.index = None
             self.document = None

         def __iter__(self):
             self.index = file(self.index_path, "r")
             for path in self.index:
                 self.document = file(path.strip(), "r")
             for line in self.document:
                 yield line
             self.document.close()
             self.document = None

         def close(self):
             if self.index:
                 self.index.close()
             if self.document:
                 self.document.close()

to be used as:

    all_lines = AllLines("index.txt")
    try:
        for line in all_lines:
            ...
    finally:
        all_lines.close()

The more convoluted solution implementing timely release, seems to offer
a precious hint. What we have done is encapsulate our traversal in an
object (iterator) with a close method.

This PEP proposes that generators should grow such a close method with
such semantics that the example could be rewritten as:

    # Today this is not valid Python: yield is not allowed between
    # try and finally, and generator type instances support no
    # close method.

    def all_lines(index_path):
        index = file(index_path, "r")
        try:
            for path in index:
                document = file(path.strip(), "r")
            try:
                for line in document:
                    yield line
            finally:
                document.close()
        finally:
            index.close()

    all = all_lines("index.txt")
    try:
        for line in all:
            ...
    finally:
        all.close() # close on generator

Currently PEP 255 disallows yield inside a try clause of a try-finally
statement, because the execution of the finally clause cannot be
guaranteed as required by try-finally semantics.

The semantics of the proposed close method should be such that while the
finally clause execution still cannot be guaranteed, it can be enforced
when required. Specifically, the close method behavior should trigger
the execution of the finally clauses inside the generator, either by
forcing a return in the generator frame or by throwing an exception in
it. In situations requiring timely resource release, close could then be
explicitly invoked.

The semantics of generator destruction on the other hand should be
extended in order to implement a best-effort policy for the general
case. Specifically, destruction should invoke close(). The best-effort
limitation comes from the fact that the destructor's execution is not
guaranteed in the first place.

This seems to be a reasonable compromise, the resulting global behavior
being similar to that of files and closing.

Possible Semantics

The built-in generator type should have a close method implemented,
which can then be invoked as:

    gen.close()

where gen is an instance of the built-in generator type. Generator
destruction should also invoke close method behavior.

If a generator is already terminated, close should be a no-op.

Otherwise, there are two alternative solutions, Return or Exception
Semantics:

A - Return Semantics: The generator should be resumed, generator
execution should continue as if the instruction at the re-entry point is
a return. Consequently, finally clauses surrounding the re-entry point
would be executed, in the case of a then allowed try-yield-finally
pattern.

Issues: is it important to be able to distinguish forced termination by
close, normal termination, exception propagation from generator or
generator-called code? In the normal case it seems not, finally clauses
should be there to work the same in all these cases, still this
semantics could make such a distinction hard.

Except-clauses, like by a normal return, are not executed, such clauses
in legacy generators expect to be executed for exceptions raised by the
generator or by code called from it. Not executing them in the close
case seems correct.

B - Exception Semantics: The generator should be resumed and execution
should continue as if a special-purpose exception (e.g. CloseGenerator)
has been raised at re-entry point. Close implementation should consume
and not propagate further this exception.

Issues: should StopIteration be reused for this purpose? Probably not.
We would like close to be a harmless operation for legacy generators,
which could contain code catching StopIteration to deal with other
generators/iterators.

In general, with exception semantics, it is unclear what to do if the
generator does not terminate or we do not receive the special exception
propagated back. Other different exceptions should probably be
propagated, but consider this possible legacy generator code:

    try:
        ...
        yield ...
        ...
    except: # or except Exception:, etc
        raise Exception("boom")

If close is invoked with the generator suspended after the yield, the
except clause would catch our special purpose exception, so we would get
a different exception propagated back, which in this case ought to be
reasonably consumed and ignored but in general should be propagated, but
separating these scenarios seems hard.

The exception approach has the advantage to let the generator
distinguish between termination cases and have more control. On the
other hand, clear-cut semantics seem harder to define.

Remarks

If this proposal is accepted, it should become common practice to
document whether a generator acquires resources, so that its close
method ought to be called. If a generator is no longer used, calling
close should be harmless.

On the other hand, in the typical scenario the code that instantiated
the generator should call close if required by it. Generic code dealing
with iterators/generators instantiated elsewhere should typically not be
littered with close calls.

The rare case of code that has acquired ownership of and need to
properly deal with all of iterators, generators and generators acquiring
resources that need timely release, is easily solved:

    if hasattr(iterator, 'close'):
        iterator.close()

Open Issues

Definitive semantics ought to be chosen. Currently Guido favors
Exception Semantics. If the generator yields a value instead of
terminating, or propagating back the special exception, a special
exception should be raised again on the generator side.

It is still unclear whether spuriously converted special exceptions (as
discussed in Possible Semantics) are a problem and what to do about
them.

Implementation issues should be explored.

Alternative Ideas

The idea that the yield placement limitation should be removed and that
generator destruction should trigger execution of finally clauses has
been proposed more than once. Alone it cannot guarantee that timely
release of resources acquired by a generator can be enforced.

PEP 288 proposes a more general solution, allowing custom exception
passing to generators. The proposal in this PEP addresses more directly
the problem of resource release. Were PEP 288 implemented, Exceptions
Semantics for close could be layered on top of it, on the other hand PEP
288 should make a separate case for the more general functionality.

Copyright

This document has been placed in the public domain.


text:
PEP: 360 Title: Externally Maintained Packages Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon<EMAIL_ADDRESS>Status:
Final Type: Process Content-Type: text/x-rst Created: 30-May-2006
Post-History:

Warning

No new modules are to be added to this PEP. It has been deemed dangerous
to codify external maintenance of any code checked into Python's code
repository. Code contributors should expect Python's development
methodology to be used for any and all code checked into Python's code
repository.

Abstract

There are many great pieces of Python software developed outside of the
Python standard library (a.k.a., the "stdlib"). Sometimes it makes sense
to incorporate these externally maintained packages into the stdlib in
order to fill a gap in the tools provided by Python.

But by having the packages maintained externally it means Python's
developers do not have direct control over the packages' evolution and
maintenance. Some package developers prefer to have bug reports and
patches go through them first instead of being directly applied to
Python's repository.

This PEP is meant to record details of packages in the stdlib that are
maintained outside of Python's repository. Specifically, it is meant to
keep track of any specific maintenance needs for each package. It should
be mentioned that changes needed in order to fix bugs and keep the code
running on all of Python's supported platforms will be done directly in
Python's repository without worrying about going through the contact
developer. This is so that Python itself is not held up by a single bug
and allows the whole process to scale as needed.

It also is meant to allow people to know which version of a package is
released with which version of Python.

Externally Maintained Packages

The section title is the name of the package as it is known outside of
the Python standard library. The "standard library name" is what the
package is named within Python. The "contact person" is the Python
developer in charge of maintaining the package. The "synchronisation
history" lists what external version of the package was included in each
version of Python (if different from the previous Python release).

ElementTree

Web site

    http://effbot.org/zone/element-index.htm

Standard library name

    xml.etree

Contact person

    Fredrik Lundh

Fredrik has ceded ElementTree maintenance to the core Python development
team[1].

Expat XML parser

Web site

    http://www.libexpat.org/

Standard library name

    N/A (this refers to the parser itself, and not the Python bindings)

Contact person

    None

Optik

Web site

    http://optik.sourceforge.net/

Standard library name

    optparse

Contact person

    Greg Ward

External development seems to have ceased. For new applications,
optparse itself has been largely superseded by argparse.

wsgiref

Web site

    None

Standard library name

    wsgiref

Contact Person

    Phillip J. Eby

This module is maintained in the standard library, but significant bug
reports and patches should pass through the Web-SIG mailing list [2] for
discussion.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Fredrik's handing over of ElementTree
(https://mail.python.org/pipermail/python-dev/2012-February/116389.html)

[2] Web-SIG mailing list
(https://mail.python.org/mailman/listinfo/web-sig)


text:
PEP: 451 Title: A ModuleSpec Type for the Import System Version:
$Revision$ Last-Modified: $Date$ Author: Eric Snow
<EMAIL_ADDRESS>BDFL-Delegate: Brett Cannon
<EMAIL_ADDRESS>Alyssa Coghlan<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 08-Aug-2013 Python-Version: 3.4 Post-History:
08-Aug-2013, 28-Aug-2013, 18-Sep-2013, 24-Sep-2013, 04-Oct-2013
Resolution:
https://mail.python.org/pipermail/python-dev/2013-November/130104.html

Abstract

This PEP proposes to add a new class to importlib.machinery called
"ModuleSpec". It will provide all the import-related information used to
load a module and will be available without needing to load the module
first. Finders will directly provide a module's spec instead of a loader
(which they will continue to provide indirectly). The import machinery
will be adjusted to take advantage of module specs, including using them
to load modules.

Terms and Concepts

The changes in this proposal are an opportunity to make several existing
terms and concepts more clear, whereas currently they are
(unfortunately) ambiguous. New concepts are also introduced in this
proposal. Finally, it's worth explaining a few other existing terms with
which people may not be so familiar. For the sake of context, here is a
brief summary of all three groups of terms and concepts. A more detailed
explanation of the import system is found at [1].

name

In this proposal, a module's "name" refers to its fully-qualified name,
meaning the fully-qualified name of the module's parent (if any) joined
to the simple name of the module by a period.

finder

A "finder" is an object that identifies the loader that the import
system should use to load a module. Currently this is accomplished by
calling the finder's find_module() method, which returns the loader.

Finders are strictly responsible for providing the loader, which they do
through their find_module() method. The import system then uses that
loader to load the module.

loader

A "loader" is an object that is used to load a module during import.
Currently this is done by calling the loader's load_module() method. A
loader may also provide APIs for getting information about the modules
it can load, as well as about data from sources associated with such a
module.

Right now loaders (via load_module()) are responsible for certain
boilerplate, import-related operations. These are:

1.  Perform some (module-related) validation
2.  Create the module object
3.  Set import-related attributes on the module
4.  "Register" the module to sys.modules
5.  Exec the module
6.  Clean up in the event of failure while loading the module

This all takes place during the import system's call to
Loader.load_module().

origin

This is a new term and concept. The idea of it exists subtly in the
import system already, but this proposal makes the concept explicit.

"origin" in an import context means the system (or resource within a
system) from which a module originates. For the purposes of this
proposal, "origin" is also a string which identifies such a resource or
system. "origin" is applicable to all modules.

For example, the origin for built-in and frozen modules is the
interpreter itself. The import system already identifies this origin as
"built-in" and "frozen", respectively. This is demonstrated in the
following module repr: "<module 'sys' (built-in)>".

In fact, the module repr is already a relatively reliable, though
implicit, indicator of a module's origin. Other modules also indicate
their origin through other means, as described in the entry for
"location".

It is up to the loader to decide on how to interpret and use a module's
origin, if at all.

location

This is a new term. However the concept already exists clearly in the
import system, as associated with the __file__ and __path__ attributes
of modules, as well as the name/term "path" elsewhere.

A "location" is a resource or "place", rather than a system at large,
from which a module is loaded. It qualifies as an "origin". Examples of
locations include filesystem paths and URLs. A location is identified by
the name of the resource, but may not necessarily identify the system to
which the resource pertains. In such cases the loader would have to
identify the system itself.

In contrast to other kinds of module origin, a location cannot be
inferred by the loader just by the module name. Instead, the loader must
be provided with a string to identify the location, usually by the
finder that generates the loader. The loader then uses this information
to locate the resource from which it will load the module. In theory you
could load the module at a given location under various names.

The most common example of locations in the import system are the files
from which source and extension modules are loaded. For these modules
the location is identified by the string in the __file__ attribute.
Although __file__ isn't particularly accurate for some modules (e.g.
zipped), it is currently the only way that the import system indicates
that a module has a location.

A module that has a location may be called "locatable".

cache

The import system stores compiled modules in the __pycache__ directory
as an optimization. This module cache that we use today was provided by
PEP 3147. For this proposal, the relevant API for module caching is the
__cache__ attribute of modules and the cache_from_source() function in
importlib.util. Loaders are responsible for putting modules into the
cache (and loading out of the cache). Currently the cache is only used
for compiled source modules. However, loaders may take advantage of the
module cache for other kinds of modules.

package

The concept does not change, nor does the term. However, the distinction
between modules and packages is mostly superficial. Packages are
modules. They simply have a __path__ attribute and import may add
attributes bound to submodules. The typically perceived difference is a
source of confusion. This proposal explicitly de-emphasizes the
distinction between packages and modules where it makes sense to do so.

Motivation

The import system has evolved over the lifetime of Python. In late 2002
PEP 302 introduced standardized import hooks via finders and loaders and
sys.meta_path. The importlib module, introduced with Python 3.1, now
exposes a pure Python implementation of the APIs described by PEP 302,
as well as of the full import system. It is now much easier to
understand and extend the import system. While a benefit to the Python
community, this greater accessibility also presents a challenge.

As more developers come to understand and customize the import system,
any weaknesses in the finder and loader APIs will be more impactful. So
the sooner we can address any such weaknesses the import system, the
better...and there are a couple we hope to take care of with this
proposal.

Firstly, any time the import system needs to save information about a
module we end up with more attributes on module objects that are
generally only meaningful to the import system. It would be nice to have
a per-module namespace in which to put future import-related information
and to pass around within the import system. Secondly, there's an API
void between finders and loaders that causes undue complexity when
encountered. The PEP 420 (namespace packages) implementation had to work
around this. The complexity surfaced again during recent efforts on a
separate proposal.[2]

The finder and loader sections above detail current responsibility of
both. Notably, loaders are not required to provide any of the
functionality of their load_module() method through other methods. Thus,
though the import-related information about a module is likely available
without loading the module, it is not otherwise exposed.

Furthermore, the requirements associated with load_module() are common
to all loaders and mostly are implemented in exactly the same way. This
means every loader has to duplicate the same boilerplate code.
importlib.util provides some tools that help with this, but it would be
more helpful if the import system simply took charge of these
responsibilities. The trouble is that this would limit the degree of
customization that load_module() could easily continue to facilitate.

More importantly, While a finder could provide the information that the
loader's load_module() would need, it currently has no consistent way to
get it to the loader. This is a gap between finders and loaders which
this proposal aims to fill.

Finally, when the import system calls a finder's find_module(), the
finder makes use of a variety of information about the module that is
useful outside the context of the method. Currently the options are
limited for persisting that per-module information past the method call,
since it only returns the loader. Popular options for this limitation
are to store the information in a module-to-info mapping somewhere on
the finder itself, or store it on the loader.

Unfortunately, loaders are not required to be module-specific. On top of
that, some of the useful information finders could provide is common to
all finders, so ideally the import system could take care of those
details. This is the same gap as before between finders and loaders.

As an example of complexity attributable to this flaw, the
implementation of namespace packages in Python 3.3 (see PEP 420) added
FileFinder.find_loader() because there was no good way for find_module()
to provide the namespace search locations.

The answer to this gap is a ModuleSpec object that contains the
per-module information and takes care of the boilerplate functionality
involved with loading the module.

Specification

The goal is to address the gap between finders and loaders while
changing as little of their semantics as possible. Though some
functionality and information is moved to the new ModuleSpec type, their
behavior should remain the same. However, for the sake of clarity the
finder and loader semantics will be explicitly identified.

Here is a high-level summary of the changes described by this PEP. More
detail is available in later sections.

importlib.machinery.ModuleSpec (new)

An encapsulation of a module's import-system-related state during
import. See the ModuleSpec section below for a more detailed
description.

-   ModuleSpec(name, loader, *, origin=None, loader_state=None,
    is_package=None)

Attributes:

-   name - a string for the fully-qualified name of the module.
-   loader - the loader to use for loading.
-   origin - the name of the place from which the module is loaded, e.g.
    "builtin" for built-in modules and the filename for modules loaded
    from source.
-   submodule_search_locations - list of strings for where to find
    submodules, if a package (None otherwise).
-   loader_state - a container of extra module-specific data for use
    during loading.
-   cached (property) - a string for where the compiled module should be
    stored.
-   parent (RO-property) - the fully-qualified name of the package to
    which the module belongs as a submodule (or None).
-   has_location (RO-property) - a flag indicating whether or not the
    module's "origin" attribute refers to a location.

importlib.util Additions

These are ModuleSpec factory functions, meant as a convenience for
finders. See the Factory Functions section below for more detail.

-   spec_from_file_location(name, location, *, loader=None,
    submodule_search_locations=None)
    -   build a spec from file-oriented information and loader APIs.
-   spec_from_loader(name, loader, *, origin=None, is_package=None)
    - build a spec with missing information filled in by using loader
    APIs.

Other API Additions

-   importlib.find_spec(name, path=None, target=None) will work exactly
    the same as importlib.find_loader() (which it replaces), but return
    a spec instead of a loader.

For finders:

-   importlib.abc.MetaPathFinder.find_spec(name, path, target) and
    importlib.abc.PathEntryFinder.find_spec(name, target) will return a
    module spec to use during import.

For loaders:

-   importlib.abc.Loader.exec_module(module) will execute a module in
    its own namespace. It replaces importlib.abc.Loader.load_module(),
    taking over its module execution functionality.
-   importlib.abc.Loader.create_module(spec) (optional) will return the
    module to use for loading.

For modules:

-   Module objects will have a new attribute: __spec__.

API Changes

-   InspectLoader.is_package() will become optional.

Deprecations

-   importlib.abc.MetaPathFinder.find_module()
-   importlib.abc.PathEntryFinder.find_module()
-   importlib.abc.PathEntryFinder.find_loader()
-   importlib.abc.Loader.load_module()
-   importlib.abc.Loader.module_repr()
-   importlib.util.set_package()
-   importlib.util.set_loader()
-   importlib.find_loader()

Removals

These were introduced prior to Python 3.4's release, so they can simply
be removed.

-   importlib.abc.Loader.init_module_attrs()
-   importlib.util.module_to_load()

Other Changes

-   The import system implementation in importlib will be changed to
    make use of ModuleSpec.
-   importlib.reload() will make use of ModuleSpec.
-   A module's import-related attributes (other than __spec__) will no
    longer be used directly by the import system during that module's
    import. However, this does not impact use of those attributes (e.g.
    __path__) when loading other modules (e.g. submodules).
-   Import-related attributes should no longer be added to modules
    directly, except by the import system.
-   The module type's __repr__() will be a thin wrapper around a pure
    Python implementation which will leverage ModuleSpec.
-   The spec for the __main__ module will reflect the appropriate name
    and origin.

Backward-Compatibility

-   If a finder does not define find_spec(), a spec is derived from the
    loader returned by find_module().
-   PathEntryFinder.find_loader() still takes priority over
    find_module().
-   Loader.load_module() is used if exec_module() is not defined.

What Will not Change?

-   The syntax and semantics of the import statement.
-   Existing finders and loaders will continue to work normally.
-   The import-related module attributes will still be initialized with
    the same information.
-   Finders will still create loaders (now storing them in specs).
-   Loader.load_module(), if a module defines it, will have all the same
    requirements and may still be called directly.
-   Loaders will still be responsible for module data APIs.
-   importlib.reload() will still overwrite the import-related
    attributes.

Responsibilities

Here's a quick breakdown of where responsibilities lie after this PEP.

finders:

-   create/identify a loader that can load the module.
-   create the spec for the module.

loaders:

-   create the module (optional).
-   execute the module.

ModuleSpec:

-   orchestrate module loading
-   boilerplate for module loading, including managing sys.modules and
    setting import-related attributes
-   create module if loader doesn't
-   call loader.exec_module(), passing in the module in which to exec
-   contain all the information the loader needs to exec the module
-   provide the repr for modules

What Will Existing Finders and Loaders Have to Do Differently?

Immediately? Nothing. The status quo will be deprecated, but will
continue working. However, here are the things that the authors of
finders and loaders should change relative to this PEP:

-   Implement find_spec() on finders.
-   Implement exec_module() on loaders, if possible.

The ModuleSpec factory functions in importlib.util are intended to be
helpful for converting existing finders. spec_from_loader() and
spec_from_file_location() are both straightforward utilities in this
regard.

For existing loaders, exec_module() should be a relatively direct
conversion from the non-boilerplate portion of load_module(). In some
uncommon cases the loader should also implement create_module().

ModuleSpec Users

ModuleSpec objects have 3 distinct target audiences: Python itself,
import hooks, and normal Python users.

Python will use specs in the import machinery, in interpreter startup,
and in various standard library modules. Some modules are
import-oriented, like pkgutil, and others are not, like pickle and
pydoc. In all cases, the full ModuleSpec API will get used.

Import hooks (finders and loaders) will make use of the spec in specific
ways. First of all, finders may use the spec factory functions in
importlib.util to create spec objects. They may also directly adjust the
spec attributes after the spec is created. Secondly, the finder may bind
additional information to the spec (in finder_extras) for the loader to
consume during module creation/execution. Finally, loaders will make use
of the attributes on a spec when creating and/or executing a module.

Python users will be able to inspect a module's __spec__ to get
import-related information about the object. Generally, Python
applications and interactive users will not be using the ModuleSpec
factory functions nor any the instance methods.

How Loading Will Work

Here is an outline of what the import machinery does during loading,
adjusted to take advantage of the module's spec and the new loader API:

    module = None
    if spec.loader is not None and hasattr(spec.loader, 'create_module'):
        module = spec.loader.create_module(spec)
    if module is None:
        module = ModuleType(spec.name)
    # The import-related module attributes get set here:
    _init_module_attrs(spec, module)

    if spec.loader is None and spec.submodule_search_locations is not None:
        # Namespace package
        sys.modules[spec.name] = module
    elif not hasattr(spec.loader, 'exec_module'):
        spec.loader.load_module(spec.name)
        # __loader__ and __package__ would be explicitly set here for
        # backwards-compatibility.
    else:
        sys.modules[spec.name] = module
        try:
            spec.loader.exec_module(module)
        except BaseException:
            try:
                del sys.modules[spec.name]
            except KeyError:
                pass
            raise
    module_to_return = sys.modules[spec.name]

These steps are exactly what Loader.load_module() is already expected to
do. Loaders will thus be simplified since they will only need to
implement exec_module().

Note that we must return the module from sys.modules. During loading the
module may have replaced itself in sys.modules. Since we don't have a
post-import hook API to accommodate the use case, we have to deal with
it. However, in the replacement case we do not worry about setting the
import-related module attributes on the object. The module writer is on
their own if they are doing this.

How Reloading Will Work

Here is the corresponding outline for reload():

    _RELOADING = {}

    def reload(module):
        try:
            name = module.__spec__.name
        except AttributeError:
            name = module.__name__
        spec = find_spec(name, target=module)

        if sys.modules.get(name) is not module:
            raise ImportError
        if spec in _RELOADING:
            return _RELOADING[name]
        _RELOADING[name] = module
        try:
            if spec.loader is None:
                # Namespace loader
                _init_module_attrs(spec, module)
                return module
            if spec.parent and spec.parent not in sys.modules:
                raise ImportError

            _init_module_attrs(spec, module)
            # Ignoring backwards-compatibility call to load_module()
            # for simplicity.
            spec.loader.exec_module(module)
            return sys.modules[name]
        finally:
            del _RELOADING[name]

A key point here is the switch to Loader.exec_module() means that
loaders will no longer have an easy way to know at execution time if it
is a reload or not. Before this proposal, they could simply check to see
if the module was already in sys.modules. Now, by the time exec_module()
is called during load (not reload) the import machinery would already
have placed the module in sys.modules. This is part of the reason why
find_spec() has the "target" parameter.

The semantics of reload will remain essentially the same as they exist
already[3]. The impact of this PEP on some kinds of lazy loading modules
was a point of discussion.[4]

ModuleSpec

Attributes

Each of the following names is an attribute on ModuleSpec objects. A
value of None indicates "not set". This contrasts with module objects
where the attribute simply doesn't exist. Most of the attributes
correspond to the import-related attributes of modules. Here is the
mapping. The reverse of this mapping describes how the import machinery
sets the module attributes right before calling exec_module().

+----------------------------+----------------+
| On ModuleSpec              | On Modules     |
+============================+================+
| name                       | __name__       |
+----------------------------+----------------+
| loader                     | __loader__     |
+----------------------------+----------------+
| parent                     | __package__    |
+----------------------------+----------------+
| origin                     | __file__*      |
+----------------------------+----------------+
| cached                     | __cached__*,** |
+----------------------------+----------------+
| submodule_search_locations | __path__**     |
+----------------------------+----------------+
| loader_state               |   -            |
+----------------------------+----------------+
| has_location               |   -            |
+----------------------------+----------------+

* Set on the module only if spec.has_location is true.
** Set on the module only if the spec attribute is not None.

While parent and has_location are read-only properties, the remaining
attributes can be replaced after the module spec is created and even
after import is complete. This allows for unusual cases where directly
modifying the spec is the best option. However, typical use should not
involve changing the state of a module's spec.

origin

"origin" is a string for the name of the place from which the module
originates. See origin above. Aside from the informational value, it is
also used in the module's repr. In the case of a spec where
"has_location" is true, __file__ is set to the value of "origin". For
built-in modules "origin" would be set to "built-in".

has_location

As explained in the location section above, many modules are
"locatable", meaning there is a corresponding resource from which the
module will be loaded and that resource can be described by a string. In
contrast, non-locatable modules can't be loaded in this fashion, e.g.
builtin modules and modules dynamically created in code. For these, the
name is the only way to access them, so they have an "origin" but not a
"location".

"has_location" is true if the module is locatable. In that case the
spec's origin is used as the location and __file__ is set to
spec.origin. If additional location information is required (e.g.
zipimport), that information may be stored in spec.loader_state.

"has_location" may be implied from the existence of a load_data() method
on the loader.

Incidentally, not all locatable modules will be cache-able, but most
will.

submodule_search_locations

The list of location strings, typically directory paths, in which to
search for submodules. If the module is a package this will be set to a
list (even an empty one). Otherwise it is None.

The name of the corresponding module attribute, __path__, is relatively
ambiguous. Instead of mirroring it, we use a more explicit attribute
name that makes the purpose clear.

loader_state

A finder may set loader_state to any value to provide additional data
for the loader to use during loading. A value of None is the default and
indicates that there is no additional data. Otherwise it can be set to
any object, such as a dict, list, or types.SimpleNamespace, containing
the relevant extra information.

For example, zipimporter could use it to pass the zip archive name to
the loader directly, rather than needing to derive it from origin or
create a custom loader for each find operation.

loader_state is meant for use by the finder and corresponding loader. It
is not guaranteed to be a stable resource for any other use.

Factory Functions

spec_from_file_location(name, location, *, loader=None,
submodule_search_locations=None)

Build a spec from file-oriented information and loader APIs.

-   "origin" will be set to the location.
-   "has_location" will be set to True.
-   "cached" will be set to the result of calling cache_from_source().
-   "origin" can be deduced from loader.get_filename() (if "location" is
    not passed in.
-   "loader" can be deduced from suffix if the location is a filename.
-   "submodule_search_locations" can be deduced from loader.is_package()
    and from os.path.dirname(location) if location is a filename.

spec_from_loader(name, loader, *, origin=None, is_package=None)

Build a spec with missing information filled in by using loader APIs.

-   "has_location" can be deduced from loader.get_data.
-   "origin" can be deduced from loader.get_filename().
-   "submodule_search_locations" can be deduced from loader.is_package()
    and from os.path.dirname(location) if location is a filename.

Backward Compatibility

ModuleSpec doesn't have any. This would be a different story if
Finder.find_module() were to return a module spec instead of loader. In
that case, specs would have to act like the loader that would have been
returned instead. Doing so would be relatively simple, but is an
unnecessary complication. It was part of earlier versions of this PEP.

Subclassing

Subclasses of ModuleSpec are allowed, but should not be necessary.
Simply setting loader_state or adding functionality to a custom finder
or loader will likely be a better fit and should be tried first.
However, as long as a subclass still fulfills the requirements of the
import system, objects of that type are completely fine as the return
value of Finder.find_spec(). The same points apply to duck-typing.

Existing Types

Module Objects

Other than adding __spec__, none of the import-related module attributes
will be changed or deprecated, though some of them could be; any such
deprecation can wait until Python 4.

A module's spec will not be kept in sync with the corresponding
import-related attributes. Though they may differ, in practice they will
typically be the same.

One notable exception is that case where a module is run as a script by
using the -m flag. In that case module.__spec__.name will reflect the
actual module name while module.__name__ will be __main__.

A module's spec is not guaranteed to be identical between two modules
with the same name. Likewise there is no guarantee that successive calls
to importlib.find_spec() will return the same object or even an
equivalent object, though at least the latter is likely.

Finders

Finders are still responsible for identifying, and typically creating,
the loader that should be used to load a module. That loader will now be
stored in the module spec returned by find_spec() rather than returned
directly. As is currently the case without the PEP, if a loader would be
costly to create, that loader can be designed to defer the cost until
later.

MetaPathFinder.find_spec(name, path=None, target=None)

PathEntryFinder.find_spec(name, target=None)

Finders must return ModuleSpec objects when find_spec() is called. This
new method replaces find_module() and find_loader() (in the
PathEntryFinder case). If a loader does not have find_spec(),
find_module() and find_loader() are used instead, for
backward-compatibility.

Adding yet another similar method to loaders is a case of practicality.
find_module() could be changed to return specs instead of loaders. This
is tempting because the import APIs have suffered enough, especially
considering PathEntryFinder.find_loader() was just added in Python 3.3.
However, the extra complexity and a less-than-explicit method name
aren't worth it.

The "target" parameter of find_spec()

A call to find_spec() may optionally include a "target" argument. This
is the module object that will be used subsequently as the target of
loading. During normal import (and by default) "target" is None, meaning
the target module has yet to be created. During reloading the module
passed in to reload() is passed through to find_spec() as the target.
This argument allows the finder to build the module spec with more
information than is otherwise available. Doing so is particularly
relevant in identifying the loader to use.

Through find_spec() the finder will always identify the loader it will
return in the spec (or return None). At the point the loader is
identified, the finder should also decide whether or not the loader
supports loading into the target module, in the case that "target" is
passed in. This decision may entail consulting with the loader.

If the finder determines that the loader does not support loading into
the target module, it should either find another loader or raise
ImportError (completely stopping import of the module). This
determination is especially important during reload since, as noted in
How Reloading Will Work, loaders will no longer be able to trivially
identify a reload situation on their own.

Two alternatives were presented to the "target" parameter:
Loader.supports_reload() and adding "target" to Loader.exec_module()
instead of find_spec(). supports_reload() was the initial approach to
the reload situation.[5] However, there was some opposition to the
loader-specific, reload-centric approach. [6]

As to "target" on exec_module(), the loader may need other information
from the target module (or spec) during reload, more than just "does
this loader support reloading this module", that is no longer available
with the move away from load_module(). A proposal on the table was to
add something like "target" to exec_module().[7] However, putting
"target" on find_spec() instead is more in line with the goals of this
PEP. Furthermore, it obviates the need for supports_reload().

Namespace Packages

Currently a path entry finder may return (None, portions) from
find_loader() to indicate it found part of a possible namespace package.
To achieve the same effect, find_spec() must return a spec with "loader"
set to None (a.k.a. not set) and with submodule_search_locations set to
the same portions as would have been provided by find_loader(). It's up
to PathFinder how to handle such specs.

Loaders

Loader.exec_module(module)

Loaders will have a new method, exec_module(). Its only job is to "exec"
the module and consequently populate the module's namespace. It is not
responsible for creating or preparing the module object, nor for any
cleanup afterward. It has no return value. exec_module() will be used
during both loading and reloading.

exec_module() should properly handle the case where it is called more
than once. For some kinds of modules this may mean raising ImportError
every time after the first time the method is called. This is
particularly relevant for reloading, where some kinds of modules do not
support in-place reloading.

Loader.create_module(spec)

Loaders may also implement create_module() that will return a new module
to exec. It may return None to indicate that the default module creation
code should be used. One use case, though atypical, for create_module()
is to provide a module that is a subclass of the builtin module type.
Most loaders will not need to implement create_module(),

create_module() should properly handle the case where it is called more
than once for the same spec/module. This may include returning None or
raising ImportError.

Note

exec_module() and create_module() should not set any import-related
module attributes. The fact that load_module() does is a design flaw
that this proposal aims to correct.

Other changes:

PEP 420 introduced the optional module_repr() loader method to limit the
amount of special-casing in the module type's __repr__(). Since this
method is part of ModuleSpec, it will be deprecated on loaders. However,
if it exists on a loader it will be used exclusively.

Loader.init_module_attr() method, added prior to Python 3.4's release,
will be removed in favor of the same method on ModuleSpec.

However, InspectLoader.is_package() will not be deprecated even though
the same information is found on ModuleSpec. ModuleSpec can use it to
populate its own is_package if that information is not otherwise
available. Still, it will be made optional.

In addition to executing a module during loading, loaders will still be
directly responsible for providing APIs concerning module-related data.

Other Changes

-   The various finders and loaders provided by importlib will be
    updated to comply with this proposal.
-   Any other implementations of or dependencies on the import-related
    APIs (particularly finders and loaders) in the stdlib will be
    likewise adjusted to this PEP. While they should continue to work,
    any such changes that get missed should be considered bugs for the
    Python 3.4.x series.
-   The spec for the __main__ module will reflect how the interpreter
    was started. For instance, with -m the spec's name will be that of
    the module used, while __main__.__name__ will still be "__main__".
-   We will add importlib.find_spec() to mirror importlib.find_loader()
    (which becomes deprecated).
-   importlib.reload() is changed to use ModuleSpec.
-   importlib.reload() will now make use of the per-module import lock.

Reference Implementation

A reference implementation is available at
http://bugs.python.org/issue18864.

Implementation Notes

* The implementation of this PEP needs to be cognizant of its impact on
pkgutil (and setuptools). pkgutil has some generic function-based
extensions to PEP 302 which may break if importlib starts wrapping
loaders without the tools' knowledge.

* Other modules to look at: runpy (and pythonrun.c), pickle, pydoc,
inspect.

For instance, pickle should be updated in the __main__ case to look at
module.__spec__.name.

Rejected Additions to the PEP

There were a few proposed additions to this proposal that did not fit
well enough into its scope.

There is no "PathModuleSpec" subclass of ModuleSpec that separates out
has_location, cached, and submodule_search_locations. While that might
make the separation cleaner, module objects don't have that distinction.
ModuleSpec will support both cases equally well.

While "ModuleSpec.is_package" would be a simple additional attribute
(aliasing self.submodule_search_locations is not None), it perpetuates
the artificial (and mostly erroneous) distinction between modules and
packages.

The module spec Factory Functions could be classmethods on ModuleSpec.
However that would expose them on all modules via __spec__, which has
the potential to unnecessarily confuse non-advanced Python users. The
factory functions have a specific use case, to support finder authors.
See ModuleSpec Users.

Likewise, several other methods could be added to ModuleSpec that expose
the specific uses of module specs by the import machinery:

-   create() - a wrapper around Loader.create_module().
-   exec(module) - a wrapper around Loader.exec_module().
-   load() - an analogue to the deprecated Loader.load_module().

As with the factory functions, exposing these methods via
module.__spec__ is less than desirable. They would end up being an
attractive nuisance, even if only exposed as "private" attributes (as
they were in previous versions of this PEP). If someone finds a need for
these methods later, we can expose the via an appropriate API (separate
from ModuleSpec) at that point, perhaps relative to PEP 406 (import
engine).

Conceivably, the load() method could optionally take a list of modules
with which to interact instead of sys.modules. Also, load() could be
leveraged to implement multi-version imports. Both are interesting
ideas, but definitely outside the scope of this proposal.

Others left out:

-   Add ModuleSpec.submodules (RO-property) - returns possible
    submodules relative to the spec.
-   Add ModuleSpec.loaded (RO-property) - the module in sys.module, if
    any.
-   Add ModuleSpec.data - a descriptor that wraps the data API of the
    spec's loader.
-   Also see[8].

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] http://docs.python.org/3/reference/import.html

[2] https://mail.python.org/pipermail/import-sig/2013-August/000658.html

[3] http://bugs.python.org/issue19413

[4] https://mail.python.org/pipermail/python-dev/2013-August/128129.html

[5] https://mail.python.org/pipermail/python-dev/2013-October/129913.html

[6] https://mail.python.org/pipermail/python-dev/2013-October/129971.html

[7] https://mail.python.org/pipermail/python-dev/2013-October/129933.html

[8] https://mail.python.org/pipermail/import-sig/2013-September/000735.html