text:
PEP: 234 Title: Iterators Author: Ka-Ping Yee<EMAIL_ADDRESS>Guido van
Rossum<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 30-Jan-2001 Python-Version: 2.1
Post-History: 30-Apr-2001

Abstract

This document proposes an iteration interface that objects can provide
to control the behaviour of for loops. Looping is customized by
providing a method that produces an iterator object. The iterator
provides a get next value operation that produces the next item in the
sequence each time it is called, raising an exception when no more items
are available.

In addition, specific iterators over the keys of a dictionary and over
the lines of a file are proposed, and a proposal is made to allow
spelling dict.has_key(key) as key in dict.

Note: this is an almost complete rewrite of this PEP by the second
author, describing the actual implementation checked into the trunk of
the Python 2.2 CVS tree. It is still open for discussion. Some of the
more esoteric proposals in the original version of this PEP have been
withdrawn for now; these may be the subject of a separate PEP in the
future.

C API Specification

A new exception is defined, StopIteration, which can be used to signal
the end of an iteration.

A new slot named tp_iter for requesting an iterator is added to the type
object structure. This should be a function of one PyObject * argument
returning a PyObject *, or NULL. To use this slot, a new C API function
PyObject_GetIter() is added, with the same signature as the tp_iter slot
function.

Another new slot, named tp_iternext, is added to the type structure, for
obtaining the next value in the iteration. To use this slot, a new C API
function PyIter_Next() is added. The signature for both the slot and the
API function is as follows, although the NULL return conditions differ:
the argument is a PyObject * and so is the return value. When the return
value is non-NULL, it is the next value in the iteration. When it is
NULL, then for the tp_iternext slot there are three possibilities:

-   No exception is set; this implies the end of the iteration.
-   The StopIteration exception (or a derived exception class) is set;
    this implies the end of the iteration.
-   Some other exception is set; this means that an error occurred that
    should be propagated normally.

The higher-level PyIter_Next() function clears the StopIteration
exception (or derived exception) when it occurs, so its NULL return
conditions are simpler:

-   No exception is set; this means iteration has ended.
-   Some exception is set; this means an error occurred, and should be
    propagated normally.

Iterators implemented in C should not implement a next() method with
similar semantics as the tp_iternext slot! When the type's dictionary is
initialized (by PyType_Ready()), the presence of a tp_iternext slot
causes a method next() wrapping that slot to be added to the type's
tp_dict. (Exception: if the type doesn't use PyObject_GenericGetAttr()
to access instance attributes, the next() method in the type's tp_dict
may not be seen.) (Due to a misunderstanding in the original text of
this PEP, in Python 2.2, all iterator types implemented a next() method
that was overridden by the wrapper; this has been fixed in Python 2.3.)

To ensure binary backwards compatibility, a new flag
Py_TPFLAGS_HAVE_ITER is added to the set of flags in the tp_flags field,
and to the default flags macro. This flag must be tested before
accessing the tp_iter or tp_iternext slots. The macro PyIter_Check()
tests whether an object has the appropriate flag set and has a non-NULL
tp_iternext slot. There is no such macro for the tp_iter slot (since the
only place where this slot is referenced should be PyObject_GetIter(),
and this can check for the Py_TPFLAGS_HAVE_ITER flag directly).

(Note: the tp_iter slot can be present on any object; the tp_iternext
slot should only be present on objects that act as iterators.)

For backwards compatibility, the PyObject_GetIter() function implements
fallback semantics when its argument is a sequence that does not
implement a tp_iter function: a lightweight sequence iterator object is
constructed in that case which iterates over the items of the sequence
in the natural order.

The Python bytecode generated for for loops is changed to use new
opcodes, GET_ITER and FOR_ITER, that use the iterator protocol rather
than the sequence protocol to get the next value for the loop variable.
This makes it possible to use a for loop to loop over non-sequence
objects that support the tp_iter slot. Other places where the
interpreter loops over the values of a sequence should also be changed
to use iterators.

Iterators ought to implement the tp_iter slot as returning a reference
to themselves; this is needed to make it possible to use an iterator (as
opposed to a sequence) in a for loop.

Iterator implementations (in C or in Python) should guarantee that once
the iterator has signalled its exhaustion, subsequent calls to
tp_iternext or to the next() method will continue to do so. It is not
specified whether an iterator should enter the exhausted state when an
exception (other than StopIteration) is raised. Note that Python cannot
guarantee that user-defined or 3rd party iterators implement this
requirement correctly.

Python API Specification

The StopIteration exception is made visible as one of the standard
exceptions. It is derived from Exception.

A new built-in function is defined, iter(), which can be called in two
ways:

-   iter(obj) calls PyObject_GetIter(obj).
-   iter(callable, sentinel) returns a special kind of iterator that
    calls the callable to produce a new value, and compares the return
    value to the sentinel value. If the return value equals the
    sentinel, this signals the end of the iteration and StopIteration is
    raised rather than returning normal; if the return value does not
    equal the sentinel, it is returned as the next value from the
    iterator. If the callable raises an exception, this is propagated
    normally; in particular, the function is allowed to raise
    StopIteration as an alternative way to end the iteration. (This
    functionality is available from the C API as
    PyCallIter_New(callable, sentinel).)

Iterator objects returned by either form of iter() have a next() method.
This method either returns the next value in the iteration, or raises
StopIteration (or a derived exception class) to signal the end of the
iteration. Any other exception should be considered to signify an error
and should be propagated normally, not taken to mean the end of the
iteration.

Classes can define how they are iterated over by defining an __iter__()
method; this should take no additional arguments and return a valid
iterator object. A class that wants to be an iterator should implement
two methods: a next() method that behaves as described above, and an
__iter__() method that returns self.

The two methods correspond to two distinct protocols:

1.  An object can be iterated over with for if it implements __iter__()
    or __getitem__().
2.  An object can function as an iterator if it implements next().

Container-like objects usually support protocol 1. Iterators are
currently required to support both protocols. The semantics of iteration
come only from protocol 2; protocol 1 is present to make iterators
behave like sequences; in particular so that code receiving an iterator
can use a for-loop over the iterator.

Dictionary Iterators

-   Dictionaries implement a sq_contains slot that implements the same
    test as the has_key() method. This means that we can write

        if k in dict: ...

    which is equivalent to

        if dict.has_key(k): ...

-   Dictionaries implement a tp_iter slot that returns an efficient
    iterator that iterates over the keys of the dictionary. During such
    an iteration, the dictionary should not be modified, except that
    setting the value for an existing key is allowed (deletions or
    additions are not, nor is the update() method). This means that we
    can write

        for k in dict: ...

    which is equivalent to, but much faster than

        for k in dict.keys(): ...

    as long as the restriction on modifications to the dictionary
    (either by the loop or by another thread) are not violated.

-   Add methods to dictionaries that return different kinds of iterators
    explicitly:

        for key in dict.iterkeys(): ...

        for value in dict.itervalues(): ...

        for key, value in dict.iteritems(): ...

    This means that for x in dict is shorthand for
    for x in dict.iterkeys().

Other mappings, if they support iterators at all, should also iterate
over the keys. However, this should not be taken as an absolute rule;
specific applications may have different requirements.

File Iterators

The following proposal is useful because it provides us with a good
answer to the complaint that the common idiom to iterate over the lines
of a file is ugly and slow.

-   Files implement a tp_iter slot that is equivalent to
    iter(f.readline, ""). This means that we can write

        for line in file:
            ...

    as a shorthand for

        for line in iter(file.readline, ""):
            ...

    which is equivalent to, but faster than

        while 1:
            line = file.readline()
            if not line:
                break
            ...

This also shows that some iterators are destructive: they consume all
the values and a second iterator cannot easily be created that iterates
independently over the same values. You could open the file for a second
time, or seek() to the beginning, but these solutions don't work for all
file types, e.g. they don't work when the open file object really
represents a pipe or a stream socket.

Because the file iterator uses an internal buffer, mixing this with
other file operations (e.g. file.readline()) doesn't work right. Also,
the following code:

    for line in file:
        if line == "\n":
            break
    for line in file:
       print line,

doesn't work as you might expect, because the iterator created by the
second for-loop doesn't take the buffer read-ahead by the first for-loop
into account. A correct way to write this is:

    it = iter(file)
    for line in it:
        if line == "\n":
            break
    for line in it:
        print line,

(The rationale for these restrictions are that for line in file ought to
become the recommended, standard way to iterate over the lines of a
file, and this should be as fast as can be. The iterator version is
considerable faster than calling readline(), due to the internal buffer
in the iterator.)

Rationale

If all the parts of the proposal are included, this addresses many
concerns in a consistent and flexible fashion. Among its chief virtues
are the following four -- no, five -- no, six -- points:

1.  It provides an extensible iterator interface.
2.  It allows performance enhancements to list iteration.
3.  It allows big performance enhancements to dictionary iteration.
4.  It allows one to provide an interface for just iteration without
    pretending to provide random access to elements.
5.  It is backward-compatible with all existing user-defined classes and
    extension objects that emulate sequences and mappings, even mappings
    that only implement a subset of {__getitem__, keys, values, items}.
6.  It makes code iterating over non-sequence collections more concise
    and readable.

Resolved Issues

The following topics have been decided by consensus or BDFL
pronouncement.

-   Two alternative spellings for next() have been proposed but
    rejected: __next__(), because it corresponds to a type object slot
    (tp_iternext); and __call__(), because this is the only operation.

    Arguments against __next__(): while many iterators are used in for
    loops, it is expected that user code will also call next() directly,
    so having to write __next__() is ugly; also, a possible extension of
    the protocol would be to allow for prev(), current() and reset()
    operations; surely we don't want to use __prev__(), __current__(),
    __reset__().

    Arguments against __call__() (the original proposal): taken out of
    context, x() is not very readable, while x.next() is clear; there's
    a danger that every special-purpose object wants to use __call__()
    for its most common operation, causing more confusion than clarity.

    (In retrospect, it might have been better to go for __next__() and
    have a new built-in, next(it), which calls it.__next__(). But alas,
    it's too late; this has been deployed in Python 2.2 since December
    2001.)

-   Some folks have requested the ability to restart an iterator. This
    should be dealt with by calling iter() on a sequence repeatedly, not
    by the iterator protocol itself. (See also requested extensions
    below.)

-   It has been questioned whether an exception to signal the end of the
    iteration isn't too expensive. Several alternatives for the
    StopIteration exception have been proposed: a special value End to
    signal the end, a function end() to test whether the iterator is
    finished, even reusing the IndexError exception.

    -   A special value has the problem that if a sequence ever contains
        that special value, a loop over that sequence will end
        prematurely without any warning. If the experience with
        null-terminated C strings hasn't taught us the problems this can
        cause, imagine the trouble a Python introspection tool would
        have iterating over a list of all built-in names, assuming that
        the special End value was a built-in name!
    -   Calling an end() function would require two calls per iteration.
        Two calls is much more expensive than one call plus a test for
        an exception. Especially the time-critical for loop can test
        very cheaply for an exception.
    -   Reusing IndexError can cause confusion because it can be a
        genuine error, which would be masked by ending the loop
        prematurely.

-   Some have asked for a standard iterator type. Presumably all
    iterators would have to be derived from this type. But this is not
    the Python way: dictionaries are mappings because they support
    __getitem__() and a handful other operations, not because they are
    derived from an abstract mapping type.

-   Regarding if key in dict: there is no doubt that the dict.has_key(x)
    interpretation of x in dict is by far the most useful
    interpretation, probably the only useful one. There has been
    resistance against this because x in list checks whether x is
    present among the values, while the proposal makes x in dict check
    whether x is present among the keys. Given that the symmetry between
    lists and dictionaries is very weak, this argument does not have
    much weight.

-   The name iter() is an abbreviation. Alternatives proposed include
    iterate(), traverse(), but these appear too long. Python has a
    history of using abbrs for common builtins, e.g. repr(), str(),
    len().

    Resolution: iter() it is.

-   Using the same name for two different operations (getting an
    iterator from an object and making an iterator for a function with a
    sentinel value) is somewhat ugly. I haven't seen a better name for
    the second operation though, and since they both return an iterator,
    it's easy to remember.

    Resolution: the builtin iter() takes an optional argument, which is
    the sentinel to look for.

-   Once a particular iterator object has raised StopIteration, will it
    also raise StopIteration on all subsequent next() calls? Some say
    that it would be useful to require this, others say that it is
    useful to leave this open to individual iterators. Note that this
    may require an additional state bit for some iterator
    implementations (e.g. function-wrapping iterators).

    Resolution: once StopIteration is raised, calling it.next()
    continues to raise StopIteration.

    Note: this was in fact not implemented in Python 2.2; there are many
    cases where an iterator's next() method can raise StopIteration on
    one call but not on the next. This has been remedied in Python 2.3.

-   It has been proposed that a file object should be its own iterator,
    with a next() method returning the next line. This has certain
    advantages, and makes it even clearer that this iterator is
    destructive. The disadvantage is that this would make it even more
    painful to implement the "sticky StopIteration" feature proposed in
    the previous bullet.

    Resolution: tentatively rejected (though there are still people
    arguing for this).

-   Some folks have requested extensions of the iterator protocol, e.g.
    prev() to get the previous item, current() to get the current item
    again, finished() to test whether the iterator is finished, and
    maybe even others, like rewind(), __len__(), position().

    While some of these are useful, many of these cannot easily be
    implemented for all iterator types without adding arbitrary
    buffering, and sometimes they can't be implemented at all (or not
    reasonably). E.g. anything to do with reversing directions can't be
    done when iterating over a file or function. Maybe a separate PEP
    can be drafted to standardize the names for such operations when
    they are implementable.

    Resolution: rejected.

-   There has been a long discussion about whether

        for x in dict: ...

    should assign x the successive keys, values, or items of the
    dictionary. The symmetry between if x in y and for x in y suggests
    that it should iterate over keys. This symmetry has been observed by
    many independently and has even been used to "explain" one using the
    other. This is because for sequences, if x in y iterates over y
    comparing the iterated values to x. If we adopt both of the above
    proposals, this will also hold for dictionaries.

    The argument against making for x in dict iterate over the keys
    comes mostly from a practicality point of view: scans of the
    standard library show that there are about as many uses of
    for x in dict.items() as there are of for x in dict.keys(), with the
    items() version having a small majority. Presumably many of the
    loops using keys() use the corresponding value anyway, by writing
    dict[x], so (the argument goes) by making both the key and value
    available, we could support the largest number of cases. While this
    is true, I (Guido) find the correspondence between for x in dict and
    if x in dict too compelling to break, and there's not much overhead
    in having to write dict[x] to explicitly get the value.

    For fast iteration over items, use
    for key, value in dict.iteritems(). I've timed the difference
    between

        for key in dict: dict[key]

    and

        for key, value in dict.iteritems(): pass

    and found that the latter is only about 7% faster.

    Resolution: By BDFL pronouncement, for x in dict iterates over the
    keys, and dictionaries have iteritems(), iterkeys(), and
    itervalues() to return the different flavors of dictionary
    iterators.

Mailing Lists

The iterator protocol has been discussed extensively in a mailing list
on SourceForge:

  http://lists.sourceforge.net/lists/listinfo/python-iterators

Initially, some of the discussion was carried out at Yahoo; archives are
still accessible:

  http://groups.yahoo.com/group/python-iter

Copyright

This document is in the public domain.

authors:
Guido van Rossum
Ka-Ping Yee


text:
PEP: 752 Title: Implicit namespaces for package repositories Author:
Ofek Lev<EMAIL_ADDRESS>Sponsor: Barry Warsaw
<EMAIL_ADDRESS>PEP-Delegate: Dustin Ingram<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/63192 Status: Draft Type:
Standards Track Topic: Packaging Created: 13-Aug-2024 Post-History:
18-Aug-2024, 07-Sep-2024,

Abstract

This PEP specifies a way for organizations to reserve package name
prefixes for future uploads.

  "Namespaces are one honking great idea -- let's do more of those!" -
  PEP 20

Motivation

The current ecosystem lacks a way for projects with many packages to
signal a verified pattern of ownership. Such projects fall into two
categories.

The first category is projects[1] that want complete control over their
namespace. A few examples:

-   Major cloud providers like Amazon, Google and Microsoft have a
    common prefix for each feature's corresponding package[2]. For
    example, most of Google's packages are prefixed by google-cloud-
    e.g. google-cloud-compute for using virtual machines.
-   OpenTelemetry is an open standard for observability with official
    packages for the core APIs and SDK with contrib packages to collect
    data from various sources. All packages are prefixed by
    opentelemetry- with child prefixes in the form
    opentelemetry-<component>-<name>-. The contrib packages live in a
    central repository and they are the only ones with the ability to
    publish.

The second category is projects[3] that want to share their namespace
such that some packages are officially maintained and third-party
developers are encouraged to participate by publishing their own. Some
examples:

-   Project Jupyter is devoted to the development of tooling for sharing
    interactive documents. They support extensions which in most cases
    (and in all cases for officially maintained extensions) are prefixed
    by jupyter-.
-   Django is one of the most widely used web frameworks in existence.
    They have the concept of reusable apps, which are commonly installed
    via third-party packages that implement a subset of functionality to
    extend Django-based websites. These packages are by convention
    prefixed by django- or dj-.

Such projects are uniquely vulnerable to name-squatting attacks which
can ultimately result in dependency confusion.

For example, say a new product is released for which monitoring would be
valuable. It would be reasonable to assume that Datadog would eventually
support it as an official integration. It takes a nontrivial amount of
time to deliver such an integration due to roadmap prioritization and
the time required for implementation. It would be impossible to reserve
the name of every potential package so in the interim an attacker may
create a package that appears legitimate which would execute malicious
code at runtime. Not only are users more likely to install such packages
but doing so taints the perception of the entire project.

Although PEP 708 attempts to address this attack vector, it is
specifically about the case of multiple repositories being considered
during dependency resolution and does not offer any protection to the
aforementioned use cases.

Namespacing also would drastically reduce the incidence of typosquatting
because typos would have to be in the prefix itself which is normalized
and likely to be a short, well-known identifier like aws-. In recent
years, typosquatting has become a popular attack vector [4].

The current protection against typosquatting used by PyPI is to
normalize similar characters but that is insufficient for these use
cases.

Rationale

Other package ecosystems have generally solved this problem by taking
one of two approaches: either minimizing or maximizing backwards
compatibility.

-   NPM has the concept of scoped packages which were introduced
    primarily to combat there being a dearth of available good package
    names (whether a real or perceived phenomenon). When a user or
    organization signs up they are given a scope that matches their
    name. For example, the package for using Google Cloud Storage is
    @google-cloud/storage where @google-cloud/ is the scope. Regular
    user accounts (non-organization) may publish unscoped packages for
    public use. This approach has the lowest amount of backwards
    compatibility because every installer and tool has to be modified to
    account for scopes.
-   NuGet has the concept of package ID prefix reservation which was
    introduced primarily to satisfy users wishing to know where a
    package came from. A package name prefix may be reserved for use by
    one or more owners. Every reserved package has a special indication
    on its page to communicate this. After reservation, any upload with
    a reserved prefix will fail if the user is not an owner of the
    prefix. Existing packages that have a prefix that is owned may
    continue to release as usual. This approach has the highest amount
    of backwards compatibility because only modifications to indices
    like PyPI are required and installers do not need to change.

This PEP specifies the NuGet approach of authorized reservation across a
flat namespace. Any solution that requires new package syntax must be
built atop the existing flat namespace and therefore implicit namespaces
acquired via a reservation mechanism would be a prerequisite to such
explicit namespaces.

Although existing packages matching a reserved namespace would be
untouched, preventing future unauthorized uploads and strategically
applying PEP 541 takedown requests for malicious cases would reduce
risks to users to a negligible level.

Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in 2119.

Organization

    Organizations are entities that own projects and have various users
    associated with them.

Grant

    A grant is a reservation of a namespace for a package repository.

Open Namespace

    An open namespace allows for uploads from any project owner.

Restricted Namespace

    A restricted namespace only allows uploads from an owner of the
    namespace.

Parent Namespace

    A namespace's parent refers to the namespace without the trailing
    hyphenated component e.g. the parent of foo-bar is foo.

Child Namespace

    A namespace's child refers to the namespace with additional trailing
    hyphenated components e.g. foo-bar is a valid child of foo as is
    foo-bar-baz.

Specification

Organizations

Any package repository that allows for the creation of projects (e.g.
non-mirrors) MAY offer the concept of organizations. Organizations are
entities that own projects and have various users associated with them.

Organizations MAY reserve one or more namespaces. Such reservations
neither confer ownership nor grant special privileges to existing
projects.

Naming

A namespace MUST be a valid project name and normalized internally e.g.
foo.bar would become foo-bar.

Semantics

A namespace grant bestows ownership over the following:

1.  A project matching the namespace itself such as the placeholder
    package microsoft.
2.  Projects that start with the namespace followed by a hyphen. For
    example, the namespace foo would match the normalized project name
    foo-bar but not the project name foobar.

Package name matching acts upon the normalized namespace.

Namespaces are per-package repository and SHALL NOT be shared between
repositories. For example, if PyPI has a namespace microsoft that is
owned by the company Microsoft, packages starting with microsoft- that
come from other non-PyPI mirror repositories do not confer the same
level of trust.

Grants MUST NOT overlap. For example, if there is an existing grant for
foo-bar then a new grant for foo would be forbidden. An overlap is
determined by comparing the normalized proposed namespace with the
normalized namespace of every existing root grant. Every comparison must
append a hyphen to the end of the proposed and existing namespace. An
overlap is detected when any existing namespace starts with the proposed
namespace.

Uploads

If the following criteria are all true for a given upload:

1.  The project does not yet exist.
2.  The name matches a reserved namespace.
3.  The project is not owned by an organization with an active grant for
    the namespace.

Then the upload MUST fail with a 403 HTTP status code.

Open Namespaces

The owner of a grant may choose to allow others the ability to release
new projects with the associated namespace. Doing so MUST allow uploads
for new projects matching the namespace from any user.

It is possible for the owner of a namespace to both make it open and
allow other organizations to use the grant. In this case, the authorized
organizations have no special permissions and are equivalent to an open
grant without ownership.

Hidden Grants

Repositories MAY create hidden grants that are not visible to the public
which prevent their namespaces from being claimed by others. Such grants
MUST NOT be open and SHOULD NOT be exposed in the API.

Hidden grants are useful for repositories that wish to enforce upload
restrictions without the need to expose the namespace to the public.

Repository Metadata

The JSON API <691> version will be incremented from 1.2 to 1.3. The
following API changes MUST be implemented by repositories that support
this PEP. Repositories that do not support this PEP MUST NOT implement
these changes so that consumers of the API are able to determine whether
the repository supports this PEP.

Project Detail

The project detail <691#project-detail> response will be modified as
follows.

The namespace key MUST be null if the project does not match an active
namespace grant. If the project does match a namespace grant, the value
MUST be a mapping with the following keys:

-   prefix: This is the associated normalized namespace e.g. foo-bar. If
    the owner of the project owns multiple matching grants then this
    MUST be the namespace with the most number of characters. For
    example, if the project name matched both foo-bar and foo-bar-baz
    then this key would be the latter.
-   authorized: This is a boolean and will be true if the project owner
    is an organization and is one of the current owners of the grant.
    This is useful for tools that wish to make a distinction between
    official and community packages.
-   open: This is a boolean indicating whether the namespace is open.

Namespace Detail

The format of this URL is /namespace/<namespace> where <namespace> is
the normalized namespace. For example, the URL for the namespace foo.bar
would be /namespace/foo-bar.

The response will be a mapping with the following keys:

-   prefix: This is the normalized version of the namespace e.g.
    foo-bar.
-   owner: This is the organization that is responsible for the
    namespace.
-   open: This is a boolean indicating whether the namespace is open.
-   parent: This is the parent namespace if it exists. For example, if
    the namespace is foo-bar and there is an active grant for foo, then
    this would be "foo". If there is no parent then this key will be
    null.
-   children: This is an array of any child namespaces. For example, if
    the namespace is foo and there are active grants for foo-bar and
    foo-bar-baz then this would be ["foo-bar", "foo-bar-baz"].

Grant Removal

When a reserved namespace becomes unclaimed, repositories MUST set the
namespace key to null in the API.

Namespaces that were previously claimed but are now not SHOULD be
eligible for claiming again by any organization.

Community Buy-in

Representatives from the following organizations have expressed support
for this PEP (with a link to the discussion):

-   Apache Airflow
-   Typeshed
-   Project Jupyter (expanded)
-   Microsoft
-   DataDog

Backwards Compatibility

There are no intrinsic concerns because there is still a flat namespace
and installers need no modification. Additionally, many projects have
already chosen to signal a shared purpose with a prefix like typeshed
has done.

Security Implications

-   There is an opportunity to build on top of PEP 740 and PEP 480 so
    that one could prove cryptographically that a specific release came
    from an owner of the associated namespace. This PEP makes no effort
    to describe how this will happen other than that work is planned for
    the future.

How to Teach This

For consumers of packages we will document how metadata is exposed in
the API and potentially in future note tooling that supports utilizing
namespaces to provide extra security guarantees during installation.

Reference Implementation

None at this time.

Rejected Ideas

Organization Scoping

The primary motivation for this PEP is to reduce dependency confusion
attacks and NPM-style scoping with an allowance of the legacy flat
namespace would increase the risk. If documentation instructed a user to
install bar in the namespace foo then the user must be careful to
install @foo/bar and not foo-bar, or vice versa. The Python packaging
ecosystem has normalization rules for names in order to maximize the
ease of communication and this would be a regression.

The runtime environment of Python is also not conducive to scoping.
Whereas multiple versions of the same JavaScript package may coexist,
Python only allows a single global namespace. Barring major changes to
the language itself, this is nearly impossible to change. Additionally,
users have come to expect that the package name is usually the same as
what they would import and eliminating the flat namespace would do away
with that convention.

Scoping would be particularly affected by organization changes which are
bound to happen over time. An organization may change their name due to
internal shuffling, an acquisition, or any other reason. Whenever this
happens every project they own would in effect be renamed which would
cause unnecessary confusion for users, frequently.

Finally, the disruption to the community would be massive because it
would require an update from every package manager, security scanner,
IDE, etc. New packages released with the scoping would be incompatible
with older tools and would cause confusion for users along with
frustration from maintainers having to triage such complaints.

Encourage Dedicated Package Repositories

Critically, this imposes a burden on projects to maintain their own
infra. This is an unrealistic expectation for the vast majority of
companies and a complete non-starter for community projects.

This does not help in most cases because the default behavior of most
package managers is to use PyPI so users attempting to perform a simple
pip install would already be vulnerable to malicious packages.

In this theoretical future every project must document how to add their
repository to dependency resolution, which would be different for each
package manager. Few package managers are able to download specific
dependencies from specific repositories and would require users to use
verbose configuration in the common case.

The ones that do not support this would instead find a given package
using an ordered enumeration of repositories, leading to dependency
confusion. For example, say a user wants two packages from two custom
repositories X and Y. If each repository has both packages but one is
malicious on X and the other is malicious on Y then the user would be
unable to satisfy their requirements without encountering a malicious
package.

Use Fixed Prefixes

The idea here would be to have one or more top-level fixed prefixes that
are used for namespace reservations:

-   com-: Reserved for corporate organizations.
-   org-: Reserved for community organizations.

Organizations would then apply for a namespace prefixed by the type of
their organization.

This would cause perpetual disruption because when projects begin it is
unknown whether a user base will be large enough to warrant a namespace
reservation. Whenever that happens the project would have to be renamed
which would put a high maintenance burden on the project maintainers and
would cause confusion for users who have to learn a new way to reference
the project's packages. The potential for this deterring projects from
reserving namespaces at all is high.

Another issue with this approach is that projects often have branding in
mind (example) and would be reluctant to change their package names.

It's unrealistic to expect every company and project to voluntarily
change their existing and future package names.

Use DNS

The idea here is to add a new metadata field to projects in the API
called domain-authority. Repositories would support a new endpoint for
verifying the domain via HTTPS. Clients would then support options to
allow certain domains.

This does not solve the problem for the target audience who do not check
where their packages are coming from and is more about checking for the
integrity of uploads which is already supported in a more secure way by
PEP 740.

Most projects do not have a domain and could not benefit from this,
unfairly favoring organizations that have the financial means to acquire
one.

Open Issues

None at this time.

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] Additional examples of projects with restricted namespaces:

-   Typeshed is a community effort to maintain type stubs for various
    packages. The stub packages they maintain mirror the package name
    they target and are prefixed by types-. For example, the package
    requests has a stub that users would depend on called
    types-requests. Unofficial stubs are not supposed to use the types-
    prefix and are expected to use a -stubs suffix instead.
-   Sphinx is a documentation framework popular for large technical
    projects such as Swift and Python itself. They have the concept of
    extensions which are prefixed by sphinxcontrib-, many of which are
    maintained within a dedicated organization.
-   Apache Airflow is a platform to programmatically orchestrate tasks
    as directed acyclic graphs (DAGs). They have the concept of plugins,
    and also providers which are prefixed by apache-airflow-providers-.

[2] The following shows the package prefixes for the major cloud
providers:

-   Amazon: aws-cdk-
-   Google: google-cloud- and others based on google-
-   Microsoft: azure-

[3] Additional examples of projects with open namespaces:

-   pytest is Python's most popular testing framework. They have the
    concept of plugins which may be developed by anyone and by
    convention are prefixed by pytest-.
-   MkDocs is a documentation framework based on Markdown files. They
    also have the concept of plugins which may be developed by anyone
    and are usually prefixed by mkdocs-.
-   Datadog offers observability as a service for organizations at any
    scale. The Datadog Agent ships out-of-the-box with official
    integrations for many products, like various databases and web
    servers, which are distributed as Python packages that are prefixed
    by datadog-. There is support for creating third-party integrations
    which customers may run.

[4] Examples of typosquatting attacks targeting Python users:

-   django- namespace was squatted, among other packages, leading to a
    postmortem by PyPI.
-   cupy- namespace was squatted by a malicious actor thousands of
    times.
-   scikit- namespace was squatted, among other packages. Notice how
    packages with a known prefix are much more prone to successful
    attacks.
-   typing- namespace was squatted and this would be useful to prevent
    as a hidden grant.

license: Public Domain


text:
PEP: 610 Title: Recording the Direct URL Origin of installed
distributions Author: Stéphane Bidoul<EMAIL_ADDRESS>Chris
Jerdonek<EMAIL_ADDRESS>Sponsor: Alyssa Coghlan
<EMAIL_ADDRESS>BDFL-Delegate: Pradyun Gedam<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/recording-the-source-url-of-an-installed-distribution/1535
Status: Final Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 21-Apr-2019 Post-History: Resolution:
https://discuss.python.org/t/1535/56

packaging:direct-url

Abstract

Following PEP 440, a distribution can be identified by a name and either
a version, or a direct URL reference (see
PEP440 Direct References <440#direct-references>). After installation,
the name and version are captured in the project metadata, but currently
there is no way to obtain details of the URL used when the distribution
was identified by a direct URL reference.

This proposal defines additional metadata, to be added to the installed
distribution by the installation front end, which records the Direct URL
Origin for use by consumers which introspect the database of installed
packages (see PEP 376).

Motivation

The original motivation of this PEP was to permit tools with a "freeze"
operation allowing a Python environment to be recreated to work in a
broader range of situations.

Specifically, the PEP originated from the desire to address pip issue
#609: i.e. improving the behavior of pip freeze in the presence of
distributions installed from direct URL references. It follows a thread
on discuss.python.org about the best course of action to implement it.

Installation from direct URL references

Python installers such as pip are capable of downloading and installing
distributions from package indexes. They are also capable of downloading
and installing source code from requirements specifying arbitrary URLs
of source archives and Version Control Systems (VCS) repositories, as
standardized in PEP440 Direct References <440#direct-references>.

In other words, two relevant installation modes exist.

1.  the package to install is specified as a name and version specifier:

  In this case, the installer looks in a package index (or optionally
  using --find-links in the case of pip) to find the distribution to
  install.

2.  The package to install is specified as a direct URL reference:

  In this case, the installer downloads whatever is specified by the URL
  (typically a wheel, a source archive or a VCS repository) and installs
  it.

  In this mode, installers typically download the source code in a
  temporary directory, invoke the PEP 517 build backend to produce a
  wheel if needed, install the wheel, and delete the temporary
  directory.

  After installation, no trace of the URL the user requested to download
  the package is left on the user system.

Freezing an environment

Pip also sports a command named pip freeze which examines the Database
of Installed Python Distributions to generate a list of requirements.
The main goal of this command is to help users generating a list of
requirements that will later allow the re-installation the same
environment with the highest possible fidelity.

As of pip version 19.3, the pip freeze command outputs a name==version
line for each installed distribution (except for editable installs). To
achieve the goal of reinstalling the same environment, this requires the
(name, version) tuple to refer to an immutable version of the
distribution. The immutability is guaranteed by package indexes such as
Warehouse. The package index to use is typically known from
environmental or command line parameters of the installer.

This freeze mechanism therefore works fine for installation mode 1 (i.e.
when the package to install was specified as a name plus version
specifier).

For installation mode 2, i.e. when the package to install was specified
as a direct URL reference, the name==version tuple is obviously not
sufficient to reinstall the same distribution and users of the freeze
command expect it to output the URL that was originally requested.

The reasoning above is equally applicable to tools, other than
pip freeze, that would attempt to generate a Pipfile.lock or any other
similar format from the Database of Installed Python Distributions.
Unless specified otherwise, "freeze" is used in this document as a
generic term for such an operation.

The importance of installing from (VCS) URLs for application integrators

For an application integrator, it is important to be able to reliably
install and freeze unreleased version of python distributions. For
instance when a developer needs to deploy an unreleased patched version
of a dependency, it is common to install the dependency directly from a
VCS branch that has the patch, while waiting for the maintainer to
release an updated version.

In such cases, it is important for "freeze" to pin the exact VCS
reference (commit-hash if available) that was installed, in order to
create reproducible builds with the highest possible fidelity.

Additional origin metadata available for VCS URLs

For VCS URLs, there is additional origin information available only at
install time useful for introspection and certain workflows. For
example, when installing a revision from a VCS URL, a tool can determine
if the revision corresponds to a branch, tag or (in the case of Git) a
ref. This information can be used when introspecting the Database of
Installed Distributions to communicate to users more information about
what version was installed (e.g. whether a branch or tag was installed
and, if so, the name of the branch or tag). This also permits one to
know whether a PEP 440 direct reference URL can be constructed using the
tag form, as only tags have the semantics of immutability.

In cases where the revision is mutable (e.g. branches and Git refs),
knowing this information enables workflows where users can e.g. update
to the latest version of a branch they are tracking, or update to the
latest version of a pull request they are reviewing locally. In
contrast, when the revision is a tag, tools can know in advance (e.g.
without network calls) that no update is needed.

As with the URL itself, if this information isn't recorded at install
time when the VCS repository is available, it would otherwise be lost.

Note about "editable" installs

The editable installation mode of pip roughly lets a user insert a local
directory in sys.path for development purpose. This mode is somewhat
abused to work around the fact that a non-editable install from a VCS
URL loses track of the origin after installation. Indeed, editable
installs implicitly record the VCS origin in the checkout directory, so
the information can be recovered when running "freeze".

The use of this workaround, although useful, is fragile, creates
confusion about the purpose of the editable mode, and works only when
the distribution can be installed with setuptools (i.e. it is not usable
with other PEP 517 build backends).

When this PEP is implemented, it will not be necessary anymore to use
editable installs for the purpose of making pip freeze work correctly
with VCS references.

Rationale

This PEP specifies a new direct_url.json metadata file in the .dist-info
directory of an installed distribution.

The fields specified are sufficient to reproduce the source archive and
VCS URLs supported by pip. They are also sufficient to reproduce
PEP440 Direct References <440#direct-references>, as well as Pipfile and
Pipfile.lock entries. Finally, they are sufficient to record the branch,
tag, and/or Git ref origin of the installed version that is already
available for editable installs by virtue of a VCS checkout being
present.

Since at least three different ways already exist to encode this type of
information, this PEP uses a dictionary format, so as not to make any
assumption on how a direct URL reference must ultimately be encoded in a
requirement or lockfile. See also the Alternatives section below for
more discussion about this choice.

Information has been taken from Ruby's bundler manual to verify it has
similar capabilities and inform the selection and naming of fields in
this specifications.

The JSON format allows for the addition of additional fields in the
future.

Specification

This PEP specifies a direct_url.json file in the .dist-info directory of
an installed distribution, to record the Direct URL Origin of the
distribution.

The canonical source for the name and semantics of this metadata file is
the Recording the Direct URL Origin of installed distributions document.

This file MUST be created by installers when installing a distribution
from a requirement specifying a direct URL reference (including a VCS
URL).

This file MUST NOT be created when installing a distribution from an
other type of requirement (i.e. name plus version specifier).

This JSON file MUST be a dictionary, compliant with 8259 and UTF-8
encoded.

If present, it MUST contain at least two fields. The first one is url,
with type string. Depending on what url refers to, the second field MUST
be one of vcs_info (if url is a VCS reference), archive_info (if url is
a source archives or a wheel), or dir_info (if url is a local
directory). These info fields have a (possibly empty) subdictionary as
value, with the possible keys defined below.

url MUST be stripped of any sensitive authentication information, for
security reasons.

The user:password section of the URL MAY however be composed of
environment variables, matching the following regular expression:

    \$\{[A-Za-z0-9-_]+\}(:\$\{[A-Za-z0-9-_]+\})?

Additionally, the user:password section of the URL MAY be a well-known,
non security sensitive string. A typical example is git in the case of
an URL such as<EMAIL_ADDRESS>
When url refers to a VCS repository, the vcs_info key MUST be present as
a dictionary with the following keys:

-   A vcs key (type string) MUST be present, containing the name of the
    VCS (i.e. one of git, hg, bzr, svn). Other VCS's SHOULD be
    registered by writing a PEP to amend this specification. The url
    value MUST be compatible with the corresponding VCS, so an installer
    can hand it off without transformation to a checkout/download
    command of the VCS.
-   A requested_revision key (type string) MAY be present naming a
    branch/tag/ref/commit/revision/etc (in a format compatible with the
    VCS) to install.
-   A commit_id key (type string) MUST be present, containing the exact
    commit/revision number that was installed. If the VCS supports
    commit-hash based revision identifiers, such commit-hash MUST be
    used as commit_id in order to reference the immutable version of the
    source code that was installed.
-   If the installer could discover additional information about the
    requested revision, it MAY add a resolved_revision and/or
    resolved_revision_type field. If no revision was provided in the
    requested URL, resolved_revision MAY contain the default branch that
    was installed, and resolved_revision_type will be branch. If the
    installer determines that requested_revision was a tag, it MAY add
    resolved_revision_type with value tag.

When url refers to a source archive or a wheel, the archive_info key
MUST be present as a dictionary with the following key:

-   A hash key (type string) SHOULD be present, with value
    <hash-algorithm>=<expected-hash>. It is RECOMMENDED that only hashes
    which are unconditionally provided by the latest version of the
    standard library's hashlib module be used for source archive hashes.
    At time of writing, that list consists of 'md5', 'sha1', 'sha224',
    'sha256', 'sha384', and 'sha512'.

When url refers to a local directory, the dir_info key MUST be present
as a dictionary with the following key:

-   editable (type: boolean): true if the distribution was installed in
    editable mode, false otherwise. If absent, default to false.

When url refers to a local directory, it MUST have the file scheme and
be compliant with 8089. In particular, the path component must be
absolute. Symbolic links SHOULD be preserved when making relative paths
absolute.

Note

When the requested URL has the file:// scheme and points to a local
directory that happens to contain a VCS checkout, installers MUST NOT
attempt to infer any VCS information and therefore MUST NOT output any
VCS related information (such as vcs_info) in direct_url.json.

A top-level subdirectory field MAY be present containing a directory
path, relative to the root of the VCS repository, source archive or
local directory, to specify where pyproject.toml or setup.py is located.

Note

As a general rule, installers should as much as possible preserve the
information that was provided in the requested URL when generating
direct_url.json. For example, user:password environment variables should
be preserved and requested_revision should reflect the revision that was
provided in the requested URL as faithfully as possible. This
information is however enriched with more precise data, such as
commit_id.

Registered VCS

This section lists the registered VCS's; expanded, VCS-specific
information on how to use the vcs, requested_revision, and other fields
of vcs_info; and in some cases additional VCS-specific fields. Tools MAY
support other VCS's although it is RECOMMENDED to register them by
writing a PEP to amend this specification. The vcs field SHOULD be the
command name (lowercased). Additional fields that would be necessary to
support such VCS SHOULD be prefixed with the VCS command name.

Git

Home page

  https://git-scm.com/

vcs command

  git

vcs field

  git

requested_revision field

  A tag name, branch name, Git ref, commit hash, shortened commit hash,
  or other commit-ish.

commit_id field

  A commit hash (40 hexadecimal characters sha1).

Note

Installers can use the git show-ref and git symbolic-ref commands to
determine if the requested_revision corresponds to a Git ref. In turn, a
ref beginning with refs/tags/ corresponds to a tag, and a ref beginning
with refs/remotes/origin/ after cloning corresponds to a branch.

Mercurial

Home page

  https://www.mercurial-scm.org/

vcs command

  hg

vcs field

  hg

requested_revision field

  A tag name, branch name, changeset ID, shortened changeset ID.

commit_id field

  A changeset ID (40 hexadecimal characters).

Bazaar

Home page

  https://bazaar.canonical.com/

vcs command

  bzr

vcs field

  bzr

requested_revision field

  A tag name, branch name, revision id.

commit_id field

  A revision id.

Subversion

Home page

  https://subversion.apache.org/

vcs command

  svn

vcs field

  svn

requested_revision field

  requested_revision must be compatible with svn checkout --revision
  option. In Subversion, branch or tag is part of url.

commit_id field

  Since Subversion does not support globally unique identifiers, this
  field is the Subversion revision number in the corresponding
  repository.

Examples

Example direct_url.json

Source archive:

    {
        "url": "https://github.com/pypa/pip/archive/1.3.1.zip",
        "archive_info": {
            "hash": "sha256=2dc6b5a470a1bde68946f263f1af1515a2574a150a30d6ce02c6ff742fcc0db8"
        }
    }

Git URL with tag and commit-hash:

    {
        "url": "https://github.com/pypa/pip.git",
        "vcs_info": {
            "vcs": "git",
            "requested_revision": "1.3.1",
            "resolved_revision_type": "tag",
            "commit_id": "7921be1537eac1e97bc40179a57f0349c2aee67d"
        }
    }

Local directory:

    {
        "url": "file:///home/user/project",
        "dir_info": {}
    }

Local directory installed in editable mode:

    {
        "url": "file:///home/user/project",
        "dir_info": {
            "editable": true
        }
    }

Example pip commands and their effect on direct_url.json

Commands that generate a direct_url.json:

-   pip install https://example.com/app-1.0.tgz
-   pip install https://example.com/app-1.0.whl
-   pip install
    "git+https://example.com/repo/app.git#egg=app&subdirectory=setup"
-   pip install ./app
-   pip install file:///home/user/app
-   pip install --editable
    "git+https://example.com/repo/app.git#egg=app&subdirectory=setup"
    (in which case, url will be the local directory where the git
    repository has been cloned to, and dir_info will be present with
    "editable": true and no vcs_info will be set)
-   pip install -e ./app

Commands that do not generate a direct_url.json

-   pip install app
-   pip install app --no-index --find-links https://example.com/

Use cases

"Freezing" an environment

  Tools, such as pip freeze, which generate requirements from the
  Database of Installed Python Distributions SHOULD exploit
  direct_url.json if it is present, and give it priority over the
  Version metadata in order to generate a higher fidelity output. In the
  presence of a vcs direct URL reference, the commit_id field SHOULD be
  used in priority in order to provide the highest possible fidelity to
  the originally installed version. If supported by their requirement
  format, tools are encouraged also to output the tag value if present,
  as it has immutable semantics. Tools MAY choose another approach,
  depending on the needs of their users.

  Note the initial iteration of this PEP does not attempt to make
  environments that include editable installs or installs from local
  directories reproducible, but it does attempt to make them readily
  identifiable. By locating the local project directory via the url and
  dir_info fields of this specification, tools can implement any
  strategy that fits their use cases.

Backwards Compatibility

Since this PEP specifies a new file in the .dist-info directory, there
are no backwards compatibility implications.

Alternatives

PEP 426 source_url

The now withdrawn PEP 426 specifies a source_url metadata entry. It is
also implemented in distlib.

It was intended for a slightly different purpose, for use in sdists.

This format lacks support for the subdirectory option of pip requirement
URLs. The same limitation is present in
PEP440 Direct References <440#direct-references>.

It also lacks explicit support for environment variables in the
user:password part of URLs.

The introduction of a key/value extensibility mechanism and support for
environment variables for user:password in PEP 440, would be necessary
for use in this PEP.

revision vs ref

The requested_revision key was retained over requested_ref as it is a
more generic term across various VCS and ref has a specific meaning for
git.

References

Acknowledgements

Various people helped make this PEP a reality. Paul F. Moore provided
the essence of the abstract. Alyssa Coghlan suggested the direct_url
name.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

authors:
Chris Jerdonek
Stéphane Bidoul


text:
PEP: 633 Title: Dependency specification in pyproject.toml using an
exploded TOML table Author: Laurie Opperman
<EMAIL_ADDRESS>Arun Babu Neelicattu
<EMAIL_ADDRESS>Sponsor: Brett Cannon<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/dependency-specification-in-pyproject-toml-using-an-exploded-toml-table/5123/
Status: Rejected Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 02-Sep-2020 Post-History: 02-Sep-2020 Resolution:
https://discuss.python.org/t/how-to-specify-dependencies-pep-508-strings-or-a-table-in-toml/5243/38

Rejection Notice

This PEP has been rejected in favour of PEP 631 due to its popularity,
consistency with the existing usage of PEP 508 strings, and
compatibility with existing packaging tool suites.

Abstract

This PEP specifies how to write a project's dependencies in a
pyproject.toml file for packaging-related tools to consume using the
fields defined in PEP 621, as an alternative to the PEP 508-based
approach defined in PEP 631.

Motivation

There are multiple benefits to using TOML tables and other data-types to
represent requirements rather than PEP 508 strings:

-   Easy initial validation via the TOML syntax.
-   Easy secondary validation using a schema, for example a JSON Schema.
-   Potential for users to guess the keys of given features, rather than
    memorising a syntax.
-   Users of multiple other popular languages may already be familiar
    with the TOML syntax.
-   TOML directly represents the same data structures as in JSON, and
    therefore a sub-set of Python literals, so users can understand the
    hierarchy and type of value

Rationale

Most of this is taken from discussions in the PEP 621 dependencies
topic. This has elements from Pipfile, Poetry, Dart's dependencies and
Rust's Cargo. A comparison document shows advantages and disadvantages
between this format and PEP 508-style specifiers.

In the specification of multiple requirements with the same distribution
name (where environment markers choose the appropriate dependency), the
chosen solution is similar to Poetry's, where an array of requirements
is allowed.

The direct-reference keys closely align with and utilise PEP 610 and PEP
440 as to reduce differences in the packaging ecosystem and rely on
previous work in specification.

Specification

As in PEP 621, if metadata is improperly specified then tools MUST raise
an error. The metadata MUST conform to the TOML specification.

To reduce confusion with this document being a specification for
specifying dependencies, the word "requirement" is used to mean a PEP
508 dependency specification.

The following tables are added to the project table specified in PEP
621.

dependencies

Format: table

The keys inside this table are the names of the required distribution.
The values can have one of the following types:

-   string: the requirement is defined only by a version requirement,
    with same specification as version in the requirement table, except
    allowing the empty string "" to place no restriction on the version.
-   table: a requirement table.
-   array: an array of requirement tables. It is an error to specify an
    empty array [] as a value.

Requirement table

The keys of the requirement table are as follows (all are optional):

-   version (string): a PEP 440 version specifier, which is a
    comma-delimited list of version specifier clauses. The string MUST
    be non-empty.
-   extras (array of strings): a list of PEP 508 extras declarations for
    the distribution. The list MUST be non-empty.
-   markers (string): a PEP 508 environment marker expression. The
    string MUST be non-empty.
-   url (string): the URL of the artifact to install and satisfy the
    requirement. Note that file:// is the prefix used for packages to be
    retrieved from the local filesystem.
-   git, hg, bzr or svn (string): the URL of a VCS repository (as
    specified in PEP 440) to clone, whose tree will be installed to
    satisfy the requirement. Further VCS keys will be added via
    amendments to PEP 610, however tools MAY opt to support other VCS's
    using their command-line command prior to the acceptance of the
    amendment.
-   revision (string): the identifier for a specific revision of the
    specified VCS repository to check-out before installation. Users
    MUST only provide this when one of git, hg, bzr, svn, or another VCS
    key is used to identify the distribution to install. Revision
    identifiers are suggested in PEP 610.

At most one of the following keys can be specified simultaneously, as
they logically conflict with each other in the requirement: version,
url, git, hg, bzr, svn, and any other VCS key.

An empty requirement table {} places no restriction on the requirement,
in addition to the empty string "".

Any keys provided which are not specified in this document MUST cause an
error in parsing.

optional-dependencies

Format: table

The keys inside this table are the names of an extra's required
distribution. The values can have one of the following types:

-   table: a requirement table.
-   array: an array of requirement tables.

These requirement tables have the same specification as above, with the
addition of the following required key:

-   for-extra (string): the name of the PEP 508 extra that this
    requirement is required for.

Reference implementation

Tools will need to convert this format to PEP 508 requirement strings.
Below is an example implementation of that conversion (assuming
validation is already performed):

    def convert_requirement_to_pep508(name, requirement):
        if isinstance(requirement, str):
            requirement = {"version": requirement}
        pep508 = name
        if "extras" in requirement:
            pep508 += " [" + ", ".join(requirement["extras"]) + "]"
        if "version" in requirement:
            pep508 += " " + requirement["version"]
        if "url" in requirement:
            pep508 += " @ " + requirement["url"]
        for vcs in ("git", "hg", "bzr", "svn"):
            if vcs in requirement:
                pep508 += " @ " + vcs + "+" + requirement[vcs]
                if "revision" in requirement:
                    pep508 += "@" + requirement["revision"]
        extra = None
        if "for-extra" in requirement:
            extra = requirement["for-extra"]
        if "markers" in requirement:
            markers = requirement["markers"]
            if extra:
                markers = "extra = '" + extra + "' and (" + markers + ")"
            pep508 += "; " + markers
        return pep508, extra

    def convert_requirements_to_pep508(dependencies):
        pep508s = []
        extras = set()
        for name, req in dependencies.items():
            if isinstance(req, list):
                for sub_req in req:
                    pep508, extra = convert_requirement_to_pep508(name, sub_req)
                    pep508s.append(pep508)
                    if extra:
                        extras.add(extra)
            else:
                pep508, extra = convert_requirement_to_pep508(name, req)
                pep508s.append(pep508)
                if extra:
                    extras.add(extra)
        return pep508s, extras

    def convert_project_requirements_to_pep508(project):
        reqs, _ = convert_requirements_to_pep508(project.get("dependencies", {}))
        optional_reqs, extras = convert_requirements_to_pep508(
            project.get("optional-dependencies", {})
        )
        reqs += optional_reqs
        return reqs, extras

JSON schema

For initial validation, a JSON-schema can be used. Not only does this
help tools have a consistent validation, but it allows code editors to
highlight validation errors as users are building the dependencies list.

    {
        "$id": "spam",
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "Project metadata",
        "type": "object",
        "definitions": {
            "requirementTable": {
                "title": "Full project dependency specification",
                "type": "object",
                "properties": {
                    "extras": {
                        "title": "Dependency extras",
                        "type": "array",
                        "items": {
                            "title": "Dependency extra",
                            "type": "string"
                        }
                    },
                    "markers": {
                        "title": "Dependency environment markers",
                        "type": "string"
                    }
                },
                "propertyNames": {
                    "enum": [
                        "extras",
                        "markers",
                        "version",
                        "url",
                        "git",
                        "hg",
                        "bzr",
                        "svn",
                        "for-extra"
                    ]
                },
                "oneOf": [
                    {
                        "title": "Version requirement",
                        "properties": {
                            "version": {
                                "title": "Version",
                                "type": "string"
                            }
                        }
                    },
                    {
                        "title": "URL requirement",
                        "properties": {
                            "url": {
                                "title": "URL",
                                "type": "string",
                                "format": "uri"
                            }
                        },
                        "required": [
                            "url"
                        ]
                    },
                    {
                        "title": "VCS requirement",
                        "properties": {
                            "revision": {
                                "title": "VCS repository revision",
                                "type": "string"
                            }
                        },
                        "oneOf": [
                            {
                                "title": "Git repository",
                                "properties": {
                                    "git": {
                                        "title": "Git URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "git"
                                ]
                            },
                            {
                                "title": "Mercurial repository",
                                "properties": {
                                    "hg": {
                                        "title": "Mercurial URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "hg"
                                ]
                            },
                            {
                                "title": "Bazaar repository",
                                "properties": {
                                    "bzr": {
                                        "title": "Bazaar URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "bzr"
                                ]
                            },
                            {
                                "title": "Subversion repository",
                                "properties": {
                                    "svn": {
                                        "title": "Subversion URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "svn"
                                ]
                            }
                        ]
                    }
                ]
            },
            "requirementVersion": {
                "title": "Version project dependency specification",
                "type": "string"
            },
            "requirement": {
                "title": "Project dependency specification",
                "oneOf": [
                    {
                        "$ref": "#/definitions/requirementVersion"
                    },
                    {
                        "$ref": "#/definitions/requirementTable"
                    },
                    {
                        "title": "Multiple specifications",
                        "type": "array",
                        "items": {
                            "$ref": "#/definitions/requirementTable"
                        },
                        "minLength": 1
                    }
                ]
            },
            "optionalRequirementTable": {
                "title": "Project optional dependency specification table",
                "allOf": [
                    {
                        "$ref": "#/definitions/requirementTable"
                    },
                    {
                        "properties": {
                            "for-extra": {
                                "title": "Dependency's extra",
                                "type": "string"
                            }
                        },
                        "required": [
                            "for-extra"
                        ]
                    }
                ]
            },
            "optionalRequirement": {
                "title": "Project optional dependency specification",
                "oneOf": [
                    {
                        "$ref": "#/definitions/optionalRequirementTable"
                    },
                    {
                        "title": "Multiple specifications",
                        "type": "array",
                        "items": {
                            "$ref": "#/definitions/optionalRequirementTable"
                        },
                        "minLength": 1
                    }
                ]
            }
        },
        "properties": {
            "dependencies": {
                "title": "Project dependencies",
                "type": "object",
                "additionalProperties": {
                    "$ref": "#/definitions/requirement"
                }
            },
            "optional-dependencies": {
                "title": "Project dependencies",
                "type": "object",
                "additionalProperties": {
                    "$ref": "#/definitions/optionalRequirement"
                }
            }
        }
    }

Examples

Full artificial example:

    [project.dependencies]
    flask = { }
    django = { }
    requests = { version = ">= 2.8.1, == 2.8.*", extras = ["security", "tests"], markers = "python_version < '2.7'" }
    pip = { url = "https://github.com/pypa/pip/archive/1.3.1.zip" }
    sphinx = { git =<EMAIL_ADDRESS>}
    numpy = "~=1.18"
    pytest = [
        { version = "<6", markers = "python_version < '3.5'" },
        { version = ">=6", markers = "python_version >= '3.5'" },
    ]

    [project.optional-dependencies]
    pytest-timout = { for-extra = "dev" }
    pytest-mock = [
        { version = "<6", markers = "python_version < '3.5'", for-extra = "dev" },
        { version = ">=6", markers = "python_version >= '3.5'", for-extra = "dev" },
    ]

In homage to PEP 631, the following is an equivalent dependencies
specification for docker-compose:

    [project.dependencies]
    cached-property = ">= 1.2.0, < 2"
    distro = ">= 1.2.0, < 2"
    docker = { extras = ["ssh"], version = ">= 4.2.2, < 5" }
    docopt = ">= 0.6.1, < 1"
    jsonschema = ">= 2.5.1, < 4"
    PyYAML = ">= 3.10, < 6"
    python-dotenv = ">= 0.13.0, < 1"
    requests = ">= 2.20.0, < 3"
    texttable = ">= 0.9.0, < 2"
    websocket-client = ">= 0.32.0, < 1"

    # Conditional
    "backports.shutil_get_terminal_size" = { version = "== 1.0.0", markers = "python_version < '3.3'" }
    "backports.ssl_match_hostname" = { version = ">= 3.5, < 4", markers = "python_version < '3.5'" }
    colorama = { version = ">= 0.4, < 1", markers = "sys_platform == 'win32'" }
    enum34 = { version = ">= 1.0.4, < 2", markers = "python_version < '3.4'" }
    ipaddress = { version = ">= 1.0.16, < 2", markers = "python_version < '3.3'" }
    subprocess32 = { version = ">= 3.5.4, < 4", markers = "python_version < '3.2'" }

    [project.optional-dependencies]
    PySocks = { version = ">= 1.5.6, != 1.5.7, < 2", for-extra = "socks" }
    ddt = { version = ">= 1.2.2, < 2", for-extra = "tests" }
    pytest = { version = "< 6", for-extra = "tests" }
    mock = { version = ">= 1.0.1, < 4", markers = "python_version < '3.4'", for-extra = "tests" }

Compatibility Examples

The authors of this PEP recognise that various tools need to both read
from and write to this format for dependency specification. This section
aims to provide direct comparison with and examples for translating
to/from the currently used standard, PEP 508.

Note

For simplicity and clarity, various ways in which TOML allows you to
specify each specification is not represented. These examples use the
standard inline representation.

For example, while following are considered equivalent in TOML, we
choose the second form for the examples in this section.

    aiohttp.version = "== 3.6.2"
    aiohttp = { version = "== 3.6.2" }

Version Constrained Dependencies

No Version Constraint

    aiohttp

    aiohttp = {}

Simple Version Constraint

    aiohttp >= 3.6.2, < 4.0.0

    aiohttp = { version = ">= 3.6.2, < 4.0.0" }

Note

This can, for conciseness, be also represented as a string.

    aiohttp = ">= 3.6.2, < 4.0.0"

Direct Reference Dependencies

URL Dependency

    aiohttp @ https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl

    aiohttp = { url = "https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl" }

VCS Dependency

    aiohttp @ git+ssh://git@github.com/aio-libs/aiohttp.git@master

    aiohttp = { git =<EMAIL_ADDRESS>revision = "master" }

Environment Markers

    aiohttp >= 3.6.1; python_version >= '3.8'

    aiohttp = { version = ">= 3.6.1", markers = "python_version >= '3.8'" }

A slightly extended example of the above, where a particular version of
aiohttp is required based on the interpreter version.

    aiohttp >= 3.6.1; python_version >= '3.8'
    aiohttp >= 3.0.0, < 3.6.1; python_version < '3.8'

    aiohttp = [
        { version = ">= 3.6.1", markers = "python_version >= '3.8'" },
        { version = ">= 3.0.0, < 3.6.1", markers = "python_version < '3.8'" }
    ]

Package Extras

Specifying dependency for a package extra

    aiohttp >= 3.6.2; extra == 'http'

    aiohttp = { version = ">= 3.6.2", for-extra = "http" }

Using extras from a dependency

    aiohttp [speedups] >= 3.6.2

    aiohttp = { version = ">= 3.6.2", extras = ["speedups"] }

Complex Examples

Version Constraint

    aiohttp [speedups] >= 3.6.2; python_version >= '3.8' and extra == 'http'

    aiohttp = { version = ">= 3.6.2", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }

Direct Reference (VCS)

    aiohttp [speedups] @ git+ssh://git@github.com/aio-libs/aiohttp.git@master ; python_version >= '3.8' and extra == 'http'

    aiohttp = { git =<EMAIL_ADDRESS>revision = "master", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }

Rejected Ideas

Switch to an array for dependencies

Use an array instead of a table in order to have each element only be a
table (with a name key) and no arrays of requirement tables. This was
very verbose and restrictive in the TOML format, and having multiple
requirements for a given distribution isn't very common.

Replace optional-dependencies with extras

Remove the optional-dependencies table in favour of both including an
optional key in the requirement and an extras table which specifies
which (optional) requirements are needed for a project's extra. This
reduces the number of table with the same specification (to 1) and
allows for requirements to be specified once but used in multiple
extras, but distances some of the requirement's properties (which
extra(s) it belongs to), groups required and optional dependencies
together (possibly mixed), and there may not be a simple way to choose a
requirement when a distribution has multiple requirements. This was
rejected as optional-dependencies has already been used in the PEP 621
draft.

direct table in requirement

Include the direct-reference keys in a direct table, have the VCS
specified as the value of a vcs key. This was more explicit and easier
to include in a JSON-schema validation, but was decided to be too
verbose and not as readable.

Include hash

Include hash in direct-reference requirements. This was only for package
lock-files, and didn't really have a place in the project's metadata.

Dependency tables for each extra

Have the optional-dependencies be a table of dependency tables for each
extra, with the table name being the extra's name. This made
optional-dependencies a different type (table of tables of requirements)
from dependencies (table of requirements), which could be jarring for
users and harder to parse.

Environment marker keys

Make each PEP 508 environment marker as a key (or child-table key) in
the requirement. This arguably increases readability and ease of
parsing. The markers key would still be allowed for more advanced
specification, with which the key-specified environment markers are
and'd with the result of. This was deferred as more design needs to be
undertaken.

Multiple extras which one requirement can satisfy

Replace the for-extra key with for-extras, with the value being an array
of extras which the requirement satisfies. This reduces some
duplication, but in this case that duplication makes explicit which
extras have which dependencies.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.