text:
PEP: 691 Title: JSON-based Simple API for Python Package Indexes Author:
Donald Stufft<EMAIL_ADDRESS>Pradyun Gedam<EMAIL_ADDRESS>Cooper Lees<EMAIL_ADDRESS>Dustin Ingram<EMAIL_ADDRESS>PEP-Delegate: Brett Cannon<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-691-json-based-simple-api-for-python-package-indexes/15553
Status: Accepted Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 04-May-2022 Post-History: 05-May-2022 Resolution:
https://discuss.python.org/t/pep-691-json-based-simple-api-for-python-package-indexes/15553/70

Abstract

The "Simple Repository API" that was defined in PEP 503 (and was in use
much longer than that) has served us reasonably well for a very long
time. However, the reliance on using HTML as the data exchange mechanism
has several shortcomings.

There are two major issues with an HTML-based API:

-   While HTML5 is a standard, it's an incredibly complex standard and
    ensuring completely correct parsing of it involves complex logic
    that does not currently exist within the Python standard library
    (nor the standard library of many other languages).

    This means that to actually accept everything that is technically
    valid, tools have to pull in large dependencies or they have to rely
    on the standard library's html.parser library, which is lighter
    weight but potentially doesn't fully support HTML5.

-   HTML5 is primarily designed as a markup language to present
    documents for human consumption. Our use of it is driven largely for
    historical and accidental reasons, and it's unlikely anyone would
    design an API that relied on it if they were starting from scratch.

    The primary issue with using a markup format designed for human
    consumption is that there's not a great way to actually encode data
    within HTML. We've gotten around this by limiting the data we put in
    this API and being creative with how we can cram data into the API
    (for instance, hashes are embedded as URL fragments, adding the
    data-yanked attribute in PEP 592).

PEP 503 was largely an attempt to standardize what was already in use,
so it did not propose any large changes to the API.

In the intervening years, we've regularly talked about an "API V2" that
would re-envision the entire API of PyPI. However, due to limited time
constraints, that effort has not gained much, if any, traction beyond
people thinking that it would be nice to do.

This PEP attempts to take a different route. It doesn't fundamentally
change the overall API structure, but instead specifies a new
serialization of the existing data contained in existing PEP 503
responses in a format that is easier for software to parse rather than
using a human centric document format.

Goals

-   Enable zero configuration discovery. Clients of the simple API MUST
    be able to gracefully determine whether a target repository supports
    this PEP without relying on any form of out of band communication
    (configuration, prior knowledge, etc). Individual clients MAY choose
    to require configuration to enable the use of this API, however.
-   Enable clients to drop support for "legacy" HTML parsing. While it
    is expected that most clients will keep supporting HTML-only
    repositories for a while, if not forever, it should be possible for
    a client to choose to support only the new API formats and no longer
    invoke an HTML parser.
-   Enable repositories to drop support for "legacy" HTML formats.
    Similar to clients, it is expected that most repositories will
    continue to support HTML responses for a long time, or forever. It
    should be possible for a repository to choose to only support the
    new formats.
-   Maintain full support for existing HTML-only clients. We MUST not
    break existing clients that are accessing the API as a strictly PEP
    503 API. The only exception to this, is if the repository itself has
    chosen to no longer support the HTML format.
-   Minimal additional HTTP requests. Using this API MUST not
    drastically increase the amount of HTTP requests an installer must
    do in order to function. Ideally it will require 0 additional
    requests, but if needed it may require one or two additional
    requests (total, not per dependency).
-   Minimal additional unique responses. Due to the nature of how large
    repositories like PyPI cache responses, this PEP should not
    introduce a significantly or combinatorially large number of
    additional unique responses that the repository may produce.
-   Supports TUF. This PEP MUST be able to function within the bounds of
    what TUF can support (PEP 458), and must be able to be secured using
    it.
-   Require only the standard library, or small external dependencies
    for clients. Parsing an API response should ideally require nothing
    but the standard library, however it would be acceptable to require
    a small, pure Python dependency.

Specification

To enable response parsing with only the standard library, this PEP
specifies that all responses (besides the files themselves, and the HTML
responses from PEP 503) should be serialized using JSON.

To enable zero configuration discovery and to minimize the amount of
additional HTTP requests, this PEP extends PEP 503 such that all of the
API endpoints (other than the files themselves) will utilize HTTP
content negotiation to allow client and server to select the correct
serialization format to serve, i.e. either HTML or JSON.

Versioning

Versioning will adhere to PEP 629 format (Major.Minor), which has
defined the existing HTML responses to be 1.0. Since this PEP does not
introduce new features into the API, rather it describes a different
serialization format for the existing features, this PEP does not change
the existing 1.0 version, and instead just describes how to serialize
that into JSON.

Similar to PEP 629, the major version number MUST be incremented if any
changes to the new format would result in no longer being able to expect
existing clients to meaningfully understand the format.

Likewise, the minor version MUST be incremented if features are added or
removed from the format, but existing clients would be expected to
continue to meaningfully understand the format.

Changes that would not result in existing clients being unable to
meaningfully understand the format and which do not represent features
being added or removed may occur without changing the version number.

This is intentionally vague, as this PEP believes it is best left up to
future PEPs that make any changes to the API to investigate and decide
whether or not that change should increment the major or minor version.

Future versions of the API may add things that can only be represented
in a subset of the available serializations of that version. All
serializations version numbers, within a major version, SHOULD be kept
in sync, but the specifics of how a feature serializes into each format
may differ, including whether or not that feature is present at all.

It is the intent of this PEP that the API should be thought of as URL
endpoints that return data, whose interpretation is defined by the
version of that data, and then serialized into the target serialization
format.

JSON Serialization

The URL structure from PEP 503 still applies, as this PEP only adds an
additional serialization format for the already existing API.

The following constraints apply to all JSON serialized responses
described in this PEP:

-   All JSON responses will always be a JSON object rather than an array
    or other type.
-   While JSON doesn't natively support an URL type, any value that
    represents an URL in this API may be either absolute or relative as
    long as they point to the correct location. If relative, they are
    relative to the current URL as if it were HTML.
-   Additional keys may be added to any dictionary objects in the API
    responses and clients MUST ignore keys that they don't understand.
-   All JSON responses will have a meta key, which contains information
    related to the response itself, rather than the content of the
    response.
-   All JSON responses will have a meta.api-version key, which will be a
    string that contains the PEP 629 Major.Minor version number, with
    the same fail/warn semantics as defined in PEP 629.
-   All requirements of PEP 503 that are not HTML specific still apply.

Project List

The root URL / for this PEP (which represents the base URL) will be a
JSON encoded dictionary which has a two keys:

-   projects: An array where each entry is a dictionary with a single
    key, name, which represents string of the project name.
-   meta: The general response metadata as described earlier.

As an example:

    {
      "meta": {
        "api-version": "1.0"
      },
      "projects": [
        {"name": "Frob"},
        {"name": "spamspamspam"}
      ]
    }

Note

The name field is the same as the one from PEP 503, which does not
specify whether it is the non-normalized display name or the normalized
name. In practice different implementations of these PEPs are choosing
differently here, so relying on it being either non-normalized or
normalized is relying on an implementation detail of the repository in
question.

Note

While the projects key is an array, and thus is required to be in some
kind of an order, neither PEP 503 nor this PEP requires any specific
ordering nor that the ordering is consistent from one request to the
next. Mentally this is best thought of as a set, but both JSON and HTML
lack the functionality to have sets.

Project Detail

The format of this URL is /<project>/ where the <project> is replaced by
the PEP 503 normalized name for that project, so a project named
"Silly_Walk" would have a URL like /silly-walk/.

This URL must respond with a JSON encoded dictionary that has three
keys:

-   name: The normalized name of the project.
-   files: A list of dictionaries, each one representing an individual
    file.
-   meta: The general response metadata as described earlier.

Each individual file dictionary has the following keys:

-   filename: The filename that is being represented.

-   url: The URL that the file can be fetched from.

-   hashes: A dictionary mapping a hash name to a hex encoded digest of
    the file. Multiple hashes can be included, and it is up to the
    client to decide what to do with multiple hashes (it may validate
    all of them or a subset of them, or nothing at all). These hash
    names SHOULD always be normalized to be lowercase.

    The hashes dictionary MUST be present, even if no hashes are
    available for the file, however it is HIGHLY recommended that at
    least one secure, guaranteed-to-be-available hash is always
    included.

    By default, any hash algorithm available via hashlib (specifically
    any that can be passed to hashlib.new() and do not require
    additional parameters) can be used as a key for the hashes
    dictionary. At least one secure algorithm from
    hashlib.algorithms_guaranteed SHOULD always be included. At the time
    of this PEP, sha256 specifically is recommended.

-   requires-python: An optional key that exposes the Requires-Python
    metadata field, specified in PEP 345. Where this is present,
    installer tools SHOULD ignore the download when installing to a
    Python version that doesn't satisfy the requirement.

    Unlike data-requires-python in PEP 503, the requires-python key does
    not require any special escaping other than anything JSON does
    naturally.

-   dist-info-metadata: An optional key that indicates that metadata for
    this file is available, via the same location as specified in PEP
    658 ({file_url}.metadata). Where this is present, it MUST be either
    a boolean to indicate if the file has an associated metadata file,
    or a dictionary mapping hash names to a hex encoded digest of the
    metadata's hash.

    When this is a dictionary of hashes instead of a boolean, then all
    the same requirements and recommendations as the hashes key hold
    true for this key as well.

    If this key is missing then the metadata file may or may not exist.
    If the key value is truthy, then the metadata file is present, and
    if it is falsey then it is not.

    It is recommended that servers make the hashes of the metadata file
    available if possible.

-   gpg-sig: An optional key that acts a boolean to indicate if the file
    has an associated GPG signature or not. The URL for the signature
    file follows what is specified in PEP 503 ({file_url}.asc). If this
    key does not exist, then the signature may or may not exist.

-   yanked: An optional key which may be either a boolean to indicate if
    the file has been yanked, or a non empty, but otherwise arbitrary,
    string to indicate that a file has been yanked with a specific
    reason. If the yanked key is present and is a truthy value, then it
    SHOULD be interpreted as indicating that the file pointed to by the
    url field has been "Yanked" as per PEP 592.

As an example:

    {
      "meta": {
        "api-version": "1.0"
      },
      "name": "holygrail",
      "files": [
        {
          "filename": "holygrail-1.0.tar.gz",
          "url": "https://example.com/files/holygrail-1.0.tar.gz",
          "hashes": {"sha256": "...", "blake2b": "..."},
          "requires-python": ">=3.7",
          "yanked": "Had a vulnerability"
        },
        {
          "filename": "holygrail-1.0-py3-none-any.whl",
          "url": "https://example.com/files/holygrail-1.0-py3-none-any.whl",
          "hashes": {"sha256": "...", "blake2b": "..."},
          "requires-python": ">=3.7",
          "dist-info-metadata": true
        }
      ]
    }

Note

While the files key is an array, and thus is required to be in some kind
of an order, neither PEP 503 nor this PEP requires any specific ordering
nor that the ordering is consistent from one request to the next.
Mentally this is best thought of as a set, but both JSON and HTML lack
the functionality to have sets.

Content-Types

This PEP proposes that all responses from the Simple API will have a
standard content type that describes what the response is (a Simple API
response), what version of the API it represents, and what serialization
format has been used.

The structure of this content type will be:

    application/vnd.pypi.simple.$version+format

Since only major versions should be disruptive to clients attempting to
understand one of these API responses, only the major version will be
included in the content type, and will be prefixed with a v to clarify
that it is a version number.

Which means that for the existing 1.0 API, the content types would be:

-   JSON: application/vnd.pypi.simple.v1+json
-   HTML: application/vnd.pypi.simple.v1+html

In addition to the above, a special "meta" version is supported named
latest, whose purpose is to allow clients to request the absolute latest
version, without having to know ahead of time what that version is. It
is recommended however, that clients be explicit about what versions
they support.

To support existing clients which expect the existing PEP 503 API
responses to use the text/html content type, this PEP further defines
text/html as an alias for the application/vnd.pypi.simple.v1+html
content type.

Version + Format Selection

Now that there is multiple possible serializations, we need a mechanism
to allow clients to indicate what serialization formats they're able to
understand. In addition, it would be beneficial if any possible new
major version to the API can be added without disrupting existing
clients expecting the previous API version.

To enable this, this PEP standardizes on the use of HTTP's Server-Driven
Content Negotiation.

While this PEP won't fully describe the entirety of server-driven
content negotiation, the flow is roughly:

1.  The client makes an HTTP request containing an Accept header listing
    all of the version+format content types that they are able to
    understand.
2.  The server inspects that header, selects one of the listed content
    types, then returns a response using that content type (treating the
    absence of an Accept header as Accept: */*).
3.  If the server does not support any of the content types in the
    Accept header then they are able to choose between 3 different
    options for how to respond:
    a.  Select a default content type other than what the client has
        requested and return a response with that.
    b.  Return a HTTP 406 Not Acceptable response to indicate that none
        of the requested content types were available, and the server
        was unable or unwilling to select a default content type to
        respond with.
    c.  Return a HTTP 300 Multiple Choices response that contains a list
        of all of the possible responses that could have been chosen.
4.  The client interprets the response, handling the different types of
    responses that the server may have responded with.

This PEP does not specify which choices the server makes in regards to
handling a content type that it isn't able to return, and clients SHOULD
be prepared to handle all of the possible responses in whatever way
makes the most sense for that client.

However, as there is no standard format for how a 300 Multiple Choices
response can be interpreted, this PEP highly discourages servers from
utilizing that option, as clients will have no way to understand and
select a different content-type to request. In addition, it's unlikely
that the client could understand a different content type anyways, so at
best this response would likely just be treated the same as a
406 Not Acceptable error.

This PEP does require that if the meta version latest is being used, the
server MUST respond with the content type for the actual version that is
contained in the response (i.e. A
Accept: application/vnd.pypi.simple.latest+json request that returns a
v1.x response should have a Content-Type of
application/vnd.pypi.simple.v1+json).

The Accept header is a comma separated list of content types that the
client understands and is able to process. It supports three different
formats for each content type that is being requested:

-   $type/$subtype
-   $type/*
-   */*

For the use of selecting a version+format, the most useful of these is
$type/$subtype, as that is the only way to actually specify the version
and format you want.

The order of the content types listed in the Accept header does not have
any specific meaning, and the server SHOULD consider all of them to be
equally valid to respond with. If a client wishes to specify that they
prefer a specific content type over another, they may use the Accept
header's quality value syntax.

This allows a client to specify a priority for a specific entry in their
Accept header, by appending a ;q= followed by a value between 0 and 1
inclusive, with up to 3 decimal digits. When interpreting this value, an
entry with a higher quality has priority over an entry with a lower
quality, and any entry without a quality present will default to a
quality of 1.

However, clients should keep in mind that a server is free to select any
of the content types they've asked for, regardless of their requested
priority, and it may even return a content type that they did not ask
for.

To aid clients in determining the content type of the response that they
have received from an API request, this PEP requires that servers always
include a Content-Type header indicating the content type of the
response. This is technically a backwards incompatible change, however
in practice pip has been enforcing this requirement so the risks for
actual breakages is low.

An example of how a client can operate would look like:

    import email.message
    import requests

    def parse_content_type(header: str) -> str:
        m = email.message.Message()
        m["content-type"] = header
        return m.get_content_type()

    # Construct our list of acceptable content types, we want to prefer
    # that we get a v1 response serialized using JSON, however we also
    # can support a v1 response serialized using HTML. For compatibility
    # we also request text/html, but we prefer it least of all since we
    # don't know if it's actually a Simple API response, or just some
    # random HTML page that we've gotten due to a misconfiguration.
    CONTENT_TYPES = [
        "application/vnd.pypi.simple.v1+json",
        "application/vnd.pypi.simple.v1+html;q=0.2",
        "text/html;q=0.01",  # For legacy compatibility
    ]
    ACCEPT = ", ".join(CONTENT_TYPES)

    # Actually make our request to the API, requesting all of the content
    # types that we find acceptable, and letting the server select one of
    # them out of the list.
    resp = requests.get("https://pypi.org/simple/", headers={"Accept": ACCEPT})

    # If the server does not support any of the content types you requested,
    # AND it has chosen to return a HTTP 406 error instead of a default
    # response then this will raise an exception for the 406 error.
    resp.raise_for_status()

    # Determine what kind of response we've gotten to ensure that it is one
    # that we can support, and if it is, dispatch to a function that will
    # understand how to interpret that particular version+serialization. If
    # we don't understand the content type we've gotten, then we'll raise
    # an exception.
    content_type = parse_content_type(resp.headers.get("content-type", ""))
    match content_type:
        case "application/vnd.pypi.simple.v1+json":
            handle_v1_json(resp)
        case "application/vnd.pypi.simple.v1+html" | "text/html":
            handle_v1_html(resp)
        case _:
            raise Exception(f"Unknown content type: {content_type}")

If a client wishes to only support HTML or only support JSON, then they
would just remove the content types that they do not want from the
Accept header, and turn receiving them into an error.

Alternative Negotiation Mechanisms

While using HTTP's Content negotiation is considered the standard way
for a client and server to coordinate to ensure that the client is
getting an HTTP response that it is able to understand, there are
situations where that mechanism may not be sufficient. For those cases
this PEP has alternative negotiation mechanisms that may optionally be
used instead.

URL Parameter

Servers that implement the Simple API may choose to support an URL
parameter named format to allow the clients to request a specific
version of the URL.

The value of the format parameter should be one of the valid content
types. Passing multiple content types, wild cards, quality values,
etc... is not supported.

Supporting this parameter is optional, and clients SHOULD NOT rely on it
for interacting with the API. This negotiation mechanism is intended to
allow for easier human based exploration of the API within a browser, or
to allow documentation or notes to link to a specific version+format.

Servers that do not support this parameter may choose to return an error
when it is present, or they may simple ignore its presence.

When a server does implement this parameter, it SHOULD take precedence
over any values in the client's Accept header, and if the server does
not support the requested format, it may choose to fall back to the
Accept header, or choose any of the error conditions that standard
server-driven content negotiation typically has (e.g. 406 Not Available,
303 Multiple Choices, or selecting a default type to return).

Endpoint Configuration

This option technically is not a special option at all, it is just a
natural consequence of using content negotiation and allowing servers to
select which of the available content types is their default.

If a server is unwilling or unable to implement the server-driven
content negotiation, and would instead rather require users to
explicitly configure their client to select the version they want, then
that is a supported configuration.

To enable this, a server should make multiple endpoints (for instance,
/simple/v1+html/ and/or /simple/v1+json/) for each version+format that
they wish to support. Under that endpoint, they can host a copy of their
repository that only supports one (or a subset) of the content-types.
When a client makes a request using the Accept header, the server can
ignore it and return the content type that corresponds to that endpoint.

For clients that wish to require specific configuration, they can keep
track of which version+format a specific repository URL was configured
for, and when making a request to that server, emit an Accept header
that only includes the correct content type.

TUF Support - PEP 458

PEP 458 requires that all API responses are hashable and that they can
be uniquely identified by a path relative to the repository root. For a
Simple API repository, the target path is the Root of our API (e.g.
/simple/ on PyPI). This creates challenges when accessing the API using
a TUF client instead of directly using a standard HTTP client, as the
TUF client cannot handle the fact that a target could have multiple
different representations that all hash differently.

PEP 458 does not specify what the target path should be for the Simple
API, but TUF requires that the target paths be "file-like", in other
words, a path like simple/PROJECT/ is not acceptable, because it
technically points to a directory.

The saving grace is that the target path does not have to actually match
the URL being fetched from the Simple API, and it can just be a sigil
that the fetching code knows how to transform into the actual URL that
needs to be fetched. This same thing can hold true for other aspects of
the actual HTTP request, such as the Accept header.

Ultimately figuring out how to map a directory to a filename is out of
scope for this PEP (but it would be in scope for PEP 458), and this PEP
defers making a decision about how exactly to represent this inside of
PEP 458 metadata.

However, it appears that the current WIP branch against pip that
attempts to implement PEP 458 is using a target path like
simple/PROJECT/index.html. This could be modified to include the API
version and serialization format using something like
simple/PROJECT/vnd.pypi.simple.vN.FORMAT. So the v1 HTML format would be
simple/PROJECT/vnd.pypi.simple.v1.html and the v1 JSON format would be
simple/PROJECT/vnd.pypi.simple.v1.json.

In this case, since text/html is an alias to
application/vnd.pypi.simple.v1+html when interacting through TUF, it
likely will make the most sense to normalize to the more explicit name.

Likewise the latest metaversion should not be included in the targets,
only explicitly declared versions should be supported.

Recommendations

This section is non-normative, and represents what the PEP authors
believe to be the best default implementation decisions for something
implementing this PEP, but it does not represent any sort of requirement
to match these decisions.

These decisions have been chosen to maximize the number of requests that
can be moved onto the newest version of an API, while maintaining the
greatest amount of compatibility. In addition, they've also tried to
make using the API provide guardrails that attempt to push clients into
making the best choices it can.

It is recommended that servers:

-   Support all 3 content types described in this PEP, using
    server-driven content negotiation, for as long as they reasonably
    can, or at least as long as they're receiving non trivial traffic
    that uses the HTML responses.
-   When encountering an Accept header that does not contain any content
    types that it knows how to work with, the server should not ever
    return a 300 Multiple Choice response, and instead return a
    406 Not Acceptable response.
    -   However, if choosing to use the endpoint configuration, you
        should prefer to return a 200 OK response in the expected
        content type for that endpoint.
-   When selecting an acceptable version, the server should choose the
    highest version that the client supports, with the most
    expressive/featureful serialization format, taking into account the
    specificity of the client requests as well as any quality priority
    values they have expressed, and it should only use the text/html
    content type as a last resort.

It is recommended that clients:

-   Support all 3 content types described in this PEP, using
    server-driven content negotiation, for as long as they reasonably
    can.

-   When constructing an Accept header, include all of the content types
    that you support.

    You should generally not include a quality priority value for your
    content types, unless you have implementation specific reasons that
    you want the server to take into account (for example, if you're
    using the standard library HTML parser and you're worried that there
    may be some kinds of HTML responses that you're unable to parse in
    some edge cases).

    The one exception to this recommendation is that it is recommended
    that you should include a ;q=0.01 value on the legacy text/html
    content type, unless it is the only content type that you are
    requesting.

-   Explicitly select what versions they are looking for, rather than
    using the latest meta version during normal operation.

-   Check the Content-Type of the response and ensure it matches
    something that you were expecting.

FAQ

Does this mean PyPI is planning to drop support for HTML/PEP 503?

No, PyPI has no plans at this time to drop support for PEP 503 or HTML
responses.

While this PEP does give repositories the flexibility to do that, that
largely exists to ensure that things like using the Endpoint
Configuration mechanism is able to work, and to ensure that clients do
not make any assumptions that would prevent, at some point in the
future, gracefully dropping support for HTML.

The existing HTML responses incur almost no maintenance burden on PyPI
and there is no pressing need to remove them. The only real benefit to
dropping them would be to reduce the number of items cached in our CDN.

If in the future PyPI does wish to drop support for them, doing so would
almost certainly be the topic of a PEP, or at a minimum a public, open,
discussion and would be informed by metrics showing any impact to end
users.

Why JSON instead of X format?

JSON parsers are widely available in most, if not every, language. A
JSON parser is also available in the Python standard library. It's not
the perfect format, but it's good enough.

Why not add X feature?

The general goal of this PEP is to change or add very little. We will
instead focus largely on translating the existing information contained
within our HTML responses into a sensible JSON representation. This will
include PEP 658 metadata required for packaging tooling.

The only real new capability that is added in this PEP is the ability to
have multiple hashes for a single file. That was done because the
current mechanism being limited to a single hash has made it painful in
the past to migrate hashes (md5 to sha256) and the cost of making the
hashes a dictionary and allowing multiple is pretty low.

The API was generally designed to allow further extension through adding
new keys, so if there's some new piece of data that an installer might
need, future PEPs can easily make that available.

Why include the filename when the URL has it already?

We could reduce the size of our responses by removing the filename key
and expecting clients to pull that information out of the URL.

Currently this PEP chooses not to do that, largely because PEP 503
explicitly required that the filename be available via the anchor tag of
the links, though that was largely because something had to be there.
It's not clear if repositories in the wild always have a filename as the
last part of the URL or if they're relying on the filename in the anchor
tag.

It also makes the responses slightly nicer to read for a human, as you
get a nice short unique identifier.

If we got reasonable confidence that mandating the filename is in the
URL, then we could drop this data and reduce the size of the JSON
response.

Why not break out other pieces of information from the filename?

Currently clients are expected to parse a number of pieces of
information from the filename such as project name, version, ABI tags,
etc. We could break these out and add them as keys to the file object.

This PEP has chosen not to do that because doing so would increase the
size of the API responses, and most clients are going to require the
ability to parse that information out of file names anyways regardless
of what the API does. Thus it makes sense to keep that functionality
inside of the clients.

Why Content Negotiation instead of multiple URLs?

Another reasonable way to implement this would be to duplicate the API
routes and include some marker in the URL itself for JSON. Such as
making the URLs be something like /simple/foo.json, /simple/_index.json,
etc.

This makes some things simpler like TUF integration and fully static
serving of a repository (since .json files can just be written out).

However, this is two pretty major issues:

-   Our current URL structure relies on the fact that there is an URL
    that represents the "root", / to serve the list of projects. If we
    want to have separate URLs for JSON and HTML, we would need to come
    up with some way to have two root URLs.

    Something like / being HTML and /_index.json being JSON, since
    _index isn't a valid project name could work. But / being HTML
    doesn't work great if a repository wants to remove support for HTML.

    Another option could be moving all of the existing HTML URLs under a
    namespace while making a new namespace for JSON. Since /<project>/
    was defined, we would have to make these namespaces not valid
    project names, so something like /_html/ and /_json/ could work,
    then just redirect the non namespaced URLs to whatever the "default"
    for that repository is (likely HTML, unless they've disabled HTML
    then JSON).

-   With separate URLs, there's no good way to support zero
    configuration discovery that a repository supports the JSON URLs
    without making additional HTTP requests to determine if the JSON URL
    exists or not.

    The most naive implementation of this would be to request the JSON
    URL and fall back to the HTML URL for every single request, but that
    would be horribly performant and violate the goal of minimal
    additional HTTP requests.

    The most likely implementation of this would be to make some sort of
    repository level configuration file that somehow indicates what is
    supported. We would have the same namespace problem as above, with
    the same solution, something like /_config.json or so could hold
    that data, and a client could first make an HTTP request to that,
    and if it exists pull it down and parse it to learn about the
    capabilities of this particular repository.

-   The use of Accept also allows us to add versioning into this field

All being said, it is the opinion of this PEP that those three issues
combined make using separate API routes a less desirable solution than
relying on content negotiation to select the most ideal representation
of the data.

Does this mean that static servers are no longer supported?

In short, no, static servers are still (almost) fully supported by this
PEP.

The specifics of how they are supported will depend on the static server
in question. For example:

-   S3: S3 fully supports custom content types, however it does not
    support any form of content negotiation. In order to have a server
    hosted on S3, you would have to use the "Endpoint configuration"
    style of negotiation, and users would have to configure their
    clients explicitly.
-   GitHub Pages: GitHub pages does not support custom content types, so
    the S3 solution is not currently workable, which means that only
    text/html repositories would function.
-   Apache: Apache fully supports server-driven content negotiation, and
    would just need to be configured to map the custom content types to
    specific extension.

Why not add an application/json alias like text/html?

This PEP believes that it is best for both clients and servers to be
explicit about the types of the API responses that are being used, and a
content type like application/json is the exact opposite of explicit.

The existence of the text/html alias exists as a compromise primarily to
ensure that existing consumers of the API continue to function as they
already do. There is no such expectation of existing clients using the
Simple API with a application/json content type.

In addition, application/json has no versioning in it, which means that
if there is ever a 2.x version of the Simple API, we will be forced to
make a decision. Should application/json preserve backwards
compatibility and continue to be an alias for
application/vnd.pypi.simple.v1+json, or should it be updated to be an
alias for application/vnd.pypi.simple.v2+json?

This problem doesn't exist for text/html, because the assumption is that
HTML will remain a legacy format, and will likely not gain any new
features, much less features that require breaking compatibility. So
having it be an alias for application/vnd.pypi.simple.v1+html is
effectively the same as having it be an alias for
application/vnd.pypi.simple.latest+html, since 1.x will likely be the
only HTML version to exist.

The largest benefit to adding the application/json content type is that
there do things that do not allow you to have custom content types, and
require you to select one of their preset content types. The main
example of this being GitHub Pages, which the lack of application/json
support in this PEP means that static repositories will no longer be
able to be hosted on GitHub Pages unless GitHub adds the
application/vnd.pypi.simple.v1+json content type.

This PEP believes that the benefits are not large enough to add that
content type alias at this time, and that its inclusion would likely be
a footgun waiting for unsuspecting people to accidentally pick it up.
Especially given that we can always add it in the future, but removing
things is a lot harder to do.

Why add a application/vnd.pypi.simple.v1+html?

The PEP expects the HTML version of the API to become legacy, so one
option it could take is not add the application/vnd.pypi.simple.v1+html
content type, and just use text/html for that.

This PEP has decided that adding the new content type is better overall,
since it makes even the legacy format more self describing and makes
them both more consistent with each other. Overall I think it's more
confusing if the +html version doesn't exist.

Why v1.0 and not v1.1 or v2.0?

This PEP is still wholly backwards compatible with clients that could
read the existing v1.0 API, can still continue to read the API after
these changes have been made. In PEP 629, the qualification for a major
version bump is:

  Incrementing the major version is used to signal a backwards
  incompatible change such that existing clients would no longer be
  expected to be able to meaningfully use the API.

The changes in this PEP do not meet that bar, nothing has changed in a
way that existing clients would no longer be expected to be able to
meaningfully use the API.

That means we should still be within the v1.x version line.

The question of whether we should be v1.1 or v1.0 is a more interesting
one, and there are a few ways of looking at it:

-   We've exposed new features to the API (the project name on the
    project page, multiple hashes), which is a sign that we should
    increment the minor version.
-   The new features exist wholly within the JSON serialization, which
    means that no client that currently is requesting the HTML 1.0 page,
    would ever see any of the new features anyways, so for them it is
    effectively still v1.0.
-   No major client has implemented support for PEP 629 yet, which means
    that the minor version numbering is largely academic at this point
    anyways, since it exists to let clients provide feedback to end
    users.

The second and third points above end up making the first point kind of
meaningless, and with that, it makes more sense to just call everything
v1.0 and be stricter about updating versions into the future.

Appendix 1: Survey of use cases to cover

This was done through a discussion between pip, PyPI, and bandersnarch
maintainers, who are the two first potential users for the new API. This
is how they use the Simple + JSON APIs today or how they currently plan
to use it:

-   pip:
    -   List of all files for a particular release
    -   Metadata of each individual artifact:
        -   was it yanked? (data-yanked)
        -   what's the python-requires? (data-python-requires)
        -   what's the hash of this file? (currently, hash in URL)
        -   Full metadata (data-dist-info-metadata)
        -   [Bonus] what are the declared dependencies, if available
            (list-of-strings, null if unavailable)?
-   bandersnatch - Only uses legacy JSON API + XMLRPC today:
    -   Generates Simple HTML rather than copying from PyPI
        -   Maybe this changes with the new API and we verbatim pull
            these API assets from PyPI
    -   List of all files for a particular release.
        -   Workout URL for release files to download
    -   Metadata of each individual artifact.
        -   Write out the JSON to mirror storage today (disk/S3)
            -   Required metadata used (via Package class):
                -   metadata["info"]
                -   metadata["last_serial"]
                -   metadata["releases"]
                    -   digests
                    -   URL
    -   XML-RPC calls (we'd love to deprecate - but we don't think
        should go in the Simple API)
        -   [Bonus] Get packages since serial X (or all)
            -   XML-RPC Call: changelog_since_serial
        -   [Bonus] Get all packages with serial
            -   XML-RPC Call: list_packages_with_serial

Appendix 2: Rough Underlying Data Models

These are not intended to perfectly match the server, client, or wire
formats. Rather, these are conceptual models, put to code to make them
more explicit as to the abstract models underlining the repository API
as it evolved through PEP 503, PEP 529, PEP 629, PEP 658, and now this
PEP, PEP 691.

The existing HTML, and the new JSON serialization of these models then
represent how these underlying conceptual models get mapped onto the
actual wire formats.

How servers or clients choose to model this data is out of scope for
this PEP.

    @dataclass
    class Meta:
        api_version: Literal["1.0"]

    @dataclass
    class Project:
        # Normalized or Non-Normalized Name
        name: str
        # Computed in JSON, Included in HTML
        url: str | None

    @dataclass
    class File:
        filename: str
        url: str
        # Limited to a len() of 1 in HTML
        hashes: dict[str, str]
        gpg_sig: bool | None
        requires_python: str | None

    @dataclass
    class PEP529File(File):
        yanked: bool | str

    @dataclass
    class PEP658File(PEP529File):
        # Limited to a len() of 1 in HTML
        dist_info_metadata: bool | dict[str, str]

    # Simple Index page (/simple/)
    @dataclass
    class PEP503_Index:
        projects: set[Project]

    @dataclass
    class PEP629_Index(PEP503_Index):
        meta: Meta

    @dataclass
    class Index(PEP629_Index):
        pass

    # Simple Detail page (/simple/$PROJECT/)
    @dataclass
    class PEP503_Detail:
        files: set[File]

    @dataclass
    class PEP529_Detail(PEP503_Detail):
        files: set[PEP529File]

    @dataclass
    class PEP629_Detail(PEP529_Detail):
        meta: Meta

    @dataclass
    class PEP658_Detail(PEP629_Detail):
      files: set[PEP658File]

    @dataclass
    class PEP691_Detail(PEP658_Detail):
        name: str  # Normalized Name

    @dataclass
    class Detail(PEP691_Detail):
        pass

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 584 Title: Add Union Operators To dict Version: $Revision$
Last-Modified: $Date$ Author: Steven D'Aprano<EMAIL_ADDRESS>Brandt Bucher<EMAIL_ADDRESS>BDFL-Delegate: Guido van Rossum
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 01-Mar-2019 Python-Version: 3.9 Post-History:
01-Mar-2019, 16-Oct-2019, 02-Dec-2019, 04-Feb-2020, 17-Feb-2020
Resolution:
<EMAIL_ADDRESS>
Abstract

This PEP proposes adding merge (|) and update (|=) operators to the
built-in dict class.

Note

After this PEP was accepted, the decision was made to also implement the
new operators for several other standard library mappings.

Motivation

The current ways to merge two dicts have several disadvantages:

dict.update

d1.update(d2) modifies d1 in-place. e = d1.copy(); e.update(d2) is not
an expression and needs a temporary variable.

{**d1, **d2}

Dict unpacking looks ugly and is not easily discoverable. Few people
would be able to guess what it means the first time they see it, or
think of it as the "obvious way" to merge two dicts.

As Guido said:

  I'm sorry for PEP 448, but even if you know about **d in simpler
  contexts, if you were to ask a typical Python user how to combine two
  dicts into a new one, I doubt many people would think of {**d1, **d2}.
  I know I myself had forgotten about it when this thread started!

{**d1, **d2} ignores the types of the mappings and always returns a
dict. type(d1)({**d1, **d2}) fails for dict subclasses such as
defaultdict that have an incompatible __init__ method.

collections.ChainMap

ChainMap is unfortunately poorly-known and doesn't qualify as "obvious".
It also resolves duplicate keys in the opposite order to that expected
("first seen wins" instead of "last seen wins"). Like dict unpacking, it
is tricky to get it to honor the desired subclass. For the same reason,
type(d1)(ChainMap(d2, d1)) fails for some subclasses of dict.

Further, ChainMaps wrap their underlying dicts, so writes to the
ChainMap will modify the original dict:

    >>> d1 = {'spam': 1}
    >>> d2 = {'eggs': 2}
    >>> merged = ChainMap(d2, d1)
    >>> merged['eggs'] = 999
    >>> d2
    {'eggs': 999}

dict(d1, **d2)

This "neat trick" is not well-known, and only works when d2 is entirely
string-keyed:

    >>> d1 = {"spam": 1}
    >>> d2 = {3665: 2}
    >>> dict(d1, **d2)
    Traceback (most recent call last):
      ...
    TypeError: keywords must be strings

Rationale

The new operators will have the same relationship to the dict.update
method as the list concatenate (+) and extend (+=) operators have to
list.extend. Note that this is somewhat different from the relationship
that |/|= have with set.update; the authors have determined that
allowing the in-place operator to accept a wider range of types (as list
does) is a more useful design, and that restricting the types of the
binary operator's operands (again, as list does) will help avoid silent
errors caused by complicated implicit type casting on both sides.

Key conflicts will be resolved by keeping the rightmost value. This
matches the existing behavior of similar dict operations, where the last
seen value always wins:

    {'a': 1, 'a': 2}
    {**d, **e}
    d.update(e)
    d[k] = v
    {k: v for x in (d, e) for (k, v) in x.items()}

All of the above follow the same rule. This PEP takes the position that
this behavior is simple, obvious, usually the behavior we want, and
should be the default behavior for dicts. This means that dict union is
not commutative; in general d | e != e | d.

Similarly, the iteration order of the key-value pairs in the dictionary
will follow the same semantics as the examples above, with each newly
added key (and its value) being appended to the current sequence.

Specification

Dict union will return a new dict consisting of the left operand merged
with the right operand, each of which must be a dict (or an instance of
a dict subclass). If a key appears in both operands, the last-seen value
(i.e. that from the right-hand operand) wins:

    >>> d = {'spam': 1, 'eggs': 2, 'cheese': 3}
    >>> e = {'cheese': 'cheddar', 'aardvark': 'Ethel'}
    >>> d | e
    {'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}
    >>> e | d
    {'cheese': 3, 'aardvark': 'Ethel', 'spam': 1, 'eggs': 2}

The augmented assignment version operates in-place:

    >>> d |= e
    >>> d
    {'spam': 1, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}

Augmented assignment behaves identically to the update method called
with a single positional argument, so it also accepts anything
implementing the Mapping protocol (more specifically, anything with the
keys and __getitem__ methods) or iterables of key-value pairs. This is
analogous to list += and list.extend, which accept any iterable, not
just lists. Continued from above:

    >>> d | [('spam', 999)]
    Traceback (most recent call last):
      ...
    TypeError: can only merge dict (not "list") to dict

    >>> d |= [('spam', 999)]
    >>> d
    {'spam': 999, 'eggs': 2, 'cheese': 'cheddar', 'aardvark': 'Ethel'}

When new keys are added, their order matches their order within the
right-hand mapping, if any exists for its type.

Reference Implementation

One of the authors has written a C implementation.

An approximate pure-Python implementation is:

    def __or__(self, other):
        if not isinstance(other, dict):
            return NotImplemented
        new = dict(self)
        new.update(other)
        return new

    def __ror__(self, other):
        if not isinstance(other, dict):
            return NotImplemented
        new = dict(other)
        new.update(self)
        return new

    def __ior__(self, other):
        dict.update(self, other)
        return self

Major Objections

Dict Union Is Not Commutative

Union is commutative, but dict union will not be (d | e != e | d).

Response

There is precedent for non-commutative unions in Python:

    >>> {0} | {False}
    {0}
    >>> {False} | {0}
    {False}

While the results may be equal, they are distinctly different. In
general, a | b is not the same operation as b | a.

Dict Union Will Be Inefficient

Giving a pipe operator to mappings is an invitation to writing code that
doesn't scale well. Repeated dict union is inefficient:
d | e | f | g | h creates and destroys three temporary mappings.

Response

The same argument applies to sequence concatenation.

Sequence concatenation grows with the total number of items in the
sequences, leading to O(N**2) (quadratic) performance. Dict union is
likely to involve duplicate keys, so the temporary mappings will not
grow as fast.

Just as it is rare for people to concatenate large numbers of lists or
tuples, the authors of this PEP believe that it will be rare for people
to merge large numbers of dicts. collections.Counter is a dict subclass
that supports many operators, and there are no known examples of people
having performance issues due to combining large numbers of Counters.
Further, a survey of the standard library by the authors found no
examples of merging more than two dicts, so this is unlikely to be a
performance problem in practice... "Everything is fast for small enough
N".

If one expects to be merging a large number of dicts where performance
is an issue, it may be better to use an explicit loop and in-place
merging:

    new = {}
    for d in many_dicts:
        new |= d

Dict Union Is Lossy

Dict union can lose data (values may disappear); no other form of union
is lossy.

Response

It isn't clear why the first part of this argument is a problem.
dict.update() may throw away values, but not keys; that is expected
behavior, and will remain expected behavior regardless of whether it is
spelled as update() or |.

Other types of union are also lossy, in the sense of not being
reversible; you cannot get back the two operands given only the union.
a | b == 365... what are a and b?

Only One Way To Do It

Dict union will violate the Only One Way koan from the Zen.

Response

There is no such koan. "Only One Way" is a calumny about Python
originating long ago from the Perl community.

More Than One Way To Do It

Okay, the Zen doesn't say that there should be Only One Way To Do It.
But it does have a prohibition against allowing "more than one way to do
it".

Response

There is no such prohibition. The "Zen of Python" merely expresses a
preference for "only one obvious way":

    There should be one-- and preferably only one --obvious way to do
    it.

The emphasis here is that there should be an obvious way to do "it". In
the case of dict update operations, there are at least two different
operations that we might wish to do:

-   Update a dict in place: The Obvious Way is to use the update()
    method. If this proposal is accepted, the |= augmented assignment
    operator will also work, but that is a side-effect of how augmented
    assignments are defined. Which you choose is a matter of taste.
-   Merge two existing dicts into a third, new dict: This PEP proposes
    that the Obvious Way is to use the | merge operator.

In practice, this preference for "only one way" is frequently violated
in Python. For example, every for loop could be re-written as a while
loop; every if block could be written as an if/ else block. List, set
and dict comprehensions could all be replaced by generator expressions.
Lists offer no fewer than five ways to implement concatenation:

-   Concatenation operator: a + b
-   In-place concatenation operator: a += b
-   Slice assignment: a[len(a):] = b
-   Sequence unpacking: [*a, *b]
-   Extend method: a.extend(b)

We should not be too strict about rejecting useful functionality because
it violates "only one way".

Dict Union Makes Code Harder To Understand

Dict union makes it harder to tell what code means. To paraphrase the
objection rather than quote anyone in specific: "If I see spam | eggs, I
can't tell what it does unless I know what spam and eggs are".

Response

This is very true. But it is equally true today, where the use of the |
operator could mean any of:

-   int/bool bitwise-or
-   set/frozenset union
-   any other overloaded operation

Adding dict union to the set of possibilities doesn't seem to make it
harder to understand the code. No more work is required to determine
that spam and eggs are mappings than it would take to determine that
they are sets, or integers. And good naming conventions will help:

    flags |= WRITEABLE  # Probably numeric bitwise-or.
    DO_NOT_RUN = WEEKENDS | HOLIDAYS  # Probably set union.
    settings = DEFAULT_SETTINGS | user_settings | workspace_settings  # Probably dict union.

What About The Full set API?

dicts are "set like", and should support the full collection of set
operators: |, &, ^, and -.

Response

This PEP does not take a position on whether dicts should support the
full collection of set operators, and would prefer to leave that for a
later PEP (one of the authors is interested in drafting such a PEP). For
the benefit of any later PEP, a brief summary follows.

Set symmetric difference (^) is obvious and natural. For example, given
two dicts:

    d1 = {"spam": 1, "eggs": 2}
    d2 = {"ham": 3, "eggs": 4}

the symmetric difference d1 ^ d2 would be {"spam": 1, "ham": 3}.

Set difference (-) is also obvious and natural, and an earlier version
of this PEP included it in the proposal. Given the dicts above, we would
have d1 - d2 be {"spam": 1} and d2 - d1 be {"ham": 3}.

Set intersection (&) is a bit more problematic. While it is easy to
determine the intersection of keys in two dicts, it is not clear what to
do with the values. Given the two dicts above, it is obvious that the
only key of d1 & d2 must be "eggs". "Last seen wins", however, has the
advantage of consistency with other dict operations (and the proposed
union operators).

What About Mapping And MutableMapping?

collections.abc.Mapping and collections.abc.MutableMapping should define
| and |=, so subclasses could just inherit the new operators instead of
having to define them.

Response

There are two primary reasons why adding the new operators to these
classes would be problematic:

-   Currently, neither defines a copy method, which would be necessary
    for | to create a new instance.
-   Adding |= to MutableMapping (or a copy method to Mapping) would
    create compatibility issues for virtual subclasses.

Rejected Ideas

Rejected Semantics

There were at least four other proposed solutions for handling
conflicting keys. These alternatives are left to subclasses of dict.

Raise

It isn't clear that this behavior has many use-cases or will be often
useful, but it will likely be annoying as any use of the dict union
operator would have to be guarded with a try/except clause.

Add The Values (As Counter Does, with +)

Too specialised to be used as the default behavior.

Leftmost Value (First-Seen) Wins

It isn't clear that this behavior has many use-cases. In fact, one can
simply reverse the order of the arguments:

    d2 | d1  # d1 merged with d2, keeping existing values in d1

Concatenate Values In A List

This is likely to be too specialised to be the default. It is not clear
what to do if the values are already lists:

    {'a': [1, 2]} | {'a': [3, 4]}

Should this give {'a': [1, 2, 3, 4]} or {'a': [[1, 2], [3, 4]]}?

Rejected Alternatives

Use The Addition Operator

This PEP originally started life as a proposal for dict addition, using
the + and += operator. That choice proved to be exceedingly
controversial, with many people having serious objections to the choice
of operator. For details, see previous versions of the PEP and the
mailing list discussions.

Use The Left Shift Operator

The << operator didn't seem to get much support on Python-Ideas, but no
major objections either. Perhaps the strongest objection was Chris
Angelico's comment

  The "cuteness" value of abusing the operator to indicate information
  flow got old shortly after C++ did it.

Use A New Left Arrow Operator

Another suggestion was to create a new operator <-. Unfortunately this
would be ambiguous, d <- e could mean d merge e or d less-than minus e.

Use A Method

A dict.merged() method would avoid the need for an operator at all. One
subtlety is that it would likely need slightly different implementations
when called as an unbound method versus as a bound method.

As an unbound method, the behavior could be similar to:

    def merged(cls, *mappings, **kw):
        new = cls()  # Will this work for defaultdict?
        for m in mappings:
            new.update(m)
        new.update(kw)
        return new

As a bound method, the behavior could be similar to:

    def merged(self, *mappings, **kw):
        new = self.copy()
        for m in mappings:
            new.update(m)
        new.update(kw)
        return new

Advantages

-   Arguably, methods are more discoverable than operators.
-   The method could accept any number of positional and keyword
    arguments, avoiding the inefficiency of creating temporary dicts.
-   Accepts sequences of (key, value) pairs like the update method.
-   Being a method, it is easy to override in a subclass if you need
    alternative behaviors such as "first wins", "unique keys", etc.

Disadvantages

-   Would likely require a new kind of method decorator which combined
    the behavior of regular instance methods and classmethod. It would
    need to be public (but not necessarily a builtin) for those needing
    to override the method. There is a proof of concept.
-   It isn't an operator. Guido discusses why operators are useful. For
    another viewpoint, see Alyssa Coghlan's blog post.

Use a Function

Instead of a method, use a new built-in function merged(). One possible
implementation could be something like this:

    def merged(*mappings, **kw):
        if mappings and isinstance(mappings[0], dict):
            # If the first argument is a dict, use its type.
            new = mappings[0].copy()
            mappings = mappings[1:]
        else:
            # No positional arguments, or the first argument is a
            # sequence of (key, value) pairs.
            new = dict()
        for m in mappings:
            new.update(m)
        new.update(kw)
        return new

An alternative might be to forgo the arbitrary keywords, and take a
single keyword parameter that specifies the behavior on collisions:

    def merged(*mappings, on_collision=lambda k, v1, v2: v2):
        # implementation left as an exercise to the reader

Advantages

-   Most of the same advantages of the method solutions above.
-   Doesn't require a subclass to implement alternative behavior on
    collisions, just a function.

Disadvantages

-   May not be important enough to be a builtin.
-   Hard to override behavior if you need something like "first wins",
    without losing the ability to process arbitrary keyword arguments.

Examples

The authors of this PEP did a survey of third party libraries for
dictionary merging which might be candidates for dict union.

This is a cursory list based on a subset of whatever arbitrary
third-party packages happened to be installed on one of the authors'
computers, and may not reflect the current state of any package. Also
note that, while further (unrelated) refactoring may be possible, the
rewritten version only adds usage of the new operators for an
apples-to-apples comparison. It also reduces the result to an expression
when it is efficient to do so.

IPython/zmq/ipkernel.py

Before:

    aliases = dict(kernel_aliases)
    aliases.update(shell_aliases)

After:

    aliases = kernel_aliases | shell_aliases

IPython/zmq/kernelapp.py

Before:

    kernel_aliases = dict(base_aliases)
    kernel_aliases.update({
        'ip' : 'KernelApp.ip',
        'hb' : 'KernelApp.hb_port',
        'shell' : 'KernelApp.shell_port',
        'iopub' : 'KernelApp.iopub_port',
        'stdin' : 'KernelApp.stdin_port',
        'parent': 'KernelApp.parent',
    })
    if sys.platform.startswith('win'):
        kernel_aliases['interrupt'] = 'KernelApp.interrupt'

    kernel_flags = dict(base_flags)
    kernel_flags.update({
        'no-stdout' : (
                {'KernelApp' : {'no_stdout' : True}},
                "redirect stdout to the null device"),
        'no-stderr' : (
                {'KernelApp' : {'no_stderr' : True}},
                "redirect stderr to the null device"),
    })

After:

    kernel_aliases = base_aliases | {
        'ip' : 'KernelApp.ip',
        'hb' : 'KernelApp.hb_port',
        'shell' : 'KernelApp.shell_port',
        'iopub' : 'KernelApp.iopub_port',
        'stdin' : 'KernelApp.stdin_port',
        'parent': 'KernelApp.parent',
    }
    if sys.platform.startswith('win'):
        kernel_aliases['interrupt'] = 'KernelApp.interrupt'

    kernel_flags = base_flags | {
        'no-stdout' : (
                {'KernelApp' : {'no_stdout' : True}},
                "redirect stdout to the null device"),
        'no-stderr' : (
                {'KernelApp' : {'no_stderr' : True}},
                "redirect stderr to the null device"),
    }

matplotlib/backends/backend_svg.py

Before:

    attrib = attrib.copy()
    attrib.update(extra)
    attrib = attrib.items()

After:

    attrib = (attrib | extra).items()

matplotlib/delaunay/triangulate.py

Before:

    edges = {}
    edges.update(dict(zip(self.triangle_nodes[border[:,0]][:,1],
                 self.triangle_nodes[border[:,0]][:,2])))
    edges.update(dict(zip(self.triangle_nodes[border[:,1]][:,2],
                 self.triangle_nodes[border[:,1]][:,0])))
    edges.update(dict(zip(self.triangle_nodes[border[:,2]][:,0],
                 self.triangle_nodes[border[:,2]][:,1])))

Rewrite as:

    edges = {}
    edges |= zip(self.triangle_nodes[border[:,0]][:,1],
                 self.triangle_nodes[border[:,0]][:,2])
    edges |= zip(self.triangle_nodes[border[:,1]][:,2],
                 self.triangle_nodes[border[:,1]][:,0])
    edges |= zip(self.triangle_nodes[border[:,2]][:,0],
                 self.triangle_nodes[border[:,2]][:,1])

matplotlib/legend.py

Before:

    hm = default_handler_map.copy()
    hm.update(self._handler_map)
    return hm

After:

    return default_handler_map | self._handler_map

numpy/ma/core.py

Before:

    _optinfo = {}
    _optinfo.update(getattr(obj, '_optinfo', {}))
    _optinfo.update(getattr(obj, '_basedict', {}))
    if not isinstance(obj, MaskedArray):
        _optinfo.update(getattr(obj, '__dict__', {}))

After:

    _optinfo = {}
    _optinfo |= getattr(obj, '_optinfo', {})
    _optinfo |= getattr(obj, '_basedict', {})
    if not isinstance(obj, MaskedArray):
        _optinfo |= getattr(obj, '__dict__', {})

praw/internal.py

Before:

    data = {'name': six.text_type(user), 'type': relationship}
    data.update(kwargs)

After:

    data = {'name': six.text_type(user), 'type': relationship} | kwargs

pygments/lexer.py

Before:

    kwargs.update(lexer.options)
    lx = lexer.__class__(**kwargs)

After:

    lx = lexer.__class__(**(kwargs | lexer.options))

requests/sessions.py

Before:

    merged_setting = dict_class(to_key_val_list(session_setting))
    merged_setting.update(to_key_val_list(request_setting))

After:

    merged_setting = dict_class(to_key_val_list(session_setting)) | to_key_val_list(request_setting)

sphinx/domains/__init__.py

Before:

    self.attrs = self.known_attrs.copy()
    self.attrs.update(attrs)

After:

    self.attrs = self.known_attrs | attrs

sphinx/ext/doctest.py

Before:

    new_opt = code[0].options.copy()
    new_opt.update(example.options)
    example.options = new_opt

After:

    example.options = code[0].options | example.options

sphinx/ext/inheritance_diagram.py

Before:

    n_attrs = self.default_node_attrs.copy()
    e_attrs = self.default_edge_attrs.copy()
    g_attrs.update(graph_attrs)
    n_attrs.update(node_attrs)
    e_attrs.update(edge_attrs)

After:

    g_attrs |= graph_attrs
    n_attrs = self.default_node_attrs | node_attrs
    e_attrs = self.default_edge_attrs | edge_attrs

sphinx/highlighting.py

Before:

    kwargs.update(self.formatter_args)
    return self.formatter(**kwargs)

After:

    return self.formatter(**(kwargs | self.formatter_args))

sphinx/quickstart.py

Before:

    d2 = DEFAULT_VALUE.copy()
    d2.update(dict(("ext_"+ext, False) for ext in EXTENSIONS))
    d2.update(d)
    d = d2

After:

    d = DEFAULT_VALUE | dict(("ext_"+ext, False) for ext in EXTENSIONS) | d

sympy/abc.py

Before:

    clash = {}
    clash.update(clash1)
    clash.update(clash2)
    return clash1, clash2, clash

After:

    return clash1, clash2, clash1 | clash2

sympy/parsing/maxima.py

Before:

    dct = MaximaHelpers.__dict__.copy()
    dct.update(name_dict)
    obj = sympify(str, locals=dct)

After:

    obj = sympify(str, locals=MaximaHelpers.__dict__|name_dict)

sympy/printing/ccode.py and sympy/printing/fcode.py

Before:

    self.known_functions = dict(known_functions)
    userfuncs = settings.get('user_functions', {})
    self.known_functions.update(userfuncs)

After:

    self.known_functions = known_functions | settings.get('user_functions', {})

sympy/utilities/runtests.py

Before:

    globs = globs.copy()
    if extraglobs is not None:
        globs.update(extraglobs)

After:

    globs = globs | (extraglobs if extraglobs is not None else {})

The above examples show that sometimes the | operator leads to a clear
increase in readability, reducing the number of lines of code and
improving clarity. However other examples using the | operator lead to
long, complex single expressions, possibly well over the PEP 8 maximum
line length of 80 columns. As with any other language feature, the
programmer should use their own judgement about whether | improves their
code.

Related Discussions

Mailing list threads (this is by no means an exhaustive list):

-   Dict joining using + and +=
-   PEP: Dict addition and subtraction
-   PEP 584: Add + and += operators to the built-in dict class.
-   Moving PEP 584 forward (dict + and += operators)
-   PEP 584: Add Union Operators To dict
-   Accepting PEP 584: Add Union Operators To dict

Ticket on the bug tracker

Merging two dictionaries in an expression is a frequently requested
feature. For example:

https://stackoverflow.com/questions/38987/how-to-merge-two-dictionaries-in-a-single-expression

https://stackoverflow.com/questions/1781571/how-to-concatenate-two-dictionaries-to-create-a-new-one-in-python

https://stackoverflow.com/questions/6005066/adding-dictionaries-together-python

Occasionally people request alternative behavior for the merge:

https://stackoverflow.com/questions/1031199/adding-dictionaries-in-python

https://stackoverflow.com/questions/877295/python-dict-add-by-valuedict-2

...including one proposal to treat dicts as sets of keys.

Ian Lee's proto-PEP, and discussion in 2015. Further discussion took
place on Python-Ideas.

(Observant readers will notice that one of the authors of this PEP was
more skeptical of the idea in 2015.)

Adding a full complement of operators to dicts.

Discussion on Y-Combinator.

https://treyhunner.com/2016/02/how-to-merge-dictionaries-in-python/

https://code.tutsplus.com/tutorials/how-to-merge-two-python-dictionaries--cms-26230

In direct response to an earlier draft of this PEP, Serhiy Storchaka
raised a ticket in the bug tracker to replace the copy(); update() idiom
with dict unpacking.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

license: Public Domain


text:
PEP: 342 Title: Coroutines via Enhanced Generators Version: $Revision$
Last-Modified: $Date$ Author: Guido van Rossum, Phillip J. Eby Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
10-May-2005 Python-Version: 2.5 Post-History:

Introduction

This PEP proposes some enhancements to the API and syntax of generators,
to make them usable as simple coroutines. It is basically a combination
of ideas from these two PEPs, which may be considered redundant if this
PEP is accepted:

-   PEP 288, Generators Attributes and Exceptions. The current PEP
    covers its second half, generator exceptions (in fact the throw()
    method name was taken from PEP 288). PEP 342 replaces generator
    attributes, however, with a concept from an earlier revision of PEP
    288, the yield expression.
-   PEP 325, Resource-Release Support for Generators. PEP 342 ties up a
    few loose ends in the PEP 325 spec, to make it suitable for actual
    implementation.

Motivation

Coroutines are a natural way of expressing many algorithms, such as
simulations, games, asynchronous I/O, and other forms of event-driven
programming or co-operative multitasking. Python's generator functions
are almost coroutines -- but not quite -- in that they allow pausing
execution to produce a value, but do not provide for values or
exceptions to be passed in when execution resumes. They also do not
allow execution to be paused within the try portion of try/finally
blocks, and therefore make it difficult for an aborted coroutine to
clean up after itself.

Also, generators cannot yield control while other functions are
executing, unless those functions are themselves expressed as
generators, and the outer generator is written to yield in response to
values yielded by the inner generator. This complicates the
implementation of even relatively simple use cases like asynchronous
communications, because calling any functions either requires the
generator to block (i.e. be unable to yield control), or else a lot of
boilerplate looping code must be added around every needed function
call.

However, if it were possible to pass values or exceptions into a
generator at the point where it was suspended, a simple co-routine
scheduler or trampoline function would let coroutines call each other
without blocking -- a tremendous boon for asynchronous applications.
Such applications could then write co-routines to do non-blocking socket
I/O by yielding control to an I/O scheduler until data has been sent or
becomes available. Meanwhile, code that performs the I/O would simply do
something like this:

    data = (yield nonblocking_read(my_socket, nbytes))

in order to pause execution until the nonblocking_read() coroutine
produced a value.

In other words, with a few relatively minor enhancements to the language
and to the implementation of the generator-iterator type, Python will be
able to support performing asynchronous operations without needing to
write the entire application as a series of callbacks, and without
requiring the use of resource-intensive threads for programs that need
hundreds or even thousands of co-operatively multitasking pseudothreads.
Thus, these enhancements will give standard Python many of the benefits
of the Stackless Python fork, without requiring any significant
modification to the CPython core or its APIs. In addition, these
enhancements should be readily implementable by any Python
implementation (such as Jython) that already supports generators.

Specification Summary

By adding a few simple methods to the generator-iterator type, and with
two minor syntax adjustments, Python developers will be able to use
generator functions to implement co-routines and other forms of
co-operative multitasking. These methods and adjustments are:

1.  Redefine yield to be an expression, rather than a statement. The
    current yield statement would become a yield expression whose value
    is thrown away. A yield expression's value is None whenever the
    generator is resumed by a normal next() call.
2.  Add a new send() method for generator-iterators, which resumes the
    generator and sends a value that becomes the result of the current
    yield-expression. The send() method returns the next value yielded
    by the generator, or raises StopIteration if the generator exits
    without yielding another value.
3.  Add a new throw() method for generator-iterators, which raises an
    exception at the point where the generator was paused, and which
    returns the next value yielded by the generator, raising
    StopIteration if the generator exits without yielding another value.
    (If the generator does not catch the passed-in exception, or raises
    a different exception, then that exception propagates to the
    caller.)
4.  Add a close() method for generator-iterators, which raises
    GeneratorExit at the point where the generator was paused. If the
    generator then raises StopIteration (by exiting normally, or due to
    already being closed) or GeneratorExit (by not catching the
    exception), close() returns to its caller. If the generator yields a
    value, a RuntimeError is raised. If the generator raises any other
    exception, it is propagated to the caller. close() does nothing if
    the generator has already exited due to an exception or normal exit.
5.  Add support to ensure that close() is called when a generator
    iterator is garbage-collected.
6.  Allow yield to be used in try/finally blocks, since garbage
    collection or an explicit close() call would now allow the finally
    clause to execute.

A prototype patch implementing all of these changes against the current
Python CVS HEAD is available as SourceForge patch #1223381
(https://bugs.python.org/issue1223381).

Specification: Sending Values into Generators

New generator method: send(value)

A new method for generator-iterators is proposed, called send(). It
takes exactly one argument, which is the value that should be sent in to
the generator. Calling send(None) is exactly equivalent to calling a
generator's next() method. Calling send() with any other value is the
same, except that the value produced by the generator's current yield
expression will be different.

Because generator-iterators begin execution at the top of the
generator's function body, there is no yield expression to receive a
value when the generator has just been created. Therefore, calling
send() with a non-None argument is prohibited when the generator
iterator has just started, and a TypeError is raised if this occurs
(presumably due to a logic error of some kind). Thus, before you can
communicate with a coroutine you must first call next() or send(None) to
advance its execution to the first yield expression.

As with the next() method, the send() method returns the next value
yielded by the generator-iterator, or raises StopIteration if the
generator exits normally, or has already exited. If the generator raises
an uncaught exception, it is propagated to send()'s caller.

New syntax: Yield Expressions

The yield-statement will be allowed to be used on the right-hand side of
an assignment; in that case it is referred to as yield-expression. The
value of this yield-expression is None unless send() was called with a
non-None argument; see below.

A yield-expression must always be parenthesized except when it occurs at
the top-level expression on the right-hand side of an assignment. So

    x = yield 42
    x = yield
    x = 12 + (yield 42)
    x = 12 + (yield)
    foo(yield 42)
    foo(yield)

are all legal, but

    x = 12 + yield 42
    x = 12 + yield
    foo(yield 42, 12)
    foo(yield, 12)

are all illegal. (Some of the edge cases are motivated by the current
legality of yield 12, 42.)

Note that a yield-statement or yield-expression without an expression is
now legal. This makes sense: when the information flow in the next()
call is reversed, it should be possible to yield without passing an
explicit value (yield is of course equivalent to yield None).

When send(value) is called, the yield-expression that it resumes will
return the passed-in value. When next() is called, the resumed
yield-expression will return None. If the yield-expression is a
yield-statement, this returned value is ignored, similar to ignoring the
value returned by a function call used as a statement.

In effect, a yield-expression is like an inverted function call; the
argument to yield is in fact returned (yielded) from the currently
executing function, and the return value of yield is the argument passed
in via send().

Note: the syntactic extensions to yield make its use very similar to
that in Ruby. This is intentional. Do note that in Python the block
passes a value to the generator using send(EXPR) rather than
return EXPR, and the underlying mechanism whereby control is passed
between the generator and the block is completely different. Blocks in
Python are not compiled into thunks; rather, yield suspends execution of
the generator's frame. Some edge cases work differently; in Python, you
cannot save the block for later use, and you cannot test whether there
is a block or not. (XXX - this stuff about blocks seems out of place
now, perhaps Guido can edit to clarify.)

Specification: Exceptions and Cleanup

Let a generator object be the iterator produced by calling a generator
function. Below, g always refers to a generator object.

New syntax: yield allowed inside try-finally

The syntax for generator functions is extended to allow a
yield-statement inside a try-finally statement.

New generator method: throw(type, value=None, traceback=None)

g.throw(type, value, traceback) causes the specified exception to be
thrown at the point where the generator g is currently suspended (i.e.
at a yield-statement, or at the start of its function body if next() has
not been called yet). If the generator catches the exception and yields
another value, that is the return value of g.throw(). If it doesn't
catch the exception, the throw() appears to raise the same exception
passed it (it falls through). If the generator raises another exception
(this includes the StopIteration produced when it returns) that
exception is raised by the throw() call. In summary, throw() behaves
like next() or send(), except it raises an exception at the suspension
point. If the generator is already in the closed state, throw() just
raises the exception it was passed without executing any of the
generator's code.

The effect of raising the exception is exactly as if the statement:

    raise type, value, traceback

was executed at the suspension point. The type argument must not be
None, and the type and value must be compatible. If the value is not an
instance of the type, a new exception instance is created using the
value, following the same rules that the raise statement uses to create
an exception instance. The traceback, if supplied, must be a valid
Python traceback object, or a TypeError occurs.

Note: The name of the throw() method was selected for several reasons.
Raise is a keyword and so cannot be used as a method name. Unlike raise
(which immediately raises an exception from the current execution
point), throw() first resumes the generator, and only then raises the
exception. The word throw is suggestive of putting the exception in
another location, and is already associated with exceptions in other
languages.

Alternative method names were considered: resolve(), signal(),
genraise(), raiseinto(), and flush(). None of these seem to fit as well
as throw().

New standard exception: GeneratorExit

A new standard exception is defined, GeneratorExit, inheriting from
Exception. A generator should handle this by re-raising it (or just not
catching it) or by raising StopIteration.

New generator method: close()

g.close() is defined by the following pseudo-code:

    def close(self):
        try:
            self.throw(GeneratorExit)
        except (GeneratorExit, StopIteration):
            pass
        else:
            raise RuntimeError("generator ignored GeneratorExit")
        # Other exceptions are not caught

New generator method: __del__()

g.__del__() is a wrapper for g.close(). This will be called when the
generator object is garbage-collected (in CPython, this is when its
reference count goes to zero). If close() raises an exception, a
traceback for the exception is printed to sys.stderr and further
ignored; it is not propagated back to the place that triggered the
garbage collection. This is consistent with the handling of exceptions
in __del__() methods on class instances.

If the generator object participates in a cycle, g.__del__() may not be
called. This is the behavior of CPython's current garbage collector. The
reason for the restriction is that the GC code needs to break a cycle at
an arbitrary point in order to collect it, and from then on no Python
code should be allowed to see the objects that formed the cycle, as they
may be in an invalid state. Objects hanging off a cycle are not subject
to this restriction.

Note that it is unlikely to see a generator object participate in a
cycle in practice. However, storing a generator object in a global
variable creates a cycle via the generator frame's f_globals pointer.
Another way to create a cycle would be to store a reference to the
generator object in a data structure that is passed to the generator as
an argument (e.g., if an object has a method that's a generator, and
keeps a reference to a running iterator created by that method). Neither
of these cases are very likely given the typical patterns of generator
use.

Also, in the CPython implementation of this PEP, the frame object used
by the generator should be released whenever its execution is terminated
due to an error or normal exit. This will ensure that generators that
cannot be resumed do not remain part of an uncollectable reference
cycle. This allows other code to potentially use close() in a
try/finally or with block (per PEP 343) to ensure that a given generator
is properly finalized.

Optional Extensions

The Extended continue Statement

An earlier draft of this PEP proposed a new continue EXPR syntax for use
in for-loops (carried over from PEP 340), that would pass the value of
EXPR into the iterator being looped over. This feature has been
withdrawn for the time being, because the scope of this PEP has been
narrowed to focus only on passing values into generator-iterators, and
not other kinds of iterators. It was also felt by some on the Python-Dev
list that adding new syntax for this particular feature would be
premature at best.

Open Issues

Discussion on python-dev has revealed some open issues. I list them
here, with my preferred resolution and its motivation. The PEP as
currently written reflects this preferred resolution.

1.  What exception should be raised by close() when the generator yields
    another value as a response to the GeneratorExit exception?

    I originally chose TypeError because it represents gross misbehavior
    of the generator function, which should be fixed by changing the
    code. But the with_template decorator class in PEP 343 uses
    RuntimeError for similar offenses. Arguably they should all use the
    same exception. I'd rather not introduce a new exception class just
    for this purpose, since it's not an exception that I want people to
    catch: I want it to turn into a traceback which is seen by the
    programmer who then fixes the code. So now I believe they should
    both raise RuntimeError. There are some precedents for that: it's
    raised by the core Python code in situations where endless recursion
    is detected, and for uninitialized objects (and for a variety of
    miscellaneous conditions).

2.  Oren Tirosh has proposed renaming the send() method to feed(), for
    compatibility with the consumer interface (see
    http://effbot.org/zone/consumer.htm for the specification.)

    However, looking more closely at the consumer interface, it seems
    that the desired semantics for feed() are different than for send(),
    because send() can't be meaningfully called on a just-started
    generator. Also, the consumer interface as currently defined doesn't
    include handling for StopIteration.

    Therefore, it seems like it would probably be more useful to create
    a simple decorator that wraps a generator function to make it
    conform to the consumer interface. For example, it could warm up the
    generator with an initial next() call, trap StopIteration, and
    perhaps even provide reset() by re-invoking the generator function.

Examples

1.  A simple consumer decorator that makes a generator function
    automatically advance to its first yield point when initially
    called:

        def consumer(func):
            def wrapper(*args,**kw):
                gen = func(*args, **kw)
                gen.next()
                return gen
            wrapper.__name__ = func.__name__
            wrapper.__dict__ = func.__dict__
            wrapper.__doc__  = func.__doc__
            return wrapper

2.  An example of using the consumer decorator to create a reverse
    generator that receives images and creates thumbnail pages, sending
    them on to another consumer. Functions like this can be chained
    together to form efficient processing pipelines of consumers that
    each can have complex internal state:

        @consumer
        def thumbnail_pager(pagesize, thumbsize, destination):
            while True:
                page = new_image(pagesize)
                rows, columns = pagesize / thumbsize
                pending = False
                try:
                    for row in xrange(rows):
                        for column in xrange(columns):
                            thumb = create_thumbnail((yield), thumbsize)
                            page.write(
                                thumb, col*thumbsize.x, row*thumbsize.y )
                            pending = True
                except GeneratorExit:
                    # close() was called, so flush any pending output
                    if pending:
                        destination.send(page)

                    # then close the downstream consumer, and exit
                    destination.close()
                    return
                else:
                    # we finished a page full of thumbnails, so send it
                    # downstream and keep on looping
                    destination.send(page)

        @consumer
        def jpeg_writer(dirname):
            fileno = 1
            while True:
                filename = os.path.join(dirname,"page%04d.jpg" % fileno)
                write_jpeg((yield), filename)
                fileno += 1

        # Put them together to make a function that makes thumbnail
        # pages from a list of images and other parameters.
        #
        def write_thumbnails(pagesize, thumbsize, images, output_dir):
            pipeline = thumbnail_pager(
                pagesize, thumbsize, jpeg_writer(output_dir)
            )

            for image in images:
                pipeline.send(image)

            pipeline.close()

3.  A simple co-routine scheduler or trampoline that lets coroutines
    call other coroutines by yielding the coroutine they wish to invoke.
    Any non-generator value yielded by a coroutine is returned to the
    coroutine that called the one yielding the value. Similarly, if a
    coroutine raises an exception, the exception is propagated to its
    caller. In effect, this example emulates simple tasklets as are used
    in Stackless Python, as long as you use a yield expression to invoke
    routines that would otherwise block. This is only a very simple
    example, and far more sophisticated schedulers are possible. (For
    example, the existing GTasklet framework for Python
    (http://www.gnome.org/~gjc/gtasklet/gtasklets.html) and the
    peak.events framework (http://peak.telecommunity.com/) already
    implement similar scheduling capabilities, but must currently use
    awkward workarounds for the inability to pass values or exceptions
    into generators.)

        import collections

        class Trampoline:
            """Manage communications between coroutines"""

            running = False

            def __init__(self):
                self.queue = collections.deque()

            def add(self, coroutine):
                """Request that a coroutine be executed"""
                self.schedule(coroutine)

            def run(self):
                result = None
                self.running = True
                try:
                    while self.running and self.queue:
                       func = self.queue.popleft()
                       result = func()
                    return result
                finally:
                    self.running = False

            def stop(self):
                self.running = False

            def schedule(self, coroutine, stack=(), val=None, *exc):
                def resume():
                    value = val
                    try:
                        if exc:
                            value = coroutine.throw(value,*exc)
                        else:
                            value = coroutine.send(value)
                    except:
                        if stack:
                            # send the error back to the "caller"
                            self.schedule(
                                stack[0], stack[1], *sys.exc_info()
                            )
                        else:
                            # Nothing left in this pseudothread to
                            # handle it, let it propagate to the
                            # run loop
                            raise

                    if isinstance(value, types.GeneratorType):
                        # Yielded to a specific coroutine, push the
                        # current one on the stack, and call the new
                        # one with no args
                        self.schedule(value, (coroutine,stack))

                    elif stack:
                        # Yielded a result, pop the stack and send the
                        # value to the caller
                        self.schedule(stack[0], stack[1], value)

                    # else: this pseudothread has ended

                self.queue.append(resume)

4.  A simple echo server, and code to run it using a trampoline
    (presumes the existence of nonblocking_read, nonblocking_write, and
    other I/O coroutines, that e.g. raise ConnectionLost if the
    connection is closed):

        # coroutine function that echos data back on a connected
        # socket
        #
        def echo_handler(sock):
            while True:
                try:
                    data = yield nonblocking_read(sock)
                    yield nonblocking_write(sock, data)
                except ConnectionLost:
                    pass  # exit normally if connection lost

        # coroutine function that listens for connections on a
        # socket, and then launches a service "handler" coroutine
        # to service the connection
        #
        def listen_on(trampoline, sock, handler):
            while True:
                # get the next incoming connection
                connected_socket = yield nonblocking_accept(sock)

                # start another coroutine to handle the connection
                trampoline.add( handler(connected_socket) )

        # Create a scheduler to manage all our coroutines
        t = Trampoline()

        # Create a coroutine instance to run the echo_handler on
        # incoming connections
        #
        server = listen_on(
            t, listening_socket("localhost","echo"), echo_handler
        )

        # Add the coroutine to the scheduler
        t.add(server)

        # loop forever, accepting connections and servicing them
        # "in parallel"
        #
        t.run()

Reference Implementation

A prototype patch implementing all of the features described in this PEP
is available as SourceForge patch #1223381
(https://bugs.python.org/issue1223381).

This patch was committed to CVS 01-02 August 2005.

Acknowledgements

Raymond Hettinger (PEP 288) and Samuele Pedroni (PEP 325) first formally
proposed the ideas of communicating values or exceptions into
generators, and the ability to close generators. Timothy Delaney
suggested the title of this PEP, and Steven Bethard helped edit a
previous version. See also the Acknowledgements section of PEP 340.

References

TBD.

Copyright

This document has been placed in the public domain.

authors:
Guido van Rossum
Phillip J. Eby