text:
PEP: 636 Title: Structural Pattern Matching: Tutorial Version:
$Revision$ Last-Modified: $Date$ Author: Daniel F Moisset
<EMAIL_ADDRESS>Sponsor: Guido van Rossum<EMAIL_ADDRESS>BDFL-Delegate: Discussions-To<EMAIL_ADDRESS>Status: Final Type:
Informational Content-Type: text/x-rst Created: 12-Sep-2020
Python-Version: 3.10 Post-History: 22-Oct-2020, 08-Feb-2021 Resolution:
<EMAIL_ADDRESS>
Abstract

This PEP is a tutorial for the pattern matching introduced by PEP 634.

PEP 622 proposed syntax for pattern matching, which received detailed
discussion both from the community and the Steering Council. A frequent
concern was about how easy it would be to explain (and learn) this
feature. This PEP addresses that concern providing the kind of document
which developers could use to learn about pattern matching in Python.

This is considered supporting material for PEP 634 (the technical
specification for pattern matching) and PEP 635 (the motivation and
rationale for having pattern matching and design considerations).

For readers who are looking more for a quick review than for a tutorial,
see Appendix A.

Tutorial

As an example to motivate this tutorial, you will be writing a text
adventure. That is a form of interactive fiction where the user enters
text commands to interact with a fictional world and receives text
descriptions of what happens. Commands will be simplified forms of
natural language like get sword, attack dragon, go north, enter shop or
buy cheese.

Matching sequences

Your main loop will need to get input from the user and split it into
words, let's say a list of strings like this:

    command = input("What are you doing next? ")
    # analyze the result of command.split()

The next step is to interpret the words. Most of our commands will have
two words: an action and an object. So you may be tempted to do the
following:

    [action, obj] = command.split()
    ... # interpret action, obj

The problem with that line of code is that it's missing something: what
if the user types more or fewer than 2 words? To prevent this problem
you can either check the length of the list of words, or capture the
ValueError that the statement above would raise.

You can use a matching statement instead:

    match command.split():
        case [action, obj]:
            ... # interpret action, obj

The match statement evaluates the "subject" (the value after the match
keyword), and checks it against the pattern (the code next to case). A
pattern is able to do two different things:

-   Verify that the subject has certain structure. In your case, the
    [action, obj] pattern matches any sequence of exactly two elements.
    This is called matching
-   It will bind some names in the pattern to component elements of your
    subject. In this case, if the list has two elements, it will bind
    action = subject[0] and obj = subject[1].

If there's a match, the statements inside the case block will be
executed with the bound variables. If there's no match, nothing happens
and the statement after match is executed next.

Note that, in a similar way to unpacking assignments, you can use either
parenthesis, brackets, or just comma separation as synonyms. So you
could write case action, obj or case (action, obj) with the same
meaning. All forms will match any sequence (for example lists or
tuples).

Matching multiple patterns

Even if most commands have the action/object form, you might want to
have user commands of different lengths. For example, you might want to
add single verbs with no object like look or quit. A match statement can
(and is likely to) have more than one case:

    match command.split():
        case [action]:
            ... # interpret single-verb action
        case [action, obj]:
            ... # interpret action, obj

The match statement will check patterns from top to bottom. If the
pattern doesn't match the subject, the next pattern will be tried.
However, once the first matching pattern is found, the body of that case
is executed, and all further cases are ignored. This is similar to the
way that an if/elif/elif/... statement works.

Matching specific values

Your code still needs to look at the specific actions and conditionally
execute different logic depending on the specific action (e.g., quit,
attack, or buy). You could do that using a chain of if/elif/elif/..., or
using a dictionary of functions, but here we'll leverage pattern
matching to solve that task. Instead of a variable, you can use literal
values in patterns (like "quit", 42, or None). This allows you to write:

    match command.split():
        case ["quit"]:
            print("Goodbye!")
            quit_game()
        case ["look"]:
            current_room.describe()
        case ["get", obj]:
            character.get(obj, current_room)
        case ["go", direction]:
            current_room = current_room.neighbor(direction)
        # The rest of your commands go here

A pattern like ["get", obj] will match only 2-element sequences that
have a first element equal to "get". It will also bind obj = subject[1].

As you can see in the go case, we also can use different variable names
in different patterns.

Literal values are compared with the == operator except for the
constants True, False and None which are compared with the is operator.

Matching multiple values

A player may be able to drop multiple items by using a series of
commands drop key, drop sword, drop cheese. This interface might be
cumbersome, and you might like to allow dropping multiple items in a
single command, like drop key sword cheese. In this case you don't know
beforehand how many words will be in the command, but you can use
extended unpacking in patterns in the same way that they are allowed in
assignments:

    match command.split():
        case ["drop", *objects]:
            for obj in objects:
                character.drop(obj, current_room)
        # The rest of your commands go here

This will match any sequences having "drop" as its first elements. All
remaining elements will be captured in a list object which will be bound
to the objects variable.

This syntax has similar restrictions as sequence unpacking: you can not
have more than one starred name in a pattern.

Adding a wildcard

You may want to print an error message saying that the command wasn't
recognized when all the patterns fail. You could use the feature we just
learned and write case [*ignored_words] as your last pattern. There's
however a much simpler way:

    match command.split():
        case ["quit"]: ... # Code omitted for brevity
        case ["go", direction]: ...
        case ["drop", *objects]: ...
        ... # Other cases
        case _:
            print(f"Sorry, I couldn't understand {command!r}")

This special pattern which is written _ (and called wildcard) always
matches but it doesn't bind any variables.

Note that this will match any object, not just sequences. As such, it
only makes sense to have it by itself as the last pattern (to prevent
errors, Python will stop you from using it before).

Composing patterns

This is a good moment to step back from the examples and understand how
the patterns that you have been using are built. Patterns can be nested
within each other, and we have been doing that implicitly in the
examples above.

There are some "simple" patterns ("simple" here meaning that they do not
contain other patterns) that we've seen:

-   Capture patterns (stand-alone names like direction, action,
    objects). We never discussed these separately, but used them as part
    of other patterns.
-   Literal patterns (string literals, number literals, True, False, and
    None)
-   The wildcard pattern _

Until now, the only non-simple pattern we have experimented with is the
sequence pattern. Each element in a sequence pattern can in fact be any
other pattern. This means that you could write a pattern like
["first", (left, right), _, *rest]. This will match subjects which are a
sequence of at least three elements, where the first one is equal to
"first" and the second one is in turn a sequence of two elements. It
will also bind left=subject[1][0], right=subject[1][1], and
rest = subject[3:]

Or patterns

Going back to the adventure game example, you may find that you'd like
to have several patterns resulting in the same outcome. For example, you
might want the commands north and go north to be equivalent. You may
also desire to have aliases for get X, pick up X and pick X up for any
X.

The | symbol in patterns combines them as alternatives. You could for
example write:

    match command.split():
        ... # Other cases
        case ["north"] | ["go", "north"]:
            current_room = current_room.neighbor("north")
        case ["get", obj] | ["pick", "up", obj] | ["pick", obj, "up"]:
            ... # Code for picking up the given object

This is called an or pattern and will produce the expected result.
Patterns are tried from left to right; this may be relevant to know what
is bound if more than one alternative matches. An important restriction
when writing or patterns is that all alternatives should bind the same
variables. So a pattern [1, x] | [2, y] is not allowed because it would
make unclear which variable would be bound after a successful match.
[1, x] | [2, x] is perfectly fine and will always bind x if successful.

Capturing matched sub-patterns

The first version of our "go" command was written with a
["go", direction] pattern. The change we did in our last version using
the pattern ["north"] | ["go", "north"] has some benefits but also some
drawbacks in comparison: the latest version allows the alias, but also
has the direction hardcoded, which will force us to actually have
separate patterns for north/south/east/west. This leads to some code
duplication, but at the same time we get better input validation, and we
will not be getting into that branch if the command entered by the user
is "go figure!" instead of a direction.

We could try to get the best of both worlds doing the following (I'll
omit the aliased version without "go" for brevity):

    match command.split():
        case ["go", ("north" | "south" | "east" | "west")]:
            current_room = current_room.neighbor(...)
            # how do I know which direction to go?

This code is a single branch, and it verifies that the word after "go"
is really a direction. But the code moving the player around needs to
know which one was chosen and has no way to do so. What we need is a
pattern that behaves like the or pattern but at the same time does a
capture. We can do so with an as pattern:

    match command.split():
        case ["go", ("north" | "south" | "east" | "west") as direction]:
            current_room = current_room.neighbor(direction)

The as-pattern matches whatever pattern is on its left-hand side, but
also binds the value to a name.

Adding conditions to patterns

The patterns we have explored above can do some powerful data filtering,
but sometimes you may wish for the full power of a boolean expression.
Let's say that you would actually like to allow a "go" command only in a
restricted set of directions based on the possible exits from the
current_room. We can achieve that by adding a guard to our case. Guards
consist of the if keyword followed by any expression:

    match command.split():
        case ["go", direction] if direction in current_room.exits:
            current_room = current_room.neighbor(direction)
        case ["go", _]:
            print("Sorry, you can't go that way")

The guard is not part of the pattern, it's part of the case. It's only
checked if the pattern matches, and after all the pattern variables have
been bound (that's why the condition can use the direction variable in
the example above). If the pattern matches and the condition is truthy,
the body of the case executes normally. If the pattern matches but the
condition is falsy, the match statement proceeds to check the next case
as if the pattern hadn't matched (with the possible side-effect of
having already bound some variables).

Adding a UI: Matching objects

Your adventure is becoming a success and you have been asked to
implement a graphical interface. Your UI toolkit of choice allows you to
write an event loop where you can get a new event object by calling
event.get(). The resulting object can have different type and attributes
according to the user action, for example:

-   A KeyPress object is generated when the user presses a key. It has a
    key_name attribute with the name of the key pressed, and some other
    attributes regarding modifiers.
-   A Click object is generated when the user clicks the mouse. It has
    an attribute position with the coordinates of the pointer.
-   A Quit object is generated when the user clicks on the close button
    for the game window.

Rather than writing multiple isinstance() checks, you can use patterns
to recognize different kinds of objects, and also apply patterns to its
attributes:

    match event.get():
        case Click(position=(x, y)):
            handle_click_at(x, y)
        case KeyPress(key_name="Q") | Quit():
            game.quit()
        case KeyPress(key_name="up arrow"):
            game.go_north()
        ...
        case KeyPress():
            pass # Ignore other keystrokes
        case other_event:
            raise ValueError(f"Unrecognized event: {other_event}")

A pattern like Click(position=(x, y)) only matches if the type of the
event is a subclass of the Click class. It will also require that the
event has a position attribute that matches the (x, y) pattern. If
there's a match, the locals x and y will get the expected values.

A pattern like KeyPress(), with no arguments will match any object which
is an instance of the KeyPress class. Only the attributes you specify in
the pattern are matched, and any other attributes are ignored.

Matching positional attributes

The previous section described how to match named attributes when doing
an object match. For some objects it could be convenient to describe the
matched arguments by position (especially if there are only a few
attributes and they have a "standard" ordering). If the classes that you
are using are named tuples or dataclasses, you can do that by following
the same order that you'd use when constructing an object. For example,
if the UI framework above defines their class like this:

    from dataclasses import dataclass

    @dataclass
    class Click:
        position: tuple
        button: Button

then you can rewrite your match statement above as:

    match event.get():
        case Click((x, y)):
            handle_click_at(x, y)

The (x, y) pattern will be automatically matched against the position
attribute, because the first argument in the pattern corresponds to the
first attribute in your dataclass definition.

Other classes don't have a natural ordering of their attributes so
you're required to use explicit names in your pattern to match with
their attributes. However, it's possible to manually specify the
ordering of the attributes allowing positional matching, like in this
alternative definition:

    class Click:
        __match_args__ = ("position", "button")
        def __init__(self, pos, btn):
            self.position = pos
            self.button = btn
            ...

The __match_args__ special attribute defines an explicit order for your
attributes that can be used in patterns like case Click((x,y)).

Matching against constants and enums

Your pattern above treats all mouse buttons the same, and you have
decided that you want to accept left-clicks, and ignore other buttons.
While doing so, you notice that the button attribute is typed as a
Button which is an enumeration built with enum.Enum. You can in fact
match against enumeration values like this:

    match event.get():
        case Click((x, y), button=Button.LEFT):  # This is a left click
            handle_click_at(x, y)
        case Click():
            pass  # ignore other clicks

This will work with any dotted name (like math.pi). However an
unqualified name (i.e. a bare name with no dots) will be always
interpreted as a capture pattern, so avoid that ambiguity by always
using qualified constants in patterns.

Going to the cloud: Mappings

You have decided to make an online version of your game. All of your
logic will be in a server, and the UI in a client which will communicate
using JSON messages. Via the json module, those will be mapped to Python
dictionaries, lists and other builtin objects.

Our client will receive a list of dictionaries (parsed from JSON) of
actions to take, each element looking for example like these:

-   {"text": "The shop keeper says 'Ah! We have Camembert, yes sir'", "color": "blue"}
-   If the client should make a pause {"sleep": 3}
-   To play a sound {"sound": "filename.ogg", "format": "ogg"}

Until now, our patterns have processed sequences, but there are patterns
to match mappings based on their present keys. In this case you could
use:

    for action in actions:
        match action:
            case {"text": message, "color": c}:
                ui.set_text_color(c)
                ui.display(message)
            case {"sleep": duration}:
                ui.wait(duration)
            case {"sound": url, "format": "ogg"}:
                ui.play(url)
            case {"sound": _, "format": _}:
                warning("Unsupported audio format")

The keys in your mapping pattern need to be literals, but the values can
be any pattern. As in sequence patterns, all subpatterns have to match
for the general pattern to match.

You can use **rest within a mapping pattern to capture additional keys
in the subject. Note that if you omit this, extra keys in the subject
will be ignored while matching, i.e. the message
{"text": "foo", "color": "red", "style": "bold"} will match the first
pattern in the example above.

Matching builtin classes

The code above could use some validation. Given that messages came from
an external source, the types of the field could be wrong, leading to
bugs or security issues.

Any class is a valid match target, and that includes built-in classes
like bool str or int. That allows us to combine the code above with a
class pattern. So instead of writing {"text": message, "color": c} we
can use {"text": str() as message, "color": str() as c} to ensure that
message and c are both strings. For many builtin classes (see PEP 634
for the whole list), you can use a positional parameter as a shorthand,
writing str(c) rather than str() as c. The fully rewritten version looks
like this:

    for action in actions:
        match action:
            case {"text": str(message), "color": str(c)}:
                ui.set_text_color(c)
                ui.display(message)
            case {"sleep": float(duration)}:
                ui.wait(duration)
            case {"sound": str(url), "format": "ogg"}:
                ui.play(url)
            case {"sound": _, "format": _}:
                warning("Unsupported audio format")

Appendix A -- Quick Intro

A match statement takes an expression and compares its value to
successive patterns given as one or more case blocks. This is
superficially similar to a switch statement in C, Java or JavaScript
(and many other languages), but much more powerful.

The simplest form compares a subject value against one or more literals:

    def http_error(status):
        match status:
            case 400:
                return "Bad request"
            case 404:
                return "Not found"
            case 418:
                return "I'm a teapot"
            case _:
                return "Something's wrong with the Internet"

Note the last block: the "variable name" _ acts as a wildcard and never
fails to match.

You can combine several literals in a single pattern using | ("or"):

    case 401 | 403 | 404:
        return "Not allowed"

Patterns can look like unpacking assignments, and can be used to bind
variables:

    # point is an (x, y) tuple
    match point:
        case (0, 0):
            print("Origin")
        case (0, y):
            print(f"Y={y}")
        case (x, 0):
            print(f"X={x}")
        case (x, y):
            print(f"X={x}, Y={y}")
        case _:
            raise ValueError("Not a point")

Study that one carefully! The first pattern has two literals, and can be
thought of as an extension of the literal pattern shown above. But the
next two patterns combine a literal and a variable, and the variable
binds a value from the subject (point). The fourth pattern captures two
values, which makes it conceptually similar to the unpacking assignment
(x, y) = point.

If you are using classes to structure your data you can use the class
name followed by an argument list resembling a constructor, but with the
ability to capture attributes into variables:

    from dataclasses import dataclass

    @dataclass
    class Point:
        x: int
        y: int

    def where_is(point):
        match point:
            case Point(x=0, y=0):
                print("Origin")
            case Point(x=0, y=y):
                print(f"Y={y}")
            case Point(x=x, y=0):
                print(f"X={x}")
            case Point():
                print("Somewhere else")
            case _:
                print("Not a point")

You can use positional parameters with some builtin classes that provide
an ordering for their attributes (e.g. dataclasses). You can also define
a specific position for attributes in patterns by setting the
__match_args__ special attribute in your classes. If it's set to ("x",
"y"), the following patterns are all equivalent (and all bind the y
attribute to the var variable):

    Point(1, var)
    Point(1, y=var)
    Point(x=1, y=var)
    Point(y=var, x=1)

Patterns can be arbitrarily nested. For example, if we have a short list
of points, we could match it like this:

    match points:
        case []:
            print("No points")
        case [Point(0, 0)]:
            print("The origin")
        case [Point(x, y)]:
            print(f"Single point {x}, {y}")
        case [Point(0, y1), Point(0, y2)]:
            print(f"Two on the Y axis at {y1}, {y2}")
        case _:
            print("Something else")

We can add an if clause to a pattern, known as a "guard". If the guard
is false, match goes on to try the next case block. Note that value
capture happens before the guard is evaluated:

    match point:
        case Point(x, y) if x == y:
            print(f"Y=X at {x}")
        case Point(x, y):
            print(f"Not on the diagonal")

Several other key features:

-   Like unpacking assignments, tuple and list patterns have exactly the
    same meaning and actually match arbitrary sequences. An important
    exception is that they don't match iterators or strings.
    (Technically, the subject must be an instance of
    collections.abc.Sequence.)

-   Sequence patterns support wildcards: [x, y, *rest] and (x, y, *rest)
    work similar to wildcards in unpacking assignments. The name after *
    may also be _, so (x, y, *_) matches a sequence of at least two
    items without binding the remaining items.

-   Mapping patterns: {"bandwidth": b, "latency": l} captures the
    "bandwidth" and "latency" values from a dict. Unlike sequence
    patterns, extra keys are ignored. A wildcard **rest is also
    supported. (But **_ would be redundant, so it is not allowed.)

-   Subpatterns may be captured using the as keyword:

        case (Point(x1, y1), Point(x2, y2) as p2): ...

-   Most literals are compared by equality, however the singletons True,
    False and None are compared by identity.

-   Patterns may use named constants. These must be dotted names to
    prevent them from being interpreted as capture variable:

        from enum import Enum
        class Color(Enum):
            RED = 0
            GREEN = 1
            BLUE = 2

        match color:
            case Color.RED:
                print("I see red!")
            case Color.GREEN:
                print("Grass is green")
            case Color.BLUE:
                print("I'm feeling the blues :(")

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 3117 Title: Postfix type declarations Version: $Revision$
Last-Modified: $Date$ Author: Georg Brandl<EMAIL_ADDRESS>Status:
Rejected Type: Standards Track Content-Type: text/x-rst Created:
01-Apr-2007 Python-Version: 3.0 Post-History:

Abstract

This PEP proposes the addition of a postfix type declaration syntax to
Python. It also specifies a new typedef statement which is used to
create new mappings between types and declarators.

Its acceptance will greatly enhance the Python user experience as well
as eliminate one of the warts that deter users of other programming
languages from switching to Python.

Rationale

Python has long suffered from the lack of explicit type declarations.
Being one of the few aspects in which the language deviates from its
Zen, this wart has sparked many a discussion between Python heretics and
members of the PSU (for a few examples, see [EX1], [EX2] or [EX3]), and
it also made it a large-scale enterprise success unlikely.

However, if one wants to put an end to this misery, a decent Pythonic
syntax must be found. In almost all languages that have them, type
declarations lack this quality: they are verbose, often needing multiple
words for a single type, or they are hard to comprehend (e.g., a certain
language uses completely unrelated[1] adjectives like dim for type
declaration).

Therefore, this PEP combines the move to type declarations with another
bold move that will once again prove that Python is not only
future-proof but future-embracing: the introduction of Unicode
characters as an integral constituent of source code.

Unicode makes it possible to express much more with much less
characters, which is in accordance with the Zen <20> ("Readability
counts."). Additionally, it eliminates the need for a separate type
declaration statement, and last but not least, it makes Python measure
up to Perl 6, which already uses Unicode for its operators.[2]

Specification

When the type declaration mode is in operation, the grammar is changed
so that each NAME must consist of two parts: a name and a type
declarator, which is exactly one Unicode character.

The declarator uniquely specifies the type of the name, and if it occurs
on the left hand side of an expression, this type is enforced: an
InquisitionError exception is raised if the returned type doesn't match
the declared type.[3]

Also, function call result types have to be specified. If the result of
the call does not have the declared type, an InquisitionError is raised.
Caution: the declarator for the result should not be confused with the
declarator for the function object (see the example below).

Type declarators after names that are only read, not assigned to, are
not strictly necessary but enforced anyway (see the Python Zen:
"Explicit is better than implicit.").

The mapping between types and declarators is not static. It can be
completely customized by the programmer, but for convenience there are
some predefined mappings for some built-in types:

  Type        Declarator
  ----------- --------------------------------------------------
  object      � (REPLACEMENT CHARACTER)
  int         ℕ (DOUBLE-STRUCK CAPITAL N)
  float       ℮ (ESTIMATED SYMBOL)
  bool        ✓ (CHECK MARK)
  complex     ℂ (DOUBLE-STRUCK CAPITAL C)
  str         ✎ (LOWER RIGHT PENCIL)
  unicode     ✒ (BLACK NIB)
  tuple       ⒯ (PARENTHESIZED LATIN SMALL LETTER T)
  list        ♨ (HOT SPRINGS)
  dict        ⧟ (DOUBLE-ENDED MULTIMAP)
  set         ∅ (EMPTY SET) (Note: this is also for full sets)
  frozenset   ☃ (SNOWMAN)
  datetime    ⌚ (WATCH)
  function    ƛ (LATIN SMALL LETTER LAMBDA WITH STROKE)
  generator   ⚛ (ATOM SYMBOL)
  Exception   ⌁ (ELECTRIC ARROW)

The declarator for the None type is a zero-width space.

These characters should be obvious and easy to remember and type for
every programmer.

Unicode replacement units

Since even in our modern, globalized world there are still some
old-fashioned rebels who can't or don't want to use Unicode in their
source code, and since Python is a forgiving language, a fallback is
provided for those:

Instead of the single Unicode character, they can type
name${UNICODE NAME OF THE DECLARATOR}$. For example, these two function
definitions are equivalent:

    def fooƛ(xℂ):
        return None

and :

    def foo${LATIN SMALL LETTER LAMBDA WITH STROKE}$(x${DOUBLE-STRUCK CAPITAL C}$):
        return None${ZERO WIDTH NO-BREAK SPACE}$

This is still easy to read and makes the full power of type-annotated
Python available to ASCII believers.

The typedef statement

The mapping between types and declarators can be extended with this new
statement.

The syntax is as follows:

    typedef_stmt  ::=  "typedef" expr DECLARATOR

where expr resolves to a type object. For convenience, the typedef
statement can also be mixed with the class statement for new classes,
like so:

    typedef class Foo☺(object�):
        pass

Example

This is the standard os.path.normpath function, converted to type
declaration syntax:

    def normpathƛ(path✎)✎:
        """Normalize path, eliminating double slashes, etc."""
        if path✎ == '':
            return '.'
        initial_slashes✓ = path✎.startswithƛ('/')✓
        # POSIX allows one or two initial slashes, but treats three or more
        # as single slash.
        if (initial_slashes✓ and
            path✎.startswithƛ('//')✓ and not path✎.startswithƛ('///')✓)✓:
            initial_slashesℕ = 2
        comps♨ = path✎.splitƛ('/')♨
        new_comps♨ = []♨
        for comp✎ in comps♨:
            if comp✎ in ('', '.')⒯:
                continue
            if (comp✎ != '..' or (not initial_slashesℕ and not new_comps♨)✓ or
                 (new_comps♨ and new_comps♨[-1]✎ == '..')✓)✓:
                new_comps♨.appendƛ(comp✎)
            elif new_comps♨:
                new_comps♨.popƛ()✎
        comps♨ = new_comps♨
        path✎ = '/'.join(comps♨)✎
        if initial_slashesℕ:
            path✎ = '/'*initial_slashesℕ + path✎
        return path✎ or '.'

As you can clearly see, the type declarations add expressiveness, while
at the same time they make the code look much more professional.

Compatibility issues

To enable type declaration mode, one has to write:

    from __future__ import type_declarations

which enables Unicode parsing of the source[4], makes typedef a keyword
and enforces correct types for all assignments and function calls.

Rejection

After careful considering, much soul-searching, gnashing of teeth and
rending of garments, it has been decided to reject this PEP.

References

Acknowledgements

Many thanks go to Armin Ronacher, Alexander Schremmer and Marek Kubica
who helped find the most suitable and mnemonic declarator for built-in
types.

Thanks also to the Unicode Consortium for including all those useful
characters in the Unicode standard.

Copyright

This document has been placed in the public domain.

  Local Variables: coding: utf-8 mode: indented-text indent-tabs-mode:
  nil sentence-end-double-space: t fill-column: 70 End:

EX1

    https://mail.python.org/pipermail/python-list/2003-June/210588.html

EX2

    https://mail.python.org/pipermail/python-list/2000-May/034685.html

EX3

    http://groups.google.com/group/comp.lang.python/browse_frm/thread/6ae8c6add913635a/de40d4ffe9bd4304?lnk=gst&q=type+declarations&rnum=6

[1] Though, if you know the language in question, it may not be that
unrelated.

[2] Well, it would, if there was a Perl 6.

[3] Since the name TypeError is already in use, this name has been
chosen for obvious reasons.

[4] The encoding in which the code is written is read from a standard
coding cookie. There will also be an autodetection mechanism, invoked by
from __future__ import encoding_hell.

license: Public Domain


text:
PEP: 3147 Title: PYC Repository Directories Author: Barry Warsaw
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 16-Dec-2009 Python-Version: 3.2 Post-History:
30-Jan-2010, 25-Feb-2010, 03-Mar-2010, 12-Apr-2010 Resolution:
https://mail.python.org/pipermail/python-dev/2010-April/099414.html

Abstract

This PEP describes an extension to Python's import mechanism which
improves sharing of Python source code files among multiple installed
different versions of the Python interpreter. It does this by allowing
more than one byte compilation file (.pyc files) to be co-located with
the Python source file (.py file). The extension described here can also
be used to support different Python compilation caches, such as JIT
output that may be produced by an Unladen Swallow (PEP 3146) enabled C
Python.

Background

CPython compiles its source code into "byte code", and for performance
reasons, it caches this byte code on the file system whenever the source
file has changes. This makes loading of Python modules much faster
because the compilation phase can be bypassed. When your source file is
foo.py, CPython caches the byte code in a foo.pyc file right next to the
source.

Byte code files contain two 32-bit big-endian numbers followed by the
marshaled[1] code object. The 32-bit numbers represent a magic number
and a timestamp. The magic number changes whenever Python changes the
byte code format, e.g. by adding new byte codes to its virtual machine.
This ensures that pyc files built for previous versions of the VM won't
cause problems. The timestamp is used to make sure that the pyc file
match the py file that was used to create it. When either the magic
number or timestamp do not match, the py file is recompiled and a new
pyc file is written.

In practice, it is well known that pyc files are not compatible across
Python major releases. A reading of import.c[2] in the Python source
code proves that within recent memory, every new CPython major release
has bumped the pyc magic number.

Rationale

Linux distributions such as Ubuntu[3] and Debian[4] provide more than
one Python version at the same time to their users. For example, Ubuntu
9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with
Python 2.6 being the default.

This causes a conflict for third party Python source files installed by
the system, because you cannot compile a single Python source file for
more than one Python version at a time. When Python finds a pyc file
with a non-matching magic number, it falls back to the slower process of
recompiling the source. Thus if your system installed a
/usr/share/python/foo.py, two different versions of Python would fight
over the pyc file and rewrite it each time the source is compiled. (The
standard library is unaffected by this, since multiple versions of the
stdlib are installed on such distributions..)

Furthermore, in order to ease the burden on operating system packagers
for these distributions, the distribution packages do not contain Python
version numbers[5]; they are shared across all Python versions installed
on the system. Putting Python version numbers in the packages would be a
maintenance nightmare, since all the packages - and their dependencies -
would have to be updated every time a new Python release was added or
removed from the distribution. Because of the sheer number of packages
available, this amount of work is infeasible.

(PEP 384 has been proposed to address binary compatibility issues of
third party extension modules across different versions of Python.)

Because these distributions cannot share pyc files, elaborate mechanisms
have been developed to put the resulting pyc files in non-shared
locations while the source code is still shared. Examples include the
symlink-based Debian regimes python-support[6] and python-central[7].
These approaches make for much more complicated, fragile, inscrutable,
and fragmented policies for delivering Python applications to a wide
range of users. Arguably more users get Python from their operating
system vendor than from upstream tarballs. Thus, solving this pyc
sharing problem for CPython is a high priority for such vendors.

This PEP proposes a solution to this problem.

Proposal

Python's import machinery is extended to write and search for byte code
cache files in a single directory inside every Python package directory.
This directory will be called __pycache__.

Further, pyc file names will contain a magic string (called a "tag")
that differentiates the Python version they were compiled for. This
allows multiple byte compiled cache files to co-exist for a single
Python source file.

The magic tag is implementation defined, but should contain the
implementation name and a version number shorthand, e.g. cpython-32. It
must be unique among all versions of Python, and whenever the magic
number is bumped, a new magic tag must be defined. An example pyc file
for Python 3.2 is thus foo.cpython-32.pyc.

The magic tag is available in the imp module via the get_tag() function.
This is parallel to the imp.get_magic() function.

This scheme has the added benefit of reducing the clutter in a Python
package directory.

When a Python source file is imported for the first time, a __pycache__
directory will be created in the package directory, if one does not
already exist. The pyc file for the imported source will be written to
the __pycache__ directory, using the magic-tag formatted name. If either
the creation of the __pycache__ directory or the pyc file inside that
fails, the import will still succeed, just as it does in a pre-PEP 3147
world.

If the py source file is missing, the pyc file inside __pycache__ will
be ignored. This eliminates the problem of accidental stale pyc file
imports.

For backward compatibility, Python will still support pyc-only
distributions, however it will only do so when the pyc file lives in the
directory where the py file would have been, i.e. not in the __pycache__
directory. pyc file outside of __pycache__ will only be imported if the
py source file is missing.

Tools such as py_compile[8] and compileall[9] will be extended to create
PEP 3147 formatted layouts automatically, but will have an option to
create pyc-only distribution layouts.

Examples

What would this look like in practice?

Let's say we have a Python package named alpha which contains a
sub-package name beta. The source directory layout before byte
compilation might look like this:

    alpha/
        __init__.py
        one.py
        two.py
        beta/
            __init__.py
            three.py
            four.py

After byte compiling this package with Python 3.2, you would see the
following layout:

    alpha/
        __pycache__/
            __init__.cpython-32.pyc
            one.cpython-32.pyc
            two.cpython-32.pyc
        __init__.py
        one.py
        two.py
        beta/
            __pycache__/
                __init__.cpython-32.pyc
                three.cpython-32.pyc
                four.cpython-32.pyc
            __init__.py
            three.py
            four.py

Note: listing order may differ depending on the platform.

Let's say that two new versions of Python are installed, one is Python
3.3 and another is Unladen Swallow. After byte compilation, the file
system would look like this:

    alpha/
        __pycache__/
            __init__.cpython-32.pyc
            __init__.cpython-33.pyc
            __init__.unladen-10.pyc
            one.cpython-32.pyc
            one.cpython-33.pyc
            one.unladen-10.pyc
            two.cpython-32.pyc
            two.cpython-33.pyc
            two.unladen-10.pyc
        __init__.py
        one.py
        two.py
        beta/
            __pycache__/
                __init__.cpython-32.pyc
                __init__.cpython-33.pyc
                __init__.unladen-10.pyc
                three.cpython-32.pyc
                three.cpython-33.pyc
                three.unladen-10.pyc
                four.cpython-32.pyc
                four.cpython-33.pyc
                four.unladen-10.pyc
            __init__.py
            three.py
            four.py

As you can see, as long as the Python version identifier string is
unique, any number of pyc files can co-exist. These identifier strings
are described in more detail below.

A nice property of this layout is that the __pycache__ directories can
generally be ignored, such that a normal directory listing would show
something like this:

    alpha/
        __pycache__/
        __init__.py
        one.py
        two.py
        beta/
            __pycache__/
            __init__.py
            three.py
            four.py

This is much less cluttered than even today's Python.

Python behavior

When Python searches for a module to import (say foo), it may find one
of several situations. As per current Python rules, the term "matching
pyc" means that the magic number matches the current interpreter's magic
number, and the source file's timestamp matches the timestamp in the pyc
file exactly.

Case 0: The steady state

When Python is asked to import module foo, it searches for a foo.py file
(or foo package, but that's not important for this discussion) along its
sys.path. If found, Python looks to see if there is a matching
__pycache__/foo.<magic>.pyc file, and if so, that pyc file is loaded.

Case 1: The first import

When Python locates the foo.py, if the __pycache__/foo.<magic>.pyc file
is missing, Python will create it, also creating the __pycache__
directory if necessary. Python will parse and byte compile the foo.py
file and save the byte code in __pycache__/foo.<magic>.pyc.

Case 2: The second import

When Python is asked to import module foo a second time (in a different
process of course), it will again search for the foo.py file along its
sys.path. When Python locates the foo.py file, it looks for a matching
__pycache__/foo.<magic>.pyc and finding this, it reads the byte code and
continues as usual.

Case 3: __pycache/foo.<magic>.pyc with no source

It's possible that the foo.py file somehow got removed, while leaving
the cached pyc file still on the file system. If the
__pycache__/foo.<magic>.pyc file exists, but the foo.py file used to
create it does not, Python will raise an ImportError when asked to
import foo. In other words, Python will not import a pyc file from the
cache directory unless the source file exists.

Case 4: legacy pyc files and source-less imports

Python will ignore all legacy pyc files when a source file exists next
to it. In other words, if a foo.pyc file exists next to the foo.py file,
the pyc file will be ignored in all cases

In order to continue to support source-less distributions though, if the
source file is missing, Python will import a lone pyc file if it lives
where the source file would have been.

Case 5: read-only file systems

When the source lives on a read-only file system, or the __pycache__
directory or pyc file cannot otherwise be written, all the same rules
apply. This is also the case when __pycache__ happens to be written with
permissions which do not allow for writing containing pyc files.

Flow chart

Here is a flow chart describing how modules are loaded:

[image]

Alternative Python implementations

Alternative Python implementations such as Jython[10], IronPython [11],
PyPy[12], Pynie[13], and Unladen Swallow can also use the __pycache__
directory to store whatever compilation artifacts make sense for their
platforms. For example, Jython could store the class file for the module
in __pycache__/foo.jython-32.class.

Implementation strategy

This feature is targeted for Python 3.2, solving the problem for those
and all future versions. It may be back-ported to Python 2.7. Vendors
are free to backport the changes to earlier distributions as they see
fit. For backports of this feature to Python 2, when the -U flag is
used, a file such as foo.cpython-27u.pyc can be written.

Effects on existing code

Adoption of this PEP will affect existing code and idioms, both inside
Python and outside. This section enumerates some of these effects.

Detecting PEP 3147 availability

The easiest way to detect whether your version of Python provides PEP
3147 functionality is to do the following check:

    >>> import imp
    >>> has3147 = hasattr(imp, 'get_tag')

------------------------------------------------------------------------

In Python 3, when you import a module, its __file__ attribute points to
its source py file (in Python 2, it points to the pyc file). A package's
__file__ points to the py file for its __init__.py. E.g.:

    >>> import foo
    >>> foo.__file__
    'foo.py'
    # baz is a package
    >>> import baz
    >>> baz.__file__
    'baz/__init__.py'

Nothing in this PEP would change the semantics of __file__.

This PEP proposes the addition of an __cached__ attribute to modules,
which will always point to the actual pyc file that was read or written.
When the environment variable $PYTHONDONTWRITEBYTECODE is set, or the -B
option is given, or if the source lives on a read-only filesystem, then
the __cached__ attribute will point to the location that the pyc file
would have been written to if it didn't exist. This location of course
includes the __pycache__ subdirectory in its path.

For alternative Python implementations which do not support pyc files,
the __cached__ attribute may point to whatever information makes sense.
E.g. on Jython, this might be the .class file for the module:
__pycache__/foo.jython-32.class. Some implementations may use multiple
compiled files to create the module, in which case __cached__ may be a
tuple. The exact contents of __cached__ are Python implementation
specific.

It is recommended that when nothing sensible can be calculated,
implementations should set the __cached__ attribute to None.

py_compile and compileall

Python comes with two modules, py_compile[14] and compileall [15] which
support compiling Python modules external to the built-in import
machinery. py_compile in particular has intimate knowledge of byte
compilation, so these will be updated to understand the new layout. The
-b flag is added to compileall for writing legacy .pyc byte-compiled
file path names.

bdist_wininst and the Windows installer

These tools also compile modules explicitly on installation. If they do
not use py_compile and compileall, then they would also have to be
modified to understand the new layout.

File extension checks

There exists some code which checks for files ending in .pyc and simply
chops off the last character to find the matching .py file. This code
will obviously fail once this PEP is implemented.

To support this use case, we'll add two new methods to the imp
package[16]:

-   imp.cache_from_source(py_path) -> pyc_path
-   imp.source_from_cache(pyc_path) -> py_path

Alternative implementations are free to override these functions to
return reasonable values based on their own support for this PEP. These
methods are allowed to return None when the implementation (or PEP 302
loader in effect) for whatever reason cannot calculate the appropriate
file name. They should not raise exceptions.

Backports

For versions of Python earlier than 3.2 (and possibly 2.7), it is
possible to backport this PEP. However, in Python 3.2 (and possibly
2.7), this behavior will be turned on by default, and in fact, it will
replace the old behavior. Backports will need to support the old layout
by default. We suggest supporting PEP 3147 through the use of an
environment variable called $PYTHONENABLECACHEDIR or the command line
switch -Xenablecachedir to enable the feature.

Makefiles and other dependency tools

Makefiles and other tools which calculate dependencies on .pyc files
(e.g. to byte-compile the source if the .pyc is missing) will have to be
updated to check the new paths.

Alternatives

This section describes some alternative approaches or details that were
considered and rejected during the PEP's development.

Hexadecimal magic tags

pyc files inside of the __pycache__ directories contain a magic tag in
their file names. These are mnemonic tags for the actual magic numbers
used by the importer. We could have used the hexadecimal
representation[17] of the binary magic number as a unique identifier.
For example, in Python 3.2:

    >>> from binascii import hexlify
    >>> from imp import get_magic
    >>> 'foo.{}.pyc'.format(hexlify(get_magic()).decode('ascii'))
    'foo.580c0d0a.pyc'

This isn't particularly human friendly though, thus the magic tag
proposed in this PEP.

PEP 304

There is some overlap between the goals of this PEP and PEP 304, which
has been withdrawn. However PEP 304 would allow a user to create a
shadow file system hierarchy in which to store pyc files. This concept
of a shadow hierarchy for pyc files could be used to satisfy the aims of
this PEP. Although the PEP 304 does not indicate why it was withdrawn,
shadow directories have a number of problems. The location of the shadow
pyc files would not be easily discovered and would depend on the proper
and consistent use of the $PYTHONBYTECODE environment variable both by
the system and by end users. There are also global implications, meaning
that while the system might want to shadow pyc files, users might not
want to, but the PEP defines only an all-or-nothing approach.

As an example of the problem, a common (though fragile) Python idiom for
locating data files is to do something like this:

    from os import dirname, join
    import foo.bar
    data_file = join(dirname(foo.bar.__file__), 'my.dat')

This would be problematic since foo.bar.__file__ will give the location
of the pyc file in the shadow directory, and it may not be possible to
find the my.dat file relative to the source directory from there.

Fat byte compilation files

An earlier version of this PEP described "fat" Python byte code files.
These files would contain the equivalent of multiple pyc files in a
single pyf file, with a lookup table keyed off the appropriate magic
number. This was an extensible file format so that the first 5 parallel
Python implementations could be supported fairly efficiently, but with
extension lookup tables available to scale pyf byte code objects as
large as necessary.

The fat byte compilation files were fairly complex, and inherently
introduced difficult race conditions, so the current simplification of
using directories was suggested. The same problem applies to using zip
files as the fat pyc file format.

Multiple file extensions

The PEP author also considered an approach where multiple thin byte
compiled files lived in the same place, but used different file
extensions to designate the Python version. E.g. foo.pyc25, foo.pyc26,
foo.pyc31 etc. This was rejected because of the clutter involved in
writing so many different files. The multiple extension approach makes
it more difficult (and an ongoing task) to update any tools that are
dependent on the file extension.

.pyc

A proposal was floated to call the __pycache__ directory .pyc or some
other dot-file name. This would have the effect on *nix systems of
hiding the directory. There are many reasons why this was rejected by
the BDFL[18] including the fact that dot-files are only special on some
platforms, and we actually do not want to hide these completely from
users.

Reference implementation

Work on this code is tracked in a Bazaar branch on Launchpad[19] until
it's ready for merge into Python 3.2. The work-in-progress diff can also
be viewed[20] and is updated automatically as new changes are uploaded.

A Rietveld code review issue[21] has been opened as of 2010-04-01 (no,
this is not an April Fools joke :).

References

[21] importlib: https://docs.python.org/3.1/library/importlib.html

ACKNOWLEDGMENTS

Barry Warsaw's original idea was for fat Python byte code files. Martin
von Loewis reviewed an early draft of the PEP and suggested the
simplification to store traditional pyc and pyo files in a directory.
Many other people reviewed early versions of this PEP and provided
useful feedback including but not limited to:

-   David Malcolm
-   Josselin Mouette
-   Matthias Klose
-   Michael Hudson
-   Michael Vogt
-   Piotr Ożarowski
-   Scott Kitterman
-   Toshio Kuratomi

Copyright

This document has been placed in the public domain.

[1] The marshal module: https://docs.python.org/3.1/library/marshal.html

[2] import.c:
https://github.com/python/cpython/blob/v3.2a1/Python/import.c

[3] Ubuntu: https://www.ubuntu.com

[4] Debian: https://www.debian.org

[5] Debian Python Policy:
https://www.debian.org/doc/packaging-manuals/python-policy/

[6] python-support:
https://web.archive.org/web/20100110123824/http://wiki.debian.org/DebianPythonFAQ#Whatispython-support.3F

[7] python-central:
https://web.archive.org/web/20100110123824/http://wiki.debian.org/DebianPythonFAQ#Whatispython-central.3F

[8] py_compile: https://docs.python.org/3.1/library/py_compile.html

[9] compileall: https://docs.python.org/3.1/library/compileall.html

[10] Jython: http://www.jython.org/

[11] IronPython: http://ironpython.net/

[12] PyPy:
https://web.archive.org/web/20100310130136/http://codespeak.net/pypy/dist/pypy/doc/

[13] Pynie: https://code.google.com/archive/p/pynie/

[14] py_compile: https://docs.python.org/3.1/library/py_compile.html

[15] compileall: https://docs.python.org/3.1/library/compileall.html

[16] imp: https://docs.python.org/3.1/library/imp.html

[17] binascii.hexlify():
https://docs.python.org/3.1/library/binascii.html#binascii.hexlify

[18]<EMAIL_ADDRESS>
[19] https://code.launchpad.net/~barry/python/pep3147

[20] https://code.launchpad.net/~barry/python/pep3147/+merge/22648

[21] http://codereview.appspot.com/842043/show


text:
PEP: 366 Title: Main module explicit relative imports Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 01-May-2007 Python-Version: 2.6, 3.0 Post-History:
01-May-2007, 04-Jul-2007, 07-Jul-2007, 23-Nov-2007

Abstract

This PEP proposes a backwards compatible mechanism that permits the use
of explicit relative imports from executable modules within packages.
Such imports currently fail due to an awkward interaction between PEP
328 and PEP 338.

By adding a new module level attribute, this PEP allows relative imports
to work automatically if the module is executed using the -m switch. A
small amount of boilerplate in the module itself will allow the relative
imports to work when the file is executed by name.

Guido accepted the PEP in November 2007[1].

Proposed Change

The major proposed change is the introduction of a new module level
attribute, __package__. When it is present, relative imports will be
based on this attribute rather than the module __name__ attribute.

As with the current __name__ attribute, setting __package__ will be the
responsibility of the PEP 302 loader used to import a module. Loaders
which use imp.new_module() to create the module object will have the new
attribute set automatically to None. When the import system encounters
an explicit relative import in a module without __package__ set (or with
it set to None), it will calculate and store the correct value
(__name__.rpartition('.')[0] for normal modules and __name__ for package
initialisation modules). If __package__ has already been set then the
import system will use it in preference to recalculating the package
name from the __name__ and __path__ attributes.

The runpy module will explicitly set the new attribute, basing it off
the name used to locate the module to be executed rather than the name
used to set the module's __name__ attribute. This will allow relative
imports to work correctly from main modules executed with the -m switch.

When the main module is specified by its filename, then the __package__
attribute will be set to None. To allow relative imports when the module
is executed directly, boilerplate similar to the following would be
needed before the first relative import statement:

    if __name__ == "__main__" and __package__ is None:
        __package__ = "expected.package.name"

Note that this boilerplate is sufficient only if the top level package
is already accessible via sys.path. Additional code that manipulates
sys.path would be needed in order for direct execution to work without
the top level package already being importable.

This approach also has the same disadvantage as the use of absolute
imports of sibling modules - if the script is moved to a different
package or subpackage, the boilerplate will need to be updated manually.
It has the advantage that this change need only be made once per file,
regardless of the number of relative imports.

Note that setting __package__ to the empty string explicitly is
permitted, and has the effect of disabling all relative imports from
that module (since the import machinery will consider it to be a top
level module in that case). This means that tools like runpy do not need
to provide special case handling for top level modules when setting
__package__.

Rationale for Change

The current inability to use explicit relative imports from the main
module is the subject of at least one open SF bug report (#1510172)[2],
and has most likely been a factor in at least a few queries on
comp.lang.python (such as Alan Isaac's question in[3]).

This PEP is intended to provide a solution which permits explicit
relative imports from main modules, without incurring any significant
costs during interpreter startup or normal module import.

The section in PEP 338 on relative imports and the main module provides
further details and background on this problem.

Reference Implementation

Rev 47142 in SVN implemented an early variant of this proposal which
stored the main module's real module name in the __module_name__
attribute. It was reverted due to the fact that 2.5 was already in beta
by that time.

Patch 1487[4] is the proposed implementation for this PEP.

Alternative Proposals

PEP 3122 proposed addressing this problem by changing the way the main
module is identified. That's a significant compatibility cost to incur
to fix something that is a pretty minor bug in the overall scheme of
things, and the PEP was rejected[5].

The advantage of the proposal in this PEP is that its only impact on
normal code is the small amount of time needed to set the extra
attribute when importing a module. Relative imports themselves should be
sped up fractionally, as the package name is cached in the module
globals, rather than having to be worked out again for each relative
import.

References

Copyright

This document has been placed in the public domain.

[1] Acceptance of the PEP
(https://mail.python.org/pipermail/python-dev/2007-November/075475.html)

[2] Absolute/relative import not working?
(https://github.com/python/cpython/issues/43535)

[3] c.l.p. question about modules and relative imports
(http://groups.google.com/group/comp.lang.python/browse_thread/thread/c44c769a72ca69fa/)

[4] PEP 366 implementation patch
(https://github.com/python/cpython/issues/45828)

[5] Guido's rejection of PEP 3122
(https://mail.python.org/pipermail/python-3000/2007-April/006793.html)


text:
PEP: 495 Title: Local Time Disambiguation Version: $Revision$
Last-Modified: $Date$ Author: Alexander Belopolsky
<EMAIL_ADDRESS>Tim Peters<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards
Track Content-Type: text/x-rst Created: 02-Aug-2015 Python-Version: 3.6
Resolution:
https://mail.python.org/pipermail/datetime-sig/2015-September/000900.html

Abstract

This PEP adds a new attribute fold to instances of the datetime.time and
datetime.datetime classes that can be used to differentiate between two
moments in time for which local times are the same. The allowed values
for the fold attribute will be 0 and 1 with 0 corresponding to the
earlier and 1 to the later of the two possible readings of an ambiguous
local time.

Rationale

In most world locations, there have been and will be times when local
clocks are moved back.[1] In those times, intervals are introduced in
which local clocks show the same time twice in the same day. In these
situations, the information displayed on a local clock (or stored in a
Python datetime instance) is insufficient to identify a particular
moment in time. The proposed solution is to add an attribute to the
datetime instances taking values of 0 and 1 that will enumerate the two
ambiguous times.

[A cartoon of a strong man struggling to stop the hands of a large clock.
The caption reads: You can't stop time... but you can turn it back one
hour at 2 a.m. Oct. 28 when daylight-saving time ends and standard time
begins."
:width: 30%]

Terminology

When clocks are moved back, we say that a fold[2] is created in time.
When the clocks are moved forward, a gap is created. A local time that
falls in the fold is called ambiguous. A local time that falls in the
gap is called missing.

Proposal

The "fold" attribute

We propose adding an attribute called fold to instances of the
datetime.time and datetime.datetime classes. This attribute should have
the value 0 for all instances except those that represent the second
(chronologically) moment in time in an ambiguous case. For those
instances, the value will be 1.[3]

Affected APIs

Attributes

Instances of datetime.time and datetime.datetime classes will get a new
attribute fold with two possible values: 0 and 1.

Constructors

The __new__ methods of the datetime.time and datetime.datetime classes
will get a new keyword-only argument called fold with the default value
0. The value of the fold argument will be used to initialize the value
of the fold attribute in the returned instance.

Methods

The replace() methods of the datetime.time and datetime.datetime classes
will get a new keyword-only argument called fold. It will behave
similarly to the other replace() arguments: if the fold argument is
specified and given a value 0 or 1, the new instance returned by
replace() will have its fold attribute set to that value. In CPython,
any non-integer value of fold will raise a TypeError, but other
implementations may allow the value None to behave the same as when fold
is not given.[4] (This is a nod to the existing difference in treatment
of None arguments in other positions of this method across Python
implementations; it is not intended to leave the door open for future
alternative interpretation of fold=None.) If the fold argument is not
specified, the original value of the fold attribute is copied to the
result.

C-API

Access macros will be defined to extract the value of fold from
PyDateTime_DateTime and PyDateTime_Time objects.

    int PyDateTime_DATE_GET_FOLD(PyDateTime_DateTime *o)

Return the value of fold as a C int.

    int PyDateTime_TIME_GET_FOLD(PyDateTime_Time *o)

Return the value of fold as a C int.

New constructors will be defined that will take an additional argument
to specify the value of fold in the created instance:

    PyObject* PyDateTime_FromDateAndTimeAndFold(
        int year, int month, int day, int hour, int minute,
        int second, int usecond, int fold)

Return a datetime.datetime object with the specified year, month, day,
hour, minute, second, microsecond and fold.

    PyObject* PyTime_FromTimeAndFold(
        int hour, int minute, int second, int usecond, int fold)

Return a datetime.time object with the specified hour, minute, second,
microsecond and fold.

Affected Behaviors

What time is it?

The datetime.now() method called without arguments will set fold=1 when
returning the second of the two ambiguous times in a system local time
fold. When called with a tzinfo argument, the value of the fold will be
determined by the tzinfo.fromutc() implementation. When an instance of
the datetime.timezone class (the stdlib's fixed-offset tzinfo subclass,
e.g. datetime.timezone.utc) is passed as tzinfo, the returned datetime
instance will always have fold=0. The datetime.utcnow() method is
unaffected.

Conversion from naive to aware

A new feature is proposed to facilitate conversion from naive datetime
instances to aware.

The astimezone() method will now work for naive self. The system local
timezone will be assumed in this case and the fold flag will be used to
determine which local timezone is in effect in the ambiguous case.

For example, on a system set to US/Eastern timezone:

    >>> dt = datetime(2014, 11, 2, 1, 30)
    >>> dt.astimezone().strftime('%D %T %Z%z')
    '11/02/14 01:30:00 EDT-0400'
    >>> dt.replace(fold=1).astimezone().strftime('%D %T %Z%z')
    '11/02/14 01:30:00 EST-0500'

An implication is that datetime.now(tz) is fully equivalent to
datetime.now().astimezone(tz) (assuming tz is an instance of a post-PEP
tzinfo implementation, i.e. one that correctly handles and sets fold).

Conversion from POSIX seconds from EPOCH

The fromtimestamp() static method of datetime.datetime will set the fold
attribute appropriately in the returned object.

For example, on a system set to US/Eastern timezone:

    >>> datetime.fromtimestamp(1414906200)
    datetime.datetime(2014, 11, 2, 1, 30)
    >>> datetime.fromtimestamp(1414906200 + 3600)
    datetime.datetime(2014, 11, 2, 1, 30, fold=1)

Conversion to POSIX seconds from EPOCH

The timestamp() method of datetime.datetime will return different values
for datetime.datetime instances that differ only by the value of their
fold attribute if and only if these instances represent an ambiguous or
a missing time.

When a datetime.datetime instance dt represents an ambiguous time, there
are two values s0 and s1 such that:

    datetime.fromtimestamp(s0) == datetime.fromtimestamp(s1) == dt

(This is because == disregards the value of fold -- see below.)

In this case, dt.timestamp() will return the smaller of s0 and s1 values
if dt.fold == 0 and the larger otherwise.

For example, on a system set to US/Eastern timezone:

    >>> datetime(2014, 11, 2, 1, 30, fold=0).timestamp()
<PHONE_NUMBER>.0
    >>> datetime(2014, 11, 2, 1, 30, fold=1).timestamp()
<PHONE_NUMBER>.0

When a datetime.datetime instance dt represents a missing time, there is
no value s for which:

    datetime.fromtimestamp(s) == dt

but we can form two "nice to know" values of s that differ by the size
of the gap in seconds. One is the value of s that would correspond to dt
in a timezone where the UTC offset is always the same as the offset
right before the gap and the other is the similar value but in a
timezone the UTC offset is always the same as the offset right after the
gap.

The value returned by dt.timestamp() given a missing dt will be the
greater of the two "nice to know" values if dt.fold == 0 and the smaller
otherwise. (This is not a typo -- it's intentionally backwards from the
rule for ambiguous times.)

For example, on a system set to US/Eastern timezone:

    >>> datetime(2015, 3, 8, 2, 30, fold=0).timestamp()
<PHONE_NUMBER>.0
    >>> datetime(2015, 3, 8, 2, 30, fold=1).timestamp()
<PHONE_NUMBER>.0

Aware datetime instances

Users of pre-PEP implementations of tzinfo will not see any changes in
the behavior of their aware datetime instances. Two such instances that
differ only by the value of the fold attribute will not be
distinguishable by any means other than an explicit access to the fold
value. (This is because these pre-PEP implementations are not using the
fold attribute.)

On the other hand, if an object's tzinfo is set to a fold-aware
implementation, then in a fold or gap the value of fold will affect the
result of several methods: utcoffset(), dst(), tzname(), astimezone(),
strftime() (if the "%Z" or "%z" directive is used in the format
specification), isoformat(), and timetuple().

Combining and splitting date and time

The datetime.datetime.combine() method will copy the value of the fold
attribute to the resulting datetime.datetime instance.

The datetime.datetime.time() method will copy the value of the fold
attribute to the resulting datetime.time instance.

Pickles

The value of the fold attribute will only be saved in pickles created
with protocol version 4 (introduced in Python 3.4) or greater.

Pickle sizes for the datetime.datetime and datetime.time objects will
not change. The fold value will be encoded in the first bit of the 3rd
byte of the datetime.datetime pickle payload; and in the first bit of
the 1st byte of the datetime.time payload. In the current implementation
these bytes are used to store the month (1-12) and hour (0-23) values
and the first bit is always 0. We picked these bytes because they are
the only bytes that are checked by the current unpickle code. Thus
loading post-PEP fold=1 pickles in a pre-PEP Python will result in an
exception rather than an instance with out of range components.

Implementations of tzinfo in the Standard Library

No new implementations of datetime.tzinfo abstract class are proposed in
this PEP. The existing (fixed offset) timezones do not introduce
ambiguous local times and their utcoffset() implementation will return
the same constant value as they do now regardless of the value of fold.

The basic implementation of fromutc() in the abstract datetime.tzinfo
class will not change. It is currently not used anywhere in the stdlib
because the only included tzinfo implementation (the datetime.timezone
class implementing fixed offset timezones) overrides fromutc(). Keeping
the default implementation unchanged has the benefit that pre-PEP 3rd
party implementations that inherit the default fromutc() are not
accidentally affected.

Guidelines for New tzinfo Implementations

Implementors of concrete datetime.tzinfo subclasses who want to support
variable UTC offsets (due to DST and other causes) should follow these
guidelines.

Ignorance is Bliss

New implementations of utcoffset(), tzname() and dst() methods should
ignore the value of fold unless they are called on the ambiguous or
missing times.

In the Fold

New subclasses should override the base-class fromutc() method and
implement it so that in all cases where two different UTC times u0 and
u1 (u0 <u1) correspond to the same local time t, fromutc(u0) will return
an instance with fold=0 and fromutc(u1) will return an instance with
fold=1. In all other cases the returned instance should have fold=0.

The utcoffset(), tzname() and dst() methods should use the value of the
fold attribute to determine whether an otherwise ambiguous time t
corresponds to the time before or after the transition. By definition,
utcoffset() is greater before and smaller after any transition that
creates a fold. The values returned by tzname() and dst() may or may not
depend on the value of the fold attribute depending on the kind of the
transition.

[Diagram of relationship between UTC and local time around a
fall-back transition – see full description on page.
:width: 60%
:class: invert-in-dark-mode]

The sketch above illustrates the relationship between the UTC and local
time around a fall-back transition. The zig-zag line is a graph of the
function implemented by fromutc(). Two intervals on the UTC axis
adjacent to the transition point and having the size of the time shift
at the transition are mapped to the same interval on the local axis. New
implementations of fromutc() method should set the fold attribute to 1
when self is in the region marked in yellow on the UTC axis. (All
intervals should be treated as closed on the left and open on the
right.)

Mind the Gap

The fromutc() method should never produce a time in the gap.

If the utcoffset(), tzname() or dst() method is called on a local time
that falls in a gap, the rules in effect before the transition should be
used if fold=0. Otherwise, the rules in effect after the transition
should be used.

[Diagram of relationship between UTC and local time around a
spring-forward transition – see full description on page.
:width: 60%
:class: invert-in-dark-mode]

The sketch above illustrates the relationship between the UTC and local
time around a spring-forward transition. At the transition, the local
clock is advanced skipping the times in the gap. For the purposes of
determining the values of utcoffset(), tzname() and dst(), the line
before the transition is extended forward to find the UTC time
corresponding to the time in the gap with fold=0 and for instances with
fold=1, the line after the transition is extended back.

Summary of Rules at a Transition

On ambiguous/missing times utcoffset() should return values according to
the following table:

+-----------------+----------------+-----------------------------+
|                 |   fold=0       |   fold=1                    |
+=================+================+=============================+
|   Fold          |   oldoff       |   newoff = oldoff - delta   |
+-----------------+----------------+-----------------------------+
|   Gap           |   oldoff       |   newoff = oldoff + delta   |
+-----------------+----------------+-----------------------------+

where oldoff (newoff) is the UTC offset before (after) the transition
and delta is the absolute size of the fold or the gap.

Note that the interpretation of the fold attribute is consistent in the
fold and gap cases. In both cases, fold=0 (fold=1) means use fromutc()
line before (after) the transition to find the UTC time. Only in the
"Fold" case, the UTC times u0 and u1 are "real" solutions for the
equation fromutc(u) == t, while in the "Gap" case they are "imaginary"
solutions.

The DST Transitions

On a missing time introduced at the start of DST, the values returned by
utcoffset() and dst() methods should be as follows

+-----------------+----------------+------------------+
|                 |   fold=0       |   fold=1         |
+=================+================+==================+
|   utcoffset()   |   stdoff       |   stdoff +       |
|                 |                |   dstoff         |
+-----------------+----------------+------------------+
|   dst()         |   zero         |   dstoff         |
+-----------------+----------------+------------------+

On an ambiguous time introduced at the end of DST, the values returned
by utcoffset() and dst() methods should be as follows

+-----------------+----------------+------------------+
|                 |   fold=0       |   fold=1         |
+=================+================+==================+
| utcoffset()     | stdoff +       |   stdoff         |
|                 | dstoff         |                  |
+-----------------+----------------+------------------+
|   dst()         |   dstoff       |   zero           |
+-----------------+----------------+------------------+

where stdoff is the standard (non-DST) offset, dstoff is the DST
correction (typically dstoff = timedelta(hours=1)) and
zero = timedelta(0).

Temporal Arithmetic and Comparison Operators

  In mathematicks he was greater
  Than Tycho Brahe, or Erra Pater:
  For he, by geometric scale,
  Could take the size of pots of ale;
  Resolve, by sines and tangents straight,
  If bread or butter wanted weight,
  And wisely tell what hour o' th' day
  The clock does strike by algebra.

    -- "Hudibras" by Samuel Butler

The value of the fold attribute will be ignored in all operations with
naive datetime instances. As a consequence, naive datetime.datetime or
datetime.time instances that differ only by the value of fold will
compare as equal. Applications that need to differentiate between such
instances should check the value of fold explicitly or convert those
instances to a timezone that does not have ambiguous times (such as
UTC).

The value of fold will also be ignored whenever a timedelta is added to
or subtracted from a datetime instance which may be either aware or
naive. The result of addition (subtraction) of a timedelta to (from) a
datetime will always have fold set to 0 even if the original datetime
instance had fold=1.

No changes are proposed to the way the difference t - s is computed for
datetime instances t and s. If both instances are naive or t.tzinfo is
the same instance as s.tzinfo (t.tzinfo is s.tzinfo evaluates to True)
then t - s is a timedelta d such that s + d == t. As explained in the
previous paragraph, timedelta addition ignores both fold and tzinfo
attributes and so does intra-zone or naive datetime subtraction.

Naive and intra-zone comparisons will ignore the value of fold and
return the same results as they do now. (This is the only way to
preserve backward compatibility. If you need an aware intra-zone
comparison that uses the fold, convert both sides to UTC first.)

The inter-zone subtraction will be defined as it is now: t - s is
computed as
(t - t.utcoffset()) - (s - s.utcoffset()).replace(tzinfo=t.tzinfo), but
the result will depend on the values of t.fold and s.fold when either
t.tzinfo or s.tzinfo is post-PEP.[5]

Aware datetime Equality Comparison

The aware datetime comparison operators will work the same as they do
now, with results indirectly affected by the value of fold whenever the
utcoffset() value of one of the operands depends on it, with one
exception. Whenever one or both of the operands in inter-zone comparison
is such that its utcoffset() depends on the value of its fold fold
attribute, the result is False.[6]

Formally, t == s when t.tzinfo is s.tzinfo evaluates to False can be
defined as follows. Let toutc(t, fold) be a function that takes an aware
datetime instance t and returns a naive instance representing the same
time in UTC assuming a given value of fold:

    def toutc(t, fold):
        u = t - t.replace(fold=fold).utcoffset()
        return u.replace(tzinfo=None)

Then t == s is equivalent to

    toutc(t, fold=0) == toutc(t, fold=1) == toutc(s, fold=0) == toutc(s, fold=1)

Backward and Forward Compatibility

This proposal will have little effect on the programs that do not read
the fold flag explicitly or use tzinfo implementations that do. The only
visible change for such programs will be that conversions to and from
POSIX timestamps will now round-trip correctly (up to floating point
rounding). Programs that implemented a work-around to the old incorrect
behavior may need to be modified.

Pickles produced by older programs will remain fully forward compatible.
Only datetime/time instances with fold=1 pickled in the new versions
will become unreadable by the older Python versions. Pickles of
instances with fold=0 (which is the default) will remain unchanged.

Questions and Answers

Why not call the new flag "isdst"?

A non-technical answer

-   Alice: Bob - let's have a stargazing party at 01:30 AM tomorrow!
-   Bob: Should I presume initially that Daylight Saving Time is or is
    not in effect for the specified time?
-   Alice: Huh?

------------------------------------------------------------------------

-   Bob: Alice - let's have a stargazing party at 01:30 AM tomorrow!
-   Alice: You know, Bob, 01:30 AM will happen twice tomorrow. Which
    time do you have in mind?
-   Bob: I did not think about it, but let's pick the first.

------------------------------------------------------------------------

(same characters, an hour later)

------------------------------------------------------------------------

-   Bob: Alice - this Py-O-Clock gadget of mine asks me to choose
    between fold=0 and fold=1 when I set it for tomorrow 01:30 AM. What
    should I do?
-   Alice: I've never hear of a Py-O-Clock, but I guess fold=0 is the
    first 01:30 AM and fold=1 is the second.

A technical reason

While the tm_isdst field of the time.struct_time object can be used to
disambiguate local times in the fold, the semantics of such
disambiguation are completely different from the proposal in this PEP.

The main problem with the tm_isdst field is that it is impossible to
know what value is appropriate for tm_isdst without knowing the details
about the time zone that are only available to the tzinfo
implementation. Thus while tm_isdst is useful in the output of methods
such as time.localtime, it is cumbersome as an input of methods such as
time.mktime.

If the programmer misspecified a non-negative value of tm_isdst to
time.mktime, the result will be time that is 1 hour off and since there
is rarely a way to know anything about DST before a call to time.mktime
is made, the only sane choice is usually tm_isdst=-1.

Unlike tm_isdst, the proposed fold attribute has no effect on the
interpretation of the datetime instance unless without that attribute
two (or no) interpretations are possible.

Since it would be very confusing to have something called isdst that
does not have the same semantics as tm_isdst, we need a different name.
Moreover, the datetime.datetime class already has a method called dst()
and if we called fold "isdst", we would necessarily have situations when
"isdst" is zero but dst() is not or the other way around.

Why "fold"?

Suggested by Guido van Rossum and favored by one (but initially
disfavored by another) author. A consensus was reached after the allowed
values for the attribute were changed from False/True to 0/1. The noun
"fold" has correct connotations and easy mnemonic rules, but at the same
time does not invite unbased assumptions.

What is "first"?

This was a working name of the attribute chosen initially because the
obvious alternative ("second") conflicts with the existing attribute. It
was rejected mostly on the grounds that it would make True a default
value.

The following alternative names have also been considered:

later

    A close contender to "fold". One author dislikes it because it is
    confusable with equally fitting "latter," but in the age of
    auto-completion everywhere this is a small consideration. A stronger
    objection may be that in the case of missing time, we will have
    later=True instance converted to an earlier time by
    .astimezone(timezone.utc) that that with later=False. Yet again,
    this can be interpreted as a desirable indication that the original
    time is invalid.

which

    The original placeholder name for the localtime function branch
    index was independently proposed for the name of the disambiguation
    attribute and received some support.

repeated

    Did not receive any support on the mailing list.

ltdf

    (Local Time Disambiguation Flag) - short and no-one will attempt to
    guess what it means without reading the docs. (This abbreviation was
    used in PEP discussions with the meaning ltdf=False is the earlier
    by those who didn't want to endorse any of the alternatives.)

Are two values enough?

Several reasons have been raised to allow a None or -1 value for the
fold attribute: backward compatibility, analogy with tm_isdst and strict
checking for invalid times.

Backward Compatibility

It has been suggested that backward compatibility can be improved if the
default value of the fold flag was None which would signal that pre-PEP
behavior is requested. Based on the analysis below, we believe that the
proposed changes with the fold=0 default are sufficiently backward
compatible.

This PEP provides only three ways for a program to discover that two
otherwise identical datetime instances have different values of fold:
(1) an explicit check of the fold attribute; (2) if the instances are
naive - conversion to another timezone using the astimezone() method;
and (3) conversion to float using the timestamp() method.

Since fold is a new attribute, the first option is not available to the
existing programs. Note that option (2) only works for naive datetimes
that happen to be in a fold or a gap in the system time zone. In all
other cases, the value of fold will be ignored in the conversion unless
the instances use a fold-aware tzinfo which would not be available in a
pre-PEP program. Similarly, the astimezone() called on a naive instance
will not be available in such program because astimezone() does not
currently work with naive datetimes.

This leaves us with only one situation where an existing program can
start producing different results after the implementation of this PEP:
when a datetime.timestamp() method is called on a naive datetime
instance that happen to be in the fold or the gap. In the current
implementation, the result is undefined. Depending on the system mktime
implementation, the programs can see different results or errors in
those cases. With this PEP in place, the value of timestamp will be
well-defined in those cases but will depend on the value of the fold
flag. We consider the change in datetime.timestamp() method behavior a
bug fix enabled by this PEP. The old behavior can still be emulated by
the users who depend on it by writing
time.mktime(dt.timetuple()) + 1e-6*dt.microsecond instead of
dt.timestamp().

Analogy with tm_isdst

The time.mktime interface allows three values for the tm_isdst flag: -1,
0, and 1. As we explained above, -1 (asking mktime to determine whether
DST is in effect for the given time from the rest of the fields) is the
only choice that is useful in practice.

With the fold flag, however, datetime.timestamp() will return the same
value as mktime with tm_isdst=-1 in 99.98% of the time for most time
zones with DST transitions. Moreover, tm_isdst=-1-like behavior is
specified regardless of the value of fold.

It is only in the 0.02% cases (2 hours per year) that the
datetime.timestamp() and mktime with tm_isdst=-1 may disagree. However,
even in this case, most of the mktime implementations will return the
fold=0 or the fold=1 value even though relevant standards allow mktime
to return -1 and set an error code in those cases.

In other words, tm_isdst=-1 behavior is not missing from this PEP. To
the contrary, it is the only behavior provided in two different
well-defined flavors. The behavior that is missing is when a given local
hour is interpreted as a different local hour because of the
misspecified tm_isdst.

For example, in the DST-observing time zones in the Northern hemisphere
(where DST is in effect in June) one can get

    >>> from time import mktime, localtime
    >>> t = mktime((2015, 6, 1, 12, 0, 0, -1, -1, 0))
    >>> localtime(t)[:]
    (2015, 6, 1, 13, 0, 0, 0, 152, 1)

Note that 12:00 was interpreted as 13:00 by mktime. With the
datetime.timestamp, datetime.fromtimestamp, it is currently guaranteed
that

    >>> t = datetime.datetime(2015, 6, 1, 12).timestamp()
    >>> datetime.datetime.fromtimestamp(t)
    datetime.datetime(2015, 6, 1, 12, 0)

This PEP extends the same guarantee to both values of fold:

    >>> t = datetime.datetime(2015, 6, 1, 12, fold=0).timestamp()
    >>> datetime.datetime.fromtimestamp(t)
    datetime.datetime(2015, 6, 1, 12, 0)

    >>> t = datetime.datetime(2015, 6, 1, 12, fold=1).timestamp()
    >>> datetime.datetime.fromtimestamp(t)
    datetime.datetime(2015, 6, 1, 12, 0)

Thus one of the suggested uses for fold=-1 -- to match the legacy
behavior -- is not needed. Either choice of fold will match the old
behavior except in the few cases where the old behavior was undefined.

Strict Invalid Time Checking

Another suggestion was to use fold=-1 or fold=None to indicate that the
program truly has no means to deal with the folds and gaps and
dt.utcoffset() should raise an error whenever dt represents an ambiguous
or missing local time.

The main problem with this proposal, is that dt.utcoffset() is used
internally in situations where raising an error is not an option: for
example, in dictionary lookups or list/set membership checks. So strict
gap/fold checking behavior would need to be controlled by a separate
flag, say dt.utcoffset(raise_on_gap=True, raise_on_fold=False). However,
this functionality can be easily implemented in user code:

    def utcoffset(dt, raise_on_gap=True, raise_on_fold=False):
        u = dt.utcoffset()
        v = dt.replace(fold=not dt.fold).utcoffset()
        if u == v:
            return u
        if (u < v) == dt.fold:
            if raise_on_fold:
                raise AmbiguousTimeError
        else:
            if raise_on_gap:
                raise MissingTimeError
        return u

Moreover, raising an error in the problem cases is only one of many
possible solutions. An interactive program can ask the user for
additional input, while a server process may log a warning and take an
appropriate default action. We cannot possibly provide functions for all
possible user requirements, but this PEP provides the means to implement
any desired behavior in a few lines of code.

Implementation

-   Github fork: https://github.com/abalkin/cpython/tree/issue24773-s3
-   Tracker issue: http://bugs.python.org/issue24773

Copyright

This document has been placed in the public domain.

Picture Credit

This image is a work of a U.S. military or Department of Defense
employee, taken or made as part of that person's official duties. As a
work of the U.S. federal government, the image is in the public domain.

[1] People who live in locations observing the Daylight Saving Time
(DST) move their clocks back (usually one hour) every Fall.

It is less common, but occasionally clocks can be moved back for other
reasons. For example, Ukraine skipped the spring-forward transition in
March 1990 and instead, moved their clocks back on July 1, 1990,
switching from Moscow Time to Eastern European Time. In that case,
standard (winter) time was in effect before and after the transition.

Both DST and standard time changes may result in time shifts other than
an hour.

[2] The term "fall-backward fold" was invented in 1990s by Paul Eggert
of UCLA who used it in various Internet discussions related to the C
language standard that culminated in a Defect Report #139.

[3] An instance that has fold=1 in a non-ambiguous case is said to
represent an invalid time (or is invalid for short), but users are not
prevented from creating invalid instances by passing fold=1 to a
constructor or to a replace() method. This is similar to the current
situation with the instances that fall in the spring-forward gap. Such
instances don't represent any valid time, but neither the constructors
nor the replace() methods check whether the instances that they produce
are valid. Moreover, this PEP specifies how various functions should
behave when given an invalid instance.

[4] PyPy and pure Python implementation distributed with CPython already
allow None to mean "no change to existing attribute" for all other
attributes in replace().

[5] Note that the new rules may result in a paradoxical situation when
s == t but s - u != t - u. Such paradoxes are not really new and are
inherent in the overloading of the minus operator differently for intra-
and inter-zone operations. For example, one can easily construct
datetime instances t and s with some variable offset tzinfo and a
datetime u with tzinfo=timezone.utc such that
(t - u) - (s - u) != t - s. The explanation for this paradox is that the
minuses inside the parentheses and the two other minuses are really
three different operations: inter-zone datetime subtraction, timedelta
subtraction, and intra-zone datetime subtraction, which each have the
mathematical properties of subtraction separately, but not when combined
in a single expression.

[6] This exception is designed to preserve the hash and equivalence
invariants in the face of paradoxes of inter-zone arithmetic.

authors:
Alexander Belopolsky
Tim Peters