text:
PEP: 675 Title: Arbitrary Literal String Type Author: Pradeep Kumar
Srinivasan<EMAIL_ADDRESS>Graham Bleaney<EMAIL_ADDRESS>Sponsor: Jelle Zijlstra<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic: Typing Created: 30-Nov-2021
Python-Version: 3.11 Post-History: 07-Feb-2022 Resolution:
<EMAIL_ADDRESS>
typing:literalstring and typing.LiteralString

Abstract

There is currently no way to specify, using type annotations, that a
function parameter can be of any literal string type. We have to specify
a precise literal string type, such as Literal["foo"]. This PEP
introduces a supertype of literal string types: LiteralString. This
allows a function to accept arbitrary literal string types, such as
Literal["foo"] or Literal["bar"].

Motivation

Powerful APIs that execute SQL or shell commands often recommend that
they be invoked with literal strings, rather than arbitrary user
controlled strings. There is no way to express this recommendation in
the type system, however, meaning security vulnerabilities sometimes
occur when developers fail to follow it. For example, a naive way to
look up a user record from a database is to accept a user id and insert
it into a predefined SQL query:

    def query_user(conn: Connection, user_id: str) -> User:
        query = f"SELECT * FROM data WHERE user_id = {user_id}"
        conn.execute(query)
        ...  # Transform data to a User object and return it

    query_user(conn, "user123")  # OK.

However, the user-controlled data user_id is being mixed with the SQL
command string, which means a malicious user could run arbitrary SQL
commands:

    # Delete the table.
    query_user(conn, "user123; DROP TABLE data;")

    # Fetch all users (since 1 = 1 is always true).
    query_user(conn, "user123 OR 1 = 1")

To prevent such SQL injection attacks, SQL APIs offer parameterized
queries, which separate the executed query from user-controlled data and
make it impossible to run arbitrary queries. For example, with sqlite3,
our original function would be written safely as a query with
parameters:

    def query_user(conn: Connection, user_id: str) -> User:
        query = "SELECT * FROM data WHERE user_id = ?"
        conn.execute(query, (user_id,))
        ...

The problem is that there is no way to enforce this discipline.
sqlite3's own documentation can only admonish the reader to not
dynamically build the sql argument from external input; the API's
authors cannot express that through the type system. Users can (and
often do) still use a convenient f-string as before and leave their code
vulnerable to SQL injection.

Existing tools, such as the popular security linter Bandit, attempt to
detect unsafe external data used in SQL APIs, by inspecting the AST or
by other semantic pattern-matching. These tools, however, preclude
common idioms like storing a large multi-line query in a variable before
executing it, adding literal string modifiers to the query based on some
conditions, or transforming the query string using a function. (We
survey existing tools in the Rejected Alternatives section.) For
example, many tools will detect a false positive issue in this benign
snippet:

    def query_data(conn: Connection, user_id: str, limit: bool) -> None:
        query = """
            SELECT
                user.name,
                user.age
            FROM data
            WHERE user_id = ?
        """
        if limit:
            query += " LIMIT 1"

        conn.execute(query, (user_id,))

We want to forbid harmful execution of user-controlled data while still
allowing benign idioms like the above and not requiring extra user work.

To meet this goal, we introduce the LiteralString type, which only
accepts string values that are known to be made of literals. This is a
generalization of the Literal["foo"] type from PEP 586. A string of type
LiteralString cannot contain user-controlled data. Thus, any API that
only accepts LiteralString will be immune to injection vulnerabilities
(with pragmatic limitations <Appendix B: Limitations_>).

Since we want the sqlite3 execute method to disallow strings built with
user input, we would make its typeshed stub accept a sql query that is
of type LiteralString:

    from typing import LiteralString

    def execute(self, sql: LiteralString, parameters: Iterable[str] = ...) -> Cursor: ...

This successfully forbids our unsafe SQL example. The variable query
below is inferred to have type str, since it is created from a format
string using user_id, and cannot be passed to execute:

    def query_user(conn: Connection, user_id: str) -> User:
        query = f"SELECT * FROM data WHERE user_id = {user_id}"
        conn.execute(query)  # Error: Expected LiteralString, got str.
        ...

The method remains flexible enough to allow our more complicated
example:

    def query_data(conn: Connection, user_id: str, limit: bool) -> None:
        # This is a literal string.
        query = """
            SELECT
                user.name,
                user.age
            FROM data
            WHERE user_id = ?
        """

        if limit:
            # Still has type LiteralString because we added a literal string.
            query += " LIMIT 1"

        conn.execute(query, (user_id,))  # OK

Notice that the user did not have to change their SQL code at all. The
type checker was able to infer the literal string type and complain only
in case of violations.

LiteralString is also useful in other cases where we want strict
command-data separation, such as when building shell commands or when
rendering a string into an HTML response without escaping (see Appendix
A: Other Uses). Overall, this combination of strictness and flexibility
makes it easy to enforce safer API usage in sensitive code without
burdening users.

Usage statistics

In a sample of open-source projects using sqlite3, we found that
conn.execute was called ~67% of the time with a safe string literal and
~33% of the time with a potentially unsafe, local string variable. Using
this PEP's literal string type along with a type checker would prevent
the unsafe portion of that 33% of cases (ie. the ones where user
controlled data is incorporated into the query), while seamlessly
allowing the safe ones to remain.

Rationale

Firstly, why use types to prevent security vulnerabilities?

Warning users in documentation is insufficient - most users either never
see these warnings or ignore them. Using an existing dynamic or static
analysis approach is too restrictive - these prevent natural idioms, as
we saw in the Motivation section (and will discuss more extensively in
the Rejected Alternatives section). The typing-based approach in this
PEP strikes a user-friendly balance between strictness and flexibility.

Runtime approaches do not work because, at runtime, the query string is
a plain str. While we could prevent some exploits using heuristics, such
as regex-filtering for obviously malicious payloads, there will always
be a way to work around them (perfectly distinguishing good and bad
queries reduces to the halting problem).

Static approaches, such as checking the AST to see if the query string
is a literal string expression, cannot tell when a string is assigned to
an intermediate variable or when it is transformed by a benign function.
This makes them overly restrictive.

The type checker, surprisingly, does better than both because it has
access to information not available in the runtime or static analysis
approaches. Specifically, the type checker can tell us whether an
expression has a literal string type, say Literal["foo"]. The type
checker already propagates types across variable assignments or function
calls.

In the current type system itself, if the SQL or shell command execution
function only accepted three possible input strings, our job would be
done. We would just say:

    def execute(query: Literal["foo", "bar", "baz"]) -> None: ...

But, of course, execute can accept any possible query. How do we ensure
that the query does not contain an arbitrary, user-controlled string?

We want to specify that the value must be of some type Literal[<...>]
where <...> is some string. This is what LiteralString represents.
LiteralString is the "supertype" of all literal string types. In effect,
this PEP just introduces a type in the type hierarchy between
Literal["foo"] and str. Any particular literal string, such as
Literal["foo"] or Literal["bar"], is compatible with LiteralString, but
not the other way around. The "supertype" of LiteralString itself is
str. So, LiteralString is compatible with str, but not the other way
around.

Note that a Union of literal types is naturally compatible with
LiteralString because each element of the Union is individually
compatible with LiteralString. So, Literal["foo", "bar"] is compatible
with LiteralString.

However, recall that we don't just want to represent exact literal
queries. We also want to support composition of two literal strings,
such as query + " LIMIT 1". This too is possible with the above concept.
If x and y are two values of type LiteralString, then x + y will also be
of type compatible with LiteralString. We can reason about this by
looking at specific instances such as Literal["foo"] and Literal["bar"];
the value of the added string x + y can only be "foobar", which has type
Literal["foobar"] and is thus compatible with LiteralString. The same
reasoning applies when x and y are unions of literal types; the result
of pairwise adding any two literal types from x and y respectively is a
literal type, which means that the overall result is a Union of literal
types and is thus compatible with LiteralString.

In this way, we are able to leverage Python's concept of a Literal
string type to specify that our API can only accept strings that are
known to be constructed from literals. More specific details follow in
the remaining sections.

Specification

Runtime Behavior

We propose adding LiteralString to typing.py, with an implementation
similar to typing.NoReturn.

Note that LiteralString is a special form used solely for type checking.
There is no expression for which type(<expr>) will produce LiteralString
at runtime. So, we do not specify in the implementation that it is a
subclass of str.

Valid Locations for LiteralString

LiteralString can be used where any other type can be used:

    variable_annotation: LiteralString

    def my_function(literal_string: LiteralString) -> LiteralString: ...

    class Foo:
        my_attribute: LiteralString

    type_argument: List[LiteralString]

    T = TypeVar("T", bound=LiteralString)

It cannot be nested within unions of Literal types:

    bad_union: Literal["hello", LiteralString]  # Not OK
    bad_nesting: Literal[LiteralString]  # Not OK

Type Inference

Inferring LiteralString

Any literal string type is compatible with LiteralString. For example,
x: LiteralString = "foo" is valid because "foo" is inferred to be of
type Literal["foo"].

As per the Rationale, we also infer LiteralString in the following
cases:

-   Addition: x + y is of type LiteralString if both x and y are
    compatible with LiteralString.
-   Joining: sep.join(xs) is of type LiteralString if sep's type is
    compatible with LiteralString and xs's type is compatible with
    Iterable[LiteralString].
-   In-place addition: If s has type LiteralString and x has type
    compatible with LiteralString, then s += x preserves s's type as
    LiteralString.
-   String formatting: An f-string has type LiteralString if and only if
    its constituent expressions are literal strings. s.format(...) has
    type LiteralString if and only if s and the arguments have types
    compatible with LiteralString.
-   Literal-preserving methods: In Appendix C, we have provided an
    exhaustive list of str methods that preserve the LiteralString type.

In all other cases, if one or more of the composed values has a
non-literal type str, the composition of types will have type str. For
example, if s has type str, then "hello" + s has type str. This matches
the pre-existing behavior of type checkers.

LiteralString is compatible with the type str. It inherits all methods
from str. So, if we have a variable s of type LiteralString, it is safe
to write s.startswith("hello").

Some type checkers refine the type of a string when doing an equality
check:

    def foo(s: str) -> None:
        if s == "bar":
            reveal_type(s)  # => Literal["bar"]

Such a refined type in the if-block is also compatible with
LiteralString because its type is Literal["bar"].

Examples

See the examples below to help clarify the above rules:

:

    literal_string: LiteralString
    s: str = literal_string  # OK

    literal_string: LiteralString = s  # Error: Expected LiteralString, got str.
    literal_string: LiteralString = "hello"  # OK

Addition of literal strings:

    def expect_literal_string(s: LiteralString) -> None: ...

    expect_literal_string("foo" + "bar")  # OK
    expect_literal_string(literal_string + "bar")  # OK

    literal_string2: LiteralString
    expect_literal_string(literal_string + literal_string2)  # OK

    plain_string: str
    expect_literal_string(literal_string + plain_string)  # Not OK.

Join using literal strings:

    expect_literal_string(",".join(["foo", "bar"]))  # OK
    expect_literal_string(literal_string.join(["foo", "bar"]))  # OK
    expect_literal_string(literal_string.join([literal_string, literal_string2]))  # OK

    xs: List[LiteralString]
    expect_literal_string(literal_string.join(xs)) # OK
    expect_literal_string(plain_string.join([literal_string, literal_string2]))
    # Not OK because the separator has type 'str'.

In-place addition using literal strings:

    literal_string += "foo"  # OK
    literal_string += literal_string2  # OK
    literal_string += plain_string # Not OK

Format strings using literal strings:

    literal_name: LiteralString
    expect_literal_string(f"hello {literal_name}")
    # OK because it is composed from literal strings.

    expect_literal_string("hello {}".format(literal_name))  # OK

    expect_literal_string(f"hello")  # OK

    username: str
    expect_literal_string(f"hello {username}")
    # NOT OK. The format-string is constructed from 'username',
    # which has type 'str'.

    expect_literal_string("hello {}".format(username))  # Not OK

Other literal types, such as literal integers, are not compatible with
LiteralString:

    some_int: int
    expect_literal_string(some_int)  # Error: Expected LiteralString, got int.

    literal_one: Literal[1] = 1
    expect_literal_string(literal_one)  # Error: Expected LiteralString, got Literal[1].

We can call functions on literal strings:

    def add_limit(query: LiteralString) -> LiteralString:
        return query + " LIMIT = 1"

    def my_query(query: LiteralString, user_id: str) -> None:
        sql_connection().execute(add_limit(query), (user_id,))  # OK

Conditional statements and expressions work as expected:

    def return_literal_string() -> LiteralString:
        return "foo" if condition1() else "bar"  # OK

    def return_literal_str2(literal_string: LiteralString) -> LiteralString:
        return "foo" if condition1() else literal_string  # OK

    def return_literal_str3() -> LiteralString:
        if condition1():
            result: Literal["foo"] = "foo"
        else:
            result: LiteralString = "bar"

        return result  # OK

Interaction with TypeVars and Generics

TypeVars can be bound to LiteralString:

    from typing import Literal, LiteralString, TypeVar

    TLiteral = TypeVar("TLiteral", bound=LiteralString)

    def literal_identity(s: TLiteral) -> TLiteral:
        return s

    hello: Literal["hello"] = "hello"
    y = literal_identity(hello)
    reveal_type(y)  # => Literal["hello"]

    s: LiteralString
    y2 = literal_identity(s)
    reveal_type(y2)  # => LiteralString

    s_error: str
    literal_identity(s_error)
    # Error: Expected TLiteral (bound to LiteralString), got str.

LiteralString can be used as a type argument for generic classes:

    class Container(Generic[T]):
        def __init__(self, value: T) -> None:
            self.value = value

    literal_string: LiteralString = "hello"
    x: Container[LiteralString] = Container(literal_string)  # OK

    s: str
    x_error: Container[LiteralString] = Container(s)  # Not OK

Standard containers like List work as expected:

    xs: List[LiteralString] = ["foo", "bar", "baz"]

Interactions with Overloads

Literal strings and overloads do not need to interact in a special way:
the existing rules work fine. LiteralString can be used as a fallback
overload where a specific Literal["foo"] type does not match:

    @overload
    def foo(x: Literal["foo"]) -> int: ...
    @overload
    def foo(x: LiteralString) -> bool: ...
    @overload
    def foo(x: str) -> str: ...

    x1: int = foo("foo")  # First overload.
    x2: bool = foo("bar")  # Second overload.
    s: str
    x3: str = foo(s)  # Third overload.

Backwards Compatibility

We propose adding typing_extensions.LiteralString for use in earlier
Python versions.

As PEP 586 mentions
<586#backwards-compatibility>, type checkers "should feel free to
experiment with more sophisticated inference techniques". So, if the
type checker infers a literal string type for an unannotated variable
that is initialized with a literal string, the following example should
be OK:

    x = "hello"
    expect_literal_string(x)
    # OK, because x is inferred to have type 'Literal["hello"]'.

This enables precise type checking of idiomatic SQL query code without
annotating the code at all (as seen in the Motivation section example).

However, like PEP 586, this PEP does not mandate the above inference
strategy. In case the type checker doesn't infer x to have type
Literal["hello"], users can aid the type checker by explicitly
annotating it as x: LiteralString:

    x: LiteralString = "hello"
    expect_literal_string(x)

Rejected Alternatives

Why not use tool X?

Tools to catch issues such as SQL injection seem to come in three
flavors: AST based, function level analysis, and taint flow analysis.

AST-based tools: Bandit has a plugin to warn when SQL queries are not
literal strings. The problem is that many perfectly safe SQL queries are
dynamically built out of string literals, as shown in the Motivation
section. At the AST level, the resultant SQL query is not going to
appear as a string literal anymore and is thus indistinguishable from a
potentially malicious string. To use these tools would require
significantly restricting developers' ability to build SQL queries.
LiteralString can provide similar safety guarantees with fewer
restrictions.

Semgrep and pyanalyze: Semgrep supports a more sophisticated function
level analysis, including constant propagation within a function. This
allows us to prevent injection attacks while permitting some forms of
safe dynamic SQL queries within a function. pyanalyze has a similar
extension. But neither handles function calls that construct and return
safe SQL queries. For example, in the code sample below,
build_insert_query is a helper function to create a query that inserts
multiple values into the corresponding columns. Semgrep and pyanalyze
forbid this natural usage whereas LiteralString handles it with no
burden on the programmer:

    def build_insert_query(
        table: LiteralString
        insert_columns: Iterable[LiteralString],
    ) -> LiteralString:
        sql = "INSERT INTO " + table

        column_clause = ", ".join(insert_columns)
        value_clause = ", ".join(["?"] * len(insert_columns))

        sql += f" ({column_clause}) VALUES ({value_clause})"
        return sql

    def insert_data(
        conn: Connection,
        kvs_to_insert: Dict[LiteralString, str]
    ) -> None:
        query = build_insert_query("data", kvs_to_insert.keys())
        conn.execute(query, kvs_to_insert.values())

    # Example usage
    data_to_insert = {
        "column_1": value_1,  # Note: values are not literals
        "column_2": value_2,
        "column_3": value_3,
    }
    insert_data(conn, data_to_insert)

Taint flow analysis: Tools such as Pysa or CodeQL are capable of
tracking data flowing from a user controlled input into a SQL query.
These tools are powerful but involve considerable overhead in setting up
the tool in CI, defining "taint" sinks and sources, and teaching
developers how to use them. They also usually take longer to run than a
type checker (minutes instead of seconds), which means feedback is not
immediate. Finally, they move the burden of preventing vulnerabilities
on to library users instead of allowing the libraries themselves to
specify precisely how their APIs must be called (as is possible with
LiteralString).

One final reason to prefer using a new type over a dedicated tool is
that type checkers are more widely used than dedicated security tooling;
for example, MyPy was downloaded over 7 million times in Jan 2022 vs
less than 2 million times for Bandit. Having security protections built
right into type checkers will mean that more developers benefit from
them.

Why not use a NewType for str?

Any API for which LiteralString would be suitable could instead be
updated to accept a different type created within the Python type
system, such as NewType("SafeSQL", str):

    SafeSQL = NewType("SafeSQL", str)

    def execute(self, sql: SafeSQL, parameters: Iterable[str] = ...) -> Cursor: ...

    execute(SafeSQL("SELECT * FROM data WHERE user_id = ?"), user_id)  # OK

    user_query: str
    execute(user_query)  # Error: Expected SafeSQL, got str.

Having to create a new type to call an API might give some developers
pause and encourage more caution, but it doesn't guarantee that
developers won't just turn a user controlled string into the new type,
and pass it into the modified API anyway:

    query = f"SELECT * FROM data WHERE user_id = f{user_id}"
    execute(SafeSQL(query))  # No error!

We are back to square one with the problem of preventing arbitrary
inputs to SafeSQL. This is not a theoretical concern either. Django uses
the above approach with SafeString and mark_safe. Issues such as
CVE-2020-13596 show how this technique can fail.

Also note that this requires invasive changes to the source code
(wrapping the query with SafeSQL) whereas LiteralString requires no such
changes. Users can remain oblivious to it as long as they pass in
literal strings to sensitive APIs.

Why not try to emulate Trusted Types?

Trusted Types is a W3C specification for preventing DOM-based Cross Site
Scripting (XSS). XSS occurs when dangerous browser APIs accept raw
user-controlled strings. The specification modifies these APIs to accept
only the "Trusted Types" returned by designated sanitizing functions.
These sanitizing functions must take in a potentially malicious string
and validate it or render it benign somehow, for example by verifying
that it is a valid URL or HTML-encoding it.

It can be tempting to assume porting the concept of Trusted Types to
Python could solve the problem. The fundamental difference, however, is
that the output of a Trusted Types sanitizer is usually intended to not
be executable code. Thus it's easy to HTML encode the input, strip out
dangerous tags, or otherwise render it inert. With a SQL query or shell
command, the end result still needs to be executable code. There is no
way to write a sanitizer that can reliably figure out which parts of an
input string are benign and which ones are potentially malicious.

Runtime Checkable LiteralString

The LiteralString concept could be extended beyond static type checking
to be a runtime checkable property of str objects. This would provide
some benefits, such as allowing frameworks to raise errors on dynamic
strings. Such runtime errors would be a more robust defense mechanism
than type errors, which can potentially be suppressed, ignored, or never
even seen if the author does not use a type checker.

This extension to the LiteralString concept would dramatically increase
the scope of the proposal by requiring changes to one of the most
fundamental types in Python. While runtime taint checking on strings,
similar to Perl's taint, has been considered and attempted in the past,
and others may consider it in the future, such extensions are out of
scope for this PEP.

Rejected Names

We considered a variety of names for the literal string type and
solicited ideas on typing-sig. Some notable alternatives were:

-   Literal[str]: This is a natural extension of the Literal["foo"] type
    name, but typing-sig objected that users could mistake this for the
    literal type of the str class.
-   LiteralStr: This is shorter than LiteralString but looks weird to
    the PEP authors.
-   LiteralDerivedString: This (along with MadeFromLiteralString) best
    captures the technical meaning of the type. It represents not just
    the type of literal expressions, such as "foo", but also that of
    expressions composed from literals, such as "foo" + "bar". However,
    both names seem wordy.
-   StringLiteral: Users might confuse this with the existing concept of
    "string literals" where the string exists as a syntactic token in
    the source code, whereas our concept is more general.
-   SafeString: While this comes close to our intended meaning, it may
    mislead users into thinking that the string has been sanitized in
    some way, perhaps by escaping HTML tags or shell-related special
    characters.
-   ConstantStr: This does not capture the idea of composing literal
    strings.
-   StaticStr: This suggests that the string is statically computable,
    i.e., computable without running the program, which is not true. The
    literal string may vary based on runtime flags, as seen in the
    Motivation examples.
-   LiteralOnly[str]: This has the advantage of being extensible to
    other literal types, such as bytes or int. However, we did not find
    the extensibility worth the loss of readability.

Overall, there was no clear winner on typing-sig over a long period, so
we decided to tip the scales in favor of LiteralString.

LiteralBytes

We could generalize literal byte types, such as Literal[b"foo"], to
LiteralBytes. However, literal byte types are used much less frequently
than literal string types and we did not find much user demand for
LiteralBytes, so we decided not to include it in this PEP. Others may,
however, consider it in future PEPs.

Reference Implementation

This is implemented in Pyre v0.9.8 and is actively being used.

The implementation simply extends the type checker with LiteralString as
a supertype of literal string types.

To support composition via addition, join, etc., it was sufficient to
overload the stubs for str in Pyre's copy of typeshed.

Appendix A: Other Uses

To simplify the discussion and require minimal security knowledge, we
focused on SQL injections throughout the PEP. LiteralString, however,
can also be used to prevent many other kinds of injection
vulnerabilities.

Command Injection

APIs such as subprocess.run accept a string which can be run as a shell
command:

    subprocess.run(f"echo 'Hello {name}'", shell=True)

If user-controlled data is included in the command string, the code is
vulnerable to "command injection"; i.e., an attacker can run malicious
commands. For example, a value of ' && rm -rf / # would result in the
following destructive command being run:

    echo 'Hello ' && rm -rf / #'

This vulnerability could be prevented by updating run to only accept
LiteralString when used in shell=True mode. Here is one simplified stub:

    def run(command: LiteralString, *args: str, shell: bool=...): ...

Cross Site Scripting (XSS)

Most popular Python web frameworks, such as Django, use a templating
engine to produce HTML from user data. These templating languages
auto-escape user data before inserting it into the HTML template and
thus prevent cross site scripting (XSS) vulnerabilities.

But a common way to bypass auto-escaping and render HTML as-is is to use
functions like mark_safe in Django or do_mark_safe in Jinja2, which
cause XSS vulnerabilities:

    dangerous_string = django.utils.safestring.mark_safe(f"<script>{user_input}</script>")
    return(dangerous_string)

This vulnerability could be prevented by updating mark_safe to only
accept LiteralString:

    def mark_safe(s: LiteralString) -> str: ...

Server Side Template Injection (SSTI)

Templating frameworks, such as Jinja, allow Python expressions which
will be evaluated and substituted into the rendered result:

    template_str = "There are {{ len(values) }} values: {{ values }}"
    template = jinja2.Template(template_str)
    template.render(values=[1, 2])
    # Result: "There are 2 values: [1, 2]"

If an attacker controls all or part of the template string, they can
insert expressions which execute arbitrary code and compromise the
application:

    malicious_str = "{{''.__class__.__base__.__subclasses__()[408]('rm - rf /',shell=True)}}"
    template = jinja2.Template(malicious_str)
    template.render()
    # Result: The shell command 'rm - rf /' is run

Template injection exploits like this could be prevented by updating the
Template API to only accept LiteralString:

    class Template:
        def __init__(self, source: LiteralString): ...

Logging Format String Injection

Logging frameworks often allow their input strings to contain formatting
directives. At its worst, allowing users to control the logged string
has led to CVE-2021-44228 (colloquially known as log4shell), which has
been described as the "most critical vulnerability of the last decade".
While no Python frameworks are currently known to be vulnerable to a
similar attack, the built-in logging framework does provide formatting
options which are vulnerable to Denial of Service attacks from
externally controlled logging strings. The following example illustrates
a simple denial of service scenario:

    external_string = "%(foo)999999999s"
    ...
    # Tries to add > 1GB of whitespace to the logged string:
    logger.info(f'Received: {external_string}', some_dict)

This kind of attack could be prevented by requiring that the format
string passed to the logger be a LiteralString and that all externally
controlled data be passed separately as arguments (as proposed in Issue
46200):

    def info(msg: LiteralString, *args: object) -> None:
        ...

Appendix B: Limitations

There are a number of ways LiteralString could still fail to prevent
users from passing strings built from non-literal data to an API:

1. If the developer does not use a type checker or does not add type
annotations, then violations will go uncaught.

2. cast(LiteralString, non_literal_string) could be used to lie to the
type checker and allow a dynamic string value to masquerade as a
LiteralString. The same goes for a variable that has type Any.

3. Comments such as # type: ignore could be used to ignore warnings
about non-literal strings.

4. Trivial functions could be constructed to convert a str to a
LiteralString:

    def make_literal(s: str) -> LiteralString:
        letters: Dict[str, LiteralString] = {
            "A": "A",
            "B": "B",
            ...
        }
        output: List[LiteralString] = [letters[c] for c in s]
        return "".join(output)

We could mitigate the above using linting, code review, etc., but
ultimately a clever, malicious developer attempting to circumvent the
protections offered by LiteralString will always succeed. The important
thing to remember is that LiteralString is not intended to protect
against malicious developers; it is meant to protect against benign
developers accidentally using sensitive APIs in a dangerous way (without
getting in their way otherwise).

Without LiteralString, the best enforcement tool API authors have is
documentation, which is easily ignored and often not seen. With
LiteralString, API misuse requires conscious thought and artifacts in
the code that reviewers and future developers can notice.

Appendix C: str methods that preserve LiteralString

The str class has several methods that would benefit from LiteralString.
For example, users might expect "hello".capitalize() to have the type
LiteralString similar to the other examples we have seen in the
Inferring LiteralString section. Inferring the type LiteralString is
correct because the string is not an arbitrary user-supplied string - we
know that it has the type Literal["HELLO"], which is compatible with
LiteralString. In other words, the capitalize method preserves the
LiteralString type. There are several other str methods that preserve
LiteralString.

We propose updating the stub for str in typeshed so that the methods are
overloaded with the LiteralString-preserving versions. This means type
checkers do not have to hardcode LiteralString behavior for each method.
It also lets us easily support new methods in the future by updating the
typeshed stub.

For example, to preserve literal types for the capitalize method, we
would change the stub as below:

    # before
    def capitalize(self) -> str: ...

    # after
    @overload
    def capitalize(self: LiteralString) -> LiteralString: ...
    @overload
    def capitalize(self) -> str: ...

The downside of changing the str stub is that the stub becomes more
complicated and can make error messages harder to understand. Type
checkers may need to special-case str to make error messages
understandable for users.

Below is an exhaustive list of str methods which, when called with
arguments of type LiteralString, must be treated as returning a
LiteralString. If this PEP is accepted, we will update these method
signatures in typeshed:

    @overload
    def capitalize(self: LiteralString) -> LiteralString: ...
    @overload
    def capitalize(self) -> str: ...

    @overload
    def casefold(self: LiteralString) -> LiteralString: ...
    @overload
    def casefold(self) -> str: ...

    @overload
    def center(self: LiteralString, __width: SupportsIndex, __fillchar: LiteralString = ...) -> LiteralString: ...
    @overload
    def center(self, __width: SupportsIndex, __fillchar: str = ...) -> str: ...

    if sys.version_info >= (3, 8):
        @overload
        def expandtabs(self: LiteralString, tabsize: SupportsIndex = ...) -> LiteralString: ...
        @overload
        def expandtabs(self, tabsize: SupportsIndex = ...) -> str: ...

    else:
        @overload
        def expandtabs(self: LiteralString, tabsize: int = ...) -> LiteralString: ...
        @overload
        def expandtabs(self, tabsize: int = ...) -> str: ...

    @overload
    def format(self: LiteralString, *args: LiteralString, **kwargs: LiteralString) -> LiteralString: ...
    @overload
    def format(self, *args: str, **kwargs: str) -> str: ...

    @overload
    def join(self: LiteralString, __iterable: Iterable[LiteralString]) -> LiteralString: ...
    @overload
    def join(self, __iterable: Iterable[str]) -> str: ...

    @overload
    def ljust(self: LiteralString, __width: SupportsIndex, __fillchar: LiteralString = ...) -> LiteralString: ...
    @overload
    def ljust(self, __width: SupportsIndex, __fillchar: str = ...) -> str: ...

    @overload
    def lower(self: LiteralString) -> LiteralString: ...
    @overload
    def lower(self) -> LiteralString: ...

    @overload
    def lstrip(self: LiteralString, __chars: LiteralString | None = ...) -> LiteralString: ...
    @overload
    def lstrip(self, __chars: str | None = ...) -> str: ...

    @overload
    def partition(self: LiteralString, __sep: LiteralString) -> tuple[LiteralString, LiteralString, LiteralString]: ...
    @overload
    def partition(self, __sep: str) -> tuple[str, str, str]: ...

    @overload
    def replace(self: LiteralString, __old: LiteralString, __new: LiteralString, __count: SupportsIndex = ...) -> LiteralString: ...
    @overload
    def replace(self, __old: str, __new: str, __count: SupportsIndex = ...) -> str: ...

    if sys.version_info >= (3, 9):
        @overload
        def removeprefix(self: LiteralString, __prefix: LiteralString) -> LiteralString: ...
        @overload
        def removeprefix(self, __prefix: str) -> str: ...

        @overload
        def removesuffix(self: LiteralString, __suffix: LiteralString) -> LiteralString: ...
        @overload
        def removesuffix(self, __suffix: str) -> str: ...

    @overload
    def rjust(self: LiteralString, __width: SupportsIndex, __fillchar: LiteralString = ...) -> LiteralString: ...
    @overload
    def rjust(self, __width: SupportsIndex, __fillchar: str = ...) -> str: ...

    @overload
    def rpartition(self: LiteralString, __sep: LiteralString) -> tuple[LiteralString, LiteralString, LiteralString]: ...
    @overload
    def rpartition(self, __sep: str) -> tuple[str, str, str]: ...

    @overload
    def rsplit(self: LiteralString, sep: LiteralString | None = ..., maxsplit: SupportsIndex = ...) -> list[LiteralString]: ...
    @overload
    def rsplit(self, sep: str | None = ..., maxsplit: SupportsIndex = ...) -> list[str]: ...

    @overload
    def rstrip(self: LiteralString, __chars: LiteralString | None = ...) -> LiteralString: ...
    @overload
    def rstrip(self, __chars: str | None = ...) -> str: ...

    @overload
    def split(self: LiteralString, sep: LiteralString | None = ..., maxsplit: SupportsIndex = ...) -> list[LiteralString]: ...
    @overload
    def split(self, sep: str | None = ..., maxsplit: SupportsIndex = ...) -> list[str]: ...

    @overload
    def splitlines(self: LiteralString, keepends: bool = ...) -> list[LiteralString]: ...
    @overload
    def splitlines(self, keepends: bool = ...) -> list[str]: ...

    @overload
    def strip(self: LiteralString, __chars: LiteralString | None = ...) -> LiteralString: ...
    @overload
    def strip(self, __chars: str | None = ...) -> str: ...

    @overload
    def swapcase(self: LiteralString) -> LiteralString: ...
    @overload
    def swapcase(self) -> str: ...

    @overload
    def title(self: LiteralString) -> LiteralString: ...
    @overload
    def title(self) -> str: ...

    @overload
    def upper(self: LiteralString) -> LiteralString: ...
    @overload
    def upper(self) -> str: ...

    @overload
    def zfill(self: LiteralString, __width: SupportsIndex) -> LiteralString: ...
    @overload
    def zfill(self, __width: SupportsIndex) -> str: ...

    @overload
    def __add__(self: LiteralString, __s: LiteralString) -> LiteralString: ...
    @overload
    def __add__(self, __s: str) -> str: ...

    @overload
    def __iter__(self: LiteralString) -> Iterator[str]: ...
    @overload
    def __iter__(self) -> Iterator[str]: ...

    @overload
    def __mod__(self: LiteralString, __x: Union[LiteralString, Tuple[LiteralString, ...]]) -> str: ...
    @overload
    def __mod__(self, __x: Union[str, Tuple[str, ...]]) -> str: ...

    @overload
    def __mul__(self: LiteralString, __n: SupportsIndex) -> LiteralString: ...
    @overload
    def __mul__(self, __n: SupportsIndex) -> str: ...

    @overload
    def __repr__(self: LiteralString) -> LiteralString: ...
    @overload
    def __repr__(self) -> str: ...

    @overload
    def __rmul__(self: LiteralString, n: SupportsIndex) -> LiteralString: ...
    @overload
    def __rmul__(self, n: SupportsIndex) -> str: ...

    @overload
    def __str__(self: LiteralString) -> LiteralString: ...
    @overload
    def __str__(self) -> str: ...

Appendix D: Guidelines for using LiteralString in Stubs

Libraries that do not contain type annotations within their source may
specify type stubs in Typeshed. Libraries written in other languages,
such as those for machine learning, may also provide Python type stubs.
This means the type checker cannot verify that the type annotations
match the source code and must trust the type stub. Thus, authors of
type stubs need to be careful when using LiteralString, since a function
may falsely appear to be safe when it is not.

We recommend the following guidelines for using LiteralString in stubs:

-   If the stub is for a pure function, we recommend using LiteralString
    in the return type of the function or of its overloads only if all
    the corresponding parameters have literal types (i.e., LiteralString
    or Literal["a", "b"]).

        # OK
        @overload
        def my_transform(x: LiteralString, y: Literal["a", "b"]) -> LiteralString: ...
        @overload
        def my_transform(x: str, y: str) -> str: ...

        # Not OK
        @overload
        def my_transform(x: LiteralString, y: str) -> LiteralString: ...
        @overload
        def my_transform(x: str, y: str) -> str: ...

-   If the stub is for a staticmethod, we recommend the same guideline
    as above.

-   If the stub is for any other kind of method, we recommend against
    using LiteralString in the return type of the method or any of its
    overloads. This is because, even if all the explicit parameters have
    type LiteralString, the object itself may be created using user data
    and thus the return type may be user-controlled.

-   If the stub is for a class attribute or global variable, we also
    recommend against using LiteralString because the untyped code may
    write arbitrary values to the attribute.

However, we leave the final call to the library author. They may use
LiteralString if they feel confident that the string returned by the
method or function or the string stored in the attribute is guaranteed
to have a literal type - i.e., the string is created by applying only
literal-preserving str operations to a string literal.

Note that these guidelines do not apply to inline type annotations since
the type checker can verify that, say, a method returning LiteralString
does in fact return an expression of that type.

Resources

Literal String Types in Scala

Scala uses Singleton as the supertype for singleton types, which
includes literal string types, such as "foo". Singleton is Scala's
generalized analogue of this PEP's LiteralString.

Tamer Abdulradi showed how Scala's literal string types can be used for
"Preventing SQL injection at compile time", Scala Days talk Literal
types: What are they good for? (slides 52 to 68).

Thanks

Thanks to the following people for their feedback on the PEP:

Edward Qiu, Jia Chen, Shannon Zhu, Gregory P. Smith, Никита Соболев, CAM
Gerlach, Arie Bovenberg, David Foster, and Shengye Wan

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

authors:
Graham Bleaney
Pradeep Kumar Srinivasan


text:
PEP: 557 Title: Data Classes Author: Eric V. Smith<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
02-Jun-2017 Python-Version: 3.7 Post-History: 08-Sep-2017, 25-Nov-2017,
30-Nov-2017, 01-Dec-2017, 02-Dec-2017, 06-Jan-2018, 04-Mar-2018
Resolution:
https://mail.python.org/pipermail/python-dev/2017-December/151034.html

Notice for Reviewers

This PEP and the initial implementation were drafted in a separate repo:
https://github.com/ericvsmith/dataclasses. Before commenting in a public
forum please at least read the discussion listed at the end of this PEP.

Abstract

This PEP describes an addition to the standard library called Data
Classes. Although they use a very different mechanism, Data Classes can
be thought of as "mutable namedtuples with defaults". Because Data
Classes use normal class definition syntax, you are free to use
inheritance, metaclasses, docstrings, user-defined methods, class
factories, and other Python class features.

A class decorator is provided which inspects a class definition for
variables with type annotations as defined in PEP 526, "Syntax for
Variable Annotations". In this document, such variables are called
fields. Using these fields, the decorator adds generated method
definitions to the class to support instance initialization, a repr,
comparison methods, and optionally other methods as described in the
Specification section. Such a class is called a Data Class, but there's
really nothing special about the class: the decorator adds generated
methods to the class and returns the same class it was given.

As an example:

    @dataclass
    class InventoryItem:
        '''Class for keeping track of an item in inventory.'''
        name: str
        unit_price: float
        quantity_on_hand: int = 0

        def total_cost(self) -> float:
            return self.unit_price * self.quantity_on_hand

The @dataclass decorator will add the equivalent of these methods to the
InventoryItem class:

    def __init__(self, name: str, unit_price: float, quantity_on_hand: int = 0) -> None:
        self.name = name
        self.unit_price = unit_price
        self.quantity_on_hand = quantity_on_hand
    def __repr__(self):
        return f'InventoryItem(name={self.name!r}, unit_price={self.unit_price!r}, quantity_on_hand={self.quantity_on_hand!r})'
    def __eq__(self, other):
        if other.__class__ is self.__class__:
            return (self.name, self.unit_price, self.quantity_on_hand) == (other.name, other.unit_price, other.quantity_on_hand)
        return NotImplemented
    def __ne__(self, other):
        if other.__class__ is self.__class__:
            return (self.name, self.unit_price, self.quantity_on_hand) != (other.name, other.unit_price, other.quantity_on_hand)
        return NotImplemented
    def __lt__(self, other):
        if other.__class__ is self.__class__:
            return (self.name, self.unit_price, self.quantity_on_hand) < (other.name, other.unit_price, other.quantity_on_hand)
        return NotImplemented
    def __le__(self, other):
        if other.__class__ is self.__class__:
            return (self.name, self.unit_price, self.quantity_on_hand) <= (other.name, other.unit_price, other.quantity_on_hand)
        return NotImplemented
    def __gt__(self, other):
        if other.__class__ is self.__class__:
            return (self.name, self.unit_price, self.quantity_on_hand) > (other.name, other.unit_price, other.quantity_on_hand)
        return NotImplemented
    def __ge__(self, other):
        if other.__class__ is self.__class__:
            return (self.name, self.unit_price, self.quantity_on_hand) >= (other.name, other.unit_price, other.quantity_on_hand)
        return NotImplemented

Data Classes save you from writing and maintaining these methods.

Rationale

There have been numerous attempts to define classes which exist
primarily to store values which are accessible by attribute lookup. Some
examples include:

-   collections.namedtuple in the standard library.
-   typing.NamedTuple in the standard library.
-   The popular attrs[1] project.
-   George Sakkis' recordType recipe[2], a mutable data type inspired by
    collections.namedtuple.
-   Many example online recipes[3], packages[4], and questions[5]. David
    Beazley used a form of data classes as the motivating example in a
    PyCon 2013 metaclass talk[6].

So, why is this PEP needed?

With the addition of PEP 526, Python has a concise way to specify the
type of class members. This PEP leverages that syntax to provide a
simple, unobtrusive way to describe Data Classes. With two exceptions,
the specified attribute type annotation is completely ignored by Data
Classes.

No base classes or metaclasses are used by Data Classes. Users of these
classes are free to use inheritance and metaclasses without any
interference from Data Classes. The decorated classes are truly "normal"
Python classes. The Data Class decorator should not interfere with any
usage of the class.

One main design goal of Data Classes is to support static type checkers.
The use of PEP 526 syntax is one example of this, but so is the design
of the fields() function and the @dataclass decorator. Due to their very
dynamic nature, some of the libraries mentioned above are difficult to
use with static type checkers.

Data Classes are not, and are not intended to be, a replacement
mechanism for all of the above libraries. But being in the standard
library will allow many of the simpler use cases to instead leverage
Data Classes. Many of the libraries listed have different feature sets,
and will of course continue to exist and prosper.

Where is it not appropriate to use Data Classes?

-   API compatibility with tuples or dicts is required.
-   Type validation beyond that provided by PEPs 484 and 526 is
    required, or value validation or conversion is required.

Specification

All of the functions described in this PEP will live in a module named
dataclasses.

A function dataclass which is typically used as a class decorator is
provided to post-process classes and add generated methods, described
below.

The dataclass decorator examines the class to find fields. A field is
defined as any variable identified in __annotations__. That is, a
variable that has a type annotation. With two exceptions described
below, none of the Data Class machinery examines the type specified in
the annotation.

Note that __annotations__ is guaranteed to be an ordered mapping, in
class declaration order. The order of the fields in all of the generated
methods is the order in which they appear in the class.

The dataclass decorator will add various "dunder" methods to the class,
described below. If any of the added methods already exist on the class,
a TypeError will be raised. The decorator returns the same class that is
called on: no new class is created.

The dataclass decorator is typically used with no parameters and no
parentheses. However, it also supports the following logical signature:

    def dataclass(*, init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False)

If dataclass is used just as a simple decorator with no parameters, it
acts as if it has the default values documented in this signature. That
is, these three uses of @dataclass are equivalent:

    @dataclass
    class C:
        ...

    @dataclass()
    class C:
        ...

    @dataclass(init=True, repr=True, eq=True, order=False, unsafe_hash=False, frozen=False)
    class C:
        ...

The parameters to dataclass are:

-   init: If true (the default), a __init__ method will be generated.

-   repr: If true (the default), a __repr__ method will be generated.
    The generated repr string will have the class name and the name and
    repr of each field, in the order they are defined in the class.
    Fields that are marked as being excluded from the repr are not
    included. For example:
    InventoryItem(name='widget', unit_price=3.0, quantity_on_hand=10).

    If the class already defines __repr__, this parameter is ignored.

-   eq: If true (the default), an __eq__ method will be generated. This
    method compares the class as if it were a tuple of its fields, in
    order. Both instances in the comparison must be of the identical
    type.

    If the class already defines __eq__, this parameter is ignored.

-   order: If true (the default is False), __lt__, __le__, __gt__, and
    __ge__ methods will be generated. These compare the class as if it
    were a tuple of its fields, in order. Both instances in the
    comparison must be of the identical type. If order is true and eq is
    false, a ValueError is raised.

    If the class already defines any of __lt__, __le__, __gt__, or
    __ge__, then ValueError is raised.

-   unsafe_hash: If False (the default), the __hash__ method is
    generated according to how eq and frozen are set.

    If eq and frozen are both true, Data Classes will generate a
    __hash__ method for you. If eq is true and frozen is false, __hash__
    will be set to None, marking it unhashable (which it is). If eq is
    false, __hash__ will be left untouched meaning the __hash__ method
    of the superclass will be used (if the superclass is object, this
    means it will fall back to id-based hashing).

    Although not recommended, you can force Data Classes to create a
    __hash__ method with unsafe_hash=True. This might be the case if
    your class is logically immutable but can nonetheless be mutated.
    This is a specialized use case and should be considered carefully.

    If a class already has an explicitly defined __hash__ the behavior
    when adding __hash__ is modified. An explicitly defined __hash__ is
    defined when:

      -   __eq__ is defined in the class and __hash__ is defined with
          any value other than None.
      -   __eq__ is defined in the class and any non-None __hash__ is
          defined.
      -   __eq__ is not defined on the class, and any __hash__ is
          defined.

    If unsafe_hash is true and an explicitly defined __hash__ is
    present, then ValueError is raised.

    If unsafe_hash is false and an explicitly defined __hash__ is
    present, then no __hash__ is added.

    See the Python documentation[7] for more information.

-   frozen: If true (the default is False), assigning to fields will
    generate an exception. This emulates read-only frozen instances. If
    either __getattr__ or __setattr__ is defined in the class, then
    ValueError is raised. See the discussion below.

fields may optionally specify a default value, using normal Python
syntax:

    @dataclass
    class C:
        a: int       # 'a' has no default value
        b: int = 0   # assign a default value for 'b'

In this example, both a and b will be included in the added __init__
method, which will be defined as:

    def __init__(self, a: int, b: int = 0):

TypeError will be raised if a field without a default value follows a
field with a default value. This is true either when this occurs in a
single class, or as a result of class inheritance.

For common and simple use cases, no other functionality is required.
There are, however, some Data Class features that require additional
per-field information. To satisfy this need for additional information,
you can replace the default field value with a call to the provided
field() function. The signature of field() is:

    def field(*, default=MISSING, default_factory=MISSING, repr=True,
              hash=None, init=True, compare=True, metadata=None)

The MISSING value is a sentinel object used to detect if the default and
default_factory parameters are provided. This sentinel is used because
None is a valid value for default.

The parameters to field() are:

-   default: If provided, this will be the default value for this field.
    This is needed because the field call itself replaces the normal
    position of the default value.

-   default_factory: If provided, it must be a zero-argument callable
    that will be called when a default value is needed for this field.
    Among other purposes, this can be used to specify fields with
    mutable default values, as discussed below. It is an error to
    specify both default and default_factory.

-   init: If true (the default), this field is included as a parameter
    to the generated __init__ method.

-   repr: If true (the default), this field is included in the string
    returned by the generated __repr__ method.

-   compare: If True (the default), this field is included in the
    generated equality and comparison methods (__eq__, __gt__, et al.).

-   hash: This can be a bool or None. If True, this field is included in
    the generated __hash__ method. If None (the default), use the value
    of compare: this would normally be the expected behavior. A field
    should be considered in the hash if it's used for comparisons.
    Setting this value to anything other than None is discouraged.

    One possible reason to set hash=False but compare=True would be if a
    field is expensive to compute a hash value for, that field is needed
    for equality testing, and there are other fields that contribute to
    the type's hash value. Even if a field is excluded from the hash, it
    will still be used for comparisons.

-   metadata: This can be a mapping or None. None is treated as an empty
    dict. This value is wrapped in types.MappingProxyType to make it
    read-only, and exposed on the Field object. It is not used at all by
    Data Classes, and is provided as a third-party extension mechanism.
    Multiple third-parties can each have their own key, to use as a
    namespace in the metadata.

If the default value of a field is specified by a call to field(), then
the class attribute for this field will be replaced by the specified
default value. If no default is provided, then the class attribute will
be deleted. The intent is that after the dataclass decorator runs, the
class attributes will all contain the default values for the fields,
just as if the default value itself were specified. For example, after:

    @dataclass
    class C:
        x: int
        y: int = field(repr=False)
        z: int = field(repr=False, default=10)
        t: int = 20

The class attribute C.z will be 10, the class attribute C.t will be 20,
and the class attributes C.x and C.y will not be set.

Field objects

Field objects describe each defined field. These objects are created
internally, and are returned by the fields() module-level method (see
below). Users should never instantiate a Field object directly. Its
documented attributes are:

-   name: The name of the field.
-   type: The type of the field.
-   default, default_factory, init, repr, hash, compare, and metadata
    have the identical meaning and values as they do in the field()
    declaration.

Other attributes may exist, but they are private and must not be
inspected or relied on.

post-init processing

The generated __init__ code will call a method named __post_init__, if
it is defined on the class. It will be called as self.__post_init__().
If no __init__ method is generated, then __post_init__ will not
automatically be called.

Among other uses, this allows for initializing field values that depend
on one or more other fields. For example:

    @dataclass
    class C:
        a: float
        b: float
        c: float = field(init=False)

        def __post_init__(self):
            self.c = self.a + self.b

See the section below on init-only variables for ways to pass parameters
to __post_init__(). Also see the warning about how replace() handles
init=False fields.

Class variables

One place where dataclass actually inspects the type of a field is to
determine if a field is a class variable as defined in PEP 526. It does
this by checking if the type of the field is typing.ClassVar. If a field
is a ClassVar, it is excluded from consideration as a field and is
ignored by the Data Class mechanisms. For more discussion, see[8]. Such
ClassVar pseudo-fields are not returned by the module-level fields()
function.

Init-only variables

The other place where dataclass inspects a type annotation is to
determine if a field is an init-only variable. It does this by seeing if
the type of a field is of type dataclasses.InitVar. If a field is an
InitVar, it is considered a pseudo-field called an init-only field. As
it is not a true field, it is not returned by the module-level fields()
function. Init-only fields are added as parameters to the generated
__init__ method, and are passed to the optional __post_init__ method.
They are not otherwise used by Data Classes.

For example, suppose a field will be initialized from a database, if a
value is not provided when creating the class:

    @dataclass
    class C:
        i: int
        j: int = None
        database: InitVar[DatabaseType] = None

        def __post_init__(self, database):
            if self.j is None and database is not None:
                self.j = database.lookup('j')

    c = C(10, database=my_database)

In this case, fields() will return Field objects for i and j, but not
for database.

Frozen instances

It is not possible to create truly immutable Python objects. However, by
passing frozen=True to the @dataclass decorator you can emulate
immutability. In that case, Data Classes will add __setattr__ and
__delattr__ methods to the class. These methods will raise a
FrozenInstanceError when invoked.

There is a tiny performance penalty when using frozen=True: __init__
cannot use simple assignment to initialize fields, and must use
object.__setattr__.

Inheritance

When the Data Class is being created by the @dataclass decorator, it
looks through all of the class's base classes in reverse MRO (that is,
starting at object) and, for each Data Class that it finds, adds the
fields from that base class to an ordered mapping of fields. After all
of the base class fields are added, it adds its own fields to the
ordered mapping. All of the generated methods will use this combined,
calculated ordered mapping of fields. Because the fields are in
insertion order, derived classes override base classes. An example:

    @dataclass
    class Base:
        x: Any = 15.0
        y: int = 0

    @dataclass
    class C(Base):
        z: int = 10
        x: int = 15

The final list of fields is, in order, x, y, z. The final type of x is
int, as specified in class C.

The generated __init__ method for C will look like:

    def __init__(self, x: int = 15, y: int = 0, z: int = 10):

Default factory functions

If a field specifies a default_factory, it is called with zero arguments
when a default value for the field is needed. For example, to create a
new instance of a list, use:

    l: list = field(default_factory=list)

If a field is excluded from __init__ (using init=False) and the field
also specifies default_factory, then the default factory function will
always be called from the generated __init__ function. This happens
because there is no other way to give the field an initial value.

Mutable default values

Python stores default member variable values in class attributes.
Consider this example, not using Data Classes:

    class C:
        x = []
        def add(self, element):
            self.x += element

    o1 = C()
    o2 = C()
    o1.add(1)
    o2.add(2)
    assert o1.x == [1, 2]
    assert o1.x is o2.x

Note that the two instances of class C share the same class variable x,
as expected.

Using Data Classes, if this code was valid:

    @dataclass
    class D:
        x: List = []
        def add(self, element):
            self.x += element

it would generate code similar to:

    class D:
        x = []
        def __init__(self, x=x):
            self.x = x
        def add(self, element):
            self.x += element

    assert D().x is D().x

This has the same issue as the original example using class C. That is,
two instances of class D that do not specify a value for x when creating
a class instance will share the same copy of x. Because Data Classes
just use normal Python class creation they also share this problem.
There is no general way for Data Classes to detect this condition.
Instead, Data Classes will raise a TypeError if it detects a default
parameter of type list, dict, or set. This is a partial solution, but it
does protect against many common errors. See Automatically support
mutable default values in the Rejected Ideas section for more details.

Using default factory functions is a way to create new instances of
mutable types as default values for fields:

    @dataclass
    class D:
        x: list = field(default_factory=list)

    assert D().x is not D().x

Module level helper functions

-   fields(class_or_instance): Returns a tuple of Field objects that
    define the fields for this Data Class. Accepts either a Data Class,
    or an instance of a Data Class. Raises ValueError if not passed a
    Data Class or instance of one. Does not return pseudo-fields which
    are ClassVar or InitVar.

-   asdict(instance, *, dict_factory=dict): Converts the Data Class
    instance to a dict (by using the factory function dict_factory).
    Each Data Class is converted to a dict of its fields, as name:value
    pairs. Data Classes, dicts, lists, and tuples are recursed into. For
    example:

        @dataclass
        class Point:
             x: int
             y: int

        @dataclass
        class C:
             l: List[Point]

        p = Point(10, 20)
        assert asdict(p) == {'x': 10, 'y': 20}

        c = C([Point(0, 0), Point(10, 4)])
        assert asdict(c) == {'l': [{'x': 0, 'y': 0}, {'x': 10, 'y': 4}]}

    Raises TypeError if instance is not a Data Class instance.

-   astuple(*, tuple_factory=tuple): Converts the Data Class instance to
    a tuple (by using the factory function tuple_factory). Each Data
    Class is converted to a tuple of its field values. Data Classes,
    dicts, lists, and tuples are recursed into.

    Continuing from the previous example:

        assert astuple(p) == (10, 20)
        assert astuple(c) == ([(0, 0), (10, 4)],)

    Raises TypeError if instance is not a Data Class instance.

-   make_dataclass(cls_name, fields, *, bases=(), namespace=None):
    Creates a new Data Class with name cls_name, fields as defined in
    fields, base classes as given in bases, and initialized with a
    namespace as given in namespace. fields is an iterable whose
    elements are either name, (name, type), or (name, type, Field). If
    just name is supplied, typing.Any is used for type. This function is
    not strictly required, because any Python mechanism for creating a
    new class with __annotations__ can then apply the dataclass function
    to convert that class to a Data Class. This function is provided as
    a convenience. For example:

        C = make_dataclass('C',
                           [('x', int),
                             'y',
                            ('z', int, field(default=5))],
                           namespace={'add_one': lambda self: self.x + 1})

    Is equivalent to:

        @dataclass
        class C:
            x: int
            y: 'typing.Any'
            z: int = 5

            def add_one(self):
                return self.x + 1

-   replace(instance, **changes): Creates a new object of the same type
    of instance, replacing fields with values from changes. If instance
    is not a Data Class, raises TypeError. If values in changes do not
    specify fields, raises TypeError.

    The newly returned object is created by calling the __init__ method
    of the Data Class. This ensures that __post_init__, if present, is
    also called.

    Init-only variables without default values, if any exist, must be
    specified on the call to replace so that they can be passed to
    __init__ and __post_init__.

    It is an error for changes to contain any fields that are defined as
    having init=False. A ValueError will be raised in this case.

    Be forewarned about how init=False fields work during a call to
    replace(). They are not copied from the source object, but rather
    are initialized in __post_init__(), if they're initialized at all.
    It is expected that init=False fields will be rarely and judiciously
    used. If they are used, it might be wise to have alternate class
    constructors, or perhaps a custom replace() (or similarly named)
    method which handles instance copying.

-   is_dataclass(class_or_instance): Returns True if its parameter is a
    dataclass or an instance of one, otherwise returns False.

    If you need to know if a class is an instance of a dataclass (and
    not a dataclass itself), then add a further check for
    not isinstance(obj, type):

        def is_dataclass_instance(obj):
            return is_dataclass(obj) and not isinstance(obj, type)

Discussion

python-ideas discussion

This discussion started on python-ideas[9] and was moved to a GitHub
repo[10] for further discussion. As part of this discussion, we made the
decision to use PEP 526 syntax to drive the discovery of fields.

Support for automatically setting __slots__?

At least for the initial release, __slots__ will not be supported.
__slots__ needs to be added at class creation time. The Data Class
decorator is called after the class is created, so in order to add
__slots__ the decorator would have to create a new class, set __slots__,
and return it. Because this behavior is somewhat surprising, the initial
version of Data Classes will not support automatically setting
__slots__. There are a number of workarounds:

-   Manually add __slots__ in the class definition.
-   Write a function (which could be used as a decorator) that inspects
    the class using fields() and creates a new class with __slots__ set.

For more discussion, see[11].

Why not just use namedtuple?

-   Any namedtuple can be accidentally compared to any other with the
    same number of fields. For example:
    Point3D(2017, 6, 2) == Date(2017, 6, 2). With Data Classes, this
    would return False.

-   A namedtuple can be accidentally compared to a tuple. For example,
    Point2D(1, 10) == (1, 10). With Data Classes, this would return
    False.

-   Instances are always iterable, which can make it difficult to add
    fields. If a library defines:

        Time = namedtuple('Time', ['hour', 'minute'])
        def get_time():
            return Time(12, 0)

    Then if a user uses this code as:

        hour, minute = get_time()

    then it would not be possible to add a second field to Time without
    breaking the user's code.

-   No option for mutable instances.

-   Cannot specify default values.

-   Cannot control which fields are used for __init__, __repr__, etc.

-   Cannot support combining fields by inheritance.

Why not just use typing.NamedTuple?

For classes with statically defined fields, it does support similar
syntax to Data Classes, using type annotations. This produces a
namedtuple, so it shares namedtuples benefits and some of its downsides.
Data Classes, unlike typing.NamedTuple, support combining fields via
inheritance.

Why not just use attrs?

-   attrs moves faster than could be accommodated if it were moved in to
    the standard library.
-   attrs supports additional features not being proposed here:
    validators, converters, metadata, etc. Data Classes makes a tradeoff
    to achieve simplicity by not implementing these features.

For more discussion, see[12].

post-init parameters

In an earlier version of this PEP before InitVar was added, the
post-init function __post_init__ never took any parameters.

The normal way of doing parameterized initialization (and not just with
Data Classes) is to provide an alternate classmethod constructor. For
example:

    @dataclass
    class C:
        x: int

        @classmethod
        def from_file(cls, filename):
            with open(filename) as fl:
                file_value = int(fl.read())
            return C(file_value)

    c = C.from_file('file.txt')

Because the __post_init__ function is the last thing called in the
generated __init__, having a classmethod constructor (which can also
execute code immediately after constructing the object) is functionally
equivalent to being able to pass parameters to a __post_init__ function.

With InitVars, __post_init__ functions can now take parameters. They are
passed first to __init__ which passes them to __post_init__ where user
code can use them as needed.

The only real difference between alternate classmethod constructors and
InitVar pseudo-fields is in regards to required non-field parameters
during object creation. With InitVars, using __init__ and the
module-level replace() function InitVars must always be specified.
Consider the case where a context object is needed to create an
instance, but isn't stored as a field. With alternate classmethod
constructors the context parameter is always optional, because you could
still create the object by going through __init__ (unless you suppress
its creation). Which approach is more appropriate will be
application-specific, but both approaches are supported.

Another reason for using InitVar fields is that the class author can
control the order of __init__ parameters. This is especially important
with regular fields and InitVar fields that have default values, as all
fields with defaults must come after all fields without defaults. A
previous design had all init-only fields coming after regular fields.
This meant that if any field had a default value, then all init-only
fields would have to have defaults values, too.

asdict and astuple function names

The names of the module-level helper functions asdict() and astuple()
are arguably not PEP 8 compliant, and should be as_dict() and
as_tuple(), respectively. However, after discussion[13] it was decided
to keep consistency with namedtuple._asdict() and attr.asdict().

Rejected ideas

Copying init=False fields after new object creation in replace()

Fields that are init=False are by definition not passed to __init__, but
instead are initialized with a default value, or by calling a default
factory function in __init__, or by code in __post_init__.

A previous version of this PEP specified that init=False fields would be
copied from the source object to the newly created object after __init__
returned, but that was deemed to be inconsistent with using __init__ and
__post_init__ to initialize the new object. For example, consider this
case:

    @dataclass
    class Square:
        length: float
        area: float = field(init=False, default=0.0)

        def __post_init__(self):
            self.area = self.length * self.length

    s1 = Square(1.0)
    s2 = replace(s1, length=2.0)

If init=False fields were copied from the source to the destination
object after __post_init__ is run, then s2 would end up begin
Square(length=2.0, area=1.0), instead of the correct
Square(length=2.0, area=4.0).

Automatically support mutable default values

One proposal was to automatically copy defaults, so that if a literal
list [] was a default value, each instance would get a new list. There
were undesirable side effects of this decision, so the final decision is
to disallow the 3 known built-in mutable types: list, dict, and set. For
a complete discussion of this and other options, see[14].

Examples

Custom __init__ method

Sometimes the generated __init__ method does not suffice. For example,
suppose you wanted to have an object to store *args and **kwargs:

    @dataclass(init=False)
    class ArgHolder:
        args: List[Any]
        kwargs: Mapping[Any, Any]

        def __init__(self, *args, **kwargs):
            self.args = args
            self.kwargs = kwargs

    a = ArgHolder(1, 2, three=3)

A complicated example

This code exists in a closed source project:

    class Application:
        def __init__(self, name, requirements, constraints=None, path='', executable_links=None, executables_dir=()):
            self.name = name
            self.requirements = requirements
            self.constraints = {} if constraints is None else constraints
            self.path = path
            self.executable_links = [] if executable_links is None else executable_links
            self.executables_dir = executables_dir
            self.additional_items = []

        def __repr__(self):
            return f'Application({self.name!r},{self.requirements!r},{self.constraints!r},{self.path!r},{self.executable_links!r},{self.executables_dir!r},{self.additional_items!r})'

This can be replaced by:

    @dataclass
    class Application:
        name: str
        requirements: List[Requirement]
        constraints: Dict[str, str] = field(default_factory=dict)
        path: str = ''
        executable_links: List[str] = field(default_factory=list)
        executable_dir: Tuple[str] = ()
        additional_items: List[str] = field(init=False, default_factory=list)

The Data Class version is more declarative, has less code, supports
typing, and includes the other generated functions.

Acknowledgements

The following people provided invaluable input during the development of
this PEP and code: Ivan Levkivskyi, Guido van Rossum, Hynek Schlawack,
Raymond Hettinger, and Lisa Roach. I thank them for their time and
expertise.

A special mention must be made about the attrs project. It was a true
inspiration for this PEP, and I respect the design decisions they made.

References

Copyright

This document has been placed in the public domain.

[1] attrs project on github (https://github.com/python-attrs/attrs)

[2] George Sakkis' recordType recipe
(http://code.activestate.com/recipes/576555-records/)

[3] DictDotLookup recipe
(http://code.activestate.com/recipes/576586-dot-style-nested-lookups-over-dictionary-based-dat/)

[4] attrdict package (https://pypi.python.org/pypi/attrdict)

[5] StackOverflow question about data container classes
(https://stackoverflow.com/questions/3357581/using-python-class-as-a-data-container)

[6] David Beazley metaclass talk featuring data classes
(https://www.youtube.com/watch?v=sPiWg5jSoZI)

[7] Python documentation for __hash__
(https://docs.python.org/3/reference/datamodel.html#object.__hash__)

[8] ClassVar discussion in PEP 526 <526#class-and-instance-variable-annotations>

[9] Start of python-ideas discussion
(https://mail.python.org/pipermail/python-ideas/2017-May/045618.html)

[10] GitHub repo where discussions and initial development took place
(https://github.com/ericvsmith/dataclasses)

[11] Support __slots__?
(https://github.com/ericvsmith/dataclasses/issues/28)

[12] why not just attrs?
(https://github.com/ericvsmith/dataclasses/issues/19)

[13] PEP 8 names for asdict and astuple
(https://github.com/ericvsmith/dataclasses/issues/110)

[14] Copying mutable defaults
(https://github.com/ericvsmith/dataclasses/issues/3)

license: Public Domain


text:
PEP: 269 Title: Pgen Module for Python Author: Jonathan Riehl
<EMAIL_ADDRESS>Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 24-Aug-2001 Python-Version: 2.2
Post-History:

Abstract

Much like the parser module exposes the Python parser, this PEP proposes
that the parser generator used to create the Python parser, pgen, be
exposed as a module in Python.

Rationale

Through the course of Pythonic history, there have been numerous
discussions about the creation of a Python compiler[1]. These have
resulted in several implementations of Python parsers, most notably the
parser module currently provided in the Python standard library[2] and
Jeremy Hylton's compiler module[3]. However, while multiple language
changes have been proposed [4][5], experimentation with the Python
syntax has lacked the benefit of a Python binding to the actual parser
generator used to build Python.

By providing a Python wrapper analogous to Fred Drake Jr.'s parser
wrapper, but targeted at the pgen library, the following assertions are
made:

1.  Reference implementations of syntax changes will be easier to
    develop. Currently, a reference implementation of a syntax change
    would require the developer to use the pgen tool from the command
    line. The resulting parser data structure would then either have to
    be reworked to interface with a custom CPython implementation, or
    wrapped as a C extension module.
2.  Reference implementations of syntax changes will be easier to
    distribute. Since the parser generator will be available in Python,
    it should follow that the resulting parser will accessible from
    Python. Therefore, reference implementations should be available as
    pure Python code, versus using custom versions of the existing
    CPython distribution, or as compilable extension modules.
3.  Reference implementations of syntax changes will be easier to
    discuss with a larger audience. This somewhat falls out of the
    second assertion, since the community of Python users is most likely
    larger than the community of CPython developers.
4.  Development of small languages in Python will be further enhanced,
    since the additional module will be a fully functional LL(1) parser
    generator.

Specification

The proposed module will be called pgen. The pgen module will contain
the following functions:

parseGrammarFile (fileName) -> AST

The parseGrammarFile() function will read the file pointed to by
fileName and create an AST object. The AST nodes will contain the
nonterminal, numeric values of the parser generator meta-grammar. The
output AST will be an instance of the AST extension class as provided by
the parser module. Syntax errors in the input file will cause the
SyntaxError exception to be raised.

parseGrammarString (text) -> AST

The parseGrammarString() function will follow the semantics of the
parseGrammarFile(), but accept the grammar text as a string for input,
as opposed to the file name.

buildParser (grammarAst) -> DFA

The buildParser() function will accept an AST object for input and
return a DFA (deterministic finite automaton) data structure. The DFA
data structure will be a C extension class, much like the AST structure
is provided in the parser module. If the input AST does not conform to
the nonterminal codes defined for the pgen meta-grammar, buildParser()
will throw a ValueError exception.

parseFile (fileName, dfa, start) -> AST

The parseFile() function will essentially be a wrapper for the
PyParser_ParseFile() C API function. The wrapper code will accept the
DFA C extension class, and the file name. An AST instance that conforms
to the lexical values in the token module and the nonterminal values
contained in the DFA will be output.

parseString (text, dfa, start) -> AST

The parseString() function will operate in a similar fashion to the
parseFile() function, but accept the parse text as an argument. Much
like parseFile() will wrap the PyParser_ParseFile() C API function,
parseString() will wrap the PyParser_ParseString() function.

symbolToStringMap (dfa) -> dict

The symbolToStringMap() function will accept a DFA instance and return a
dictionary object that maps from the DFA's numeric values for its
nonterminals to the string names of the nonterminals as found in the
original grammar specification for the DFA.

stringToSymbolMap (dfa) -> dict

The stringToSymbolMap() function output a dictionary mapping the
nonterminal names of the input DFA to their corresponding numeric
values.

Extra credit will be awarded if the map generation functions and parsing
functions are also methods of the DFA extension class.

Implementation Plan

A cunning plan has been devised to accomplish this enhancement:

1.  Rename the pgen functions to conform to the CPython naming
    standards. This action may involve adding some header files to the
    Include subdirectory.
2.  Move the pgen C modules in the Makefile.pre.in from unique pgen
    elements to the Python C library.
3.  Make any needed changes to the parser module so the AST extension
    class understands that there are AST types it may not understand.
    Cursory examination of the AST extension class shows that it keeps
    track of whether the tree is a suite or an expression.
4.  Code an additional C module in the Modules directory. The C
    extension module will implement the DFA extension class and the
    functions outlined in the previous section.
5.  Add the new module to the build process. Black magic, indeed.

Limitations

Under this proposal, would be designers of Python 3000 will still be
constrained to Python's lexical conventions. The addition, subtraction
or modification of the Python lexer is outside the scope of this PEP.

Reference Implementation

No reference implementation is currently provided. A patch was provided
at some point in
http://sourceforge.net/tracker/index.php?func=detail&aid=599331&group_id=5470&atid=305470
but that patch is no longer maintained.

References

Copyright

This document has been placed in the public domain.

[1] The (defunct) Python Compiler-SIG
http://www.python.org/sigs/compiler-sig/

[2] Parser Module Documentation
http://docs.python.org/library/parser.html

[3] Hylton, Jeremy. http://docs.python.org/library/compiler.html

[4] Pelletier, Michel. "Python Interface Syntax", PEP 245

[5] The Python Types-SIG http://www.python.org/sigs/types-sig/


text:
PEP: 271 Title: Prefixing sys.path by command line option Author:
Frédéric B. Giacometti<EMAIL_ADDRESS>Status: Rejected Type:
Standards Track Content-Type: text/x-rst Created: 15-Aug-2001
Python-Version: 2.2 Post-History:

Abstract

At present, setting the PYTHONPATH environment variable is the only
method for defining additional Python module search directories.

This PEP introduces the '-P' valued option to the python command as an
alternative to PYTHONPATH.

Rationale

On Unix:

    python -P $SOMEVALUE

will be equivalent to:

    env PYTHONPATH=$SOMEVALUE python

On Windows 2K:

    python -P %SOMEVALUE%

will (almost) be equivalent to:

    set __PYTHONPATH=%PYTHONPATH% && set PYTHONPATH=%SOMEVALUE%\
       && python && set PYTHONPATH=%__PYTHONPATH%

Other Information

This option is equivalent to the 'java -classpath' option.

When to use this option

This option is intended to ease and make more robust the use of Python
in test or build scripts, for instance.

Reference Implementation

A patch implementing this is available from SourceForge:

    http://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=6916&aid=429614

with the patch discussion at:

    http://sourceforge.net/tracker/?func=detail&atid=305470&aid=429614&group_id=5470

Copyright

This document has been placed in the public domain.


text:
PEP: 682 Title: Format Specifier for Signed Zero Author: John Belmonte
<EMAIL_ADDRESS>Sponsor: Mark Dickinson<EMAIL_ADDRESS>PEP-Delegate: Mark Dickinson Discussions-To:
https://discuss.python.org/t/pep-682-format-specifier-for-signed-zero/13596
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
29-Jan-2022 Python-Version: 3.11 Post-History: 08-Feb-2022 Resolution:
https://discuss.python.org/t/accepting-pep-682-format-specifier-for-signed-zero/14088

Abstract

Though float and Decimal types can represent signed zero, in many fields
of mathematics negative zero is surprising or unwanted -- especially in
the context of displaying an (often rounded) numerical result. This PEP
proposes an extension to the string format specification allowing
negative zero to be normalized to positive zero.

Motivation

Here is negative zero:

    >>> x = -0.
    >>> x
    -0.0

When formatting a number, negative zero can result from rounding.
Assuming the user's intention is truly to discard precision, the
distinction between negative and positive zero of the rounded result
might be considered an unwanted artifact:

    >>> for x in (.002, -.001, .060):
    ...     print(f'{x: .1f}')
     0.0
    -0.0
     0.1

There are various approaches to clearing the sign of a negative zero. It
can be achieved without a conditional by adding positive zero:

    >>> x = -0.
    >>> x + 0.
    0.0

To normalize negative zero when formatting, it is necessary to perform a
redundant (and error-prone) pre-rounding of the input:

    >>> for x in (.002, -.001, .060):
    ...     print(f'{round(x, 1) + 0.: .1f}')
     0.0
     0.0
     0.1

There is ample evidence that, regardless of the language, programmers
are often looking for a way to suppress negative zero, and landing on a
variety of workarounds (pre-round, post-regex, etc.). A sampling:

-   How to have negative zero always formatted as positive zero in a
    python string? (Python, post-regex)
-   (Iron)Python formatting issue with modulo operator & "negative zero"
    (Python, pre-round)
-   Negative sign in case of zero in java (Java, post-regex)
-   Prevent small negative numbers printing as "-0" (Objective-C, custom
    number formatter)

What we would like instead is a first-class option to normalize negative
zero, on top of everything else that numerical string formatting already
offers.

Rationale

There are use cases where negative zero is unwanted in formatted number
output -- arguably, not wanting it is more common. Expanding the format
specification is the best way to support this because number formatting
already incorporates rounding, and the normalization of negative zero
must happen after rounding.

While it is possible to pre-round and normalize a number before
formatting, it's tedious and prone to error if the rounding doesn't
precisely match that of the format spec. Furthermore, functions that
wrap formatting would find themselves having to parse format specs to
extract the precision information. For example, consider how this
utility for formatting one-dimensional numerical arrays would be
complicated by such pre-rounding:

    def format_vector(v, format_spec='8.2f'):
        """Format a vector (any iterable) using given per-term format string."""
        return f"[{','.join(f'{term:{format_spec}}' for term in v)}]"

To date, there doesn't appear to be any other widely-used language or
library providing a formatting option for negative zero. However, the
same z option syntax and semantics specified below have been proposed
for C++ std::format(). While the proposal was withdrawn for C++20, a
consensus proposal is promised for C++23. (The original feature request
prompting this PEP was argued without knowledge of the C++ proposal.)

When Rust developers debated whether to suppress negative zero in print
output, they took a small survey of other languages. Notably, it didn't
mention any language providing an option for negative zero handling.

Specification

An optional, literal z is added to the Format Specification
Mini-Language following sign:

    [[fill]align][sign][z][#][0][width][grouping_option][.precision][type]

where z is allowed for floating-point presentation types (f, g, etc., as
defined by the format specification documentation). Support for z is
provided by the .__format__() method of each numeric type, allowing the
specifier to be used in f-strings, built-in format(), and str.format().

When z is present, negative zero (whether the original value or the
result of rounding) will be normalized to positive zero.

Synopsis:

    >>> x = -.00001
    >>> f'{x:z.1f}'
    '0.0'

    >>> x = decimal.Decimal('-.00001')
    >>> '{:+z.1f}'.format(x)
    '+0.0'

Design Notes

The solution must be opt-in, because we can't change the behavior of
programs that may be expecting or relying on negative zero when
formatting numbers.

The proposed extension is intentionally [sign][z] rather than [sign[z]].
The default for sign (-) is not widely known or explicitly written, so
this avoids everyone having to learn it just to use the z option.

While f-strings, built-in format(), and str.format() can access the new
option, %-formatting cannot. There is already precedent for not
extending %-formatting with new options, as was the case for the ,
option (PEP 378).

C99 printf already uses the z option character for another purpose:
qualifying the unsigned type (u) to match the length of size_t. However,
since the signed zero option specifically disallows z for integer
presentation types, it's possible to disambiguate the two uses, should C
want to adopt this new option.

Backwards Compatibility

The new formatting behavior is opt-in, so numerical formatting of
existing programs will not be affected.

How to Teach This

A typical introductory Python course will not cover string formatting in
full detail. For such a course, no adjustments would need to be made.
For a course that does go into details of the string format
specification, a single example demonstrating the effect of the z option
on a negative value that's rounded to zero by the formatting should be
enough. For an independent developer encountering the feature in someone
else's code, reference to the Format Specification Mini-Language section
of the library reference manual should suffice.

Reference Implementation

A reference implementation exists at pull request #30049.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 398 Title: Python 3.3 Release Schedule Version: $Revision$
Last-Modified: $Date$ Author: Georg Brandl<EMAIL_ADDRESS>Status:
Final Type: Informational Topic: Release Content-Type: text/x-rst
Created: 23-Mar-2011 Python-Version: 3.3

Abstract

This document describes the development and release schedule for Python
3.3. The schedule primarily concerns itself with PEP-sized items.

Release Manager and Crew

-   3.3 Release Managers: Georg Brandl, Ned Deily (3.3.7+)
-   Windows installers: Martin v. Löwis
-   Mac installers: Ronald Oussoren/Ned Deily
-   Documentation: Georg Brandl

3.3 Lifespan

3.3 will receive bugfix updates approximately every 4-6 months for
approximately 18 months. After the release of 3.4.0 final, a final 3.3
bugfix update will be released. After that, security updates (source
only) will be released until 5 years after the release of 3.3 final,
which will be September 2017.

As of 2017-09-29, Python 3.3.x reached end-of-life status.

Release Schedule

3.3.0 schedule

-   3.3.0 alpha 1: March 5, 2012
-   3.3.0 alpha 2: April 2, 2012
-   3.3.0 alpha 3: May 1, 2012
-   3.3.0 alpha 4: May 31, 2012
-   3.3.0 beta 1: June 27, 2012

(No new features beyond this point.)

-   3.3.0 beta 2: August 12, 2012
-   3.3.0 candidate 1: August 24, 2012
-   3.3.0 candidate 2: September 9, 2012
-   3.3.0 candidate 3: September 24, 2012
-   3.3.0 final: September 29, 2012

3.3.1 schedule

-   3.3.1 candidate 1: March 23, 2013
-   3.3.1 final: April 6, 2013

3.3.2 schedule

-   3.3.2 final: May 13, 2013

3.3.3 schedule

-   3.3.3 candidate 1: October 27, 2013
-   3.3.3 candidate 2: November 9, 2013
-   3.3.3 final: November 16, 2013

3.3.4 schedule

-   3.3.4 candidate 1: January 26, 2014
-   3.3.4 final: February 9, 2014

3.3.5 schedule

Python 3.3.5 was the last regular maintenance release before 3.3 entered
security-fix only mode.

-   3.3.5 candidate 1: February 22, 2014
-   3.3.5 candidate 2: March 1, 2014
-   3.3.5 final: March 8, 2014

3.3.6 schedule

Security fixes only

-   3.3.6 candidate 1 (source-only release): October 4, 2014
-   3.3.6 final (source-only release): October 11, 2014

3.3.7 schedule

Security fixes only

-   3.3.7 candidate 1 (source-only release): September 6, 2017
-   3.3.7 final (source-only release): September 19, 2017

3.3.x end-of-life

-   September 29, 2017

Features for 3.3

Implemented / Final PEPs:

-   PEP 362: Function Signature Object
-   PEP 380: Syntax for Delegating to a Subgenerator
-   PEP 393: Flexible String Representation
-   PEP 397: Python launcher for Windows
-   PEP 399: Pure Python/C Accelerator Module Compatibility Requirements
-   PEP 405: Python Virtual Environments
-   PEP 409: Suppressing exception context
-   PEP 412: Key-Sharing Dictionary
-   PEP 414: Explicit Unicode Literal for Python 3.3
-   PEP 415: Implement context suppression with exception attributes
-   PEP 417: Including mock in the Standard Library
-   PEP 418: Add monotonic time, performance counter, and process time
    functions
-   PEP 420: Implicit Namespace Packages
-   PEP 421: Adding sys.implementation
-   PEP 3118: Revising the buffer protocol (protocol semantics
    finalised)
-   PEP 3144: IP Address manipulation library
-   PEP 3151: Reworking the OS and IO exception hierarchy
-   PEP 3155: Qualified name for classes and functions

Other final large-scale changes:

-   Addition of the "faulthandler" module
-   Addition of the "lzma" module, and lzma/xz support in tarfile
-   Implementing __import__ using importlib
-   Addition of the C decimal implementation
-   Switch of Windows build toolchain to VS 2010

Candidate PEPs:

-   None

Other planned large-scale changes:

-   None

Deferred to post-3.3:

-   PEP 395: Qualified Names for Modules
-   PEP 3143: Standard daemon process library
-   PEP 3154: Pickle protocol version 4
-   Breaking out standard library and docs in separate repos
-   Addition of the "packaging" module, deprecating "distutils"
-   Addition of the "regex" module
-   Email version 6
-   A standard event-loop interface (PEP by Jim Fulton pending)

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: