text:
PEP: 669 Title: Low Impact Monitoring for CPython Author: Mark Shannon
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/
Status: Final Type: Standards Track Created: 18-Aug-2021 Python-Version:
3.12 Post-History: 07-Dec-2021, 10-Jan-2022, Resolution:
https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/42

python:sys.monitoring

Abstract

Using a profiler or debugger in CPython can have a severe impact on
performance. Slowdowns by an order of magnitude are common.

This PEP proposes an API for monitoring Python programs running on
CPython that will enable monitoring at low cost.

Although this PEP does not specify an implementation, it is expected
that it will be implemented using the quickening step of PEP 659.

A sys.monitoring namespace will be added, which will contain the
relevant functions and constants.

Motivation

Developers should not have to pay an unreasonable cost to use debuggers,
profilers and other similar tools.

C++ and Java developers expect to be able to run a program at full speed
(or very close to it) under a debugger. Python developers should expect
that too.

Rationale

The quickening mechanism provided by PEP 659 provides a way to
dynamically modify executing Python bytecode. These modifications have
little cost beyond the parts of the code that are modified and a
relatively low cost to those parts that are modified. We can leverage
this to provide an efficient mechanism for monitoring that was not
possible in 3.10 or earlier.

By using quickening, we expect that code run under a debugger on 3.12
should outperform code run without a debugger on 3.11. Profiling will
still slow down execution, but by much less than in 3.11.

Specification

Monitoring of Python programs is done by registering callback functions
for events and by activating a set of events.

Activating events and registering callback functions are independent of
each other.

Both registering callbacks and activating events are done on a per-tool
basis. It is possible to have multiple tools that respond to different
sets of events.

Note that, unlike sys.settrace(), events and callbacks are per
interpreter, not per thread.

Events

As a code object executes various events occur that might be of interest
to tools. By activating events and by registering callback functions
tools can respond to these events in any way that suits them. Events can
be set globally, or for individual code objects.

For 3.12, CPython will support the following events:

-   PY_START: Start of a Python function (occurs immediately after the
    call, the callee's frame will be on the stack)
-   PY_RESUME: Resumption of a Python function (for generator and
    coroutine functions), except for throw() calls.
-   PY_THROW: A Python function is resumed by a throw() call.
-   PY_RETURN: Return from a Python function (occurs immediately before
    the return, the callee's frame will be on the stack).
-   PY_YIELD: Yield from a Python function (occurs immediately before
    the yield, the callee's frame will be on the stack).
-   PY_UNWIND: Exit from a Python function during exception unwinding.
-   CALL: A call in Python code (event occurs before the call).
-   C_RETURN: Return from any callable, except Python functions (event
    occurs after the return).
-   C_RAISE: Exception raised from any callable, except Python functions
    (event occurs after the exit).
-   RAISE: An exception is raised, except those that cause a
    STOP_ITERATION event.
-   EXCEPTION_HANDLED: An exception is handled.
-   LINE: An instruction is about to be executed that has a different
    line number from the preceding instruction.
-   INSTRUCTION -- A VM instruction is about to be executed.
-   JUMP -- An unconditional jump in the control flow graph is made.
-   BRANCH -- A conditional branch is taken (or not).
-   STOP_ITERATION -- An artificial StopIteration is raised; see the
    STOP_ITERATION event.

More events may be added in the future.

All events will be attributes of the events namespace in sys.monitoring.
All events will represented by a power of two integer, so that they can
be combined with the | operator.

Events are divided into three groups:

Local events

Local events are associated with normal execution of the program and
happen at clearly defined locations. All local events can be disabled.
The local events are:

-   PY_START
-   PY_RESUME
-   PY_RETURN
-   PY_YIELD
-   CALL
-   LINE
-   INSTRUCTION
-   JUMP
-   BRANCH
-   STOP_ITERATION

Ancilliary events

Ancillary events can be monitored like other events, but are controlled
by another event:

-   C_RAISE
-   C_RETURN

The C_RETURN and C_RAISE events are are controlled by the CALL event.
C_RETURN and C_RAISE events will only be seen if the corresponding CALL
event is being monitored.

Other events

Other events are not necessarily tied to a specific location in the
program and cannot be individually disabled.

The other events that can be monitored are:

-   PY_THROW
-   PY_UNWIND
-   RAISE
-   EXCEPTION_HANDLED

The STOP_ITERATION event

PEP 380 <380#use-of-stopiteration-to-return-values> specifies that a
StopIteration exception is raised when returning a value from a
generator or coroutine. However, this is a very inefficient way to
return a value, so some Python implementations, notably CPython 3.12+,
do not raise an exception unless it would be visible to other code.

To allow tools to monitor for real exceptions without slowing down
generators and coroutines, the STOP_ITERATION event is provided.
STOP_ITERATION can be locally disabled, unlike RAISE.

Tool identifiers

The VM can support up to 6 tools at once. Before registering or
activating events, a tool should choose an identifier. Identifiers are
integers in the range 0 to 5.

    sys.monitoring.use_tool_id(id, name:str) -> None
    sys.monitoring.free_tool_id(id) -> None
    sys.monitoring.get_tool(id) ->  str | None

sys.monitoring.use_tool_id raises a ValueError if id is in use.
sys.monitoring.get_tool returns the name of the tool if id is in use,
otherwise it returns None.

All IDs are treated the same by the VM with regard to events, but the
following IDs are pre-defined to make co-operation of tools easier:

    sys.monitoring.DEBUGGER_ID = 0
    sys.monitoring.COVERAGE_ID = 1
    sys.monitoring.PROFILER_ID = 2
    sys.monitoring.OPTIMIZER_ID = 5

There is no obligation to set an ID, nor is there anything preventing a
tool from using an ID even it is already in use. However, tools are
encouraged to use a unique ID and respect other tools.

For example, if a debugger were attached and DEBUGGER_ID were in use, it
should report an error, rather than carrying on regardless.

The OPTIMIZER_ID is provided for tools like Cinder or PyTorch that want
to optimize Python code, but need to decide what to optimize in a way
that depends on some wider context.

Setting events globally

Events can be controlled globally by modifying the set of events being
monitored:

-   sys.monitoring.get_events(tool_id:int)->int Returns the int
    representing all the active events.
-   sys.monitoring.set_events(tool_id:int, event_set: int) Activates all
    events which are set in event_set. Raises a ValueError if tool_id is
    not in use.

No events are active by default.

Per code object events

Events can also be controlled on a per code object basis:

-   sys.monitoring.get_local_events(tool_id:int, code: CodeType)->int
    Returns all the local events for code
-   sys.monitoring.set_local_events(tool_id:int, code: CodeType, event_set: int)
    Activates all the local events for code which are set in event_set.
    Raises a ValueError if tool_id is not in use.

Local events add to global events, but do not mask them. In other words,
all global events will trigger for a code object, regardless of the
local events.

Register callback functions

To register a callable for events call:

    sys.monitoring.register_callback(tool_id:int, event: int, func: Callable | None) -> Callable | None

If another callback was registered for the given tool_id and event, it
is unregistered and returned. Otherwise register_callback returns None.

Functions can be unregistered by calling
sys.monitoring.register_callback(tool_id, event, None).

Callback functions can be registered and unregistered at any time.

Registering or unregistering a callback function will generate a
sys.audit event.

Callback function arguments

When an active event occurs, the registered callback function is called.
Different events will provide the callback function with different
arguments, as follows:

-   PY_START and PY_RESUME:

        func(code: CodeType, instruction_offset: int) -> DISABLE | Any

-   PY_RETURN and PY_YIELD:

      func(code: CodeType, instruction_offset: int, retval: object) -> DISABLE | Any

-   CALL, C_RAISE and C_RETURN:

      func(code: CodeType, instruction_offset: int, callable: object, arg0: object | MISSING) -> DISABLE | Any

      If there are no arguments, arg0 is set to MISSING.

-   RAISE and EXCEPTION_HANDLED:

      func(code: CodeType, instruction_offset: int, exception: BaseException) -> DISABLE | Any

-   LINE:

      func(code: CodeType, line_number: int) -> DISABLE | Any

-   BRANCH:

      func(code: CodeType, instruction_offset: int, destination_offset: int) -> DISABLE | Any

    Note that the destination_offset is where the code will next
    execute. For an untaken branch this will be the offset of the
    instruction following the branch.

-   INSTRUCTION:

      func(code: CodeType, instruction_offset: int) -> DISABLE | Any

If a callback function returns DISABLE, then that function will no
longer be called for that (code, instruction_offset) until
sys.monitoring.restart_events() is called. This feature is provided for
coverage and other tools that are only interested seeing an event once.

Note that sys.monitoring.restart_events() is not specific to one tool,
so tools must be prepared to receive events that they have chosen to
DISABLE.

Events in callback functions

Events are suspended in callback functions and their callees for the
tool that registered that callback.

That means that other tools will see events in the callback functions
for other tools. This could be useful for debugging a profiling tool,
but would produce misleading profiles, as the debugger tool would show
up in the profile.

Order of events

If an instructions triggers several events they occur in the following
order:

-   LINE
-   INSTRUCTION
-   All other events (only one of these events can occur per
    instruction)

Each event is delivered to tools in ascending order of ID.

The "call" event group

Most events are independent; setting or disabling one event has no
effect on the others. However, the CALL, C_RAISE and C_RETURN events
form a group. If any of those events are set or disabled, then all
events in the group are. Disabling a CALL event will not disable the
matching C_RAISE or C_RETURN, but will disable all subsequent events.

Attributes of the sys.monitoring namespace

-   def use_tool_id(id)->None
-   def free_tool_id(id)->None
-   def get_events(tool_id: int)->int
-   def set_events(tool_id: int, event_set: int)->None
-   def get_local_events(tool_id: int, code: CodeType)->int
-   def set_local_events(tool_id: int, code: CodeType, event_set: int)->None
-   def register_callback(tool_id: int, event: int, func: Callable)->Optional[Callable]
-   def restart_events()->None
-   DISABLE: object
-   MISSING: object

Access to "debug only" features

Some features of the standard library are not accessible to normal code,
but are accessible to debuggers. For example, setting local variables,
or the line number.

These features will be available to callback functions.

Backwards Compatibility

This PEP is mostly backwards compatible.

There are some compatibility issues with PEP 523, as the behavior of PEP
523 plugins is outside of the VM's control. It is up to PEP 523 plugins
to ensure that they respect the semantics of this PEP. Simple plugins
that do not change the state of the VM, and defer execution to
_PyEval_EvalFrameDefault() should continue to work.

sys.settrace and sys.setprofile will act as if they were tools 6 and 7
respectively, so can be used alongside this PEP.

This means that sys.settrace and sys.setprofile may not work correctly
with all PEP 523 plugins. Although, simple PEP 523 plugins, as described
above, should be fine.

Performance

If no events are active, this PEP should have a small positive impact on
performance. Experiments show between 1 and 2% speedup from not
supporting sys.settrace directly.

The performance of sys.settrace will be about the same. The performance
of sys.setprofile should be better. However, tools relying on
sys.settrace and sys.setprofile can be made a lot faster by using the
API provided by this PEP.

If a small set of events are active, e.g. for a debugger, then the
overhead of callbacks will be orders of magnitudes less than for
sys.settrace and much cheaper than using PEP 523.

Coverage tools can be implemented at very low cost, by returning DISABLE
in all callbacks.

For heavily instrumented code, e.g. using LINE, performance should be
better than sys.settrace, but not by that much as performance will be
dominated by the time spent in callbacks.

For optimizing virtual machines, such as future versions of CPython (and
PyPy should they choose to support this API), changes to the set active
events in the midst of a long running program could be quite expensive,
possibly taking hundreds of milliseconds as it triggers
de-optimizations. Once such de-optimization has occurred, performance
should recover as the VM can re-optimize the instrumented code.

In general these operations can be considered to be fast:

-   def get_events(tool_id: int)->int
-   def get_local_events(tool_id: int, code: CodeType)->int
-   def register_callback(tool_id: int, event: int, func: Callable)->Optional[Callable]
-   def get_tool(tool_id) -> str | None

These operations are slower, but not especially so:

-   def set_local_events(tool_id: int, code: CodeType, event_set: int)->None

And these operations should be regarded as slow:

-   def use_tool_id(id, name:str)->None
-   def free_tool_id(id)->None
-   def set_events(tool_id: int, event_set: int)->None
-   def restart_events()->None

How slow the slow operations are depends on when they happen. If done
early in the program, before modules are loaded, they should be fairly
inexpensive.

Memory Consumption

When not in use, this PEP will have a negligible change on memory
consumption.

How memory is used is very much an implementation detail. However, we
expect that for 3.12 the additional memory consumption per code object
will be roughly as follows:

+-------------+--------+--------+-------------+
|             |        |        |             |
|             |        | Events |             |
+=============+========+========+=============+
|   Tools     | Others |   LINE | INSTRUCTION |
+-------------+--------+--------+-------------+
|   One       | None   |   ≈40% |   ≈80%      |
+-------------+--------+--------+-------------+
| Two or more |   ≈40% | ≈120%  |   ≈200%     |
+-------------+--------+--------+-------------+
|             |        |        |             |
+-------------+--------+--------+-------------+

Security Implications

Allowing modification of running code has some security implications,
but no more than the ability to generate and call new code.

All the new functions listed above will trigger audit hooks.

Implementation

This outlines the proposed implementation for CPython 3.12. The actual
implementation for later versions of CPython and other Python
implementations may differ considerably.

The proposed implementation of this PEP will be built on top of the
quickening step of CPython 3.11, as described in
PEP 659 <659#quickening>. Instrumentation works in much the same way as
quickening, bytecodes are replaced with instrumented ones as needed.

For example, if the CALL event is turned on, then all call instructions
will be replaced with a INSTRUMENTED_CALL instruction.

Note that this will interfere with specialization, which will result in
some performance degradation in addition to the overhead of calling the
registered callable.

When the set of active events changes, the VM will immediately update
all code objects present on the call stack of any thread. It will also
set in place traps to ensure that all code objects are correctly
instrumented when called. Consequently changing the set of active events
should be done as infrequently as possible, as it could be quite an
expensive operation.

Other events, such as RAISE can be turned on or off cheaply, as they do
not rely on code instrumentation, but runtime checks when the underlying
event occurs.

The exact set of events that require instrumentation is an
implementation detail, but for the current design, the following events
will require instrumentation:

-   PY_START
-   PY_RESUME
-   PY_RETURN
-   PY_YIELD
-   CALL
-   LINE
-   INSTRUCTION
-   JUMP
-   BRANCH

Each instrumented bytecode will require an additional 8 bits of
information to note which tool the instrumentation applies to. LINE and
INSTRUCTION events require additional information, as they need to store
the original instruction, or even the instrumented instruction if they
overlap other instrumentation.

Implementing tools

It is the philosophy of this PEP that it should be possible for
third-party monitoring tools to achieve high-performance, not that it
should be easy for them to do so.

Converting events into data that is meaningful to the users is the
responsibility of the tool.

All events have a cost, and tools should attempt to the use set of
events that trigger the least often and still provide the necessary
information.

Debuggers

Inserting breakpoints

Breakpoints can be inserted setting per code object events, either LINE
or INSTRUCTION, and returning DISABLE for any events not matching a
breakpoint.

Stepping

Debuggers usually offer the ability to step execution by a single
instruction or line.

Like breakpoints, stepping can be implemented by setting per code object
events. As soon as normal execution is to be resumed, the local events
can be unset.

Attaching

Debuggers can use the PY_START and PY_RESUME events to be informed when
a code object is first encountered, so that any necessary breakpoints
can be inserted.

Coverage Tools

Coverage tools need to track which parts of the control graph have been
executed. To do this, they need to register for the PY_ events, plus
JUMP and BRANCH.

This information can be then be converted back into a line based report
after execution has completed.

Profilers

Simple profilers need to gather information about calls. To do this
profilers should register for the following events:

-   PY_START
-   PY_RESUME
-   PY_THROW
-   PY_RETURN
-   PY_YIELD
-   PY_UNWIND
-   CALL
-   C_RAISE
-   C_RETURN

Line based profilers

Line based profilers can use the LINE and JUMP events. Implementers of
profilers should be aware that instrumenting LINE events will have a
large impact on performance.

Note

Instrumenting profilers have significant overhead and will distort the
results of profiling. Unless you need exact call counts, consider using
a statistical profiler.

Rejected ideas

A draft version of this PEP proposed making the user responsible for
inserting the monitoring instructions, rather than have VM do it.
However, that puts too much of a burden on the tools, and would make
attaching a debugger nearly impossible.

An earlier version of this PEP, proposed storing events as enums:

    class Event(enum.IntFlag):
        PY_START = ...

However, that would prevent monitoring of code before the enum module
was loaded and could cause unnecessary overhead.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 327 Title: Decimal Data Type Version: $Revision$ Last-Modified:
$Date$ Author: Facundo Batista<EMAIL_ADDRESS>Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 17-Oct-2003
Python-Version: 2.4 Post-History: 30-Nov-2003, 02-Jan-2004, 29-Jan-2004

Abstract

The idea is to have a Decimal data type, for every use where decimals
are needed but binary floating point is too inexact.

The Decimal data type will support the Python standard functions and
operations, and must comply with the decimal arithmetic ANSI standard
X3.274-1996[1].

Decimal will be floating point (as opposed to fixed point) and will have
bounded precision (the precision is the upper limit on the number of
significant digits in a result). However, precision is user-settable,
and a notion of significant trailing zeroes is supported so that
fixed-point usage is also possible.

This work is based on code and test functions written by Eric Price,
Aahz and Tim Peters. Just before Python 2.4a1, the decimal.py reference
implementation was moved into the standard library; along with the
documentation and the test suite, this was the work of Raymond
Hettinger. Much of the explanation in this PEP is taken from Cowlishaw's
work[2], comp.lang.python and python-dev.

Motivation

Here I'll expose the reasons of why I think a Decimal data type is
needed and why other numeric data types are not enough.

I wanted a Money data type, and after proposing a pre-PEP in
comp.lang.python, the community agreed to have a numeric data type with
the needed arithmetic behaviour, and then build Money over it: all the
considerations about quantity of digits after the decimal point,
rounding, etc., will be handled through Money. It is not the purpose of
this PEP to have a data type that can be used as Money without further
effort.

One of the biggest advantages of implementing a standard is that someone
already thought out all the creepy cases for you. And to a standard GvR
redirected me: Mike Cowlishaw's General Decimal Arithmetic
specification[3]. This document defines a general purpose decimal
arithmetic. A correct implementation of this specification will conform
to the decimal arithmetic defined in ANSI/IEEE standard 854-1987, except
for some minor restrictions, and will also provide unrounded decimal
arithmetic and integer arithmetic as proper subsets.

The problem with binary float

In decimal math, there are many numbers that can't be represented with a
fixed number of decimal digits, e.g. 1/3 = 0.3333333333.......

In base 2 (the way that standard floating point is calculated), 1/2 =
0.1, 1/4 = 0.01, 1/8 = 0.001, etc. Decimal 0.2 equals 2/10 equals 1/5,
resulting in the binary fractional number 0.001100110011001... As you
can see, the problem is that some decimal numbers can't be represented
exactly in binary, resulting in small roundoff errors.

So we need a decimal data type that represents exactly decimal numbers.
Instead of a binary data type, we need a decimal one.

Why floating point?

So we go to decimal, but why floating point?

Floating point numbers use a fixed quantity of digits (precision) to
represent a number, working with an exponent when the number gets too
big or too small. For example, with a precision of 5:

    1234 ==>   1234e0
    12345 ==>  12345e0
    123456 ==>  12346e1

(note that in the last line the number got rounded to fit in five
digits).

In contrast, we have the example of a long integer with infinite
precision, meaning that you can have the number as big as you want, and
you'll never lose any information.

In a fixed point number, the position of the decimal point is fixed. For
a fixed point data type, check Tim Peter's FixedPoint at SourceForge[4].
I'll go for floating point because it's easier to implement the
arithmetic behaviour of the standard, and then you can implement a fixed
point data type over Decimal.

But why can't we have a floating point number with infinite precision?
It's not so easy, because of inexact divisions. E.g.: 1/3 =
0.3333333333333... ad infinitum. In this case you should store an
infinite amount of 3s, which takes too much memory, ;).

John Roth proposed to eliminate the division operator and force the user
to use an explicit method, just to avoid this kind of trouble. This
generated adverse reactions in comp.lang.python, as everybody wants to
have support for the / operator in a numeric data type.

With this exposed maybe you're thinking "Hey! Can we just store the 1
and the 3 as numerator and denominator?", which takes us to the next
point.

Why not rational?

Rational numbers are stored using two integer numbers, the numerator and
the denominator. This implies that the arithmetic operations can't be
executed directly (e.g. to add two rational numbers you first need to
calculate the common denominator).

Quoting Alex Martelli:

  The performance implications of the fact that summing two rationals
  (which take O(M) and O(N) space respectively) gives a rational which
  takes O(M+N) memory space is just too troublesome. There are excellent
  Rational implementations in both pure Python and as extensions (e.g.,
  gmpy), but they'll always be a "niche market" IMHO. Probably worth
  PEPping, not worth doing without Decimal -- which is the right way to
  represent sums of money, a truly major use case in the real world.

Anyway, if you're interested in this data type, you maybe will want to
take a look at PEP 239: Adding a Rational Type to Python.

So, what do we have?

The result is a Decimal data type, with bounded precision and floating
point.

Will it be useful? I can't say it better than Alex Martelli:

  Python (out of the box) doesn't let you have binary floating point
  numbers with whatever precision you specify: you're limited to what
  your hardware supplies. Decimal, be it used as a fixed or floating
  point number, should suffer from no such limitation: whatever bounded
  precision you may specify on number creation (your memory permitting)
  should work just as well. Most of the expense of programming
  simplicity can be hidden from application programs and placed in a
  suitable decimal arithmetic type. As per
  http://speleotrove.com/decimal/, a single data type can be used for
  integer, fixed-point, and floating-point decimal arithmetic -- and for
  money arithmetic which doesn't drive the application programmer crazy.

There are several uses for such a data type. As I said before, I will
use it as base for Money. In this case the bounded precision is not an
issue; quoting Tim Peters:

  A precision of 20 would be way more than enough to account for total
  world economic output, down to the penny, since the beginning of time.

General Decimal Arithmetic Specification

Here I'll include information and descriptions that are part of the
specification[5] (the structure of the number, the context, etc.). All
the requirements included in this section are not for discussion
(barring typos or other mistakes), as they are in the standard, and the
PEP is just for implementing the standard.

Because of copyright restrictions, I can not copy here explanations
taken from the specification, so I'll try to explain it in my own words.
I firmly encourage you to read the original specification document[6]
for details or if you have any doubt.

The Arithmetic Model

The specification is based on a decimal arithmetic model, as defined by
the relevant standards: IEEE 854[7], ANSI X3-274[8], and the proposed
revision[9] of IEEE 754[10].

The model has three components:

-   Numbers: just the values that the operation uses as input or output.
-   Operations: addition, multiplication, etc.
-   Context: a set of parameters and rules that the user can select and
    which govern the results of operations (for example, the precision
    to be used).

Numbers

Numbers may be finite or special values. The former can be represented
exactly. The latter are infinites and undefined (such as 0/0).

Finite numbers are defined by three parameters:

-   Sign: 0 (positive) or 1 (negative).
-   Coefficient: a non-negative integer.
-   Exponent: a signed integer, the power of ten of the coefficient
    multiplier.

The numerical value of a finite number is given by:

    (-1)**sign * coefficient * 10**exponent

Special values are named as following:

-   Infinity: a value which is infinitely large. Could be positive or
    negative.
-   Quiet NaN ("qNaN"): represent undefined results (Not a Number). Does
    not cause an Invalid operation condition. The sign in a NaN has no
    meaning.
-   Signaling NaN ("sNaN"): also Not a Number, but will cause an Invalid
    operation condition if used in any operation.

Context

The context is a set of parameters and rules that the user can select
and which govern the results of operations (for example, the precision
to be used).

The context gets that name because it surrounds the Decimal numbers,
with parts of context acting as input to, and output of, operations.
It's up to the application to work with one or several contexts, but
definitely the idea is not to get a context per Decimal number. For
example, a typical use would be to set the context's precision to 20
digits at the start of a program, and never explicitly use context
again.

These definitions don't affect the internal storage of the Decimal
numbers, just the way that the arithmetic operations are performed.

The context is mainly defined by the following parameters (see Context
Attributes for all context attributes):

-   Precision: The maximum number of significant digits that can result
    from an arithmetic operation (integer > 0). There is no maximum for
    this value.
-   Rounding: The name of the algorithm to be used when rounding is
    necessary, one of "round-down", "round-half-up", "round-half-even",
    "round-ceiling", "round-floor", "round-half-down", and "round-up".
    See Rounding Algorithms below.
-   Flags and trap-enablers: Exceptional conditions are grouped into
    signals, controllable individually, each consisting of a flag
    (boolean, set when the signal occurs) and a trap-enabler (a boolean
    that controls behavior). The signals are: "clamped",
    "division-by-zero", "inexact", "invalid-operation", "overflow",
    "rounded", "subnormal" and "underflow".

Default Contexts

The specification defines two default contexts, which should be easily
selectable by the user.

Basic Default Context:

-   flags: all set to 0
-   trap-enablers: inexact, rounded, and subnormal are set to 0; all
    others are set to 1
-   precision: is set to 9
-   rounding: is set to round-half-up

Extended Default Context:

-   flags: all set to 0
-   trap-enablers: all set to 0
-   precision: is set to 9
-   rounding: is set to round-half-even

Exceptional Conditions

The table below lists the exceptional conditions that may arise during
the arithmetic operations, the corresponding signal, and the defined
result. For details, see the specification[11].

  Condition           Signal              Result
  ------------------- ------------------- ------------------------------------------------------------------------
  Clamped             clamped             see spec[12]
  Division by zero    division-by-zero    [sign,inf]
  Inexact             inexact             unchanged
  Invalid operation   invalid-operation   [0,qNaN] (or [s,qNaN] or [s,qNaN,d] when the cause is a signaling NaN)
  Overflow            overflow            depends on the rounding mode
  Rounded             rounded             unchanged
  Subnormal           subnormal           unchanged
  Underflow           underflow           see spec[13]

Note: when the standard talks about "Insufficient storage", as long as
this is implementation-specific behaviour about not having enough
storage to keep the internals of the number, this implementation will
raise MemoryError.

Regarding Overflow and Underflow, there's been a long discussion in
python-dev about artificial limits. The general consensus is to keep the
artificial limits only if there are important reasons to do that. Tim
Peters gives us three:

  ...eliminating bounds on exponents effectively means overflow (and
  underflow) can never happen. But overflow is a valuable safety net in
  real life fp use, like a canary in a coal mine, giving danger signs
  early when a program goes insane.

  Virtually all implementations of 854 use (and as IBM's standard even
  suggests) "forbidden" exponent values to encode non-finite numbers
  (infinities and NaNs). A bounded exponent can do this at virtually no
  extra storage cost. If the exponent is unbounded, then additional bits
  have to be used instead. This cost remains hidden until more time- and
  space- efficient implementations are attempted.

  Big as it is, the IBM standard is a tiny start at supplying a complete
  numeric facility. Having no bound on exponent size will enormously
  complicate the implementations of, e.g., decimal sin() and cos()
  (there's then no a priori limit on how many digits of pi effectively
  need to be known in order to perform argument reduction).

Edward Loper give us an example of when the limits are to be crossed:
probabilities.

That said, Robert Brewer and Andrew Lentvorski want the limits to be
easily modifiable by the users. Actually, this is quite possible:

    >>> d1 = Decimal("1e999999999")     # at the exponent limit
    >>> d1
    Decimal("1E+999999999")
    >>> d1 * 10                         # exceed the limit, got infinity
    Traceback (most recent call last):
      File "<pyshell#3>", line 1, in ?
        d1 * 10
      ...
      ...
    Overflow: above Emax
    >>> getcontext().Emax =<PHONE_NUMBER>  # increase the limit
    >>> d1 * 10                         # does not exceed any more
    Decimal("1.0E+1000000000")
    >>> d1 * 100                        # exceed again
    Traceback (most recent call last):
      File "<pyshell#3>", line 1, in ?
        d1 * 100
      ...
      ...
    Overflow: above Emax

Rounding Algorithms

round-down: The discarded digits are ignored; the result is unchanged
(round toward 0, truncate):

    1.123 --> 1.12
    1.128 --> 1.12
    1.125 --> 1.12
    1.135 --> 1.13

round-half-up: If the discarded digits represent greater than or equal
to half (0.5) then the result should be incremented by 1; otherwise the
discarded digits are ignored:

    1.123 --> 1.12
    1.128 --> 1.13
    1.125 --> 1.13
    1.135 --> 1.14

round-half-even: If the discarded digits represent greater than half
(0.5) then the result coefficient is incremented by 1; if they represent
less than half, then the result is not adjusted; otherwise the result is
unaltered if its rightmost digit is even, or incremented by 1 if its
rightmost digit is odd (to make an even digit):

    1.123 --> 1.12
    1.128 --> 1.13
    1.125 --> 1.12
    1.135 --> 1.14

round-ceiling: If all of the discarded digits are zero or if the sign is
negative the result is unchanged; otherwise, the result is incremented
by 1 (round toward positive infinity):

    1.123 -->  1.13
    1.128 -->  1.13
    -1.123 --> -1.12
    -1.128 --> -1.12

round-floor: If all of the discarded digits are zero or if the sign is
positive the result is unchanged; otherwise, the absolute value of the
result is incremented by 1 (round toward negative infinity):

    1.123 -->  1.12
    1.128 -->  1.12
    -1.123 --> -1.13
    -1.128 --> -1.13

round-half-down: If the discarded digits represent greater than half
(0.5) then the result is incremented by 1; otherwise the discarded
digits are ignored:

    1.123 --> 1.12
    1.128 --> 1.13
    1.125 --> 1.12
    1.135 --> 1.13

round-up: If all of the discarded digits are zero the result is
unchanged, otherwise the result is incremented by 1 (round away from 0):

    1.123 --> 1.13
    1.128 --> 1.13
    1.125 --> 1.13
    1.135 --> 1.14

Rationale

I must separate the requirements in two sections. The first is to comply
with the ANSI standard. All the requirements for this are specified in
the Mike Cowlishaw's work[14]. He also provided a very large suite of
test cases.

The second section of requirements (standard Python functions support,
usability, etc.) is detailed from here, where I'll include all the
decisions made and why, and all the subjects still being discussed.

Explicit construction

The explicit construction does not get affected by the context (there is
no rounding, no limits by the precision, etc.), because the context
affects just operations' results. The only exception to this is when
you're Creating from Context.

From int or long

There's no loss and no need to specify any other information:

    Decimal(35)
    Decimal(-124)

From string

Strings containing Python decimal integer literals and Python float
literals will be supported. In this transformation there is no loss of
information, as the string is directly converted to Decimal (there is
not an intermediate conversion through float):

    Decimal("-12")
    Decimal("23.2e-7")

Also, you can construct in this way all special values (Infinity and Not
a Number):

    Decimal("Inf")
    Decimal("NaN")

From float

The initial discussion on this item was what should happen when passing
floating point to the constructor:

1.  Decimal(1.1) == Decimal('1.1')
2.  Decimal(1.1) == Decimal('110000000000000008881784197001252...e-51')
3.  an exception is raised

Several people alleged that (1) is the better option here, because it's
what you expect when writing Decimal(1.1). And quoting John Roth, it's
easy to implement:

  It's not at all difficult to find where the actual number ends and
  where the fuzz begins. You can do it visually, and the algorithms to
  do it are quite well known.

But If I really want my number to be
Decimal('110000000000000008881784197001252...e-51'), why can't I write
Decimal(1.1)? Why should I expect Decimal to be "rounding" it? Remember
that 1.1 is binary floating point, so I can predict the result. It's not
intuitive to a beginner, but that's the way it is.

Anyway, Paul Moore showed that (1) can't work, because:

    (1) says  D(1.1) == D('1.1')
    but       1.1 == 1.1000000000000001
    so        D(1.1) == D(1.1000000000000001)
    together: D(1.1000000000000001) == D('1.1')

which is wrong, because if I write Decimal('1.1') it is exact, not
D(1.1000000000000001). He also proposed to have an explicit conversion
to float. bokr says you need to put the precision in the constructor and
mwilson agreed:

    d = Decimal (1.1, 1)  # take float value to 1 decimal place
    d = Decimal (1.1)  # gets `places` from pre-set context

But Alex Martelli says that:

  Constructing with some specified precision would be fine. Thus, I
  think "construction from float with some default precision" runs a
  substantial risk of tricking naive users.

So, the accepted solution through c.l.p is that you can not call Decimal
with a float. Instead you must use a method: Decimal.from_float(). The
syntax:

    Decimal.from_float(floatNumber, [decimal_places])

where floatNumber is the float number origin of the construction and
decimal_places are the number of digits after the decimal point where
you apply a round-half-up rounding, if any. In this way you can do, for
example:

    Decimal.from_float(1.1, 2): The same as doing Decimal('1.1').
    Decimal.from_float(1.1, 16): The same as doing Decimal('1.1000000000000001').
    Decimal.from_float(1.1): The same as doing Decimal('1100000000000000088817841970012523233890533447265625e-51').

Based on later discussions, it was decided to omit from_float() from the
API for Py2.4. Several ideas contributed to the thought process:

-   Interactions between decimal and binary floating point force the
    user to deal with tricky issues of representation and round-off.
    Avoidance of those issues is a primary reason for having the module
    in the first place.

-   The first release of the module should focus on that which is safe,
    minimal, and essential.

-   While theoretically nice, real world use cases for interactions
    between floats and decimals are lacking. Java included float/decimal
    conversions to handle an obscure case where calculations are best
    performed in decimal even though a legacy data structure requires
    the inputs and outputs to be stored in binary floating point.

-   If the need arises, users can use string representations as an
    intermediate type. The advantage of this approach is that it makes
    explicit the assumptions about precision and representation (no
    wondering what is going on under the hood).

-   The Java docs for BigDecimal(double val) reflected their experiences
    with the constructor:

        The results of this constructor can be somewhat
        unpredictable and its use is generally not recommended.

From tuples

Aahz suggested to construct from tuples: it's easier to implement
eval()'s round trip and "someone who has numeric values representing a
Decimal does not need to convert them to a string."

The structure will be a tuple of three elements: sign, number and
exponent. The sign is 1 or 0, the number is a tuple of decimal digits
and the exponent is a signed int or long:

    Decimal((1, (3, 2, 2, 5), -2))     # for -32.25

Of course, you can construct in this way all special values:

    Decimal( (0, (0,), 'F') )          # for Infinity
    Decimal( (0, (0,), 'n') )          # for Not a Number

From Decimal

No mystery here, just a copy.

Syntax for All Cases

    Decimal(value1)
    Decimal.from_float(value2, [decimal_places])

where value1 can be int, long, string, 3-tuple or Decimal, value2 can
only be float, and decimal_places is an optional non negative int.

Creating from Context

This item arose in python-dev from two sources in parallel. Ka-Ping Yee
proposes to pass the context as an argument at instance creation (he
wants the context he passes to be used only in creation time: "It would
not be persistent"). Tony Meyer asks from_string to honor the context if
it receives a parameter "honour_context" with a True value. (I don't
like it, because the doc specifies that the context be honored and I
don't want the method to comply with the specification regarding the
value of an argument.)

Tim Peters gives us a reason to have a creation that uses context:

  In general number-crunching, literals may be given to high precision,
  but that precision isn't free and usually isn't needed

Casey Duncan wants to use another method, not a bool arg:

  I find boolean arguments a general anti-pattern, especially given we
  have class methods. Why not use an alternate constructor like
  Decimal.rounded_to_context("3.14159265").

In the process of deciding the syntax of that, Tim came up with a better
idea: he proposes not to have a method in Decimal to create with a
different context, but having instead a method in Context to create a
Decimal instance. Basically, instead of:

    D.using_context(number, context)

it will be:

    context.create_decimal(number)

From Tim:

  While all operations in the spec except for the two to-string
  operations use context, no operations in the spec support an optional
  local context. That the Decimal() constructor ignores context by
  default is an extension to the spec. We must supply a context-honoring
  from-string operation to meet the spec. I recommend against any
  concept of "local context" in any operation -- it complicates the
  model and isn't necessary.

So, we decided to use a context method to create a Decimal that will use
(only to be created) that context in particular (for further operations
it will use the context of the thread). But, a method with what name?

Tim Peters proposes three methods to create from diverse sources
(from_string, from_int, from_float). I proposed to use one method,
create_decimal(), without caring about the data type. Michael Chermside:
"The name just fits my brain. The fact that it uses the context is
obvious from the fact that it's Context method".

The community agreed with that. I think that it's OK because a newbie
will not be using the creation method from Context (the separate method
in Decimal to construct from float is just to prevent newbies from
encountering binary floating point issues).

So, in short, if you want to create a Decimal instance using a
particular context (that will be used just at creation time and not any
further), you'll have to use a method of that context:

    # n is any datatype accepted in Decimal(n) plus float
    mycontext.create_decimal(n)

Example:

    >>> # create a standard decimal instance
    >>> Decimal("11.2233445566778899")
    Decimal("11.2233445566778899")
    >>>
    >>> # create a decimal instance using the thread context
    >>> thread_context = getcontext()
    >>> thread_context.prec
    28
    >>> thread_context.create_decimal("11.2233445566778899")
    Decimal("11.2233445566778899")
    >>>
    >>> # create a decimal instance using other context
    >>> other_context = thread_context.copy()
    >>> other_context.prec = 4
    >>> other_context.create_decimal("11.2233445566778899")
    Decimal("11.22")

Implicit construction

As the implicit construction is the consequence of an operation, it will
be affected by the context as is detailed in each point.

John Roth suggested that "The other type should be handled in the same
way the decimal() constructor would handle it". But Alex Martelli thinks
that

  this total breach with Python tradition would be a terrible mistake.
  23+"43" is NOT handled in the same way as 23+int("45"), and a VERY
  good thing that is too. It's a completely different thing for a user
  to EXPLICITLY indicate they want construction (conversion) and to just
  happen to sum two objects one of which by mistake could be a string.

So, here I define the behaviour again for each data type.

From int or long

An int or long is a treated like a Decimal explicitly constructed from
Decimal(str(x)) in the current context (meaning that the to-string rules
for rounding are applied and the appropriate flags are set). This
guarantees that expressions like Decimal('1234567') + 13579 match the
mental model of Decimal('1234567') + Decimal('13579'). That model works
because all integers are representable as strings without representation
error.

From string

Everybody agrees to raise an exception here.

From float

Aahz is strongly opposed to interact with float, suggesting an explicit
conversion:

  The problem is that Decimal is capable of greater precision, accuracy,
  and range than float.

The example of the valid python expression, 35 + 1.1, seems to suggest
that Decimal(35) + 1.1 should also be valid. However, a closer look
shows that it only demonstrates the feasibility of integer to floating
point conversions. Hence, the correct analog for decimal floating point
is 35 + Decimal(1.1). Both coercions, int-to-float and int-to-Decimal,
can be done without incurring representation error.

The question of how to coerce between binary and decimal floating point
is more complex. I proposed allowing the interaction with float, making
an exact conversion and raising ValueError if exceeds the precision in
the current context (this is maybe too tricky, because for example with
a precision of 9, Decimal(35) + 1.2 is OK but Decimal(35) + 1.1 raises
an error).

This resulted to be too tricky. So tricky, that c.l.p agreed to raise
TypeError in this case: you could not mix Decimal and float.

From Decimal

There isn't any issue here.

Use of Context

In the last pre-PEP I said that "The Context must be omnipresent,
meaning that changes to it affects all the current and future Decimal
instances". I was wrong. In response, John Roth said:

  The context should be selectable for the particular usage. That is, it
  should be possible to have several different contexts in play at one
  time in an application.

In comp.lang.python, Aahz explained that the idea is to have a "context
per thread". So, all the instances of a thread belongs to a context, and
you can change a context in thread A (and the behaviour of the instances
of that thread) without changing nothing in thread B.

Also, and again correcting me, he said:

  (the) Context applies only to operations, not to Decimal instances;
  changing the Context does not affect existing instances if there are
  no operations on them.

Arguing about special cases when there's need to perform operations with
other rules that those of the current context, Tim Peters said that the
context will have the operations as methods. This way, the user "can
create whatever private context object(s) it needs, and spell arithmetic
as explicit method calls on its private context object(s), so that the
default thread context object is neither consulted nor modified".

Python Usability

-   Decimal should support the basic arithmetic
    (+, -, *, /, //, **, %, divmod) and comparison
    (==, !=, <, >, <=, >=, cmp) operators in the following cases (check
    Implicit Construction to see what types could OtherType be, and what
    happens in each case):

    -   Decimal op Decimal
    -   Decimal op otherType
    -   otherType op Decimal
    -   Decimal op= Decimal
    -   Decimal op= otherType

-   Decimal should support unary operators (-, +, abs).

-   repr() should round trip, meaning that:

        m = Decimal(...)
        m == eval(repr(m))

-   Decimal should be immutable.

-   Decimal should support the built-in methods:

    -   min, max
    -   float, int, long
    -   str, repr
    -   hash
    -   bool (0 is false, otherwise true)

There's been some discussion in python-dev about the behaviour of
hash(). The community agrees that if the values are the same, the hashes
of those values should also be the same. So, while Decimal(25) == 25 is
True, hash(Decimal(25)) should be equal to hash(25).

The detail is that you can NOT compare Decimal to floats or strings, so
we should not worry about them giving the same hashes. In short:

    hash(n) == hash(Decimal(n))   # Only if n is int, long, or Decimal

Regarding str() and repr() behaviour, Ka-Ping Yee proposes that repr()
have the same behaviour as str() and Tim Peters proposes that str()
behave like the to-scientific-string operation from the Spec.

This is possible, because (from Aahz): "The string form already contains
all the necessary information to reconstruct a Decimal object".

And it also complies with the Spec; Tim Peters:

  There's no requirement to have a method named "to_sci_string", the
  only requirement is that some way to spell to-sci-string's
  functionality be supplied. The meaning of to-sci-string is precisely
  specified by the standard, and is a good choice for both str(Decimal)
  and repr(Decimal).

Documentation

This section explains all the public methods and attributes of Decimal
and Context.

Decimal Attributes

Decimal has no public attributes. The internal information is stored in
slots and should not be accessed by end users.

Decimal Methods

Following are the conversion and arithmetic operations defined in the
Spec, and how that functionality can be achieved with the actual
implementation.

-   to-scientific-string: Use builtin function str():

        >>> d = Decimal('123456789012.345')
        >>> str(d)
        '1.23456789E+11'

-   to-engineering-string: Use method to_eng_string():

        >>> d = Decimal('123456789012.345')
        >>> d.to_eng_string()
        '123.456789E+9'

-   to-number: Use Context method create_decimal(). The standard
    constructor or from_float() constructor cannot be used because these
    do not use the context (as is specified in the Spec for this
    conversion).

-   abs: Use builtin function abs():

        >>> d = Decimal('-15.67')
        >>> abs(d)
        Decimal('15.67')

-   add: Use operator +:

        >>> d = Decimal('15.6')
        >>> d + 8
        Decimal('23.6')

-   subtract: Use operator -:

        >>> d = Decimal('15.6')
        >>> d - 8
        Decimal('7.6')

-   compare: Use method compare(). This method (and not the built-in
    function cmp()) should only be used when dealing with special
    values:

        >>> d = Decimal('-15.67')
        >>> nan = Decimal('NaN')
        >>> d.compare(23)
        '-1'
        >>> d.compare(nan)
        'NaN'
        >>> cmp(d, 23)
        -1
        >>> cmp(d, nan)
        1

-   divide: Use operator /:

        >>> d = Decimal('-15.67')
        >>> d / 2
        Decimal('-7.835')

-   divide-integer: Use operator //:

        >>> d = Decimal('-15.67')
        >>> d // 2
        Decimal('-7')

-   max: Use method max(). Only use this method (and not the built-in
    function max()) when dealing with special values:

        >>> d = Decimal('15')
        >>> nan = Decimal('NaN')
        >>> d.max(8)
        Decimal('15')
        >>> d.max(nan)
        Decimal('NaN')

-   min: Use method min(). Only use this method (and not the built-in
    function min()) when dealing with special values:

        >>> d = Decimal('15')
        >>> nan = Decimal('NaN')
        >>> d.min(8)
        Decimal('8')
        >>> d.min(nan)
        Decimal('NaN')

-   minus: Use unary operator -:

        >>> d = Decimal('-15.67')
        >>> -d
        Decimal('15.67')

-   plus: Use unary operator +:

        >>> d = Decimal('-15.67')
        >>> +d
        Decimal('-15.67')

-   multiply: Use operator *:

        >>> d = Decimal('5.7')
        >>> d * 3
        Decimal('17.1')

-   normalize: Use method normalize():

        >>> d = Decimal('123.45000')
        >>> d.normalize()
        Decimal('123.45')
        >>> d = Decimal('120.00')
        >>> d.normalize()
        Decimal('1.2E+2')

-   quantize: Use method quantize():

        >>> d = Decimal('2.17')
        >>> d.quantize(Decimal('0.001'))
        Decimal('2.170')
        >>> d.quantize(Decimal('0.1'))
        Decimal('2.2')

-   remainder: Use operator %:

        >>> d = Decimal('10')
        >>> d % 3
        Decimal('1')
        >>> d % 6
        Decimal('4')

-   remainder-near: Use method remainder_near():

        >>> d = Decimal('10')
        >>> d.remainder_near(3)
        Decimal('1')
        >>> d.remainder_near(6)
        Decimal('-2')

-   round-to-integral-value: Use method to_integral():

        >>> d = Decimal('-123.456')
        >>> d.to_integral()
        Decimal('-123')

-   same-quantum: Use method same_quantum():

        >>> d = Decimal('123.456')
        >>> d.same_quantum(Decimal('0.001'))
        True
        >>> d.same_quantum(Decimal('0.01'))
        False

-   square-root: Use method sqrt():

        >>> d = Decimal('123.456')
        >>> d.sqrt()
        Decimal('11.1110756')

-   power: User operator **:

        >>> d = Decimal('12.56')
        >>> d ** 2
        Decimal('157.7536')

Following are other methods and why they exist:

-   adjusted(): Returns the adjusted exponent. This concept is defined
    in the Spec: the adjusted exponent is the value of the exponent of a
    number when that number is expressed as though in scientific
    notation with one digit before any decimal point:

        >>> d = Decimal('12.56')
        >>> d.adjusted()
        1

-   from_float(): Class method to create instances from float data
    types:

        >>> d = Decimal.from_float(12.35)
        >>> d
        Decimal('12.3500000')

-   as_tuple(): Show the internal structure of the Decimal, the triple
    tuple. This method is not required by the Spec, but Tim Peters
    proposed it and the community agreed to have it (it's useful for
    developing and debugging):

        >>> d = Decimal('123.4')
        >>> d.as_tuple()
        (0, (1, 2, 3, 4), -1)
        >>> d = Decimal('-2.34e5')
        >>> d.as_tuple()
        (1, (2, 3, 4), 3)

Context Attributes

These are the attributes that can be changed to modify the context.

-   prec (int): the precision:

        >>> c.prec
        9

-   rounding (str): rounding type (how to round):

        >>> c.rounding
        'half_even'

-   trap_enablers (dict): if trap_enablers[exception] = 1, then an
    exception is raised when it is caused:

        >>> c.trap_enablers[Underflow]
        0
        >>> c.trap_enablers[Clamped]
        0

-   flags (dict): when an exception is caused, flags[exception] is
    incremented (whether or not the trap_enabler is set). Should be
    reset by the user of Decimal instance:

        >>> c.flags[Underflow]
        0
        >>> c.flags[Clamped]
        0

-   Emin (int): minimum exponent:

        >>> c.Emin
        -999999999

-   Emax (int): maximum exponent:

        >>> c.Emax
        999999999

-   capitals (int): boolean flag to use 'E' (True/1) or 'e' (False/0) in
    the string (for example, '1.32e+2' or '1.32E+2'):

        >>> c.capitals
        1

Context Methods

The following methods comply with Decimal functionality from the Spec.
Be aware that the operations that are called through a specific context
use that context and not the thread context.

To use these methods, take note that the syntax changes when the
operator is binary or unary, for example:

    >>> mycontext.abs(Decimal('-2'))
    '2'
    >>> mycontext.multiply(Decimal('2.3'), 5)
    '11.5'

So, the following are the Spec operations and conversions and how to
achieve them through a context (where d is a Decimal instance and n a
number that can be used in an Implicit construction):

-   to-scientific-string: to_sci_string(d)
-   to-engineering-string: to_eng_string(d)
-   to-number: create_decimal(number), see Explicit construction for
    number.
-   abs: abs(d)
-   add: add(d, n)
-   subtract: subtract(d, n)
-   compare: compare(d, n)
-   divide: divide(d, n)
-   divide-integer: divide_int(d, n)
-   max: max(d, n)
-   min: min(d, n)
-   minus: minus(d)
-   plus: plus(d)
-   multiply: multiply(d, n)
-   normalize: normalize(d)
-   quantize: quantize(d, d)
-   remainder: remainder(d)
-   remainder-near: remainder_near(d)
-   round-to-integral-value: to_integral(d)
-   same-quantum: same_quantum(d, d)
-   square-root: sqrt(d)
-   power: power(d, n)

The divmod(d, n) method supports decimal functionality through Context.

These are methods that return useful information from the Context:

-   Etiny(): Minimum exponent considering precision. :

        >>> c.Emin
        -999999999
        >>> c.Etiny()
        -1000000007

-   Etop(): Maximum exponent considering precision. :

        >>> c.Emax
        999999999
        >>> c.Etop()
        999999991

-   copy(): Returns a copy of the context.

Reference Implementation

As of Python 2.4-alpha, the code has been checked into the standard
library. The latest version is available from:

http://svn.python.org/view/python/trunk/Lib/decimal.py

The test cases are here:

http://svn.python.org/view/python/trunk/Lib/test/test_decimal.py

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] ANSI standard X3.274-1996 (Programming Language REXX):
http://www.rexxla.org/Standards/ansi.html

[2] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

[3] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

[4] Tim Peter's FixedPoint at SourceForge:
http://fixedpoint.sourceforge.net/

[5] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

[6] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

[7] ANSI/IEEE standard 854-1987 (Radix-Independent Floating-Point
Arithmetic):
http://www.cs.berkeley.edu/~ejr/projects/754/private/drafts/854-1987/dir.html
(unofficial text; official copies can be ordered from
http://standards.ieee.org/catalog/ordering.html)

[8] ANSI standard X3.274-1996 (Programming Language REXX):
http://www.rexxla.org/Standards/ansi.html

[9] IEEE 754 revision: http://grouper.ieee.org/groups/754/revision.html

[10] IEEE 754 references:
http://babbage.cs.qc.edu/courses/cs341/IEEE-754references.html

[11] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

[12] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

[13] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

[14] General Decimal Arithmetic specification (Cowlishaw):
http://speleotrove.com/decimal/decarith.html (related documents and
links at http://speleotrove.com/decimal/)

license: Public Domain


text:
PEP: 519 Title: Adding a file system path protocol Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon<EMAIL_ADDRESS>Koos
Zevenhoven<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 11-May-2016 Python-Version: 3.6
Post-History: 11-May-2016, 12-May-2016, 13-May-2016 Resolution:
https://mail.python.org/pipermail/python-dev/2016-May/144646.html

Abstract

This PEP proposes a protocol for classes which represent a file system
path to be able to provide a str or bytes representation. Changes to
Python's standard library are also proposed to utilize this protocol
where appropriate to facilitate the use of path objects where
historically only str and/or bytes file system paths are accepted. The
goal is to facilitate the migration of users towards rich path objects
while providing an easy way to work with code expecting str or bytes.

Rationale

Historically in Python, file system paths have been represented as
strings or bytes. This choice of representation has stemmed from C's own
decision to represent file system paths as const char *[1]. While that
is a totally serviceable format to use for file system paths, it's not
necessarily optimal. At issue is the fact that while all file system
paths can be represented as strings or bytes, not all strings or bytes
represent a file system path. This can lead to issues where any e.g.
string duck-types to a file system path whether it actually represents a
path or not.

To help elevate the representation of file system paths from their
representation as strings and bytes to a richer object representation,
the pathlib module[2] was provisionally introduced in Python 3.4 through
PEP 428. While considered by some as an improvement over strings and
bytes for file system paths, it has suffered from a lack of adoption.
Typically the key issue listed for the low adoption rate has been the
lack of support in the standard library. This lack of support required
users of pathlib to manually convert path objects to strings by calling
str(path) which many found error-prone.

One issue in converting path objects to strings comes from the fact that
the only generic way to get a string representation of the path was to
pass the object to str(). This can pose a problem when done blindly as
nearly all Python objects have some string representation whether they
are a path or not, e.g. str(None) will give a result that
builtins.open()[3] will happily use to create a new file.

Exacerbating this whole situation is the DirEntry object[4]. While path
objects have a representation that can be extracted using str(),
DirEntry objects expose a path attribute instead. Having no common
interface between path objects, DirEntry, and any other third-party path
library has become an issue. A solution that allows any
path-representing object to declare that it is a path and a way to
extract a low-level representation that all path objects could support
is desired.

This PEP then proposes to introduce a new protocol to be followed by
objects which represent file system paths. Providing a protocol allows
for explicit signaling of what objects represent file system paths as
well as a way to extract a lower-level representation that can be used
with older APIs which only support strings or bytes.

Discussions regarding path objects that led to this PEP can be found in
multiple threads on the python-ideas mailing list archive [5] for the
months of March and April 2016 and on the python-dev mailing list
archives[6] during April 2016.

Proposal

This proposal is split into two parts. One part is the proposal of a
protocol for objects to declare and provide support for exposing a file
system path representation. The other part deals with changes to
Python's standard library to support the new protocol. These changes
will also lead to the pathlib module dropping its provisional status.

Protocol

The following abstract base class defines the protocol for an object to
be considered a path object:

    import abc
    import typing as t

    class PathLike(abc.ABC):

        """Abstract base class for implementing the file system path protocol."""

        @abc.abstractmethod
        def __fspath__(self) -> t.Union[str, bytes]:
            """Return the file system path representation of the object."""
            raise NotImplementedError

Objects representing file system paths will implement the __fspath__()
method which will return the str or bytes representation of the path.
The str representation is the preferred low-level path representation as
it is human-readable and what people historically represent paths as.

Standard library changes

It is expected that most APIs in Python's standard library that
currently accept a file system path will be updated appropriately to
accept path objects (whether that requires code or simply an update to
documentation will vary). The modules mentioned below, though, deserve
specific details as they have either fundamental changes that empower
the ability to use path objects, or entail additions/removal of APIs.

builtins

open()[7] will be updated to accept path objects as well as continue to
accept str and bytes.

os

The fspath() function will be added with the following semantics:

    import typing as t

    def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
        """Return the string representation of the path.

        If str or bytes is passed in, it is returned unchanged. If __fspath__()
        returns something other than str or bytes then TypeError is raised. If
        this function is given something that is not str, bytes, or os.PathLike
        then TypeError is raised.
        """
        if isinstance(path, (str, bytes)):
            return path

        # Work from the object's type to match method resolution of other magic
        # methods.
        path_type = type(path)
        try:
            path = path_type.__fspath__(path)
        except AttributeError:
            if hasattr(path_type, '__fspath__'):
                raise
        else:
            if isinstance(path, (str, bytes)):
                return path
            else:
                raise TypeError("expected __fspath__() to return str or bytes, "
                                "not " + type(path).__name__)

        raise TypeError("expected str, bytes or os.PathLike object, not "
                        + path_type.__name__)

The os.fsencode()[8] and os.fsdecode()[9] functions will be updated to
accept path objects. As both functions coerce their arguments to bytes
and str, respectively, they will be updated to call __fspath__() if
present to convert the path object to a str or bytes representation, and
then perform their appropriate coercion operations as if the return
value from __fspath__() had been the original argument to the coercion
function in question.

The addition of os.fspath(), the updates to os.fsencode()/os.fsdecode(),
and the current semantics of pathlib.PurePath provide the semantics
necessary to get the path representation one prefers. For a path object,
pathlib.PurePath/Path can be used. To obtain the str or bytes
representation without any coercion, then os.fspath() can be used. If a
str is desired and the encoding of bytes should be assumed to be the
default file system encoding, then os.fsdecode() should be used. If a
bytes representation is desired and any strings should be encoded using
the default file system encoding, then os.fsencode() is used. This PEP
recommends using path objects when possible and falling back to string
paths as necessary and using bytes as a last resort.

Another way to view this is as a hierarchy of file system path
representations (highest- to lowest-level): path → str → bytes. The
functions and classes under discussion can all accept objects on the
same level of the hierarchy, but they vary in whether they promote or
demote objects to another level. The pathlib.PurePath class can promote
a str to a path object. The os.fspath() function can demote a path
object to a str or bytes instance, depending on what __fspath__()
returns. The os.fsdecode() function will demote a path object to a
string or promote a bytes object to a str. The os.fsencode() function
will demote a path or string object to bytes. There is no function that
provides a way to demote a path object directly to bytes while bypassing
string demotion.

The DirEntry object[10] will gain an __fspath__() method. It will return
the same value as currently found on the path attribute of DirEntry
instances.

The Protocol ABC will be added to the os module under the name
os.PathLike.

os.path

The various path-manipulation functions of os.path[11] will be updated
to accept path objects. For polymorphic functions that accept both bytes
and strings, they will be updated to simply use os.fspath().

During the discussions leading up to this PEP it was suggested that
os.path not be updated using an "explicit is better than implicit"
argument. The thinking was that since __fspath__() is polymorphic itself
it may be better to have code working with os.path extract the path
representation from path objects explicitly. There is also the
consideration that adding support this deep into the low-level OS APIs
will lead to code magically supporting path objects without requiring
any documentation updated, leading to potential complaints when it
doesn't work, unbeknownst to the project author.

But it is the view of this PEP that "practicality beats purity" in this
instance. To help facilitate the transition to supporting path objects,
it is better to make the transition as easy as possible than to worry
about unexpected/undocumented duck typing support for path objects by
projects.

There has also been the suggestion that os.path functions could be used
in a tight loop and the overhead of checking or calling __fspath__()
would be too costly. In this scenario only path-consuming APIs would be
directly updated and path-manipulating APIs like the ones in os.path
would go unmodified. This would require library authors to update their
code to support path objects if they performed any path manipulations,
but if the library code passed the path straight through then the
library wouldn't need to be updated. It is the view of this PEP and
Guido, though, that this is an unnecessary worry and that performance
will still be acceptable.

pathlib

The constructor for pathlib.PurePath and pathlib.Path will be updated to
accept PathLike objects. Both PurePath and Path will continue to not
accept bytes path representations, and so if __fspath__() returns bytes
it will raise an exception.

The path attribute will be removed as this PEP makes it redundant (it
has not been included in any released version of Python and so is not a
backwards-compatibility concern).

C API

The C API will gain an equivalent function to os.fspath():

    /*
        Return the file system path representation of the object.

        If the object is str or bytes, then allow it to pass through with
        an incremented refcount. If the object defines __fspath__(), then
        return the result of that method. All other types raise a TypeError.
    */
    PyObject *
    PyOS_FSPath(PyObject *path)
    {
        _Py_IDENTIFIER(__fspath__);
        PyObject *func = NULL;
        PyObject *path_repr = NULL;

        if (PyUnicode_Check(path) || PyBytes_Check(path)) {
            Py_INCREF(path);
            return path;
        }

        func = _PyObject_LookupSpecial(path, &PyId___fspath__);
        if (NULL == func) {
            return PyErr_Format(PyExc_TypeError,
                                "expected str, bytes or os.PathLike object, "
                                "not %S",
                                path->ob_type);
        }

        path_repr = PyObject_CallFunctionObjArgs(func, NULL);
        Py_DECREF(func);
        if (!PyUnicode_Check(path_repr) && !PyBytes_Check(path_repr)) {
            Py_DECREF(path_repr);
            return PyErr_Format(PyExc_TypeError,
                                "expected __fspath__() to return str or bytes, "
                                "not %S",
                                path_repr->ob_type);
        }

        return path_repr;
    }

Backwards compatibility

There are no explicit backwards-compatibility concerns. Unless an object
incidentally already defines a __fspath__() method there is no reason to
expect the pre-existing code to break or expect to have its semantics
implicitly changed.

Libraries wishing to support path objects and a version of Python prior
to Python 3.6 and the existence of os.fspath() can use the idiom of
path.__fspath__() if hasattr(path, "__fspath__") else path.

Implementation

This is the task list for what this PEP proposes to be changed in Python
3.6:

1.  Remove the path attribute from pathlib (done)
2.  Remove the provisional status of pathlib (done)
3.  Add os.PathLike (code and docs done)
4.  Add PyOS_FSPath() (code and docs done)
5.  Add os.fspath() (done)
6.  Update os.fsencode() (done)
7.  Update os.fsdecode() (done)
8.  Update pathlib.PurePath and pathlib.Path (done)
    1.  Add __fspath__()
    2.  Add os.PathLike support to the constructors
9.  Add __fspath__() to DirEntry (done)
10. Update builtins.open() (done)
11. Update os.path (done)
12. Add a glossary entry for "path-like" (done)
13. Update "What's New" (done)

Rejected Ideas

Other names for the protocol's method

Various names were proposed during discussions leading to this PEP,
including __path__, __pathname__, and __fspathname__. In the end people
seemed to gravitate towards __fspath__ for being unambiguous without
being unnecessarily long.

Separate str/bytes methods

At one point it was suggested that __fspath__() only return strings and
another method named __fspathb__() be introduced to return bytes. The
thinking is that by making __fspath__() not be polymorphic it could make
dealing with the potential string or bytes representations easier. But
the general consensus was that returning bytes will more than likely be
rare and that the various functions in the os module are the better
abstraction to promote over direct calls to __fspath__().

Providing a path attribute

To help deal with the issue of pathlib.PurePath not inheriting from str,
originally it was proposed to introduce a path attribute to mirror what
os.DirEntry provides. In the end, though, it was determined that a
protocol would provide the same result while not directly exposing an
API that most people will never need to interact with directly.

Have __fspath__() only return strings

Much of the discussion that led to this PEP revolved around whether
__fspath__() should be polymorphic and return bytes as well as str or
only return str. The general sentiment for this view was that bytes are
difficult to work with due to their inherent lack of information about
their encoding and PEP 383 makes it possible to represent all file
system paths using str with the surrogateescape handler. Thus, it would
be better to forcibly promote the use of str as the low-level path
representation for high-level path objects.

In the end, it was decided that using bytes to represent paths is simply
not going to go away and thus they should be supported to some degree.
The hope is that people will gravitate towards path objects like pathlib
and that will move people away from operating directly with bytes.

A generic string encoding mechanism

At one point there was a discussion of developing a generic mechanism to
extract a string representation of an object that had semantic meaning
(__str__() does not necessarily return anything of semantic significance
beyond what may be helpful for debugging). In the end, it was deemed to
lack a motivating need beyond the one this PEP is trying to solve in a
specific fashion.

Have __fspath__ be an attribute

It was briefly considered to have __fspath__ be an attribute instead of
a method. This was rejected for two reasons. One, historically protocols
have been implemented as "magic methods" and not "magic methods and
attributes". Two, there is no guarantee that the lower-level
representation of a path object will be pre-computed, potentially
misleading users that there was no expensive computation behind the
scenes in case the attribute was implemented as a property.

This also indirectly ties into the idea of introducing a path attribute
to accomplish the same thing. This idea has an added issue, though, of
accidentally having any object with a path attribute meet the protocol's
duck typing. Introducing a new magic method for the protocol helpfully
avoids any accidental opting into the protocol.

Provide specific type hinting support

There was some consideration to providing a generic typing.PathLike
class which would allow for e.g. typing.PathLike[str] to specify a type
hint for a path object which returned a string representation. While
potentially beneficial, the usefulness was deemed too small to bother
adding the type hint class.

This also removed any desire to have a class in the typing module which
represented the union of all acceptable path-representing types as that
can be represented with typing.Union[str, bytes, os.PathLike] easily
enough and the hope is users will slowly gravitate to path objects only.

Provide os.fspathb()

It was suggested that to mirror the structure of e.g.
os.getcwd()/os.getcwdb(), that os.fspath() only return str and that
another function named os.fspathb() be introduced that only returned
bytes. This was rejected as the purposes of the *b() functions are tied
to querying the file system where there is a need to get the raw bytes
back. As this PEP does not work directly with data on a file system (but
which may be), the view was taken this distinction is unnecessary. It's
also believed that the need for only bytes will not be common enough to
need to support in such a specific manner as os.fsencode() will provide
similar functionality.

Call __fspath__() off of the instance

An earlier draft of this PEP had os.fspath() calling path.__fspath__()
instead of type(path).__fspath__(path). The changed to be consistent
with how other magic methods in Python are resolved.

Acknowledgements

Thanks to everyone who participated in the various discussions related
to this PEP that spanned both python-ideas and python-dev. Special
thanks to Stephen Turnbull for direct feedback on early drafts of this
PEP. More special thanks to Koos Zevenhoven and Ethan Furman for not
only feedback on early drafts of this PEP but also helping to drive the
overall discussion on this topic across the two mailing lists.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] open() documentation for the C standard library
(http://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html)

[2] The pathlib module
(https://docs.python.org/3/library/pathlib.html#module-pathlib)

[3] The builtins.open() function
(https://docs.python.org/3/library/functions.html#open)

[4] The os.DirEntry class
(https://docs.python.org/3/library/os.html#os.DirEntry)

[5] The python-ideas mailing list archive
(https://mail.python.org/pipermail/python-ideas/)

[6] The python-dev mailing list archive
(https://mail.python.org/pipermail/python-dev/)

[7] The builtins.open() function
(https://docs.python.org/3/library/functions.html#open)

[8] The os.fsencode() function
(https://docs.python.org/3/library/os.html#os.fsencode)

[9] The os.fsdecode() function
(https://docs.python.org/3/library/os.html#os.fsdecode)

[10] The os.DirEntry class
(https://docs.python.org/3/library/os.html#os.DirEntry)

[11] The os.path module
(https://docs.python.org/3/library/os.path.html#module-os.path)


text:
PEP: 702 Title: Marking deprecations using the type system Author: Jelle
Zijlstra<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-702-marking-deprecations-using-the-type-system/23036
Status: Final Type: Standards Track Topic: Typing Created: 30-Dec-2022
Python-Version: 3.13 Post-History: 01-Jan-2023, 22-Jan-2023 Resolution:
07-Nov-2023

typing:deprecated and @warnings.deprecated<warnings.deprecated>

Abstract

This PEP adds an @warnings.deprecated() decorator that marks a class or
function as deprecated, enabling static checkers to warn when it is
used. By default, this decorator will also raise a runtime
DeprecationWarning.

Motivation

As software evolves, new functionality is added and old functionality
becomes obsolete. Library developers want to work towards removing
obsolete code while giving their users time to migrate to new APIs.
Python provides a mechanism for achieving these goals: the
DeprecationWarning warning class, which is used to show warnings when
deprecated functionality is used. This mechanism is widely used: as of
the writing of this PEP, the CPython main branch contains about 150
distinct code paths that raise !DeprecationWarning. Many third-party
libraries also use !DeprecationWarning to mark deprecations. In the top
5000 PyPI packages, there are:

-   1911 matches for the regex warnings\.warn.*\bDeprecationWarning\b,
    indicating use of !DeprecationWarning (not including cases where the
    warning is split over multiple lines);
-   1661 matches for the regex ^\s*@deprecated, indicating use of some
    sort of deprecation decorator.

However, the current mechanism is often insufficient to ensure that
users of deprecated functionality update their code in time. For
example, the removal of various long-deprecated unittest features had to
be reverted from Python 3.11 to give users more time to update their
code. Users may run their test suite with warnings disabled for
practical reasons, or deprecations may be triggered in code paths that
are not covered by tests.

Providing more ways for users to find out about deprecated functionality
can speed up the migration process. This PEP proposes to leverage static
type checkers to communicate deprecations to users. Such checkers have a
thorough semantic understanding of user code, enabling them to detect
and report deprecations that a single grep invocation could not find. In
addition, many type checkers integrate with IDEs, enabling users to see
deprecation warnings right in their editors.

Rationale

At first glance, deprecations may not seem like a topic that type
checkers should touch. After all, type checkers are concerned with
checking whether code will work as is, not with potential future
changes. However, the analysis that type checkers perform on code to
find type errors is very similar to the analysis that would be needed to
detect usage of many deprecations. Therefore, type checkers are well
placed to find and report deprecations.

Other languages already have similar functionality:

-   GCC supports a deprecated attribute on function declarations. This
    powers CPython's Py_DEPRECATED macro.
-   GraphQL supports marking fields as @deprecated.
-   Kotlin supports a Deprecated annotation.
-   Scala supports an @deprecated annotation.
-   Swift supports using the @available attribute to mark APIs as
    deprecated.
-   TypeScript uses the @deprecated JSDoc tag to issue a hint marking
    use of deprecated functionality.

Several users have requested support for such a feature:

-   typing-sig thread
-   Pyright feature request
-   mypy feature request

There are similar existing third-party tools:

-   Deprecated provides a decorator to mark classes, functions, or
    methods as deprecated. Access to decorated objects raises a runtime
    warning, but is not detected by type checkers.
-   flake8-deprecated is a linter plugin that warns about use of
    deprecated features. However, it is limited to a short, hard-coded
    list of deprecations.

Specification

A new decorator @deprecated() is added to the warnings module. This
decorator can be used on a class, function or method to mark it as
deprecated. This includes typing.TypedDict and typing.NamedTuple
definitions. With overloaded functions, the decorator may be applied to
individual overloads, indicating that the particular overload is
deprecated. The decorator may also be applied to the overload
implementation function, indicating that the entire function is
deprecated.

The decorator takes the following arguments:

-   A required positional-only argument representing the deprecation
    message.
-   Two keyword-only arguments, category and stacklevel, controlling
    runtime behavior (see under "Runtime behavior" below).

The positional-only argument is of type str and contains a message that
should be shown by the type checker when it encounters a usage of the
decorated object. Tools may clean up the deprecation message for
display, for example by using inspect.cleandoc or equivalent logic. The
message must be a string literal. The content of deprecation messages is
up to the user, but it may include the version in which the deprecated
object is to be removed, and information about suggested replacement
APIs.

Type checkers should produce a diagnostic whenever they encounter a
usage of an object marked as deprecated. For deprecated overloads, this
includes all calls that resolve to the deprecated overload. For
deprecated classes and functions, this includes:

-   References through module, class, or instance attributes
    (module.deprecated_object, module.SomeClass.deprecated_method,
    module.SomeClass().deprecated_method)
-   Any usage of deprecated objects in their defining module
    (x = deprecated_object() in module.py)
-   If import * is used, usage of deprecated objects from the module
    (from module import *; x = deprecated_object())
-   from imports (from module import deprecated_object)
-   Any syntax that indirectly triggers a call to the function. For
    example, if the __add__ method of a class C is deprecated, then the
    code C() + C() should trigger a diagnostic. Similarly, if the setter
    of a property is marked deprecated, attempts to set the property
    should trigger a diagnostic.

If a method is marked with the typing.override decorator from PEP 698
and the base class method it overrides is deprecated, the type checker
should produce a diagnostic.

There are additional scenarios where deprecations could come into play.
For example, an object may implement a typing.Protocol, but one of the
methods required for protocol compliance is deprecated. As scenarios
such as this one appear complex and relatively unlikely to come up in
practice, this PEP does not mandate that type checkers detect them.

Example

As an example, consider this library stub named library.pyi:

    from warnings import deprecated

    @deprecated("Use Spam instead")
    class Ham: ...

    @deprecated("It is pining for the fiords")
    def norwegian_blue(x: int) -> int: ...

    @overload
    @deprecated("Only str will be allowed")
    def foo(x: int) -> str: ...
    @overload
    def foo(x: str) -> str: ...

    class Spam:
        @deprecated("There is enough spam in the world")
        def __add__(self, other: object) -> object: ...

        @property
        @deprecated("All spam will be equally greasy")
        def greasy(self) -> float: ...

        @property
        def shape(self) -> str: ...
        @shape.setter
        @deprecated("Shapes are becoming immutable")
        def shape(self, value: str) -> None: ...

Here is how type checkers should handle usage of this library:

    from library import Ham  # error: Use of deprecated class Ham. Use Spam instead.

    import library

    library.norwegian_blue(1)  # error: Use of deprecated function norwegian_blue. It is pining for the fiords.
    map(library.norwegian_blue, [1, 2, 3])  # error: Use of deprecated function norwegian_blue. It is pining for the fiords.

    library.foo(1)  # error: Use of deprecated overload for foo. Only str will be allowed.
    library.foo("x")  # no error

    ham = Ham()  # no error (already reported above)

    spam = library.Spam()
    spam + 1  # error: Use of deprecated method Spam.__add__. There is enough spam in the world.
    spam.greasy  # error: Use of deprecated property Spam.greasy. All spam will be equally greasy.
    spam.shape  # no error
    spam.shape = "cube"  # error: Use of deprecated property setter Spam.shape. Shapes are becoming immutable.

The exact wording of the diagnostics is up to the type checker and is
not part of the specification.

Runtime behavior

In addition to the positional-only message argument, the @deprecated
decorator takes two keyword-only arguments:

-   category: A warning class. Defaults to DeprecationWarning. If this
    is set to None, no warning is issued at runtime and the decorator
    returns the original object, except for setting the __deprecated__
    attribute (see below).
-   stacklevel: The number of stack frames to skip when issuing the
    warning. Defaults to 1, indicating that the warning should be issued
    at the site where the deprecated object is called. Internally, the
    implementation will add the number of stack frames it uses in
    wrapper code.

If the decorated object is a class, the decorator wraps the __new__
method such that instantiating the class issues a warning. If the
decorated object is a callable, the decorator returns a new callable
that wraps the original callable but raises a warning when called.
Otherwise, the decorator raises a TypeError (unless category=None is
passed).

There are several scenarios where use of the decorated object cannot
issue a warning, including overloads, Protocol classes, and abstract
methods. Type checkers may show a warning if @deprecated is used without
category=None in these cases.

To accommodate runtime introspection, the decorator sets an attribute
__deprecated__ on the object it is passed, as well as on the wrapper
callables it generates for deprecated classes and functions. The value
of the attribute is the message passed to the decorator. Decorating
objects that do not allow setting this attribute is not supported.

If a Protocol with the @runtime_checkable decorator is marked as
deprecated, the __deprecated__ attribute should not be considered a
member of the protocol, so its presence should not affect isinstance
checks.

For compatibility with typing.get_overloads, the @deprecated decorator
should be placed after the @overload decorator.

Type checker behavior

This PEP does not specify exactly how type checkers should present
deprecation diagnostics to their users. However, some users (e.g.,
application developers targeting only a specific version of Python) may
not care about deprecations, while others (e.g., library developers who
want their library to remain compatible with future versions of Python)
would want to catch any use of deprecated functionality in their CI
pipeline. Therefore, it is recommended that type checkers provide
configuration options that cover both use cases. As with any other type
checker error, it is also possible to ignore deprecations using
# type: ignore comments.

Deprecation policy

We propose that CPython's deprecation policy (PEP 387) is updated to
require that new deprecations use the functionality in this PEP to alert
users about the deprecation, if possible. Concretely, this means that
new deprecations should be accompanied by a change to the typeshed repo
to add the @deprecated decorator in the appropriate place. This
requirement does not apply to deprecations that cannot be expressed
using this PEP's functionality.

Backwards compatibility

Creating a new decorator poses no backwards compatibility concerns. As
with all new typing functionality, the @deprecated decorator will be
added to the typing_extensions module, enabling its use in older
versions of Python.

How to teach this

For users who encounter deprecation warnings in their IDE or type
checker output, the messages they receive should be clear and
self-explanatory. Usage of the @deprecated decorator will be an advanced
feature mostly relevant to library authors. The decorator should be
mentioned in relevant documentation (e.g., PEP 387 and the
DeprecationWarning documentation) as an additional way to mark
deprecated functionality.

Reference implementation

A runtime implementation of the @deprecated decorator is available in
the typing-extensions library since version 4.5.0. The pyanalyze type
checker has prototype support for emitting deprecation errors, as does
Pyright.

Rejected ideas

Deprecation of modules and attributes

This PEP covers deprecations of classes, functions and overloads. This
allows type checkers to detect many but not all possible deprecations.
To evaluate whether additional functionality would be worthwhile, I
examined all current deprecations in the CPython standard library.

I found:

-   74 deprecations of functions, methods and classes (supported by this
    PEP)
-   28 deprecations of whole modules (largely due to PEP 594)
-   9 deprecations of function parameters (supported by this PEP through
    decorating overloads)
-   1 deprecation of a constant
-   38 deprecations that are not easily detectable in the type system
    (for example, for calling asyncio.get_event_loop without an active
    event loop)

Modules could be marked as deprecated by adding a __deprecated__
module-level constant. However, the need for this is limited, and it is
relatively easy to detect usage of deprecated modules simply by
grepping. Therefore, this PEP omits support for whole-module
deprecations. As a workaround, users could mark all module-level classes
and functions with @deprecated.

For deprecating module-level constants, object attributes, and function
parameters, a Deprecated[type, message] type modifier, similar to
Annotated could be added. However, this would create a new place in the
type system where strings are just strings, not forward references,
complicating the implementation of type checkers. In addition, my data
show that this feature is not commonly needed.

Features for deprecating more kinds of objects could be added in a
future PEP.

Placing the decorator in the typing module

An earlier version of this PEP proposed placing the @deprecated
decorator in the typing module. However, there was feedback that it
would be unexpected for a decorator in the typing module to have runtime
behavior. Therefore, the PEP now proposes adding the decorator the
warnings module instead.

Acknowledgments

A call with the typing-sig meetup group led to useful feedback on this
proposal.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.