text:
PEP: 3002 Title: Procedure for Backwards-Incompatible Changes Version:
$Revision$ Last-Modified: $Date$ Author: Steven Bethard
<EMAIL_ADDRESS>Status: Final Type: Process Content-Type:
text/x-rst Created: 27-Mar-2006 Post-History: 27-Mar-2006, 13-Apr-2006

Abstract

This PEP describes the procedure for changes to Python that are
backwards-incompatible between the Python 2.X series and Python 3000.
All such changes must be documented by an appropriate Python 3000 PEP
and must be accompanied by code that can identify when pieces of Python
2.X code may be problematic in Python 3000.

Rationale

Python 3000 will introduce a number of backwards-incompatible changes to
Python, mainly to streamline the language and to remove some previous
design mistakes. But Python 3000 is not intended to be a new and
completely different language from the Python 2.X series, and it is
expected that much of the Python user community will make the transition
to Python 3000 when it becomes available.

To encourage this transition, it is crucial to provide a clear and
complete guide on how to upgrade Python 2.X code to Python 3000 code.
Thus, for any backwards-incompatible change, two things are required:

-   An official Python Enhancement Proposal (PEP)
-   Code that can identify pieces of Python 2.X code that may be
    problematic in Python 3000

Python Enhancement Proposals

Every backwards-incompatible change must be accompanied by a PEP. This
PEP should follow the usual PEP guidelines and explain the purpose and
reasoning behind the backwards incompatible change. In addition to the
usual PEP sections, all PEPs proposing backwards-incompatible changes
must include an additional section: Compatibility Issues. This section
should describe what is backwards incompatible about the proposed change
to Python, and the major sorts of breakage to be expected.

While PEPs must still be evaluated on a case-by-case basis, a PEP may be
inappropriate for Python 3000 if its Compatibility Issues section
implies any of the following:

-   Most or all instances of a Python 2.X construct are incorrect in
    Python 3000, and most or all instances of the Python 3000 construct
    are incorrect in Python 2.X.

    So for example, changing the meaning of the for-loop else-clause
    from "executed when the loop was not broken out of" to "executed
    when the loop had zero iterations" would mean that all Python 2.X
    for-loop else-clauses would be broken, and there would be no way to
    use a for-loop else-clause in a Python-3000-appropriate manner. Thus
    a PEP for such an idea would likely be rejected.

-   Many instances of a Python 2.X construct are incorrect in Python
    3000 and the PEP fails to demonstrate real-world use-cases for the
    changes.

    Backwards incompatible changes are allowed in Python 3000, but not
    to excess. A PEP that proposes backwards-incompatible changes should
    provide good examples of code that visibly benefits from the
    changes.

PEP-writing is time-consuming, so when a number of
backwards-incompatible changes are closely related, they should be
proposed in the same PEP. Such PEPs will likely have longer
Compatibility Issues sections, however, since they must now describe the
sorts of breakage expected from all the proposed changes.

Identifying Problematic Code

In addition to the PEP requirement, backwards incompatible changes to
Python must also be accompanied by code to issue warnings for pieces of
Python 2.X code that will behave differently in Python 3000. Such
warnings will be enabled in Python 2.X using a new command-line switch:
-3. All backwards incompatible changes should be accompanied by a patch
for Python 2.X that, when -3 is specified, issues warnings for each
construct that is being changed.

For example, if dict.keys() returns an iterator in Python 3000, the
patch to the Python 2.X branch should do something like:

  If -3 was specified, change dict.keys() to return a subclass of list
  that issues warnings whenever you use any methods other than
  __iter__().

Such a patch would mean that warnings are only issued when features that
will not be present in Python 3000 are used, and almost all existing
code should continue to work. (Code that relies on dict.keys() always
returning a list and not a subclass should be pretty much non-existent.)

References

TBD

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 421 Title: Adding sys.implementation Version: $Revision$
Last-Modified: $Date$ Author: Eric Snow<EMAIL_ADDRESS>BDFL-Delegate: Barry Warsaw Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 26-Apr-2012 Python-Version: 3.3
Post-History: 26-Apr-2012 Resolution:
https://mail.python.org/pipermail/python-dev/2012-May/119683.html

Abstract

This PEP introduces a new attribute for the sys module:
sys.implementation. The attribute holds consolidated information about
the implementation of the running interpreter. Thus sys.implementation
is the source to which the standard library may look for
implementation-specific information.

The proposal in this PEP is in line with a broader emphasis on making
Python friendlier to alternate implementations. It describes the new
variable and the constraints on what that variable contains. The PEP
also explains some immediate use cases for sys.implementation.

Motivation

For a number of years now, the distinction between Python-the-language
and CPython (the reference implementation) has been growing. Most of
this change is due to the emergence of Jython, IronPython, and PyPy as
viable alternate implementations of Python.

Consider, however, the nearly two decades of CPython-centric Python
(i.e. most of its existence). That focus has understandably contributed
to quite a few CPython-specific artifacts both in the standard library
and exposed in the interpreter. Though the core developers have made an
effort in recent years to address this, quite a few of the artifacts
remain.

Part of the solution is presented in this PEP: a single namespace in
which to consolidate implementation specifics. This will help focus
efforts to differentiate the implementation specifics from the language.
Additionally, it will foster a multiple-implementation mindset.

Proposal

We will add a new attribute to the sys module, called
sys.implementation, as an object with attribute-access (as opposed to a
mapping). It will contain implementation-specific information.

The attributes of this object will remain fixed during interpreter
execution and through the course of an implementation version. This
ensures behaviors don't change between versions which depend on
attributes of sys.implementation.

The object has each of the attributes described in the Required
Attributes section below. Those attribute names will never start with an
underscore. The standard library and the language definition will rely
only on those required attributes.

This proposal takes a conservative approach in requiring only a small
number of attributes. As more become appropriate, they may be added with
discretion, as described in Adding New Required Attributes.

While this PEP places no other constraints on sys.implementation, it
also recommends that no one rely on capabilities outside those described
here. The only exception to that recommendation is for attributes
starting with an underscore. Implementers may use those as appropriate
to store per-implementation data.

Required Attributes

These are attributes in sys.implementation on which the standard library
and language definition will rely, meaning implementers must define
them:

name

    A lower-case identifier representing the implementation. Examples
    include 'pypy', 'jython', 'ironpython', and 'cpython'.

version

    The version of the implementation, as opposed to the version of the
    language it implements. This value conforms to the format described
    in Version Format.

hexversion

    The version of the implementation in the same hexadecimal format as
    sys.hexversion.

cache_tag

    A string used for the PEP 3147 cache tag. It would normally be a
    composite of the name and version (e.g. 'cpython-33' for CPython
    3.3). However, an implementation may explicitly use a different
    cache tag. If cache_tag is set to None, it indicates that module
    caching should be disabled.

Adding New Required Attributes

In time more required attributes will be added to sys.implementation.
However, each must have a meaningful use case across all Python
implementations in order to be considered. This is made most clear by a
use case in the standard library or language specification.

All proposals for new required attributes will go through the normal PEP
process. Such a PEP need not be long, just long enough. It will need to
sufficiently spell out the rationale for the new attribute, its use
cases, and the impact it will have on the various Python
implementations.

Version Format

A main point of sys.implementation is to contain information that will
be used internally in the standard library. In order to facilitate the
usefulness of the version attribute, its value should be in a consistent
format across implementations.

As such, the format of sys.implementation.version will follow that of
sys.version_info, which is effectively a named tuple. It is a familiar
format and generally consistent with normal version format conventions.

Rationale

The status quo for implementation-specific information gives us that
information in a more fragile, harder to maintain way. It is spread out
over different modules or inferred from other information, as we see
with platform.python_implementation().

This PEP is the main alternative to that approach. It consolidates the
implementation-specific information into a single namespace and makes
explicit that which was implicit.

Type Considerations

It's very easy to get bogged down in discussions about the type of
sys.implementation. However, its purpose is to support the standard
library and language definition. As such, there isn't much that really
matters regarding its type, as opposed to a feature that would be more
generally used. Thus characteristics like immutability and sequence-ness
have been disregarded.

The only real choice has been between an object with attribute access
and a mapping with item access. This PEP espouses dotted access to
reflect the relatively fixed nature of the namespace.

Non-Required Attributes

Earlier versions of this PEP included a required attribute called
metadata that held any non-required, per-implementation data [1].
However, this proved to be an unnecessary addition considering the
purpose of sys.implementation.

Ultimately, non-required attributes are virtually ignored in this PEP.
They have no impact other than that careless use may collide with future
required attributes. That, however, is but a marginal concern for
sys.implementation.

Why a Part of sys?

The sys module holds the new namespace because sys is the depot for
interpreter-centric variables and functions. Many
implementation-specific attributes are already found in sys.

Why Strict Constraints on Any of the Values?

As already noted in Version Format, values in sys.implementation are
intended for use by the standard library. Constraining those values,
essentially specifying an API for them, allows them to be used
consistently, regardless of how they are otherwise implemented. However,
care should be taken to not over-specify the constraints.

Discussion

The topic of sys.implementation came up on the python-ideas list in
2009, where the reception was broadly positive[2]. I revived the
discussion recently while working on a pure-python imp.get_tag()[3].
Discussion has been ongoing [4]. The messages in issue #14673 are also
relevant.

A good part of the recent discussion centered on the type to use for
sys.implementation.

Use-cases

platform.python_implementation()

"explicit is better than implicit"

The platform module determines the python implementation by looking for
clues in a couple different sys variables[5]. However, this approach is
fragile, requiring changes to the standard library each time an
implementation changes. Beyond that, support in platform is limited to
those implementations that core developers have blessed by
special-casing them in the platform module.

With sys.implementation the various implementations would explicitly set
the values in their own version of the sys module.

Another concern is that the platform module is part of the stdlib, which
ideally should minimize implementation details such as would be moved to
sys.implementation.

Any overlap between sys.implementation and the platform module would
simply defer to sys.implementation (with the same interface in platform
wrapping it).

Cache Tag Generation in Frozen Importlib

PEP 3147 defined the use of a module cache and cache tags for file
names. The importlib bootstrap code, frozen into the Python binary as of
3.3, uses the cache tags during the import process. Part of the project
to bootstrap importlib has been to clean code out of Python/import.c
that did not need to be there any longer.

The cache tag defined in Python/import.c was hard-coded <3147#proposal>
to "cpython" MAJOR MINOR. For importlib the options are either
hard-coding it in the same way, or guessing the implementation in the
same way as does platform.python_implementation().

As long as the hard-coded tag is limited to CPython-specific code, it is
livable. However, inasmuch as other Python implementations use the
importlib code to work with the module cache, a hard-coded tag would
become a problem.

Directly using the platform module in this case is a non-starter. Any
module used in the importlib bootstrap must be built-in or frozen,
neither of which apply to the platform module. This is the point that
led to the recent interest in sys.implementation.

Regardless of the outcome for the implementation name used, another
problem relates to the version used in the cache tag. That version is
likely to be the implementation version rather than the language
version. However, the implementation version is not readily identified
anywhere in the standard library.

Implementation-Specific Tests

Currently there are a number of implementation-specific tests in the
test suite under Lib/test. The test support module (Lib/test/support.py)
provides some functionality for dealing with these tests. However, like
the platform module, test.support must do some guessing that
sys.implementation would render unnecessary.

Jython's os.name Hack

In Jython, os.name is set to 'java' to accommodate special treatment of
the java environment in the standard library[6] [7]. Unfortunately it
masks the os name that would otherwise go there. sys.implementation
would help obviate the need for this special case. Currently Jython sets
os._name for the normal os.name value.

The Problem With sys.(version|version_info|hexversion)

Earlier versions of this PEP made the mistake of calling
sys.version_info (and friends) the version of the Python language, in
contrast to the implementation. However, this is not the case. Instead,
it is the version of the CPython implementation. Incidentally, the first
two components of sys.version_info (major and minor) also reflect the
version of the language definition.

As Barry Warsaw noted, the "semantics of sys.version_info have been
sufficiently squishy in the past"[8]. With sys.implementation we have
the opportunity to improve this situation by first establishing an
explicit location for the version of the implementation.

This PEP makes no other effort to directly clarify the semantics of
sys.version_info. Regardless, having an explicit version for the
implementation will definitely help to clarify the distinction from the
language version.

Feedback From Other Python Implementers

IronPython

Jeff Hardy responded to a request for feedback[9]. He said, "I'll
probably add it the day after it's approved" [10]. He also gave useful
feedback on both the type of sys.implementation and on the metadata
attribute (which has since been removed from the PEP).

Jython

In 2009 Frank Wierzbicki said this (relative to Jython implementing the
required attributes)[11]:

    Speaking for Jython, so far it looks like something we would adopt
    soonish after it was accepted (it looks pretty useful to me).

PyPy

Some of the PyPy developers have responded to a request for feedback
[12]. Armin Rigo said the following[13]:

    For myself, I can only say that it looks like a good idea, which we
    will happily adhere to when we migrate to Python 3.3.

He also expressed support for keeping the required list small. Both
Armin and Laura Creighton indicated that an effort to better catalog
Python's implementation would be welcome. Such an effort, for which this
PEP is a small start, will be considered separately.

Past Efforts

PEP 3139

PEP 3139, from 2008, recommended a clean-up of the sys module in part by
extracting implementation-specific variables and functions into a
separate module. PEP 421 is less ambitious version of that idea. While
PEP 3139 was rejected, its goals are reflected in PEP 421 to a large
extent, though with a much lighter approach.

PEP 399

PEP 399 dictates policy regarding the standard library, helping to make
it friendlier to alternate implementations. PEP 421 is proposed in that
same spirit.

The Bigger Picture

It's worth noting again that this PEP is a small part of a larger
ongoing effort to identify the implementation-specific parts of Python
and mitigate their impact on alternate implementations.

sys.implementation is a focal point for implementation-specific data,
acting as a nexus for cooperation between the language, the standard
library, and the different implementations. As time goes by it is
feasible that sys.implementation will assume current attributes of sys
and other builtin/stdlib modules, where appropriate. In this way, it is
a PEP 3137-lite, but starting as small as possible.

However, as already noted, many other efforts predate
sys.implementation. Neither is it necessarily a major part of the
effort. Rather, consider it as part of the infrastructure of the effort
to make Python friendlier to alternate implementations.

Alternatives

Since the single-namespace-under-sys approach is relatively
straightforward, no alternatives have been considered for this PEP.

Examples of Other Attributes

These are examples only and not part of the proposal. Most of them were
suggested during previous discussions, but did not fit into the goals of
this PEP. (See Adding New Required Attributes if they get you excited.)

common_name

    The case-sensitive name by which the implementation is known.

vcs_url

    A URL for the main VCS repository for the implementation project.

vcs_revision_id

    A value that identifies the VCS revision of the implementation.

build_toolchain

    The tools used to build the interpreter.

build_date

    The timestamp of when the interpreter was built.

homepage

    The URL of the implementation's website.

site_prefix

    The preferred site prefix for the implementation.

runtime

    The run-time environment in which the interpreter is running, as in
    "Common Language Runtime" (.NET CLR) or "Java Runtime Executable".

gc_type

    The type of garbage collection used, like "reference counting" or
    "mark and sweep".

Open Issues

Currently none.

Implementation

The implementation of this PEP is covered in issue #14673.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Alyssa (Nick) Coghlan's proposal for sys.implementation.metadata:
https://mail.python.org/pipermail/python-ideas/2012-May/014984.html

[2] The 2009 sys.implementation discussion:
https://mail.python.org/pipermail/python-dev/2009-October/092893.html

[3] The initial 2012 discussion:
https://mail.python.org/pipermail/python-ideas/2012-March/014555.html
(and
https://mail.python.org/pipermail/python-ideas/2012-April/014878.html)

[4] Feedback on the PEP:
https://mail.python.org/pipermail/python-ideas/2012-April/014954.html

[5] The platform code which divines the implementation name:
http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247

[6] The standard library entry for os.name:
http://docs.python.org/3.3/library/os.html#os.name

[7] The use of os.name as 'java' in the stdlib test suite.
http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l512

[8] Feedback from Barry Warsaw:
https://mail.python.org/pipermail/python-dev/2012-May/119374.html

[9] Feedback from the IronPython developers:
https://mail.python.org/pipermail/ironpython-users/2012-May/015980.html

[10] (2012) Jeff Hardy offers his opinion:
https://mail.python.org/pipermail/ironpython-users/2012-May/015981.html

[11] (2009) Frank Wierzbicki offers his opinion:
https://mail.python.org/pipermail/python-dev/2009-October/092974.html

[12] Feedback from the PyPy developers:
https://mail.python.org/pipermail/pypy-dev/2012-May/009883.html

[13] (2012) Armin Rigo offers his opinion:
https://mail.python.org/pipermail/pypy-dev/2012-May/009884.html

license: Public Domain


text:
PEP: 277 Title: Unicode file name support for Windows NT Author: Neil
Hodgson<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 11-Jan-2002 Python-Version: 2.3
Post-History:

Abstract

This PEP discusses supporting access to all files possible on Windows NT
by passing Unicode file names directly to the system's wide-character
functions.

Rationale

Python 2.2 on Win32 platforms converts Unicode file names passed to open
and to functions in the os module into the 'mbcs' encoding before
passing the result to the operating system. This is often successful in
the common case where the script is operating with the locale set to the
same value as when the file was created. Most machines are set up as one
locale and rarely if ever changed from this locale. For some users,
locale is changed more often and on servers there are often files saved
by users using different locales.

On Windows NT and descendent operating systems, including Windows 2000
and Windows XP, wide-character APIs are available that provide direct
access to all file names, including those that are not representable
using the current locale. The purpose of this proposal is to provide
access to these wide-character APIs through the standard Python file
object and posix module and so provide access to all files on Windows
NT.

Specification

On Windows platforms which provide wide-character file APIs, when
Unicode arguments are provided to file APIs, wide-character calls are
made instead of the standard C library and posix calls.

The Python file object is extended to use a Unicode file name argument
directly rather than converting it. This affects the file object
constructor file(filename[, mode[, bufsize]]) and also the open function
which is an alias of this constructor. When a Unicode filename argument
is used here then the name attribute of the file object will be Unicode.
The representation of a file object, repr(f) will display Unicode file
names as an escaped string in a similar manner to the representation of
Unicode strings.

The posix module contains functions that take file or directory names:
chdir, listdir, mkdir, open, remove, rename, rmdir, stat, and
_getfullpathname. These will use Unicode arguments directly rather than
converting them. For the rename function, this behaviour is triggered
when either of the arguments is Unicode and the other argument converted
to Unicode using the default encoding.

The listdir function currently returns a list of strings. Under this
proposal, it will return a list of Unicode strings when its path
argument is Unicode.

Restrictions

On the consumer Windows operating systems, Windows 95, Windows 98, and
Windows ME, there are no wide-character file APIs so behaviour is
unchanged under this proposal. It may be possible in the future to
extend this proposal to cover these operating systems as the VFAT-32
file system used by them does support Unicode file names but access is
difficult and so implementing this would require much work. The
"Microsoft Layer for Unicode" could be a starting point for implementing
this.

Python can be compiled with the size of Unicode characters set to 4
bytes rather than 2 by defining PY_UNICODE_TYPE to be a 4 byte type and
Py_UNICODE_SIZE to be 4. As the Windows API does not accept 4 byte
characters, the features described in this proposal will not work in
this mode so the implementation falls back to the current 'mbcs'
encoding technique. This restriction could be lifted in the future by
performing extra conversions using PyUnicode_AsWideChar but for now that
would add too much complexity for a very rarely used feature.

Reference Implementation

The implementation is available at[1].

References

[1] Microsoft Windows APIs https://msdn.microsoft.com/

Copyright

This document has been placed in the public domain.

[1] https://github.com/python/cpython/issues/37017


text:
PEP: 588 Title: GitHub Issues Migration Plan Author: Mariatta
<EMAIL_ADDRESS>BDFL-Delegate: Barry Warsaw<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/13791 Status: Draft Type:
Informational Content-Type: text/x-rst Created: 27-Mar-2019

Abstract

This PEP describes the detailed plan for migrating from Python's issue
tracker on Roundup to GitHub issues. See PEP 581 for rationale and
background. PEP 588 also describes the detailed timeline for the
migration.

Migration Plan

Here we outline the tasks, steps, and core decisions we need to make in
order to migrate bug tracking to GitHub, with the least impact on
CPython developer productivity.

Hire a professional project manager

Having a professional project manager to handle the migration, similar
to how the Warehouse project was managed, would help ensure the success
of this project.

Create a playground CPython issue tracker on GitHub

We should create a playground issue tracker on GitHub where we can
experiment and test out the new workflow.

Backup of GitHub data

This effort has been started and is being tracked as an issue in
core-workflow[1]. We're using GitHub's Migrations API[2] to download
GitHub data for CPython on a daily basis. The archives will be dropped
in a S3 bucket.

Thanks to Ee Durbin for working on this.

Update the CLA host

At the moment, the CLA is hosted within bpo. It needs to be updated such
that signing the CLA does not require a bpo account, and it should be
hosted outside of the bpo.

The current CLA process itself is not ideal. Currently, contributors to
devguide, peps, and core-workflow need to sign a CLA, and it requires a
bpo account. A bpo account should not be required for those projects.

There is an ongoing effort to start using our own instance of CLA
assistant instead of the current CLA process in place. Discussion about
this has been started in core-workflow mailing list as well as on
Discourse.

This effort is currently stalled because cla-assistant does not yet
support CLA signed on behalf of organization.

Create "Python Triage" team on GitHub

The bug triagers on bpo are valuable to the core Python workflow, and we
definitely would need even more help with triaging issues on GitHub.

It has been proposed on Discourse for us to create a "bug triage" team
on GitHub to help with closing issues, notifying the appropriate
parties, as well as applying labels to issues and pull requests.

The new Triage role on GitHub is currently in beta, and the Python
organization has been granted access to this role, and we can begin
taking advantage of it.

The "Python Triage" team has been created. A description and
expectations of the triage role have been added to Devguide.

Progress of this project can be tracked in "Adding Triagers" project
board.

Create labels for issue triage

In bpo, we currently have the following fields for each issue:

Types: behavior, crash, compile error, resource usage, security,
performance, enhancement.

Components: 2to3, Argument Clinic, asyncio, Build, Cross-build, ctypes,
...

Priority: release blocker, deferred blocker, critical, high, normal, low

We will create the corresponding labels:

    type-behavior, type-crash, type-compile error, type-resource usage, ...

    components-2to3, components-argument clinic, components-asyncio, ...

    priority-release blocker, priority-deferred blocker, priority-critical, ...

In addition, we'll create a needs triage label.

The final "labels" to be created can be decided at a later time when it
is time to start switching to GitHub issues.

A test repository containing all possible labels and color schema has
been created by Carol Willing and can be reviewed at
https://github.com/willingc/test-581/labels.

Create issue templates

We will create an issue template and add the following headers:

    ---
    Type: behavior | crash | compile error | resource usage (choose one)
    Components: 2to3 | Argument Clinic | asyncio | Build | ... (can select more than one)
    Priority: release blocker | deferred blocker | critical | ...
    Needs backport to: 2.7 | 3.6 | 3.7
    ---

The idea is to allow the issue creator to help us triage the issue. The
above values are pre-filled in the template. The issue creator will
remove texts that do not apply to their issue.

Based on the above headers, bedevere-bot can apply the necessary labels
to the issue. If the issue creator did not supply the above headers, the
bot will apply the needs triage label. At that point, it will require a
core developer to properly label the issue.

We can also take advantage of GitHub's multiple issue template feature,
and the ability to automatically set issue assignee and labels by using
issue templates.

Updates to bedevere

Bedevere-bot will need to be updated to recognize the issue headers
described above and apply the proper labels.

Bedevere-bot can also copy over the labels to pull requests that
correspond to the issue.

Update the devguide

Devguide should be updated with information about the new workflow of
using GitHub issues. It can be done as a separate branch, and it should
be done ahead of the migration, not after.

Add a button in bpo to migrate the issue to GitHub

This will require the bpo to be updated. But I believe the effort needed
for this is much less than a complete overhaul.

We will create a button in bpo, that will copy over the issue
description and associated comments into a GitHub issue.

We need to add a new status: "moved" with the url of the GitHub issue.

We should not be moving all open issues to GitHub. Only when someone is
interested in continuing work or discussion about the issue, that the
issue should be "moved" to GitHub.

Migrated issues

When an issue is marked as "moved", this issue should be in read-only
mode. bpo should forbid the edition of the issue.

Make bpo read-only

This should be the final step. Once we start using GitHub issues, make
bpo read-only, instead of shutting it down. Do not accept new
registrations. Do not allow comments or issues to be created.

Mapping between issues from bpo and GitHub

Usually when we reference an issue from bpo, we use bpo-XYZ but with
Github, we will have a new reference with this format
https://github.com/python/cpython/issue/XYZ.

Because we will migrate the issues from bpo to GitHub, we need to have a
new field on bpo for the reference to the issues on GitHub, and the same
thing on Github for the 'eventual' reference from bpo.

For GitHub, we need to add origin: https://bugs.python.org/issueXYZ. For
bpo, add a new field
moved to: https://github.com/python/cpython/issue/XYZ.

Nosy-ing the expert

A current functionality in bpo is to automatically nosy people who are
listed as an expert of certain area. Several Python core developers have
expressed that they prefer not having to subscribe to everything on
GitHub, but only getting notified for issues related to their area of
interest and expertise.

To help with this situation, we can develop a bot that can notify people
whenever an issue has been categorized using labels. For example, when
an issue was labeled with area-windows, the windows experts can be
notified. The notification can be in the form of email notification, or
@-mention on GitHub.

Open issues

A GitHub account should not be a requirement

Back when moving the CPython codebase from Mercurial to GitHub was being
discussed[3][4], it was brought up that we still needed to allow
uploading of patches on bpo, and that a GitHub account should not be a
requirement in order to contribute to Python.

If bpo is made read-only, we'll need to come up with a different
solution to allow people to contribute when they don't have a GitHub
account.

One solution is to create a new "python-issues" mailing list, similar to
the<EMAIL_ADDRESS>mailing list, to allow people to submit their
issues there.

Related to this, since the migration to GitHub in 2017, I recall one
case[6] where there was a contributor, who submitted a patch to
Mercurial and refused to create a GitHub account. Because of this, our
bot was unable to detect whether they had signed the CLA. Another person
had volunteered to upload their patch to GitHub. But it was still
required that both people sign the CLA.

That particular situation was complicated. It took up five core
developers' time to investigate and manually check the CLA, causing
confusion.

Trim off the "Components" list

Is the current "components" list still making sense and relevant? Can
the list be shortened?

Priority list

Is the current "priority" list useful? Alyssa Coghlan noted that perhaps
only release blocker and deferred blocker are useful.

Further questions and discussions

You can post questions on Discourse under the Core-Workflow category.

Acknowledgements

Thanks to Guido van Rossum, Brett Cannon, and Alyssa Coghlan, who were
consulted in the early stage and research of this PEP. Their feedback,
concerns, input, and ideas have been valuable.

References

Copyright

This document has been placed in the public domain.

[1] Backup GitHub information
(https://github.com/python/core-workflow/issues/20)

[2] GitHub's Migrations API
(https://developer.github.com/v3/migrations/orgs/)

[3] Python-committers email
(https://mail.python.org/pipermail/python-committers/2015-December/003642.html)

[4] Python-committers email
(https://mail.python.org/pipermail/python-committers/2015-December/003645.html)

[5] docs mailing list (https://mail.python.org/mailman/listinfo/docs)

[6] CPython GitHub pull request 1505
(https://github.com/python/cpython/pull/1505)


text:
PEP: 506 Title: Adding A Secrets Module To The Standard Library Version:
$Revision$ Last-Modified: $Date$ Author: Steven D'Aprano
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 19-Sep-2015 Python-Version: 3.6 Post-History:

Abstract

This PEP proposes the addition of a module for common security-related
functions such as generating tokens to the Python standard library.

Definitions

Some common abbreviations used in this proposal:

-   PRNG:

    Pseudo Random Number Generator. A deterministic algorithm used to
    produce random-looking numbers with certain desirable statistical
    properties.

-   CSPRNG:

    Cryptographically Strong Pseudo Random Number Generator. An
    algorithm used to produce random-looking numbers which are resistant
    to prediction.

-   MT:

    Mersenne Twister. An extensively studied PRNG which is currently
    used by the random module as the default.

Rationale

This proposal is motivated by concerns that Python's standard library
makes it too easy for developers to inadvertently make serious security
errors. Theo de Raadt, the founder of OpenBSD, contacted Guido van
Rossum and expressed some concern[1] about the use of MT for generating
sensitive information such as passwords, secure tokens, session keys and
similar.

Although the documentation for the random module explicitly states that
the default is not suitable for security purposes[2], it is strongly
believed that this warning may be missed, ignored or misunderstood by
many Python developers. In particular:

-   developers may not have read the documentation and consequently not
    seen the warning;
-   they may not realise that their specific use of the module has
    security implications; or
-   not realising that there could be a problem, they have copied code
    (or learned techniques) from websites which don't offer best
    practises.

The first[3] hit when searching for "python how to generate passwords"
on Google is a tutorial that uses the default functions from the random
module[4]. Although it is not intended for use in web applications, it
is likely that similar techniques find themselves used in that
situation. The second hit is to a StackOverflow question about
generating passwords[5]. Most of the answers given, including the
accepted one, use the default functions. When one user warned that the
default could be easily compromised, they were told "I think you worry
too much."[6]

This strongly suggests that the existing random module is an attractive
nuisance when it comes to generating (for example) passwords or secure
tokens.

Additional motivation (of a more philosophical bent) can be found in the
post which first proposed this idea[7].

Proposal

Alternative proposals have focused on the default PRNG in the random
module, with the aim of providing "secure by default" cryptographically
strong primitives that developers can build upon without thinking about
security. (See Alternatives below.) This proposes a different approach:

-   The standard library already provides cryptographically strong
    primitives, but many users don't know they exist or when to use
    them.
-   Instead of requiring crypto-naive users to write secure code, the
    standard library should include a set of ready-to-use "batteries"
    for the most common needs, such as generating secure tokens. This
    code will both directly satisfy a need ("How do I generate a
    password reset token?"), and act as an example of acceptable
    practises which developers can learn from[8].

To do this, this PEP proposes that we add a new module to the standard
library, with the suggested name secrets. This module will contain a set
of ready-to-use functions for common activities with security
implications, together with some lower-level primitives.

The suggestion is that secrets becomes the go-to module for dealing with
anything which should remain secret (passwords, tokens, etc.) while the
random module remains backward-compatible.

API and Implementation

This PEP proposes the following functions for the secrets module:

-   Functions for generating tokens suitable for use in (e.g.) password
    recovery, as session keys, etc., in the following formats:
    -   as bytes, secrets.token_bytes;
    -   as text, using hexadecimal digits, secrets.token_hex;
    -   as text, using URL-safe base-64 encoding, secrets.token_urlsafe.
-   A limited interface to the system CSPRNG, using either os.urandom
    directly or random.SystemRandom. Unlike the random module, this does
    not need to provide methods for seeding, getting or setting the
    state, or any non-uniform distributions. It should provide the
    following:
    -   A function for choosing items from a sequence, secrets.choice.
    -   A function for generating a given number of random bits and/or
        bytes as an integer, secrets.randbits.
    -   A function for returning a random integer in the half-open range
        0 to the given upper limit, secrets.randbelow[9].
-   A function for comparing text or bytes digests for equality while
    being resistant to timing attacks, secrets.compare_digest.

The consensus appears to be that there is no need to add a new CSPRNG to
the random module to support these uses, SystemRandom will be
sufficient.

Some illustrative implementations have been given by Alyssa (Nick)
Coghlan[10] and a minimalist API by Tim Peters[11]. This idea has also
been discussed on the issue tracker for the "cryptography" module[12].
The following pseudo-code should be taken as the starting point for the
real implementation:

    from random import SystemRandom
    from hmac import compare_digest

    _sysrand = SystemRandom()

    randbits = _sysrand.getrandbits
    choice = _sysrand.choice

    def randbelow(exclusive_upper_bound):
        return _sysrand._randbelow(exclusive_upper_bound)

    DEFAULT_ENTROPY = 32  # bytes

    def token_bytes(nbytes=None):
        if nbytes is None:
            nbytes = DEFAULT_ENTROPY
        return os.urandom(nbytes)

    def token_hex(nbytes=None):
        return binascii.hexlify(token_bytes(nbytes)).decode('ascii')

    def token_urlsafe(nbytes=None):
        tok = token_bytes(nbytes)
        return base64.urlsafe_b64encode(tok).rstrip(b'=').decode('ascii')

The secrets module itself will be pure Python, and other Python
implementations can easily make use of it unchanged, or adapt it as
necessary. An implementation can be found on BitBucket[13].

Default arguments

One difficult question is "How many bytes should my token be?". We can
help with this question by providing a default amount of entropy for the
"token*" functions. If the nbytes argument is None or not given, the
default entropy will be used. This default value should be large enough
to be expected to be secure for medium-security uses, but is expected to
change in the future, possibly even in a maintenance release[14].

Naming conventions

One question is the naming conventions used in the module[15], whether
to use C-like naming conventions such as "randrange" or more Pythonic
names such as "random_range".

Functions which are simply bound methods of the private SystemRandom
instance (e.g. randrange), or a thin wrapper around such, should keep
the familiar names. Those which are something new (such as the various
token_* functions) will use more Pythonic names.

Alternatives

One alternative is to change the default PRNG provided by the random
module[16]. This received considerable scepticism and outright
opposition:

-   There is fear that a CSPRNG may be slower than the current PRNG
    (which in the case of MT is already quite slow).
-   Some applications (such as scientific simulations, and replaying
    gameplay) require the ability to seed the PRNG into a known state,
    which a CSPRNG lacks by design.
-   Another major use of the random module is for simple "guess a
    number" games written by beginners, and many people are loath to
    make any change to the random module which may make that harder.
-   Although there is no proposal to remove MT from the random module,
    there was considerable hostility to the idea of having to opt-in to
    a non-CSPRNG or any backwards-incompatible changes.
-   Demonstrated attacks against MT are typically against PHP
    applications. It is believed that PHP's version of MT is a
    significantly softer target than Python's version, due to a poor
    seeding technique[17]. Consequently, without a proven attack against
    Python applications, many people object to a backwards-incompatible
    change.

Alyssa Coghlan made an earlier suggestion <504> for a globally
configurable PRNG which uses the system CSPRNG by default, but has since
withdrawn it in favour of this proposal.

Comparison To Other Languages

-   PHP

    PHP includes a function uniqid[18] which by default returns a
    thirteen character string based on the current time in microseconds.
    Translated into Python syntax, it has the following signature:

        def uniqid(prefix='', more_entropy=False)->str

    The PHP documentation warns that this function is not suitable for
    security purposes. Nevertheless, various mature, well-known PHP
    applications use it for that purpose (citation needed).

    PHP 5.3 and better also includes a function
    openssl_random_pseudo_bytes [19]. Translated into Python syntax, it
    has roughly the following signature:

        def openssl_random_pseudo_bytes(length:int)->Tuple[str, bool]

    This function returns a pseudo-random string of bytes of the given
    length, and a boolean flag giving whether the string is considered
    cryptographically strong. The PHP manual suggests that returning
    anything but True should be rare except for old or broken platforms.

-   JavaScript

    Based on a rather cursory search[20], there do not appear to be any
    well-known standard functions for producing strong random values in
    JavaScript. Math.random is often used, despite serious weaknesses
    making it unsuitable for cryptographic purposes[21]. In recent years
    the majority of browsers have gained support for
    window.crypto.getRandomValues[22].

    Node.js offers a rich cryptographic module, crypto[23], most of
    which is beyond the scope of this PEP. It does include a single
    function for generating random bytes, crypto.randomBytes.

-   Ruby

    The Ruby standard library includes a module SecureRandom[24] which
    includes the following methods:

    -   base64 - returns a Base64 encoded random string.
    -   hex - returns a random hexadecimal string.
    -   random_bytes - returns a random byte string.
    -   random_number - depending on the argument, returns either a
        random integer in the range(0, n), or a random float between 0.0
        and 1.0.
    -   urlsafe_base64 - returns a random URL-safe Base64 encoded
        string.
    -   uuid - return a version 4 random Universally Unique IDentifier.

What Should Be The Name Of The Module?

There was a proposal to add a "random.safe" submodule, quoting the Zen
of Python "Namespaces are one honking great idea" koan. However, the
author of the Zen, Tim Peters, has come out against this idea[25], and
recommends a top-level module.

In discussion on the python-ideas mailing list so far, the name
"secrets" has received some approval, and no strong opposition.

There is already an existing third-party module with the same name[26],
but it appears to be unused and abandoned.

Frequently Asked Questions

-   Q: Is this a real problem? Surely MT is random enough that nobody
    can predict its output.

    A: The consensus among security professionals is that MT is not safe
    in security contexts. It is not difficult to reconstruct the
    internal state of MT[27][28] and so predict all past and future
    values. There are a number of known, practical attacks on systems
    using MT for randomness[29].

-   Q: Attacks on PHP are one thing, but are there any known attacks on
    Python software?

    A: Yes. There have been vulnerabilities in Zope and Plone at the
    very least. Hanno Schlichting commented[30]:

        "In the context of Plone and Zope a practical attack was
        demonstrated, but I can't find any good non-broken links about
        this anymore.  IIRC Plone generated a random number and exposed
        this on each error page along the lines of 'Sorry, you encountered
        an error, your problem has been filed as <random number>, please
        include this when you contact us'.  This allowed anyone to do large
        numbers of requests to this page and get enough random values to
        reconstruct the MT state.  A couple of security related modules used
        random instead of system random (cookie session ids, password reset
        links, auth token), so the attacker could break all of those."

    Christian Heimes reported this issue to the Zope security team in
    2012[31], there are at least two related CVE vulnerabilities[32],
    and at least one work-around for this issue in Django[33].

-   Q: Is this an alternative to specialist cryptographic software such
    as SSL?

    A: No. This is a "batteries included" solution, not a full-featured
    "nuclear reactor". It is intended to mitigate against some basic
    security errors, not be a solution to all security-related issues.
    To quote Alyssa Coghlan referring to her earlier proposal[34]:

        "...folks really are better off learning to use things like
        cryptography.io for security sensitive software, so this change
        is just about harm mitigation given that it's inevitable that a
        non-trivial proportion of the millions of current and future
        Python developers won't do that."

-   Q: What about a password generator?

    A: The consensus is that the requirements for password generators
    are too variable for it to be a good match for the standard
    library[35]. No password generator will be included in the initial
    release of the module, instead it will be given in the documentation
    as a recipe (à la the recipes in the itertools module)[36].

-   Q: Will secrets use /dev/random (which blocks) or /dev/urandom
    (which doesn't block) on Linux? What about other platforms?

    A: secrets will be based on os.urandom and random.SystemRandom,
    which are interfaces to your operating system's best source of
    cryptographic randomness. On Linux, that may be /dev/urandom[37], on
    Windows it may be CryptGenRandom(), but see the documentation and/or
    source code for the detailed implementation details.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] https://mail.python.org/pipermail/python-ideas/2015-September/035820.html

[2] https://docs.python.org/3/library/random.html

[3] As of the date of writing. Also, as Google search terms may be
automatically customised for the user without their knowledge, some
readers may see different results.

[4] http://interactivepython.org/runestone/static/everyday/2013/01/3_password.html

[5] http://stackoverflow.com/questions/3854692/generate-password-in-python

[6] http://stackoverflow.com/questions/3854692/generate-password-in-python/3854766#3854766

[7] https://mail.python.org/pipermail/python-ideas/2015-September/036238.html

[8] At least those who are motivated to read the source code and
documentation.

[9] After considerable discussion, Guido ruled that the module need only
provide randbelow, and not similar functions randrange or randint.
http://code.activestate.com/lists/python-dev/138375/

[10] https://mail.python.org/pipermail/python-ideas/2015-September/036271.html

[11] https://mail.python.org/pipermail/python-ideas/2015-September/036350.html

[12] https://github.com/pyca/cryptography/issues/2347

[13] https://bitbucket.org/sdaprano/secrets

[14] https://mail.python.org/pipermail/python-ideas/2015-September/036517.html
https://mail.python.org/pipermail/python-ideas/2015-September/036515.html

[15] https://mail.python.org/pipermail/python-ideas/2015-September/036474.html

[16] Link needed.

[17] By default PHP seeds the MT PRNG with the time (citation needed),
which is exploitable by attackers, while Python seeds the PRNG with
output from the system CSPRNG, which is believed to be much harder to
exploit.

[18] http://php.net/manual/en/function.uniqid.php

[19] http://php.net/manual/en/function.openssl-random-pseudo-bytes.php

[20] Volunteers and patches are welcome.

[21] http://ifsec.blogspot.fr/2012/05/cross-domain-mathrandom-prediction.html

[22] https://developer.mozilla.org/en-US/docs/Web/API/RandomSource/getRandomValues

[23] https://nodejs.org/api/crypto.html

[24] http://ruby-doc.org/stdlib-2.1.2/libdoc/securerandom/rdoc/SecureRandom.html

[25] https://mail.python.org/pipermail/python-ideas/2015-September/036254.html

[26] https://pypi.python.org/pypi/secrets

[27] https://jazzy.id.au/2010/09/22/cracking_random_number_generators_part_3.html

[28] https://mail.python.org/pipermail/python-ideas/2015-September/036077.html

[29] https://media.blackhat.com/bh-us-12/Briefings/Argyros/BH_US_12_Argyros_PRNG_WP.pdf

[30] Personal communication, 2016-08-24.

[31] https://bugs.launchpad.net/zope2/+bug/1071067

[32] http://www.cvedetails.com/cve/CVE-2012-5508/
http://www.cvedetails.com/cve/CVE-2012-6661/

[33] https://github.com/django/django/commit/1525874238fd705ec17a066291935a9316bd3044

[34] https://mail.python.org/pipermail/python-ideas/2015-September/036157.html

[35] https://mail.python.org/pipermail/python-ideas/2015-September/036476.html
https://mail.python.org/pipermail/python-ideas/2015-September/036478.html

[36] https://mail.python.org/pipermail/python-ideas/2015-September/036488.html

[37] http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/
http://www.2uo.de/myths-about-urandom/


text:
PEP: 3110 Title: Catching Exceptions in Python 3000 Version: $Revision$
Last-Modified: $Date$ Author: Collin Winter<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
16-Jan-2006 Python-Version: 3.0 Post-History:

Abstract

This PEP introduces changes intended to help eliminate ambiguities in
Python's grammar, simplify exception classes, simplify garbage
collection for exceptions and reduce the size of the language in Python
3.0.

Rationale

1.  except clauses in Python 2.x present a syntactic ambiguity where the
    parser cannot differentiate whether :

        except <expression>, <expression>:

    should be interpreted as :

        except <type>, <type>:

    or :

        except <type>, <name>:

    Python 2 opts for the latter semantic, at the cost of requiring the
    former to be parenthesized, like so :

        except (<type>, <type>):

2.  As specified in PEP 352, the ability to treat exceptions as tuples
    will be removed, meaning this code will no longer work :

        except os.error, (errno, errstr):

    Because the automatic unpacking will no longer be possible, it is
    desirable to remove the ability to use tuples as except targets.

3.  As specified in PEP 344, exception instances in Python 3 will
    possess a __traceback__ attribute. The Open Issues section of that
    PEP includes a paragraph on garbage collection difficulties caused
    by this attribute, namely a "exception -> traceback -> stack frame
    -> exception" reference cycle, whereby all locals are kept in scope
    until the next GC run. This PEP intends to resolve this issue by
    adding a cleanup semantic to except clauses in Python 3 whereby the
    target name is deleted at the end of the except suite.

4.  In the spirit of "there should be one -- and preferably only one
    -- obvious way to do it" <20>, it is desirable to consolidate
    duplicate functionality. To this end, the exc_value, exc_type and
    exc_traceback attributes of the sys module[1] will be removed in
    favor of sys.exc_info(), which provides the same information. These
    attributes are already listed in PEP 3100 as targeted for removal.

Grammar Changes

In Python 3, the grammar for except statements will change from[2] :

    except_clause: 'except' [test [',' test]]

to :

    except_clause: 'except' [test ['as' NAME]]

The use of as in place of the comma token means that :

    except (AttributeError, os.error):

can be clearly understood as a tuple of exception classes. This new
syntax was first proposed by Greg Ewing[3] and endorsed ([4],[5]) by the
BDFL.

Further, the restriction of the token following as from test to NAME
means that only valid identifiers can be used as except targets.

Note that the grammar above always requires parenthesized tuples as
exception classes. That way, the ambiguous :

    except A, B:

which would mean different things in Python 2.x and 3.x -- leading to
hard-to-catch bugs -- cannot legally occur in 3.x code.

Semantic Changes

In order to resolve the garbage collection issue related to PEP 344,
except statements in Python 3 will generate additional bytecode to
delete the target, thus eliminating the reference cycle. The
source-to-source translation, as suggested by Phillip J. Eby [6], is :

    try:
        try_body
    except E as N:
        except_body
    ...

gets translated to (in Python 2.5 terms) :

    try:
        try_body
    except E, N:
        try:
            except_body
        finally:
            N = None
            del N
    ...

An implementation has already been checked into the py3k (formerly
"p3yk") branch[7].

Compatibility Issues

Nearly all except clauses will need to be changed. except clauses with
identifier targets will be converted from :

    except E, N:

to :

    except E as N:

except clauses with non-tuple, non-identifier targets (e.g., a.b.c[d])
will need to be converted from :

    except E, T:

to :

    except E as t:
        T = t

Both of these cases can be handled by Guido van Rossum's 2to3 utility[8]
using the except fixer[9].

except clauses with tuple targets will need to be converted manually, on
a case-by-case basis. These changes will usually need to be accompanied
by changes to the exception classes themselves. While these changes
generally cannot be automated, the 2to3 utility is able to point out
cases where the target of an except clause is a tuple, simplifying
conversion.

Situations where it is necessary to keep an exception instance around
past the end of the except suite can be easily translated like so :

    try:
        ...
    except E as N:
        ...
    ...

becomes :

    try:
        ...
    except E as N:
        n = N
        ...
    ...

This way, when N is deleted at the end of the block, n will persist and
can be used as normal.

Lastly, all uses of the sys module's exc_type, exc_value and
exc_traceback attributes will need to be removed. They can be replaced
with sys.exc_info()[0], sys.exc_info()[1] and sys.exc_info()[2]
respectively, a transformation that can be performed by 2to3's
sysexcattrs fixer.

2.6 - 3.0 Compatibility

In order to facilitate forwards compatibility between Python 2.6 and
3.0, the except ... as ...: syntax will be backported to the 2.x series.
The grammar will thus change from:

    except_clause: 'except' [test [',' test]]

to:

    except_clause: 'except' [test [('as' | ',') test]]

The end-of-suite cleanup semantic for except statements will not be
included in the 2.x series of releases.

Open Issues

Replacing or Dropping "sys.exc_info()"

The idea of dropping sys.exc_info() or replacing it with a sys.exception
attribute or a sys.get_exception() function has been raised several
times on python-3000 ([10], [11]) and mentioned in PEP 344's "Open
Issues" section.

While a 2to3 fixer to replace calls to sys.exc_info() and some attribute
accesses would be trivial, it would be far more difficult for static
analysis to find and fix functions that expect the values from
sys.exc_info() as arguments. Similarly, this does not address the need
to rewrite the documentation for all APIs that are defined in terms of
sys.exc_info().

Implementation

This PEP was implemented in revisions 53342[12] and 53349 [13]. Support
for the new except syntax in 2.6 was implemented in revision 55446[14].

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] http://docs.python.org/library/sys.html

[2] http://docs.python.org/reference/compound_stmts.html#try

[3] https://mail.python.org/pipermail/python-dev/2006-March/062449.html

[4] https://mail.python.org/pipermail/python-dev/2006-March/062449.html

[5] https://mail.python.org/pipermail/python-dev/2006-March/062640.html

[6] https://mail.python.org/pipermail/python-3000/2007-January/005395.html

[7] http://svn.python.org/view?rev=53342&view=rev

[8] https://hg.python.org/sandbox/guido/file/2.7/Lib/lib2to3/

[9] https://hg.python.org/sandbox/guido/file/2.7/Lib/lib2to3/fixes/fix_except.py

[10] https://mail.python.org/pipermail/python-3000/2007-January/005385.html

[11] https://mail.python.org/pipermail/python-3000/2007-January/005604.html

[12] http://svn.python.org/view?view=revision&revision=53342

[13] http://svn.python.org/view?view=revision&revision=53349

[14] http://svn.python.org/view/python/trunk/?view=rev&rev=55446


text:
PEP: 565 Title: Show DeprecationWarning in __main__ Author: Alyssa
Coghlan<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 12-Nov-2017 Python-Version: 3.7
Post-History: 12-Nov-2017, 25-Nov-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-December/151224.html

Abstract

In Python 2.7 and Python 3.2, the default warning filters were updated
to hide DeprecationWarning by default, such that deprecation warnings in
development tools that were themselves written in Python (e.g. linters,
static analysers, test runners, code generators), as well as any other
applications that merely happened to be written in Python, wouldn't be
visible to their users unless those users explicitly opted in to seeing
them.

However, this change has had the unfortunate side effect of making
DeprecationWarning markedly less effective at its primary intended
purpose: providing advance notice of breaking changes in APIs (whether
in CPython, the standard library, or in third party libraries) to users
of those APIs.

To improve this situation, this PEP proposes a single adjustment to the
default warnings filter: displaying deprecation warnings attributed to
the main module by default.

This change will mean that code entered at the interactive prompt and
code in single file scripts will revert to reporting these warnings by
default, while they will continue to be silenced by default for packaged
code distributed as part of an importable module.

The PEP also proposes a number of small adjustments to the reference
interpreter and standard library documentation to help make the warnings
subsystem more approachable for new Python developers.

As part of the documentation updates, it will be made clearer that the
unittest test runner displays all warnings by default when executing
test cases, and that other test runners are advised to follow that
example.

Specification

New default warnings filter entry

The current set of default warnings filters consists of:

    ignore::DeprecationWarning
    ignore::PendingDeprecationWarning
    ignore::ImportWarning
    ignore::BytesWarning
    ignore::ResourceWarning

The default unittest test runner then uses warnings.catch_warnings()
warnings.simplefilter('default') to override the default filters while
running test cases.

The change proposed in this PEP is to update the default warning filter
list to be:

    default::DeprecationWarning:__main__
    ignore::DeprecationWarning
    ignore::PendingDeprecationWarning
    ignore::ImportWarning
    ignore::BytesWarning
    ignore::ResourceWarning

This means that in cases where the nominal location of the warning (as
determined by the stacklevel parameter to warnings.warn) is in the
__main__ module, the first occurrence of each DeprecationWarning will
once again be reported.

This change will lead to DeprecationWarning being displayed by default
for:

-   code executed directly at the interactive prompt
-   code executed directly as part of a single-file script

While continuing to be hidden by default for:

-   code imported from another module in a zipapp archive's __main__.py
    file
-   code imported from another module in an executable package's
    __main__ submodule
-   code imported from an executable script wrapper generated at
    installation time based on a console_scripts or gui_scripts entry
    point definition

This means that tool developers that create an installable or executable
artifact (such as a zipapp archive) for distribution to their users
shouldn't see any change from the status quo, while users of more ad hoc
personal or locally distributed scripts are likely to start seeing
relevant deprecation warnings again (as they did in Python 2.6 and
earlier).

Additional use case for FutureWarning

The standard library documentation will be updated to explicitly
recommend the use of FutureWarning (rather than DeprecationWarning) for
backwards compatibility warnings that are intended to be seen by users
of an application. (This will be in addition to the existing use of
FutureWarning to warn about constructs that will remain valid code in
the future, but will have different semantics).

This will give the following three distinct categories of backwards
compatibility warning, with three different intended audiences:

-   PendingDeprecationWarning: hidden by default for all code. The
    intended audience is Python developers that take an active interest
    in ensuring the future compatibility of their software (e.g.
    professional Python application developers with specific support
    obligations).
-   DeprecationWarning: reported by default for code that runs directly
    in the __main__ module (as such code is considered relatively
    unlikely to have a dedicated test suite), but hidden by default for
    code in other modules. The intended audience is Python developers
    that are at risk of upgrades to their dependencies (including
    upgrades to Python itself) breaking their software (e.g. developers
    using Python to script environments where someone else is in control
    of the timing of dependency upgrades).
-   FutureWarning: reported by default for all code. The intended
    audience is users of applications written in Python, rather than
    other Python developers (e.g. warning about use of a deprecated
    setting in a configuration file format).

For library and framework authors that want to ensure their API
compatibility warnings are more reliably seen by their users, the
recommendation is to use a custom warning class that derives from
DeprecationWarning in Python 3.7+, and from FutureWarning in earlier
versions.

Recommended filter settings for test runners

Developers of test runners are advised to implement logic equivalent to
the following when determining their default warnings filters:

    if not sys.warnoptions:
        warnings.simplefilter("default")

This effectively enables all warnings by default, as if the -Wd command
line option had been passed.

Note that actually enabling BytesWarning in a test suite still requires
passing the -b option to the interpreter at the command line. For
implicit bytes conversion and bytes comparison warnings, the warnings
filter machinery is only used to determine whether they should be
printed as warnings or raised as exceptions - when the command line flag
isn't set, the interpreter doesn't even emit the warning in the first
place.

Recommended filter settings for interactive shells

Developers of interactive shells are advised to add a filter that
enables DeprecationWarning in the namespace where user code is entered
and executed.

If that namespace is __main__ (as it is for the default CPython REPL),
then no changes are needed beyond those in this PEP.

Interactive shell implementations which use a namespace other than
__main__ will need to add their own filter. For example, IPython uses
the following command ([1]) to set up a suitable filter:

    warnings.filterwarnings("default", category=DeprecationWarning,
                                       module=self.user_ns.get("__name__"))

Other documentation updates

The current reference documentation for the warnings system is
relatively short on specific examples of possible settings for the -W
command line option or the PYTHONWARNINGS environment variably that
achieve particular end results.

The following improvements are proposed as part of the implementation of
this PEP:

-   Explicitly list the following entries under the description of the
    PYTHONWARNINGS environment variable:

        PYTHONWARNINGS=error # Convert to exceptions
        PYTHONWARNINGS=always # Warn every time
        PYTHONWARNINGS=default # Warn once per call location
        PYTHONWARNINGS=module # Warn once per calling module
        PYTHONWARNINGS=once # Warn once per Python process
        PYTHONWARNINGS=ignore # Never warn

-   Explicitly list the corresponding short options (-We, -Wa, -Wd, -Wm,
    -Wo, -Wi) for each of the warning actions listed under the -W
    command line switch documentation

-   Explicitly list the default filter set in the warnings module
    documentation, using the action::category and
    action::category:module notation

-   Explicitly list the following snippet in the warnings.simplefilter
    documentation as a recommended approach to turning off all warnings
    by default in a Python application while still allowing them to be
    turned back on via PYTHONWARNINGS or the -W command line switch:

        if not sys.warnoptions:
            warnings.simplefilter("ignore")

None of these are new (they already work in all still supported Python
versions), but they're not especially obvious given the current
structure of the related documentation.

Reference Implementation

A reference implementation is available in the PR[2] linked from the
related tracker issue for this PEP[3].

As a side-effect of implementing this PEP, the internal warnings filter
list will start allowing the use of plain strings as part of filter
definitions (in addition to the existing use of compiled regular
expressions). When present, the plain strings will be compared for exact
matches only. This approach allows the new default filter to be added
during interpreter startup without requiring early access to the re
module.

Motivation

As discussed in[4] and mentioned in[5], Python 2.7 and Python 3.2
changed the default handling of DeprecationWarning such that:

-   the warning was hidden by default during normal code execution
-   the unittest test runner was updated to re-enable it when running
    tests

The intent was to avoid cases of tooling output like the following:

    $ devtool mycode/
    /usr/lib/python3.6/site-packages/devtool/cli.py:1: DeprecationWarning: 'async' and 'await' will become reserved keywords in Python 3.7
      async = True
    ... actual tool output ...

Even when devtool is a tool specifically for Python programmers, this is
not a particularly useful warning, as it will be shown on every
invocation, even though the main helpful step an end user can take is to
report a bug to the developers of devtool.

The warning is even less helpful for general purpose developer tools
that are used across more languages than just Python, and almost
entirely *un*helpful for applications that simply happen to be written
in Python, and aren't necessarily intended for a developer audience at
all.

However, this change proved to have unintended consequences for the
following audiences:

-   anyone using a test runner other than the default one built into
    unittest (the request for third party test runners to change their
    default warnings filters was never made explicitly, so many of them
    still rely on the interpreter defaults that are designed to suit
    deployed applications)
-   anyone using the default unittest test runner to test their Python
    code in a subprocess (since even unittest only adjusts the warnings
    settings in the current process)
-   anyone writing Python code at the interactive prompt or as part of a
    directly executed script that didn't have a Python level test suite
    at all

In these cases, DeprecationWarning ended up become almost entirely
equivalent to PendingDeprecationWarning: it was simply never seen at
all.

Limitations on PEP Scope

This PEP exists specifically to explain both the proposed addition to
the default warnings filter for 3.7, and to more clearly articulate the
rationale for the original change to the handling of DeprecationWarning
back in Python 2.7 and 3.2.

This PEP does not solve all known problems with the current approach to
handling deprecation warnings. Most notably:

-   The default unittest test runner does not currently report
    deprecation warnings emitted at module import time, as the warnings
    filter override is only put in place during test execution, not
    during test discovery and loading.
-   The default unittest test runner does not currently report
    deprecation warnings in subprocesses, as the warnings filter
    override is applied directly to the loaded warnings module, not to
    the PYTHONWARNINGS environment variable.
-   The standard library doesn't provide a straightforward way to opt-in
    to seeing all warnings emitted by a particular dependency prior to
    upgrading it (the third-party warn module[6] does provide this, but
    enabling it involves monkeypatching the standard library's warnings
    module).
-   When software has been factored out into support modules, but those
    modules have little or no automated test coverage, re-enabling
    deprecation warnings by default in __main__ isn't likely to help
    find API compatibility problems. Near term, the best currently
    available answer is to run affected applications with
    PYTHONWARNINGS=default::DeprecationWarning or
    python -W default::DeprecationWarning and pay attention to their
    stderr output. Longer term, this is really a question for
    researchers working on static analysis of Python code: how to
    reliably find usage of deprecated APIs, and how to infer that an API
    or parameter is deprecated based on warnings.warn calls, without
    actually running either the code providing the API or the code
    accessing it.

While these are real problems with the status quo, they're excluded from
consideration in this PEP because they're going to require more complex
solutions than a single additional entry in the default warnings filter,
and resolving them at least potentially won't require going through the
PEP process.

For anyone interested in pursuing them further, the first two would be
unittest module enhancement requests, the third would be a warnings
module enhancement request, while the last would only require a PEP if
inferring API deprecations from their contents was deemed to be an
intractable code analysis problem, and an explicit function and
parameter marker syntax in annotations was proposed instead.

The CPython reference implementation will also include the following
related changes in 3.7:

-   a new -X dev command line option that combines several developer
    centric settings (including -Wd) into one command line flag:
    https://github.com/python/cpython/issues/76224
-   changing the behaviour in debug builds to show more of the warnings
    that are off by default in regular interpreter builds:
    https://github.com/python/cpython/issues/76269

Independently of the proposed changes to the default filters in this
PEP, issue 32229[7] is a proposal to add a warnings.hide_warnings API to
make it simpler for application developers to hide warnings during
normal operation, while easily making them visible when testing.

References

-   First python-dev discussion thread
-   Second python-dev discussion thread

Copyright

This document has been placed in the public domain.

[1] IPython's DeprecationWarning auto-configuration
(https://github.com/ipython/ipython/blob/6.2.x/IPython/core/interactiveshell.py#L619)

[2] GitHub PR for PEP 565 implementation
(https://github.com/python/cpython/pull/4458)

[3] Tracker issue for PEP 565 implementation
(https://github.com/python/cpython/issues/76156)

[4] stdlib-sig thread proposing the original default filter change
(https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html)

[5] Python 2.7 notification of the default warnings filter change
(https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-deprecation-warnings)

[6] Emitting warnings based on the location of the warning itself
(https://pypi.org/project/warn/)

[7] warnings.hide_warnings API proposal
(https://github.com/python/cpython/issues/76410)


text:
PEP: 372 Title: Adding an ordered dictionary to collections Author:
Armin Ronacher<EMAIL_ADDRESS>Raymond Hettinger
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 15-Jun-2008 Python-Version: 2.7, 3.1 Post-History:

Abstract

This PEP proposes an ordered dictionary as a new data structure for the
collections module, called "OrderedDict" in this PEP. The proposed API
incorporates the experiences gained from working with similar
implementations that exist in various real-world applications and other
programming languages.

Patch

A working Py3.1 patch including tests and documentation is at:

  OrderedDict patch

The check-in was in revisions: 70101 and 70102

Rationale

In current Python versions, the widely used built-in dict type does not
specify an order for the key/value pairs stored. This makes it hard to
use dictionaries as data storage for some specific use cases.

Some dynamic programming languages like PHP and Ruby 1.9 guarantee a
certain order on iteration. In those languages, and existing Python
ordered-dict implementations, the ordering of items is defined by the
time of insertion of the key. New keys are appended at the end, but keys
that are overwritten are not moved to the end.

The following example shows the behavior for simple assignments:

  >>> d = OrderedDict() >>> d['parrot'] = 'dead' >>> d['penguin'] =
  'exploded' >>> d.items() [('parrot', 'dead'), ('penguin', 'exploded')]

That the ordering is preserved makes an OrderedDict useful for a couple
of situations:

-   XML/HTML processing libraries currently drop the ordering of
    attributes, use a list instead of a dict which makes filtering
    cumbersome, or implement their own ordered dictionary. This affects
    ElementTree, html5lib, Genshi and many more libraries.

-   There are many ordered dict implementations in various libraries and
    applications, most of them subtly incompatible with each other.
    Furthermore, subclassing dict is a non-trivial task and many
    implementations don't override all the methods properly which can
    lead to unexpected results.

    Additionally, many ordered dicts are implemented in an inefficient
    way, making many operations more complex then they have to be.

-   PEP 3115 allows metaclasses to change the mapping object used for
    the class body. An ordered dict could be used to create ordered
    member declarations similar to C structs. This could be useful, for
    example, for future ctypes releases as well as ORMs that define
    database tables as classes, like the one the Django framework ships.
    Django currently uses an ugly hack to restore the ordering of
    members in database models.

-   The RawConfigParser class accepts a dict_type argument that allows
    an application to set the type of dictionary used internally. The
    motivation for this addition was expressly to allow users to provide
    an ordered dictionary.[1]

-   Code ported from other programming languages such as PHP often
    depends on an ordered dict. Having an implementation of an
    ordering-preserving dictionary in the standard library could ease
    the transition and improve the compatibility of different libraries.

Ordered Dict API

The ordered dict API would be mostly compatible with dict and existing
ordered dicts. Note: this PEP refers to the 2.7 and 3.0 dictionary API
as described in collections.Mapping abstract base class.

The constructor and update() both accept iterables of tuples as well as
mappings like a dict does. Unlike a regular dictionary, the insertion
order is preserved.

  >>> d = OrderedDict([('a', 'b'), ('c', 'd')]) >>> d.update({'foo':
  'bar'}) >>> d collections.OrderedDict([('a', 'b'), ('c', 'd'), ('foo',
  'bar')])

If ordered dicts are updated from regular dicts, the ordering of new
keys is of course undefined.

All iteration methods as well as keys(), values() and items() return the
values ordered by the time the key was first inserted:

  >>> d['spam'] = 'eggs' >>> d.keys() ['a', 'c', 'foo', 'spam'] >>>
  d.values() ['b', 'd', 'bar', 'eggs'] >>> d.items() [('a', 'b'), ('c',
  'd'), ('foo', 'bar'), ('spam', 'eggs')]

New methods not available on dict:

OrderedDict.__reversed__()

    Supports reverse iteration by key.

Questions and Answers

What happens if an existing key is reassigned?

  The key is not moved but assigned a new value in place. This is
  consistent with existing implementations.

What happens if keys appear multiple times in the list passed to the
constructor?

  The same as for regular dicts -- the latter item overrides the former.
  This has the side-effect that the position of the first key is used
  because only the value is actually overwritten:

      >>> OrderedDict([('a', 1), ('b', 2), ('a', 3)])
      collections.OrderedDict([('a', 3), ('b', 2)])

  This behavior is consistent with existing implementations in Python,
  the PHP array and the hashmap in Ruby 1.9.

Is the ordered dict a dict subclass? Why?

  Yes. Like defaultdict, an ordered dictionary subclasses dict. Being a
  dict subclass make some of the methods faster (like __getitem__ and
  __len__). More importantly, being a dict subclass lets ordered
  dictionaries be usable with tools like json that insist on having dict
  inputs by testing isinstance(d, dict).

Do any limitations arise from subclassing dict?

  Yes. Since the API for dicts is different in Py2.x and Py3.x, the
  OrderedDict API must also be different. So, the Py2.7 version will
  need to override iterkeys, itervalues, and iteritems.

Does OrderedDict.popitem() return a particular key/value pair?

  Yes. It pops-off the most recently inserted new key and its
  corresponding value. This corresponds to the usual LIFO behavior
  exhibited by traditional push/pop pairs. It is semantically equivalent
  to k=list(od)[-1]; v=od[k]; del od[k]; return (k,v). The actual
  implementation is more efficient and pops directly from a sorted list
  of keys.

Does OrderedDict support indexing, slicing, and whatnot?

  As a matter of fact, OrderedDict does not implement the Sequence
  interface. Rather, it is a MutableMapping that remembers the order of
  key insertion. The only sequence-like addition is support for
  reversed.

  A further advantage of not allowing indexing is that it leaves open
  the possibility of a fast C implementation using linked lists.

Does OrderedDict support alternate sort orders such as alphabetical?

  No. Those wanting different sort orders really need to be using
  another technique. The OrderedDict is all about recording insertion
  order. If any other order is of interest, then another structure (like
  an in-memory dbm) is likely a better fit.

How well does OrderedDict work with the json module, PyYAML, and
ConfigParser?

  For json, the good news is that json's encoder respects OrderedDict's
  iteration order:

      >>> items = [('one', 1), ('two', 2), ('three',3), ('four',4), ('five',5)]
      >>> json.dumps(OrderedDict(items))
      '{"one": 1, "two": 2, "three": 3, "four": 4, "five": 5}'

  In Py2.6, the object_hook for json decoders passes-in an already built
  dictionary so order is lost before the object hook sees it. This
  problem is being fixed for Python 2.7/3.1 by adding a new hook that
  preserves order (see https://github.com/python/cpython/issues/49631 ).
  With the new hook, order can be preserved:

      >>> jtext = '{"one": 1, "two": 2, "three": 3, "four": 4, "five": 5}'
      >>> json.loads(jtext, object_pairs_hook=OrderedDict)
      OrderedDict({'one': 1, 'two': 2, 'three': 3, 'four': 4, 'five': 5})

  For PyYAML, a full round-trip is problem free:

      >>> ytext = yaml.dump(OrderedDict(items))
      >>> print ytext
      !!python/object/apply:collections.OrderedDict
      - - [one, 1]
        - [two, 2]
        - [three, 3]
        - [four, 4]
        - [five, 5]

      >>> yaml.load(ytext)
      OrderedDict({'one': 1, 'two': 2, 'three': 3, 'four': 4, 'five': 5})

  For the ConfigParser module, round-tripping is also problem free.
  Custom dicts were added in Py2.6 specifically to support ordered
  dictionaries:

      >>> config = ConfigParser(dict_type=OrderedDict)
      >>> config.read('myconfig.ini')
      >>> config.remove_option('Log', 'error')
      >>> config.write(open('myconfig.ini', 'w'))

How does OrderedDict handle equality testing?

  Comparing two ordered dictionaries implies that the test will be
  order-sensitive so that list (od1.items())==list(od2.items()).

  When ordered dicts are compared with other Mappings, their order
  insensitive comparison is used. This allows ordered dictionaries to be
  substituted anywhere regular dictionaries are used.

How __repr__ format will maintain order during a repr/eval round-trip?

  OrderedDict([('a', 1), ('b', 2)])

What are the trade-offs of the possible underlying data structures?

  -   Keeping a sorted list of keys is fast for all operations except
      __delitem__() which becomes an O(n) exercise. This data structure
      leads to very simple code and little wasted space.
  -   Keeping a separate dictionary to record insertion sequence numbers
      makes the code a little bit more complex. All of the basic
      operations are O(1) but the constant factor is increased for
      __setitem__() and __delitem__() meaning that every use case will
      have to pay for this speedup (since all buildup go through
      __setitem__). Also, the first traversal incurs a one-time
      O(n log n) sorting cost. The storage costs are double that for the
      sorted-list-of-keys approach.
  -   A version written in C could use a linked list. The code would be
      more complex than the other two approaches but it would conserve
      space and would keep the same big-oh performance as regular
      dictionaries. It is the fastest and most space efficient.

Reference Implementation

An implementation with tests and documentation is at:

  OrderedDict patch

The proposed version has several merits:

-   Strict compliance with the MutableMapping API and no new methods so
    that the learning curve is near zero. It is simply a dictionary that
    remembers insertion order.
-   Generally good performance. The big-oh times are the same as regular
    dictionaries except that key deletion is O(n).

Other implementations of ordered dicts in various Python projects or
standalone libraries, that inspired the API proposed here, are:

-   odict in Python
-   odict in Babel
-   OrderedDict in Django
-   The odict module
-   ordereddict (a C implementation of the odict module)
-   StableDict
-   Armin Rigo's OrderedDict

Future Directions

With the availability of an ordered dict in the standard library, other
libraries may take advantage of that. For example, ElementTree could
return odicts in the future that retain the attribute ordering of the
source file.

References

Copyright

This document has been placed in the public domain.

[1] https://github.com/python/cpython/issues/42649