text:
PEP: 3131 Title: Supporting Non-ASCII Identifiers Version: $Revision$
Last-Modified: $Date$ Author: Martin von Löwis<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
01-May-2007 Python-Version: 3.0 Post-History:

Abstract

This PEP suggests to support non-ASCII letters (such as accented
characters, Cyrillic, Greek, Kanji, etc.) in Python identifiers.

Rationale

Python code is written by many people in the world who are not familiar
with the English language, or even well-acquainted with the Latin
writing system. Such developers often desire to define classes and
functions with names in their native languages, rather than having to
come up with an (often incorrect) English translation of the concept
they want to name. By using identifiers in their native language, code
clarity and maintainability of the code among speakers of that language
improves.

For some languages, common transliteration systems exist (in particular,
for the Latin-based writing systems). For other languages, users have
larger difficulties to use Latin to write their native words.

Common Objections

Some objections are often raised against proposals similar to this one.

People claim that they will not be able to use a library if to do so
they have to use characters they cannot type on their keyboards.
However, it is the choice of the designer of the library to decide on
various constraints for using the library: people may not be able to use
the library because they cannot get physical access to the source code
(because it is not published), or because licensing prohibits usage, or
because the documentation is in a language they cannot understand. A
developer wishing to make a library widely available needs to make a
number of explicit choices (such as publication, licensing, language of
documentation, and language of identifiers). It should always be the
choice of the author to make these decisions - not the choice of the
language designers.

In particular, projects wishing to have wide usage probably might want
to establish a policy that all identifiers, comments, and documentation
is written in English (see the GNU coding style guide for an example of
such a policy). Restricting the language to ASCII-only identifiers does
not enforce comments and documentation to be English, or the identifiers
actually to be English words, so an additional policy is necessary,
anyway.

Specification of Language Changes

The syntax of identifiers in Python will be based on the Unicode
standard annex UAX-31[1], with elaboration and changes as defined below.

Within the ASCII range (U+0001..U+007F), the valid characters for
identifiers are the same as in Python 2.5. This specification only
introduces additional characters from outside the ASCII range. For other
characters, the classification uses the version of the Unicode Character
Database as included in the unicodedata module.

The identifier syntax is <XID_Start> <XID_Continue>*.

The exact specification of what characters have the XID_Start or
XID_Continue properties can be found in the DerivedCoreProperties file
of the Unicode data in use by Python (4.1 at the time this PEP was
written), see[2]. For reference, the construction rules for these sets
are given below. The XID* properties are derived from
ID_Start/ID_Continue, which are derived themselves.

ID_Start is defined as all characters having one of the general
categories uppercase letters (Lu), lowercase letters (Ll), titlecase
letters (Lt), modifier letters (Lm), other letters (Lo), letter numbers
(Nl), the underscore, and characters carrying the Other_ID_Start
property. XID_Start then closes this set under normalization, by
removing all characters whose NFKC normalization is not of the form
ID_Start ID_Continue* anymore.

ID_Continue is defined as all characters in ID_Start, plus nonspacing
marks (Mn), spacing combining marks (Mc), decimal number (Nd), connector
punctuations (Pc), and characters carrying the Other_ID_Continue
property. Again, XID_Continue closes this set under NFKC-normalization;
it also adds U+00B7 to support Catalan.

All identifiers are converted into the normal form NFKC while parsing;
comparison of identifiers is based on NFKC.

A non-normative HTML file listing all valid identifier characters for
Unicode 4.1 can be found at
http://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html.

Policy Specification

As an addition to the Python Coding style, the following policy is
prescribed: All identifiers in the Python standard library MUST use
ASCII-only identifiers, and SHOULD use English words wherever feasible
(in many cases, abbreviations and technical terms are used which aren't
English). In addition, string literals and comments must also be in
ASCII. The only exceptions are (a) test cases testing the non-ASCII
features, and (b) names of authors. Authors whose names are not based on
the Latin alphabet MUST provide a Latin transliteration of their names.

As an option, this specification can be applied to Python 2.x. In that
case, ASCII-only identifiers would continue to be represented as byte
string objects in namespace dictionaries; identifiers with non-ASCII
characters would be represented as Unicode strings.

Implementation

The following changes will need to be made to the parser:

1.  If a non-ASCII character is found in the UTF-8 representation of the
    source code, a forward scan is made to find the first ASCII
    non-identifier character (e.g. a space or punctuation character)
2.  The entire UTF-8 string is passed to a function to normalize the
    string to NFKC, and then verify that it follows the identifier
    syntax. No such callout is made for pure-ASCII identifiers, which
    continue to be parsed the way they are today. The Unicode database
    must start including the Other_ID{Start|Continue} property.
3.  If this specification is implemented for 2.x, reflective libraries
    (such as pydoc) must be verified to continue to work when Unicode
    strings appear in __dict__ slots as keys.

Open Issues

John Nagle suggested consideration of Unicode Technical Standard #39,
[3], which discusses security mechanisms for Unicode identifiers. It's
not clear how that can precisely apply to this PEP; possible
consequences are

-   warn about characters listed as "restricted" in xidmodifications.txt
-   warn about identifiers using mixed scripts
-   somehow perform Confusable Detection

In the latter two approaches, it's not clear how precisely the algorithm
should work. For mixed scripts, certain kinds of mixing should probably
allowed - are these the "Common" and "Inherited" scripts mentioned in
section 5? For Confusable Detection, it seems one needs two identifiers
to compare them for confusion - is it possible to somehow apply it to a
single identifier only, and warn?

In follow-up discussion, it turns out that John Nagle actually meant to
suggest UTR#36, level "Highly Restrictive",[4].

Several people suggested to allow and ignore formatting control
characters (general category Cf), as is done in Java, JavaScript, and
C#. It's not clear whether this would improve things (it might for RTL
languages); if there is a need, these can be added later.

Some people would like to see an option on selecting support for this
PEP at run-time; opinions vary on what precisely that option should be,
and what precisely its default value should be. Guido van Rossum
commented in[5] that a global flag passed to the interpreter is not
acceptable, as it would apply to all modules.

Discussion

Ka-Ping Yee summarizes discussion and further objection in[6] as such:

A.  Should identifiers be allowed to contain any Unicode letter?

    Drawbacks of allowing non-ASCII identifiers wholesale:

    1.  Python will lose the ability to make a reliable round trip to a
        human-readable display on screen or on paper.
    2.  Python will become vulnerable to a new class of security
        exploits; code and submitted patches will be much harder to
        inspect.
    3.  Humans will no longer be able to validate Python syntax.
    4.  Unicode is young; its problems are not yet well understood and
        solved; tool support is weak.
    5.  Languages with non-ASCII identifiers use different character
        sets and normalization schemes; PEP 3131's choices are
        non-obvious.
    6.  The Unicode bidi algorithm yields an extremely confusing display
        order for RTL text when digits or operators are nearby.

B.  Should the default behaviour accept only ASCII identifiers, or
    should it accept identifiers containing non-ASCII characters?

    Arguments for ASCII only by default:

    1.  Non-ASCII identifiers by default makes common
        practice/assumptions subtly/unknowingly wrong; rarely wrong is
        worse than obviously wrong.
    2.  Better to raise a warning than to fail silently when
        encountering a probably unexpected situation.
    3.  All of current usage is ASCII-only; the vast majority of future
        usage will be ASCII-only.
    4.  It is the pockets of Unicode adoption that are parochial, not
        the ASCII advocates.
    5.  Python should audit for ASCII-only identifiers for the same
        reasons that it audits for tab-space consistency
    6.  Incremental change is safer.
    7.  An ASCII-only default favors open-source development and sharing
        of source code.
    8.  Existing projects won't have to waste any brainpower worrying
        about the implications of Unicode identifiers.

C.  Should non-ASCII identifiers be optional?

    Various voices in support of a flag (although there's been debate
    over which should be the default, no one seems to be saying that
    there shouldn't be an off switch)

D.  Should the identifier character set be configurable?

    Various voices proposing and supporting a selectable character set,
    so that users can get all the benefits of using their own language
    without the drawbacks of confusable/unfamiliar characters

E.  Which identifier characters should be allowed?

    1.  What to do about bidi format control characters?
    2.  What about other ID_Continue characters? What about characters
        that look like punctuation? What about other recommendations in
        UTS #39? What about mixed-script identifiers?

F.  Which normalization form should be used, NFC or NFKC?

G.  Should source code be required to be in normalized form?

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] http://www.unicode.org/reports/tr31/

[2] http://www.unicode.org/Public/4.1.0/ucd/DerivedCoreProperties.txt

[3] http://www.unicode.org/reports/tr39/

[4] http://www.unicode.org/reports/tr36/

[5] https://mail.python.org/pipermail/python-3000/2007-May/007925.html

[6] https://mail.python.org/pipermail/python-3000/2007-June/008161.html


text:
PEP: 524 Title: Make os.urandom() blocking on Linux Version: $Revision$
Last-Modified: $Date$ Author: Victor Stinner<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
20-Jun-2016 Python-Version: 3.6

Abstract

Modify os.urandom() to block on Linux 3.17 and newer until the OS
urandom is initialized to increase the security.

Add also a new os.getrandom() function (for Linux and Solaris) to be
able to choose how to handle when os.urandom() is going to block on
Linux.

The bug

Original bug

Python 3.5.0 was enhanced to use the new getrandom() syscall introduced
in Linux 3.17 and Solaris 11.3. The problem is that users started to
complain that Python 3.5 blocks at startup on Linux in virtual machines
and embedded devices: see issues #25420 and #26839.

On Linux, getrandom(0) blocks until the kernel initialized urandom with
128 bits of entropy. The issue #25420 describes a Linux build platform
blocking at import random. The issue #26839 describes a short Python
script used to compute a MD5 hash, systemd-cron, script called very
early in the init process. The system initialization blocks on this
script which blocks on getrandom(0) to initialize Python.

The Python initialization requires random bytes to implement a
counter-measure against the hash denial-of-service (hash DoS), see:

-   Issue #13703: Hash collision security issue
-   PEP 456: Secure and interchangeable hash algorithm
    <456>

Importing the random module creates an instance of random.Random:
random._inst. On Python 3.5, random.Random constructor reads 2500 bytes
from os.urandom() to seed a Mersenne Twister RNG (random number
generator).

Other platforms may be affected by this bug, but in practice, only Linux
systems use Python scripts to initialize the system.

Status in Python 3.5.2

Python 3.5.2 behaves like Python 2.7 and Python 3.4. If the system
urandom is not initialized, the startup does not block, but os.urandom()
can return low-quality entropy (even it is not easily guessable).

Use Cases

The following use cases are used to help to choose the right compromise
between security and practicability.

Use Case 1: init script

Use a Python 3 script to initialize the system, like systemd-cron. If
the script blocks, the system initialize is stuck too. The issue #26839
is a good example of this use case.

Use case 1.1: No secret needed

If the init script doesn't have to generate any secure secret, this use
case is already handled correctly in Python 3.5.2: Python startup
doesn't block on system urandom anymore.

Use case 1.2: Secure secret required

If the init script has to generate a secure secret, there is no safe
solution.

Falling back to weak entropy is not acceptable, it would reduce the
security of the program.

Python cannot produce itself secure entropy, it can only wait until
system urandom is initialized. But in this use case, the whole system
initialization is blocked by this script, so the system fails to boot.

The real answer is that the system initialization must not be blocked by
such script. It is ok to start the script very early at system
initialization, but the script may blocked a few seconds until it is
able to generate the secret.

Reminder: in some cases, the initialization of the system urandom never
occurs and so programs waiting for system urandom blocks forever.

Use Case 2: Web server

Run a Python 3 web server serving web pages using HTTP and HTTPS
protocols. The server is started as soon as possible.

The first target of the hash DoS attack was web server: it's important
that the hash secret cannot be easily guessed by an attacker.

If serving a web page needs a secret to create a cookie, create an
encryption key, ..., the secret must be created with good entropy:
again, it must be hard to guess the secret.

A web server requires security. If a choice must be made between
security and running the server with weak entropy, security is more
important. If there is no good entropy: the server must block or fail
with an error.

The question is if it makes sense to start a web server on a host before
system urandom is initialized.

The issues #25420 and #26839 are restricted to the Python startup, not
to generate a secret before the system urandom is initialized.

Fix system urandom

Load entropy from disk at boot

Collecting entropy can take up to several minutes. To accelerate the
system initialization, operating systems store entropy on disk at
shutdown, and then reload entropy from disk at the boot.

If a system collects enough entropy at least once, the system urandom
will be initialized quickly, as soon as the entropy is reloaded from
disk.

Virtual machines

Virtual machines don't have a direct access to the hardware and so have
less sources of entropy than bare metal. A solution is to add a
virtio-rng device to pass entropy from the host to the virtual machine.

Embedded devices

A solution for embedded devices is to plug an hardware RNG.

For example, Raspberry Pi have an hardware RNG but it's not used by
default. See: Hardware RNG on Raspberry Pi.

Denial-of-service when reading random

Don't use /dev/random but /dev/urandom

The /dev/random device should only used for very specific use cases.
Reading from /dev/random on Linux is likely to block. Users don't like
when an application blocks longer than 5 seconds to generate a secret.
It is only expected for specific cases like generating explicitly an
encryption key.

When the system has no available entropy, choosing between blocking
until entropy is available or falling back on lower quality entropy is a
matter of compromise between security and practicability. The choice
depends on the use case.

On Linux, /dev/urandom is secure, it should be used instead of
/dev/random. See Myths about /dev/urandom by Thomas Hühn: "Fact:
/dev/urandom is the preferred source of cryptographic randomness on
UNIX-like systems"

getrandom(size, 0) can block forever on Linux

The origin of the Python issue #26839 is the Debian bug report #822431:
in fact, getrandom(size, 0) blocks forever on the virtual machine. The
system succeeded to boot because systemd killed the blocked process
after 90 seconds.

Solutions like Load entropy from disk at boot reduces the risk of this
bug.

Rationale

On Linux, reading the /dev/urandom can return "weak" entropy before
urandom is fully initialized, before the kernel collected 128 bits of
entropy. Linux 3.17 adds a new getrandom() syscall which allows to block
until urandom is initialized.

On Python 3.5.2, os.urandom() uses the getrandom(size, GRND_NONBLOCK),
but falls back on reading the non-blocking /dev/urandom if
getrandom(size, GRND_NONBLOCK) fails with EAGAIN.

Security experts promotes os.urandom() to generate cryptographic keys
because it is implemented with a Cryptographically secure pseudo-random
number generator (CSPRNG). By the way, os.urandom() is preferred over
ssl.RAND_bytes() for different reasons.

This PEP proposes to modify os.urandom() to use getrandom() in blocking
mode to not return weak entropy, but also ensure that Python will not
block at startup.

Changes

Make os.urandom() blocking on Linux

All changes described in this section are specific to the Linux
platform.

Changes:

-   Modify os.urandom() to block until system urandom is initialized:
    os.urandom() (C function _PyOS_URandom()) is modified to always call
    getrandom(size, 0) (blocking mode) on Linux and Solaris.
-   Add a new private _PyOS_URandom_Nonblocking() function: try to call
    getrandom(size, GRND_NONBLOCK) on Linux and Solaris, but falls back
    on reading /dev/urandom if it fails with EAGAIN.
-   Initialize hash secret from non-blocking system urandom:
    _PyRandom_Init() is modified to call _PyOS_URandom_Nonblocking().
-   random.Random constructor now uses non-blocking system urandom: it
    is modified to use internally the new _PyOS_URandom_Nonblocking()
    function to seed the RNG.

Add a new os.getrandom() function

A new os.getrandom(size, flags=0) function is added: use getrandom()
syscall on Linux and getrandom() C function on Solaris.

The function comes with 2 new flags:

-   os.GRND_RANDOM: read bytes from /dev/random rather than reading
    /dev/urandom
-   os.GRND_NONBLOCK: raise a BlockingIOError if os.getrandom() would
    block

The os.getrandom() is a thin wrapper on the getrandom() syscall/C
function and so inherit of its behaviour. For example, on Linux, it can
return less bytes than requested if the syscall is interrupted by a
signal.

Examples using os.getrandom()

Best-effort RNG

Example of a portable non-blocking RNG function: try to get random bytes
from the OS urandom, or fallback on the random module.

    def best_effort_rng(size):
        # getrandom() is only available on Linux and Solaris
        if not hasattr(os, 'getrandom'):
            return os.urandom(size)

        result = bytearray()
        try:
            # need a loop because getrandom() can return less bytes than
            # requested for different reasons
            while size:
                data = os.getrandom(size, os.GRND_NONBLOCK)
                result += data
                size -= len(data)
        except BlockingIOError:
            # OS urandom is not initialized yet:
            # fallback on the Python random module
            data = bytes(random.randrange(256) for byte in range(size))
            result += data
        return bytes(result)

This function can block in theory on a platform where os.getrandom() is
not available but os.urandom() can block.

wait_for_system_rng()

Example of function waiting timeout seconds until the OS urandom is
initialized on Linux or Solaris:

    def wait_for_system_rng(timeout, interval=1.0):
        if not hasattr(os, 'getrandom'):
            return

        deadline = time.monotonic() + timeout
        while True:
            try:
                os.getrandom(1, os.GRND_NONBLOCK)
            except BlockingIOError:
                pass
            else:
                return

            if time.monotonic() > deadline:
                raise Exception('OS urandom not initialized after %s seconds'
                                % timeout)

            time.sleep(interval)

This function is not portable. For example, os.urandom() can block on
FreeBSD in theory, at the early stage of the system initialization.

Create a best-effort RNG

Simpler example to create a non-blocking RNG on Linux: choose between
Random.SystemRandom and Random.Random depending if getrandom(size) would
block.

    def create_nonblocking_random():
        if not hasattr(os, 'getrandom'):
            return random.Random()

        try:
            os.getrandom(1, os.GRND_NONBLOCK)
        except BlockingIOError:
            return random.Random()
        else:
            return random.SystemRandom()

This function is not portable. For example, random.SystemRandom can
block on FreeBSD in theory, at the early stage of the system
initialization.

Alternative

Leave os.urandom() unchanged, add os.getrandom()

os.urandom() remains unchanged: never block, but it can return weak
entropy if system urandom is not initialized yet.

Only add the new os.getrandom() function (wrapper to the getrandom()
syscall/C function).

The secrets.token_bytes() function should be used to write portable
code.

The problem with this change is that it expects that users understand
well security and know well each platforms. Python has the tradition of
hiding "implementation details". For example, os.urandom() is not a thin
wrapper to the /dev/urandom device: it uses CryptGenRandom() on Windows,
it uses getentropy() on OpenBSD, it tries getrandom() on Linux and
Solaris or falls back on reading /dev/urandom. Python already uses the
best available system RNG depending on the platform.

This PEP does not change the API:

-   os.urandom(), random.SystemRandom and secrets for security
-   random module (except random.SystemRandom) for all other usages

Raise BlockingIOError in os.urandom()

Proposition

PEP 522: Allow BlockingIOError in security sensitive APIs on Linux
<522>.

Python should not decide for the developer how to handle The bug:
raising immediately a BlockingIOError if os.urandom() is going to block
allows developers to choose how to handle this case:

-   catch the exception and falls back to a non-secure entropy source:
    read /dev/urandom on Linux, use the Python random module (which is
    not secure at all), use time, use process identifier, etc.
-   don't catch the error, the whole program fails with this fatal
    exception

More generally, the exception helps to notify when sometimes goes wrong.
The application can emit a warning when it starts to wait for
os.urandom().

Criticism

For the use case 2 (web server), falling back on non-secure entropy is
not acceptable. The application must handle BlockingIOError: poll
os.urandom() until it completes. Example:

    def secret(n=16):
        try:
            return os.urandom(n)
        except BlockingIOError:
            pass

        print("Wait for system urandom initialization: move your "
              "mouse, use your keyboard, use your disk, ...")
        while 1:
            # Avoid busy-loop: sleep 1 ms
            time.sleep(0.001)
            try:
                return os.urandom(n)
            except BlockingIOError:
                pass

For correctness, all applications which must generate a secure secret
must be modified to handle BlockingIOError even if The bug is unlikely.

The case of applications using os.urandom() but don't really require
security is not well defined. Maybe these applications should not use
os.urandom() at the first place, but always the non-blocking random
module. If os.urandom() is used for security, we are back to the use
case 2 described above: Use Case 2: Web server. If a developer doesn't
want to drop os.urandom(), the code should be modified. Example:

    def almost_secret(n=16):
        try:
            return os.urandom(n)
        except BlockingIOError:
            return bytes(random.randrange(256) for byte in range(n))

The question is if The bug is common enough to require that so many
applications have to be modified.

Another simpler choice is to refuse to start before the system urandom
is initialized:

    def secret(n=16):
        try:
            return os.urandom(n)
        except BlockingIOError:
            print("Fatal error: the system urandom is not initialized")
            print("Wait a bit, and rerun the program later.")
            sys.exit(1)

Compared to Python 2.7, Python 3.4 and Python 3.5.2 where os.urandom()
never blocks nor raise an exception on Linux, such behaviour change can
be seen as a major regression.

Add an optional block parameter to os.urandom()

See the issue #27250: Add os.urandom_block().

Add an optional block parameter to os.urandom(). The default value may
be True (block by default) or False (non-blocking).

The first technical issue is to implement os.urandom(block=False) on all
platforms. Only Linux 3.17 (and newer) and Solaris 11.3 (and newer) have
a well defined non-blocking API (getrandom(size, GRND_NONBLOCK)).

As Raise BlockingIOError in os.urandom(), it doesn't seem worth it to
make the API more complex for a theoretical (or at least very rare) use
case.

As Leave os.urandom() unchanged, add os.getrandom(), the problem is that
it makes the API more complex and so more error-prone.

Acceptance

The PEP was accepted on 2016-08-08 by Guido van Rossum.

Annexes

Operating system random functions

os.urandom() uses the following functions:

-   OpenBSD: getentropy() (OpenBSD 5.6)
-   Linux: getrandom() (Linux 3.17) -- see also A system call for random
    numbers: getrandom()
-   Solaris: getentropy(), getrandom() (both need Solaris 11.3)
-   UNIX, BSD: /dev/urandom, /dev/random
-   Windows: CryptGenRandom() (Windows XP)

On Linux, commands to get the status of /dev/random (results are number
of bytes):

    $ cat /proc/sys/kernel/random/entropy_avail
    2850
    $ cat /proc/sys/kernel/random/poolsize
    4096

Why using os.urandom()?

Since os.urandom() is implemented in the kernel, it doesn't have issues
of user-space RNG. For example, it is much harder to get its state. It
is usually built on a CSPRNG, so even if its state is "stolen", it is
hard to compute previously generated numbers. The kernel has a good
knowledge of entropy sources and feed regularly the entropy pool.

That's also why os.urandom() is preferred over ssl.RAND_bytes().

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 529 Title: Change Windows filesystem encoding to UTF-8 Version:
$Revision$ Last-Modified: $Date$ Author: Steve Dower
<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 27-Aug-2016 Python-Version: 3.6
Post-History: 01-Sep-2016, 04-Sep-2016 Resolution:
https://mail.python.org/pipermail/python-dev/2016-September/146277.html

Abstract

Historically, Python uses the ANSI APIs for interacting with the Windows
operating system, often via C Runtime functions. However, these have
been long discouraged in favor of the UTF-16 APIs. Within the operating
system, all text is represented as UTF-16, and the ANSI APIs perform
encoding and decoding using the active code page. See Naming Files,
Paths, and Namespaces for more details.

This PEP proposes changing the default filesystem encoding on Windows to
utf-8, and changing all filesystem functions to use the Unicode APIs for
filesystem paths. This will not affect code that uses strings to
represent paths, however those that use bytes for paths will now be able
to correctly round-trip all valid paths in Windows filesystems.
Currently, the conversions between Unicode (in the OS) and bytes (in
Python) were lossy and would fail to round-trip characters outside of
the user's active code page.

Notably, this does not impact the encoding of the contents of files.
These will continue to default to locale.getpreferredencoding() (for
text files) or plain bytes (for binary files). This only affects the
encoding used when users pass a bytes object to Python where it is then
passed to the operating system as a path name.

Background

File system paths are almost universally represented as text with an
encoding determined by the file system. In Python, we expose these paths
via a number of interfaces, such as the os and io modules. Paths may be
passed either direction across these interfaces, that is, from the
filesystem to the application (for example, os.listdir()), or from the
application to the filesystem (for example, os.unlink()).

When paths are passed between the filesystem and the application, they
are either passed through as a bytes blob or converted to/from str using
os.fsencode() and os.fsdecode() or explicit encoding using
sys.getfilesystemencoding(). The result of encoding a string with
sys.getfilesystemencoding() is a blob of bytes in the native format for
the default file system.

On Windows, the native format for the filesystem is utf-16-le. The
recommended platform APIs for accessing the filesystem all accept and
return text encoded in this format. However, prior to Windows NT (and
possibly further back), the native format was a configurable machine
option and a separate set of APIs existed to accept this format. The
option (the "active code page") and these APIs (the "*A functions")
still exist in recent versions of Windows for backwards compatibility,
though new functionality often only has a utf-16-le API (the "*W
functions").

In Python, str is recommended because it can correctly round-trip all
characters used in paths (on POSIX with surrogateescape handling; on
Windows because str maps to the native representation). On Windows bytes
cannot round-trip all characters used in paths, as Python internally
uses the *A functions and hence the encoding is "whatever the active
code page is". Since the active code page cannot represent all Unicode
characters, the conversion of a path into bytes can lose information
without warning or any available indication.

As a demonstration of this:

    >>> open('test\uAB00.txt', 'wb').close()
    >>> import glob
    >>> glob.glob('test*')
    ['test\uab00.txt']
    >>> glob.glob(b'test*')
    [b'test?.txt']

The Unicode character in the second call to glob has been replaced by a
'?', which means passing the path back into the filesystem will result
in a FileNotFoundError. The same results may be observed with
os.listdir() or any function that matches the return type to the
parameter type.

While one user-accessible fix is to use str everywhere, POSIX systems
generally do not suffer from data loss when using bytes exclusively as
the bytes are the canonical representation. Even if the encoding is
"incorrect" by some standard, the file system will still map the bytes
back to the file. Making use of this avoids the cost of decoding and
reencoding, such that (theoretically, and only on POSIX), code such as
this may be faster because of the use of b'.' compared to using '.':

    >>> for f in os.listdir(b'.'):
    ...     os.stat(f)
    ...

As a result, POSIX-focused library authors prefer to use bytes to
represent paths. For some authors it is also a convenience, as their
code may receive bytes already known to be encoded correctly, while
others are attempting to simplify porting their code from Python 2.
However, the correctness assumptions do not carry over to Windows where
Unicode is the canonical representation, and errors may result. This
potential data loss is why the use of bytes paths on Windows was
deprecated in Python 3.3 - all of the above code snippets produce
deprecation warnings on Windows.

Proposal

Currently the default filesystem encoding is 'mbcs', which is a
meta-encoder that uses the active code page. However, when bytes are
passed to the filesystem they go through the *A APIs and the operating
system handles encoding. In this case, paths are always encoded using
the equivalent of 'mbcs:replace' with no opportunity for Python to
override or change this.

This proposal would remove all use of the *A APIs and only ever call the
*W APIs. When Windows returns paths to Python as str, they will be
decoded from utf-16-le and returned as text (in whatever the minimal
representation is). When Python code requests paths as bytes, the paths
will be transcoded from utf-16-le into utf-8 using surrogatepass
(Windows does not validate surrogate pairs, so it is possible to have
invalid surrogates in filenames). Equally, when paths are provided as
bytes, they are transcoded from utf-8 into utf-16-le and passed to the
*W APIs.

The use of utf-8 will not be configurable, except for the provision of a
"legacy mode" flag to revert to the previous behaviour.

The surrogateescape error mode does not apply here, as the concern is
not about retaining nonsensical bytes. Any path returned from the
operating system will be valid Unicode, while invalid paths created by
the user should raise a decoding error (currently these would raise
OSError or a subclass).

The choice of utf-8 bytes (as opposed to utf-16-le bytes) is to ensure
the ability to round-trip path names and allow basic manipulation (for
example, using the os.path module) when assuming an ASCII-compatible
encoding. Using utf-16-le as the encoding is more pure, but will cause
more issues than are resolved.

This change would also undeprecate the use of bytes paths on Windows. No
change to the semantics of using bytes as a path is required - as
before, they must be encoded with the encoding specified by
sys.getfilesystemencoding().

Specific Changes

Update sys.getfilesystemencoding

Remove the default value for Py_FileSystemDefaultEncoding and set it in
initfsencoding() to utf-8, or if the legacy-mode switch is enabled to
mbcs.

Update the implementations of PyUnicode_DecodeFSDefaultAndSize() and
PyUnicode_EncodeFSDefault() to use the utf-8 codec, or if the
legacy-mode switch is enabled the existing mbcs codec.

Add sys.getfilesystemencodeerrors

As the error mode may now change between surrogatepass and replace,
Python code that manually performs encoding also needs access to the
current error mode. This includes the implementation of os.fsencode()
and os.fsdecode(), which currently assume an error mode based on the
codec.

Add a public Py_FileSystemDefaultEncodeErrors, similar to the existing
Py_FileSystemDefaultEncoding. The default value on Windows will be
surrogatepass or in legacy mode, replace. The default value on all other
platforms will be surrogateescape.

Add a public sys.getfilesystemencodeerrors() function that returns the
current error mode.

Update the implementations of PyUnicode_DecodeFSDefaultAndSize() and
PyUnicode_EncodeFSDefault() to use the variable for error mode rather
than constant strings.

Update the implementations of os.fsencode() and os.fsdecode() to use
sys.getfilesystemencodeerrors() instead of assuming the mode.

Update path_converter

Update the path converter to always decode bytes or buffer objects into
text using PyUnicode_DecodeFSDefaultAndSize().

Change the narrow field from a char* string into a flag that indicates
whether the original object was bytes. This is required for functions
that need to return paths using the same type as was originally
provided.

Remove unused ANSI code

Remove all code paths using the narrow field, as these will no longer be
reachable by any caller. These are only used within posixmodule.c. Other
uses of paths should have use of bytes paths replaced with decoding and
use of the *W APIs.

Add legacy mode

Add a legacy mode flag, enabled by the environment variable
PYTHONLEGACYWINDOWSFSENCODING or by a function call to
sys._enablelegacywindowsfsencoding(). The function call can only be used
to enable the flag and should be used by programs as close to
initialization as possible. Legacy mode cannot be disabled while Python
is running.

When this flag is set, the default filesystem encoding is set to mbcs
rather than utf-8, and the error mode is set to replace rather than
surrogatepass. Paths will continue to decode to wide characters and only
*W APIs will be called, however, the bytes passed in and received from
Python will be encoded the same as prior to this change.

Undeprecate bytes paths on Windows

Using bytes as paths on Windows is currently deprecated. We would
announce that this is no longer the case, and that paths when encoded as
bytes should use whatever is returned from sys.getfilesystemencoding()
rather than the user's active code page.

Beta experiment

To assist with determining the impact of this change, we propose
applying it to 3.6.0b1 provisionally with the intent being to make a
final decision before 3.6.0b4.

During the experiment period, decoding and encoding exception messages
will be expanded to include a link to an active online discussion and
encourage reporting of problems.

If it is decided to revert the functionality for 3.6.0b4, the
implementation change would be to permanently enable the legacy mode
flag, change the environment variable to PYTHONWINDOWSUTF8FSENCODING and
function to sys._enablewindowsutf8fsencoding() to allow enabling the
functionality on a case-by-case basis, as opposed to disabling it.

It is expected that if we cannot feasibly make the change for 3.6 due to
compatibility concerns, it will not be possible to make the change at
any later time in Python 3.x.

Affected Modules

This PEP implicitly includes all modules within the Python that either
pass path names to the operating system, or otherwise use
sys.getfilesystemencoding().

As of 3.6.0a4, the following modules require modification:

-   os
-   _overlapped
-   _socket
-   subprocess
-   zipimport

The following modules use sys.getfilesystemencoding() but do not need
modification:

-   gc (already assumes bytes are utf-8)
-   grp (not compiled for Windows)
-   http.server (correctly includes codec name with transmitted data)
-   idlelib.editor (should not be needed; has fallback handling)
-   nis (not compiled for Windows)
-   pwd (not compiled for Windows)
-   spwd (not compiled for Windows)
-   _ssl (only used for ASCII constants)
-   tarfile (code unused on Windows)
-   _tkinter (already assumes bytes are utf-8)
-   wsgiref (assumed as the default encoding for unknown environments)
-   zipapp (code unused on Windows)

The following native code uses one of the encoding or decoding
functions, but do not require any modification:

-   Parser/parsetok.c (docs already specify sys.getfilesystemencoding())
-   Python/ast.c (docs already specify sys.getfilesystemencoding())
-   Python/compile.c (undocumented, but Python filesystem encoding
    implied)
-   Python/errors.c (docs already specify os.fsdecode())
-   Python/fileutils.c (code unused on Windows)
-   Python/future.c (undocumented, but Python filesystem encoding
    implied)
-   Python/import.c (docs already specify utf-8)
-   Python/importdl.c (code unused on Windows)
-   Python/pythonrun.c (docs already specify
    sys.getfilesystemencoding())
-   Python/symtable.c (undocumented, but Python filesystem encoding
    implied)
-   Python/thread.c (code unused on Windows)
-   Python/traceback.c (encodes correctly for comparing strings)
-   Python/_warnings.c (docs already specify os.fsdecode())

Rejected Alternatives

Use strict mbcs decoding

This is essentially the same as the proposed change, but instead of
changing sys.getfilesystemencoding() to utf-8 it is changed to mbcs
(which dynamically maps to the active code page).

This approach allows the use of new functionality that is only available
as *W APIs and also detection of encoding/decoding errors. For example,
rather than silently replacing Unicode characters with '?', it would be
possible to warn or fail the operation.

Compared to the proposed fix, this could enable some new functionality
but does not fix any of the problems described initially. New runtime
errors may cause some problems to be more obvious and lead to fixes,
provided library maintainers are interested in supporting Windows and
adding a separate code path to treat filesystem paths as strings.

Making the encoding mbcs without strict errors is equivalent to the
legacy-mode switch being enabled by default. This is a possible course
of action if there is significant breakage of actual code and a need to
extend the deprecation period, but still a desire to have the
simplifications to the CPython source.

Make bytes paths an error on Windows

By preventing the use of bytes paths on Windows completely we prevent
users from hitting encoding issues.

However, the motivation for this PEP is to increase the likelihood that
code written on POSIX will also work correctly on Windows. This
alternative would move the other direction and make such code completely
incompatible. As this does not benefit users in any way, we reject it.

Make bytes paths an error on all platforms

By deprecating and then disable the use of bytes paths on all platforms
we prevent users from hitting encoding issues regardless of where the
code was originally written. This would require a full deprecation
cycle, as there are currently no warnings on platforms other than
Windows.

This is likely to be seen as a hostile action against Python developers
in general, and as such is rejected at this time.

Code that may break

The following code patterns may break or see different behaviour as a
result of this change. Each of these examples would have been fragile in
code intended for cross-platform use. The suggested fixes demonstrate
the most compatible way to handle path encoding issues across all
platforms and across multiple Python versions.

Note that all of these examples produce deprecation warnings on Python
3.3 and later.

Not managing encodings across boundaries

Code that does not manage encodings when crossing protocol boundaries
may currently be working by chance, but could encounter issues when
either encoding changes. Note that the source of filename may be any
function that returns a bytes object, as illustrated in a second example
below:

    >>> filename = open('filename_in_mbcs.txt', 'rb').read()
    >>> text = open(filename, 'r').read()

To correct this code, the encoding of the bytes in filename should be
specified, either when reading from the file or before using the value:

    >>> # Fix 1: Open file as text (default encoding)
    >>> filename = open('filename_in_mbcs.txt', 'r').read()
    >>> text = open(filename, 'r').read()

    >>> # Fix 2: Open file as text (explicit encoding)
    >>> filename = open('filename_in_mbcs.txt', 'r', encoding='mbcs').read()
    >>> text = open(filename, 'r').read()

    >>> # Fix 3: Explicitly decode the path
    >>> filename = open('filename_in_mbcs.txt', 'rb').read()
    >>> text = open(filename.decode('mbcs'), 'r').read()

Where the creator of filename is separated from the user of filename,
the encoding is important information to include:

    >>> some_object.filename = r'C:\Users\Steve\Documents\my_file.txt'.encode('mbcs')

    >>> filename = some_object.filename
    >>> type(filename)
    <class 'bytes'>
    >>> text = open(filename, 'r').read()

To fix this code for best compatibility across operating systems and
Python versions, the filename should be exposed as str:

    >>> # Fix 1: Expose as str
    >>> some_object.filename = r'C:\Users\Steve\Documents\my_file.txt'

    >>> filename = some_object.filename
    >>> type(filename)
    <class 'str'>
    >>> text = open(filename, 'r').read()

Alternatively, the encoding used for the path needs to be made available
to the user. Specifying os.fsencode() (or sys.getfilesystemencoding())
is an acceptable choice, or a new attribute could be added with the
exact encoding:

    >>> # Fix 2: Use fsencode
    >>> some_object.filename = os.fsencode(r'C:\Users\Steve\Documents\my_file.txt')

    >>> filename = some_object.filename
    >>> type(filename)
    <class 'bytes'>
    >>> text = open(filename, 'r').read()

    >>> # Fix 3: Expose as explicit encoding
    >>> some_object.filename = r'C:\Users\Steve\Documents\my_file.txt'.encode('cp437')
    >>> some_object.filename_encoding = 'cp437'

    >>> filename = some_object.filename
    >>> type(filename)
    <class 'bytes'>
    >>> filename = filename.decode(some_object.filename_encoding)
    >>> type(filename)
    <class 'str'>
    >>> text = open(filename, 'r').read()

Explicitly using 'mbcs'

Code that explicitly encodes text using 'mbcs' before passing to file
system APIs is now passing incorrectly encoded bytes. Note that the
source of filename in this example is not relevant, provided that it is
a str:

    >>> filename = open('files.txt', 'r').readline().rstrip()
    >>> text = open(filename.encode('mbcs'), 'r')

To correct this code, the string should be passed without explicit
encoding, or should use os.fsencode():

    >>> # Fix 1: Do not encode the string
    >>> filename = open('files.txt', 'r').readline().rstrip()
    >>> text = open(filename, 'r')

    >>> # Fix 2: Use correct encoding
    >>> filename = open('files.txt', 'r').readline().rstrip()
    >>> text = open(os.fsencode(filename), 'r')

References

Copyright

This document has been placed in the public domain.


text:
PEP: 507 Title: Migrate CPython to Git and GitLab Author: Barry Warsaw
<EMAIL_ADDRESS>Status: Rejected Type: Process Content-Type:
text/x-rst Created: 30-Sep-2015 Post-History: Resolution:
https://mail.python.org/pipermail/core-workflow/2016-January/000345.html

Abstract

This PEP proposes migrating the repository hosting of CPython and the
supporting repositories to Git. Further, it proposes adopting a hosted
GitLab instance as the primary way of handling merge requests, code
reviews, and code hosting. It is similar in intent to PEP 481 but
proposes an open source alternative to GitHub and omits the proposal to
run Phabricator. As with PEP 481, this particular PEP is offered as an
alternative to PEP 474 and PEP 462.

Rationale

CPython is an open source project which relies on a number of volunteers
donating their time. As with any healthy, vibrant open source project,
it relies on attracting new volunteers as well as retaining existing
developers. Given that volunteer time is the most scarce resource,
providing a process that maximizes the efficiency of contributors and
reduces the friction for contributions, is of vital importance for the
long-term health of the project.

The current tool chain of the CPython project is a custom and unique
combination of tools. This has two critical implications:

-   The unique nature of the tool chain means that contributors must
    remember or relearn, the process, workflow, and tools whenever they
    contribute to CPython, without the advantage of leveraging long-term
    memory and familiarity they retain by working with other projects in
    the FLOSS ecosystem. The knowledge they gain in working with CPython
    is unlikely to be applicable to other projects.
-   The burden on the Python/PSF infrastructure team is much greater in
    order to continue to maintain custom tools, improve them over time,
    fix bugs, address security issues, and more generally adapt to new
    standards in online software development with global collaboration.

These limitations act as a barrier to contribution both for highly
engaged contributors (e.g. core Python developers) and especially for
more casual "drive-by" contributors, who care more about getting their
bug fix than learning a new suite of tools and workflows.

By proposing the adoption of both a different version control system and
a modern, well-maintained hosting solution, this PEP addresses these
limitations. It aims to enable a modern, well-understood process that
will carry CPython development for many years.

Version Control System

Currently the CPython and supporting repositories use Mercurial. As a
modern distributed version control system, it has served us well since
the migration from Subversion. However, when evaluating the VCS we must
consider the capabilities of the VCS itself as well as the network
effect and mindshare of the community around that VCS.

There are really only two real options for this, Mercurial and Git. The
technical capabilities of the two systems are largely equivalent,
therefore this PEP instead focuses on their social aspects.

It is not possible to get exact numbers for the number of projects or
people which are using a particular VCS, however we can infer this by
looking at several sources of information for what VCS projects are
using.

The Open Hub (previously Ohloh) statistics[1] show that 37% of the
repositories indexed by The Open Hub are using Git (second only to
Subversion which has 48%) while Mercurial has just 2%, beating only
Bazaar which has 1%. This has Git being just over 18 times as popular as
Mercurial on The Open Hub.

Another source of information on VCS popularity is PyPI itself. This
source is more targeted at the Python community itself since it
represents projects developed for Python. Unfortunately PyPI does not
have a standard location for representing this information, so this
requires manual processing. If we limit our search to the top 100
projects on PyPI (ordered by download counts) we can see that 62% of
them use Git, while 22% of them use Mercurial, and 13% use something
else. This has Git being just under 3 times as popular as Mercurial for
the top 100 projects on PyPI.

These numbers back up the anecdotal evidence for Git as the far more
popular DVCS for open source projects. Choosing the more popular VCS has
a number of positive benefits.

For new contributors it increases the likelihood that they will have
already learned the basics of Git as part of working with another
project or if they are just now learning Git, that they'll be able to
take that knowledge and apply it to other projects. Additionally a
larger community means more people writing how to guides, answering
questions, and writing articles about Git which makes it easier for a
new user to find answers and information about the tool they are trying
to learn and use. Given its popularity, there may also be more auxiliary
tooling written around Git. This increases options for everything from
GUI clients, helper scripts, repository hosting, etc.

Further, the adoption of Git as the proposed back-end repository format
doesn't prohibit the use of Mercurial by fans of that VCS! Mercurial
users have the[2] plugin which allows them to push and pull from a Git
server using the Mercurial front-end. It's a well-maintained and highly
functional plugin that seems to be well-liked by Mercurial users.

Repository Hosting

Where and how the official repositories for CPython are hosted is in
someways determined by the choice of VCS. With Git there are several
options. In fact, once the repository is hosted in Git, branches can be
mirrored in many locations, within many free, open, and proprietary code
hosting sites.

It's still important for CPython to adopt a single, official repository,
with a web front-end that allows for many convenient and common
interactions entirely through the web, without always requiring local
VCS manipulations. These interactions include as a minimum, code review
with inline comments, branch diffing, CI integration, and auto-merging.

This PEP proposes to adopt a[3] instance, run within the python.org
domain, accessible to and with ultimate control from the PSF and the
Python infrastructure team, but donated, hosted, and primarily
maintained by GitLab, Inc.

Why GitLab? Because it is a fully functional Git hosting system, that
sports modern web interactions, software workflows, and CI integration.
GitLab's Community Edition (CE) is open source software, and thus is
closely aligned with the principles of the CPython community.

Code Review

Currently CPython uses a custom fork of Rietveld modified to not run on
Google App Engine and which is currently only really maintained by one
person. It is missing common features present in many modern code review
tools.

This PEP proposes to utilize GitLab's built-in merge requests and online
code review features to facilitate reviews of all proposed changes.

GitLab merge requests

The normal workflow for a GitLab hosted project is to submit a merge
request asking that a feature or bug fix branch be merged into a target
branch, usually one or more of the stable maintenance branches or the
next-version master branch for new features. GitLab's merge requests are
similar in form and function to GitHub's pull requests, so anybody who
is already familiar with the latter should be able to immediately
utilize the former.

Once submitted, a conversation about the change can be had between the
submitter and reviewer. This includes both general comments, and inline
comments attached to a particular line of the diff between the source
and target branches. Projects can also be configured to automatically
run continuous integration on the submitted branch, the results of which
are readily visible from the merge request page. Thus both the reviewer
and submitter can immediately see the results of the tests, making it
much easier to only land branches with passing tests. Each new push to
the source branch (e.g. to respond to a commenter's feedback or to fix a
failing test) results in a new run of the CI, so that the state of the
request always reflects the latest commit.

Merge requests have a fairly major advantage over the older "submit a
patch to a bug tracker" model. They allow developers to work completely
within the VCS using standard VCS tooling, without requiring the
creation of a patch file or figuring out the right location to upload
the patch to. This lowers the barrier for sending a change to be
reviewed.

Merge requests are far easier to review. For example, they provide nice
syntax highlighted diffs which can operate in either unified or side by
side views. They allow commenting inline and on the merge request as a
whole and they present that in a nice unified way which will also hide
comments which no longer apply. Comments can be hidden and revealed.

Actually merging a merge request is quite simple, if the source branch
applies cleanly to the target branch. A core reviewer simply needs to
press the "Merge" button for GitLab to automatically perform the merge.
The source branch can be optionally rebased, and once the merge is
completed, the source branch can be automatically deleted.

GitLab also has a good workflow for submitting pull requests to a
project completely through their web interface. This would enable the
Python documentation to have "Edit on GitLab" buttons on every page and
people who discover things like typos, inaccuracies, or just want to
make improvements to the docs they are currently reading. They can
simply hit that button and get an in browser editor that will let them
make changes and submit a merge request all from the comfort of their
browser.

Criticism

X is not written in Python

One feature that the current tooling (Mercurial, Rietveld) has is that
the primary language for all of the pieces are written in Python. This
PEP focuses more on the best tools for the job and not necessarily on
the best tools that happen to be written in Python. Volunteer time is
the most precious resource for any open source project and we can best
respect and utilize that time by focusing on the benefits and downsides
of the tools themselves rather than what language their authors happened
to write them in.

One concern is the ability to modify tools to work for us, however one
of the Goals here is to not modify software to work for us and instead
adapt ourselves to a more standardized workflow. This standardization
pays off in the ability to re-use tools out of the box freeing up
developer time to actually work on Python itself as well as enabling
knowledge sharing between projects.

However, if we do need to modify the tooling, Git itself is largely
written in C the same as CPython itself. It can also have commands
written for it using any language, including Python. GitLab itself is
largely written in Ruby and since it is Open Source software, we would
have the ability to submit merge requests to the upstream Community
Edition, albeit in language potentially unfamiliar to most Python
programmers.

Mercurial is better than Git

Whether Mercurial or Git is better on a technical level is a highly
subjective opinion. This PEP does not state whether the mechanics of Git
or Mercurial are better, and instead focuses on the network effect that
is available for either option. While this PEP proposes switching to
Git, Mercurial users are not left completely out of the loop. By using
the hg-git extension for Mercurial, working with server-side Git
repositories is fairly easy and straightforward.

CPython Workflow is too Complicated

One sentiment that came out of previous discussions was that the
multi-branch model of CPython was too complicated for GitLab style merge
requests. This PEP disagrees with that sentiment.

Currently any particular change requires manually creating a patch for
2.7 and 3.x which won't change at all in this regards.

If someone submits a fix for the current stable branch (e.g. 3.5) the
merge request workflow can be used to create a request to merge the
current stable branch into the master branch, assuming there is no merge
conflicts. As always, merge conflicts must be manually and locally
resolved. Because developers also have the option of performing the
merge locally, this provides an improvement over the current situation
where the merge must always happen locally.

For fixes in the current development branch that must also be applied to
stable release branches, it is possible in many situations to locally
cherry pick and apply the change to other branches, with merge requests
submitted for each stable branch. It is also possible just cherry pick
and complete the merge locally. These are all accomplished with standard
Git commands and techniques, with the advantage that all such changes
can go through the review and CI test workflows, even for merges to
stable branches. Minor changes may be easily accomplished in the GitLab
web editor.

No system can hide all the complexities involved in maintaining several
long lived branches. The only thing that the tooling can do is make it
as easy as possible to submit and commit changes.

Open issues

-   What level of hosted support will GitLab offer? The PEP author has
    been in contact with the GitLab CEO, with positive interest on their
    part. The details of the hosting offer would have to be discussed.
-   What happens to Roundup and do we switch to the GitLab issue
    tracker? Currently, this PEP is not suggesting we move from Roundup
    to GitLab issues. We have way too much invested in Roundup right now
    and migrating the data would be a huge effort. GitLab does support
    webhooks, so we will probably want to use webhooks to integrate
    merges and other events with updates to Roundup (e.g. to include
    pointers to commits, close issues, etc. similar to what is currently
    done).
-   What happens to wiki.python.org? Nothing! While GitLab does support
    wikis in repositories, there's no reason for us to migration our
    Moin wikis.
-   What happens to the existing GitHub mirrors? We'd probably want to
    regenerate them once the official upstream branches are natively
    hosted in Git. This may change commit ids, but after that, it should
    be easy to mirror the official Git branches and repositories far and
    wide.
-   Where would the GitLab instance live? Physically, in whatever
    hosting provider GitLab chooses. We would point gitlab.python.org
    (or git.python.org?) to this host.

References

Copyright

This document has been placed in the public domain.

[1] Open Hub Statistics

[2] Hg-Git mercurial plugin

[3] https://about.gitlab.com


text:
PEP: 553 Title: Built-in breakpoint() Author: Barry Warsaw
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 05-Sep-2017 Python-Version: 3.7 Post-History:
05-Sep-2017, 07-Sep-2017, 13-Sep-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-October/149705.html

Abstract

This PEP proposes adding a new built-in function called breakpoint()
which enters a Python debugger at the point of the call. Additionally,
two new names are added to the sys module to make the choice of which
debugger is entered configurable.

Rationale

Python has long had a great debugger in its standard library called pdb.
Setting a break point is commonly written like this:

    foo()
    import pdb; pdb.set_trace()
    bar()

Thus after executing foo() and before executing bar(), Python will enter
the debugger. However this idiom has several disadvantages.

-   It's a lot to type (27 characters).
-   It's easy to typo. The PEP author often mistypes this line, e.g.
    omitting the semicolon, or typing a dot instead of an underscore.
-   It ties debugging directly to the choice of pdb. There might be
    other debugging options, say if you're using an IDE or some other
    development environment.
-   Python linters (e.g. flake8 [linters]) complain about this line
    because it contains two statements. Breaking the idiom up into two
    lines complicates its use because there are more opportunities for
    mistakes at clean up time. I.e. you might forget to delete one of
    those lines when you no longer need to debug the code.

Python developers also have many other debuggers to choose from, but
remembering how to invoke them can be problematic. For example, even
when IDEs have user interface for setting breakpoints, it may still be
more convenient to just edit the code. The APIs for entering the
debugger programmatically are inconsistent, so it can be difficult to
remember exactly what to type.

We can solve all these problems by providing a universal API for
entering the debugger, as proposed in this PEP.

Proposal

The JavaScript language provides a debugger statement [js-debugger]
which enters the debugger at the point where the statement appears.

This PEP proposes a new built-in function called breakpoint() which
enters a Python debugger at the call site. Thus the example above would
be written like so:

    foo()
    breakpoint()
    bar()

Further, this PEP proposes two new name bindings for the sys module,
called sys.breakpointhook() and sys.__breakpointhook__. By default,
sys.breakpointhook() implements the actual importing and entry into
pdb.set_trace(), and it can be set to a different function to change the
debugger that breakpoint() enters.

sys.__breakpointhook__ is initialized to the same function as
sys.breakpointhook() so that you can always easily reset
sys.breakpointhook() to the default value (e.g. by doing
sys.breakpointhook = sys.__breakpointhook__). This is exactly the same
as how the existing sys.displayhook() / sys.__displayhook__ and
sys.excepthook() / sys.__excepthook__ work [hooks].

The signature of the built-in is breakpoint(*args, **kws). The
positional and keyword arguments are passed straight through to
sys.breakpointhook() and the signatures must match or a TypeError will
be raised. The return from sys.breakpointhook() is passed back up to,
and returned from breakpoint().

The rationale for this is based on the observation that the underlying
debuggers may accept additional optional arguments. For example, IPython
allows you to specify a string that gets printed when the break point is
entered [ipython-embed]. As of Python 3.7, the pdb module also supports
an optional header argument [pdb-header].

Environment variable

The default implementation of sys.breakpointhook() consults a new
environment variable called PYTHONBREAKPOINT. This environment variable
can have various values:

-   PYTHONBREAKPOINT=0 disables debugging. Specifically, with this value
    sys.breakpointhook() returns None immediately.
-   PYTHONBREAKPOINT= (i.e. the empty string). This is the same as not
    setting the environment variable at all, in which case
    pdb.set_trace() is run as usual.
-   PYTHONBREAKPOINT=some.importable.callable. In this case,
    sys.breakpointhook() imports the some.importable module and gets the
    callable object from the resulting module, which it then calls. The
    value may be a string with no dots, in which case it names a
    built-in callable, e.g. PYTHONBREAKPOINT=int. (Guido has expressed
    the preference for normal Python dotted-paths, not setuptools-style
    entry point syntax [syntax].)

This environment variable allows external processes to control how
breakpoints are handled. Some uses cases include:

-   Completely disabling all accidental breakpoint() calls pushed to
    production. This could be accomplished by setting PYTHONBREAKPOINT=0
    in the execution environment. Another suggestion by reviewers of the
    PEP was to set PYTHONBREAKPOINT=sys.exit in this case.
-   IDE integration with specialized debuggers for embedded execution.
    The IDE would run the program in its debugging environment with
    PYTHONBREAKPOINT set to their internal debugging hook.

PYTHONBREAKPOINT is re-interpreted every time sys.breakpointhook() is
reached. This allows processes to change its value during the execution
of a program and have breakpoint() respond to those changes. It is not
considered a performance critical section since entering a debugger by
definition stops execution. Thus, programs can do the following:

    os.environ['PYTHONBREAKPOINT'] = 'foo.bar.baz'
    breakpoint()    # Imports foo.bar and calls foo.bar.baz()

Overriding sys.breakpointhook defeats the default consultation of
PYTHONBREAKPOINT. It is up to the overriding code to consult
PYTHONBREAKPOINT if they want.

If access to the PYTHONBREAKPOINT callable fails in any way (e.g. the
import fails, or the resulting module does not contain the callable), a
RuntimeWarning is issued, and no breakpoint function is called.

Note that as with all other PYTHON* environment variables,
PYTHONBREAKPOINT is ignored when the interpreter is started with -E.
This means the default behavior will occur (i.e. pdb.set_trace() will
run). There was some discussion about alternatively treating
PYTHONBREAKPOINT=0 when -E as in effect, but the opinions were
inconclusive, so it was decided that this wasn't special enough for a
special case.

Implementation

A pull request exists with the proposed implementation [impl].

While the actual implementation is in C, the Python pseudo-code for this
feature looks roughly like the following:

    # In builtins.
    def breakpoint(*args, **kws):
        import sys
        missing = object()
        hook = getattr(sys, 'breakpointhook', missing)
        if hook is missing:
            raise RuntimeError('lost sys.breakpointhook')
        return hook(*args, **kws)

    # In sys.
    def breakpointhook(*args, **kws):
        import importlib, os, warnings
        hookname = os.getenv('PYTHONBREAKPOINT')
        if hookname is None or len(hookname) == 0:
            hookname = 'pdb.set_trace'
        elif hookname == '0':
            return None
        modname, dot, funcname = hookname.rpartition('.')
        if dot == '':
            modname = 'builtins'
        try:
            module = importlib.import_module(modname)
            hook = getattr(module, funcname)
        except:
            warnings.warn(
                'Ignoring unimportable $PYTHONBREAKPOINT: {}'.format(
                    hookname),
                RuntimeWarning)
            return None
        return hook(*args, **kws)

    __breakpointhook__ = breakpointhook

Rejected alternatives

A new keyword

Originally, the author considered a new keyword, or an extension to an
existing keyword such as break here. This is rejected on several fronts.

-   A brand new keyword would require a __future__ to enable it since
    almost any new keyword could conflict with existing code. This
    negates the ease with which you can enter the debugger.
-   An extended keyword such as break here, while more readable and not
    requiring a __future__ would tie the keyword extension to this new
    feature, preventing more useful extensions such as those proposed in
    PEP 548.
-   A new keyword would require a modified grammar and likely a new
    bytecode. Each of these makes the implementation more complex. A new
    built-in breaks no existing code (since any existing module global
    would just shadow the built-in) and is quite easy to implement.

sys.breakpoint()

Why not sys.breakpoint()? Requiring an import to invoke the debugger is
explicitly rejected because sys is not imported in every module. That
just requires more typing and would lead to:

    import sys; sys.breakpoint()

which inherits several of the problems this PEP aims to solve.

Version History

-   2019-10-13
    -   Add missing return None in except clause to pseudo-code.
-   2017-09-13
    -   The PYTHONBREAKPOINT environment variable is made a first class
        feature.
-   2017-09-07
    -   debug() renamed to breakpoint()
    -   Signature changed to breakpoint(*args, **kws) which is passed
        straight through to sys.breakpointhook().

References

Copyright

This document has been placed in the public domain.

hooks

    https://docs.python.org/3/library/sys.html#sys.displayhook

impl

    https://github.com/python/cpython/pull/3355

ipython-embed

    http://ipython.readthedocs.io/en/stable/api/generated/IPython.terminal.embed.html

js-debugger

    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/debugger

linters

    http://flake8.readthedocs.io/en/latest/

pdb-header

    https://docs.python.org/3.7/library/pdb.html#pdb.set_trace

syntax

    http://setuptools.readthedocs.io/en/latest/setuptools.html?highlight=console#automatic-script-creation


text:
PEP: 358 Title: The "bytes" Object Version: $Revision$ Last-Modified:
$Date$ Author: Neil Schemenauer<EMAIL_ADDRESS>Guido van Rossum
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 15-Feb-2006 Python-Version: 2.6, 3.0 Post-History:

Update

This PEP has partially been superseded by PEP 3137.

Abstract

This PEP outlines the introduction of a raw bytes sequence type. Adding
the bytes type is one step in the transition to Unicode-based str
objects which will be introduced in Python 3.0.

The PEP describes how the bytes type should work in Python 2.6, as well
as how it should work in Python 3.0. (Occasionally there are differences
because in Python 2.6, we have two string types, str and unicode, while
in Python 3.0 we will only have one string type, whose name will be str
but whose semantics will be like the 2.6 unicode type.)

Motivation

Python's current string objects are overloaded. They serve to hold both
sequences of characters and sequences of bytes. This overloading of
purpose leads to confusion and bugs. In future versions of Python,
string objects will be used for holding character data. The bytes object
will fulfil the role of a byte container. Eventually the unicode type
will be renamed to str and the old str type will be removed.

Specification

A bytes object stores a mutable sequence of integers that are in the
range 0 to 255. Unlike string objects, indexing a bytes object returns
an integer. Assigning or comparing an object that is not an integer to
an element causes a TypeError exception. Assigning an element to a value
outside the range 0 to 255 causes a ValueError exception. The .__len__()
method of bytes returns the number of integers stored in the sequence
(i.e. the number of bytes).

The constructor of the bytes object has the following signature:

    bytes([initializer[, encoding]])

If no arguments are provided then a bytes object containing zero
elements is created and returned. The initializer argument can be a
string (in 2.6, either str or unicode), an iterable of integers, or a
single integer. The pseudo-code for the constructor (optimized for clear
semantics, not for speed) is:

    def bytes(initializer=0, encoding=None):
        if isinstance(initializer, int): # In 2.6, int -> (int, long)
            initializer = [0]*initializer
        elif isinstance(initializer, basestring):
            if isinstance(initializer, unicode): # In 3.0, "if True"
                if encoding is None:
                    # In 3.0, raise TypeError("explicit encoding required")
                    encoding = sys.getdefaultencoding()
                initializer = initializer.encode(encoding)
            initializer = [ord(c) for c in initializer]
        else:
            if encoding is not None:
                raise TypeError("no encoding allowed for this initializer")
            tmp = []
            for c in initializer:
                if not isinstance(c, int):
                    raise TypeError("initializer must be iterable of ints")
                if not 0 <= c < 256:
                    raise ValueError("initializer element out of range")
                tmp.append(c)
            initializer = tmp
        new = <new bytes object of length len(initializer)>
        for i, c in enumerate(initializer):
            new[i] = c
        return new

The .__repr__() method returns a string that can be evaluated to
generate a new bytes object containing a bytes literal:

    >>> bytes([10, 20, 30])
    b'\n\x14\x1e'

The object has a .decode() method equivalent to the .decode() method of
the str object. The object has a classmethod .fromhex() that takes a
string of characters from the set [0-9a-fA-F ] and returns a bytes
object (similar to binascii.unhexlify). For example:

    >>> bytes.fromhex('5c5350ff')
    b'\\SP\xff'
    >>> bytes.fromhex('5c 53 50 ff')
    b'\\SP\xff'

The object has a .hex() method that does the reverse conversion (similar
to binascii.hexlify):

    >> bytes([92, 83, 80, 255]).hex()
    '5c5350ff'

The bytes object has some methods similar to list methods, and others
similar to str methods. Here is a complete list of methods, with their
approximate signatures:

    .__add__(bytes) -> bytes
    .__contains__(int | bytes) -> bool
    .__delitem__(int | slice) -> None
    .__delslice__(int, int) -> None
    .__eq__(bytes) -> bool
    .__ge__(bytes) -> bool
    .__getitem__(int | slice) -> int | bytes
    .__getslice__(int, int) -> bytes
    .__gt__(bytes) -> bool
    .__iadd__(bytes) -> bytes
    .__imul__(int) -> bytes
    .__iter__() -> iterator
    .__le__(bytes) -> bool
    .__len__() -> int
    .__lt__(bytes) -> bool
    .__mul__(int) -> bytes
    .__ne__(bytes) -> bool
    .__reduce__(...) -> ...
    .__reduce_ex__(...) -> ...
    .__repr__() -> str
    .__reversed__() -> bytes
    .__rmul__(int) -> bytes
    .__setitem__(int | slice, int | iterable[int]) -> None
    .__setslice__(int, int, iterable[int]) -> Bote
    .append(int) -> None
    .count(int) -> int
    .decode(str) -> str | unicode # in 3.0, only str
    .endswith(bytes) -> bool
    .extend(iterable[int]) -> None
    .find(bytes) -> int
    .index(bytes | int) -> int
    .insert(int, int) -> None
    .join(iterable[bytes]) -> bytes
    .partition(bytes) -> (bytes, bytes, bytes)
    .pop([int]) -> int
    .remove(int) -> None
    .replace(bytes, bytes) -> bytes
    .rindex(bytes | int) -> int
    .rpartition(bytes) -> (bytes, bytes, bytes)
    .split(bytes) -> list[bytes]
    .startswith(bytes) -> bool
    .reverse() -> None
    .rfind(bytes) -> int
    .rindex(bytes | int) -> int
    .rsplit(bytes) -> list[bytes]
    .translate(bytes, [bytes]) -> bytes

Note the conspicuous absence of .isupper(), .upper(), and friends. (But
see "Open Issues" below.) There is no .__hash__() because the object is
mutable. There is no use case for a .sort() method.

The bytes type also supports the buffer interface, supporting reading
and writing binary (but not character) data.

Out of Scope Issues

-   Python 3k will have a much different I/O subsystem. Deciding how
    that I/O subsystem will work and interact with the bytes object is
    out of the scope of this PEP. The expectation however is that binary
    I/O will read and write bytes, while text I/O will read strings.
    Since the bytes type supports the buffer interface, the existing
    binary I/O operations in Python 2.6 will support bytes objects.
-   It has been suggested that a special method named .__bytes__() be
    added to the language to allow objects to be converted into byte
    arrays. This decision is out of scope.
-   A bytes literal of the form b"..." is also proposed. This is the
    subject of PEP 3112.

Open Issues

-   The .decode() method is redundant since a bytes object b can also be
    decoded by calling unicode(b, <encoding>) (in 2.6) or
    str(b, <encoding>) (in 3.0). Do we need encode/decode methods at
    all? In a sense the spelling using a constructor is cleaner.
-   Need to specify the methods still more carefully.
-   Pickling and marshalling support need to be specified.
-   Should all those list methods really be implemented?
-   A case could be made for supporting .ljust(), .rjust(), .center()
    with a mandatory second argument.
-   A case could be made for supporting .split() with a mandatory
    argument.
-   A case could even be made for supporting .islower(), .isupper(),
    .isspace(), .isalpha(), .isalnum(), .isdigit() and the corresponding
    conversions (.lower() etc.), using the ASCII definitions for
    letters, digits and whitespace. If this is accepted, the cases for
    .ljust(), .rjust(), .center() and .split() become much stronger, and
    they should have default arguments as well, using an ASCII space or
    all ASCII whitespace (for .split()).

Frequently Asked Questions

Q: Why have the optional encoding argument when the encode method of
Unicode objects does the same thing?

A: In the current version of Python, the encode method returns a str
object and we cannot change that without breaking code. The construct
bytes(s.encode(...)) is expensive because it has to copy the byte
sequence multiple times. Also, Python generally provides two ways of
converting an object of type A into an object of type B: ask an A
instance to convert itself to a B, or ask the type B to create a new
instance from an A. Depending on what A and B are, both APIs make sense;
sometimes reasons of decoupling require that A can't know about B, in
which case you have to use the latter approach; sometimes B can't know
about A, in which case you have to use the former.

Q: Why does bytes ignore the encoding argument if the initializer is a
str? (This only applies to 2.6.)

A: There is no sane meaning that the encoding can have in that case. str
objects are byte arrays and they know nothing about the encoding of
character data they contain. We need to assume that the programmer has
provided a str object that already uses the desired encoding. If you
need something other than a pure copy of the bytes then you need to
first decode the string. For example:

    bytes(s.decode(encoding1), encoding2)

Q: Why not have the encoding argument default to Latin-1 (or some other
encoding that covers the entire byte range) rather than ASCII?

A: The system default encoding for Python is ASCII. It seems least
confusing to use that default. Also, in Py3k, using Latin-1 as the
default might not be what users expect. For example, they might prefer a
Unicode encoding. Any default will not always work as expected. At least
ASCII will complain loudly if you try to encode non-ASCII data.

Copyright

This document has been placed in the public domain.

authors:
Guido van Rossum
Neil Schemenauer


text:
PEP: 3140 Title: str(container) should call str(item), not repr(item)
Version: $Revision$ Last-Modified: $Date$ Author: Oleg Broytman
<EMAIL_ADDRESS>Jim J. Jewett<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 27-May-2008 Post-History: 28-May-2008

Rejection

Guido said this would cause too much disturbance too close to beta.
See[1].

Abstract

This document discusses the advantages and disadvantages of the current
implementation of str(container). It also discusses the pros and cons of
a different approach - to call str(item) instead of repr(item).

Motivation

Currently str(container) calls repr on items. Arguments for it:

-   containers refuse to guess what the user wants to see on
    str(container) - surroundings, delimiters, and so on;
-   repr(item) usually displays type information - apostrophes around
    strings, class names, etc.

Arguments against:

-   it's illogical; str() is expected to call __str__ if it exists, not
    __repr__;
-   there is no standard way to print a container's content calling
    items' __str__, that's inconvenient in cases where __str__ and
    __repr__ return different results;
-   repr(item) sometimes do wrong things (hex-escapes non-ASCII strings,
    e.g.)

This PEP proposes to change how str(container) works. It is proposed to
mimic how repr(container) works except one detail - call str on items
instead of repr. This allows a user to choose what results she want to
get - from item.__repr__ or item.__str__.

Current situation

Most container types (tuples, lists, dicts, sets, etc.) do not implement
__str__ method, so str(container) calls container.__repr__, and
container.__repr__, once called, forgets it is called from str and
always calls repr on the container's items.

This behaviour has advantages and disadvantages. One advantage is that
most items are represented with type information - strings are
surrounded by apostrophes, instances may have both class name and
instance data:

    >>> print([42, '42'])
    [42, '42']
    >>> print([Decimal('42'), datetime.now()])
    [Decimal("42"), datetime.datetime(2008, 5, 27, 19, 57, 43, 485028)]

The disadvantage is that __repr__ often returns technical data (like
'<object at address>') or unreadable string (hex-encoded string if the
input is non-ASCII string):

    >>> print(['тест'])
    ['\xd4\xc5\xd3\xd4']

One of the motivations for PEP 3138 is that neither repr nor str will
allow the sensible printing of dicts whose keys are non-ASCII text
strings. Now that Unicode identifiers are allowed, it includes Python's
own attribute dicts. This also includes JSON serialization (and caused
some hoops for the json lib).

PEP 3138 proposes to fix this by breaking the "repr is safe ASCII"
invariant, and changing the way repr (which is used for persistence)
outputs some objects, with system-dependent failures.

Changing how str(container) works would allow easy debugging in the
normal case, and retain the safety of ASCII-only for the
machine-readable case. The only downside is that str(x) and repr(x)
would more often be different -- but only in those cases where the
current almost-the-same version is insufficient.

It also seems illogical that str(container) calls repr on items instead
of str. It's only logical to expect following code:

    class Test:
        def __str__(self):
            return "STR"

        def __repr__(self):
            return "REPR"

    test = Test()
    print(test)
    print(repr(test))
    print([test])
    print(str([test]))

to print:

    STR
    REPR
    [STR]
    [STR]

where it actually prints:

    STR
    REPR
    [REPR]
    [REPR]

Especially it is illogical to see that print in Python 2 uses str if it
is called on what seems to be a tuple:

    >>> print Decimal('42'), datetime.now()
    42 2008-05-27 20:16:22.534285

where on an actual tuple it prints:

    >>> print((Decimal('42'), datetime.now()))
    (Decimal("42"), datetime.datetime(2008, 5, 27, 20, 16, 27, 937911))

A different approach - call str(item)

For example, with numbers it is often only the value that people care
about.

    >>> print Decimal('3')
    3

But putting the value in a list forces users to read the type
information, exactly as if repr had been called for the benefit of a
machine:

    >>> print [Decimal('3')]
    [Decimal("3")]

After this change, the type information would not clutter the str
output:

    >>> print "%s".format([Decimal('3')])
    [3]
    >>> str([Decimal('3')])  # ==
    [3]

But it would still be available if desired:

    >>> print "%r".format([Decimal('3')])
    [Decimal('3')]
    >>> repr([Decimal('3')])  # ==
    [Decimal('3')]

There is a number of strategies to fix the problem. The most radical is
to change __repr__ so it accepts a new parameter (flag) "called from
str, so call str on items, not repr". The drawback of the proposal is
that every __repr__ implementation must be changed. Introspection could
help a bit (inspect __repr__ before calling if it accepts 2 or 3
parameters), but introspection doesn't work on classes written in C,
like all built-in containers.

Less radical proposal is to implement __str__ methods for built-in
container types. The obvious drawback is a duplication of effort - all
those __str__ and __repr__ implementations are only differ in one small
detail - if they call str or repr on items.

The most conservative proposal is not to change str at all but to allow
developers to implement their own application- or library-specific
pretty-printers. The drawback is again a multiplication of effort and
proliferation of many small specific container-traversal algorithms.

Backward compatibility

In those cases where type information is more important than usual, it
will still be possible to get the current results by calling repr
explicitly.

References

Copyright

This document has been placed in the public domain.

[1] Guido van Rossum, PEP: str(container) should call str(item), not
repr(item)
https://mail.python.org/pipermail/python-3000/2008-May/013876.html


text:
PEP: 3137 Title: Immutable Bytes and Mutable Buffer Version: $Revision$
Last-Modified: $Date$ Author: Guido van Rossum<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
26-Sep-2007 Python-Version: 3.0 Post-History: 26-Sep-2007, 30-Sep-2007

Introduction

After releasing Python 3.0a1 with a mutable bytes type, pressure mounted
to add a way to represent immutable bytes. Gregory P. Smith proposed a
patch that would allow making a bytes object temporarily immutable by
requesting that the data be locked using the new buffer API from PEP
3118. This did not seem the right approach to me.

Jeffrey Yasskin, with the help of Adam Hupp, then prepared a patch to
make the bytes type immutable (by crudely removing all mutating APIs)
and fix the fall-out in the test suite. This showed that there aren't
all that many places that depend on the mutability of bytes, with the
exception of code that builds up a return value from small pieces.

Thinking through the consequences, and noticing that using the array
module as an ersatz mutable bytes type is far from ideal, and recalling
a proposal put forward earlier by Talin, I floated the suggestion to
have both a mutable and an immutable bytes type. (This had been brought
up before, but until seeing the evidence of Jeffrey's patch I wasn't
open to the suggestion.)

Moreover, a possible implementation strategy became clear: use the old
PyString implementation, stripped down to remove locale support and
implicit conversions to/from Unicode, for the immutable bytes type, and
keep the new PyBytes implementation as the mutable bytes type.

The ensuing discussion made it clear that the idea is welcome but needs
to be specified more precisely. Hence this PEP.

Advantages

One advantage of having an immutable bytes type is that code objects can
use these. It also makes it possible to efficiently create hash tables
using bytes for keys; this may be useful when parsing protocols like
HTTP or SMTP which are based on bytes representing text.

Porting code that manipulates binary data (or encoded text) in Python
2.x will be easier using the new design than using the original 3.0
design with mutable bytes; simply replace str with bytes and change
'...' literals into b'...' literals.

Naming

I propose the following type names at the Python level:

-   bytes is an immutable array of bytes (PyString)
-   bytearray is a mutable array of bytes (PyBytes)
-   memoryview is a bytes view on another object (PyMemory)

The old type named buffer is so similar to the new type memoryview,
introduce by PEP 3118, that it is redundant. The rest of this PEP
doesn't discuss the functionality of memoryview; it is just mentioned
here to justify getting rid of the old buffer type. (An earlier version
of this PEP proposed buffer as the new name for PyBytes; in the end this
name was deemed to confusing given the many other uses of the word
buffer.)

While eventually it makes sense to change the C API names, this PEP
maintains the old C API names, which should be familiar to all.

Summary

Here's a simple ASCII-art table summarizing the type names in various
Python versions:

    +--------------+-------------+------------+--------------------------+
    | C name       | 2.x    repr | 3.0a1 repr | 3.0a2               repr |
    +--------------+-------------+------------+--------------------------+
    | PyUnicode    | unicode u'' | str     '' | str                   '' |
    | PyString     | str      '' | str8   s'' | bytes                b'' |
    | PyBytes      | N/A         | bytes  b'' | bytearray bytearray(b'') |
    | PyBuffer     | buffer      | buffer     | N/A                      |
    | PyMemoryView | N/A         | memoryview | memoryview         <...> |
    +--------------+-------------+------------+--------------------------+

Literal Notations

The b'...' notation introduced in Python 3.0a1 returns an immutable
bytes object, whatever variation is used. To create a mutable array of
bytes, use bytearray(b'...') or bytearray([...]). The latter form takes
a list of integers in range(256).

Functionality

PEP 3118 Buffer API

Both bytes and bytearray implement the PEP 3118 buffer API. The bytes
type only implements read-only requests; the bytearray type allows
writable and data-locked requests as well. The element data type is
always 'B' (i.e. unsigned byte).

Constructors

There are four forms of constructors, applicable to both bytes and
bytearray:

-   bytes(<bytes>), bytes(<bytearray>), bytearray(<bytes>),
    bytearray(<bytearray>): simple copying constructors, with the note
    that bytes(<bytes>) might return its (immutable) argument, but
    bytearray(<bytearray>) always makes a copy.
-   bytes(<str>, <encoding>[, <errors>]),
    bytearray(<str>, <encoding>[, <errors>]): encode a text string. Note
    that the str.encode() method returns an immutable bytes object. The
    <encoding> argument is mandatory; <errors> is optional. <encoding>
    and <errors>, if given, must be str instances.
-   bytes(<memory view>), bytearray(<memory view>): construct a bytes or
    bytearray object from anything that implements the PEP 3118 buffer
    API.
-   bytes(<iterable of ints>), bytearray(<iterable of ints>): construct
    a bytes or bytearray object from a stream of integers in range(256).
-   bytes(<int>), bytearray(<int>): construct a zero-initialized bytes
    or bytearray object of a given length.

Comparisons

The bytes and bytearray types are comparable with each other and
orderable, so that e.g. b'abc' == bytearray(b'abc') < b'abd'.

Comparing either type to a str object for equality returns False
regardless of the contents of either operand. Ordering comparisons with
str raise TypeError. This is all conformant to the standard rules for
comparison and ordering between objects of incompatible types.

(Note: in Python 3.0a1, comparing a bytes instance with a str instance
would raise TypeError, on the premise that this would catch the
occasional mistake quicker, especially in code ported from Python 2.x.
However, a long discussion on the python-3000 list pointed out so many
problems with this that it is clearly a bad idea, to be rolled back in
3.0a2 regardless of the fate of the rest of this PEP.)

Slicing

Slicing a bytes object returns a bytes object. Slicing a bytearray
object returns a bytearray object.

Slice assignment to a bytearray object accepts anything that implements
the PEP 3118 buffer API, or an iterable of integers in range(256).

Indexing

Indexing bytes and bytearray returns small ints (like the bytes type in
3.0a1, and like lists or array.array('B')).

Assignment to an item of a bytearray object accepts an int in
range(256). (To assign from a bytes sequence, use a slice assignment.)

Str() and Repr()

The str() and repr() functions return the same thing for these objects.
The repr() of a bytes object returns a b'...' style literal. The repr()
of a bytearray returns a string of the form "bytearray(b'...')".

Operators

The following operators are implemented by the bytes and bytearray
types, except where mentioned:

-   b1 + b2: concatenation. With mixed bytes/bytearray operands, the
    return type is that of the first argument (this seems arbitrary
    until you consider how += works).
-   b1 += b2: mutates b1 if it is a bytearray object.
-   b * n, n * b: repetition; n must be an integer.
-   b *= n: mutates b if it is a bytearray object.
-   b1 in b2, b1 not in b2: substring test; b1 can be any object
    implementing the PEP 3118 buffer API.
-   i in b, i not in b: single-byte membership test; i must be an
    integer (if it is a length-1 bytes array, it is considered to be a
    substring test, with the same outcome).
-   len(b): the number of bytes.
-   hash(b): the hash value; only implemented by the bytes type.

Note that the % operator is not implemented. It does not appear worth
the complexity.

Methods

The following methods are implemented by bytes as well as bytearray,
with similar semantics. They accept anything that implements the PEP
3118 buffer API for bytes arguments, and return the same type as the
object whose method is called ("self"):

    .capitalize(), .center(), .count(), .decode(), .endswith(),
    .expandtabs(), .find(), .index(), .isalnum(), .isalpha(), .isdigit(),
    .islower(), .isspace(), .istitle(), .isupper(), .join(), .ljust(),
    .lower(), .lstrip(), .partition(), .replace(), .rfind(), .rindex(),
    .rjust(), .rpartition(), .rsplit(), .rstrip(), .split(),
    .splitlines(), .startswith(), .strip(), .swapcase(), .title(),
    .translate(), .upper(), .zfill()

This is exactly the set of methods present on the str type in Python
2.x, with the exclusion of .encode(). The signatures and semantics are
the same too. However, whenever character classes like letter,
whitespace, lower case are used, the ASCII definitions of these classes
are used. (The Python 2.x str type uses the definitions from the current
locale, settable through the locale module.) The .encode() method is
left out because of the more strict definitions of encoding and decoding
in Python 3000: encoding always takes a Unicode string and returns a
bytes sequence, and decoding always takes a bytes sequence and returns a
Unicode string.

In addition, both types implement the class method .fromhex(), which
constructs an object from a string containing hexadecimal values (with
or without spaces between the bytes).

The bytearray type implements these additional methods from the
MutableSequence ABC (see PEP 3119):

  .extend(), .insert(), .append(), .reverse(), .pop(), .remove().

Bytes and the Str Type

Like the bytes type in Python 3.0a1, and unlike the relationship between
str and unicode in Python 2.x, attempts to mix bytes (or bytearray)
objects and str objects without specifying an encoding will raise a
TypeError exception. (However, comparing bytes/bytearray and str objects
for equality will simply return False; see the section on Comparisons
above.)

Conversions between bytes or bytearray objects and str objects must
always be explicit, using an encoding. There are two equivalent APIs:
str(b, <encoding>[, <errors>]) is equivalent to
b.decode(<encoding>[, <errors>]), and bytes(s, <encoding>[, <errors>])
is equivalent to s.encode(<encoding>[, <errors>]).

There is one exception: we can convert from bytes (or bytearray) to str
without specifying an encoding by writing str(b). This produces the same
result as repr(b). This exception is necessary because of the general
promise that any object can be printed, and printing is just a special
case of conversion to str. There is however no promise that printing a
bytes object interprets the individual bytes as characters (unlike in
Python 2.x).

The str type currently implements the PEP 3118 buffer API. While this is
perhaps occasionally convenient, it is also potentially confusing,
because the bytes accessed via the buffer API represent a
platform-depending encoding: depending on the platform byte order and a
compile-time configuration option, the encoding could be UTF-16-BE,
UTF-16-LE, UTF-32-BE, or UTF-32-LE. Worse, a different implementation of
the str type might completely change the bytes representation, e.g. to
UTF-8, or even make it impossible to access the data as a contiguous
array of bytes at all. Therefore, the PEP 3118 buffer API will be
removed from the str type.

The basestring Type

The basestring type will be removed from the language. Code that used to
say isinstance(x, basestring) should be changed to use
isinstance(x, str) instead.

Pickling

Left as an exercise for the reader.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: