text:
PEP: 102 Title: Doing Python Micro Releases Author: Anthony Baxter
<EMAIL_ADDRESS>Barry Warsaw<EMAIL_ADDRESS>Guido van
Rossum<EMAIL_ADDRESS>Status: Superseded Type: Informational
Content-Type: text/x-rst Created: 09-Jan-2002 Post-History:
Superseded-By: 101

Replacement Note

Although the size of the to-do list in this PEP is much less scary than
that in PEP 101, it turns out not to be enough justification for the
duplication of information, and with it, the danger of one of the copies
to become out of date. Therefore, this PEP is not maintained anymore,
and micro releases are fully covered by PEP 101.

Abstract

Making a Python release is an arduous process that takes a minimum of
half a day's work even for an experienced releaser. Until recently, most
-- if not all -- of that burden was borne by Guido himself. But several
recent releases have been performed by other folks, so this PEP attempts
to collect, in one place, all the steps needed to make a Python bugfix
release.

The major Python release process is covered in PEP 101 - this PEP is
just PEP 101, trimmed down to only include the bits that are relevant
for micro releases, a.k.a. patch, or bug fix releases.

It is organized as a recipe and you can actually print this out and
check items off as you complete them.

How to Make A Release

Here are the steps taken to make a Python release. Some steps are more
fuzzy than others because there's little that can be automated (e.g.
writing the NEWS entries). Where a step is usually performed by An
Expert, the name of that expert is given. Otherwise, assume the step is
done by the Release Manager (RM), the designated person performing the
release. Almost every place the RM is mentioned below, this step can
also be done by the BDFL of course!

XXX: We should include a dependency graph to illustrate the steps that
can be taken in parallel, or those that depend on other steps.

We use the following conventions in the examples below. Where a release
number is given, it is of the form X.Y.MaA, e.g. 2.1.2c1 for Python
2.1.2 release candidate 1, where "a" == alpha, "b" == beta, "c" ==
release candidate. Final releases are tagged with "releaseXYZ" in CVS.
The micro releases are made from the maintenance branch of the major
release, e.g. Python 2.1.2 is made from the release21-maint branch.

1.  Send an email to<EMAIL_ADDRESS>indicating the release is
    about to start.

2.  Put a freeze on check ins into the maintenance branch. At this
    point, nobody except the RM should make any commits to the branch
    (or his duly assigned agents, i.e. Guido the BDFL, Fred Drake for
    documentation, or Thomas Heller for Windows). If the RM screwed up
    and some desperate last minute change to the branch is necessary, it
    can mean extra work for Fred and Thomas. So try to avoid this!

3.  On the branch, change Include/patchlevel.h in two places, to reflect
    the new version number you've just created. You'll want to change
    the PY_VERSION macro, and one or several of the version subpart
    macros just above PY_VERSION, as appropriate.

4.  Change the "%define version" line of Misc/RPM/python-2.3.spec to the
    same string as PY_VERSION was changed to above. E.g:

        %define version 2.3.1

    You also probably want to reset the %define release line to
    '1pydotorg' if it's not already that.

5.  If you're changing the version number for Python (e.g. from Python
    2.1.1 to Python 2.1.2), you also need to update the README file,
    which has a big banner at the top proclaiming its identity. Don't do
    this if you're just releasing a new alpha or beta release, but /do/
    do this if you're release a new micro, minor or major release.

6.  The LICENSE file also needs to be changed, due to several references
    to the release number. As for the README file, changing these are
    necessary for a new micro, minor or major release.

    The LICENSE file contains a table that describes the legal heritage
    of Python; you should add an entry for the X.Y.Z release you are now
    making. You should update this table in the LICENSE file on the CVS
    trunk too.

7.  When the year changes, copyright legends need to be updated in many
    places, including the README and LICENSE files.

8.  For the Windows build, additional files have to be updated.

    PCbuild/BUILDno.txt contains the Windows build number, see the
    instructions in this file how to change it. Saving the project file
    PCbuild/pythoncore.dsp results in a change to PCbuild/pythoncore.dsp
    as well.

    PCbuild/python20.wse sets up the Windows installer version resource
    (displayed when you right-click on the installer .exe and select
    Properties), and also contains the Python version number.

    (Before version 2.3.2, it was required to manually edit
    PC/python_nt.rc, this step is now automated by the build process.)

9.  After starting the process, the most important thing to do next is
    to update the Misc/NEWS file. Thomas will need this in order to do
    the Windows release and he likes to stay up late. This step can be
    pretty tedious, so it's best to get to it immediately after making
    the branch, or even before you've made the branch. The sooner the
    better (but again, watch for new checkins up until the release is
    made!)

    Add high level items new to this release. E.g. if we're releasing
    2.2a3, there must be a section at the top of the file explaining
    "What's new in Python 2.2a3". It will be followed by a section
    entitled "What's new in Python 2.2a2".

    Note that you /hope/ that as developers add new features to the
    trunk, they've updated the NEWS file accordingly. You can't be
    positive, so double check. If you're a Unix weenie, it helps to
    verify with Thomas about changes on Windows, and Jack Jansen about
    changes on the Mac.

    This command should help you (but substitute the correct -r tag!):

        % cvs log -rr22a1: | python Tools/scripts/logmerge.py > /tmp/news.txt

    IOW, you're printing out all the cvs log entries from the previous
    release until now. You can then troll through the news.txt file
    looking for interesting things to add to NEWS.

10. Check your NEWS changes into the maintenance branch. It's easy to
    forget to update the release date in this file!

11. Check in any changes to IDLE's NEWS.txt. Update the header in
    Lib/idlelib/NEWS.txt to reflect its release version and date. Update
    the IDLE version in Lib/idlelib/idlever.py to match.

12. Once the release process has started, the documentation needs to be
    built and posted on python.org according to the instructions in PEP
    101.

    Note that Fred is responsible both for merging doc changes from the
    trunk to the branch AND for merging any branch changes from the
    branch to the trunk during the cleaning up phase. Basically, if it's
    in Doc/ Fred will take care of it.

13. Thomas compiles everything with MSVC 6.0 SP5, and moves the
    python23.chm file into the src/chm directory. The installer
    executable is then generated with Wise Installation System.

    The installer includes the MSVC 6.0 runtime in the files MSVCRT.DLL
    and MSVCIRT.DLL. It leads to disaster if these files are taken from
    the system directory of the machine where the installer is built,
    instead it must be absolutely made sure that these files come from
    the VCREDIST.EXE redistributable package contained in the MSVC
    SP5 CD. VCREDIST.EXE must be unpacked with winzip, and the Wise
    Installation System prompts for the directory.

    After building the installer, it should be opened with winzip, and
    the MS dlls extracted again and check for the same version number as
    those unpacked from VCREDIST.EXE.

    Thomas uploads this file to the starship. He then sends the RM a
    notice which includes the location and MD5 checksum of the Windows
    executable.

    Note that Thomas's creation of the Windows executable may generate a
    few more commits on the branch. Thomas will be responsible for
    merging Windows-specific changes from trunk to branch, and from
    branch to trunk.

14. Sean performs his Red Hat magic, generating a set of RPMs. He
    uploads these files to python.org. He then sends the RM a notice
    which includes the location and MD5 checksum of the RPMs.

15. It's Build Time!

    Now, you're ready to build the source tarball. First cd to your
    working directory for the branch. E.g. % cd .../python-22a3

16. Do a "cvs update" in this directory. Do NOT include the -A flag!

    You should not see any "M" files, but you may see several "P" and/or
    "U" files. I.e. you better not have any uncommitted changes in your
    working directory, but you may pick up some of Fred's or Thomas's
    last minute changes.

17. Now tag the branch using a symbolic name like "rXYMaZ", e.g. r212

        % cvs tag r212

    Be sure to tag only the python/dist/src subdirectory of the Python
    CVS tree!

18. Change to a neutral directory, i.e. one in which you can do a fresh,
    virgin, cvs export of the branch. You will be creating a new
    directory at this location, to be named "Python-X.Y.M". Do a CVS
    export of the tagged branch.

        % cd ~
        % cvs -d cvs.sf.net:/cvsroot/python export -rr212 \
                              -d Python-2.1.2 python/dist/src

19. Generate the tarball. Note that we're not using the 'z' option on
    the tar command because 1) that's only supported by GNU tar as far
    as we know, and 2) we're going to max out the compression level,
    which isn't a supported option. We generate both tar.gz tar.bz2
    formats, as the latter is about 1/6th smaller.

        % tar -cf - Python-2.1.2 | gzip -9 > Python-2.1.2.tgz
        % tar -cf - Python-2.1.2 | bzip2 -9 > Python-2.1.2.tar.bz2

20. Calculate the MD5 checksum of the tgz and tar.bz2 files you just
    created

        % md5sum Python-2.1.2.tgz

    Note that if you don't have the md5sum program, there is a Python
    replacement in the Tools/scripts/md5sum.py file.

21. Create GPG keys for each of the files.

        % gpg -ba Python-2.1.2.tgz
        % gpg -ba Python-2.1.2.tar.bz2
        % gpg -ba Python-2.1.2.exe

22. Now you want to perform the very important step of checking the
    tarball you just created, to make sure a completely clean, virgin
    build passes the regression test. Here are the best steps to take:

        % cd /tmp
        % tar zxvf ~/Python-2.1.2.tgz
        % cd Python-2.1.2
        % ls
        (Do things look reasonable?)
        % ./configure
        (Loads of configure output)
        % make test
        (Do all the expected tests pass?)

    If the tests pass, then you can feel good that the tarball is fine.
    If some of the tests fail, or anything else about the freshly
    unpacked directory looks weird, you better stop now and figure out
    what the problem is.

23. You need to upload the tgz and the exe file to creosote.python.org.
    This step can take a long time depending on your network bandwidth.
    scp both files from your own machine to creosote.

24. While you're waiting, you can start twiddling the web pages to
    include the announcement.

    1.  In the top of the python.org web site CVS tree, create a
        subdirectory for the X.Y.Z release. You can actually copy an
        earlier patch release's subdirectory, but be sure to delete the
        X.Y.Z/CVS directory and "cvs add X.Y.Z", for example:

            % cd .../pydotorg
            % cp -r 2.2.2 2.2.3
            % rm -rf 2.2.3/CVS
            % cvs add 2.2.3
            % cd 2.2.3

    2.  Edit the files for content: usually you can globally replace
        X.Ya(Z-1) with X.YaZ. However, you'll need to think about the
        "What's New?" section.

    3.  Copy the Misc/NEWS file to NEWS.txt in the X.Y.Z directory for
        python.org; this contains the "full scoop" of changes to Python
        since the previous release for this version of Python.

    4.  Copy the .asc GPG signatures you created earlier here as well.

    5.  Also, update the MD5 checksums.

    6.  Preview the web page by doing a "make" or "make install" (as
        long as you've created a new directory for this release!)

    7.  Similarly, edit the ../index.ht file, i.e. the python.org home
        page. In the Big Blue Announcement Block, move the paragraph for
        the new version up to the top and boldify the phrase "Python
        X.YaZ is out". Edit for content, and preview locally, but do NOT
        do a "make install" yet!

25. Now we're waiting for the scp to creosote to finish. Da de da, da de
    dum, hmm, hmm, dum de dum.

26. Once that's done you need to go to creosote.python.org and move all
    the files in place over there. Our policy is that every Python
    version gets its own directory, but each directory may contain
    several releases. We keep all old releases, moving them into a
    "prev" subdirectory when we have a new release.

    So, there's a directory called "2.2" which contains Python-2.2a2.exe
    and Python-2.2a2.tgz, along with a "prev" subdirectory containing
    Python-2.2a1.exe and Python-2.2a1.tgz.

    So...

    1.  On creosote, cd to ~ftp/pub/python/X.Y creating it if necessary.
    2.  Move the previous release files to a directory called "prev"
        creating the directory if necessary (make sure the directory has
        g+ws bits on). If this is the first alpha release of a new
        Python version, skip this step.
    3.  Move the .tgz file and the .exe file to this directory. Make
        sure they are world readable. They should also be group
        writable, and group-owned by webmaster.
    4.  md5sum the files and make sure they got uploaded intact.

27. the X.Y/bugs.ht file if necessary. It is best to get BDFL input for
    this step.

28. Go up to the parent directory (i.e. the root of the web page
    hierarchy) and do a "make install" there. You're release is now
    live!

29. Now it's time to write the announcement for the mailing lists. This
    is the fuzzy bit because not much can be automated. You can use one
    of Guido's earlier announcements as a template, but please edit it
    for content!

    Once the announcement is ready, send it to the following addresses:

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
30. Send a SourceForge News Item about the release. From the project's
    "menu bar", select the "News" link; once in News, select the
    "Submit" link. Type a suitable subject (e.g. "Python 2.2c1 released"
    :-) in the Subject box, add some text to the Details box (at the
    very least including the release URL at www.python.org and the fact
    that you're happy with the release) and click the SUBMIT button.

    Feel free to remove any old news items.

Now it's time to do some cleanup. These steps are very important!

1.  Edit the file Include/patchlevel.h so that the PY_VERSION string
    says something like "X.YaZ+". Note the trailing '+' indicating that
    the trunk is going to be moving forward with development. E.g. the
    line should look like:

        #define PY_VERSION              "2.1.2+"

    Make sure that the other PY_ version macros contain the correct
    values. Commit this change.

2.  For the extra paranoid, do a completely clean test of the release.
    This includes downloading the tarball from www.python.org.

3.  Make sure the md5 checksums match. Then unpack the tarball, and do a
    clean make test.

        % make distclean
        % ./configure
        % make test

    To ensure that the regression test suite passes. If not, you screwed
    up somewhere!

Step 5 ...

Verify! This can be interleaved with Step 4. Pretend you're a user:
download the files from python.org, and make Python from it. This step
is too easy to overlook, and on several occasions we've had useless
release files. Once a general server problem caused mysterious
corruption of all files; once the source tarball got built incorrectly;
more than once the file upload process on SF truncated files; and so on.

What Next?

Rejoice. Drink. Be Merry. Write a PEP like this one. Or be like unto
Guido and take A Vacation.

You've just made a Python release!

Actually, there is one more step. You should turn over ownership of the
branch to Jack Jansen. All this means is that now he will be responsible
for making commits to the branch. He's going to use this to build the
MacOS versions. He may send you information about the Mac release that
should be merged into the informational pages on www.python.org. When
he's done, he'll tag the branch something like "rX.YaZ-mac". He'll also
be responsible for merging any Mac-related changes back into the trunk.

Final Release Notes

The Final release of any major release, e.g. Python 2.2 final, has
special requirements, specifically because it will be one of the longest
lived releases (i.e. betas don't last more than a couple of weeks, but
final releases can last for years!).

For this reason we want to have a higher coordination between the three
major releases: Windows, Mac, and source. The Windows and source
releases benefit from the close proximity of the respective
release-bots. But the Mac-bot, Jack Jansen, is 6 hours away. So we add
this extra step to the release process for a final release:

1.  Hold up the final release until Jack approves, or until we lose
    patience <wink>.

The python.org site also needs some tweaking when a new bugfix release
is issued.

2.  The documentation should be installed at doc/<version>/.
3.  Add a link from doc/<previous-minor-release>/index.ht to the
    documentation for the new version.
4.  All older doc/<old-release>/index.ht files should be updated to
    point to the documentation for the new version.
5.  /robots.txt should be modified to prevent the old version's
    documentation from being crawled by search engines.

Windows Notes

Windows has a GUI installer, various flavors of Windows have "special
limitations", and the Windows installer also packs precompiled "foreign"
binaries (Tcl/Tk, expat, etc). So Windows testing is tiresome but very
necessary.

Concurrent with uploading the installer, Thomas installs Python from it
twice: once into the default directory suggested by the installer, and
later into a directory with embedded spaces in its name. For each
installation, he runs the full regression suite from a DOS box, and both
with and without -0.

He also tries every shortcut created under Start -> Menu -> the Python
group. When trying IDLE this way, you need to verify that Help -> Python
Documentation works. When trying pydoc this way (the "Module Docs" Start
menu entry), make sure the "Start Browser" button works, and make sure
you can search for a random module (Thomas uses "random" <wink>) and
then that the "go to selected" button works.

It's amazing how much can go wrong here -- and even more amazing how
often last-second checkins break one of these things. If you're "the
Windows geek", keep in mind that you're likely the only person routinely
testing on Windows, and that Windows is simply a mess.

Repeat all of the above on at least one flavor of Win9x, and one of
NT/2000/XP. On NT/2000/XP, try both an Admin and a plain User (not Power
User) account.

WRT Step 5 above (verify the release media), since by the time release
files are ready to download Thomas has generally run many Windows tests
on the installer he uploaded, he usually doesn't do anything for Step 5
except a full byte-comparison ("fc /b" if using a Windows shell) of the
downloaded file against the file he uploaded.

Copyright

This document has been placed in the public domain.


text:
PEP: 214 Title: Extended Print Statement Author: Barry Warsaw
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 24-Jul-2000 Python-Version: 2.0 Post-History:
16-Aug-2000

Introduction

This PEP describes a syntax to extend the standard 'print' statement so
that it can be used to print to any file-like object, instead of the
default sys.stdout. This PEP tracks the status and ownership of this
feature. It contains a description of the feature and outlines changes
necessary to support the feature. This PEP summarizes discussions held
in mailing list forums, and provides URLs for further information, where
appropriate. The CVS revision history of this file contains the
definitive historical record.

Proposal

This proposal introduces a syntax extension to the print statement,
which allows the programmer to optionally specify the output file
target. An example usage is as follows:

    print >> mylogfile, 'this message goes to my log file'

Formally, the syntax of the extended print statement is:

    print_stmt: ... | '>>' test [ (',' test)+ [','] ] )

where the ellipsis indicates the original print_stmt syntax unchanged.
In the extended form, the expression just after >> must yield an object
with a write() method (i.e. a file-like object). Thus these two
statements are equivalent:

    print 'hello world'
    print >> sys.stdout, 'hello world'

As are these two statements:

    print
    print >> sys.stdout

These two statements are syntax errors:

    print ,
    print >> sys.stdout,

Justification

'print' is a Python keyword and introduces the print statement as
described in section 6.6 of the language reference manual[1]. The print
statement has a number of features:

-   it auto-converts the items to strings
-   it inserts spaces between items automatically
-   it appends a newline unless the statement ends in a comma

The formatting that the print statement performs is limited; for more
control over the output, a combination of sys.stdout.write(), and string
interpolation can be used.

The print statement by definition outputs to sys.stdout. More
specifically, sys.stdout must be a file-like object with a write()
method, but it can be rebound to redirect output to files other than
specifically standard output. A typical idiom is:

    save_stdout = sys.stdout
    try:
        sys.stdout = mylogfile
        print 'this message goes to my log file'
    finally:
        sys.stdout = save_stdout

The problem with this approach is that the binding is global, and so
affects every statement inside the try: clause. For example, if we added
a call to a function that actually did want to print to stdout, this
output too would get redirected to the logfile.

This approach is also very inconvenient for interleaving prints to
various output streams, and complicates coding in the face of legitimate
try/except or try/finally clauses.

Reference Implementation

A reference implementation, in the form of a patch against the Python
2.0 source tree, is available on SourceForge's patch manager[2]. This
approach adds two new opcodes, PRINT_ITEM_TO and PRINT_NEWLINE_TO, which
simply pop the file like object off the top of the stack and use it
instead of sys.stdout as the output stream.

(This reference implementation has been adopted in Python 2.0.)

Alternative Approaches

An alternative to this syntax change has been proposed (originally by
Moshe Zadka) which requires no syntax changes to Python. A writeln()
function could be provided (possibly as a builtin), that would act much
like extended print, with a few additional features:

    def writeln(*args, **kws):
        import sys
        file = sys.stdout
        sep = ' '
        end = '\n'
        if kws.has_key('file'):
            file = kws['file']
            del kws['file']
        if kws.has_key('nl'):
            if not kws['nl']:
                end = ' '
            del kws['nl']
        if kws.has_key('sep'):
            sep = kws['sep']
            del kws['sep']
        if kws:
            raise TypeError('unexpected keywords')
        file.write(sep.join(map(str, args)) + end)

writeln() takes a three optional keyword arguments. In the context of
this proposal, the relevant argument is 'file' which can be set to a
file-like object with a write() method. Thus:

    print >> mylogfile, 'this goes to my log file'

would be written as:

    writeln('this goes to my log file', file=mylogfile)

writeln() has the additional functionality that the keyword argument
'nl' is a flag specifying whether to append a newline or not, and an
argument 'sep' which specifies the separator to output in between each
item.

More Justification by the BDFL

The proposal has been challenged on the newsgroup. One series of
challenges doesn't like '>>' and would rather see some other symbol.

-   Challenge: Why not one of these?

        print in stderr items,....
        print + stderr items,.......
        print[stderr] items,.....
        print to stderr items,.....

    Response: If we want to use a special symbol (print <symbol>
    expression), the Python parser requires that it is not already a
    symbol that can start an expression -- otherwise it can't decide
    which form of print statement is used. (The Python parser is a
    simple LL(1) or recursive descent parser.)

    This means that we can't use the "keyword only in context trick"
    that was used for "import as", because an identifier can start an
    expression. This rules out +stderr, [sterr], and to stderr. It
    leaves us with binary operator symbols and other miscellaneous
    symbols that are currently illegal here, such as 'import'.

    If I had to choose between 'print in file' and 'print >> file' I
    would definitely choose '>>'. In part because 'in' would be a new
    invention (I know of no other language that uses it, while '>>' is
    used in sh, awk, Perl, and C++), in part because '>>', being
    non-alphabetic, stands out more so is more likely to catch the
    reader's attention.

-   Challenge: Why does there have to be a comma between the file and
    the rest?

    Response: The comma separating the file from the following
    expression is necessary! Of course you want the file to be an
    arbitrary expression, not just a single word. (You definitely want
    to be able to write print >>sys.stderr.) Without the expression the
    parser wouldn't be able to distinguish where that expression ends
    and where the next one begins, e.g.

        print >>i +1, 2
        print >>a [1], 2
        print >>f (1), 2

-   Challenge: Why do you need a syntax extension? Why not writeln(file,
    item, ...)?

    Response: First of all, this is lacking a feature of the print
    statement: the trailing comma to print which suppresses the final
    newline. Note that 'print a,' still isn't equivalent to
    'sys.stdout.write(a)' -- print inserts a space between items, and
    takes arbitrary objects as arguments; write() doesn't insert a space
    and requires a single string.

    When you are considering an extension for the print statement, it's
    not right to add a function or method that adds a new feature in one
    dimension (where the output goes) but takes away in another
    dimension (spaces between items, and the choice of trailing newline
    or not). We could add a whole slew of methods or functions to deal
    with the various cases but that seems to add more confusion than
    necessary, and would only make sense if we were to deprecate the
    print statement altogether.

    I feel that this debate is really about whether print should have
    been a function or method rather than a statement. If you are in the
    function camp, of course adding special syntax to the existing print
    statement is not something you like. I suspect the objection to the
    new syntax comes mostly from people who already think that the print
    statement was a bad idea. Am I right?

    About 10 years ago I debated with myself whether to make the most
    basic from of output a function or a statement; basically I was
    trying to decide between "print(item, ...)" and "print item, ...". I
    chose to make it a statement because printing needs to be taught
    very early on, and is very important in the programs that beginners
    write. Also, because ABC, which lead the way for so many things,
    made it a statement. In a move that's typical for the interaction
    between ABC and Python, I changed the name from WRITE to print, and
    reversed the convention for adding newlines from requiring extra
    syntax to add a newline (ABC used trailing slashes to indicate
    newlines) to requiring extra syntax (the trailing comma) to suppress
    the newline. I kept the feature that items are separated by
    whitespace on output.

    Full example: in ABC,

        WRITE 1
        WRITE 2/

    has the same effect as:

        print 1,
        print 2

    has in Python, outputting in effect "1 2n".

    I'm not 100% sure that the choice for a statement was right (ABC had
    the compelling reason that it used statement syntax for anything
    with side effects, but Python doesn't have this convention), but I'm
    also not convinced that it's wrong. I certainly like the economy of
    the print statement. (I'm a rabid Lisp-hater -- syntax-wise, not
    semantics-wise! -- and excessive parentheses in syntax annoy me.
    Don't ever write return(i) or if(x==y): in your Python code! :-)

    Anyway, I'm not ready to deprecate the print statement, and over the
    years we've had many requests for an option to specify the file.

-   Challenge: Why not > instead of >>?

    Response: To DOS and Unix users, >> suggests "append", while >
    suggests "overwrite"; the semantics are closest to append. Also, for
    C++ programmers, >> and << are I/O operators.

-   Challenge: But in C++, >> is input and << is output!

    Response: doesn't matter; C++ clearly took it from Unix and reversed
    the arrows. The important thing is that for output, the arrow points
    to the file.

-   Challenge: Surely you can design a println() function can do all
    what print>>file can do; why isn't that enough?

    Response: I think of this in terms of a simple programming exercise.
    Suppose a beginning programmer is asked to write a function that
    prints the tables of multiplication. A reasonable solution is:

        def tables(n):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print i, 'x', j, '=', i*j
                print

    Now suppose the second exercise is to add printing to a different
    file. With the new syntax, the programmer only needs to learn one
    new thing: print >> file, and the answer can be like this:

        def tables(n, file=sys.stdout):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print >> file, i, 'x', j, '=', i*j
                print >> file

    With only a print statement and a println() function, the programmer
    first has to learn about println(), transforming the original
    program to using println():

        def tables(n):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    println(i, 'x', j, '=', i*j)
                println()

    and then about the file keyword argument:

        def tables(n, file=sys.stdout):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    println(i, 'x', j, '=', i*j, file=sys.stdout)
                println(file=sys.stdout)

    Thus, the transformation path is longer:

        (1) print
        (2) print >> file

    vs.

        (1) print
        (2) println()
        (3) println(file=...)

    Note: defaulting the file argument to sys.stdout at compile time is
    wrong, because it doesn't work right when the caller assigns to
    sys.stdout and then uses tables() without specifying the file. This
    is a common problem (and would occur with a println() function too).
    The standard solution so far has been:

        def tables(n, file=None):
            if file is None:
                file = sys.stdout
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print >> file, i, 'x', j, '=', i*j
                print >> file

    I've added a feature to the implementation (which I would also
    recommend to println()) whereby if the file argument is None,
    sys.stdout is automatically used. Thus,

        print >> None, foo bar

    (or, of course, print >> x where x is a variable whose value is
    None) means the same as

        print foo, bar

    and the tables() function can be written as follows:

        def tables(n, file=None):
            for j in range(1, n+1):
                for i in range(1, n+1):
                    print >> file, i, 'x', j, '=', i*j
                print >> file

References

[1] http://docs.python.org/reference/simple_stmts.html#print

[2] http://sourceforge.net/patch/download.php?id=100970

license: Public Domain


text:
PEP: 343 Title: The "with" Statement Version: $Revision$ Last-Modified:
$Date$ Author: Guido van Rossum, Alyssa Coghlan Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 13-May-2005
Python-Version: 2.5 Post-History: 02-Jun-2005, 16-Oct-2005, 29-Oct-2005,
23-Apr-2006, 01-May-2006, 30-Jul-2006

Abstract

This PEP adds a new statement "with" to the Python language to make it
possible to factor out standard uses of try/finally statements.

In this PEP, context managers provide __enter__() and __exit__() methods
that are invoked on entry to and exit from the body of the with
statement.

Author's Note

This PEP was originally written in first person by Guido, and
subsequently updated by Alyssa (Nick) Coghlan to reflect later
discussion on python-dev. Any first person references are from Guido's
original.

Python's alpha release cycle revealed terminology problems in this PEP
and in the associated documentation and implementation[1]. The PEP
stabilised around the time of the first Python 2.5 beta release.

Yes, the verb tense is messed up in a few places. We've been working on
this PEP for over a year now, so things that were originally in the
future are now in the past :)

Introduction

After a lot of discussion about PEP 340 and alternatives, I decided to
withdraw PEP 340 and proposed a slight variant on PEP 310. After more
discussion, I have added back a mechanism for raising an exception in a
suspended generator using a throw() method, and a close() method which
throws a new GeneratorExit exception; these additions were first
proposed on python-dev in [2] and universally approved of. I'm also
changing the keyword to 'with'.

After acceptance of this PEP, the following PEPs were rejected due to
overlap:

-   PEP 310, Reliable Acquisition/Release Pairs. This is the original
    with-statement proposal.
-   PEP 319, Python Synchronize/Asynchronize Block. Its use cases can be
    covered by the current PEP by providing suitable with-statement
    controllers: for 'synchronize' we can use the "locking" template
    from example 1; for 'asynchronize' we can use a similar "unlocking"
    template. I don't think having an "anonymous" lock associated with a
    code block is all that important; in fact it may be better to always
    be explicit about the mutex being used.

PEP 340 and PEP 346 also overlapped with this PEP, but were voluntarily
withdrawn when this PEP was submitted.

Some discussion of earlier incarnations of this PEP took place on the
Python Wiki[3].

Motivation and Summary

PEP 340, Anonymous Block Statements, combined many powerful ideas: using
generators as block templates, adding exception handling and
finalization to generators, and more. Besides praise it received a lot
of opposition from people who didn't like the fact that it was, under
the covers, a (potential) looping construct. This meant that break and
continue in a block-statement would break or continue the
block-statement, even if it was used as a non-looping resource
management tool.

But the final blow came when I read Raymond Chen's rant about
flow-control macros[4]. Raymond argues convincingly that hiding flow
control in macros makes your code inscrutable, and I find that his
argument applies to Python as well as to C. I realized that PEP 340
templates can hide all sorts of control flow; for example, its example 4
(auto_retry()) catches exceptions and repeats the block up to three
times.

However, the with-statement of PEP 310 does not hide control flow, in my
view: while a finally-suite temporarily suspends the control flow, in
the end, the control flow resumes as if the finally-suite wasn't there
at all.

Remember, PEP 310 proposes roughly this syntax (the "VAR =" part is
optional):

    with VAR = EXPR:
        BLOCK

which roughly translates into this:

    VAR = EXPR
    VAR.__enter__()
    try:
        BLOCK
    finally:
        VAR.__exit__()

Now consider this example:

    with f = open("/etc/passwd"):
        BLOCK1
    BLOCK2

Here, just as if the first line was "if True" instead, we know that if
BLOCK1 completes without an exception, BLOCK2 will be reached; and if
BLOCK1 raises an exception or executes a non-local goto (a break,
continue or return), BLOCK2 is not reached. The magic added by the
with-statement at the end doesn't affect this.

(You may ask, what if a bug in the __exit__() method causes an
exception? Then all is lost -- but this is no worse than with other
exceptions; the nature of exceptions is that they can happen anywhere,
and you just have to live with that. Even if you write bug-free code, a
KeyboardInterrupt exception can still cause it to exit between any two
virtual machine opcodes.)

This argument almost led me to endorse PEP 310, but I had one idea left
from the PEP 340 euphoria that I wasn't ready to drop: using generators
as "templates" for abstractions like acquiring and releasing a lock or
opening and closing a file is a powerful idea, as can be seen by looking
at the examples in that PEP.

Inspired by a counter-proposal to PEP 340 by Phillip Eby I tried to
create a decorator that would turn a suitable generator into an object
with the necessary __enter__() and __exit__() methods. Here I ran into a
snag: while it wasn't too hard for the locking example, it was
impossible to do this for the opening example. The idea was to define
the template like this:

    @contextmanager
    def opening(filename):
        f = open(filename)
        try:
            yield f
        finally:
            f.close()

and used it like this:

    with f = opening(filename):
        ...read data from f...

The problem is that in PEP 310, the result of calling EXPR is assigned
directly to VAR, and then VAR's __exit__() method is called upon exit
from BLOCK1. But here, VAR clearly needs to receive the opened file, and
that would mean that __exit__() would have to be a method on the file.

While this can be solved using a proxy class, this is awkward and made
me realize that a slightly different translation would make writing the
desired decorator a piece of cake: let VAR receive the result from
calling the __enter__() method, and save the value of EXPR to call its
__exit__() method later. Then the decorator can return an instance of a
wrapper class whose __enter__() method calls the generator's next()
method and returns whatever next() returns; the wrapper instance's
__exit__() method calls next() again but expects it to raise
StopIteration. (Details below in the section Optional Generator
Decorator.)

So now the final hurdle was that the PEP 310 syntax:

    with VAR = EXPR:
        BLOCK1

would be deceptive, since VAR does not receive the value of EXPR.
Borrowing from PEP 340, it was an easy step to:

    with EXPR as VAR:
        BLOCK1

Additional discussion showed that people really liked being able to
"see" the exception in the generator, even if it was only to log it; the
generator is not allowed to yield another value, since the
with-statement should not be usable as a loop (raising a different
exception is marginally acceptable). To enable this, a new throw()
method for generators is proposed, which takes one to three arguments
representing an exception in the usual fashion (type, value, traceback)
and raises it at the point where the generator is suspended.

Once we have this, it is a small step to proposing another generator
method, close(), which calls throw() with a special exception,
GeneratorExit. This tells the generator to exit, and from there it's
another small step to proposing that close() be called automatically
when the generator is garbage-collected.

Then, finally, we can allow a yield-statement inside a try-finally
statement, since we can now guarantee that the finally-clause will
(eventually) be executed. The usual cautions about finalization apply --
the process may be terminated abruptly without finalizing any objects,
and objects may be kept alive forever by cycles or memory leaks in the
application (as opposed to cycles or leaks in the Python implementation,
which are taken care of by GC).

Note that we're not guaranteeing that the finally-clause is executed
immediately after the generator object becomes unused, even though this
is how it will work in CPython. This is similar to auto-closing files:
while a reference-counting implementation like CPython deallocates an
object as soon as the last reference to it goes away, implementations
that use other GC algorithms do not make the same guarantee. This
applies to Jython, IronPython, and probably to Python running on Parrot.

(The details of the changes made to generators can now be found in PEP
342 rather than in the current PEP)

Use Cases

See the Examples section near the end.

Specification: The 'with' Statement

A new statement is proposed with the syntax:

    with EXPR as VAR:
        BLOCK

Here, 'with' and 'as' are new keywords; EXPR is an arbitrary expression
(but not an expression-list) and VAR is a single assignment target. It
can not be a comma-separated sequence of variables, but it can be a
parenthesized comma-separated sequence of variables. (This restriction
makes a future extension possible of the syntax to have multiple
comma-separated resources, each with its own optional as-clause.)

The "as VAR" part is optional.

The translation of the above statement is:

    mgr = (EXPR)
    exit = type(mgr).__exit__  # Not calling it yet
    value = type(mgr).__enter__(mgr)
    exc = True
    try:
        try:
            VAR = value  # Only if "as VAR" is present
            BLOCK
        except:
            # The exceptional case is handled here
            exc = False
            if not exit(mgr, *sys.exc_info()):
                raise
            # The exception is swallowed if exit() returns true
    finally:
        # The normal and non-local-goto cases are handled here
        if exc:
            exit(mgr, None, None, None)

Here, the lowercase variables (mgr, exit, value, exc) are internal
variables and not accessible to the user; they will most likely be
implemented as special registers or stack positions.

The details of the above translation are intended to prescribe the exact
semantics. If either of the relevant methods are not found as expected,
the interpreter will raise AttributeError, in the order that they are
tried (__exit__, __enter__). Similarly, if any of the calls raises an
exception, the effect is exactly as it would be in the above code.
Finally, if BLOCK contains a break, continue or return statement, the
__exit__() method is called with three None arguments just as if BLOCK
completed normally. (I.e. these "pseudo-exceptions" are not seen as
exceptions by __exit__().)

If the "as VAR" part of the syntax is omitted, the "VAR =" part of the
translation is omitted (but mgr.__enter__() is still called).

The calling convention for mgr.__exit__() is as follows. If the
finally-suite was reached through normal completion of BLOCK or through
a non-local goto (a break, continue or return statement in BLOCK),
mgr.__exit__() is called with three None arguments. If the finally-suite
was reached through an exception raised in BLOCK, mgr.__exit__() is
called with three arguments representing the exception type, value, and
traceback.

IMPORTANT: if mgr.__exit__() returns a "true" value, the exception is
"swallowed". That is, if it returns "true", execution continues at the
next statement after the with-statement, even if an exception happened
inside the with-statement. However, if the with-statement was left via a
non-local goto (break, continue or return), this non-local return is
resumed when mgr.__exit__() returns regardless of the return value. The
motivation for this detail is to make it possible for mgr.__exit__() to
swallow exceptions, without making it too easy (since the default return
value, None, is false and this causes the exception to be re-raised).
The main use case for swallowing exceptions is to make it possible to
write the @contextmanager decorator so that a try/except block in a
decorated generator behaves exactly as if the body of the generator were
expanded in-line at the place of the with-statement.

The motivation for passing the exception details to __exit__(), as
opposed to the argument-less __exit__() from PEP 310, was given by the
transactional() use case, example 3 below. The template in that example
must commit or roll back the transaction depending on whether an
exception occurred or not. Rather than just having a boolean flag
indicating whether an exception occurred, we pass the complete exception
information, for the benefit of an exception-logging facility for
example. Relying on sys.exc_info() to get at the exception information
was rejected; sys.exc_info() has very complex semantics and it is
perfectly possible that it returns the exception information for an
exception that was caught ages ago. It was also proposed to add an
additional boolean to distinguish between reaching the end of BLOCK and
a non-local goto. This was rejected as too complex and unnecessary; a
non-local goto should be considered unexceptional for the purposes of a
database transaction roll-back decision.

To facilitate chaining of contexts in Python code that directly
manipulates context managers, __exit__() methods should not re-raise the
error that is passed in to them. It is always the responsibility of the
caller of the __exit__() method to do any reraising in that case.

That way, if the caller needs to tell whether the __exit__() invocation
failed (as opposed to successfully cleaning up before propagating the
original error), it can do so.

If __exit__() returns without an error, this can then be interpreted as
success of the __exit__() method itself (regardless of whether or not
the original error is to be propagated or suppressed).

However, if __exit__() propagates an exception to its caller, this means
that __exit__() itself has failed. Thus, __exit__() methods should avoid
raising errors unless they have actually failed. (And allowing the
original error to proceed isn't a failure.)

Transition Plan

In Python 2.5, the new syntax will only be recognized if a future
statement is present:

    from __future__ import with_statement

This will make both 'with' and 'as' keywords. Without the future
statement, using 'with' or 'as' as an identifier will cause a Warning to
be issued to stderr.

In Python 2.6, the new syntax will always be recognized; 'with' and 'as'
are always keywords.

Generator Decorator

With PEP 342 accepted, it is possible to write a decorator that makes it
possible to use a generator that yields exactly once to control a
with-statement. Here's a sketch of such a decorator:

    class GeneratorContextManager(object):

       def __init__(self, gen):
           self.gen = gen

       def __enter__(self):
           try:
               return self.gen.next()
           except StopIteration:
               raise RuntimeError("generator didn't yield")

       def __exit__(self, type, value, traceback):
           if type is None:
               try:
                   self.gen.next()
               except StopIteration:
                   return
               else:
                   raise RuntimeError("generator didn't stop")
           else:
               try:
                   self.gen.throw(type, value, traceback)
                   raise RuntimeError("generator didn't stop after throw()")
               except StopIteration:
                   return True
               except:
                   # only re-raise if it's *not* the exception that was
                   # passed to throw(), because __exit__() must not raise
                   # an exception unless __exit__() itself failed.  But
                   # throw() has to raise the exception to signal
                   # propagation, so this fixes the impedance mismatch
                   # between the throw() protocol and the __exit__()
                   # protocol.
                   #
                   if sys.exc_info()[1] is not value:
                       raise

    def contextmanager(func):
       def helper(*args, **kwds):
           return GeneratorContextManager(func(*args, **kwds))
       return helper

This decorator could be used as follows:

    @contextmanager
    def opening(filename):
       f = open(filename) # IOError is untouched by GeneratorContext
       try:
           yield f
       finally:
           f.close() # Ditto for errors here (however unlikely)

A robust implementation of this decorator will be made part of the
standard library.

Context Managers in the Standard Library

It would be possible to endow certain objects, like files, sockets, and
locks, with __enter__() and __exit__() methods so that instead of
writing:

    with locking(myLock):
        BLOCK

one could write simply:

    with myLock:
        BLOCK

I think we should be careful with this; it could lead to mistakes like:

    f = open(filename)
    with f:
        BLOCK1
    with f:
        BLOCK2

which does not do what one might think (f is closed before BLOCK2 is
entered).

OTOH such mistakes are easily diagnosed; for example, the generator
context decorator above raises RuntimeError when a second with-statement
calls f.__enter__() again. A similar error can be raised if __enter__ is
invoked on a closed file object.

For Python 2.5, the following types have been identified as context
managers:

    - file
    - thread.LockType
    - threading.Lock
    - threading.RLock
    - threading.Condition
    - threading.Semaphore
    - threading.BoundedSemaphore

A context manager will also be added to the decimal module to support
using a local decimal arithmetic context within the body of a with
statement, automatically restoring the original context when the with
statement is exited.

Standard Terminology

This PEP proposes that the protocol consisting of the __enter__() and
__exit__() methods be known as the "context management protocol", and
that objects that implement that protocol be known as "context
managers".[5]

The expression immediately following the with keyword in the statement
is a "context expression" as that expression provides the main clue as
to the runtime environment the context manager establishes for the
duration of the statement body.

The code in the body of the with statement and the variable name (or
names) after the as keyword don't really have special terms at this
point in time. The general terms "statement body" and "target list" can
be used, prefixing with "with" or "with statement" if the terms would
otherwise be unclear.

Given the existence of objects such as the decimal module's arithmetic
context, the term "context" is unfortunately ambiguous. If necessary, it
can be made more specific by using the terms "context manager" for the
concrete object created by the context expression and "runtime context"
or (preferably) "runtime environment" for the actual state modifications
made by the context manager. When simply discussing use of the with
statement, the ambiguity shouldn't matter too much as the context
expression fully defines the changes made to the runtime environment.
The distinction is more important when discussing the mechanics of the
with statement itself and how to go about actually implementing context
managers.

Caching Context Managers

Many context managers (such as files and generator-based contexts) will
be single-use objects. Once the __exit__() method has been called, the
context manager will no longer be in a usable state (e.g. the file has
been closed, or the underlying generator has finished execution).

Requiring a fresh manager object for each with statement is the easiest
way to avoid problems with multi-threaded code and nested with
statements trying to use the same context manager. It isn't coincidental
that all of the standard library context managers that support reuse
come from the threading module - they're all already designed to deal
with the problems created by threaded and nested usage.

This means that in order to save a context manager with particular
initialisation arguments to be used in multiple with statements, it will
typically be necessary to store it in a zero-argument callable that is
then called in the context expression of each statement rather than
caching the context manager directly.

When this restriction does not apply, the documentation of the affected
context manager should make that clear.

Resolved Issues

The following issues were resolved by BDFL approval (and a lack of any
major objections on python-dev).

1.  What exception should GeneratorContextManager raise when the
    underlying generator-iterator misbehaves? The following quote is the
    reason behind Guido's choice of RuntimeError for both this and for
    the generator close() method in PEP 342 (from[6]):

    "I'd rather not introduce a new exception class just for this
    purpose, since it's not an exception that I want people to catch: I
    want it to turn into a traceback which is seen by the programmer who
    then fixes the code. So now I believe they should both raise
    RuntimeError. There are some precedents for that: it's raised by the
    core Python code in situations where endless recursion is detected,
    and for uninitialized objects (and for a variety of miscellaneous
    conditions)."

2.  It is fine to raise AttributeError instead of TypeError if the
    relevant methods aren't present on a class involved in a with
    statement. The fact that the abstract object C API raises TypeError
    rather than AttributeError is an accident of history, rather than a
    deliberate design decision[7].

3.  Objects with __enter__/__exit__ methods are called "context
    managers" and the decorator to convert a generator function into a
    context manager factory is contextlib.contextmanager. There were
    some other suggestions[8] during the 2.5 release cycle but no
    compelling arguments for switching away from the terms that had been
    used in the PEP implementation were made.

Rejected Options

For several months, the PEP prohibited suppression of exceptions in
order to avoid hidden flow control. Implementation revealed this to be a
right royal pain, so Guido restored the ability[9].

Another aspect of the PEP that caused no end of questions and
terminology debates was providing a __context__() method that was
analogous to an iterable's __iter__() method[10][11][12]. The ongoing
problems[13][14] with explaining what it was and why it was and how it
was meant to work eventually lead to Guido killing the concept
outright[15] (and there was much rejoicing!).

The notion of using the PEP 342 generator API directly to define the
with statement was also briefly entertained[16], but quickly dismissed
as making it too difficult to write non-generator based context
managers.

Examples

The generator based examples rely on PEP 342. Also, some of the examples
are unnecessary in practice, as the appropriate objects, such as
threading.RLock, are able to be used directly in with statements.

The tense used in the names of the example contexts is not arbitrary.
Past tense ("-ed") is used when the name refers to an action which is
done in the __enter__ method and undone in the __exit__ method.
Progressive tense ("-ing") is used when the name refers to an action
which is to be done in the __exit__ method.

1.  A template for ensuring that a lock, acquired at the start of a
    block, is released when the block is left:

        @contextmanager
        def locked(lock):
            lock.acquire()
            try:
                yield
            finally:
                lock.release()

    Used as follows:

        with locked(myLock):
            # Code here executes with myLock held.  The lock is
            # guaranteed to be released when the block is left (even
            # if via return or by an uncaught exception).

2.  A template for opening a file that ensures the file is closed when
    the block is left:

        @contextmanager
        def opened(filename, mode="r"):
            f = open(filename, mode)
            try:
                yield f
            finally:
                f.close()

    Used as follows:

        with opened("/etc/passwd") as f:
            for line in f:
                print line.rstrip()

3.  A template for committing or rolling back a database transaction:

        @contextmanager
        def transaction(db):
            db.begin()
            try:
                yield None
            except:
                db.rollback()
                raise
            else:
                db.commit()

4.  Example 1 rewritten without a generator:

        class locked:
           def __init__(self, lock):
               self.lock = lock
           def __enter__(self):
               self.lock.acquire()
           def __exit__(self, type, value, tb):
               self.lock.release()

    (This example is easily modified to implement the other relatively
    stateless examples; it shows that it is easy to avoid the need for a
    generator if no special state needs to be preserved.)

5.  Redirect stdout temporarily:

        @contextmanager
        def stdout_redirected(new_stdout):
            save_stdout = sys.stdout
            sys.stdout = new_stdout
            try:
                yield None
            finally:
                sys.stdout = save_stdout

    Used as follows:

        with opened(filename, "w") as f:
            with stdout_redirected(f):
                print "Hello world"

    This isn't thread-safe, of course, but neither is doing this same
    dance manually. In single-threaded programs (for example, in
    scripts) it is a popular way of doing things.

6.  A variant on opened() that also returns an error condition:

        @contextmanager
        def opened_w_error(filename, mode="r"):
            try:
                f = open(filename, mode)
            except IOError, err:
                yield None, err
            else:
                try:
                    yield f, None
                finally:
                    f.close()

    Used as follows:

        with opened_w_error("/etc/passwd", "a") as (f, err):
            if err:
                print "IOError:", err
            else:
                f.write("guido::0:0::/:/bin/sh\n")

7.  Another useful example would be an operation that blocks signals.
    The use could be like this:

        import signal

        with signal.blocked():
            # code executed without worrying about signals

    An optional argument might be a list of signals to be blocked; by
    default all signals are blocked. The implementation is left as an
    exercise to the reader.

8.  Another use for this feature is the Decimal context. Here's a simple
    example, after one posted by Michael Chermside:

        import decimal

        @contextmanager
        def extra_precision(places=2):
            c = decimal.getcontext()
            saved_prec = c.prec
            c.prec += places
            try:
                yield None
            finally:
                c.prec = saved_prec

    Sample usage (adapted from the Python Library Reference):

        def sin(x):
            "Return the sine of x as measured in radians."
            with extra_precision():
                i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1
                while s != lasts:
                    lasts = s
                    i += 2
                    fact *= i * (i-1)
                    num *= x * x
                    sign *= -1
                    s += num / fact * sign
            # The "+s" rounds back to the original precision,
            # so this must be outside the with-statement:
            return +s

9.  Here's a simple context manager for the decimal module:

        @contextmanager
        def localcontext(ctx=None):
            """Set a new local decimal context for the block"""
            # Default to using the current context
            if ctx is None:
                ctx = getcontext()
            # We set the thread context to a copy of this context
            # to ensure that changes within the block are kept
            # local to the block.
            newctx = ctx.copy()
            oldctx = decimal.getcontext()
            decimal.setcontext(newctx)
            try:
                yield newctx
            finally:
                # Always restore the original context
                decimal.setcontext(oldctx)

    Sample usage:

        from decimal import localcontext, ExtendedContext

        def sin(x):
            with localcontext() as ctx:
                ctx.prec += 2
                # Rest of sin calculation algorithm
                # uses a precision 2 greater than normal
            return +s # Convert result to normal precision

        def sin(x):
            with localcontext(ExtendedContext):
                # Rest of sin calculation algorithm
                # uses the Extended Context from the
                # General Decimal Arithmetic Specification
            return +s # Convert result to normal context

10. A generic "object-closing" context manager:

        class closing(object):
            def __init__(self, obj):
                self.obj = obj
            def __enter__(self):
                return self.obj
            def __exit__(self, *exc_info):
                try:
                    close_it = self.obj.close
                except AttributeError:
                    pass
                else:
                    close_it()

    This can be used to deterministically close anything with a close
    method, be it file, generator, or something else. It can even be
    used when the object isn't guaranteed to require closing (e.g., a
    function that accepts an arbitrary iterable):

        # emulate opening():
        with closing(open("argument.txt")) as contradiction:
           for line in contradiction:
               print line

        # deterministically finalize an iterator:
        with closing(iter(data_source)) as data:
           for datum in data:
               process(datum)

    (Python 2.5's contextlib module contains a version of this context
    manager)

11. PEP 319 gives a use case for also having a released() context to
    temporarily release a previously acquired lock; this can be written
    very similarly to the locked context manager above by swapping the
    acquire() and release() calls:

        class released:
          def __init__(self, lock):
              self.lock = lock
          def __enter__(self):
              self.lock.release()
          def __exit__(self, type, value, tb):
              self.lock.acquire()

    Sample usage:

        with my_lock:
            # Operations with the lock held
            with released(my_lock):
                # Operations without the lock
                # e.g. blocking I/O
            # Lock is held again here

12. A "nested" context manager that automatically nests the supplied
    contexts from left-to-right to avoid excessive indentation:

        @contextmanager
        def nested(*contexts):
            exits = []
            vars = []
            try:
                try:
                    for context in contexts:
                        exit = context.__exit__
                        enter = context.__enter__
                        vars.append(enter())
                        exits.append(exit)
                    yield vars
                except:
                    exc = sys.exc_info()
                else:
                    exc = (None, None, None)
            finally:
                while exits:
                    exit = exits.pop()
                    try:
                        exit(*exc)
                    except:
                        exc = sys.exc_info()
                    else:
                        exc = (None, None, None)
                if exc != (None, None, None):
                    # sys.exc_info() may have been
                    # changed by one of the exit methods
                    # so provide explicit exception info
                    raise exc[0], exc[1], exc[2]

    Sample usage:

        with nested(a, b, c) as (x, y, z):
            # Perform operation

    Is equivalent to:

        with a as x:
            with b as y:
                with c as z:
                    # Perform operation

    (Python 2.5's contextlib module contains a version of this context
    manager)

Reference Implementation

This PEP was first accepted by Guido at his EuroPython keynote, 27 June
2005. It was accepted again later, with the __context__ method added.
The PEP was implemented in Subversion for Python 2.5a1 The __context__()
method was removed in Python 2.5b1

Acknowledgements

Many people contributed to the ideas and concepts in this PEP, including
all those mentioned in the acknowledgements for PEP 340 and PEP 346.

Additional thanks goes to (in no meaningful order): Paul Moore, Phillip
J. Eby, Greg Ewing, Jason Orendorff, Michael Hudson, Raymond Hettinger,
Walter Dörwald, Aahz, Georg Brandl, Terry Reedy, A.M. Kuchling, Brett
Cannon, and all those that participated in the discussions on
python-dev.

References

Copyright

This document has been placed in the public domain.

[1] A simple question kickstarts a thorough review of PEP 343
https://mail.python.org/pipermail/python-dev/2006-April/063859.html

[2] Guido suggests some generator changes that ended up in PEP 342
https://mail.python.org/pipermail/python-dev/2005-May/053885.html

[3] Wiki discussion of PEP 343 http://wiki.python.org/moin/WithStatement

[4] Raymond Chen's article on hidden flow control
https://devblogs.microsoft.com/oldnewthing/20050106-00/?p=36783

[5] Early draft of some documentation for the with statement
https://mail.python.org/pipermail/python-dev/2005-July/054658.html

[6] Guido raises some exception handling questions
https://mail.python.org/pipermail/python-dev/2005-June/054064.html

[7] Guido says AttributeError is fine for missing special methods
https://mail.python.org/pipermail/python-dev/2005-October/057625.html

[8] Proposal to use 'context guard' instead of 'context manager'
https://mail.python.org/pipermail/python-dev/2006-May/064676.html

[9] Guido restores the ability to suppress exceptions
https://mail.python.org/pipermail/python-dev/2006-February/061909.html

[10] Proposal to add the __with__ method
https://mail.python.org/pipermail/python-dev/2005-October/056947.html

[11] Guido lets me (Alyssa Coghlan) talk him into a bad idea ;)
https://mail.python.org/pipermail/python-dev/2005-October/057018.html

[12] Guido answers some questions about the __context__ method
https://mail.python.org/pipermail/python-dev/2005-October/057520.html

[13] Guido answers more questions about the __context__ method
https://mail.python.org/pipermail/python-dev/2005-October/057535.html

[14] Guido restores the ability to suppress exceptions
https://mail.python.org/pipermail/python-dev/2006-February/061909.html

[15] Guido kills the __context__() method
https://mail.python.org/pipermail/python-dev/2006-April/064632.html

[16] Proposal to use the PEP 342 enhanced generator API directly
https://mail.python.org/pipermail/python-dev/2005-October/056969.html

authors:
Alyssa Coghlan
Guido van Rossum


text:
PEP: 523 Title: Adding a frame evaluation API to CPython Author: Brett
Cannon<EMAIL_ADDRESS>Dino Viehland<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
16-May-2016 Python-Version: 3.6 Post-History: 16-May-2016 Resolution:
https://mail.python.org/pipermail/python-dev/2016-August/145937.html

Abstract

This PEP proposes to expand CPython's C API[1] to allow for the
specification of a per-interpreter function pointer to handle the
evaluation of frames[2]. This proposal also suggests adding a new field
to code objects[3] to store arbitrary data for use by the frame
evaluation function.

Rationale

One place where flexibility has been lacking in Python is in the direct
execution of Python code. While CPython's C API[4] allows for
constructing the data going into a frame object and then evaluating it
via PyEval_EvalFrameEx()[5], control over the execution of Python code
comes down to individual objects instead of a holistic control of
execution at the frame level.

While wanting to have influence over frame evaluation may seem a bit too
low-level, it does open the possibility for things such as a
method-level JIT to be introduced into CPython without CPython itself
having to provide one. By allowing external C code to control frame
evaluation, a JIT can participate in the execution of Python code at the
key point where evaluation occurs. This then allows for a JIT to
conditionally recompile Python bytecode to machine code as desired while
still allowing for executing regular CPython bytecode when running the
JIT is not desired. This can be accomplished by allowing interpreters to
specify what function to call to evaluate a frame. And by placing the
API at the frame evaluation level it allows for a complete view of the
execution environment of the code for the JIT.

This ability to specify a frame evaluation function also allows for
other use-cases beyond just opening CPython up to a JIT. For instance,
it would not be difficult to implement a tracing or profiling function
at the call level with this API. While CPython does provide the ability
to set a tracing or profiling function at the Python level, this would
be able to match the data collection of the profiler and quite possibly
be faster for tracing by simply skipping per-line tracing support.

It also opens up the possibility of debugging where the frame evaluation
function only performs special debugging work when it detects it is
about to execute a specific code object. In that instance the bytecode
could be theoretically rewritten in-place to inject a breakpoint
function call at the proper point for help in debugging while not having
to do a heavy-handed approach as required by sys.settrace().

To help facilitate these use-cases, we are also proposing the adding of
a "scratch space" on code objects via a new field. This will allow
per-code object data to be stored with the code object itself for easy
retrieval by the frame evaluation function as necessary. The field
itself will simply be a PyObject * type so that any data stored in the
field will participate in normal object memory management.

Proposal

All proposed C API changes below will not be part of the stable ABI.

Expanding PyCodeObject

One field is to be added to the PyCodeObject struct [6]:

    typedef struct {
       ...
       void *co_extra;  /* "Scratch space" for the code object. */
    } PyCodeObject;

The co_extra will be NULL by default and only filled in as needed.
Values stored in the field are expected to not be required in order for
the code object to function, allowing the loss of the data of the field
to be acceptable.

A private API has been introduced to work with the field:

    PyAPI_FUNC(Py_ssize_t) _PyEval_RequestCodeExtraIndex(freefunc);
    PyAPI_FUNC(int) _PyCode_GetExtra(PyObject *code, Py_ssize_t index,
                                     void **extra);
    PyAPI_FUNC(int) _PyCode_SetExtra(PyObject *code, Py_ssize_t index,
                                     void *extra);

Users of the field are expected to call _PyEval_RequestCodeExtraIndex()
to receive (what should be considered) an opaque index value to adding
data into co-extra. With that index, users can set data using
_PyCode_SetExtra() and later retrieve the data with _PyCode_GetExtra().
The API is purposefully listed as private to communicate the fact that
there are no semantic guarantees of the API between Python releases.

Using a list and tuple were considered but was found to be less
performant, and with a key use-case being JIT usage the performance
consideration won out for using a custom struct instead of a Python
object.

A dict was also considered, but once again performance was more
important. While a dict will have constant overhead in looking up data,
the overhead for the common case of a single object being stored in the
data structure leads to a tuple having better performance
characteristics (i.e. iterating a tuple of length 1 is faster than the
overhead of hashing and looking up an object in a dict).

Expanding PyInterpreterState

The entrypoint for the frame evaluation function is per-interpreter:

    // Same type signature as PyEval_EvalFrameEx().
    typedef PyObject* (*_PyFrameEvalFunction)(PyFrameObject*, int);

    typedef struct {
        ...
        _PyFrameEvalFunction eval_frame;
    } PyInterpreterState;

By default, the eval_frame field will be initialized to a function
pointer that represents what PyEval_EvalFrameEx() currently is (called
_PyEval_EvalFrameDefault(), discussed later in this PEP). Third-party
code may then set their own frame evaluation function instead to control
the execution of Python code. A pointer comparison can be used to detect
if the field is set to _PyEval_EvalFrameDefault() and thus has not been
mutated yet.

Changes to Python/ceval.c

PyEval_EvalFrameEx()[7] as it currently stands will be renamed to
_PyEval_EvalFrameDefault(). The new PyEval_EvalFrameEx() will then
become:

    PyObject *
    PyEval_EvalFrameEx(PyFrameObject *frame, int throwflag)
    {
        PyThreadState *tstate = PyThreadState_GET();
        return tstate->interp->eval_frame(frame, throwflag);
    }

This allows third-party code to place themselves directly in the path of
Python code execution while being backwards-compatible with code already
using the pre-existing C API.

Updating python-gdb.py

The generated python-gdb.py file used for Python support in GDB makes
some hard-coded assumptions about PyEval_EvalFrameEx(), e.g. the names
of local variables. It will need to be updated to work with the proposed
changes.

Performance impact

As this PEP is proposing an API to add pluggability, performance impact
is considered only in the case where no third-party code has made any
changes.

Several runs of pybench[8] consistently showed no performance cost from
the API change alone.

A run of the Python benchmark suite[9] showed no measurable cost in
performance.

In terms of memory impact, since there are typically not many CPython
interpreters executing in a single process that means the impact of
co_extra being added to PyCodeObject is the only worry. According
to[10], a run of the Python test suite results in about 72,395 code
objects being created. On a 64-bit CPU that would result in 579,160
bytes of extra memory being used if all code objects were alive at once
and had nothing set in their co_extra fields.

Example Usage

A JIT for CPython

Pyjion

The Pyjion project[11] has used this proposed API to implement a JIT for
CPython using the CoreCLR's JIT[12]. Each code object has its co_extra
field set to a PyjionJittedCode object which stores four pieces of
information:

1.  Execution count
2.  A boolean representing whether a previous attempt to JIT failed
3.  A function pointer to a trampoline (which can be type tracing or
    not)
4.  A void pointer to any JIT-compiled machine code

The frame evaluation function has (roughly) the following algorithm:

    def eval_frame(frame, throw_flag):
        pyjion_code = frame.code.co_extra
        if not pyjion_code:
            frame.code.co_extra = PyjionJittedCode()
        elif not pyjion_code.jit_failed:
            if not pyjion_code.jit_code:
                return pyjion_code.eval(pyjion_code.jit_code, frame)
            elif pyjion_code.exec_count > 20_000:
                if jit_compile(frame):
                    return pyjion_code.eval(pyjion_code.jit_code, frame)
                else:
                    pyjion_code.jit_failed = True
        pyjion_code.exec_count += 1
        return _PyEval_EvalFrameDefault(frame, throw_flag)

The key point, though, is that all of this work and logic is separate
from CPython and yet with the proposed API changes it is able to provide
a JIT that is compliant with Python semantics (as of this writing,
performance is almost equivalent to CPython without the new API). This
means there's nothing technically preventing others from implementing
their own JITs for CPython by utilizing the proposed API.

Other JITs

It should be mentioned that the Pyston team was consulted on an earlier
version of this PEP that was more JIT-specific and they were not
interested in utilizing the changes proposed because they want control
over memory layout they had no interest in directly supporting CPython
itself. An informal discussion with a developer on the PyPy team led to
a similar comment.

Numba[13], on the other hand, suggested that they would be interested in
the proposed change in a post-1.0 future for themselves[14].

The experimental Coconut JIT[15] could have benefitted from this PEP. In
private conversations with Coconut's creator we were told that our API
was probably superior to the one they developed for Coconut to add JIT
support to CPython.

Debugging

In conversations with the Python Tools for Visual Studio team (PTVS)
[16], they thought they would find these API changes useful for
implementing more performant debugging. As mentioned in the Rationale
section, this API would allow for switching on debugging functionality
only in frames where it is needed. This could allow for either skipping
information that sys.settrace() normally provides and even go as far as
to dynamically rewrite bytecode prior to execution to inject e.g.
breakpoints in the bytecode.

It also turns out that Google provides a very similar API internally. It
has been used for performant debugging purposes.

Implementation

A set of patches implementing the proposed API is available through the
Pyjion project[17]. In its current form it has more changes to CPython
than just this proposed API, but that is for ease of development instead
of strict requirements to accomplish its goals.

Open Issues

Allow eval_frame to be NULL

Currently the frame evaluation function is expected to always be set. It
could very easily simply default to NULL instead which would signal to
use _PyEval_EvalFrameDefault(). The current proposal of not
special-casing the field seemed the most straightforward, but it does
require that the field not accidentally be cleared, else a crash may
occur.

Rejected Ideas

A JIT-specific C API

Originally this PEP was going to propose a much larger API change which
was more JIT-specific. After soliciting feedback from the Numba
team[18], though, it became clear that the API was unnecessarily large.
The realization was made that all that was truly needed was the
opportunity to provide a trampoline function to handle execution of
Python code that had been JIT-compiled and a way to attach that compiled
machine code along with other critical data to the corresponding Python
code object. Once it was shown that there was no loss in functionality
or in performance while minimizing the API changes required, the
proposal was changed to its current form.

Is co_extra needed?

While discussing this PEP at PyCon US 2016, some core developers
expressed their worry of the co_extra field making code objects mutable.
The thinking seemed to be that having a field that was mutated after the
creation of the code object made the object seem mutable, even though no
other aspect of code objects changed.

The view of this PEP is that the co_extra field doesn't change the fact
that code objects are immutable. The field is specified in this PEP to
not contain information required to make the code object usable, making
it more of a caching field. It could be viewed as similar to the UTF-8
cache that string objects have internally; strings are still considered
immutable even though they have a field that is conditionally set.

Performance measurements were also made where the field was not
available for JIT workloads. The loss of the field was deemed too costly
to performance when using an unordered map from C++ or Python's dict to
associated a code object with JIT-specific data objects.

References

Copyright

This document has been placed in the public domain.

[1] CPython's C API (https://docs.python.org/3/c-api/index.html)

[2] PyEval_EvalFrameEx()
(https://docs.python.org/3/c-api/veryhigh.html?highlight=pyframeobject#c.PyEval_EvalFrameEx)

[3] PyCodeObject
(https://docs.python.org/3/c-api/code.html#c.PyCodeObject)

[4] CPython's C API (https://docs.python.org/3/c-api/index.html)

[5] PyEval_EvalFrameEx()
(https://docs.python.org/3/c-api/veryhigh.html?highlight=pyframeobject#c.PyEval_EvalFrameEx)

[6] PyCodeObject
(https://docs.python.org/3/c-api/code.html#c.PyCodeObject)

[7] PyEval_EvalFrameEx()
(https://docs.python.org/3/c-api/veryhigh.html?highlight=pyframeobject#c.PyEval_EvalFrameEx)

[8] pybench (https://hg.python.org/cpython/file/default/Tools/pybench)

[9] Python benchmark suite (https://hg.python.org/benchmarks)

[10] [Python-Dev] Opcode cache in ceval loop
(https://mail.python.org/pipermail/python-dev/2016-February/143025.html)

[11] Pyjion project (https://github.com/microsoft/pyjion)

[12] .NET Core Runtime (CoreCLR) (https://github.com/dotnet/coreclr)

[13] Numba (http://numba.pydata.org/)

[14] numba-users mailing list: "Would the C API for a JIT entrypoint
being proposed by Pyjion help out Numba?"
(https://groups.google.com/a/continuum.io/forum/#!topic/numba-users/yRl_0t8-m1g)

[15] Coconut (https://github.com/davidmalcolm/coconut)

[16] Python Tools for Visual Studio (http://microsoft.github.io/PTVS/)

[17] Pyjion project (https://github.com/microsoft/pyjion)

[18] Numba (http://numba.pydata.org/)


text:
PEP: 581 Title: Using GitHub Issues for CPython Author: Mariatta
<EMAIL_ADDRESS>BDFL-Delegate: Barry Warsaw<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/535 Status: Final Type:
Process Created: 20-Jun-2018 Post-History: 07-Mar-2019 Resolution:
https://mail.python.org/pipermail/python-dev/2019-May/157399.html

Abstract

This PEP outlines the rationale for migration from Python's issue
tracker on Roundup to GitHub issues. See PEP 588 for the detailed
migration plan.

Rationale

CPython's development moved to GitHub in February 2017. All other
projects within the PSF's organization are hosted on GitHub and are
using GitHub issues. CPython is still using Roundup as the issue tracker
on bugs.python.org (bpo)[1].

Why GitHub?

GitHub has a lot of nice features, readily available out of the box.
Some of these aren't readily available on Roundup / bpo.

-   APIs that can be used to build integrations and automations. There
    are various existing integrations and applications available from
    GitHub Marketplace to help with the workflow. New applications are
    easily installed and enabled. In addition, we've had great success
    with building our own GitHub bots, like miss-islington[2],
    bedevere[3], and the-knights-who-say-ni[4].
-   Ability to embed/drag and drop screenshots and debug log files into
    GitHub pull requests and issues.
-   Administrators and core developers can edit issues, comments, and
    pull requests.
-   Ability to reply to issue and pull request conversations via email.
-   Support for two factor authentication.
-   Support for markdown and emoji.
-   Preview tab, showing how a comment will be rendered, prior to
    actually posting.
-   Support for voting via reactions.
-   Support for permalinks[5], allowing easy quoting and copying &
    pasting of source code. It is possible to automatically embed code
    snippet on GitHub by pasting a permalink.
-   Core developers, volunteers, and the PSF don't have to maintain the
    issue infrastructure/site, giving us more time and resources to
    focus on the development of Python.
-   Ability to automatically close issues when a PR has been merged[6].
    -   Note that this feature exists in bpo too.
-   Lower barrier to contribution. With more than 28 million users, an
    open source contributor is more likely to already have an account
    and be familiar with GitHub's interface, making it easier to start
    contributing.
-   Email notifications containing metadata[7], integrated with Gmail,
    allowing systematic filtering of emails. While Roundup emails
    contain some metadata, they're not as extensive.
-   Additional privacy, such as offering the user a choice to hide an
    email address, while still allowing communication with the user
    through @-mentions.

Issues with Roundup / bpo

-   Less than five people maintain bpo. Some of them are core
    developers.

-   The upstream Roundup code is in Mercurial. Without any CI available,
    it puts heavy burden on the few existing maintainers in terms of
    reviewing, testing, and applying patches. While there is an
    unofficial mirror of Roundup on GitHub[8], Mercurial patches are
    still the main way to contribute to it.

    There is an open discussion about moving the source code of bpo to
    GitHub[9]. If the source code of bpo does move to GitHub, it will
    become difficult to update patches from upstream. But as long as it
    is in Mercurial, it is difficult to maintain and onboard new
    contributors.

-   In its current state, the project is not equipped to accept lots of
    contributions from people who aren't already familiar with the code
    base.

-   The user interface needs an update and redesign. It will require
    UX/UI research to keep it up to date with current web standards,
    including accessibility.

-   Users email addresses were exposed.

    -   Note: Exposing email address to registered and logged in users
        was a decision taken when bpo instance was setup. This behavior
        has been recently modified after PEP 581's acceptance.

-   REST API is not currently available in bpo. There was an open issue
    in Roundup for adding REST API[10]. At the time PEP 581 was
    proposed, the ticket received no activity since 2016. REST API has
    been integrated in Roundup in February 2019, however it is not yet
    integrated to bpo.

-   It sends a number of unnecessary emails and notifications. An
    example is the nosy email, where email notifications are sent
    whenever someone adds themselves as "nosy". An issue has been filed
    in upstream Roundup about this since 2012 with little traction[11].
    While it can be configured, the request to configure it was not
    addressed/ignored.

-   Creating an account has been a hassle. There have been reports of
    people having trouble creating accounts or logging in. Some examples
    of open tickets: "Commas in username causes error in nosy list"[12],
    "An error has occurred .."[13], "It is not sending me the
    confirmation email ..."[14].

Why not GitLab?

Had we migrated to GitLab instead of GitHub in 2017, this PEP would have
been titled "Using GitLab Issues for CPython".

Why not another issue tracker?

Using another issue tracker will require yet another learning curve, for
having to learn and get used to a different interface. We'll also need
to learn and figure out how to build the integrations with GitHub.

By using GitHub issues, where the CPython source code is currently
hosted and where pull requests are taking place, we'll be providing
consistent experience to contributors and maintainers, while not having
to jump from one interface to another.

Why not focus on improving Roundup / bpo?

GitHub has many features we like that are already available. We still
need to build out additional integrations and update our bots, but this
is something we already know how to do.

In order to really improve Roundup / bpo, it needs to first migrate to
GitHub and add CI and bots. As I understand it, there is hesitation
because upstream Roundup is still in Mercurial. Someone more familiar
with Roundup / bpo needs to champion this effort. (I'm not volunteering,
I'm sorry).

I believe the effort of creating and maintaining GitHub integrations and
bots is much less than the effort needed to get Roundup up to speed and
then to continue maintaining it.

Downsides of GitHub

GitHub is not the perfect issue tracker. Several issues we need to be
aware of:

-   Fear of uncertainty and vendor lock in. GitHub is proprietary and
    there is risk of vendor lock-in. However, this is an existing
    problem since CPython's codebase is already on GitHub. This is also
    not a unique problem to CPython. As a precautionary step, CPython's
    repository on GitHub has been backed up on daily basis since June
    2018.[15]
-   Bots maintenance costs money, and also takes up volunteer time. We
    would be shifting maintenance burden from Roundup to the bots. At
    the very least, so far we've been able to address any bugs/issues
    related to the bots/GitHub APIs rather quickly, in a matter of days,
    instead of months or years. GitHub APIs are extensive, and used not
    only by CPython's bots but also by the wider Python community. It
    makes GitHub APIs more approachable compared to the maintenance of
    Roundup/bpo.
-   Using GitHub could possibly increase the triaging effort. This was
    first raised as a Zulip topic[16], and also brought up during Core
    Python sprint in September 2018[17]. A few solutions have been
    proposed and considered, such as creating a special triage team[18].
    After PEP 581's acceptance, GitHub released a new triaging role,
    currently in beta. The PSF has been in touch with GitHub to have
    this enabled for Python organization. This is pending GitHub's
    review[19].
-   Using GitHub could make it easier for people to post disruptive or
    spammy comments. It is true that there has been incidents where core
    developers have to moderate and lock disruptive discussions on
    GitHub. Thankfully, GitHub interface makes it easy for core
    developers to moderate discussions. In addition, incidents can be
    escalated to GitHub.
-   Manually editing issue templates can be cumbersome and error-prone.
    However, for most people, creating issues on GitHub will be a much
    better experience than creating issue on bpo. The numerous fields
    and textbox to choose from can be confusing and intimidating to a
    newcomer, and it is not possible to "edit" a message. On GitHub,
    issue creator can preview their submission, and edit their mistake
    after posting.
-   bpo uses a number of fields to specify several metadata, and these
    might not be easily transferable to GitHub. The intended way to
    handle custom metadata on GitHub is by using labels. The details of
    which labels to create will be further discussed in PEP 588.

Further questions and discussions

You can post questions on Discourse under the Core-Workflow category.

Acknowledgements

Thanks to Guido van Rossum, Brett Cannon, and Alyssa Coghlan, who were
consulted in the early stage and research of this PEP. Their feedback,
concerns, input, and ideas have been valuable.

References

Copyright

This document has been placed in the public domain.

[1] bugs.python.org (https://bugs.python.org/)

[2] miss-islington (https://github.com/python/miss-islington)

[3] bedevere (https://github.com/python/bedevere)

[4] the-knights-who-say-ni
(https://github.com/python/the-knights-who-say-ni)

[5] Getting permanent links to files
(https://help.github.com/articles/getting-permanent-links-to-files/)

[6] Closing issues using keywords
(https://help.github.com/articles/closing-issues-using-keywords/)

[7] About GitHub email notifications
(https://help.github.com/articles/about-email-notifications/)

[8] Unofficial GitHub mirror of Roundup
(https://github.com/roundup-tracker/roundup)

[9] Consider whether or not to migrate bugs.python.org source code to
GitHub repo (https://github.com/python/bugs.python.org/issues/2)

[10] Roundup issue 2550734: Expose roundup via a RESTful interface
(http://issues.roundup-tracker.org/issue2550734)

[11] Roundup issue 2550742: Do not send email by default when adding or
removing oneself from the Nosy list
(http://issues.roundup-tracker.org/issue2550742)

[12] Commas in username causes error
(https://github.com/python/bugs.python.org/issues/26)

[13] It is not sending me the confirmation email
(https://github.com/python/bugs.python.org/issues/27)

[14] An error has occurred
(https://github.com/python/bugs.python.org/issues/26)

[15] Backup GitHub information
(https://github.com/python/core-workflow/issues/20#issuecomment-396709762)

[16] s/n ratio -- Python -- Zulip
(https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20ratio)

[17] PEP 581 discussion at Python Core Sprint 2018
(https://mariatta.ca/core-sprint-2018-part-2.html)

[18] Proposal: Create Bug Triage team on GitHub
(https://discuss.python.org/t/proposal-create-bug-triage-team-on-github/992)

[19] Request to enable triage role on GitHub
(https://discuss.python.org/t/proposal-create-bug-triage-team-on-github/992/16)


text:
PEP: 212 Title: Loop Counter Iteration Author: Peter Schneider-Kamp
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 22-Aug-2000 Python-Version: 2.1
Post-History:

Rejection Notice

This PEP has been rejected. enumerate(), introduced in PEP 279, covers
the use-case proposed in this PEP, and the PEP author has been
unreachable.

Introduction

This PEP describes the often proposed feature of exposing the loop
counter in for-loops. This PEP tracks the status and ownership of this
feature. It contains a description of the feature and outlines changes
necessary to support the feature. This PEP summarizes discussions held
in mailing list forums, and provides URLs for further information, where
appropriate. The CVS revision history of this file contains the
definitive historical record.

Motivation

Standard for-loops in Python iterate over the elements of a sequence[1].
Often it is desirable to loop over the indices or both the elements and
the indices instead.

The common idioms used to accomplish this are unintuitive. This PEP
proposes two different ways of exposing the indices.

Loop counter iteration

The current idiom for looping over the indices makes use of the built-in
range function:

    for i in range(len(sequence)):
        # work with index i

Looping over both elements and indices can be achieved either by the old
idiom or by using the new zip built-in function[2]:

    for i in range(len(sequence)):
        e = sequence[i]
        # work with index i and element e

or:

    for i, e in zip(range(len(sequence)), sequence):
       # work with index i and element e

The Proposed Solutions

There are three solutions that have been discussed. One adds a
non-reserved keyword, the other adds two built-in functions. A third
solution adds methods to sequence objects.

Non-reserved keyword indexing

This solution would extend the syntax of the for-loop by adding an
optional <variable> indexing clause which can also be used instead of
the <variable> in clause.

Looping over the indices of a sequence would thus become:

    for i indexing sequence:
        # work with index i

Looping over both indices and elements would similarly be:

    for i indexing e in sequence:
        # work with index i and element e

Built-in functions indices and irange

This solution adds two built-in functions indices and irange. The
semantics of these can be described as follows:

    def indices(sequence):
        return range(len(sequence))

    def irange(sequence):
        return zip(range(len(sequence)), sequence)

These functions could be implemented either eagerly or lazily and should
be easy to extend in order to accept more than one sequence argument.

The use of these functions would simplify the idioms for looping over
the indices and over both elements and indices:

    for i in indices(sequence):
        # work with index i

    for i, e in irange(sequence):
        # work with index i and element e

Methods for sequence objects

This solution proposes the addition of indices, items and values methods
to sequences, which enable looping over indices only, both indices and
elements, and elements only respectively.

This would immensely simplify the idioms for looping over indices and
for looping over both elements and indices:

    for i in sequence.indices():
        # work with index i

    for i, e in sequence.items():
        # work with index i and element e

Additionally it would allow to do looping over the elements of sequences
and dictionaries in a consistent way:

    for e in sequence_or_dict.values():
        # do something with element e

Implementations

For all three solutions some more or less rough patches exist as patches
at SourceForge:

-   for i indexing a in l: exposing the for-loop counter[3]
-   add indices() and irange() to built-ins[4]
-   add items() method to listobject[5]

All of them have been pronounced on and rejected by the BDFL.

Note that the indexing keyword is only a NAME in the grammar and so does
not hinder the general use of indexing.

Backward Compatibility Issues

As no keywords are added and the semantics of existing code remains
unchanged, all three solutions can be implemented without breaking
existing code.

Copyright

This document has been placed in the public domain.

References

[1] http://docs.python.org/reference/compound_stmts.html#for

[2] Lockstep Iteration, PEP 201

[3] http://sourceforge.net/patch/download.php?id=101138

[4] http://sourceforge.net/patch/download.php?id=101129

[5] http://sourceforge.net/patch/download.php?id=101178