Implementing "nonlocal" in Tauthon: Part I
February 27, 2017
Tauthon is a fork of Python 2.7 with syntax, builtins, and libraries backported from Python 3. It aspires to be able to run all valid Python 2 and 3 code. In this article, I begin discussing how I was able to backport the "nonlocal" keyword from Python 3. I hope this post is useful for people who are interested in hacking on the CPython interpreter or CPython forks: it sounds hard, and it can be a bit tedious, but it's actually a lot easier than you'd think.
Why nonlocal is important
The nonlocal keyword allows you to modify variables defined in an enclosing scope. For example:
>>> def f(): ... x = 0 ... def g(): ... x += 1 ... def h(): ... nonlocal x ... x += 1 ... print x ... return g, h ... >>> g, h = f() >>> g() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in g UnboundLocalError: local variable 'x' referenced before assignment >>> h() 1 >>> h() 2 >>> h() 3
In practice, I've found nonlocal used most often in tests. The typical pattern is that a function is defined in a unittest, and then modifies a nonlocal variable when something of interest occurs. After the function completes, the test verifies that the event occurs by checking that the value of the variable changed. For example, the code would look something like this:
import unittest
class TestStuff(unittest.TestCase):
def test_something(self):
call_count = 0
def inner():
nonlocal call_count
call_count += 1
# Do something exciting here
def outer():
# Do something complicated here and call inner multiple times
# in non-obvious ways.
inner()
outer()
self.assertEqual(call_count, 1)
if __name__ == "__main__":
unittest.main()
For a pile of examples that show off this usage pattern, (as well as the drudgery associated with backporting it!), see this commit. This came from my work on backporting asyncio, which I (unfortunately) started work on before backporting the nonlocal keyword.
As you can see from that commit, language additions like nonlocal make backporting libraries from Python 3 pretty tiresome. And pretty error-prone, too, since I do them by hand. The experience of trying to backport asyncio made it more clear to me that the proper technical strategy for this project was to first backport new syntax and language features, then new builtins, and finally the new libraries. This is because the libraries and builtins often require the new language features and syntax.
In Python 3.x, nonlocal is a keyword. This means that code like "nonlocal = True" is a SyntaxError. Since Tauthon is completely backwards compatible with Python 2.7 code, when backporting the nonlocal keyword I needed to do so in such a way that using "nonlocal" as an identifier (i.e. variable, function, or attribute name) continued to be valid. My strategy was to first implement nonlocal as a keyword, just as in Python 3.x, and after that to relax nonlocal so that you can use it as an identifier. The reason for this is that I can easily follow the implementation of nonlocal in Python 3.
Finding Python 3.x's nonlocal implementation
So my first step was to find the commits in Python 3 that touched nonlocal. I did this with
$ git log 3.6 --reverse --stat --grep "nonlocal"
as well as other variants of search text (like "PEP 3104"), to find relevant commits. (Of course, someone might have made a relevant change, and used a poor commit message like "save commit" or "fixed bug", and I wouldn't find it with this method. There are a couple of ways of getting around this issue; more details to come). Here are those relevant commits from above, with my own annotations about them for ease of following, (and emails redacted so the authors don't get spammed). Warning: this can be pretty tedious; feel free to skip to Backporting Python 3's nonlocal implementation if you're not interested in every single time "nonlocal" has been touched for the last ten years.
commit 8110a89fde7e127e8234d14226521a586aceff0f
Author: Jeremy Hylton <*****@******.***>
Date: Tue Feb 27 06:50:52 2007 +0000
Provisional implementation of PEP 3104.
Add nonlocal_stmt to Grammar and Nonlocal node to AST. They both
parallel the definitions for globals. The symbol table treats
variables declared as nonlocal just like variables that are free
implicitly.
This change is missing the language spec changes, but makes some
decisions about what the spec should say via the unittests. The PEP
is silent on a number of decisions, so we should review those before
claiming that nonlocal is complete.
Thomas Wouters made the grammer and ast changes. Jeremy Hylton added
the symbol table changes and the tests. Pete Shinners and Neal
Norwitz helped review the code.
Grammar/Grammar | 3 +-
Include/Python-ast.h | 10 +-
Include/graminit.h | 101 +--
Include/symtable.h | 21 +-
Lib/test/test_scope.py | 84 +++
Lib/test/test_syntax.py | 40 ++
Parser/Python.asdl | 1 +
Python/Python-ast.c | 31 +
Python/ast.c | 27 +-
Python/compile.c | 1 +
Python/graminit.c | 1598 ++++++++++++++++++++++++-----------------------
Python/symtable.c | 77 ++-
12 files changed, 1130 insertions(+), 864 deletions(-)
NOTES: This is the implementation of nonlocal. All the commits after this are bugfixes or similar.
commit e5d64c782e26c90fee17265c9414a93edcc3f9bc
Author: Jeremy Hylton <*****@******.***>
Date: Tue Feb 27 15:53:28 2007 +0000
Add news about nonlocal statement
Misc/NEWS | 15 +++++++++++++++
1 file changed, 15 insertions(+)
NOTES: I haven't been keeping the NEWS file updated in Tauthon, so no need to update this.
commit 3fb7381faecb0f675cfe40f4558f45f17e72b9e4
Author: Jack Diederich <*****@******.***>
Date: Wed Feb 28 20:21:30 2007 +0000
regenerated to reflect the addition of 'nonlocal' and removal of 'print'
Lib/keyword.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
NOTES: I'm not going to make nonlocal a keyword, so I can ignore this.
commit f63534524c0d210f0c33c4138ecc9f31d4c5eda1
Author: Guido van Rossum <*****@******.***>
Date: Mon Mar 19 17:56:01 2007 +0000
Fix the compiler package w.r.t. the new metaclass syntax.
(It is still broken w.r.t. the new nonlocal keyword.)
Remove a series of debug prints I accidentally left in test_ast.py.
Lib/compiler/ast.py | 24 +++++++++++++++++++-----
Lib/compiler/pyassem.py | 3 ++-
Lib/compiler/pycodegen.py | 16 +++++++---------
Lib/compiler/symbols.py | 2 +-
Lib/compiler/transformer.py | 11 ++++++-----
Lib/test/test_ast.py | 10 +++-------
Lib/test/test_compiler.py | 4 ++--
7 files changed, 40 insertions(+), 30 deletions(-)
NOTES: This commit is for the new metaclass syntax, which I've actually already done. It does suggest that more work will need to be done on the compiler module.
commit c9001d179c76cf3e7805862400e28b74bec67743
Author: Nick Coghlan <*****@******.***>
Date: Mon Apr 23 10:14:27 2007 +0000
Don't crash when nonlocal is used at module level (fixes SF#1705365)
Lib/test/test_syntax.py | 6 ++++++
Python/symtable.c | 15 ++++++++++-----
2 files changed, 16 insertions(+), 5 deletions(-)
NOTES: Looks like a nonlocal bug. I'll need to backport this.
commit a5d53022c4b03625db47da5523b66e26d25c1ad6
Author: Georg Brandl <*****@******.***>
Date: Tue Dec 4 18:11:03 2007 +0000
Document nonlocal statement. Written for GHOP by "Canadabear".
Doc/reference/simple_stmts.rst | 36 ++++++++++++++++++++++++---
Doc/tutorial/classes.rst | 56 +++++++++++++++++++++++++++++++++++++++---
2 files changed, 85 insertions(+), 7 deletions(-)
NOTES: I haven't been backporting documentation, so no need to do this.
commit 116d63cae91b48ef31450de1ca58b0e25079b04f
Author: Georg Brandl <*****@******.***>
Date: Mon Jul 21 18:26:21 2008 +0000
nonlocal is not in 2.6.
Doc/tutorial/classes.rst | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
NOTES: Ditto, documentation.
commit e79abcda80b78610dd671d1d858e4008d7a3a46c
Author: Georg Brandl <*****@******.***>
Date: Mon Jul 21 18:26:48 2008 +0000
Blocked revisions 65172 via svnmerge
........
r65172 | georg.brandl | 2008-07-21 20:26:21 +0200 (Mon, 21 Jul 2008) | 2 lines
nonlocal is not in 2.6.
........
NOTES: CPython is a sufficiently old project (over 25 years) that it's spanned multiple version control systems. That I know of, it's used CVS, Subversion, Mercurial, and Git (in that order; see PEP's 347, 385, and 512 if you enjoy reading about these migrations). Empty commits like this, I believe, come from idiosyncrasies of these previous version control systems. But I'm not sure.
commit fe44b94c1078f81d5deefb259792ec29435a95ca
Author: Benjamin Peterson <*****@******.***>
Date: Fri Oct 24 22:16:39 2008 +0000
add grammar tests for nonlocal
Lib/test/test_grammar.py | 8 ++++++++
1 file changed, 8 insertions(+)
NOTES: This is a test I should backport.
commit aee9cb6b7231a54c30223654ff9ddb60c0457415
Author: Benjamin Peterson <*****@******.***>
Date: Sat Oct 25 02:53:28 2008 +0000
give a py3k warning when 'nonlocal' is used as a variable name
Lib/test/test_py3kwarn.py | 64 +++++++++++++++++------------------------------
Misc/NEWS | 3 +++
Python/ast.c | 11 +++++---
3 files changed, 34 insertions(+), 44 deletions(-)
NOTES: This is already in Python 2.7 and Tauthon, no need to do anything here.
commit bda5da94834de53cecccb6ae72cc6a6d7cfbacb4
Author: Benjamin Peterson <*****@******.***>
Date: Sat Oct 25 02:56:18 2008 +0000
Blocked revisions 67013 via svnmerge
........
r67013 | benjamin.peterson | 2008-10-24 21:53:28 -0500 (Fri, 24 Oct 2008) | 1 line
give a py3k warning when 'nonlocal' is used as a variable name
........
NOTES: No files changed; old SVN stuff I guess.
commit 009189ef4c344081efc60d6849cddac01d899de2
Author: Georg Brandl <*****@******.***>
Date: Fri Nov 7 08:57:11 2008 +0000
Blocked revisions 66822-66823,66832,66836,66852,66868,66878,66894,66902,66912,66989,66994,67013,67015,67049,67065 via svnmerge
........
r66822 | skip.montanaro | 2008-10-07 03:55:20 +0200 (Tue, 07 Oct 2008) | 2 lines
Simplify individual tests by defining setUp and tearDown methods.
........
(...snipped...)
........
r67013 | benjamin.peterson | 2008-10-25 04:53:28 +0200 (Sat, 25 Oct 2008) | 1 line
give a py3k warning when 'nonlocal' is used as a variable name
........
(...snipped...)
NOTES: This is a bunch of subversion commits squashed together, like the two above, for example. Snipped out the other commits to save space; ignoring this as it's the same py3k warnings as above.
commit 40f777b58da83f1212d3cc25cdee234b3419d453
Author: Georg Brandl <*****@******.***>
Date: Fri Dec 5 18:06:58 2008 +0000
#4549: Mention nonlocal statement in tutorial section about scoping.
Doc/tutorial/classes.rst | 11 +++++------
1 file changed, 5 insertions(+), 6 deletions(-)
NOTES: Docs; ignoring.
commit 23d13f0618fca6d66c26fc3986811d6d60c2b1ca
Author: Georg Brandl <*****@******.***>
Date: Tue May 18 23:45:21 2010 +0000
Blocked revisions 66721-66722,66744-66745,66752,66756,66763-66765,66768,66791-66792,66822-66823,66832,66836,66852,66857,66868,66878,66894,66902,66912,66989,66994,67013,67015,67049,67065,67171,67226,67234,67287,67342,67348-67349,67353,67396,67407,67411,67442,67511,67521,67536-67537,67543,67572,67584,67587,67601,67614,67628,67818,67822,67850,67857,67902,67946,67954,67976,67978-67980,67985,68089,68092,68119,68150,68153,68156,68158,68163,68167,68176,68203,68208-68209,68231,68238,68240,68243,68296,68299,68302,68304,68311,68314,68319,68381,68395,68415,68425,68432,68455,68458-68462,68476,68484-68485,68487,68496,68498,68532,68542,68544-68546,68559-68560,68562,68565-68569,68571,68592,68596-68597,68603-68604,68607,68618,68648,68665,68667,68676,68722,68739,68763-68764,68766,68772-68773,68785,68789,68792-68793,68803,68807,68826,68829,68831,68839-68840,68843,68845,68850,68853,68881,68884,68892,68925,68927,68929,68933,68941-68943,68953,68964,68985,68998,69001,69003,69010,69012,69014,69018,69023,69039,69050,69053,69060-69063,69070,69074,69080,69085,69087,69112-69113,69129-69130,69134,69139,69143,69146,69149,69154,69156,69158,69169,69195,69211-69212,69227,69237,69242,69252-69253,69257,69260,69262,69268,69285,69302-69303,69305,69315,69322,69324,69330-69332,69342,69356,69360,69364-69366,69373-69374,69377,69385,69389,69394,69404,69410,69413,69415,69419-69420,69425,69443,69447,69459-69460,69466-69467,69470,69473-69474,69480-69481,69495,69498,69516,69521-69522,69525,69528,69530,69561,69566,69578-69580,69582-69583,69591,69594,69602,69604,69609-69610,69617,69619,69634,69639,69666,69685,69688-69690,69692-69693,69700,69709-69710,69715-69716,69724,69739,69743,69748,69751,69757,69761,69765,69770,69772,69777,69795,69811,69837-69838,69855,69861,69870-69871,69874,69878,69881,69889,69901-69902,69907-69908,69937,69946-69947,69952-69953,69955,69959,69974,69976,69981,69983,69994,70000 via svnmerge
(...snipped...)
........
r67013 | benjamin.peterson | 2008-10-25 02:53:28 +0000 (Sa, 25 Okt 2008) | 1 line
give a py3k warning when 'nonlocal' is used as a variable name
........
(...snipped...)
NOTES: Snipped, as in the squashed commits above. This time there were over a thousand lines in the squashed commit message! No need to do anything here.
commit 390f59c3bb7b8adfd4be7177837cc495cb673bb0
Author: Mark Dickinson <*****@******.***>
Date: Mon Jun 28 21:14:17 2010 +0000
Update Demo/parser/unparse.py to current Python 3.x syntax. Additions:
- relative imports
- keyword-only arguments
- function annotations
- class decorators
- raise ... from ...
- except ... as ...
- nonlocal
- bytes literals
- set literals
- set comprehensions
- dict comprehensions
Removals:
- print statement.
Some of this should be backported to 2.x.
Demo/parser/test_unparse.py | 83 ++++++++++++++++++++--
Demo/parser/unparse.py | 165 ++++++++++++++++++++++++++++++--------------
2 files changed, 194 insertions(+), 54 deletions(-)
NOTES: This is a demo, not officially part of the Python distribution. (It's not in the standard library). I backported some earlier changes to this file previously, but haven't been updating it since.
commit 74799b493386c17529d5dc7ec6f09783b9f11b22
Author: Benjamin Peterson <*****@******.***>
Date: Tue Jun 29 18:36:39 2010 +0000
update for nonlocal keyword
Doc/glossary.rst | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
NOTES: Docs, ignoring.
commit 8000806ccad06cd21f7cec2432c9c796b57f628d
Author: Benjamin Peterson <*****@******.***>
Date: Tue Jun 29 18:40:09 2010 +0000
Merged revisions 82376 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r82376 | benjamin.peterson | 2010-06-29 13:36:39 -0500 (Tue, 29 Jun 2010) | 1 line
update for nonlocal keyword
........
Doc/glossary.rst | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
NOTES: Docs, ignoring.
commit 04c5ac48e73ffe7972cb1d38890af59320d0288d
Author: Benjamin Peterson <*****@******.***>
Date: Tue Aug 31 14:31:01 2010 +0000
add nonlocal to pydoc topics #9724
Doc/tools/sphinxext/pyspecific.py | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
NOTES: Docs, ignoring.
commit b9e6b88ae2750d78a44879e0b0aee70fa52b6650
Author: Benjamin Peterson <*****@******.***>
Date: Tue Aug 31 14:32:27 2010 +0000
Merged revisions 84376 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r84376 | benjamin.peterson | 2010-08-31 09:31:01 -0500 (Tue, 31 Aug 2010) | 1 line
add nonlocal to pydoc topics #9724
........
Doc/tools/sphinxext/pyspecific.py | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
NOTES: Docs, ignoring.
commit 8312de473362a358391165fa1aa152cd09bb95a0
Author: Georg Brandl <*****@******.***>
Date: Sat Nov 20 19:54:36 2010 +0000
#9724: add nonlocal to pydoc topics.
Lib/pydoc.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
NOTES: Docs, ignoring.
commit a5b9e5da5858dcbe0b9d517e2d7e3b1b6e591a74
Author: Georg Brandl <*****@******.***>
Date: Fri Nov 26 08:25:00 2010 +0000
Blocked revisions 86479-86480,86537,86550,86608,86619,86725 via svnmerge
........
r86479 | georg.brandl | 2010-11-16 16:15:29 +0100 (Di, 16 Nov 2010) | 1 line
Add stub for PEP 3148.
........
r86480 | georg.brandl | 2010-11-16 16:15:56 +0100 (Di, 16 Nov 2010) | 1 line
Post-release bumps.
........
r86537 | georg.brandl | 2010-11-19 23:09:04 +0100 (Fr, 19 Nov 2010) | 1 line
Do not put a raw REPLACEMENT CHARACTER in the document.
........
r86550 | georg.brandl | 2010-11-20 11:24:34 +0100 (Sa, 20 Nov 2010) | 1 line
Fix rst markup errors.
........
r86608 | georg.brandl | 2010-11-20 20:54:36 +0100 (Sa, 20 Nov 2010) | 1 line
#9724: add nonlocal to pydoc topics.
........
r86619 | georg.brandl | 2010-11-20 23:40:10 +0100 (Sa, 20 Nov 2010) | 1 line
Add error handling in range_count.
........
r86725 | georg.brandl | 2010-11-24 10:09:29 +0100 (Mi, 24 Nov 2010) | 1 line
Remove UTF-8 BOM.
........
NOTES: I snipped these kinds of empty commits above, but this one is small enough that I thought I'd show you what they look like un-snipped.
commit ca50ac7e9e4e3e0dd83ae81f887812107035717a
Author: Georg Brandl <*****@******.***>
Date: Fri Nov 26 09:03:14 2010 +0000
Blocked revisions 86256,86324,86409,86427,86429,86444-86446,86451,86479-86480,86608,86619,86725 via svnmerge
(...snipped...)
NOTES: Snipping this one entirely.
commit f617d0302adc99fdb8a5041141a488f2f75786df
Author: Éric Araujo <*****@******.***>
Date: Fri Aug 19 01:27:00 2011 +0200
Synchronize glossary with py3k.
This update includes new entries that apply to 2.7 too, mention of class
decorators, mention of nonlocal, notes about bytecode, markup fixes and
some rewrappings. Future backports of changes should be slightly
easier.
Doc/glossary.rst | 97 +++++++++++++++++++++++++++++++++++++++++---------------
1 file changed, 71 insertions(+), 26 deletions(-)
NOTES: Docs, ignoring.
commit df042cada442b9d05da096017efc453b04cfd8c0
Author: Éric Araujo <*****@******.***>
Date: Thu Sep 1 18:45:50 2011 +0200
Document that True/False/None don’t use :keyword: in doc.
This was discussed some months ago on python-dev. Having tons of links
to the definition of True would be annoying, contrary to links to e.g.
the nonlocal or with statements doc.
Doc/documenting/markup.rst | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
NOTES: Docs, ignoring.
commit d2ff3bd6fd7f75ef55031a75bc504d3b6d0c1dd5
Author: Mark Dickinson <*****@******.***>
Date: Sun Apr 29 22:18:31 2012 +0100
Issue #14696: Fix parser module to understand 'nonlocal' declarations.
Lib/test/test_parser.py | 10 ++++++++++
Misc/NEWS | 2 ++
Modules/parsermodule.c | 41 ++++++++++++++++++++++++++++++++++++-----
3 files changed, 48 insertions(+), 5 deletions(-)
NOTES: This one is an actual bugfix I need to backport.
commit de94a556a046dcc32643fc0e422da691d147626e
Author: Benjamin Peterson <*****@******.***>
Date: Wed Oct 31 20:26:20 2012 -0400
point errors related to nonlocals and globals to the statement declaring them (closes #10189)
Include/symtable.h | 1 +
Misc/NEWS | 3 +++
Python/symtable.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++--------
3 files changed, 53 insertions(+), 8 deletions(-)
NOTES: This is an enhancement that's worth backporting; it adds additional context to these errors so that the error messages have more detailed information about where the error is.
commit 72d561c9690b6d43797fba65b3f576d51424f4cc
Author: Benjamin Peterson <*****@******.***>
Date: Sat Mar 23 10:09:24 2013 -0500
nonlocal isn't a 2.x topic
Doc/tools/sphinxext/pyspecific.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
NOTES: Docs, ignoring.
commit 58df7b9d23bdf09d8c35011622050c407437bd78
Author: Benjamin Peterson <*****@******.***>
Date: Tue Dec 29 10:08:34 2015 -0600
make recording and reporting errors and nonlocal and global directives more robust (closes #25973)
Lib/test/test_syntax.py | 8 ++++++++
Misc/NEWS | 3 +++
Python/symtable.c | 24 ++++++++++++++++--------
3 files changed, 27 insertions(+), 8 deletions(-)
NOTES: Looks like a bugfix or improvement; need to backport.
commit 47baf563fa8377e746c232a2eb9e91eaec1db335
Author: Guido van Rossum <*****@******.***>
Date: Fri Sep 9 09:36:26 2016 -0700
Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.
Patch by Ivan Levkivskyi.
Doc/reference/simple_stmts.rst | 5 +-
Lib/test/test_syntax.py | 18 ++++++-
Misc/NEWS | 3 ++
Python/symtable.c | 104 ++++++++++++++---------------------------
4 files changed, 59 insertions(+), 71 deletions(-)
NOTES: Reading the attached bug report shows that this introduces a backwards incompatibility, ("x = 3; global x" changes from a SyntaxWarning to a SyntaxError), so I won't backport this. This is actually a nice example of the philosophy behind introducing breaking changes to Tauthon; we don't do it, even in cases like this where it looks like it's not a big deal, and perhaps even a language improvement. Somewhere, somebody's working code will break from this change.
Backporting Python 3's nonlocal implementation
As you can see, there are only a few relevant commits, and the first one contains the meat of the changes. Let's look at the first commit in detail:
commit 8110a89fde7e127e8234d14226521a586aceff0f
Author: Jeremy Hylton <*****@******.***>
Date: Tue Feb 27 06:50:52 2007 +0000
Provisional implementation of PEP 3104.
Add nonlocal_stmt to Grammar and Nonlocal node to AST. They both
parallel the definitions for globals. The symbol table treats
variables declared as nonlocal just like variables that are free
implicitly.
This change is missing the language spec changes, but makes some
decisions about what the spec should say via the unittests. The PEP
is silent on a number of decisions, so we should review those before
claiming that nonlocal is complete.
Thomas Wouters made the grammer and ast changes. Jeremy Hylton added
the symbol table changes and the tests. Pete Shinners and Neal
Norwitz helped review the code.
Grammar/Grammar | 3 +-
Include/Python-ast.h | 10 +-
Include/graminit.h | 101 +--
Include/symtable.h | 21 +-
Lib/test/test_scope.py | 84 +++
Lib/test/test_syntax.py | 40 ++
Parser/Python.asdl | 1 +
Python/Python-ast.c | 31 +
Python/ast.c | 27 +-
Python/compile.c | 1 +
Python/graminit.c | 1598 ++++++++++++++++++++++++-----------------------
Python/symtable.c | 77 ++-
12 files changed, 1130 insertions(+), 864 deletions(-)
This commit looks pretty intimidating at first--a thousand lines each of additions and deletions! But actually, the bulk of the changes (by line count) happen in one file, Python/graminit.c, which is generated based upon Grammar/Grammar by Parser/pgen. (How can you tell? Well, helpfully the first line of Python/graminit.c says so, and you can also find where it happens in the Makefile). In fact, Include/graminit.h is generated as well, in the same way. So the three line change to Grammar/Grammar takes care of most of the changed lines, giving us a much more manageable 295 line diff. Let's take a closer look at the change:
$ git show 8110a89fde7e127e8234d14226521a586aceff0f --oneline -- Grammar/Grammar
8110a89 Provisional implementation of PEP 3104.
diff --git a/Grammar/Grammar b/Grammar/Grammar
index 7606d6e..0277799 100644
--- a/Grammar/Grammar
+++ b/Grammar/Grammar
@@ -39,7 +39,7 @@ vfplist: vfpdef (',' vfpdef)* [',']
stmt: simple_stmt | compound_stmt
simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
- import_stmt | global_stmt | assert_stmt)
+ import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
expr_stmt: testlist (augassign (yield_expr|testlist) |
('=' (yield_expr|testlist))*)
augassign: ('+=' | '-=' | '*=' | '/=' | '%=' | '&=' | '|=' | '^=' |
@@ -63,6 +63,7 @@ import_as_names: import_as_name (',' import_as_name)* [',']
dotted_as_names: dotted_as_name (',' dotted_as_name)*
dotted_name: NAME ('.' NAME)*
global_stmt: 'global' NAME (',' NAME)*
+nonlocal_stmt: 'nonlocal' NAME (',' NAME)*
assert_stmt: 'assert' test [',' test]
compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef
Grammar/Grammar describes Python's grammar. As you can see, this change is quite simple and easy to understand on a high level: We're adding a new kind of "small statement", which is a kind of statement that includes things like import or assert statements. Our "nonlocal" statement has the exact same form as the "global" statement, i.e., it's the word "nonlocal" followed by one or more comma-seperated variable names. As we'll see, the similarity of nonlocal's grammar and global's grammar will actually be helpful when updating some of other files in nonlocal's implementation. Many times the code for nonlocal is exactly the same as for global, but with "nonlocal" instead of "global".
The Grammar/Grammar files for Python 2 and Python 3 are quite similar, so it's easy to make the corresponding Grammar/Grammar changes in Tauthon. The only difference in the relevant lines is that Python 2 has an "exec" statement that was removed in Python 3, (it became a built-in function).
Of the remaining files in the initial nonlocal implementation, two of them are tests (Lib/test/test_scope.py and Lib/test/test_syntax.py), which are easy to port. Of the actual interpreter changes, I typically focus on the header files (*.h) first, since that way I can often continually compile Tauthon while I'm working. There are two remaining header files, Include/Python-ast.h and Include/symtable.h. And in fact, Include/Python-ast.h is also generated!
The changes to Parser/Python.asdl are what create the changes to this file, (as well as to Python/Python-ast.c):
$ git show 8110a89fde7e127e8234d14226521a586aceff0f -U50 --oneline -- Parser/Python.asdl
8110a89 Provisional implementation of PEP 3104.
diff --git a/Parser/Python.asdl b/Parser/Python.asdl
index c5b64a9..3dc3c60 100644
--- a/Parser/Python.asdl
+++ b/Parser/Python.asdl
@@ -1,86 +1,87 @@
(...snipped...)
stmt = FunctionDef(identifier name, arguments args,
stmt* body, expr* decorators, expr? returns)
| ClassDef(identifier name, expr* bases, stmt* body)
| Return(expr? value)
(...snipped...)
| Global(identifier* names)
+ | Nonlocal(identifier* names)
| Expr(expr value)
| Pass | Break | Continue
(...snipped...)
Again, the change is pretty straightforward: we're adding a new statement type, analogous to global statements. The statement operates on a list of identifiers, (i.e. variable names).
What's the difference between Grammar/Grammar and Parser/Python.asdl? As I understand it, Grammar/Grammar describes the possible sequences of tokens that are valid Python, while Parser/Python.asdl describes Python's internal Abstract Syntax Tree (AST). More detail in the Python devguide.
In any case, the AST for Python 2 is also very similar to that for Python 3, so it's trivial to backport this change. We're doing pretty darn well so far; we've only changed four lines, and it's already changed six files (half of the original twelve) and the great bulk of the ~2000 changed line of code!
I'm going to stop this post here, since it's already a lot of content for one blog post. (Although how I managed to write this much to describe four lines of code is a bit of mystery). Let's take stock of the files we've discussed so far:
Grammar/Grammar | 3 +- (Already discussed) Include/Python-ast.h | 10 +- (generated by Parser/Python.asdl) Include/graminit.h | 101 +-- (generated by Grammar/Grammar) Include/symtable.h | 21 +- Lib/test/test_scope.py | 84 +++ (testing code; easy to port) Lib/test/test_syntax.py | 40 ++ (testing code; easy to port) Parser/Python.asdl | 1 + (Already discussed) Python/Python-ast.c | 31 + (generated by Parser/Python.asdl) Python/ast.c | 27 +- Python/compile.c | 1 + Python/graminit.c | 1598 ... (generated by Grammar/Grammar) Python/symtable.c | 77 ++-
In my next post on nonlocal, I'll discuss the remaining changes in this commit, to the symtable, the compile.c, and ast.c. Then I'll talk about backporting ten years of bugfixes for nonlocal back to Tauthon, as well as my discovery of a small bug in Python 2.7 and my submission of an upstream patch for it. And then finally I'll show how I was able to make nonlocal not be a keyword in Tauthon, so that Python 2.7 code like "nonlocal = True" will continue to work. Until then! :-)
