A Curious Comprehension
The Oddity
For the first real post of the decade let’s start with a coding curiousity.
Why do these two functions behave differently?
return [e for e in [(lambda: i)() for i in range(10)]]
def oddity():
return [e() for e in [(lambda: i) for i in range(10)]]
You can almost certainly anticipate what the first, expectation function would return:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
And if you could anticipate what the oddity should return, then this blog entry might be a waste of your time.
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
However, if you were surprised by this, as I was, then we both have an opportunity to learn something new about the nuances of Python.
The instinctive next step is to inspect the lambdas before invoking them. We can peel back the comprehension one level with a new function:
return [(lambda: i) for i in range(10)]
Then we can inspect the returned functions:
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D898>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D8E0>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D928>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D970>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D610>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117DA00>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D658>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D9B8>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D5C8>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117DA48>
The oddity is creating unique lambdas which all return 9, which is a clear indication that they are bound to the last value of i. The relevant question is why?
The Explanation
After an afternoon of research, I have concluded that it has to do with Python's use of names. Let's run the Python REPL and run an experiment.
... return x
...
We have not defined anything with the name x at this point, and yet there is no error! This may be obvious to some of you, but it was a subtle point to me that we do not encounter issues until we invoke f.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
NameError: name 'x' is not defined
Let's define x and invoke the function again.
>>> f()
1729
Now the behavior should seem more intuitive than before: the function f returns the variable with the name x.[1]
In fact, if we re-assign x, we find that f returns the new value of x:
>>> f()
7920
We can now apply this logic to our original problem.
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
We'll seperate out the inner comprehension and take a few liberties to better illustrate the issue:
inner_comprehension = []
while i < 10:
inner_comprehension.append(lambda: i)
i += 1
[e() for e in inner_comprehension]
If we unroll the loop the issue becomes even more obvious:
inner_comprehension = []
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
# [...]
In our earlier example with f() and x this would be equivalent to:
def f():
return x
x = 1
def g():
return x
x = 2
def h():
return x
x = 3
def i():
return x
x = 4
def j():
return x
x = 5
# [...]
Every function is identical, as they all look up x and return it. The same is true in our oddity, where every lambda returns i, and the iteration through range simply updates i.
Thinking in these terms makes our original code much more palatable.
return [e for e in [(lambda: i)() for i in range(10)]]
def oddity():
return [e() for e in [(lambda: i) for i in range(10)]]
In the expectation case we are making functions which will return i based on the scope of the inner comprehension, In the oddity case we are invoking the functions as i takes values from 0 to 10 (exclusive).
Now for a more difficult problem: can we modify values in the scope of the inner comprehension? In our case this would involve modifying i and seeing those changes reflected in the lambdas.
In Python 2 list comprehensions have a "leaky" scope, so this would be trivial, but in Python 3 it may not be possible. If I do find a way to accomplish this in Python 3 I will follow up this entry with my findings.
Footnotes
[1] Of course, there are scoping rules which would affect this in a more complex context, but for our purposes we're working in the singular context of the REPL.