A Curious Comprehension

10/01/20

The Oddity

For the first real post of the decade let’s start with a coding curiousity.

Why do these two functions behave differently?

def expectation():
    return [e for e in [(lambda: i)() for i in range(10)]]

def oddity():
    return [e() for e in [(lambda: i) for i in range(10)]]

You can almost certainly anticipate what the first, expectation function would return:

>>> expectation()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

And if you could anticipate what the oddity should return, then this blog entry might be a waste of your time.

>>> oddity()
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

However, if you were surprised by this, as I was, then we both have an opportunity to learn something new about the nuances of Python.

The instinctive next step is to inspect the lambdas before invoking them. We can peel back the comprehension one level with a new function:

def oddity_functions():
    return [(lambda: i) for i in range(10)]

Then we can inspect the returned functions:

>>> print('\n'.join([str(e) for e in oddity_functions()]))
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D898>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D8E0>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D928>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D970>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D610>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117DA00>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D658>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D9B8>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117D5C8>
<function oddity_functions.<locals>.<listcomp>.<lambda> at 0x0117DA48>

The oddity is creating unique lambdas which all return 9, which is a clear indication that they are bound to the last value of i. The relevant question is why?

The Explanation

After an afternoon of research, I have concluded that it has to do with Python's use of names. Let's run the Python REPL and run an experiment.

>>> def f():
... return x
...

We have not defined anything with the name x at this point, and yet there is no error! This may be obvious to some of you, but it was a subtle point to me that we do not encounter issues until we invoke f.

>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
NameError: name 'x' is not defined

Let's define x and invoke the function again.

>>> x = 1729
>>> f()
1729

Now the behavior should seem more intuitive than before: the function f returns the variable with the name x.[1]

In fact, if we re-assign x, we find that f returns the new value of x:

>>> x = 7920
>>> f()
7920

We can now apply this logic to our original problem.

>>> [e() for e in [(lambda: i) for i in range(10)]]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

We'll seperate out the inner comprehension and take a few liberties to better illustrate the issue:

i = 0
inner_comprehension = []
while i < 10:
    inner_comprehension.append(lambda: i)
i += 1

[e() for e in inner_comprehension]

If we unroll the loop the issue becomes even more obvious:

i = 0
inner_comprehension = []
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
inner_comprehension.append(lambda: i)
i += 1
# [...]

In our earlier example with f() and x this would be equivalent to:

x = 0
def f():
    return x
x = 1
def g():
    return x
x = 2
def h():
    return x
x = 3
def i():
    return x
x = 4
def j():
    return x
x = 5
# [...]

Every function is identical, as they all look up x and return it. The same is true in our oddity, where every lambda returns i, and the iteration through range simply updates i.

Thinking in these terms makes our original code much more palatable.

def expectation():
    return [e for e in [(lambda: i)() for i in range(10)]]

def oddity():
    return [e() for e in [(lambda: i) for i in range(10)]]

In the expectation case we are making functions which will return i based on the scope of the inner comprehension, In the oddity case we are invoking the functions as i takes values from 0 to 10 (exclusive).

Now for a more difficult problem: can we modify values in the scope of the inner comprehension? In our case this would involve modifying i and seeing those changes reflected in the lambdas.

In Python 2 list comprehensions have a "leaky" scope, so this would be trivial, but in Python 3 it may not be possible. If I do find a way to accomplish this in Python 3 I will follow up this entry with my findings.

Footnotes

[1] Of course, there are scoping rules which would affect this in a more complex context, but for our purposes we're working in the singular context of the REPL.

The Next Ten Years

31/12/19

It will soon be 2020 and I feel that the next decade will pass as quickly as the last year has. I would like to decide on the kind of person I want to become and the kind of things I want to do in the next ten years.

First, I do not want to be embarrassed by the things I do in the next decade. I will not change the kinds of things I do, instead, I will decide not to be embarrassed by them. I will start with this entry: I might have been embarrassed by this kind of journaling in another decade, but I will not be embarrassed by it in this one.

Second, I want to spend less time “working” and more time doing things which interest me. I am concerned that I have already spent so much time working that it is my most prominent memory of all the years passed. Yet the most rewarding activities are those which I never considered to be “work”. Even something as indolent as the daydreams I would have en route to my university’s campus proved valuable as they became the basis of programming projects which spanned months and taught me more than any class I took at the university. Paradoxically, the most intensive work I have done seems to have been the least productive: when I crammed for exams all the things I learned were forgotten soon afterwards, yet when I read about the same topics in my spare time I enjoyed them and still remember the most minute details. Those hours I spent learning – for no other reason than to learn – are some of my most cherished memories, and there are many more things I enjoyed but avoided because I thought they were not effective uses of my time. Mountain biking, video games, and sports, are all among the victims of my obsession with productivity. I particularly regret losing mountain biking, as it let me find the edge of my comfort zone in the drops and berms, and I had a chance to step beyond it. However, even if I recognized the value of mountain biking earlier, I would have had to compute the logistics – where would I practice? Who would drop me off? What if I got hurt? These questions daunted me as they did not have an obvious answer.

Which is why I want to become dauntless in the face of uncertainty. I do not want to do this through courage – on some days I will wake up brave and other days I will not – I want to do this as though there is no other possibility. If I want to go mountain biking, I should take my bike, get to a trail, and not worry about what the next steps would be. This would not be an act of bravery; it would be an act of idiocy, as there are valid concerns about what I should do if I get injured, or lost, or worse, but if I think about these things too seriously, then I might not do anything at all. I have missed out on too many things because I scared myself out of them by obsessing about the details. Instead, I will begin with the instinctual first step, knowing that the latter steps will have my attention when they deserve it.

And so, this entry is my instinctual first step into the next decade. I have so many ideas which have been burning a hole in pocket for years, and the few ideas I have pursued have rewarded me infinitely. I want to pursue more of those ideas without regard for embarrassment, without worry about productivity, and without thinking beyond the first step. I aim to succeed splendidly but am also ready to fail spectacularly. Over the next ten years I want to build more than I have before.