How and when does an embedded function store nonlocals?

Question

Example:

def outer(x):
def inner():
print(x)
return inner

I am wondering: How does the returned 'inner' remember x?

When a function is defined, only the definition line with default args and such is said to be evaluated; but when we call the returned 'inner', its locals will contain x. When and how has that gotten in there?

Answer 1

HonFu There is an answer in that SO link posted by Ulisses Cruz that brilliantly explains how the enclosed variable is persisted and accessible when the returned inner function is eventually called.

https://stackoverflow.com/a/20898085

Essentially, when a free variable is a) defined in the outer function and b) is referenced in the inner function, a func_closure member is appended to the inner function. A property on func_closure called content_cell will have the value of the enclosed variable.

Definitely review that linked SO answer I included here and let me know if you still have questions.

Answer 2

Ulisses Cruz, let me once again try to clarify why that still doesn't make sense to me.

def f():
x = y

Write that in your code, it will fly anytime, no matter if y is defined anywhere in the code or not. And that makes sense, because who knows when and how it will be called, right?

But call the function, then it will raise an error if there isn't really a y anywhere.

The moment make_printer is called, the inner function is not even yet defined. And when it is defined, it seems still not to be checked: you can write x=y again, return that function, call that inner function from outside - and only then you get your error.

As far as I know, the code of a function is initially only checked for SyntaxError, and that was it. But at some point the crucial information - inner accesses a value from envelope - must be gathered somehow in order to decide if it will be a closure or not.

Answer 3

HonFu That is a good question. Based on the output of each cell_contents property of each __closure__ item, it appears only a memory address is stored. I assume the original symbol name is not carried over because the variables in the inner function have also been converted to memory addresses, which would be consistent to what happens during compilation.

Answer 4

HonFu That sounds about right to me.

Here is a quick code demo showing the address of the outer is used in the inner.

https://code.sololearn.com/c3Hf9cCrEgJS/?ref=app

Answer 5

Checkout the first answer to this stackoverflow question:

https://stackoverflow.com/questions/4020419/why-arent-JUMP_LINK__&&__python__&&__JUMP_LINK-nested-functions-called-closures

Answer 6

Thank you, David Carroll, that second answer really pointed more in the direction of what I wanted to know. Playing around with __closure__ actually illuminates the matter some more.

I also found this, which gives more details:

https://stackoverflow.com/questions/14413946/what-exactly-is-contained-within-a-obj-closure

I played around with it a bit and saw that immediately after definition of the inner function, __closure__ can be accessed and will show the references if it is a closure case.

So seemingly together with the definition, there must be enough name checking going on (except I'm missing something) to figure out if free outer-scope names are referenced; and if so, __closure__ will 'take notes'.

Answer 7

David Carroll, thank you, that's an interesting read; but I still feel that my question is not fully clarified.

'When make_printer is called, a new frame is put on the stack with the compiled code for the printer function as a constant and the value of msg as a local. It then creates and returns the function.'

The answer doesn't seem to specify, how and when this happens. To see that a name from an enclosing scope is referred to, the inner function needs to be checked for that, right?

But if embedded functions are not treated differently than free functions, this shouldn't happen with the def.

Is it the func return statement that triggers this? Or is my understanding of def wrong and the logic of referred names is checked already with definition?

Answer 8

David Carroll, in a practical sense, since in Python everything is a reference, and speaking for CPython, would it be correct to sum it up like this:

If a variable of an outer scope is referenced in the body, the function will at def time 'add 1' to the reference count of that variable, so when the outer function returns, the object won't be garbage collected?

Answer 9

HonFu,
I think the 'when' is the part that says: "when make_printer is called".
And the 'how' is the rest.