[SOLVED] Confused with Python functions...

Question

Consider this code snippet:

def foo1(x1):
def foo2(x2):
return x1 * x2
return foo2
a = foo1(3)
print(a(4))

# Prints 12 as the output

How does this even work? Here's what I understand from this code:

"""
First, the function object 'foo1' gets initialized in the heap.

<< Q.1) Is 'foo2' initialized during the creation of 'foo1' or only when 'foo1()' gets called? >>

Now, 'foo1' gets called. It returns the reference to the function object 'foo2' and variable 'a' stores it.

<< Q.2) Does the function 'foo2' get created to return 3 * 'x2' or 'x1' * 'x2'? What I mean by this is; consider another code snippet:

foo = lambda x: return t * x
print(foo)
t = 2
print(foo(3))

# Prints 6

Here, the function having its reference stored in the 'foo' variable is created in the heap to return the product of its parameter 'x' and the variable 't', which is created later; a variable named 't' is to be searched and substituted by the interpreter if found later in the code. Continued in the comments --
"""

Answer 1

Calvin Thomas Again... in Python, it's not going to be implemented the way you think and definitely will be different from other languages. So... just be careful about learning the Python way and then thinking this is the way in other languages.

The concept of references and heap may actually be managed by another structure in the C code. It's essentially a layer of abstraction that may behave differently from lower level languages.

In Python... the enclosed variables from the outer function are stored in a special function attribute called __closure__, which is actually where x1 is stored and made accessible to foo2: a.__closure__.

Here are some links that go deeper into this:

https://medium.com/techtofreedom/5-levels-of-understanding-closures-in-python-a0e1212baf6d

https://gist.github.com/DmitrySoshnikov/700292

https://stackoverflow.com/questions/14413946/what-exactly-is-contained-within-a-obj-closure

https://zetcode.com/python/python-closures/

Answer 2

Calvin Thomas - First... let me say... most learners here could learn a thing or two by the way you try to understand a concept and then ask your questions. Thanks for keeping things interesting here for people like me.

What you're asking about will make much more sense when you understand concepts like higher order functions, closures, and decorators.

NOTE: Python's implementation of closures aren't really the same as closures in other languages. But, that doesn't necessarily matter to understand nested functions.

I've included a few links to review.

NOTE: The links I provided aren't for beginners who aren't thinking beyond the surface level understanding. I think you'll be fine as someone who seems to dig beyond the surface.

https://code.sololearn.com/cA23A9a20A16

https://www.programiz.com/python-programming/closure

https://stackoverflow.com/questions/4020419/why-arent-python-nested-functions-called-closures

https://www.python.org/dev/peps/pep-0227/

Answer 3

Calvin Thomas To be honest... you'll likely not be able to piece all this together until you step through the code using a debugger and also reviewing the C source code for Python.

I can say that the mechanics of Python in how it implements variable scope and lexical environments is very different from what I originally presumed. It's far more convoluted and less sophisticated than you might think.

That said, don't let Python be the standard for which you try to understand closures, lexical environments, and scope boundaries. It's a language that appeals to learners and people who aren't trying to be professional software engineers.

Those who try to make it more than that will only be left disappointed when they pull back the curtains and see the mess behind the scenes.

That said, it is fun to dig deeper to see how it all works. 😉

Answer 4

Slick I gotcha. 👌

The last line in your code doesn't actually store the reference to MyNum(4) in the main call stack.

Rather it is only assigned to x2 in foo2.

The stack for foo2 is popped after x2 has gone bye bye 👋🤣.

This happens after the return value is passed into the print function, but before the print function executes.

That's why you see 4 is garbage collected before the output.

You'll probably see what you were expecting if you replace the last line:

print("\nOUTPUT -->", a(MyNum(4)))

with... these lines:

four = MyNum(4)
print("\nOUTPUT -->", a(four))

Answer 5

Omar Nurhusien Idris That link will be great in your activity feed. I'll be happy to share my thoughts there if you use the mention feature. 😉

It's probably best to remove from here to keep it "case closed". 👌

Answer 6

Calvin Thomas low-level languages such as C don't have garbage collector mechanism so they only preserve local scope during function execution... and works as you describe

high level languages such as Python have garbage collector, so create an object local scope wich is keeped while there's still reference to some variable/function inside it... without that mechanism, you cannot return nested created function without allocating memory stuff, as you must do in C ^^

Answer 7

x1 is not destroyed when foo1 return...

when foo1 is called a scope object is created, where is stored x1 and foo2... as foo2 is returned and you store it in a variable, the scope is saved along the foo2 reference.

when foo1 is called a second time, another scope is created by the same way...

when using foo2 reference from scope 1, x1 from scope 1 is refered to foo2

when using foo2 reference from scope 2, x2 from scope 2 is refered to foo2

when no more reference to foo2 from scope 1 exists, both foo2 and its scope (1) are garbage collected

same with foo2 from scope 2 ^^

Answer 8

Thanks David Carroll! I was expecting this, except i didn't know wich variable would be destroyed first. I figured it would all happen after the calls because x1 was passed to foo2()

I was trying to make it clear that the local variables wouldn't be destroyed until they were both returned in foo2(). It did take me by suprise that x1 was destroyed last though, but that's most likely because i didn't follow the program 100% through before running it. Led to me learning too haha!

Answer 9

Tibor Santa I agree... Java isn't dying. It's just dead to me. 🤣😂

Answer 10

https://code.sololearn.com/cGSDi3GO8bMj/?ref=app

Look at when each is destroyed. Even though the 4 was given to foo2() it is destroyed first! while x1 in foo1() only garbage collects once foo2() is complete

Answer 11

Slick I like the demo. It seems to present the expected behavior. Or were you not expecting this output?

Answer 12

Slick You might actually find this thread relevant as we recently discussed this same scenario in C#.

https://www.sololearn.com/Discuss/2803651/?ref=app

Answer 13

This is going a bit offtopic now...
Omar Nurhusien Idris You CAN do a lot with Python in various programming contexts. There are some things it is really good at, and some where it is suboptimal but people still use it. It is just a bit overhyped because of the low entry barrier it provides into the AI/ML space. Yet, I have heard about companies whose business is serious AI-based service, who are switching their codebase from Python to Java because of performance issues.

Java is certainly not dying. Java is not just a language but a large ecosystem, and the JVM is also part of it. Java Virtual Machine is constantly improved and tweaked with new garbage collector options every release.
https://blogs.oracle.com/javamagazine/understanding-the-jdks-new-superfast-garbage-collectors
And many other languages thrive on the success of the JVM, including Kotlin, Scala, Clojure and Groovy.

Answer 14

[Part 1 of 3]
Omar Nurhusien Idris It was a little difficult for me to follow everything you said in your response to me. However, I think I got the gist of your question.

First, it seems that many beginners focus a lot on associating programming languages with their careers.

I suppose it's because many learners believe that the programming language is the most challenging and critical skill for their career and will remain so throughout their careers.

However, the programming language eventually becomes naturally familiar, automatic, and effortless.

You begin to learn new languages and start to choose based on capabilities and preferences rather than on familiarity with human languages.

I suspect that opinions and preferences about one language compared to another will change as one becomes more experienced.

Answer 15

[Part 2 of 3]
Omar Nurhusien Idris Perhaps the appeal of Python for learners starts with it being more similar to grammatical sentence structures. However, logical vs human expressions are very different. Once you become more proficient with thinking abstractly and logically, you may begin to see the efficiencies in other languages over Python.

You'll begin to understand the value of static types, design patterns, coding practices, explicit terminating block delimiters, enhanced scoping, clean coding, SOLID and DRY Principles, etc, etc, etc.

Python is great for simple scripting. But the bigger the project gets, the bigger the challenges become.

This is a huge reason why it's simply not the preferred tool of choice for core enterprise development.

Still... there will likely always be a market for Python. I just think it has a low ceiling for opportunities if that's all you can work with.

I also think many will struggle transitioning from Python to other languages in higher demand.

Answer 16

[Part 3 of 3]
Omar Nurhusien Idris I remember when C# and VB.NET were both released. Many VB6 developers were excited about being able do work on enterprise .NET projects without having to learn C#. For many, the learning curve was steep and they wanted to continue working with their English like syntax.

Over time, a stigma developed about the quality of VB.NET developers compared to C# developers. While this is purely anecdotal, I personally saw that exclusive VB.NET developers implemented their code with less abstractions than C# developers.

This was not due to any limitations with VB.NET.

Rather, I think the less talented developers remained in VB.NET because they were still stuck with syntax challenges and never evolved their logical thought processes beyond explicitly procedural control flow.

So... it became a red flag when looking at candidates with more VB.NET experience.

I can't help but wonder if a similar reputation will exist for developers who primarily work with Python over the years. 🤔

Answer 17

Note: Comment marked as the best to make it appear as the first one.

Keeping this in mind, does this mean that 'foo2' is also created to find the product of its parameter 'x2' and another variable 'x1' whose name is to be searched by the interpreter later while calling the function? If this conclusion is true, then how does the 'foo2' function return the product of parameter 'x2' (i.e., 4) and the to-be-searched-and-replaced-by variable 'x1' if x1 is destroyed by the previous function ('foo1') after it exits? >>

Thank yoy for reading all these 1525 characters.

P.S. << Q.3) Aren't function objects local to the function's scope and destroyed even after there is a reference to it? >>

Answer 18

no more x1 is destroyed than foo2... wich is created at running time, and not the same object at each call...

Answer 19

I think about it like so...

def foo1(x1):
def foo2(x2):
return x1 * x2
return foo2
a = foo1(3)
print(a(4))

You've got foo1, all it does it take an integer and pass it to the function within because they're in the same variable scope. Then, returns the reference to foo2.

i scrapped the variable 'a', and just worked with the raw code:

foo1(3)(4)

foo1(3) stores 3 as an int and returns foo2 so it looks like:

foo2(4) *** while being in the same scope as the other variable.

foo2() returns the product of both variables so

foo2(4) ends up being 12 or 3*4 (when using the outer functions scope to hold the first variable)

Answer 20

Calvin Thomas wrote "Note: Comment marked as the best to make it appear as the first one" and allow you to not give best answer mark? ;P