This was very helpful. I'm still trying to wrap my head around generators, iterators, etc., and the step-by-step walkthrough in this article clarified several things for me. Also, interestingly, when I ran sys.getsizeof(square_genexpr) in Python 3.11.4, the result was 208 bytes instead of 104. Running it on other recent Python versions, the results were 120 (v3.7.9), 112 (v3.8.10), 112 (v3.9.2), 104 (v3.10.4).
Thanks for the feedback, I'm really happy to know that the level of detail here is helpful.
That's an interesting observation. I've been using Python since 2006, and I've been going to Python conferences since 2012. It's incredible to see how heavily optimized some sections of the Python codebase are. There are some fantastic videos from the mid 2010s about optimizations that were being made to the way dictionaries are implemented. Dictionaries are just under the surface in much of Python, so any optimization there has a lot of compound effects. If you're curious about this, try watching The Dictionary Even Mightier by Brandon Rhodes, from 2017: https://www.youtube.com/watch?v=66P5FMkWoVU.
Most of the time optimizations are made by making things smaller and faster. But there's also a tradeoff where you can make something slightly larger, but make up for that by gaining speed. Here's an excerpt from the Python 3.11 release notes:
---
CPython bytecode changes
The bytecode now contains inline cache entries, which take the form of the newly-added CACHE instructions. Many opcodes expect to be followed by an exact number of caches, and instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data.
New opcodes
ASYNC_GEN_WRAP, RETURN_GENERATOR and SEND, used in generators and co-routines.
(Apparently I'm hitting a length limit on comments)
My interpretation of this is that there's a little caching going on when a generator is defined, and that's taking up a little more space around generator definitions. That extra memory is used to make programs execute even faster.
This interpretation might not be entirely accurate, but I would bet that the increase in memory usage you noticed is related to this change.
This was very helpful. I'm still trying to wrap my head around generators, iterators, etc., and the step-by-step walkthrough in this article clarified several things for me. Also, interestingly, when I ran sys.getsizeof(square_genexpr) in Python 3.11.4, the result was 208 bytes instead of 104. Running it on other recent Python versions, the results were 120 (v3.7.9), 112 (v3.8.10), 112 (v3.9.2), 104 (v3.10.4).
Thanks for the feedback, I'm really happy to know that the level of detail here is helpful.
That's an interesting observation. I've been using Python since 2006, and I've been going to Python conferences since 2012. It's incredible to see how heavily optimized some sections of the Python codebase are. There are some fantastic videos from the mid 2010s about optimizations that were being made to the way dictionaries are implemented. Dictionaries are just under the surface in much of Python, so any optimization there has a lot of compound effects. If you're curious about this, try watching The Dictionary Even Mightier by Brandon Rhodes, from 2017: https://www.youtube.com/watch?v=66P5FMkWoVU.
Most of the time optimizations are made by making things smaller and faster. But there's also a tradeoff where you can make something slightly larger, but make up for that by gaining speed. Here's an excerpt from the Python 3.11 release notes:
---
CPython bytecode changes
The bytecode now contains inline cache entries, which take the form of the newly-added CACHE instructions. Many opcodes expect to be followed by an exact number of caches, and instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data.
New opcodes
ASYNC_GEN_WRAP, RETURN_GENERATOR and SEND, used in generators and co-routines.
https://docs.python.org/3/whatsnew/3.11.html#cpython-bytecode-changes
---
(Apparently I'm hitting a length limit on comments)
My interpretation of this is that there's a little caching going on when a generator is defined, and that's taking up a little more space around generator definitions. That extra memory is used to make programs execute even faster.
This interpretation might not be entirely accurate, but I would bet that the increase in memory usage you noticed is related to this change.
Interesting. I had heard that 3.11 was significantly faster than 3.10, so a tradeoff in memory usage makes a lot of sense.
Thanks for the Brandon Rhodes video. I'm a big fan of his talks and haven't seen that one yet. Added to my watchlist.