Python Generator Iterator and Decorator

December 23, 2024 · 10 min read · Page View:

If you have any questions, feel free to comment below.

This article will review the basic knowledge of Python, including generator, iterator, and decorator.

CH9 Generator, Iterator, and Decorator #

Generator #

ls = [ i**2 for i in range(1, 1000001)]
for i in ls:
    pass

Disadvantage: Occupying a lot of memory

Use lazy calculation
Do not need to store a large amount of data at once
Calculate on the fly, only calculate the value needed each time
Actually always executing the next() operation until there is no value left

Generator Expression #

Do not need to store all data

sum((i for i in range(101))) # Sum, inside a generator
# 5050

Generator Function `yield` #

Generate Fibonacci sequence

def fib(max):
    n, a, b = 0, 1, 1
    while n < max:
        print(a)
        a, b = b, a + b
        n = n + 1

Construct generator function, execute each time when next() is called, return when encountering yield statement, continue execution from the yield statement last returned.

Difference:

A normal function executes all code in the function body at once when called, returning a result (if there is a return value) and then ends.
The function containing yield (i.e., the generator function) executes until yield, returns a value, and then continues execution multiple times based on needs, possibly returning new values each time, until the function is executed to completion (e.g., when reaching the end of the function or encountering a return statement). It seems like the function has memory effect.

def fib(max):
    n, a, b = 0, 1, 1
    while n < max:
        yield a
        a, b = b, a + b
        n = n + 1
 
fib(10)
# <generator object fib at 0x000001BE11B19048>
for i in fib(10):
    print(i)
# 1
# 1
# 2
# 3
# 5
# 8
# 13
# ...

Iterator #

Iterable #

Objects that can be directly used in for loops are collectively referred to as Iterable:

List, tuple, string, dictionary, set, file

Use isinstance() to determine if an object is an Iterable object

from collections import Iterable
isinstance([1, 2, 3], Iterable)
# True

Generator

Generators can not only be used in for loops but also be called by the next() function

squares = (i**2 for i in range(5))
isinstance(squares, Iterable)
# True
print(next(squares))
# Until there is no data to take, throw StopIteration
print(next(squares)) # StopIteration:

Objects that can be called by the next() function and return the next value until there is no data to take are called Iterators: Iterator

Iterator #

Use isinstance() to determine if an object is an Iterator object

Generators are Iterators

from collections import Iterator

squares = (i**2 for i in range(5))
isinstance(squares, Iterator)
# True

List, tuple, string, dictionary, set are not Iterators

isinstance([1, 2, 3], Iterator)
# False

Can create an Iterator by iter(Iterable)

isinstance(iter([1, 2, 3]), Iterator)
# True

for item in Iterable is equivalent to: First get the Iterator of the Iterable by iter() function, then call next() method on the obtained Iterator to get the next value and assign it to item, and the loop ends when encountering the StopIteration exception.

zip enumerate and other functions in itertools are Iterators

x = [1, 2]
y = ["a", "b"]
zip(x, y) # <zip at 0x1be11b13c48>
for i in zip(x, y):
    print(i)
isinstance(zip(x, y), Iterator) # True

numbers = [1, 2, 3, 4, 5]
enumerate(numbers) # <enumerate at 0x1be11b39990>
for i in enumerate(numbers):
    print(i)

isinstance(enumerate(numbers), Iterator) # True

File is an Iterator

with open("测试文件.txt", "r", encoding = "utf-8") as f:
    print(isinstance(f, Iterator)) # True

Iterator is consumable

squares = (i**2 for i in range(3))
for square in squares:
    print(square)
#  0
# 1
# 4
for square in squares:
    print(square)
# Cannot iterate anymore, because it has been exhausted

range() is not an Iterator

Can be called range() as a lazy sequence, it is a sequence, but does not contain any content in memory, but answers questions through calculation.

numbers = range(10)
isinstance(numbers, Iterator) # False
print(len(numbers))   # Has length 10
print(numbers[0])     # Can be indexed 0
print(9 in numbers)   # Can exist calculation True
next(numbers)         # Cannot be called by next() TypeError: 'range' object is not an iterator
# Will not be exhausted

Decorator #

Template #

Talk is cheap, just show the code

def decorator(func):
    # The wrapper is the function to be decorated(must keep the same parameters with the original function)
    def wrapper(*args, **kwargs):
        # Do something before the function is called
        res = func(*args, **kwargs) # Call the original function
        # Do something after the function is called
        return res
    return wrapper # Return the wrapper(if the original function returns value, the wrapper must return the value)

Demand #

(1) Need to add some features to the already developed program (2) Cannot modify the source code of the function in the program (3) Cannot change the calling method of the function in the program

Function Object #

You can assign a function to a variable and call the variable to realize the function of the original function.
Functions can be passed as parameters

def square(x):
    return x**2

print(type(square))      # square is an instance of the function class
# <class 'function'>
pow_2 = square          # It can be understood that this function is given an alias pow 2
print(pow_2(5)) #  25
print(square(5)) #  25

High-order Function #

One of the following is sufficient:

Receives a function as a parameter
Or returns a function

def square(x):
    return x**2

def pow_2(fun):
    return fun

f = pow_2(square)
f(8) # 64
print(f == square) # True

Nested Function #

Define a function inside a function

def outer():
    print("outer is running")
    
    def inner():
        print("inner is running")
        
    inner()

outer()

Closure #

def outer():
    x = 1
    z = 10
    
    def inner():
        y = x + 100
        return y, z
        
    return inner


f = outer()   # In fact, f contains the inner function itself + the environment of the outer function
print(f)
# <function outer.<locals>.inner at 0x000001BE11B1D730>

print(f.__closure__)         # __closure__ property contains information from the outer function
for i in f.__closure__:
    print(i.cell_contents)
#     (<cell at 0x000001BE0FDE06D8: int object at 0x00007FF910D59340>, <cell at 0x000001BE0FDE0A98: int object at 0x00007FF910D59460>)
#     1
#     10

res = f()
print(res)
# (101, 10)

Closure: Function that extends the scope

If a function is defined in the scope of another function and references a variable in the outer function, then this function is called a closure
A closure is an entity composed of a function and its related reference environment (i.e., closure = function + reference environment)
Once a variable with the same name is redefined inside the inner function, it becomes a local variable

def outer_function(x):
    outer_variable = x

    def inner_function(y):
        return outer_variable + y

    return inner_function

closure = outer_function(10)  # x is 10 closure = inner_function

result = closure(5)  # y is 5
print(result)  # 15

def outer():
    x = 1
    
    def inner():
        x = x+100
        return x
        
    return inner


f = outer()             
f()
#     <ipython-input-87-d2da1048af8b> in inner()
#          3 
#          4     def inner():
#     ----> 5         x = x+100
#          6         return x
# UnboundLocalError: local variable 'x' referenced before assignment

nonlocal allows the inner function to modify the closure variable, indicating that it is not an internal variable, and uses the variable of the outer function.

def outer():
    x = 1
    
    def inner():
        nonlocal x
        x = x+100
        return x  
    return inner


f = outer()             
f()
# 1
# 101

LEGB rules #

In order to understand actually how the scope resolution happening in Python, we should analyze the LEGB rule because, it is the sequence of names(variables, functions, objects and so on) that Python resolves in a program.

LEGB stanas for：（L） Local scope （E） Enclosed scope （G） Global scope （B） Built-in scope

So the sequence of the scope resolution is: Local scope -> Enclosed scope -> Global scope -> Built-in scope. In a closure environment, if firstly search the local scope, if not, it will search the enclosed scope, you can refer to this example:

#foo.py
filename = "foo.py"
print(filename)
def call_func(f):
    filename = "foo_call.py"
    print(filename)
    return f()

#func.py
import foo

filename = "func_global.py"
print(filename)
def wrapper():
    filename = "func_enclosed.py" 
    print(filename)
    def show_filename():
        filename = "func_inner.py"
        print(filename)
        return f"filename: {filename}"
    print(foo.call_func(show_filename))

if __name__ == "__main__":
    wrapper()

Guess what the output will be? Can you explain the print sequence?

the main call the foo.call_func first, so the foo will be loaded first, the foo.py will be printed when import foo.
then the func.py will be loaded, the func_global.py will be printed.
Now the wrapper is called, the func_enclosed.py will be printed.
And the call_func is a closure, it contains the f() and its environment. Now execute the closure, so the foo_call.py will be printed.
Now execute the return, actually the return is the show_filename function, so the func_inner.py will be printed. And the filename return is the func_inner.py. (Local scope)

So the output is:

foo.py
func_global.py
func_enclosed.py
foo_call.py
func_inner.py
filename: func_inner.py

Now we delete the func_inner.py, can you guess the filename?

filename: func_enclosed.py

As we can see, the closure contains the show_filename function and its environment, there is no filename inside the function, so it search the env filename is func_enclosed.py. (Enclosed scope)

If we delete the func_enclosed.py, the filename will be func_global.py. (Global scope)

Now if we delete the func_global.py, can you guess the output? Should the filename be foo_call.py?

Absolutely not, because the return is the show_filename function, and there is no filename inside both function and its environment, so the function cannot be executed originally, let alone called any other way. It will just appear the NameError: name 'filename' is not defined.

Hope this example can help you better understand the concepts.

A Simple Decorator #

Implemented with nested functions

import time

def timer(func):
    
    def inner():
        print("inner run")
        start = time.time()
        func()
        end = time.time()
        print("{} function running time: {:.2f} seconds".format(func.__name__, (end-start)))
    
    return inner


def f1():
    print("f1 run")
    time.sleep(1)

f1 = timer(f1) # Contains inner() and the environment of timer, such as the passed parameter func
f1()
# inner run
# f1 run
# f1 function running time: 1.00 seconds

Syntax Sugar #

import time

def timer(func):
    
    def inner():
        print("inner run")
        start = time.time()
        func()
        end = time.time()
        print("{} function running time: {:.2f} seconds".format(func.__name__, (end-start)))
    
    return inner

@timer     # Equivalent to implementing f1 = timer(f1)
def f1():
    print("f1 run")
    time.sleep(1)

Decorate Function with Parameters #

import time


def timer(func):
    
    def inner(*args, **kwargs):
        print("inner run")
        start = time.time()
        func(*args, **kwargs)
        end = time.time()
        print("{} function running time: {:.2f} seconds".format(func.__name__, (end-start)))
    
    return inner


@timer       # Equivalent to implementing f1 = timer(f1)
def f1(n):
    print("f1 run")
    time.sleep(n)

    
f1(2)

Function decorated with return value

import time


def timer(func):
    
    def inner(*args, **kwargs):
        print("inner run")
        start = time.time()
        res = func(*args, **kwargs)
        end = time.time()
        print("{} function running time: {:.2f} seconds".format(func.__name__, (end-start)))
        return res
    
    return inner


@timer     # Equivalent to implementing f1 = timer(f1)
def f1(n):
    print("f1 run")
    time.sleep(n)
    return "wake up"
    
res = f1(2)
print(res)
# inner run
# f1 run
# f1 function running time: 2.00 seconds
# wake up

Decorator with Parameters #

Decorators themselves need to pass some additional parameters

Requirement: Sometimes you need to count the absolute time, sometimes you need to count the absolute time twice

def timer(method):
    
    def outer(func):
    
        def inner(*args, **kwargs):
            print("inner run")
            if method == "origin":
                print("origin_inner run")
                start = time.time()
                res = func(*args, **kwargs)
                end = time.time()
                print("{} function running time: {:.2f} seconds".format(func.__name__, (end-start)))
            elif method == "double":
                print("double_inner run")
                start = time.time()
                res = func(*args, **kwargs)
                end = time.time()
                print("{} function running time: {:.2f} seconds".format(func.__name__, 2*(end-start)))
            return res
    
        return inner
    
    return outer


@timer(method="origin")  # Equivalent to timer = timer(method = "origin")   f1 = timer(f1)
def f1():
    print("f1 run")
    time.sleep(1)
    
    
@timer(method="double")
def f2():
    print("f2 run")
    time.sleep(1)


f1()
print()
f2()
# inner run
# origin_inner run
# f1 run
# f1 function running time: 1.00 seconds
    
# inner run
# double_inner run
# f2 run
# f2 function running time: 2.00 seconds

Understanding closures is key!

9、When does the decorator execute

Execute immediately when decorated, no need to wait for the call

func_names=[]
def find_function(func):
    print("run")
    func_names.append(func)
    return func


@find_function
def f1():
    print("f1 run")
    

@find_function
def f2():
    print("f2 run")
    
# run
# run

for func in func_names:
    print(func.__name__)
    func()
    print()

# f1
# f1 run
# f2
# f2 run

Return to the Source #

The properties of the original function are hidden

import time

def timer(func):
    def inner():
        print("inner run")
        start = time.time()
        func()
        end = time.time()
        print("{} function running time: {:.2f} seconds".format(func.__name__, (end-start)))
    
    return inner

@timer                # Equivalent to implementing f1 = timer(f1)
def f1():
    time.sleep(1)
    print("f1 run")

print(f1.__name__)
# inner

Return to the source

import time
from functools import wraps

def timer(func):
    @wraps(func)
    def inner():
        print("inner run")
        start = time.time()
        func()
        end = time.time()
        print("{} function running time: {:.2f} seconds".format(func.__name__, (end-start)))
    
    return inner

@timer  # Equivalent to implementing f1 = timer(f1)
def f1():
    time.sleep(1)
    print("f1 run")

print(f1.__name__) 
f1()
# f1
# inner run
# f1 run
# f1 function running time: 1.00 seconds

Python Generator Iterator and Decorator

December 23, 2024 · 10 min read · Page View:

CH9 Generator, Iterator, and Decorator #

Generator #

Generator Expression #

Generator Function yield #

Iterator #

Iterable #

Iterator #

Decorator #

Template #

Demand #

Function Object #

High-order Function #

Nested Function #

Closure #

LEGB rules #

A Simple Decorator #

Syntax Sugar #

Decorate Function with Parameters #

Decorator with Parameters #

Return to the Source #

Related readings

Generator Function `yield` #