In Python, generators are a simple way of creating iterators. Generators allows us to iterate over a set of values without needing to create a list in memory. Generators are created using functions and the yield keyword instead of return.
Generators in Python
Generators in Python provide a powerful and efficient way to work with data streams or large datasets without loading everything into memory at once. Generators allows us to iterate over values lazily, meaning each value is produced only when needed. This makes generators both memory-efficient and convenient for a variety of use cases.
Generators are a type of iterable, like lists or tuples. But unlike lists, generators don’t store all their values in memory. Instead, they generate values one at a time, which is particularly useful when dealing with large datasets or streams of data. The key feature of a generator is that it yields a value and pauses its state so that it can resume later to yield the next value.
Create Generators in Python
There are two main ways to create generators in Python.
- Generator Functions using the
yieldkeyword. - Generator Expressions that resemble list comprehensions but use parentheses instead of square brackets.
Generator Function Example
# Generator Function Example
def simple_function_generator(): # function to define generator
yield "1- One"
yield "2 - Two"
yield "3 - Three"
gen_object = simple_function_generator() # object of generator
# Iterating through the generator
for value in gen_object: # iterate over the generator values
print(value)
# Output
# 1- One
# 2 - Two
# 3 - Three- In this example, the function
simple_function_generator()produces values one at a time, returning them viayield. gen_object = simple_function_generator()– This creates an object reference to the generator function- Using
forloop to iterate through the values of generator. Generator generates values one at a time.
Using yield vs return
Generators are created using functions and the yield keyword instead of return.
return=> Terminates the function and returns a value.yield=> Pauses the function, saving its state, and returns a value. When resumed, execution starts from where it was paused.
# Example of yield
def count_down_generator(n): # function to define generator
while n > 0:
yield n
n -= 1
for value in count_down_generator(5): # iterate over the generator values
print(value)- In this example, the function
count_down_generator()produces count down values one at a time, returning them viayield. yieldis used to produce values from a countdown without terminating the function.
Program Output
5
4
3
2
1Using for loop to create and iterate through Generators
We can use for loop to generate values for a generator and once generator is created, we can use a for loop to iterate through its values.
# Using for loop to create and iterate through Generators
def generate_even_numbers(limit): # function to define generator
for num in range(2, limit+1, 2): # iterate to get next even number
yield num # yield number
for even in generate_even_numbers(20): # iterate over the generator values
print(even, end=" ")
# Output => 2 4 6 8 10 12 14 16 18 20 - In this example, the function
generate_even_numbers()produces even numbers using for loop returning them viayield. - In generator function, we can also perform any operation (like normal function) before yielding the next number.
for even in generate_even_numbers(20)=> Using for loop, we can iterate through the generator values and perform operations on them.
Using Generators with next()
We can manually control the flow of values using the next() function, which retrieves the next value from the generator.
# Using Generators with next()
def generate_even_numbers(limit): # function to define generator
for num in range(2, limit+1, 2): # iterate to get next even number
yield num # yield number
gen_object = generate_even_numbers(20) # object - reference to the generator function
print(next(gen_object)) # Output => 2 # using next() - get next value from generator
print(next(gen_object)) # Output => 4 # using next() - get next value from generator
print(next(gen_object)) # Output => 6 # using next() - get next value from generator
# similarly we can get other value and can control the flow of values from generator- In this example, the function
generate_even_numbers()produces even numbers using for loop returning them viayield. gen_object = generate_even_numbers(20)=> This creates an object reference to the generator function- Using
next(gen_object), we can get the next value from the generator and can control the flow of values from generator. - If the generator has no more values, calling
next()raises aStopIterationexception.
Benefits of Generators
There are multiple benefits of using generators.
- Memory Efficiency – Generators do not store all their values at once. They generate each value only when needed. Because of lazily access of value, it does not consume high memory at once and does not load the system. This is very helpful in efficient and on-demand use of memory.
- Improved Performance – Generators are particularly useful for large datasets because generators does not create values in advance and does not store previously generated values. Creating a full list in advance would consume too much memory and will slow down the execution of program.
- Infinite Sequences – We can use generators to represent infinite data streams without worrying about memory consumption.
Example of an Infinite Generator
# Example of an Infinite Generator
def generate_infinite_count(start=0): # function to define generator
while True: # loop always True
yield start # yield number
start += 1
gen_object = generate_infinite_count() # object - reference to the generator function
# Printing the first 10 values of an infinite generator
for _ in range(10):
print(next(gen_object), end=" ") # Output => 0 1 2 3 4 5 6 7 8 9 Generator – Real-world Use Cases
There are multiple real-world use cases in which generators are used.
Reading Large Files Line by Line
Using generators to read large files allows us to handle large datasets without loading the entire file into memory.
# Reading Large Files Line by Line
def read_large_file(file_name): # function to read large file
with open(file_name) as f: # open file in read and text mode
for line in f:
yield line.strip() # yield line from the file
for line in read_large_file("large_file.txt"):
print(line) # print yielded line from the fileStreaming Data – Generators are excellent for streaming data from APIs or databases where we don’t want to load everything into memory at once.
Generator Expressions
Generator expressions are a shorthand for creating generators. They are similar to list comprehensions, but they use parentheses () instead of square brackets [].
# list comprehension - each term multiply by 2
square_number_list = [item*2 for item in range(5)] # list comprehension using square brackets []
print(square_number_list) # print list elements
# generator expression - surrounded by parenthesis ()
generator_object = (item*2 for item in range(5)) # generator expression using paranthesis ()
print(generator_object) # print generator object
print(next(generator_object)) # next value from generator
print(next(generator_object)) # next value from generator
print(next(generator_object)) # next value from generatorProgram Output
[0, 2, 4, 6, 8]
<generator object <genexpr> at 0x00000211D4EF3850>
0
2
4Generator expressions are often more memory-efficient than list comprehensions, especially with large datasets.
Generator Pipelines
We can chain multiple generators together to create a pipeline of data transformations. This is particularly useful in data processing workflows.
# Generator Pipelines
def generate_integers(): # generator function for integers
for i in range(1, 11):
yield i # yield integer values
def generate_squared_numbers(numbers): # generator function for square numbers
for number in numbers:
yield number ** 2 # yield square values
def generate_even_numbers(numbers): # generator function for even numbers
for number in numbers:
if number % 2 == 0:
yield number # yield even numbers
# Chaining 3 generators together
pipeline = generate_even_numbers(generate_squared_numbers(generate_integers()))
for value in pipeline:
print(value, end=" ") # Output => 4 16 36 64 100 - In this example, we have three generator functions
generate_integers()– This is used to yield numbers using the range function.generate_squared_numbers(numbers)– This takes the number as an argument and yield square of that number.generate_even_numbers(numbers)– This check for the number is even or not. yield is number is even number.pipeline = generate_even_numbers(generate_squared_numbers(generate_integers()))=> We have created a pipeline using all three generator functions.- Using
forloop, we can iterate through the generated values and print them in console.
Summary
Generators in Python are a powerful tool for handling data streams and large datasets efficiently. They allow us to produce values lazily, meaning they compute each value on demand rather than holding everything in memory at once. Whether we are processing large files, working with APIs, or building data pipelines, generators can help us write more efficient and scalable code.
In this article, we learned about Generators in Python. Following topics were covered:
Code – Github Repository
All code snippets and programs for this article and for Python tutorial, can be accessed from Github repository – Comments and Docstring in Python.