Today we’ll be talking about the basics of the iteration protocol, and, to be more specific, about the concepts of iterables and iterators, which often get confused. So, iterable vs iterator – what’s the difference?
First of all, what is iteration? Simply put, iteration is scanning objects from left to right. So, if we iterate over a list, we scan all the elements of the list from left to right. A list is an example of an iterable. But we also know other iterables like strings or tuples, and not only sequences. A dictionary, for example, is an iterable as well.
And here we’re getting at the concept of iterables and iterators. These two notions are often used interchangeably, even in literature, but, although related to each other, they do not mean the same thing.
Before we try to define the two terms, let’s have a look at the following example:
>>> colors = ["blue", "green", "red", "yellow", "pink", "brown"]This is a list, which is an iterable. We can use the for loop to iterate over the elements of the list:
>>> for color in colors:
... print(color)
...
blue
green
red
yellow
pink
brown
Learn how to make beautiful GUI apps
in Python using the Kivy framework.
Comprehensive, for Kivy beginners, easy to follow.
Get the book here (PDF) or on Amazon:
ebook / paperback (black and white) / paperback (full color)
Table of Contents
Iterable vs Iterator
But we can also iterate over the elements of the list manually and retrieve one element at a time. The two elements we need are: an iterator and an iterable. We already have an iterable, which is the list, so we need an iterator. So, our first step to take is to make the iterable return an iterator. Each iterable either is an iterator at the same time as being an iterable or has an iterator, which we’ll discuss in a moment. For the time being, all we need to know is that a list is not an iterator, so it has an iterator.
In order to get the iterator from an iterable, we use either the __iter__ method or the built-in iter function on the iterable, like so:
>>> c = iter(colors)
>>> c
<list_iterator object at 0x000001AB9091AA90>
As we can see, c is now a list iterator. So, we can define an iterable as an object that supports the iter call. This will do for now.
Now that we have the iterator, we can use it to manually iterate over the elements of the iterable. To do so, we use either the __next__ method or the built-in next function:
>>> c.__next__()
'blue'
>>> c.__next__()
'green'
>>> next(c)
'red'
>>> next(c)
'yellow'
>>> next(c)
'pink'
>>> next(c)
'brown'
>>> next(c)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Each time we call one of these functions, the following element is returned. When there are no more elements left, the StopIteration error is raised.
Iterators are Exhaustive!
List iterators are exhaustive. It means, the elements get used up as they are returned. We can use the list function to collect all the elements remaining in the iterator at any given time:
>>> colors = ["blue", "green", "red", "yellow", "pink", "brown"]
>>> c = iter(colors)
>>> next(c)
'blue'
>>> next(c)
'green'
>>> list(c)
['red', 'yellow', 'pink', 'brown']
After all the elements are used up, we get an empty list:
>>> colors = ["blue", "green", "red", "yellow", "pink", "brown"]
>>> c = iter(colors)
>>> next(c)
'blue'
>>> next(c)
'green'
>>> next(c)
'red'
>>> next(c)
'yellow'
>>> next(c)
'pink'
>>> next(c)
'brown'
>>> list(c)
[]
Make Awesome Games and Other 3D Apps
with Panda3D and Blender using Python.
Cool stuff, easy to follow articles.
Get the magazine here (PDF).
What is an Iterator?
At this moment we’re ready to define an iterator. An iterator is an object returned by an iterable that supports the next call to return the elements one by one, and raises the StopIteration error when it runs out of the elements.
I mentioned before that there are two kinds of iterables: ones which also are iterators and ones that only have an iterator. As we know, a list is an example of the latter. We can easily test whether an iterable is an iterator at the same time by trying to call the next function on it:
>>> colors = ["blue", "green", "red", "yellow", "pink", "brown"]
>>> next(colors)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'list' object is not an iterator
Let’s have a look at some other iterables then.
Learn the basics of Python, including OOP.
with lots of exercises, easy to follow
The course is available on Udemy.
Iterables That HAVE an Iterator
Iterables that are not iterators themselves include:
– lists, which we just examined
– tuples:
>>> numbers = (1, 4, 6, 8, 9)
>>> next(numbers)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object is not an iterator
>>> n = iter(numbers)
>>> next(n)
1
>>> next(n)
4
>>> next(n)
6
>>> next(n)
8
>>> next(n)
9
>>> next(n)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
– strings
>>> text = "hello"
>>> next(text)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'str' object is not an iterator
>>> t = text.__iter__()
>>> next(t)
'h'
>>> next(t)
'e'
>>> next(t)
'l'
>>> next(t)
'l'
>>> next(t)
'o'
>>> next(t)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
– dictionaries (you iterate over the keys)
>>> students = {"Sarah": 18, "Ben": 19, "Dorothy": 20}
>>> next(students)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'dict' object is not an iterator
>>> s = iter(students)
>>> next(s)
'Sarah'
>>> next(s)
'Ben'
>>> next(s)
'Dorothy'
>>> next(s)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
– ranges:
>>> nums = range(5)
>>> next(nums)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'range' object is not an iterator
>>> n = iter(nums)
>>> next(n)
0
>>> next(n)
1
>>> next(n)
2
>>> next(n)
3
>>> next(n)
4
>>> next(n)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Learn the basics of 3D modeling in Blender.
step-by-step, easy to follow, visually rich
The course is available on Udemy and on Skillshare.
Iterables That ARE Iterators
Iterables that are iterators at the same time include:
– files
Let’s create a text file called workers.txt in the current directory with the following content:
John Smith, 36, programmer
Emma Jenkins, 38, tester
William Harper, 32, designer
Tom Parker, 44, programmer
Now we can use the iterator, which is the file itself, to iterate over the lines in the file. So, we don’t need to use the readline method. The iterator is all we need. We can use it manually like so:
>>> f = open("workers.txt")
>>> next(f)
'John Smith, 36, programmer\n'
>>> next(f)
'Emma Jenkins, 38, tester\n'
>>> next(f)
'William Harper, 32, designer\n'
>>> next(f)
'Tom Parker, 44, programmer'
>>> next(f)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
or we can use a for loop, which uses the iterator automatically:
>>> f = open("workers.txt")
>>> for line in f:
... print(line)
...
John Smith, 36, programmer
Emma Jenkins, 38, tester
William Harper, 32, designer
Tom Parker, 44, programmer
We don’t need to call the iter function, because a file is an iterator already.
– enumerates:
>>> p = enumerate("help")
>>> next(p)
(0, 'h')
>>> next(p)
(1, 'e')
>>> next(p)
(2, 'l')
>>> next(p)
(3, 'p')
>>> next(p)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
– zips:
>>> countries = zip(["Germany", "USA", "Poland"], ["Berlin", "Washington", "Warsaw"])
>>> next(countries)
('Germany', 'Berlin')
>>> next(countries)
('USA', 'Washington')
>>> next(countries)
('Poland', 'Warsaw')
>>> next(countries)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
– maps:
>>> nums = [1, 2, 3, 4, 5]
>>> squares = map(lambda n: n ** 2, nums)
>>> next(squares)
1
>>> next(squares)
4
>>> next(squares)
9
>>> next(squares)
16
>>> next(squares)
25
>>> next(squares)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
– filters:
>>> words = ["good", "bad", "old", "young", "new"]
>>> short_words = filter(lambda word: len(word) < 4, words)
>>> next(short_words)
'bad'
>>> next(short_words)
'old'
>>> next(short_words)
'new'
>>> next(short_words)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Each of the iterators can be used manually, like before, or automatically in a for loop. For example:
>>> symbols = map(lambda s: s[:3], ["Canada", "France", "Spain", "Japan"])
>>> for symbol in symbols:
... print(symbol)
...
Can
Fra
Spa
Jap
I hope the iterable vs iterator problem has been solved for you.
Here’s the video version: