MISSING DATA
In real-life scenarios data is seldom complete. How does pandas handle missing data? Generally, it uses two null values, None and NaN, the latter standing for Not a Number. The problem with None, though, is that it’s a Python type. To be precise, its dtype is object. If you use it in a numpy array along with ints, where all elements must be of the same type, the nearest common type for int and None will be object:
import numpy as np
import pandas as pd
nums = np.array([1, 2, 3, None, 4, 5])
nums
It takes much more time to process objects than ints. So, what about the other null value, NaN? Well, the dtype of NaN is float64 and it’s defined in numpy:
nums = np.array([1, 2, 3, np.nan, 4, 5])
nums, nums.dtype
If used in operations, the best common type for NaN and any other type is NaN:
np.nan + 10
5 * np.nan
What about aggregates? Let’s see. We’re going to use the nums array with the NaN value defined a moment ago:
# minimum value
nums.min()
# maximum value
nums.max()
# sum
nums.sum()
It works, so we don’t get an error. But we don’t get anything useful at all, to be exact. Fortunately, there are methods in numpy that ignore NaN values and take only the values that are not NaN into account:
# minimum value
np.nanmin(nums)
# maximum value
np.nanmax(nums)
# sum
np.nansum(nums)
TESTING FOR NULL VALUES
In practice None and NaN are used in a similar way. There are a couple operations that we can perform on data with null value. The first thing we can do is test for null values. The two methods we can use to this end are isnull and notnull, with pretty self-explanatory names. They both return a mask over the data. Let’s check them out:
# let's create a Series object from the nums array
num_series = pd.Series(nums)
# and now let's use the isnull method
num_series.isnull()
# and now the notnull method
num_series.notnull()
# and now let's use the masks to index the data with the isnull method
num_series[num_series.isnull()]
# and with the notnull method
num_series[num_series.notnull()]
DROPPING NULL VALUES
You can easily remove null values from a Series object using the dropna method:
num_series.dropna()
With a DataFrame the problem is that default a whole row or column is removed if at least one value in that row or column is a null value. Let’s create a simple DataFrame object to demonstrate it:
df = pd.DataFrame([[4, 3, np.nan],
[7, 8, 9],
[9, np.nan, 7]])
df
As you can see, there are NaN values in two rows and in two columns. So, let’s use the dropna method to see how it works:
df.dropna()
As you can see, the first and last rows have been removed. This is because the method removes all rows with null values by default. If you want it to remove columns with null values instead, you just have to specify the axis:
df.dropna(axis=1)
# Alternatively, you can use the explicit name of the axis.
df.dropna(axis='columns')
If there are lots of rows and columns and quite a few null values, it may happen that most or even all of them will be removed. This way we’ll lose most valuable data as well. This is why we can use the parameter how (which is by default set to ‘any’) with the value ‘all’ if we want only those rows or columns to be removed where all values are null.
Let’s modify our DataFrame a bit so that the second column contains only null values:
df[1] = np.nan
df
Now we’ll remove only the columns where all values are null:
df.dropna(axis='columns', how='all')
There’s also the thresh parameter which you can use to set a minimum number of null values in a row or column. Only rows and and columns with minimum that number of null values will be removed. So, here’s how we can remove rows with minimum 2 null values, which in our case means just the first row:
df.dropna(thresh=2)
FILLING NULL VALUES
Sometimes you may want to replace null values with some default value like 0, or by an interpolated or repeated value. This is where the fillna and interpolate methods comes in handy. Let’s have a look at our Series and DataFrame objects again:
num_series
df
Let’s have a look at these two methods in action:
# Let's fill all null values in the DataFrame with zeros.
df.fillna(0)
# Let's propagate the previous values forward in the Series.
num_series.fillna(method='ffill')
# Let's propagate the next values backward in the DataFrame.
df.fillna(method='bfill')
As you can see in the last example, the values are propagated only if the next value isn’t a NaN itself. This is also true about forward fill that we saw before. Some more examples:
# Let's fill NaN values using the forward fill method and a specific axis.
df.fillna(method='ffill', axis='columns')
# And now let's use interpolated values in the Series object.
num_series.interpolate()
# And finally let's use interpolated values in the Dataframe object with a specified axis (1 is the same as 'columns').
df.interpolate(axis=1)
Learn how to make beautiful GUI apps
in Python using the Kivy framework.
Comprehensive, for Kivy beginners, easy to follow.
Get the book here (PDF) or on Amazon:
ebook / paperback (black and white) / paperback (full color)
Make Awesome Games and Other 3D Apps
with Panda3D and Blender using Python.
Cool stuff, easy to follow articles.
Get the magazine here (PDF).
Learn the basics of Python, including OOP.
with lots of exercises, easy to follow
The course is available on Udemy.
Learn the basics of 3D modeling in Blender.
step-by-step, easy to follow, visually rich
The course is available on Udemy and on Skillshare.
Here’s the video version of this article: