Skip to Content

Drop Rows with NaNs in Pandas DataFrame

Pandas DataFrame provides a number of powerful features to manipulate tabular data. While working with your data, it may happen that there are NaNs present in it. NaNs are used as a placeholder for missing data and it’s better (and in a lot of cases required) to treat these NaNs before you proceed to your next steps. One of the ways to do it is to simply remove the rows that contain such values. In this tutorial we’ll look at how to drop rows with NaN values in a pandas dataframe using the dropna() function.

The pandas dataframe function dropna() is used to remove missing values from a dataframe. It drops rows by default (as axis is set to 0 by default) and can be used in a number of use-cases (discussed below). The following is the syntax:

df.dropna()

It returns a dataframe with the NA entries dropped. To modify the dataframe in-place pass inplace=True

Examples of some of the use cases are –

By default, the dropna() function drops rows containing any NA value.

import numpy as np
np.random.seed(0)
import pandas as pd

# create a sample dataframe
df = pd.DataFrame(np.random.randint(1,9, (6,3)), columns=['A', 'B', 'C'])
df.iloc[::2,0] = np.nan
df.iloc[::3,1] = np.nan
df.iloc[::4,2] = np.nan
# print the dataframe
print("Before dropping rows:\n", df)

# drop rows with NaNs
df_dropped = df.dropna()
print("\nAfter dropping rows:\n", df_dropped)

Output:

Before dropping rows:
      A    B    C
0  NaN  NaN  NaN
1  1.0  4.0  4.0
2  NaN  8.0  2.0
3  4.0  NaN  3.0
4  NaN  8.0  NaN
5  1.0  1.0  5.0

After dropping rows:
      A    B    C
1  1.0  4.0  4.0
5  1.0  1.0  5.0

In the above example, you can see that using dropna() with default parameters resulted in all the rows that contained any NA values getting dropped. This behavior of the dropna() function is controlled by the parameter how which takes either 'any' or 'all' and is 'any' by default as was the case in the above example.

import numpy as np
np.random.seed(0)
import pandas as pd

# create a sample dataframe
df = pd.DataFrame(np.random.randint(1,9, (6,3)), columns=['A', 'B', 'C'])
df.iloc[::2,0] = np.nan
df.iloc[::3,1] = np.nan
df.iloc[::4,2] = np.nan
# print the dataframe
print("Before dropping rows:\n", df)

# drop rows with NaNs
df_dropped = df.dropna(how='all')
print("\nAfter dropping rows:\n", df_dropped)

Output:

Before dropping rows:
      A    B    C
0  NaN  NaN  NaN
1  1.0  4.0  4.0
2  NaN  8.0  2.0
3  4.0  NaN  3.0
4  NaN  8.0  NaN
5  1.0  1.0  5.0

After dropping rows:
      A    B    C
1  1.0  4.0  4.0
2  NaN  8.0  2.0
3  4.0  NaN  3.0
4  NaN  8.0  NaN
5  1.0  1.0  5.0

In the above example you can see that only the row which had all columns as NaN was dropped. This was because the argument how=all was passed to the dropna() function.

import numpy as np
np.random.seed(0)
import pandas as pd

# create a sample dataframe
df = pd.DataFrame(np.random.randint(1,9, (6,3)), columns=['A', 'B', 'C'])
df.iloc[::2,0] = np.nan
df.iloc[::3,1] = np.nan
df.iloc[::4,2] = np.nan
# print the dataframe
print("Before dropping rows:\n", df)

# drop rows with NaNs
df_dropped = df.dropna(thresh=2)
print("\nAfter dropping rows:\n", df_dropped)

Output:

Before dropping rows:
      A    B    C
0  NaN  NaN  NaN
1  1.0  4.0  4.0
2  NaN  8.0  2.0
3  4.0  NaN  3.0
4  NaN  8.0  NaN
5  1.0  1.0  5.0

After dropping rows:
      A    B    C
1  1.0  4.0  4.0
2  NaN  8.0  2.0
3  4.0  NaN  3.0
5  1.0  1.0  5.0

You can also drop rows that have a certain minimum number of NAs by passing a threshold to the dropna() function. In the above example, we pass the argument thresh=2 to drop only those rows that had 2 or more NaNs.

import numpy as np
np.random.seed(0)
import pandas as pd

# create a sample dataframe
df = pd.DataFrame(np.random.randint(1,9, (6,3)), columns=['A', 'B', 'C'])
df.iloc[::2,0] = np.nan
df.iloc[::3,1] = np.nan
df.iloc[::4,2] = np.nan
# print the dataframe
print("Before dropping rows:\n", df)

# drop rows with NaNs
df_dropped = df.dropna(subset=['B'])
print("\nAfter dropping rows:\n", df_dropped)

Output:

Before dropping rows:
      A    B    C
0  NaN  NaN  NaN
1  1.0  4.0  4.0
2  NaN  8.0  2.0
3  4.0  NaN  3.0
4  NaN  8.0  NaN
5  1.0  1.0  5.0

After dropping rows:
      A    B    C
1  1.0  4.0  4.0
2  NaN  8.0  2.0
4  NaN  8.0  NaN
5  1.0  1.0  5.0

You can use dropna() such that it drops rows only if NAs are present in certain column(s). You can pass the columns to check for as a list to the subset parameter. In the above example, we drop only the rows that had column B as NaN.

For more on the dropna() function check out its official documentation.

With this, we come to the end of this tutorial. The code examples and results presented in this tutorial have been implemented in a Jupyter Notebook with a python (version 3.8.3) kernel having numpy version 1.18.5 and pandas version 1.0.5


Subscribe to our newsletter for more informative guides and tutorials.
We do not spam and you can opt out any time.


Author

  • Piyush is a data scientist passionate about using data to understand things better and make informed decisions. In the past, he's worked as a Data Scientist for ZS and holds an engineering degree from IIT Roorkee. His hobbies include watching cricket, reading, and working on side projects.