Drop Duplicates from a Pandas DataFrame

While working with data there can be situations where your dataframe has duplicate rows. Knowing how to remove such rows quickly can be quite handy. In this tutorial, we’ll look at how to drop duplicates from a pandas dataframe through some examples.

The pandas dataframe drop_duplicates() function can be used to remove duplicate rows from a dataframe. It also gives you the flexibility to identify duplicates based on certain columns through the subset parameter. The following is its syntax:

df.drop_duplicates()

It returns a dataframe with the duplicate rows removed. It drops the duplicates except for the first occurrence by default. You can change this behavior through the parameter keep which takes in 'first', 'last', or False. To modify the dataframe in-place pass the argument inplace=True.

Let’s look at some of the use-cases of the drop_duplicates() function through examples –

By default, the drop_duplicates() function identifies the duplicates taking all the columns into consideration. It then, drops the duplicate rows and just keeps their first occurrence.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates()
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

In the above example, you can see that the rows with index 1 and 2 have the same values for all the three columns. On applying the drop_duplicates() function, the first row is retained and the remaining duplicate rows are dropped. As a result, the dataframe returned does not have a continuous index. If you want the returned dataframe to have a continuous index pass ignore_index=True to the drop_duplicates() function or reset the index of the returned dataframe.

You can also instruct the drop_duplicates() function to identify the duplicates based on only certain columns by passing them as a list to the subset argument.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates(subset=['Pet', 'Color'])
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
4  Cat   Black  Green

In the above example, we identify the duplicates based on just the columns Pet and Color by passing them as a list to the drop_duplicates() function. With this criteria, rows with index 1, 2, and 3 are now duplicates with the returned dataframe only retaining the first row.

If you want to retain the last duplicate row instead of the first one pass keep='last' to the drop_duplicates() function.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates(keep='last')
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

In the above example, we retain the last duplicate instead of the first one.

If you do not want to retain any of the duplicate rows pass keep=False to the drop_duplicates() function.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates(keep=False)
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

In the above example, none of the duplicates are retained.

For more on the pandas dataframe drop_duplicates() function refer to its official documentation.

With this, we come to the end of this tutorial. The code examples and results presented in this tutorial have been implemented in a Jupyter Notebook with a python (version 3.8.3) kernel having pandas version 1.0.5


Subscribe to our newsletter for more informative guides and tutorials.
We do not spam and you can opt out any time.