Drop Duplicates from a Pandas DataFrame

While working with data there can be situations where your dataframe has duplicate rows. Knowing how to remove such rows quickly can be quite handy. In this tutorial, we’ll look at how to drop duplicates from a pandas dataframe through some examples.

The drop_duplicates() function

The pandas dataframe drop_duplicates() function can be used to remove duplicate rows from a dataframe. It also gives you the flexibility to identify duplicates based on certain columns through the subset parameter. The following is its syntax:

df.drop_duplicates()

It returns a dataframe with the duplicate rows removed. It drops the duplicates except for the first occurrence by default. You can change this behavior through the parameter keep which takes in 'first', 'last', or False. To modify the dataframe in-place pass the argument inplace=True.

Examples

Let’s look at some of the use-cases of the drop_duplicates() function through examples –

1. Drop duplicate rows based on all columns

By default, the drop_duplicates() function identifies the duplicates taking all the columns into consideration. It then, drops the duplicate rows and just keeps their first occurrence.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates()
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

In the above example, you can see that the rows with index 1 and 2 have the same values for all the three columns. On applying the drop_duplicates() function, the first row is retained and the remaining duplicate rows are dropped. As a result, the dataframe returned does not have a continuous index. If you want the returned dataframe to have a continuous index pass ignore_index=True to the drop_duplicates() function or reset the index of the returned dataframe.

2. Drop duplicate rows based on certain columns

You can also instruct the drop_duplicates() function to identify the duplicates based on only certain columns by passing them as a list to the subset argument.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates(subset=['Pet', 'Color'])
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
4  Cat   Black  Green

In the above example, we identify the duplicates based on just the columns Pet and Color by passing them as a list to the drop_duplicates() function. With this criteria, rows with index 1, 2, and 3 are now duplicates with the returned dataframe only retaining the first row.

3. Remove duplicates and retain the last occurrence

If you want to retain the last duplicate row instead of the first one pass keep='last' to the drop_duplicates() function.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates(keep='last')
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

In the above example, we retain the last duplicate instead of the first one.

4. Remove duplicates and do not retain any occurrences

If you do not want to retain any of the duplicate rows pass keep=False to the drop_duplicates() function.

import pandas as pd

# create a sample dataframe with duplicate rows
data = {
    'Pet': ['Cat', 'Dog', 'Dog', 'Dog', 'Cat'],
    'Color': ['Brown', 'Golden', 'Golden', 'Golden', 'Black'],
    'Eyes': ['Black', 'Black', 'Black', 'Brown', 'Green']
}

df = pd.DataFrame(data)

# print the dataframe
print("The original dataframe:\n")
print(df)

# drop duplicates
df_unique = df.drop_duplicates(keep=False)
print("\nAfter dropping duplicates:\n")
print(df_unique)

Output:

The original dataframe:

   Pet   Color   Eyes
0  Cat   Brown  Black
1  Dog  Golden  Black
2  Dog  Golden  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

After dropping duplicates:

   Pet   Color   Eyes
0  Cat   Brown  Black
3  Dog  Golden  Brown
4  Cat   Black  Green

In the above example, none of the duplicates are retained.

For more on the pandas dataframe drop_duplicates() function refer to its official documentation.

With this, we come to the end of this tutorial. The code examples and results presented in this tutorial have been implemented in a Jupyter Notebook with a python (version 3.8.3) kernel having pandas version 1.0.5