Python Remove Duplicates from a List

Lists are an ordered collection of objects in python. It may happen that you require to remove duplicates from a list. In this tutorial, we’ll look at how to remove these duplicates entries from a python list.

Before we proceed, here’s a quick refresher on python lists – Lists are used to store an ordered collection of items. These items can be any type of object from numbers to strings or even another list. This makes lists are one of the most versatile data structures in python to store a collection of objects. For more, check out our guide on lists and other data structures in python.

How to remove duplicates from a list?

The most intuitive way in python to remove duplicates from a list is to iterate through the list items and append the unique elements to another list. However, there are other ways as well. The following are some of the different ways in which you can remove duplicates from a list:

1. Using a loop

As mentioned above, in this straightforward approach we create an empty list and append the unique elements of the original list to this list as we iterate through them. Example:

# Original list
list1 = [1,2,3,3,4,5,6,6]
# New list to store the unique items
list2 = []
# Iterate through the original list 
for item in list1:
    # using the membership operator in check for the presence of the element in the list
    if item not in list2:
        # append to the new list if not already present
        list2.append(item)

print("The original list:", list1)
print("The new list:", list2)

Output:

The original list: [1, 2, 3, 3, 4, 5, 6, 6]
The new list: [1, 2, 3, 4, 5, 6]

In the above example, the original list list1 contains some duplicate entries. To remove these duplicates, we create an empty list list2 and iterate through the items of the original list. If the item is not present in list2 which we check using the membership operator in, we append that element to list2.

This approach is simple and straightforward. Also, the order of the elements in the original list remains intact in the new list with unique entries.

2. By converting the list to a set

If you’re not concerned about the order of elements in the list, converting the list to a set and then back to a list does the job of removing duplicates pretty quickly. Example:

# Original list
list1 = [1,2,3,3,4,5,6,6]
# New list to store the unique items created via list comprehension
list2 = list(set(list1))
print("The original list:", list1)
print("The new list:", list2)

Output:

The original list: [1, 2, 3, 3, 4, 5, 6, 6]
The new list: [1, 2, 3, 4, 5, 6]

In the above example, the list list1 is first converted to a set and then back to a list. Since a set holds only unique items, the duplicates from the original list get dropped. Now, when we convert the set back to a list, we get a list of the unique elements.

Note: Sets are an unordered collection and it may happen that you get an arbitrary order of elements when converting a list to a set.

3. Using Ordered Dictionary

If the order is important, another way of removing duplicates from a list is to create an ordered dictionary using the elements of the list as keys. A dictionary can only have unique keys hence the duplicates get dropped in the process and since it’s an ordered dictionary, the order of the keys is also maintained. Example:

from collections import OrderedDict

# Original list
list1 = [1,2,3,3,4,5,6,6]
# New list to store the unique items created via list comprehension
list2 = list(OrderedDict.fromkeys(list1))
print("The original list:", list1)
print("The new list:", list2)

Output:

The original list: [1, 2, 3, 3, 4, 5, 6, 6]
The new list: [1, 2, 3, 4, 5, 6]

In the above example, we use OrderedDict imported from the collections library in python. An ordered dictionary is created with its keys as the elements of list1. On converting the ordered dictionary to a list, we get a list with the duplicates removed and order of elements intact.

For a more detailed discussion on the different methods of removing duplicates from a list, check out this Stack Overflow answer.