Merge Two Lists in Python Without Duplicates

In this tutorial, we will look at how to merge (or combine) two lists in Python such that the resulting list does not contain any duplicate elements.

For example, let’s say you have the following two lists –

ls1 = [1, 2, 2, 3]
ls2 = [3, 4, 4, 5]
after combining them the resulting list should look like,
res = [1, 2, 3, 4, 5]

How to combine two lists without duplicates in Python?

Merging two lists in Python without duplicates can be accomplished in a few different ways. In this tutorial, we will cover two different methods to merge two lists without duplicates:

1. Combine the lists using the + operator and then remove the duplicates using a set.
2. Convert the two lists to sets and then take the union of the sets and convert the resulting set to a list.

Let’s now look at both the above methods with the help of some examples.

Method 1 – Concatenate the lists and then remove the duplicates using a set

# lists to be merged
ls1 = [1, 2, 2, 3]
ls2 = [3, 4, 4, 5]

# merge the lists
res = ls1 + ls2
print("The merged list, ", res)

# remove duplicates from the merged list
res = list(set(res))
print("The merged list without duplicates, ", res)

Output:

The merged list,  [1, 2, 2, 3, 3, 4, 4, 5]
The merged list without duplicates,  [1, 2, 3, 4, 5]

Here, we are first combining the two lists, ls1 and ls2 using the + operator which results in a list containing the elements from both lists (including all the duplicate elements). We then use a set to remove the duplicate elements from the resulting list.

Method 2 – Converting the lists to sets and then joining the sets

# lists to be merged
ls1 = [1, 2, 2, 3]
ls2 = [3, 4, 4, 5]

# merge the lists without duplicates
res = list(set(ls1) | set(ls2))

print("The merged list without duplicates, ", res)

Output:

The merged list without duplicates,  [1, 2, 3, 4, 5]

Here, we convert the individual lists to sets and then perform a union operation on the sets using the | operator (you can also use the set union() function). We then convert the resulting set to a list using the list() function.

Let’s now look at another example where we apply both methods.

# lists to be merged
ls1 = [9, 8, 8, 3]
ls2 = [7, 3, 3, 4]

# method 1
res1 = list(set(ls1 + ls2))
print("The merged list without duplicates, ", res1)

# method 2
res2 = list(set(ls1) | set(ls2))
print("The merged list without duplicates, ", res2)

Output:

The merged list without duplicates,  [3, 4, 7, 8, 9]
The merged list without duplicates,  [3, 4, 7, 8, 9]

We get the same (and the correct result) from both methods. Notice that the order of the elements from the original list is not preserved in the resulting list. This is because, we are converting the list to a set (in both methods) and since set is an unordered data structure, the order information is not preserved.

How to merge lists without duplicates and preserve the order of the elements?

You can merge two lists without duplicates and preserve the original order of elements by iterating through the lists. Let’s see an example. We’ll take the same use case as above.

# lists to be merged
ls1 = [9, 8, 8, 3]
ls2 = [7, 3, 3, 4]

# resulting list
res = []

# iterate over ls1
for val in ls1+ls2:
    if val not in res:
        res.append(val)

print("The merged list without duplicates, ", res)

Output:

The merged list without duplicates,  [9, 8, 3, 7, 4]

Here, we iterate over the elements of the concatenated list (which includes) duplicates and then add each element to element to the resulting list only if it is not already present. You can see that the original order of elements is preserved in the resulting list.

Frequently Asked Questions

You might also be interested in –

• Merge Two Lists Alternatively in Python
• Python – Flatten a list of lists to a single list
• Python List Comprehension – With Examples