How to Merge Lists Without Duplicates in Python?

Today, I will explain a tricky topic in Python: merging lists without duplicates. Someone in my machine learning team faced the exact issue while working on a project. In this tutorial, I will explain how to merge lists without duplicates in Python.

To merge two lists in Python without duplicates, you can use the set() method. By converting the concatenated lists into a set, duplicates are automatically removed. Here’s a quick example: merged_list = list(set(list1 + list2)). This method is efficient and straightforward, though it doesn’t preserve the order of elements. For instance, merging list1 = [1, 2, 3] and list2 = [3, 4, 5] results in merged_list = [1, 2, 3, 4, 5].

Merge Lists Without Duplicates in Python

Merging lists without duplicates in Python requires combining data from multiple sources while ensuring that each element appears only once.

There are various methods in Python to merge two lists without duplicates.

Method 1: Using set()

The simplest way to merge two lists and remove duplicates in Python is by converting them into a set. Sets are collections that automatically discard duplicate values.

Syntax

merged_list = list(set(list1 + list2))

Now, let me show you an example.

Example

list1 = [1, 2, 3, 4] list2 = [3, 4, 5, 6] merged_list = list(set(list1 + list2)) print(merged_list) # Output: [1, 2, 3, 4, 5, 6]

This method is quick and easy, but it does not preserve the order of elements.

You can see the exact output in the screenshot below:

Check out Convert a Dictionary to a List in Python

Method 2: Using List Comprehension

List comprehension is a concise way to merge lists while maintaining order and removing duplicates.

Syntax

Here is the syntax.

merged_list = [x for i, x in enumerate(list1 + list2) if x not in (list1 + list2)[:i]]

Example

Let me show you an example.

list1 = [1, 2, 3, 4] list2 = [3, 4, 5, 6] merged_list = [x for i, x in enumerate(list1 + list2) if x not in (list1 + list2)[:i]] print(merged_list) # Output: [1, 2, 3, 4, 5, 6]

This method preserves the order of elements and ensures that each element appears only once.

Here is the output in the screenshot below:

Read Get the Index of an Element in a List in Python

Method 3: Using dict.fromkeys()

Another efficient way to merge lists without duplicates in Python is by using dict.fromkeys(). This method leverages the fact that dictionaries cannot have duplicate keys.

Syntax

merged_list = list(dict.fromkeys(list1 + list2))

Example

Now, let me show you an example.

list1 = ['apple', 'banana', 'cherry'] list2 = ['banana', 'cherry', 'date'] merged_list = list(dict.fromkeys(list1 + list2)) print(merged_list) # Output: ['apple', 'banana', 'cherry', 'date']

This method is both efficient and straightforward, preserving the order of elements.

You can see the output in the screenshot below:

Method 4: Using a Loop

We can also use a loop to merge lists without duplicates in Python.

Syntax

Here is the syntax.

merged_list = [] for item in list1 + list2: if item not in merged_list: merged_list.append(item)

Example

Now, let me show you an example.

list1 = [7, 8, 9] list2 = [9, 10, 11] merged_list = [] for item in list1 + list2: if item not in merged_list: merged_list.append(item) print(merged_list) # Output: [7, 8, 9, 10, 11]

This method is more verbose but clearly explains how the merging process works.

Here is the output in the screenshot below:

Read How to Sum Elements in a List in Python

Method 5: Using itertools.chain

For those who prefer using libraries, itertools.chain can be used to merge lists without duplicates in Python.

Syntax

import itertools merged_list = list(dict.fromkeys(itertools.chain(list1, list2)))

Example

Here is an example.

import itertools list1 = [12, 13, 14] list2 = [14, 15, 16] merged_list = list(dict.fromkeys(itertools.chain(list1, list2))) print(merged_list) # Output: [12, 13, 14, 15, 16]

This method combines the power of itertools.chain and dict.fromkeys() to achieve the desired result.

Conclusion

In this tutorial, I explained how to merge lists without duplicates in Python, such as using sets, list comprehension, dictionaries, loops, libraries like itertools, etc. I hope all the above examples also help you.

You may also like the following tutorials:

• Add Elements in List in Python using For Loop
• Compare Two Lists Python
• Find Mean of a List Python
• Sum a List in Python Without Sum Function