Python Remove Duplicates from a List

Removing duplicates from a list in Python

There are several ways to python remove duplicates from list inputs depending on whether you need to preserve order, handle unhashable items, or optimize for speed and memory. Below are common, practical approaches with short explanations and examples so you can pick the best method for your use case.

Method 1 — Manual loop with temporary list (preserve order)

This simple approach keeps the first occurrence of each element and preserves order. It works for any element that supports equality checks.

items = [1, 2, 3, 2, 4, 3]
result = []
for x in items:
    if x not in result:
        result.append(x)
print(result)  # [1, 2, 3, 4]

Method 2 — Using set() (fast, unordered)

Converting a list to a set quickly removes duplicates because sets only store unique elements. Note that a set does not preserve insertion order, so this method is best when order doesn't matter.

items = [1, 2, 3, 2, 4, 3]
unique = list(set(items))
print(unique)  # order may vary, e.g. [1, 2, 3, 4]

Method 3 — dict.fromkeys (preserve order & concise)

In modern Python (3.7+), dictionaries preserve insertion order. dict.fromkeys creates keys from the iterable, which effectively removes duplicates while keeping the first occurrence.

items = [1, 2, 3, 2, 4, 3]
unique = list(dict.fromkeys(items))
print(unique)  # [1, 2, 3, 4]

Method 4 — List comprehension with a seen set (preserve order, efficient)

This pattern is a common way to remove duplicates from list while preserving order and giving good performance for large lists.

items = [1, 2, 3, 2, 4, 3]
seen = set()
unique = [x for x in items if not (x in seen or seen.add(x))]
print(unique)  # [1, 2, 3, 4]

Method 5 — collections.OrderedDict (compatibility for older Python)

Before insertion-order was guaranteed in dict, OrderedDict provided a stable way to deduplicate while preserving order.

from collections import OrderedDict
items = [1, 2, 3, 2, 4, 3]
unique = list(OrderedDict.fromkeys(items))
print(unique)  # [1, 2, 3, 4]

When not to use list.count or remove during iteration

Using count() combined with remove() while iterating the same list is error-prone and can skip or keep duplicates unexpectedly. Avoid patterns that mutate the list while looping over it.

# Not recommended
values = [1, 2, 2, 3]
for v in values:
    if values.count(v) > 1:
        values.remove(v)
# results can be unpredictable

Removing duplicates from lists of unhashable items (lists, dicts)

Lists and dictionaries are unhashable and can't be put directly into a set. Convert them to a hashable representation first:

• List of lists: convert inner lists to tuples.
• List of dicts: convert each dict to a tuple of items (or use a stable serialization like JSON).

# List of lists
items = [[1,2], [1,2], [2,3]]
unique = [list(t) for t in set(tuple(x) for x in items)]
print(unique)  # order not preserved

# List of dicts (preserve first occurrence)
import json
items = [{'a':1}, {'a':1}, {'b':2}]
seen = set()
unique = []
for d in items:
    key = json.dumps(d, sort_keys=True)
    if key not in seen:
        seen.add(key)
        unique.append(d)
print(unique)  # [{'a':1}, {'b':2}]

Choosing the best approach

Use these guidelines when deciding how to python remove duplicates from list:

• If order doesn't matter and elements are hashable: use set() for simplicity and speed.
• If you must preserve the first occurrence: use dict.fromkeys or the list-comprehension with a seen set.
• For unhashable elements: transform them into hashable forms (tuple, JSON string) or use a manual loop with membership checks.
• Avoid mutating the list while iterating and avoid count()-based removal for reliability.

Additional scenarios

For arrays (e.g., when using numpy), use specialized functions like numpy.unique for numeric arrays. For lists of tuples, tuples are hashable and can be deduplicated with set or dict.fromkeys. These patterns cover python remove duplicates from array and python unique list needs.

Summary

There is no single best answer for how to remove duplicates from a list in python. For most cases where you want to preserve order and keep the first occurrence, dict.fromkeys or the seen-set list comprehension are the recommended, white-hat solutions. Use set() for speed when order is irrelevant. For complex, unhashable items, transform them to a hashable representation or apply a manual deduplication loop.