How to remove duplicates in two list python

I am trying to remove duplicates from 2 lists. so I wrote this function:

a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]

b = ["ijk", "lmn", "opq", "rst", "123", "456", ]

for i in b:
    if i in a:
        print "found " + i
        b.remove(i)

print b

But I find that the matching items following a matched item does not get remove.

I get result like this:

found ijk
found opq
['lmn', 'rst', '123', '456']

but i expect result like this:

['123', '456']

How can I fix my function to do what I want?

Thank you.

Paolo

18.8k21 gold badges73 silver badges110 bronze badges

asked Aug 12, 2013 at 19:18

michaelmichael

102k114 gold badges239 silver badges342 bronze badges

0

Here is what's going on. Suppose you have this list:

['a', 'b', 'c', 'd']

and you are looping over every element in the list. Suppose you are currently at index position 1:

['a', 'b', 'c', 'd']
       ^
       |
   index = 1

...and you remove the element at index position 1, giving you this:

['a',      'c', 'd']
       ^
       |
    index 1

After removing the item, the other items slide to the left, giving you this:

['a', 'c', 'd']
       ^
       |
    index 1

Then when the loop runs again, the loop increments the index to 2, giving you this:

['a', 'c', 'd']
            ^ 
            |
         index = 2

See how you skipped over 'c'? The lesson is: never delete an element from a list that you are looping over.

answered Aug 12, 2013 at 19:28

0

Your problem seems to be that you're changing the list you're iterating over. Iterate over a copy of the list instead.

for i in b[:]:
    if i in a:
        b.remove(i)


>>> b
['123', '456']

But, How about using a list comprehension instead?

>>> a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]
>>> b = ["ijk", "lmn", "opq", "rst", "123", "456", ]
>>> [elem for elem in b if elem not in a ]
['123', '456']

answered Aug 12, 2013 at 19:20

Sukrit KalraSukrit Kalra

31.4k7 gold badges65 silver badges70 bronze badges

1

What about

b= set(b) - set(a)

If you need possible repetitions in b to also appear repeated in the result and/or order to be preserved, then

b= [ x for x in b if not x in a ] 

would do.

answered Aug 12, 2013 at 19:24

Mario RossiMario Rossi

7,54124 silver badges37 bronze badges

3

You asked to remove both the lists duplicates, here's my solution:

from collections import OrderedDict
a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]
b = ["ijk", "lmn", "opq", "rst", "123", "456", ]

x = OrderedDict.fromkeys(a)
y = OrderedDict.fromkeys(b)

for k in x:
    if k in y:
        x.pop(k)
        y.pop(k)


print x.keys()
print y.keys()

Result:

['abc', 'def', 'xyz']
['123', '456']

The nice thing here is that you keep the order of both lists items

answered Aug 12, 2013 at 19:26

DevLoungeDevLounge

8,2273 gold badges30 silver badges43 bronze badges

or a set

set(b).difference(a)

be forewarned sets will not preserve order if that is important

answered Aug 12, 2013 at 19:22

Joran BeasleyJoran Beasley

105k12 gold badges148 silver badges174 bronze badges

You can use lambda functions.

f = lambda list1, list2: list(filter(lambda element: element not in list2, list1))

The duplicated elements in list2 are removed from list1.

>>> a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]
>>> b = ["ijk", "lmn", "opq", "rst", "123", "456"]
>>> f(a, b)
['abc', 'def', 'xyz']
>>> f(b, a)
['123', '456']

BcK

2,3801 gold badge11 silver badges26 bronze badges

answered Dec 14, 2020 at 12:31

GolvinGolvin

311 silver badge2 bronze badges

One way of avoiding the problem of editing a list while you iterate over it, is to use comprehensions:

a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]
b = ["ijk", "lmn", "opq", "rst", "123", "456", ]
b = [x for x in b if not x in a]

answered Aug 12, 2013 at 20:24

Mayur PatelMayur Patel

9051 gold badge7 silver badges15 bronze badges

2

There are already many answers on "how can you fix it?", so this is a "how can you improve it and be more pythonic?": since what you want to achieve is to get the difference between list b and list a, you should use difference operation on sets (operations on sets):

>>> a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]
>>> b = ["ijk", "lmn", "opq", "rst", "123", "456", ]
>>> s1 = set(a)
>>> s2 = set(b)
>>> s2 - s1
set(['123', '456'])

answered Aug 12, 2013 at 20:29

Vincenzo PiiVincenzo Pii

18k8 gold badges37 silver badges49 bronze badges

Along the lines of 7stud, if you go through the list in reversed order, you don't have the problem you encountered:

a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]

b = ["ijk", "lmn", "opq", "rst", "123", "456", ]

for i in reversed(b):
    if i in a:
        print "found " + i
        b.remove(i)

print b

Output:
found rst
found opq
found lmn
found ijk
['123', '456']

answered Dec 14, 2020 at 13:09

MarkoMarko

3622 silver badges7 bronze badges

You can use the list comprehensive

a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]
b = ["ijk", "lmn", "opq", "rst", "123", "456", ]

duplicates value removed from a

c=[value for value in a if value not in b]

duplicate value removed from b

c=[value for value in b if value not in a]

answered Sep 8, 2021 at 11:44

a = ["abc", "def", "ijk", "lmn", "opq", "rst", "xyz"]

b = ["ijk", "lmn", "opq", "rst", "123", "456","abc"]

for i in a:
    if i in b:
        print("found", i)
        b.remove(i)
print(b)

output:
found abc
found ijk
found lmn
found opq
found rst
['123', '456']

answered May 1 at 13:36

1

A simple fix would be to instead iterate through a range, look at the element at the index, delete that element, then decrement the counter by 1.
Mock untested code

for i in range(0, len(b)):
    if b[i] in a:
        del b[i]
        i -= 1

answered Aug 27, 2021 at 10:14

2

How do you remove duplicates between two lists in Python?

How do you remove common items from two lists in Python?

How do I remove duplicates from a list?

How do you remove duplicates from a consecutive list in Python?