Python remove duplicates from list of dictionaries by key

I am trying to remove the duplicates from following list:

distinct_cur = [
    {'rtc': 0, 'vf': 0, 'mtc': 0, 'doc': 'good job', 'foc': 195, 'st': 0.0, 'htc': 2, '_id': ObjectId('58e86a550a0aeff4e14ca6bb'), 'ftc': 0}, 
    {'rtc': 0, 'vf': 0, 'mtc': 0, 'doc': 'good job', 'foc': 454, 'st': 0.8, 'htc': 1, '_id': ObjectId('58e8d03958ae6d179c2b4413'), 'ftc': 1},
    {'rtc': 0, 'vf': 2, 'mtc': 1, 'doc': 'test', 'foc': 45, 'st': 0.8, 'htc': 12, '_id': ObjectId('58e8d03958ae6d180c2b4446'), 'ftc': 0}
]

Of dictionaries based on condition that if 'doc' key value text is same then one of the dictionary should be removed. I have tried the following solution:

distinct_cur = [dict(y) for y in set(tuple(x.items()) for x in cur)]

But duplicates are still present in the final list.

Below is the desired output as in 1st and 2nd distinct_cur text of key 'doc' value is same (good job):

[
    {'rtc': 0, 'vf': 0, 'mtc': 0, 'doc': 'good job', 'foc': 195, 'st': 0.0, 'htc': 2, '_id': ObjectId('58e86a550a0aeff4e14ca6bb'), 'ftc': 0}, 
    {'rtc': 0, 'vf': 2, 'mtc': 1, 'doc': 'test', 'foc': 45, 'st': 0.8, 'htc': 12, '_id': ObjectId('58e8d03958ae6d180c2b4446'), 'ftc': 0}
]

Removal of duplicates is essential in many applications. List of dictionaries are quite common and sometimes we require to duplicate the duplicated. Lets discuss certain ways in which we can removing duplicate dictionaries in a list.

Using loop for Removing duplicate dictionaries in a list

The basic method that comes to our mind while performing this operation is the naive method of iterating the list of dictionaries in Python and manually removing the duplicate dictionary and append in new list. 

Python3

test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},

             {"Kil" : 2}, {"Akshat" : 3}]

print ("Original list : " + str(test_list))

res_list = []

for i in range(len(test_list)):

    if test_list[i] not in test_list[i + 1:]:

        res_list.append(test_list[i])

print ("Resultant list is : " + str(res_list))

Output :

Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}]

Resultant list is : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}]

Using list comprehension for Removing duplicate dictionaries in a list

The use of list comprehension and enumerate can possibly allow to achieve this particular task in a single line and hence is of a good utility. 

Python3

test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},

             {"Kil" : 2}, {"Akshat" : 3}]

print ("Original list : " + str(test_list))

res_list = [i for n, i in enumerate(test_list)

            if i not in test_list[n + 1:]]

print ("Resultant list is : " + str(res_list))

Output :

Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}]

Resultant list is : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}]

Using frozenset for Removing duplicate dictionaries in a list

frozenset is used to assign a value to key in dictionary as a set. The repeated entries of dictionary are hence ignored and hence solving this particular task. 

Python3

test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},

             {"Kil" : 2}, {"Akshat" : 3}]

print ("Original list : " + str(test_list))

res_list = {frozenset(item.items()) :

            item for item in test_list}.values()

print ("Resultant list is : " + str(res_list))

Output :

Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}]

Resultant list is : [{‘Kil’: 2}, {‘Akshat’: 3}, {‘Akash’: 1}]

Using unique everseen() for Removing duplicate dictionaries in a list

everseen() function is used to find all the unique elements present in the iterable and preserving their order of occurrence. Hence it remembers all elements ever seen in the iterable.

Python3

from iteration_utilities import unique_everseen

test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},

             {"Kil" : 2}, {"Akshat" : 3}]

res_list=list(unique_everseen(test_list))

print ("Original list : " + str(test_list))

print ("Resultant list is : " + str(res_list))

Output:

Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}]
Resultant list is : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}]

How do I remove duplicates from a dictionary list?

How do you remove duplicate keys in Python?

Can you duplicate keys in dictionaries?

How do you count duplicates in a list in Python?