The first thing to be aware of is that you don't have two different dictionaries. You have two different lists of dictionaries. The second is that you don't explain exactly what counts as a duplicate. The third is that you don't say what to do with the relevance key. I'll assume that two dictionaries with equivalent type and name keys are identical, and that you want the relevance values to be merged into a list. Then later you could average them, or
whatever. def gen_key(d):
return (d['name'], d['type'])
def merge_dupes(dlist):
relevance = [float(d['relevance']) for d in dlist]
name, type = dlist[0]['name'], dlist[0]['type']
return {'name':name, 'type':type, 'relevance':relevance}
to_merge = {}
for l in (x, y):
for d in l:
to_merge.setdefault(gen_key(d), []).append(d)
# if you want another list
merged_list = [merge_dupes(l) for l in to_merge.itervalues()]
# if you'd prefer a dictionary
merged_dict = dict((k, merge_dupes(v)) for k, v in to_merge.iteritems())
Output: >>> pprint(merged_list)
[{'name': u'Rob Hirschfeld',
'relevance': [0.44458599999999998],
'type': u'Person'},
{'name': 'VMs',
'relevance': [0.314, 0.52216899999999999],
'type': 'OperatingSystem'},
{'name': 'Greg Althaus',
'relevance': [0.32700000000000001, 0.41398800000000002],
'type': 'Person'},
{'name': 'Storage Hardware',
'relevance': [0.17399999999999999],
'type': 'Company'},
{'name': u'iSCSI',
'relevance': [0.37648900000000002],
'type': u'FieldTerminology'},
{'name': 'Force10',
'relevance': [0.26600000000000001, 0.31405899999999998],
'type': 'Company'},
{'name': 'http://Dell.com/OpenStack',
'relevance': [0.085000000000000006],
'type': 'URL'},
{'name': 'Dell',
'relevance': [0.72199999999999998, 0.87406499999999998],
'type': 'Company'},
{'name': 'iSCSI', 'relevance': [0.122], 'type': 'Technology'}]
>>> pprint(merged_dict)
{('Dell', 'Company'): {'name': 'Dell',
'relevance': [0.72199999999999998,
0.87406499999999998],
'type': 'Company'},
('Force10', 'Company'): {'name': 'Force10',
'relevance': [0.26600000000000001,
0.31405899999999998],
'type': 'Company'},
('Greg Althaus', 'Person'): {'name': 'Greg Althaus',
'relevance': [0.32700000000000001,
0.41398800000000002],
'type': 'Person'},
(u'Rob Hirschfeld', u'Person'): {'name': u'Rob Hirschfeld',
'relevance': [0.44458599999999998],
'type': u'Person'},
('Storage Hardware', 'Company'): {'name': 'Storage Hardware',
'relevance': [0.17399999999999999],
'type': 'Company'},
('VMs', 'OperatingSystem'): {'name': 'VMs',
'relevance': [0.314, 0.52216899999999999],
'type': 'OperatingSystem'},
('http://Dell.com/OpenStack', 'URL'): {'name': 'http://Dell.com/OpenStack',
'relevance': [0.085000000000000006],
'type': 'URL'},
(u'iSCSI', u'FieldTerminology'): {'name': u'iSCSI',
'relevance': [0.37648900000000002],
'type': u'FieldTerminology'},
('iSCSI', 'Technology'): {'name': 'iSCSI',
'relevance': [0.122],
'type': 'Technology'}}
Removal of duplicates is essential in many applications. List of dictionaries are quite common and sometimes we require to duplicate the duplicated. Lets discuss certain ways in which we can removing duplicate dictionaries in a list. Using loop for Removing duplicate dictionaries in a listThe basic method that comes to our mind while performing this operation is the naive method of iterating the list of dictionaries in
Python and manually removing the duplicate dictionary and append in new list. Python3test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat"
: 3},
{"Kil" : 2}, {"Akshat" : 3}]
print ("Original list : " + str(test_list))
res_list = []
for i in range(len(test_list)):
if test_list[i] not in test_list[i + 1:]:
res_list.append(test_list[i])
print ("Resultant list is : " + str(res_list))
Output :
Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}] Resultant list is : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}]
Using list comprehension for Removing duplicate dictionaries in a listThe use of list comprehension and enumerate can possibly allow to achieve this particular task in a single line and
hence is of a good utility. Python3test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},
{"Kil" : 2}, {"Akshat" : 3}]
print ("Original list : " + str(test_list))
res_list = [i for n, i in enumerate(test_list)
if i not in
test_list[n + 1:]]
print ("Resultant list is : " + str(res_list))
Output : Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}] Resultant list is : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}]
Using frozenset for Removing duplicate dictionaries in a listfrozenset is used to assign a value to key in dictionary as a set. The repeated entries of dictionary are hence ignored and hence solving this particular task. Python3test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},
{"Kil" : 2}, {"Akshat"
: 3}]
print ("Original list : " + str(test_list))
res_list = {frozenset(item.items()) :
item for item in test_list}.values()
print ("Resultant list is : " + str(res_list))
Output : Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}] Resultant list is : [{‘Kil’: 2}, {‘Akshat’: 3}, {‘Akash’: 1}]
Using
unique everseen() for Removing duplicate dictionaries in a listeverseen() function is used to find all the unique elements present in the iterable and preserving their order of occurrence. Hence it remembers all elements ever seen in the iterable. Python3from iteration_utilities import unique_everseen
test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat"
: 3},
{"Kil" : 2}, {"Akshat" : 3}]
res_list=list(unique_everseen(test_list))
print ("Original list : " + str(test_list))
print ("Resultant list is : " + str(res_list))
Output: Original list : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}, {‘Kil’: 2}, {‘Akshat’: 3}]
Resultant list is : [{‘Akash’: 1}, {‘Kil’: 2}, {‘Akshat’: 3}]
How do you remove duplicates from two dictionaries in Python?
“python combine two dicts into one without duplicates” Code Answer's. dict1 = {'color': 'blue', 'shape': 'square'}. dict2 = {'color': 'red', 'edges': 4}. dict1. update(dict2) #if a key exists in both, it takes the value of the second dict.. How do I remove duplicates from a dictionary list?
Using unique everseen() for Removing duplicate dictionaries in a list. everseen() function is used to find all the unique elements present in the iterable and preserving their order of occurrence. Hence it remembers all elements ever seen in the iterable.
How do you know if two dictionaries have the same value?
Use == to check if two dictionaries contain the same set of key: value pairs.. dictionary1 = {"a": 1, "b": 2}. dictionary2 = {"a": 1, "b": 2}. print(is_equal). Can you zip two dictionaries in Python?
Now you can: Use the zip() function in both Python 3 and Python 2. Loop over multiple iterables and perform different actions on their items in parallel. Create and update dictionaries on the fly by zipping two input iterables together.
| 
