Simple Python script without the use of heavy text processing libraries to extract most common words from a corpus.
What is the most used word in all of Shakespeare plays? Was ‘king’ more often used than ‘Lord’ or vice versa?
To answer these type of fun questions, one often needs to quickly examine and plot most frequent words in a text file (often downloaded from open source portals such as Project Gutenberg). However, if you search on the web or on Stackoverflow, you will most probably see examples of nltk and use of CountVectorizer. While they are incredibly powerful and fun to use, the matter of the fact is, you don’t need them if the only thing you want is to extract most common words appearing in a single text corpus.
Below, I am showing a very simple Python 3 code snippet to do just that — using only a dictionary and simple string manipulation methods.
Feel free to copy the code and use your own stopwords to make it better!
import collectionsimport pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline# Read input file, note the encoding is specified here
# It may be different in your text file
file = open('PrideandPrejudice.txt', encoding="utf8")
a= file.read()# Stopwords
stopwords = set(line.strip() for line in open('stopwords.txt'))
stopwords = stopwords.union(set(['mr','mrs','one','two','said']))# Instantiate a dictionary, and for every word in the file,
# Add to the dictionary if it doesn't exist. If it does, increase the count.
wordcount = {}# To eliminate duplicates, remember to split by punctuation, and use case demiliters.
for word in a.lower().split():
word = word.replace(".","")
word = word.replace(",","")
word = word.replace(":","")
word = word.replace("\"","")
word = word.replace("!","")
word = word.replace("“","")
word = word.replace("‘","")
word = word.replace("*","")
if word not in stopwords:
if word not in wordcount:
wordcount[word] = 1
else:
wordcount[word] += 1# Print most common word
n_print = int(input("How many most common words to print: "))
print("\nOK. The {} most common words are as follows\n".format(n_print))
word_counter = collections.Counter(wordcount)
for word, count in word_counter.most_common(n_print):
print(word, ": ", count)# Close the file
file.close()# Create a data frame of the most common words
# Draw a bar chart
lst = word_counter.most_common(n_print)
df = pd.DataFrame(lst, columns = ['Word', 'Count'])
df.plot.bar(x='Word',y='Count')
An example of the code output and plot of the 10 most frequently used words in the corpus. The text is ‘Pride and Prejudice’ and you can see the familiar names of Elizabeth and Mr. Darcy! :)
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.
Given the data set, we can find k number of most frequent words.
The solution of this problem already present as Find the k most frequent words from a file. But we can solve this problem very efficiently in Python with the help of some high performance modules.
In order to do this, we’ll use a high performance data type module, which is collections. This
module got some specialized container datatypes and we will use counter class from this module.
Examples :
Recommended: Please try your approach on {IDE} first, before moving on to the solution.
Approach :
- Import Counter class from collections module.
- Split the string into list using split(), it will return the lists of words.
- Now pass the list to the instance of Counter class
- The function 'most-common()' inside Counter will return the list of most frequent words from list and its count.
Below is Python implementation of above approach :
from collections import Counter
data_set = "Welcome to the world of Geeks " \
"This portal has been created to provide well written well" \
"thought and well explained solutions for selected questions " \
"If you like Geeks for Geeks and would like to contribute " \
"here is your chance You can write article and mail your article " \
" to contribute at geeksforgeeks org See your article appearing on " \
"the Geeks for Geeks main page and help thousands of other Geeks. " \
split_it = data_set.split()
Counter = Counter(split_it)
most_occur = Counter.most_common(4)
print(most_occur)
Output :
[('Geeks', 5), ('to', 4), ('and', 4), ('article', 3)]