Write a Python program to
- Perform Morphological Analysis using NLTK library
- Generate n-grams using NLTK N-Grams library
- Implement N-Grams Smoothing
Solution :
DESCRIPTION:
a) Morphological Analysis :
Morphology is the study of the structure of words. Morphological analysis splits a word into its root (stem) and its affixes. The program uses two NLTK stemmers: RegexpStemmer, which removes the suffixes given by a regular expression (here -ness, -ing, -ed, -ly, -s), and PorterStemmer, which applies the Porter rules. The suffix is the part of the word that was removed.
b) N-Grams :
An n-gram is a sequence of n consecutive words of a text: 1 word is a unigram, 2 words a bigram, 3 words a trigram. N-grams are used in language modelling, text prediction and spelling correction. NLTK generates them with nltk.ngrams(tokens, n).
c) N-Grams Smoothing :
A language model estimates the bigram probability as P(w2 | w1) = C(w1 w2) / C(w1). A bigram that never occurred in the training text gets the probability 0, which makes the probability of every sentence containing it 0. Smoothing gives such unseen n-grams a small non-zero probability. In add-one (Laplace) smoothing 1 is added to every count: P(w2 | w1) = (C(w1 w2) + 1) / (C(w1) + V), where V is the size of the vocabulary.
a) Morphological Analysis
PROGRAM: ( morphological_analysis.py )
from nltk.stem import PorterStemmer, RegexpStemmer
porter = PorterStemmer()
# removes the suffixes -ness, -ing, -ed, -ly and -s from the end of a word
suffix_stemmer = RegexpStemmer("ness$|ing$|ed$|ly$|s$", min=4)
words = ["walking", "walked", "cars", "quickly", "kindness", "happiness", "unhappiness"]
print("{:<13} {:<12} {:<9} {:<12}".format("Word", "Root", "Suffix", "Porter stem"))
print("-" * 48)
for w in words:
root = suffix_stemmer.stem(w)
suffix = w[len(root):] if len(root) < len(w) else "-"
print("{:<13} {:<12} {:<9} {:<12}".format(w, root, suffix, porter.stem(w)))
OUTPUT:
$ python morphological_analysis.py
Word Root Suffix Porter stem
------------------------------------------------
walking walk ing walk
walked walk ed walk
cars car s car
quickly quick ly quickli
kindness kind ness kind
happiness happi ness happi
unhappiness unhappi ness unhappi
b) N-Grams
PROGRAM: ( ngrams.py )
from nltk import ngrams
text = input("Enter a sentence: ")
tokens = text.lower().split()
for n, name in [(1, "Unigrams"), (2, "Bigrams"), (3, "Trigrams")]:
grams = list(ngrams(tokens, n))
print("\n" + name + " (" + str(len(grams)) + "):")
print(grams)
# n-grams for any value of n
n = int(input("\nEnter the value of n: "))
grams = list(ngrams(tokens, n))
print(str(n) + "-grams (" + str(len(grams)) + "):")
for g in grams:
print(" ".join(g))
OUTPUT:
$ python ngrams.py
Enter a sentence: Natural language processing is fun to learn
Unigrams (7):
[('natural',), ('language',), ('processing',), ('is',), ('fun',), ('to',), ('learn',)]
Bigrams (6):
[('natural', 'language'), ('language', 'processing'), ('processing', 'is'), ('is', 'fun'), ('fun', 'to'), ('to', 'learn')]
Trigrams (5):
[('natural', 'language', 'processing'), ('language', 'processing', 'is'), ('processing', 'is', 'fun'), ('is', 'fun', 'to'), ('fun', 'to', 'learn')]
Enter the value of n: 4
4-grams (4):
natural language processing is
language processing is fun
processing is fun to
is fun to learn
c) N-Grams Smoothing
PROGRAM: ( ngram_smoothing.py )
from collections import Counter
from nltk import ngrams
corpus = "the cat sat on the mat the dog sat on the log"
tokens = corpus.split()
unigram_counts = Counter(tokens)
bigram_counts = Counter(ngrams(tokens, 2))
V = len(unigram_counts) # vocabulary size
print("Corpus :", corpus)
print("Vocabulary size (V) =", V)
def p_mle(w1, w2):
# probability without smoothing: C(w1 w2) / C(w1)
return bigram_counts[(w1, w2)] / unigram_counts[w1]
def p_laplace(w1, w2):
# add-one (Laplace) smoothing: (C(w1 w2) + 1) / (C(w1) + V)
return (bigram_counts[(w1, w2)] + 1) / (unigram_counts[w1] + V)
test_bigrams = [("the", "cat"), ("the", "dog"), ("sat", "on"),
("cat", "dog"), ("dog", "mat")]
print()
print("{:<12} {:>6} {:>12} {:>14}".format("Bigram", "Count", "P (no smooth)", "P (add-one)"))
print("-" * 47)
for w1, w2 in test_bigrams:
print("{:<12} {:>6} {:>13.4f} {:>14.4f}".format(w1 + " " + w2, bigram_counts[(w1, w2)],
p_mle(w1, w2), p_laplace(w1, w2)))
OUTPUT:
$ python ngram_smoothing.py
Corpus : the cat sat on the mat the dog sat on the log
Vocabulary size (V) = 7
Bigram Count P (no smooth) P (add-one)
-----------------------------------------------
the cat 1 0.2500 0.1818
the dog 1 0.2500 0.1818
sat on 2 1.0000 0.3333
cat dog 0 0.0000 0.1250
dog mat 0 0.0000 0.1250
๐ป Run This Python Program
Write a Python program to
The selected Python program is loaded
automatically into the editor.
Edit the program, provide input through
STDIN if required
and click Run.
nltk.download()).
If NLTK is not available in the embedded compiler, use
Copy Program and run the program on your
computer (pip install nltk) or in Google Colab.