๐Ÿ’ฌ
B.TECH III YEAR โ€ข II SEMESTER โ€ข R22

Natural Language Processing - Lab Manual

Natural Language Processing Laboratory โ€ข Program Solution

๐Ÿ“˜ R22 Regulation ๐ŸŽ“ III Year ๐Ÿ“š II Semester ๐Ÿ’ฌ NLP Using Python ๐Ÿงช Lab Program
Selected Program

Write a Python program to

โ† All NLP Programs

Solution :

DESCRIPTION:

a) Morphological Analysis :
Morphology is the study of the structure of words. Morphological analysis splits a word into its root (stem) and its affixes. The program uses two NLTK stemmers: RegexpStemmer, which removes the suffixes given by a regular expression (here -ness, -ing, -ed, -ly, -s), and PorterStemmer, which applies the Porter rules. The suffix is the part of the word that was removed.

b) N-Grams :
An n-gram is a sequence of n consecutive words of a text: 1 word is a unigram, 2 words a bigram, 3 words a trigram. N-grams are used in language modelling, text prediction and spelling correction. NLTK generates them with nltk.ngrams(tokens, n).

c) N-Grams Smoothing :
A language model estimates the bigram probability as P(w2 | w1) = C(w1 w2) / C(w1). A bigram that never occurred in the training text gets the probability 0, which makes the probability of every sentence containing it 0. Smoothing gives such unseen n-grams a small non-zero probability. In add-one (Laplace) smoothing 1 is added to every count: P(w2 | w1) = (C(w1 w2) + 1) / (C(w1) + V), where V is the size of the vocabulary.

a) Morphological Analysis

PROGRAM: ( morphological_analysis.py )

 
from nltk.stem import PorterStemmer, RegexpStemmer

porter = PorterStemmer()

# removes the suffixes -ness, -ing, -ed, -ly and -s from the end of a word
suffix_stemmer = RegexpStemmer("ness$|ing$|ed$|ly$|s$", min=4)

words = ["walking", "walked", "cars", "quickly", "kindness", "happiness", "unhappiness"]

print("{:<13} {:<12} {:<9} {:<12}".format("Word", "Root", "Suffix", "Porter stem"))
print("-" * 48)
for w in words:
    root = suffix_stemmer.stem(w)
    suffix = w[len(root):] if len(root) < len(w) else "-"
    print("{:<13} {:<12} {:<9} {:<12}".format(w, root, suffix, porter.stem(w)))

OUTPUT:

 
$ python morphological_analysis.py
Word          Root         Suffix    Porter stem 
------------------------------------------------
walking       walk         ing       walk        
walked        walk         ed        walk        
cars          car          s         car         
quickly       quick        ly        quickli     
kindness      kind         ness      kind        
happiness     happi        ness      happi       
unhappiness   unhappi      ness      unhappi     

b) N-Grams

PROGRAM: ( ngrams.py )

 
from nltk import ngrams

text = input("Enter a sentence: ")
tokens = text.lower().split()

for n, name in [(1, "Unigrams"), (2, "Bigrams"), (3, "Trigrams")]:
    grams = list(ngrams(tokens, n))
    print("\n" + name + " (" + str(len(grams)) + "):")
    print(grams)

# n-grams for any value of n
n = int(input("\nEnter the value of n: "))
grams = list(ngrams(tokens, n))
print(str(n) + "-grams (" + str(len(grams)) + "):")
for g in grams:
    print(" ".join(g))

OUTPUT:

 
$ python ngrams.py
Enter a sentence: Natural language processing is fun to learn

Unigrams (7):
[('natural',), ('language',), ('processing',), ('is',), ('fun',), ('to',), ('learn',)]

Bigrams (6):
[('natural', 'language'), ('language', 'processing'), ('processing', 'is'), ('is', 'fun'), ('fun', 'to'), ('to', 'learn')]

Trigrams (5):
[('natural', 'language', 'processing'), ('language', 'processing', 'is'), ('processing', 'is', 'fun'), ('is', 'fun', 'to'), ('fun', 'to', 'learn')]

Enter the value of n: 4
4-grams (4):
natural language processing is
language processing is fun
processing is fun to
is fun to learn

c) N-Grams Smoothing

PROGRAM: ( ngram_smoothing.py )

 
from collections import Counter
from nltk import ngrams

corpus = "the cat sat on the mat the dog sat on the log"
tokens = corpus.split()

unigram_counts = Counter(tokens)
bigram_counts = Counter(ngrams(tokens, 2))
V = len(unigram_counts)          # vocabulary size

print("Corpus :", corpus)
print("Vocabulary size (V) =", V)

def p_mle(w1, w2):
    # probability without smoothing: C(w1 w2) / C(w1)
    return bigram_counts[(w1, w2)] / unigram_counts[w1]

def p_laplace(w1, w2):
    # add-one (Laplace) smoothing: (C(w1 w2) + 1) / (C(w1) + V)
    return (bigram_counts[(w1, w2)] + 1) / (unigram_counts[w1] + V)

test_bigrams = [("the", "cat"), ("the", "dog"), ("sat", "on"),
                ("cat", "dog"), ("dog", "mat")]

print()
print("{:<12} {:>6} {:>12} {:>14}".format("Bigram", "Count", "P (no smooth)", "P (add-one)"))
print("-" * 47)
for w1, w2 in test_bigrams:
    print("{:<12} {:>6} {:>13.4f} {:>14.4f}".format(w1 + " " + w2, bigram_counts[(w1, w2)],
                                                 p_mle(w1, w2), p_laplace(w1, w2)))

OUTPUT:

 
$ python ngram_smoothing.py
Corpus : the cat sat on the mat the dog sat on the log
Vocabulary size (V) = 7

Bigram        Count P (no smooth)    P (add-one)
-----------------------------------------------
the cat           1        0.2500         0.1818
the dog           1        0.2500         0.1818
sat on            2        1.0000         0.3333
cat dog           0        0.0000         0.1250
dog mat           0        0.0000         0.1250


๐Ÿ’ป Run This Python Program

Write a Python program to
The selected Python program is loaded automatically into the editor. Edit the program, provide input through STDIN if required and click Run.

The compiler is embedded using OneCompiler. Your Python program runs inside the embedded Python environment; Study Glance does not execute Python code directly on your web server. If the program asks for input, type the value in the STDIN box before clicking Run. Programs that use the NLTK library need the package and its data (nltk.download()). If NLTK is not available in the embedded compiler, use Copy Program and run the program on your computer (pip install nltk) or in Google Colab.
โ† Back to NLP Lab Programs ๐Ÿ“š All Lab Manuals