136 lines
3.0 KiB
Markdown
136 lines
3.0 KiB
Markdown
# A7 - Text Analytics
|
|
|
|
✅ Tested and working as intended.
|
|
|
|
---
|
|
|
|
## Pre-requisites
|
|
|
|
- Install required libraries: `nltk`
|
|
|
|
```shell
|
|
pip install nltk
|
|
```
|
|
|
|
---
|
|
|
|
1. Import libraries:
|
|
|
|
```python3
|
|
import nltk
|
|
from nltk.tokenize import *
|
|
from nltk.corpus import *
|
|
from nltk.stem import *
|
|
import re
|
|
```
|
|
|
|
2. Download resources:
|
|
|
|
```python3
|
|
nltk.download('all') # WARNING: ABOUT 2GBs
|
|
```
|
|
|
|
> OR IF YOU'RE FEELING FANCY YOU CAN DOWNLOAD ONLY SPECIFIC RESOURCES:
|
|
```python3
|
|
nltk.download('punkt') # For splitting text into sentences or words
|
|
nltk.download('stopwords') # Common stop words
|
|
nltk.download('wordnet') # Synonyms
|
|
nltk.download('averaged_perceptron_tagger') # part-of-speech (POS) tagger
|
|
nltk.download('punkt_tab') # For tokenizing text that is formatted in tabular form
|
|
```
|
|
|
|
3. Write text to perform preprocessing on:
|
|
|
|
```python3
|
|
text = "Hello everyone! I am first name last name. I am a loyal KSKA Git user all the way from Sangamwadi Empire. I have considerable knowledge about life, Python, C++, Java, Rust, Golang and Blockchain. For every smart contract, I lose one strand of my hair. In my free time, which by the way, I barely get, I like to swim."
|
|
```
|
|
|
|
4. Sentence tokenization:
|
|
|
|
```python3
|
|
var1 = sent_tokenize(text)
|
|
print(var1)
|
|
```
|
|
|
|
5. Word tokenization:
|
|
|
|
```python3
|
|
var2 = word_tokenize(text)
|
|
print(var2)
|
|
```
|
|
|
|
6. Removing punctuation:
|
|
|
|
```python3
|
|
text = re.sub('[^a-zA-Z]',' ',text)
|
|
print("After removing punctuation from text:\n", text)
|
|
```
|
|
|
|
7. Removing stop words:
|
|
|
|
```python3
|
|
var3 = set(stopwords.words('english'))
|
|
print("Stop words:\n", var3)
|
|
print("==============================================================")
|
|
tokens = word_tokenize(text.lower())
|
|
filtered_text = []
|
|
for word in tokens:
|
|
if word not in var3:
|
|
filtered_text.append(word)
|
|
print("Tokenized Sentence:\n", tokens)
|
|
print("\nFiltered Sentence:\n", filtered_text)
|
|
```
|
|
|
|
8. Stemmatization:
|
|
|
|
```python3
|
|
var = ["write", "writing", "wrote", "writes","reading","reads"]
|
|
ps = PorterStemmer() # brings word to its root form
|
|
for w in var:
|
|
root_word = ps.stem(w)
|
|
print(root_word)
|
|
```
|
|
|
|
9. Lemmatization:
|
|
|
|
```python3
|
|
wordnet_lemmatizer = WordNetLemmatizer()
|
|
text = "studies studying cries cry"
|
|
tt = nltk.word_tokenize(text)
|
|
print("Text is:\t", tt)
|
|
for w in tt:
|
|
print("Lemma for {} is {}".format(w, wordnet_lemmatizer.lemmatize(w)))
|
|
```
|
|
|
|
10. POS Tagging:
|
|
|
|
```python3
|
|
text = "Hello everyone this is a sample text! Earth."
|
|
text = nltk.word_tokenize(text)
|
|
nltk.pos_tag(text)
|
|
```
|
|
|
|
11. TF-IDF (Term Frequency & Inverse Document Frequency):
|
|
|
|
```python3
|
|
# TF-IDF (Term Frequency & Inverse Document Frequency)
|
|
from sklearn.feature_extraction.text import TfidfVectorizer
|
|
|
|
new_sentence = "This is an example of term frequency. Meow meow meow meow meow!"
|
|
|
|
def calculate_tfIdf(document):
|
|
tokenizer = TfidfVectorizer()
|
|
tf_matrix = tokenizer.fit_transform(document)
|
|
features_names = tokenizer.get_feature_names_out()
|
|
return tf_matrix, features_names
|
|
|
|
# Wrap the new_sentence in a list
|
|
document = [new_sentence]
|
|
tf_matrix, feature_names = calculate_tfIdf(document)
|
|
|
|
print('TF-IDF')
|
|
print(feature_names, tf_matrix.toarray())
|
|
```
|
|
|
|
---
|