TF-IDF (Term Frequency & Inverse Document Frequency)
TF-IDF stands for Term Frequency and Inverse Document Frequency. This method is better than the BoW (Bag of Words) because BoW gives the number of occurance through numeric vector of each word in the document but TF-IDF gives the importance of each word throuogh numeric value in the document.
"TF-IDF (Term Frequency–Inverse Document Frequency) is a numerical technique used in Natural Language Processing (NLP) to measure the importance of a word in a document compared with a collection of documents."
Term Frequency: It is a measure of how many times a particular word appears in a document compared to the total number of words in that document. It can be found from document vector table.
Formula:
Document Frequency: It is the number of documents in a collection that contain a particular word.
Example: If the word “Artificial Intelligence” appears in 3 out of 5 documents.
Then DF(computer) = 3
Inverse Document Frequency: It measures how rare or unique a word is across a collection of documents.
Example:
Let us go through an example:
Document 1: the cat sat on the mat
Document 2: the dog sat on the rug
Document 3: the cat chased the mouse
Applications of TF-IDF
- Document Ranking in Search Engines – Helps rank documents based on how relevant their words are to a search query.
- Keyword Extraction – Identifies the most important words in a document.
- Text Classification – Converts text into numerical values so that machine learning algorithms can classify it.
- Document Clustering – Helps group similar documents based on the words they contain.
- Recommendation Systems – Helps recommend similar documents, articles, or content based on important words.
- Spam Detection – Helps identify spam messages by analyzing the words commonly used in them.
NLTK (Natural Language ToolKit)
Natural Language Toolkit (NLTK) is one of the leading open-source NLP toolkit made up of Python libraries and used for building Python programs that can work with human language data.