Natural Language Processing (NLP) is the bridge between human language and machine understanding. It involves a series of steps to convert messy, unstructured text into a format a computer can process.
Text Preprocessing & Tokenization
Tokenization is the process of breaking text into smaller units (tokens). Modern NLP uses subword tokenization to handle out-of-vocabulary words effectively.
- BPE (Byte Pair Encoding): Iteratively merges the most frequent pairs of characters.
- WordPiece: Used in BERT, chooses merges that maximize the likelihood of the training data.
- Stop-word Removal: Filtering out common words (the, a, is) that may not carry much semantic meaning.
Word Embeddings
Embeddings are dense vector representations of words where similar words are closer in vector space. While early methods (Word2Vec) assigned one vector per word, modern models (BERT, GPT) provide contextual embeddings where a word's vector changes based on surrounding words.
Common NLP Tasks
- NER (Named Entity Recognition): Identifying names, dates, and locations.
- Sentiment Analysis: Determining the emotional tone of text.
- Text Summarization: Condensing long documents while preserving key info.
- Machine Translation: Converting text between different languages.