5 · POS tagging & Named Entity Recognition
Classify each token's grammatical role and extract people, places, and organizations from raw text.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
5 · POS tagging & Named Entity Recognition
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.POS tagging classifies each token's part of speech; NER finds named entities (people, places, orgs) — both are structured outputs from raw text.
Without this:
You'd miss the structural information that powers every information-extraction system.
So far we've treated text as a bag of words — tokens without grammatical context. But language is structured: "bank" as a noun means a financial institution; "bank" as a verb means to tilt. "Apple" in an earnings report refers to the company; "apple" in a recipe refers to the fruit.
Part-of-Speech (POS) tagging assigns each token a grammatical tag: noun (NN), verb (VB), adjective (JJ), adverb (RB), etc. These tags are useful for:
- Providing the right POS hint to
WordNetLemmatizer - Keyword extraction (keep only nouns and verbs)
- Feature engineering for text classification
Named Entity Recognition (NER) is a step further: it identifies and classifies multi-token phrases that refer to real-world entities — people (PERSON), places (GPE), organizations (ORG), monetary values (MONEY), dates (DATE).
NLTK uses the Penn Treebank tagset for POS and a Maximum Entropy chunker for NER. These are solid for learning but slower and less accurate than modern alternatives like spaCy or transformer-based models.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Penn Treebank tag reference — the 16 most useful tags
NER with nltk.ne_chunk — reference code (maxent_ne_chunker not available in Pyodide)
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
spaCy and Hugging Face — the production-grade NER alternatives to NLTK
What does the Penn Treebank POS tag 'NNP' mean?
- `nltk.pos_tag(tokens)` assigns Penn Treebank tags (NN, VB, JJ, RB, NNP, etc.) to each token — essential for correct lemmatization and noun extraction.
- NLTK's `ne_chunk` provides NER but requires the `maxent_ne_chunker` corpus which cannot be downloaded in Pyodide — use spaCy or Hugging Face for production NER.
- Filtering tokens by POS tag (keeping only `tag.startswith('NN')`) is a simple but powerful keyword extraction technique.
- The NNP tag is the bridge between tokenization and NER — proper nouns are the primary candidates for named entities.
Information extraction pipelines; chatbot intent detection (POS helps identify action verbs); resume parsers (NER extracts skills, companies, job titles); biomedical literature mining (drug names, gene symbols); financial news analysis (company names, monetary values).
If you remove it: Without POS tags, lemmatization defaults to noun — 'running' stays 'running' instead of reducing to 'run'. Without NER, extracting 'Apple', 'U.K.', '$1 billion' from text requires hand-written regex rules for every entity pattern.