NLP-FOR-HACKERS - Page 3 of 4 - A blog about simple and effective Natural Language Processing.

Training a NER System Using a Large Dataset

In a previous article, we studied training a NER (Named-Entity-Recognition) system from the ground up, using the Groningen Meaning Bank Corpus. This article is a continuation of that tutorial. The main purpose of this extension to training a NER is to:

Replace the classifier with a Scikit-Learn Classifier
Train a NER on a larger subset of the training data
Increase accuracy
Understand Out Of Core Learning

What was wrong with the initial system you might ask. There wasn’t anything fundamentally wrong with the process. In fact, it’s a great didactical example, and we can build upon it. This is where it was lacking:
Read More

Date October 26, 2016
Author bogdani
Comments 60 Comments

Building a NLP pipeline in NLTK

If you have been working with NLTK for some time now, you probably find the task of preprocessing the text a bit cumbersome. In this post, I will walk you through a simple and fun approach for performing repetitive tasks using coroutines. The coroutines concept is a pretty obscure one but very useful indeed. You can check out this awesome presentation by David Beazley to grasp all the stuff needed to get you through this (plus much, much more).
Read More

Date October 19, 2016
Author bogdani
Comments 2 Comments

Text Chunking with NLTK

What is chunking

Text chunking, also referred to as shallow parsing, is a task that follows Part-Of-Speech Tagging and that adds more structure to the sentence. The result is a grouping of the words in “chunks”. Here’s a quick example:
Read More

Date October 15, 2016
Author bogdani
Comments 12 Comments

Complete guide to build your own Named Entity Recognizer with Python

Updates

29-Apr-2018 – Added Gist for the entire code

NER, short for Named Entity Recognition is probably the first step towards information extraction from unstructured text. It basically means extracting what is a real world entity from the text (Person, Organization, Event etc …).
Read More

Date September 27, 2016
Author bogdani
Comments 75 Comments

Stemmers vs. Lemmatizers

Stackoverflow is full of questions about why stemmers and lemmatizers don’t work as expected. The root cause of the confusion is that their role is often misunderstood. Here’s a comparison:
Read More

Date September 8, 2016
Author bogdani
Comments 2 Comments

Wordnet, getting your hands dirty

Wordnet is a lexical database created at Princeton University. Its size and several properties it holds make Wordnet one of the most useful tools you can have in your NLP arsenal.

Here are a few properties that make Wordnet so useful:
Read More

Date August 31, 2016
Author bogdani
Comments No Comments

Compute sentence similarity using Wordnet

It’s common in the world on Natural Language Processing to need to compute sentence similarity. Wordnet is an awesome tool and you should always keep it in mind when working with text. It’s of great help for the task we’re trying to tackle.

Suppose we have these sentences:
Read More

Date August 25, 2016
Author bogdani
Comments 11 Comments

Complete guide for training your own Part-Of-Speech Tagger

Part-Of-Speech tagging (or POS tagging, for short) is one of the main components of almost any NLP analysis. The task of POS-tagging simply implies labelling words with their appropriate Part-Of-Speech (Noun, Verb, Adjective, Adverb, Pronoun, …).

Date August 21, 2016
Author bogdani
Comments 38 Comments

Recipe: Text clustering using NLTK and scikit-learn

Simple recipe for text clustering. This sometimes creates issues in scikit-learn because text has sparse features.

Date August 18, 2016
Author bogdani
Comments 16 Comments

Recipe: Text classification using NLTK and scikit-learn

Text classification is most probably, the most encountered Natural Language Processing task. It can be described as assigning texts to an appropriate bucket. A sports article should go in SPORT_NEWS, and a medical prescription should go in MEDICAL_PRESCRIPTIONS.

To train a text classifier, we need some annotated data. This training data can be obtained through several methods. Suppose you want to build a spam classifier. You would export the contents of your mailbox. You’d label the email in the inbox folder as NOT_SPAM and the contents of your spam folder as SPAM.
Read More