tokenization Archives - NLP-FOR-HACKERS

tokenization

Introduction to NLTK

NLTK (Natural Language ToolKit) is the most popular Python framework for working with human language. There’s a bit of controversy around the question whether NLTK is appropriate or not for production environments. Here’s my take on the matter:

Date February 22, 2017
Author bogdani
Comments 11 Comments

Splitting text into sentences

Few people realise how tricky splitting text into sentences can be. Most of the NLP frameworks out there already have English models created for this task. You might encounter issues with the pretrained models if: