分詞
nltk.sent_tokenize(text) #按句子分割
nltk.word_tokenize(sentence) #分詞
nltk的分詞是句子級別的，所以對於一篇文檔首先要將文章按句子進行分割，然後句子進行分詞：
詞性標註
nltk.pos_tag(tokens) #對分詞後的句子進行詞性標註
tags = [nltk.pos_tag(tokens) for tokens in words]
>>>tags
[[('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('text', 'NN'), ('for', 'IN'), ('test', 'NN'), ('.', '.')], [('And', 'CC'), ('I', 'PRP'), ('want', 'VBP'), ('to', 'TO'), ('learn', 'VB'), ('how', 'WRB'), ('to', 'TO'), ('use', 'VB'), ('nltk', 'NN'), ('.', '.')]]
分塊 Chunking
用於實體識別的基本技術是分塊，可以理解爲把對個token組成詞組。
名詞短語分塊
NP-chunking，尋找單獨名詞短語對應的塊。爲了創建NP分塊，首先需要定義分塊語法，規定句子如何分塊。下面使用的是一個簡單的正則，這條規則規定由可選的且後面跟着任意數量形容詞（JJ）的限定詞（DJ）和名詞（NN）組成。
>>> text = "the little yellow dog barked at the cat"
>>> sentence = nltk.word_tokenize(text)
>>> sentence = nltk.pos_tag(sentence)
>>> sentence
[('the', 'DT'), ('little', 'JJ'), ('yellow', 'NN'), ('dog', 'NN'), ('barked', 'VBD'), ('at', 'IN'), ('the', 'DT'), ('cat', 'NN')]
>>> grammar = "NP: {<DT>?<JJ>*<NN>}"
>>> cp = nltk.RegexpParser(grammar)
>>> result = cp.parse(sentence)
>>> result
Tree('S', [Tree('NP', [('the', 'DT'), ('little', 'JJ'), ('yellow', 'NN')]), Tree('NP', [('dog', 'NN')]), ('barked', 'VBD'), ('at', 'IN'), Tree('NP', [('the', 'DT'), ('cat', 'NN')])])
>>> result.draw()