5個流行的自然語言處理庫及入門用法

{"type":"doc","content":[{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文介紹了5個流行的Python NLP庫和它們的入門用法，這些庫涵蓋了語言數據可視化、數據預處理、多任務功能、一流語言建模等用例。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"本文並不是要從這些解決方案中指定一個最優集合，而是給出一篇概述，介紹精選的5個流行庫，希望能幫助解決你的問題。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"1."},{"type":"link","attrs":{"href":"https:\/\/github.com\/huggingface\/datasets","title":null,"type":null},"content":[{"type":"text","text":"Hugging Face Datasets"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hugging Face的Datasets庫本質上是一個對公開可用的NLP數據集的打包集合，帶有一組通用的API和數據格式，以及一些輔助功能。以下是關於它的介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"收集了最多的用於ML模型的即開即用NLP數據集，具有快速、易用且高效的數據操作工具。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"你可以通過以下方式輕鬆安裝Datasets庫："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"pip install datasets"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"根據介紹，Datasets提供了兩大特性："},{"type":"text","marks":[{"type":"strong"}],"text":"用於許多公共數據集的單行數據加載器，以及高效的數據預處理"},{"type":"text","text":"。但它的介紹沒有提到這個庫的另一大特性：與NLP任務相關的許多內置評估指標。這個庫還有其他一些特性，例如數據集的後端內存管理以及與流行的Python工具（如NumPy、Pandas）和主流機器學習平臺（TensorFlow和PyTorch）的互操作性。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"我們先來看看如何加載一個數據集："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"from datasets import load_dataset, list_datasets\nprint(f\"The Hugging Face datasets library contains {len(list_datasets())} datasets\")\nsquad_dataset = load_dataset('squad')\nprint(squad_dataset['train'][0])\nprint(squad_dataset)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"The Hugging Face datasets library contains 635 datasets\nReusing dataset squad (\/home\/matt\/.cache\/huggingface\/datasets\/squad\/plain_text\/1.0.0\/4c81550d83a2ac7c7ce23783bd8ff36642800e6633c1f18417fb58c3ff50cdd7)\n{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']}, 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.', 'id': '5733be284776f41900661182', 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?', 'title': 'University_of_Notre_Dame'}\nDatasetDict({\n train: Dataset({\n features: ['id', 'title', 'context', 'question', 'answers'],\n num_rows: 87599\n })\n validation: Dataset({\n features: ['id', 'title', 'context', 'question', 'answers'],\n num_rows: 10570\n })\n})"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"加載指標也很簡單："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"from datasets import load_metric, list_metrics\nprint(f\"The Hugging Face datasets library contains {len(list_metrics())} metrics\")\nprint(f\"Available metrics are: {list_metrics()}\")\n# Load a metric\nsquad_metric = load_metric('squad')"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"The Hugging Face datasets library contains 19 metrics\nAvailable metrics are: ['accuracy', 'bertscore', 'bleu', 'bleurt', 'comet', 'coval', 'f1', 'gleu', 'glue', 'indic_glue', 'meteor', 'precision', 'recall', 'rouge', 'sacrebleu', 'seqeval', 'squad', 'squad_v2', 'xnli']"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"你想用它們做什麼都隨意，但有了這個庫，你就可以輕鬆加載可公開訪問的數據集，和久經考驗的真實評估指標了。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"2."},{"type":"link","attrs":{"href":"https:\/\/github.com\/jbesomi\/texthero","title":null,"type":null},"content":[{"type":"text","text":"TextHero"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TextHero在其GitHub存儲庫中的介紹很簡單："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"文本預處理、表示和可視化，助你從零邁向大師。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這幾句話很好地解釋了這個庫可以解決的問題，下面我們再深入研究一下爲什麼我們就要用它。從repo中我們可以看到更具體的說明："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"TextHero只有一個非常務實的目標：爲開發人員騰出空閒時間。文本數據處理起來可能會很痛苦，在大多數情況下有一個默認管道的話上手起來就輕鬆多了。總有時間回來改進以前的工作。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在你知道了爲什麼要使用TextHero，它的安裝方法如下："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"pip install texthero"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"link","attrs":{"href":"https:\/\/texthero.org\/docs\/getting-started","title":null,"type":null},"content":[{"type":"text","text":"入門指南"}]},{"type":"text","text":"介紹了你可以用它做的工作，幾行代碼就可以搞定。使用TextHero Github存儲庫中的以下示例，我們會加載一個數據集，清理它，並創建一個TF-IDF表示，執行主成分分析（PCA）並繪製PCA的結果。"}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"def text_texthero():\n\timport texthero as hero\n\timport pandas as pd\n\tdf = pd.read_csv(\"https:\/\/github.com\/jbesomi\/texthero\/raw\/master\/dataset\/bbcsport.csv\")\n\tdf['pca'] = (\n\t\tdf['text']\n\t\t\t.pipe(hero.clean)\n\t\t\t.pipe(hero.tfidf)\n\t\t\t.pipe(hero.pca)\n\t\t)\n\thero.scatterplot(df, 'pca', color='topic', title=\"PCA BBC Sport news\")"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/d9\/1b\/d9002dbb81fda60d82d3a3e765107f1b.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"用TextHero可以完成的工作還有很多，請繼續查閱"},{"type":"link","attrs":{"href":"https:\/\/texthero.org\/docs\/api-preprocessing","title":null,"type":null},"content":[{"type":"text","text":"文檔"}]},{"type":"text","text":"，瞭解數據清理和預處理、可視化、表示、基本NLP任務等相關信息。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"3."},{"type":"link","attrs":{"href":"https:\/\/spacy.io\/","title":null,"type":null},"content":[{"type":"text","text":"spaCy"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"spaCy是專門設計的，其宗旨是成爲一個用於實現生產就緒系統的有用庫。以下是關於它的介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"spaCy旨在幫助你完成真正的工作——構建真正的產品，或收集真正的見解。這個庫尊重你的寶貴時間，並儘量避免浪費它。它很容易安裝，其API簡單而高效。我們願意將spaCy視爲自然語言處理領域的Ruby on Rails。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"所以當你準備開始做一些真正的工作時，你需要先安裝spaCy和至少一個語言模型。在下面這個例子中我們將使用它的英語語言模型。庫和語言模型只需幾行代碼即可安裝："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"pip install spacy\npython -m spacy download en"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"要開始使用spaCy，我們將使用示例文本的這句話："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"sample = u\"I can't imagine spending $3000 for a single bedroom apartment in N.Y.C.\""}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在我們導入spaCy和一個英文停用詞列表。我們還將英語語言模型作爲Language對象加載（根據spaCy約定，我們將其稱爲“nlp”），然後在示例文本上調用nlp對象，它會返回一個經過處理的Doc對象（我們將其稱爲“doc”）。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"import spacy\nfrom spacy.lang.en.stop_words import STOP_WORDS\nnlp = spacy.load('en')\ndoc = nlp(sample)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這樣就行了？根據spaCy"},{"type":"link","attrs":{"href":"https:\/\/spacy.io\/usage\/spacy-101","title":null,"type":null},"content":[{"type":"text","text":"文檔"}]},{"type":"text","text":"："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"即使一個Doc經過了處理——例如拆分爲單個單詞並註釋——它仍"},{"type":"text","marks":[{"type":"strong"}],"text":"包含原始文本的所有信息"},{"type":"text","text":"，如空格字符。你隨時可以將一個符號（token）的偏移量獲取到原始字符串中，或​​者將符號及其尾隨空格連接起來來重建原始字符串。這樣，在使用spaCy處理文本時，你永遠不會丟失任何信息。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"現在我們來看看處理過的樣本："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"# Print out tokens\nprint(\"Tokens:\\n=======)\nfor token in doc:\n print(token)\n# Identify stop words\nprint(\"Stop words:\\n===========\")\nfor word in doc:\n if word.is_stop == True:\n print(word)\n# POS tagging\nprint(\"POS tagging:\\n============\")\nfor token in doc:\n print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n token.shape_, token.is_alpha, token.is_stop)\n# Print out named entities\nprint(\"Named entities:\\n===============\")\nfor ent in doc.ents:\n print(ent.text, ent.start_char, ent.end_char, ent.label_)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"Tokens:\n=======\nI\nca\nn't\nimagine\nspending\n$\n3000\nfor\na\nsingle\nbedroom\napartment\nin\nN.Y.C.\nStop words:\n===========\nca\nfor\na\nin\nPOS tagging:\n============\nI -PRON- PRON PRP nsubj X True False\nca can VERB MD aux xx True True\nn't not ADV RB neg x'x False False\nimagine imagine VERB VB ROOT xxxx True False\nspending spend VERB VBG xcomp xxxx True False\n$ $ SYM $ nmod $ False False\n3000 3000 NUM CD dobj dddd False False\nfor for ADP IN prep xxx True True\na a DET DT det x True True\nsingle single ADJ JJ amod xxxx True False\nbedroom bedroom NOUN NN compound xxxx True False\napartment apartment NOUN NN pobj xxxx True False\nin in ADP IN prep xx True True\nN.Y.C. n.y.c. PROPN NNP pobj X.X.X. False False\nNamed entities:\n===============\n3000 26 30 MONEY\nN.Y.C. 65 71 GPE"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"spaCy功能強大、堅持己見（opinionated），可用於從預處理到表示再到建模的各種NLP任務。查看spaCy"},{"type":"link","attrs":{"href":"https:\/\/spacy.io\/usage\/spacy-101","title":null,"type":null},"content":[{"type":"text","text":"文檔"}]},{"type":"text","text":"，看看你可以用它做哪些事情。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"4."},{"type":"link","attrs":{"href":"https:\/\/github.com\/huggingface\/transformers","title":null,"type":null},"content":[{"type":"text","text":"Hugging Face Transformers"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Hugging Face的Transformers庫已成爲NLP實踐不可或缺的一部分，這一點再怎麼強調也不爲過。根據GitHub存儲庫的介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"用於PyTorch和TensorFlow 2.0的一流自然語言處理"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Transformers提供了數千個預訓練模型，可以對100多種語言的文本執行分類、信息提取、問答、摘要、翻譯、文本生成等任務。它的目標是讓所有人都更容易使用尖端的NLP技術。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Transformers提供了很多API，可以用來在給定文本上快速下載和使用這些預訓練模型，在你自己的數據集上對它們進行微調，然後在我們的模型中心與社區共享。同時，定義架構的python模塊都可以作爲一個獨立的模塊進行修改，以實現快速的研究實驗。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Transformers由兩個最流行的深度學習庫PyTorch和TensorFlow提供支持，它們之間無縫集成，允許你使用一個模型來訓練你的模型，然後加載它來推理另一個。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"你可以使用"},{"type":"link","attrs":{"href":"https:\/\/transformer.huggingface.co\/","title":null,"type":null},"content":[{"type":"text","text":"Write With Transformer"}]},{"type":"text","text":"在線測試Transformer庫，這是該庫的官方功能演示。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這個很複雜的庫安裝起來卻很簡單："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"pip install transformers"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Transformers庫包羅萬象，你可以花很多時間學習它的所有細節。然而，它自帶的管道API讓你可以立即使用模型，幾乎不需要配置。以下是使用Transformers管道進行分類的一個示例（請注意，應先安裝TensorFlow或PyTorch才能繼續）："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"from transformers import pipeline\n# Allocate a pipeline for sentiment-analysis\nclassifier = pipeline('sentiment-analysis')\n# Classify text\nprint(classifier('I am a fan of KDnuggets, its useful content, and its helpful editors!'))"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"[{'label': 'POSITIVE', 'score': 0.9954679012298584}]"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"多簡單，很有趣吧。這個管道使用了一個預訓練的模型以及用於該模型的預處理，即使沒有微調，結果也非常不錯。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"下面是第二個管道示例，這次是問題回答："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"from transformers import pipeline\n# Allocate a pipeline for question-answering\nquestion_answerer = pipeline('question-answering')\n# Ask a question\nanswer = question_answerer({\n\t'question': 'Where is KDnuggets headquartered?',\n\t'context': 'KDnuggets was founded in February of 1997 by Gregory Piatetsky in Brookline, Massachusetts.'\n})\n# Print the answer\nprint(answer)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"{'score': 0.9153624176979065, 'start': 66, 'end': 90, 'answer': 'Brookline, Massachusetts'}"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"當然上面是一對簡單的示例，但這些管道非常強大，絕不止是解決一些與KDnuggets相關的瑣碎任務那麼簡單！你可以在"},{"type":"link","attrs":{"href":"https:\/\/huggingface.co\/transformers\/main_classes\/pipelines.html","title":null,"type":null},"content":[{"type":"text","text":"此處"}]},{"type":"text","text":"閱讀有關管道的更多信息。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Transformers讓最先進的模型也能輕鬆供所有人使用。請訪問它的GitHub"},{"type":"link","attrs":{"href":"https:\/\/github.com\/huggingface\/transformers","title":null,"type":null},"content":[{"type":"text","text":"存儲庫"}]},{"type":"text","text":"，探索更多精彩。"}]},{"type":"heading","attrs":{"align":null,"level":2},"content":[{"type":"text","text":"5."},{"type":"link","attrs":{"href":"https:\/\/github.com\/JasonKessler\/scattertext","title":null,"type":null},"content":[{"type":"text","text":"Scattertext"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Scattertext用於創建吸引人的可視化圖像，來描述語言在不同文檔類型之間的差異。根據其GitHub倉庫的介紹："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"這是一種用於在語料庫中查找區分術語，並將它們呈現在交互式HTML散點圖中的工具。與術語對應的點被有選擇地標記出來，防止它們與其他標籤或點重疊。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"還沒搞懂的話，我們先來安裝它："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"pip install scattertext"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"以下示例來自它的GitHub存儲庫，可視化了2012年美國大選中使用的術語。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"blockquote","content":[{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"2,000個與黨派最相關的一元分詞（unigram）顯示爲散點圖中的點。它們的x軸和y軸分別是共和黨和民主黨發言人使用它們的密集等級。"}]}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"請注意，運行示例代碼會生成一個HTML文件，然後可以在瀏覽器中查看該文件並與之交互。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"codeblock","attrs":{"lang":null},"content":[{"type":"text","text":"import scattertext as st\ndf = st.SampleCorpora.ConventionData2012.get_data().assign(\n parse=lambda df: df.text.apply(st.whitespace_nlp_with_sentences)\n)\ncorpus = st.CorpusFromParsedDocuments(\n df, category_col='party', parsed_col='parse'\n).build().get_unigram_corpus().compact(st.AssociationCompactor(2000))\nhtml = st.produce_scattertext_explorer(\n corpus,\n category='democrat', category_name='Democratic', not_category_name='Republican',\n minimum_term_frequency=0, pmi_threshold_coefficient=0,\n width_in_pixels=1000, metadata=corpus.get_df()['speaker'],\n transform=st.Scalers.dense_rank\n)\nopen('.\/demo_compact.html', 'w').write(html)"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"查看時保存的HTML文件的結果（下面顯示的是靜態圖像，因此不是交互式的）："}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"image","attrs":{"src":"https:\/\/static001.infoq.cn\/resource\/image\/0c\/82\/0c65161f3060d4a07ebc027731b5a882.jpg","alt":null,"title":"","style":[{"key":"width","value":"75%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"Scattertext的用途很窄，但效果很好。它的可視化輸出絕對很漂亮，而且富含見解。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":" "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"原文鏈接："},{"type":"link","attrs":{"href":"https:\/\/www.kdnuggets.com\/2021\/02\/getting-started-5-essential-nlp-libraries.html","title":null,"type":null},"content":[{"type":"text","text":"https:\/\/www.kdnuggets.com\/2021\/02\/getting-started-5-essential-nlp-libraries.html"}]}]}]}