SuPar-UniDic

Tokenizer, POS-tagger, lemmatizer, and dependency-parser for modern and contemporary Japanese with BERT models.

Basic usage

>>> import suparunidic
>>> nlp=suparunidic.load()
>>> doc=nlp("太郎は花子が読んでいる本を次郎に渡した")
>>> print(type(doc))
<class 'spacy.tokens.doc.Doc'>
>>> print(suparunidic.to_conllu(doc))
1    太郎    タロウ    PROPN    名詞-固有名詞-人名-名    _    12    nsubj    _    SpaceAfter=No|Translit=タロー
2    は    は    ADP    助詞-係助詞    _    1    case    _    SpaceAfter=No|Translit=ワ
3    花子    ハナコ    PROPN    名詞-固有名詞-人名-名    _    5    nsubj    _    SpaceAfter=No|Translit=ハナコ
4    が    が    ADP    助詞-格助詞    _    3    case    _    SpaceAfter=No|Translit=ガ
5    読ん    読む    VERB    動詞-一般    _    8    acl    _    SpaceAfter=No|Translit=ヨン
6    で    て    SCONJ    助詞-接続助詞    _    5    mark    _    SpaceAfter=No|Translit=デ
7    いる    居る    AUX    動詞-非自立可能    _    5    aux    _    SpaceAfter=No|Translit=イル
8    本    本    NOUN    名詞-普通名詞-一般    _    12    obj    _    SpaceAfter=No|Translit=ホン
9    を    を    ADP    助詞-格助詞    _    8    case    _    SpaceAfter=No|Translit=オ
10    次郎    ジロウ    PROPN    名詞-固有名詞-人名-名    _    12    obl    _    SpaceAfter=No|Translit=ジロー
11    に    に    ADP    助詞-格助詞    _    10    case    _    SpaceAfter=No|Translit=ニ
12    渡し    渡す    VERB    動詞-一般    _    0    root    _    SpaceAfter=No|Translit=ワタシ
13    た    た    AUX    助動詞    _    12    aux    _    SpaceAfter=No|Translit=タ
>>> import deplacy
>>> deplacy.render(doc,Japanese=True)
太郎 PROPN ═╗<════════╗ nsubj(主語)
は   ADP   <╝         ║ case(格表示)
花子 PROPN ═╗<══╗     ║ nsubj(主語)
が   ADP   <╝   ║     ║ case(格表示)
読ん VERB  ═╗═╗═╝<╗   ║ acl(連体修飾節)
で   SCONJ <╝ ║   ║   ║ mark(標識)
いる AUX   <══╝   ║   ║ aux(動詞補助成分)
本   NOUN  ═╗═════╝<╗ ║ obj(目的語)
を   ADP   <╝       ║ ║ case(格表示)
次郎 PROPN ═╗<╗     ║ ║ obl(斜格補語)
に   ADP   <╝ ║     ║ ║ case(格表示)
渡し VERB  ═╗═╝═════╝═╝ ROOT(親)
た   AUX   <╝           aux(動詞補助成分)
>>> from deplacy.deprelja import deprelja
>>> for b in suparunidic.bunsetu_spans(doc):
...   for t in b.lefts:
...     print(suparunidic.bunsetu_span(t),"->",b,"("+deprelja[t.dep_]+")")
...
花子が -> 読んでいる (主語)
読んでいる -> 本を (連体修飾節)
太郎は -> 渡した (主語)
本を -> 渡した (目的語)
次郎に -> 渡した (斜格補語)

suparunidic.load(UniDic,BERT) loads a natural language processor pipeline, which uses UniDic for tokenizer POS-tagger and lemmatizer, then uses BERT for Biaffine dependency-parser of SuPar. Available UniDic options are:

• UniDic="gendai": Use 現代書き言葉UniDic
• UniDic="spoken": Use 現代話し言葉UniDic
• UniDic="novel": Use 近現代口語小説UniDic.
• UniDic="qkana": Use 旧仮名口語UniDic
• UniDic="kindai": Use 近代文語UniDic
• UniDic="kinsei": Use 近世江戸口語UniDic
• UniDic="kyogen": Use 中世口語UniDic
• UniDic="wakan": Use 中世文語UniDic
• UniDic="wabun": Use 中古和文UniDic
• UniDic="manyo": Use 上代語UniDic
• UniDic=None unidic-lite (default)

Available BERT options are:

• BERT="bert-japanese-aozora6m3m-unidic32k-2m" from bert-japanese-aozora (default)
• BERT="roberta-large-japanese-aozora" from roberta-large-japanese-aozora
• BERT="roberta-large-japanese-aozora-char" from roberta-large-japanese-aozora-char
• BERT="roberta-base-japanese-aozora" from roberta-base-japanese-aozora
• BERT="roberta-base-japanese-aozora-char" from roberta-base-japanese-aozora-char
• BERT="roberta-small-japanese-aozora" from roberta-small-japanese-aozora
• BERT="roberta-small-japanese-aozora-char" from roberta-small-japanese-aozora-char
• BERT="deberta-large-japanese-aozora" from deberta-large-japanese-aozora
• BERT="deberta-base-japanese-aozora" from deberta-base-japanese-aozora
• BERT="deberta-small-japanese-aozora" from deberta-small-japanese-aozora
• BERT="deberta-base-japanese-unidic" from deberta-base-japanese-aozora (fugashi required)
• BERT="bert-base-japanese-char-extended" from bert-base-japanese-char-extended
• BERT="bert-large-japanese-char-extended" from bert-large-japanese-char-extended
• BERT="bert-base-japanese-char" from cl-tohoku (fugashi and ipadic required)
• BERT="bert-base-japanese-whole-word-masking" from cl-tohoku (fugashi and ipadic required)
• BERT="bert-large-japanese" from cl-tohoku (fugashi required)
• BERT="bert-large-japanese-char" from cl-tohoku (fugashi required)
• BERT="roberta-base-japanese" from nlp-waseda (SentencePiece required)
• BERT="roberta-large-japanese" from nlp-waseda (SentencePiece required)
• BERT="electra-base-japanese-discriminator" from izumi-lab (fugashi and ipadic required)
• BERT="bert-small-japanese" from izumi-lab (fugashi and ipadic required)
• BERT="electra-base-japanese-generator" from izumi-lab (fugashi and ipadic required)
• BERT="japanese-roberta-base" from rinna (SentencePiece required)
• BERT="albert-japanese-v2" from alinear-corp (SentencePiece required)
• BERT="albert-base-japanese-v1" from ken11
• BERT="electra-small-japanese-discriminator" from Cinnamon AI
• BERT="electra-small-japanese-generator" from Cinnamon AI
• BERT="ku-bert-japanese-large" from ku-bert-japanese
• BERT="bert-base-ja-cased" from Geotrend
• BERT="laboro-bert-japanese-large" from Laboro AI
• BERT="nict-bert-base-japanese-100k" from NICT BERT
• BERT="unihanlm-base" from microsoft/unihanlm-base
• BERT="distilbert-base-japanese" from bandainamco-mirai

Installation for Linux

pip3 install suparunidic --user

Installation for Cygwin64

Make sure to get python37-devel python37-pip python37-cython python37-numpy python37-wheel gcc-g++ mingw64-x86_64-gcc-g++ git curl make cmake, and then:

curl -L https://raw.githubusercontent.com/KoichiYasuoka/CygTorch/master/installer/supar.sh | sh
pip3.7 install suparunidic

Benchmarks

Results of 舞姬/雪國/荒野より-Benchmarks

BERT=”bert-japanese-aozora6m3m-unidic32k-2m”

BERT=”roberta-large-japanese-aozora”

BERT=”roberta-large-japanese-aozora-char”

BERT=”roberta-base-japanese-aozora”

BERT=”roberta-base-japanese-aozora-char”

BERT=”roberta-small-japanese-aozora”

BERT=”roberta-small-japanese-aozora-char”

BERT=”deberta-large-japanese-aozora”

BERT=”deberta-base-japanese-aozora”

BERT=”deberta-small-japanese-aozora”

BERT=”deberta-base-japanese-unidic”

BERT=”bert-base-japanese-char-extended”

BERT=”bert-large-japanese-char-extended”

BERT=”bert-base-japanese-char”

BERT=”bert-base-japanese-whole-word-masking”

BERT=”bert-large-japanese”

BERT=”bert-large-japanese-char”

BERT=”roberta-base-japanese”

BERT=”roberta-large-japanese”

BERT=”electra-base-japanese-discriminator”

BERT=”bert-small-japanese”

BERT=”electra-base-japanese-generator”

BERT=”japanese-roberta-base”

BERT=”albert-japanese-v2”

BERT=”albert-base-japanese-v1”

BERT=”electra-small-japanese-discriminator”

BERT=”electra-small-japanese-generator”

BERT=”ku-bert-japanese-large”

BERT=”bert-base-ja-cased”

BERT=”laboro-bert-japanese-large”

BERT=”nict-bert-base-japanese-100k”

BERT=”unihanlm-base”

BERT=”distilbert-base-japanese”

Author

Koichi Yasuoka (安岡孝一)