数据沙盘>> sentencepiece-jni>> 返回
项目作者： levyfan

项目描述 ：
Java JNI wrapper for SentencePiece: unsupervised text tokenizer for Neural Network-based text generation.
高级语言： C++
项目地址: git://github.com/levyfan/sentencepiece-jni.git
创建时间： 2019-05-12T10:37:21Z
项目社区：https://github.com/levyfan/sentencepiece-jni
开源协议：MIT License
下载


SentencePiece Java Wrapper

Build

Java wrapper for SentencePiece with JNI. This module wraps
sentencepiece::SentencePieceProcessor class with the following modifications:

  • Encode and Decode methods are re-defined as EncodeAsIds, EncodeAsPieces, DecodeIds and DecodePieces respectively.
  • SentencePieceText proto is not supported.

SentencePiece Version

v0.1.96

Build and Install SentencePiece

To build and install the Java wrapper from source, please try the following commands:

  1. % mvn clean install

Using sentencepiece-jni as a dependency

Because the resulting JAR is platform-dependent, resolving this dependency is managed by the os-maven-plugin. Follow the instructions there to use this platform-dependent JAR.

Please note you need to have a C++ compiler and cmake installed.

Usage

See SentencePieceProcessorTest for more.