项目作者: meta-toolkit

项目描述 :
A Modern C++ Data Sciences Toolkit
高级语言: C++
项目地址: git://github.com/meta-toolkit/meta.git
创建时间: 2014-02-02T23:54:34Z
项目社区:https://github.com/meta-toolkit/meta

开源协议:Other

下载


MeTA: ModErn Text Analysis

Please visit our web page for information and tutorials
about MeTA!

Build Status (by branch)

  • master: Build
Status
    Windows Build
Status
  • develop: Build
Status
    Windows Build
Status

Outline

Intro

MeTA is a modern C++ data sciences toolkit featuring

  • text tokenization, including deep semantic features like parse trees
  • inverted and forward indexes with compression and various caching strategies
  • a collection of ranking functions for searching the indexes
  • topic models
  • classification algorithms
  • graph algorithms
  • language models
  • CRF implementation (POS-tagging, shallow parsing)
  • wrappers for liblinear and libsvm (including libsvm dataset parsers)
  • UTF8 support for analysis on various languages
  • multithreaded algorithms

Documentation

Doxygen documentation can be found here.

Tutorials

We have walkthroughs for a few different parts of MeTA on the
MeTA homepage.

Citing

If you used MeTA in your research, we would greatly appreciate a citation for
our ACL demo paper:

@InProceedings{meta-toolkit, author = {Massung, Sean and Geigle, Chase and Zhai, Cheng{X}iang}, title = {{MeTA: A Unified Toolkit for Text Retrieval and Analysis}}, booktitle = {Proceedings of ACL-2016 System Demonstrations}, month = {August}, year = {2016}, address = {Berlin, Germany}, publisher = {Association for Computational Linguistics}, pages = {91--96}, url = {http://anthology.aclweb.org/P16-4016} }

Project setup

Mac OS X Build Guide

Mac OS X 10.6 or higher is required. You may have success with 10.5, but
this is not tested.

You will need to have homebrew installed, as well as the
Command Line Tools for Xcode (homebrew requires these as well, and it will
prompt for them during install, or you can install them with xcode-select --install on recent versions of OS X).

Once you have homebrew installed, run the following commands to get the
dependencies for MeTA:

  1. brew update
  2. brew install cmake jemalloc lzlib icu4c

To get started, run the following commands:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. CXX=clang++ cmake ../ -DCMAKE_BUILD_TYPE=Release -DICU_ROOT=/usr/local/opt/icu4c
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

Ubuntu Build Guide

The directions here depend greatly on your installed version of Ubuntu. To
check what version you are on, run the following command:

  1. cat /etc/issue

Based on what you see, you should proceed with one of the following guides:

If your version is less than 12.04 LTS, your operating system is not
supported (even by your vendor!) and you should upgrade to at least 12.04
LTS (or 14.04 LTS, if possible).

Ubuntu 12.04 LTS Build Guide

Building on Ubuntu 12.04 LTS requires more work than its more up-to-date
14.04 sister, but it can be done relatively easily. You will, however, need
to install a newer C++ compiler from a ppa, and switch to it in order to
build meta. We will also need to install a newer CMake version than is
natively available.

Start by running the following commands to get the dependencies that we
will need for building MeTA.

  1. # this might take a while
  2. sudo apt-get update
  3. sudo apt-get install python-software-properties
  4. # add the ppa that contains an updated g++
  5. sudo add-apt-repository ppa:ubuntu-toolchain-r/test
  6. sudo apt-get update
  7. # this will probably take a while
  8. sudo apt-get install g++ g++-4.8 git make wget libjemalloc-dev zlib1g-dev
  9. wget http://www.cmake.org/files/v3.2/cmake-3.2.0-Linux-x86_64.sh
  10. sudo sh cmake-3.2.0-Linux-x86_64.sh --prefix=/usr/local

During CMake installation, you should agree to the license and then say “n”
to including the subdirectory. You should be able to run the following
commands and see the following output:

  1. g++-4.8 --version

should print

  1. g++-4.8 (Ubuntu 4.8.1-2ubuntu1~12.04) 4.8.1
  2. Copyright (C) 2013 Free Software Foundation, Inc.
  3. This is free software; see the source for copying conditions. There is NO
  4. warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

and

  1. /usr/local/bin/cmake --version

should print

  1. cmake version 3.2.0
  2. CMake suite maintained and supported by Kitware (kitware.com/cmake).

Once the dependencies are all installed, you should be ready to build. Run
the following commands to get started:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. CXX=g++-4.8 /usr/local/bin/cmake ../ -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

Ubuntu 14.04 LTS Build Guide

Ubuntu 14.04 has a recent enough GCC for building MeTA, but we’ll need to
add a ppa for a more recent version of CMake.

Start by running the following commands to install the dependencies for
MeTA.

  1. # this might take a while
  2. sudo apt-get update
  3. sudo apt-get install software-properties-common
  4. # add the ppa for cmake
  5. sudo add-apt-repository ppa:george-edison55/cmake-3.x
  6. sudo apt-get update
  7. # install dependencies
  8. sudo apt-get install g++ cmake libicu-dev git libjemalloc-dev zlib1g-dev

Once the dependencies are all installed, you should double check your
versions by running the following commands.

  1. g++ --version

should output

  1. g++ (Ubuntu 4.8.2-19ubuntu1) 4.8.2
  2. Copyright (C) 2013 Free Software Foundation, Inc.
  3. This is free software; see the source for copying conditions. There is NO
  4. warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

and

  1. cmake --version

should output

  1. cmake version 3.2.2
  2. CMake suite maintained and supported by Kitware (kitware.com/cmake).

Once the dependencies are all installed, you should be ready to build. Run
the following commands to get started:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. cmake ../ -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

Ubuntu 15.10 Build Guide

Ubuntu’s non-LTS desktop offering in 15.10 has enough modern software in
its repositories to build MeTA without much trouble. To install the
dependencies, run the following commands.

  1. apt update
  2. apt install g++ git cmake make libjemalloc-dev zlib1g-dev

Once the dependencies are all installed, you should be ready to build. Run
the following commands to get started:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. cmake ../ -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

Arch Linux Build Guide

Arch Linux consistently has the most up to date packages due to its rolling
release setup, so it’s often the easiest platform to get set up on.

To install the dependencies, run the following commands.

  1. sudo pacman -Sy
  2. sudo pacman -S clang cmake git icu libc++ make jemalloc zlib

Once the dependencies are all installed, you should be ready to build. Run
the following commands to get started:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. CXX=clang++ cmake ../ -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

Fedora Build Guide

This has been tested with Fedora 22+ (the oldest currently supported Fedora
as of the time of writing). You may have success with earlier versions, but
this is not tested. (If you’re on an older version of Fedora, use yum
instead of dnf for the commands given below.)

To get started, install some dependencies:

  1. # These may be already installed
  2. sudo dnf install make git wget gcc-c++ jemalloc-devel cmake zlib-devel

You should be able to run the following commands and see the following
output:

  1. g++ --version

should print

  1. g++ (GCC) 5.3.1 20151207 (Red Hat 5.3.1-2)
  2. Copyright (C) 2015 Free Software Foundation, Inc.
  3. This is free software; see the source for copying conditions. There is NO
  4. warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

and

  1. cmake --version

should print

  1. cmake version 3.3.2
  2. CMake suite maintained and supported by Kitware (kitware.com/cmake).

Once the dependencies are all installed, you should be ready to build. Run
the following commands to get started:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. cmake ../ -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system with the following command:

  1. ./unit-test --reporter=spec

CentOS Build Guide

MeTA can be built in CentOS 7 and above. CentOS 7 comes with a recent
enough compiler (GCC 4.8.5), but too old a version of CMake. We’ll thus
install the compiler and related libraries from the package manager and
install our own more recent cmake ourselves.

  1. # install build dependencies (this will probably take a while)
  2. sudo yum install gcc gcc-c++ git make wget zlib-devel epel-release
  3. sudo yum install jemalloc-devel
  4. wget http://www.cmake.org/files/v3.2/cmake-3.2.0-Linux-x86_64.sh
  5. sudo sh cmake-3.2.0-Linux-x86_64.sh --prefix=/usr/local --exclude-subdir

You should be able to run the following commands and see the following
output:

  1. g++ --version

should print

  1. g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
  2. Copyright (C) 2015 Free Software Foundation, Inc.
  3. This is free software; see the source for copying conditions. There is NO
  4. warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

and

  1. /usr/local/bin/cmake --version

should print

  1. cmake version 3.2.0
  2. CMake suite maintained and supported by Kitware (kitware.com/cmake).

Once the dependencies are all installed, you should be ready to build. Run
the following commands to get started:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. /usr/local/bin/cmake ../ -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

EWS/EngrIT Build Guide

Note: Please don’t do this if you are able to get MeTA working in any
other possible way
, as the EWS filesystem has a habit of being
unbearably slow and increasing compile times by several orders of
magnitude. For example, comparing the cmake, make, and unit-test
steps on my desktop vs. EWS gives the following:

system cmake time make time unit-test time
my desktop 0m7.523s 2m30.715s 0m36.631s
EWS 1m28s 11m28.473s 1m25.326s

If you are on a machine managed by Engineering IT at UIUC, you should
follow this guide. These systems have software that is much too old for
building MeTA, but EngrIT has been kind enough to package updated versions
of research software as modules. The modules provided for GCC and CMake are
recent enough to build MeTA, so it is actually mostly straightforward.

To set up your dependencies (you will need to do this every time you log
back in to the system
), run the following commands:

  1. module load gcc
  2. module load cmake/3.5.0

Once you have done this, double check your versions by running the
following commands.

  1. g++ --version

should output

  1. g++ (GCC) 5.3.0
  2. Copyright (C) 2015 Free Software Foundation, Inc.
  3. This is free software; see the source for copying conditions. There is NO
  4. warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

and

  1. cmake --version

should output

  1. cmake version 3.5.0
  2. CMake suite maintained and supported by Kitware (kitware.com/cmake).

If your versions are correct, you should be ready to build. To get started,
run the following commands:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta/
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. CXX=`which g++` CC=`which gcc` cmake ../ -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

Windows Build Guide

MeTA can be built on Windows using the MinGW-w64 toolchain with gcc. We
strongly recommend using MSYS2 as this makes fetching the compiler
and related libraries significantly easier than it would be otherwise, and
it tends to have very up-to-date packages relative to other similar MinGW
distributions.

Note: If you find yourself confused or lost by the instructions below,
please refer to our visual setup guide for
Windows
which includes
screenshots for every step, including updating MSYS2 and the MinGW-w64
toolchain.

To start, download the installer for MSYS2 from the linked
website and follow the instructions on that page. Once you’ve got it
installed, you should use the MinGW shell to start a new terminal, in which
you should run the following commands to download dependencies and related
software needed for building:

  1. pacman -Syu git make patch mingw-w64-x86_64-{gcc,cmake,icu,jemalloc,zlib} --force

(the --force is needed to work around a bug with the latest MSYS2
installer as of the time of writing.)

Then, exit the shell and launch the “MinGW-w64 Win64” shell. You can obtain
the toolkit and get started with:

  1. # clone the project
  2. git clone https://github.com/meta-toolkit/meta.git
  3. cd meta
  4. # set up submodules
  5. git submodule update --init --recursive
  6. # set up a build directory
  7. mkdir build
  8. cd build
  9. cp ../config.toml .
  10. # configure and build the project
  11. cmake .. -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release
  12. make

You can now test the system by running the following command:

  1. ./unit-test --reporter=spec

If everything passes, congratulations! MeTA seems to be working on your
system.

Generic Setup Notes

  • There are rules for clean, tidy, and doc. After you run the cmake
    command once, you will be able to just run make as usual
    when you’re
    developing—-it’ll detect when the CMakeLists.txt file has changed and
    rebuild Makefiles if it needs to.

  • To compile in debug mode, just replace Release with Debug in the
    appropriate cmake command for your OS above and rebuild using make
    after.

  • Don’t hesitate to reach out on the forum if you encounter
    problems getting set up. We routinely build with a wide variety of
    compilers and operating systems through our continuous integration
    setups (travis-ci for Linux and OS X and
    Appveyor for Windows), so we can be fairly certain that
    things should build on nearly all major platforms.