项目作者: mponza

项目描述 :
Salient Open Information Extraction
高级语言: Java
项目地址: git://github.com/mponza/SalIE.git
创建时间: 2018-08-06T07:07:12Z
项目社区:https://github.com/mponza/SalIE

开源协议:GNU General Public License v3.0

下载


SalIE - Salient Open Information Extraction

This repository hosts SalIE, the first framework addressing the extraction of salient open facts from arbitrary text.

Building

Requirements. For running SalIE out-of-the-box, your machine needs only to have Java and sbt preinstalled.

Setting-up. Download this repository recursively, apply the patch with:

  1. git clone --recursive https://github.com/mponza/SalIE
  2. cd SalIE
  3. bash src/main/bash/patch.sh

Then download the precrafted embeddings from here in path/to/downloaded/zip and unzip the archive into data/embeddings with:

  1. mkdir -p data/embeddings
  2. unzip path/to/downloaded/zip -d data

Now you are ready to run SalIE.

Running. Given a data collection stored in path/to/input/data folder, where each element of the folder is a text file (i.e., .txt), you can extract its
salient open facts with SalIE by typing:

  1. src/main/bash/salient-extraction.sh path/to/input/data path/to/output/data minieMode

where minieMode can be agg, safe, dict or comp (for, respectively, aggressive, safe, dictionary and complete MinIE’s mode) and path/to/output/data is the folder on which the salient open facts will be stored in the following JSON format:

  1. {
  2. "docID": string document ID
  3. "text": string content of the document
  4. "openfacts": list list of salient open facts, sorted by descending salience score
  5. [
  6. {
  7. "text": string text of the open fact
  8. "salience": float salience score
  9. }
  10. ]
  11. }

Using SalIE within your Code. An Example is provied in src/main/scala/de/mpg/mpi/runners/RunExample.scala.
The code has been developed on the top of DkPro/UIMA frameworks, for more information, please check the documentation in their official websites.

Embeddings Wikipedia Open Facts via GloVe and Data Compression

We describe here the procedure used for generating the embedding vectors from the set of open facts extracted from the whole Wikipedia. If you have generated your own embeddings, you can easily adapt this procedure to plug them into SalIE.

Setting-up. After you have clone this repository with --recursive option you have to install and create a
virtualenv environment in the venv directory:

  1. virtualenv venv

and install the Python requirements:

  1. source venv/bin/activate
  2. pip install -r src/main/python/requirements.txt

Embeddings Generation & Compression. Given a file of open facts in JSON format (e.g., path/to/agg-wikipedia.json,
section Dataset of Wikipedia Open Facts for the description of the format), the output embeddings file (e.g., path/to/agg-wikipedia.glove) can be generated from scratch with:

  1. bash src/main/bash/facts2glove.sh path/to/agg-wikipedia.json path/to/agg-wikipedia.glove

In this example, the output filename will be path/to/agg-wikipedia.glove.bin. For setting up different GloVe parameters check src/main/bash/facts2glove.sh.

Evaluation

If you want to evaluate the performance of SalIE on a data collection, just set-up and run the following steps.

Setting-up. After the creation and activation of your virtualenv environment, you need to install the Python requirements:

  1. pip install -r src/main/python/requirements.txt

Then, you have to configure pyrouge by creating the file ~/.pyrouge/settings.ini
with the content:

  1. [pyrouge settings]
  2. home_dir = path/to/src/main/python/summarization/tools/ROUGE-1.5.5/

Evaluation. For evaluating a set of extracted salient facts with respect to documents’ abstracts you need to run:

  1. python src/main/python/summarization evaluate path/to/open/facts/dir path/to/abstracts path/to/scores.json

where path/to/open/facts/dir is the path to a directory of a set of documents, each one with the following JSON format:

  1. {
  2. "docID": string id of the document
  3. "text": string content of the document
  4. "abstract": string abstract of the document
  5. "openfacts": list list of open facts
  6. [
  7. {
  8. "text": string text of the open fact
  9. "salience": float salience score
  10. }
  11. ]
  12. }

and the path/to/abstracts is the path to a directory of a set of documents, each one containing the document’s abstract.
On the NYT dataset results are slightly different from the ones in the paper because minor changes for this release.

Known Error (and How to Fix). Running ROUGE can raise the “Cannot open exception db file for reading” exception. For fix it, just type:

  1. bash src/main/fixROUGE.sh

and then re-run the evaluation script.

NYT Dataset. This dataset can be bought from LDC, while the document IDs used in our testbed can be downloaded here.

Dataset of Wikipedia Open Facts

You can download the datasets containing the set of open facts extracted from Wikipedia (dump of August 2017) with different MinIE’s modes: aggressive, safe, dictionary and complete.
Each file size is about 9GB (compressed) and each line is a Wikipedia page with the following JSON format:

  1. {
  2. "wikiID": string id of the Wikipedia page
  3. "text": string raw text of the Wikipedia page
  4. "sentences": list list of sentences containing the extracted open facts
  5. [
  6. {
  7. "text": string sentence text
  8. "begin": int begin character offset of the sentence in the Wikipedia text
  9. "end": int end character offset of the sentence in the Wikipedia text
  10. "openFacts": list list of open facts extracted from the sentence (warning: F of facts is uppercase here!)
  11. [
  12. {
  13. "subject":
  14. {
  15. "text": string text of the subject
  16. "head": string head of the subject
  17. "begin": int begin character offset of the subject
  18. "end": int end character offset of the subject
  19. }
  20. "relation": same object structure of subject
  21. "object": same object structure of subject
  22. }
  23. ]
  24. }
  25. ]
  26. }

Citation and Further Reading

If you find any resource (code or data) of this repository useful, please cite our paper:

Marco Ponza, Luciano Del Corro, Gerhard Waikum

Facts That Matter

In Proceedings of the 2018 Conference of Empirical Methods in Natural Language Processing (EMNLP 2018)

In the following we list other relevant papers describing the tools we used in this work.

  • MinIE, the open information extraction tool default used by SalIE:

    Kiril Gashteovski, Rainer Gemulla and Luciano Del Corro

    MinIE: Minimizing Facts in Open Information Extraction

    In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)

  • Algorithms used for compressing word embeddings:

    Roi Blanco, Giuseppe Ottaviano and Edgar Meij

    Fast and Space-Efficient Entity Linking for Queries

    In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (WSDM 2015)

License

The code in this repository has been released under GNU General Public License v.3.0