Run the code

BERT models need to be dowloaded (with the exception of CamemBERT)

Training:

CUDA_VISIBLE_DEVICES=1,2,3 python main.py 
--directory TEMP_MODEL # path to save the model; predictions on test/dev will be automatically saved here at the end of training
--pre_trained_model PRETRAINED_MODEL_NAME #bert-base-cased
--train_dataset train.tsv 
--test_dataset test.tsv 
--dev_dataset valid.tsv 
--batch_size 4 
--do_train 
--no_cpu 5
--language french #for CamemBERT; english for other models
--model stacked # or bert 
--num_layers 2 #2 Transformer layers

Predicting:

python main.py 
--directory TEMP_MODEL #same param as train.py
--pre_trained_model PRETRAINED_MODEL_NAME #same param as main.py
--train_dataset train.tsv #same param as main.py
--test_dataset test.tsv #same param as main.py
--dev_dataset valid.tsv #same param as main.py
--dataset_dir DIR_DATA_TEST #directory with .tsv to be predicted
--output_dir DIR_DATA_TEST_PREDICTIONS #directory where predictions will be saved
--batch_size 4 
--do_eval 
--saved_model TEMP_MODEL/best/best_ #best model after training
--no_cpu 5
--language french #for CamemBERT; english for other; same as main.py
--model stacked # or bert; same as main.py
--num_layers 2 #2 Transformer layers; same as main.py

Dataset Annotation

TOKEN    NE-COARSE-LIT    NE-COARSE-METO    NE-FINE-LIT    NE-FINE-METO    NE-FINE-COMP    NE-NESTED    NEL-LIT    NEL-METO    MISC
# language = fr
# newspaper = GDL
# date = 1878-02-22
# document_id = GDL-1878-02-22-a-i0014
# segment_iiif_link = _
LAUSANNE    B-loc    O    B-loc.adm.town    O    O    O    Q807    _    EndOfLine
On    O    O    O    O    O    O    _    _    _
nous    O    O    O    O    O    O    _    _    _
prie    O    O    O    O    O    O    _    _    _
de    O    O    O    O    O    O    _    _    _
faire    O    O    O    O    O    O    _    _    _
connaître    O    O    O    O    O    O    _    _    _
le    O    O    O    O    O    O    _    _    _
résultat    O    O    O    O    O    O    _    _    EndOfLine
Sécuniaire    O    O    O    O    O    O    _    _    _
des    O    O    O    O    O    O    _    _    _
quatre    O    O    O    O    O    O    _    _    _
conférences    O    O    O    O    O    O    _    _    _
sur    O    O    O    O    O    O    _    _    _
l'    O    O    O    O    O    O    _    _    NoSpaceAfter
Orient    B-loc    O    B-loc.adm.sup    O    O    O    Q205653    _    EndOfLine
M    B-pers    O    B-pers.ind    O    B-comp.title    O    Q123894    _    NoSpaceAfter
.    I-pers    O    I-pers.ind    O    I-comp.title    O    Q123894    _    _
le    I-pers    O    I-pers.ind    O    O    O    Q123894    _    _
professeur    I-pers    O    I-pers.ind    O    B-comp.function    O    Q123894    _    _
Gilliéron    I-pers    O    I-pers.ind    O    B-comp.name    O    Q123894    _    NoSpaceAfter
.    O    O    O    O    O    O    _    _    EndOfLine

Requirements

pip install -r requirements.txt

How to citate:

@inproceedings{boros2020robust,
  title={Robust named entity recognition and linking on historical multilingual documents},
  author={Boros, Emanuela and Pontes, Elvys Linhares and Cabrera-Diego, Luis Adri{\'a}n and Hamdi, Ahmed and Moreno, Jos{\'e} and Sid{\`e}re, Nicolas and Doucet, Antoine},
  booktitle={Conference and Labs of the Evaluation Forum (CLEF 2020)},
  volume={2696},
  number={Paper 171},
  pages={1--17},
  year={2020},
  organization={CEUR-WS Working Notes}
}

@inproceedings{borocs2020alleviating,
  title={Alleviating digitization errors in named entity recognition for historical documents},
  author={Boro{\c{s}}, Emanuela and Hamdi, Ahmed and Pontes, Elvys Linhares and Cabrera-Diego, Luis-Adri{\'a}n and Moreno, Jose G and Sidere, Nicolas and Doucet, Antoine},
  booktitle={Proceedings of the 24th Conference on Computational Natural Language Learning},
  pages={431--441},
  year={2020}
}