Data augmentation pipeline that can add fuzzy target translations to an input sentence to improve the performance of an MT system. Fuzzy matches can be found with edit distance, set similarity and semantic matching.
For basic usage you can simply install the library via clone from git and pip install.
git clone https://github.com/lt3/nfr.gitcd nfrpip install .
By default, semantic matching capabilities with sent2vec and Sentence Transformers are not enabled because the
dependencies are considerably large. If you want to enable semantic matching, you need to install FAISS and one of
Sentence Transformers or Sent2Vec.
pip install faiss-cpu or pip install faiss-gpu)pip install sentence-transformers)torch will betorch beforesentence-transformers.After installation, four commands are exposed. In all cases, you can type <command> -h for these usage instructions.
nfr-create-faiss-index: Creates a FAISS index for semantic matches with Sent2Vec or Sentence Transformers.
usage: nfr-create-faiss-index [-h] -c CORPUS_F -p MODEL_NAME_OR_PATH -oOUTPUT_F [-m {sent2vec,stransformers}][-b BATCH_SIZE] [--use_cuda]Create a FAISS index based on the semantic representation of an existing textcorpus. To do so, the text will be embedded by means of a sent2vec model or asentence-transformers model. The index is (basically) an efficient list thatcontains all the representations of the training corpus sentences (the TM). assuch, this index can later be used to find those entries that are most similarto a given representation of a sentence. The index is saved to a binary fileso that it can be reused later on to calculate cosine similarity scores and toretrieve the most resembling entries.optional arguments:-h, --help show this help message and exit-c CORPUS_F, --corpus_f CORPUS_FPath to the corpus to turn into vectors and add to theindex. This is typically your TM or training file foran MT system containing text, one sentence per line-p MODEL_NAME_OR_PATH, --model_name_or_path MODEL_NAME_OR_PATHPath to sent2vec model (when `method` is sent2vec) orsentence-transformers model name when method isstransformers (seehttps://www.sbert.net/docs/pretrained_models.html)-o OUTPUT_F, --output_f OUTPUT_FPath to the output file to write the FAISS index to-m {sent2vec,stransformers}, --mode {sent2vec,stransformers}Whether to use 'sent2vec' or 'stransformers'(sentence-transformers)-b BATCH_SIZE, --batch_size BATCH_SIZEBatch size to use to create sent2vec embeddings orsentence-transformers embeddings. A larger value willresult in faster creation, but may lead to an out-of-memory error. If you get such an error, lower thevalue.--use_cuda Whether to use GPU when using sentence-transformers.Requires PyTorch installation with CUDA support and aCUDA-enabled device
nfr-extract-fuzzy-matches: Here, fuzzy matches can be extracted from the training set. A variety of options are
usage: nfr-extract-fuzzy-matches [-h] --tmsrc TMSRC --tmtgt TMTGT --insrcINSRC --method{editdist,setsim,setsimeditdist,sent2vec,stransformers}--minscore MINSCORE --maxmatch MAXMATCH[--model_name_or_path MODEL_NAME_OR_PATH][--faiss FAISS] [--threads THREADS][--n_setsim_candidates N_SETSIM_CANDIDATES][--setsim_function SETSIM_FUNCTION][--use_cuda] [-q QUERY_MULTIPLIER][-v {info,debug}]Given source and target TM files, extract fuzzy matches for a new input fileby using a variety of methods. You can use formal matching methods such asedit distance and set similarity, as well as semantic fuzzy matching withsent2vec and Sentence Transformers.optional arguments:-h, --help show this help message and exit--tmsrc TMSRC Source text of the TM from which fuzzy matches will beextracted--tmtgt TMTGT Target text of the TM from which fuzzy matches will beextracted--insrc INSRC Input source file to extract matches for (insrc isqueried against tmsrc)--method {editdist,setsim,setsimeditdist,sent2vec,stransformers}Method to find fuzzy matches--minscore MINSCORE Min fuzzy match score. Only matches with a similarityscore of at least 'minscore' will be included--maxmatch MAXMATCH Max number of fuzzy matches kept per source segment--model_name_or_path MODEL_NAME_OR_PATHPath to sent2vec model (when `method` is sent2vec) orsentence-transformers model name when method isstransformers (seehttps://www.sbert.net/docs/pretrained_models.html)--faiss FAISS Path to faiss index. Must be provided when `method` issent2vec or stransformers--threads THREADS Number of threads. Must be 0 or 1 when using`use_cuda`--n_setsim_candidates N_SETSIM_CANDIDATESNumber of fuzzy match candidates extracted by setsim--setsim_function SETSIM_FUNCTIONSimilarity function used by setsimsearch--use_cuda Whether to use GPU for FAISS indexing and sentence-transformers. For this to work properly `threads`should be 0 or 1.-q QUERY_MULTIPLIER, --query_multiplier QUERY_MULTIPLIER(applies only to FAISS) Initially look for`query_multiplier * maxmatch` matches to ensure thatwe find enough hits after filtering. If still notenough matches, search the whole index-v {info,debug}, --logging_level {info,debug}Set the information level of the logger. 'info' showstrivial information about the process. 'debug' alsonotifies you when less matches are found thanrequested during semantic matching
nfr-add-training-features: Adds features to the input. These involve the side of a token (source token or fuzzy
usage: nfr-add-training-features [-h] [-o OUT] [-l] [-v] fin falignGiven a file containing source, fuzzy source and fuzzy target columns, findsthe tokens in fuzzy_src that match with src according to the edit distancemetric. Then the indices of those matches are used together with the wordalignments (GIZA) between fuzzy_src and fuzzy_tgt to mark fuzzy target tokenswith ￨m (match) or ￨nm (no match). This feature indicates whether or not thefuzzy_src token that is aligned with said fuzzy target token has a match inthe original source sentence. The feature is also added to source tokens whena match was found according to the methodology described above. In addition, a"side" feature is added. This indicates which side the token is from, ￨S(source) or ￨T (target). So, in sum, every source and fuzzy target token willhave two features: match/no-match and its side. These features can be filteredin the next processing step, nfr-augment-data.positional arguments:fin Input filefalign Alignment fileoptional arguments:-h, --help show this help message and exit-o OUT, --out OUT Output file. If not given, will use the input file with'.trainfeats' before the suffix-l, --lazy Whether to use lazy processing. Useful for very largefiles-v, --verbose Whether to print intermediate results to stdout
nfr-augment-data: Prepares the dataset to be used in an MT system. Allows you to combine fuzzy matches and choose
usage: nfr-augment-data [-h] --src SRC --tgt TGT --fm FM --outdir OUTDIR--minscore MINSCORE --n_matches N_MATCHES --combine{nbest,max_coverage} [--is_trainset] [--out_ranges][-sf {side,matched} [{side,matched} ...]][-ftf {side,matched} [{side,matched} ...]]Prepares your data for training an MT system. The script creates combinationsof source and (possibly multiple) fuzzy target sentences, based on theinitially created matches (cf. extraxt-fuzzy-matches). The current script canalso filter features that need to be retained in the final files.Corresponding translations are also saved as well as those entries for whichno matches were found.optional arguments:-h, --help show this help message and exit--src SRC Input source file--tgt TGT Input target file--fm FM File containing fuzzy matches for the input source--outdir OUTDIR Output directory--minscore MINSCORE Min. fuzzy match score threshold--n_matches N_MATCHESNumber of fuzzy target to be used in augmented source--combine {nbest,max_coverage}Method of combining fuzzy matches--is_trainset Whether the input file the training set for the MTsystem--out_ranges Whether to save augmented data for different fuzzymatch range categories (considering the best fuzzymatch score)-sf {side,matched} [{side,matched} ...], --src_feats {side,matched} [{side,matched} ...]Features to retain in the source tokens-ftf {side,matched} [{side,matched} ...], --fuzzy_tgt_feats {side,matched} [{side,matched} ...]Features to retain in the fuzzy target tokens
The best configuration (for majoirty of the language pairs tested in Tezcan, Bulté & Vanroy; 2021) consists of the following parameters/properties:
Note: You can follow along with this step-by-step guide by making use of a dummy data set that we provide on the
release page. This dataset and the supplementary models and indices
are purposefully kept small in size. Therefore, their performance is not great. They merely are intended to show you how
to use our code.
Fuzzy matches need to be extracted for the training, test and development sets separately.
Fuzzy matching using sent2vec requires a sent2vec model built on the source side (source language) of the (preprocessed) training set and a FAISS index for this model. Please see sent2vec documentation on how to build a sent2vec model and our paper for the parameters we used in our experiments.
To generate a FAISS index (for the source side of the training data using the sent2vec model ‘sent2vec.model.bin’)
nfr-create-faiss-index -c ./0_preprocessing/bpe_merged/train.tok.truec.bpe.en --model_name_or_path ./sent2vec/sent2vec.train.tok.truec.10dim.bpe.bin -o ./sent2vec/sent2vec.faiss.en
To extract fuzzy matches (for the training set):
nfr-extract-fuzzy-matches --tmsrc ./0_preprocessing/bpe_merged/train.tok.truec.bpe.en --tmtgt ./0_preprocessing/bpe_merged/train.tok.truec.bpe.nl --insrc ./0_preprocessing/bpe_merged/train.tok.truec.bpe.en --method sent2vec --faiss ./sent2vec/sent2vec.faiss.10dim.en --model_name_or_path ./sent2vec/sent2vec.train.tok.truec.10dim.bpe.bin --maxmatch 40 --minscore 0.5 --threads 1
*Note 1: This command generates the ‘fuzzy match file’ (train.tok.truec.bpe.en.matches.mins0.5.maxm40.sent2vec.txt) in the same folder as the original file (--insrc).
**Note 2: Modify --insrc parameter to extract fuzzy matches for the development or test sets separately.
Note 3:* To run the process on GPU, remove --threads parameter and use --use_cuda instead.
Features are added to fuzzy match files for the training, test and development sets separately.
This step requires a word alignment file in Pharaoh format (only for the training set!), where the the source and target token indices are separated by a dash. For instance: 0-0 1-1 2-2 2-3 3-4 4-5
We used GIZA++ to obtain alignments for the training set (tokenized, truecased, byte-pair encoded). Please see the paper for the parameters we used for GIZA++.
To add features to the fuzzy match file of the training set:
nfr-add-training-features ./1_fuzzy_matches/train.tok.truec.bpe.en.matches.mins0.5.maxm40.sent2vec.txt ./word_alignments/train.bpe.alignments -l
*Note 1: This command generates a new ‘fuzzy match file’ with features (train.tok.truec.bpe.en.matches.mins0.5.maxm40.sent2vec.trainfeats.txt) in the same folder as the input file (--insrc).
**Note 2: Change the first positional argument to add features to the fuzzy match files for the development or test set separately.
Note 3: Only the word alignment file obtained on the training set is used (also for adding features to test or dev sets)!
Note 4:* In this step, both “side (source/target)” and “match/nomatch” features are added to the fuzzy matche files. While in the final augmented data we do not use the “match/nomatch” features on the source tokens, this feature is still necessary to apply “max. coverage” during the data augmentation step (Step 3).
To augment training set:
nfr-augment-data --src ./0_preprocessing/bpe_merged/train.tok.truec.bpe.en --tgt ./0_preprocessing/bpe_merged/train.tok.truec.bpe.nl --fm ./2_fuzzy_matches_w_features/train.tok.truec.bpe.en.matches.mins0.5.maxm40.sent2vec.trainfeats.txt --minscore 0.5 --n_matches 2 -sf side -ftf matched --outdir ./3_augment_data/train/ --combine max_coverage --is_trainset
*Note 1: When augmenting the training set we need to use the --is_trainset parameter as the training set is augmented in a different way compared to the test and dev sets. Please see the paper for details on data augmentation.
**Note 2:* This command creates the augmented training set (source/target) in the output directory, which can be used to train the NMT model.
To augment test (or dev) set:
nfr-augment-data --src ./0_preprocessing/bpe_merged/test.tok.truec.bpe.en --tgt ./0_preprocessing/bpe_merged/test.tok.truec.bpe.nl --fm ./2_fuzzy_matches_w_features/test.tok.truec.bpe.en.matches.mins0.5.maxm40.sent2vec.trainfeats.txt --minscore 0.5 --n_matches 2 -sf side -ftf matched --outdir ./3_augment_data/test/ --combine max_coverage
*Note 1: Modify --src, --tgt, --fm and the output directory (--outdir) to augment the dev set.
**Note 2: This command creates the augmented test/dev set (source/target) in the output directory, which can be used to translate using the NMT model trained on augmented training set.
Note 3:* If you want to generate documents containing sentences per fuzzy match range, add the parameter --out_ranges.
In our experiments we used OpenNMT-py (version 1.x) to train NMT models but the augmented data sets can be used to train models with any toolkit provided that it supports source-side features (OpenNMT-py version 2 does not support source-side features at the time of writing this guide).
Some parameters to pay attention to during the “preprocessing” step in OpenNMT:
onmt_preprocess: src-seq_length is increased based on the no. of fuzzy matches used for data augmentation (x2 when a single fuzzy match is used; x3 when 2 fuzzy matches are used etc.)onmt_train: word_vec_size + feat_vec_size = rnn_size. For ex. word_vec_size = 506, feat_vec_size = 6 and rnn_size = 512. If The sum of word_vec_size and feat_vec_size is not equal to rnn_size it gives an error (source).Please see the paper for a list of all the paramater values we used in our experiments.
Please cite our paper(s) when you use this library.
If you perform automatic or manual evaluations or analyse how similar translations influence the MT output:
Tezcan, A., Bulté, B. (2022). Evaluating the Impact of Integrating Similar Translations into Neural Machine Translation. Informatics, 13(1). https://www.mdpi.com/2078-2489/13/1/19
@Article{info13010019,AUTHOR = {Tezcan, Arda and Bulté, Bram},TITLE = {Evaluating the Impact of Integrating Similar Translations into Neural Machine Translation},JOURNAL = {Information},VOLUME = {13},YEAR = {2022},NUMBER = {1},ARTICLE-NUMBER = {19},URL = {https://www.mdpi.com/2078-2489/13/1/19},ISSN = {2078-2489},ABSTRACT = {Previous research has shown that simple methods of augmenting machine translation training data and input sentences with translations of similar sentences (or fuzzy matches), retrieved from a translation memory or bilingual corpus, lead to considerable improvements in translation quality, as assessed by a limited set of automatic evaluation metrics. In this study, we extend this evaluation by calculating a wider range of automated quality metrics that tap into different aspects of translation quality and by performing manual MT error analysis. Moreover, we investigate in more detail how fuzzy matches influence translations and where potential quality improvements could still be made by carrying out a series of quantitative analyses that focus on different characteristics of the retrieved fuzzy matches. The automated evaluation shows that the quality of NFR translations is higher than the NMT baseline in terms of all metrics. However, the manual error analysis did not reveal a difference between the two systems in terms of total number of translation errors; yet, different profiles emerged when considering the types of errors made. Finally, in our analysis of how fuzzy matches influence NFR translations, we identified a number of features that could be used to improve the selection of fuzzy matches for NFR data augmentation.},DOI = {10.3390/info13010019}}
If you use semantic fuzzy matching (sent2vec, sentence-transformers), sub-word segmentation, max. coverage for combining fuzzy matches, source-side features for training NMT models:
Tezcan, A., Bulté, B., & Vanroy, B. (2021). Towards a better integration of fuzzy matches in neural machine
translation through data augmentation. Informatics, 8(1). https://doi.org/10.3390/informatics8010007
@article{tezcan2021integration,AUTHOR = {Tezcan, Arda and Bulté, Bram and Vanroy, Bram},TITLE = {Towards a Better Integration of Fuzzy Matches in Neural Machine Translation through Data Augmentation},JOURNAL = {Informatics},VOLUME = {8},YEAR = {2021},NUMBER = {1},ARTICLE-NUMBER = {7},URL = {https://www.mdpi.com/2227-9709/8/1/7},ISSN = {2227-9709},ABSTRACT = {We identify a number of aspects that can boost the performance of Neural Fuzzy Repair (NFR), an easy-to-implement method to integrate translation memory matches and neural machine translation (NMT). We explore various ways of maximising the added value of retrieved matches within the NFR paradigm for eight language combinations, using Transformer NMT systems. In particular, we test the impact of different fuzzy matching techniques, sub-word-level segmentation methods and alignment-based features on overall translation quality. Furthermore, we propose a fuzzy match combination technique that aims to maximise the coverage of source words. This is supplemented with an analysis of how translation quality is affected by input sentence length and fuzzy match score. The results show that applying a combination of the tested modifications leads to a significant increase in estimated translation quality over all baselines for all language combinations.},DOI = {10.3390/informatics8010007}}
If you use lexical fuzzy matching (editdist, setsim, setsimeditdist):
Bulte, B., & Tezcan, A. (2019). Neural Fuzzy Repair: Integrating Fuzzy Matches into Neural Machine
Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational
Linguistics (pp. 1800–1809). Association for Computational Linguistics. https://www.aclweb.org/anthology/P19-1175
@inproceedings{bulte2019neural,AUTHOR = {Bulte, Bram and Tezcan, Arda},TITLE = {Neural Fuzzy Repair: Integrating Fuzzy Matches into Neural Machine Translation},BOOKTITLE = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},MONTH = jul,YEAR = {2019},ADDRESS = {Florence, Italy},PUBLISHER = {Association for Computational Linguistics},URL = {https://www.aclweb.org/anthology/P19-1175},PAGES = {1800--1809},ABSTRACT = {We present a simple yet powerful data augmentation method for boosting Neural Machine Translation (NMT) performance by leveraging information retrieved from a Translation Memory (TM). We propose and test two methods for augmenting NMT training data with fuzzy TM matches. Tests on the DGT-TM data set for two language pairs show consistent and substantial improvements over a range of baseline systems. The results suggest that this method is promising for any translation environment in which a sizeable TM is available and a certain amount of repetition across translations is to be expected, especially considering its ease of implementation.},DOI = {10.18653/v1/P19-1175},}
After larger refactors and before new releases, always run the following commands in the root of the current directory
to ensure a consistent code style. We use additional plugins to help with that. They can automatically be installed
by using the dev option.
pip install .[dev]
For a consistent coding style, the following command will reformat the files in nfr/.
make style
We use black-style conventions alongside isort.
For quality checking we use flake8. Run the following command and make sure to fix all warnings. Only
publish a new release when no more warnings or errors are present.
make quality