FST Lookup

Implements lookup for Foma finite state transducers.

Supports Python 3.5 and up.

Install

pip install fst-lookup

Usage

Import the library, and load an FST from a file:

Hint: Test this module by downloading the eat FST!

>>> from fst_lookup import FST
>>> fst = FST.from_file('eat.fomabin')

Assumed format of the FSTs

fst_lookup assumes that the lower label corresponds to the surface
form, while the upper label corresponds to the lemma, and linguistic
tags and features: e.g., your LEXC will look something like
this—note what is on each side of the colon (:):

Multichar_Symbols +N +Sg +Pl
Lexicon Root
    cow+N+Sg:cow #;
    cow+N+Pl:cows #;
    goose+N+Sg:goose #;
    goose+N+Pl:geese #;
    sheep+N+Sg:sheep #;
    sheep+N+Pl:sheep #;

If your FST has labels on the opposite sides—e.g., the upper label
corresponds to the surface form and the upper label corresponds to
the lemma and linguistic tags—then instantiate the FST by providing
the labels="invert" keyword argument:

fst = FST.from_file('eat-inverted.fomabin', labels="invert")

Hint: FSTs originating from the HFST suite are often inverted, so
try to loading the FST inverted first if .generate() or .analyze()
aren’t working correctly!

Analyze a word form

To analyze a form (take a word form, and get its linguistic analyzes)
call the analyze() function:

def analyze(self, surface_form: str) -> Iterator[Analysis]

This will yield all possible linguistic analyses produced by the FST.

An analysis is a tuple of strings. The strings are either linguistic
tags, or the lemma (base form of the word).

FST.analyze() is a generator, so you must call list() to get a list.

>>> list(sorted(fst.analyze('eats')))
[('eat', '+N', '+Mass'),
 ('eat', '+V', '+3P', '+Sg')]

Generate a word form

To generate a form (take a linguistic analysis, and get its concrete
word forms), call the generate() function:

def generate(self, analysis: str) -> Iterator[str]

FST.generate() is a Python generator, so you must call list() to get
a list.

>>> list(fst.generate('eat+V+Past')))
['ate']

Contributing

If you plan to contribute code, it is recommended you use Poetry.
Fork and clone this repository, then install development dependencies
by typing:

poetry install

Then, do all your development within a virtual environment, managed by
Poetry:

poetry shell

Type-checking

This project uses mypy to check static types. To invoke it on this
package, type the following:

mypy -p fst_lookup

Running tests

To run this project’s tests, we use py.test:

poetry run pytest

C Extension

Building the C extension is handled in build.py

To disable building the C extension, add the following line to .env:

export FST_LOOKUP_BUILD_EXT=False

(by default, this is True).

To enable debugging flags while working on the C extension, add the
following line to .env:

export FST_LOOKUP_DEBUG=TRUE

(by default, this is False).

Fixtures

If you are creating or modifying existing test fixtures (i.e., mostly
pre-built FSTs used for testing), you will need the following
dependencies:

GNU make
Foma

Fixtures are stored in tests/data/. Here, you will use make to
compile all pre-built FSTs from source:

make

License

Licensed under the MIT license.