Adding a User Defined Fst to OpenFst

In the post I will show how to add user defined Fst type to OpenFst. openfst.org doesn't go into detail on the process to add a new Fst type. The Fst type is memory mappable immutable transducers and this should allow for the same Fst instance to be shared across processes that load the same Fst.
All the code for the mmap-fst is avaliable in this gist. The implementation of the mmap-fst is contained in the file mmap-fst.h, the wrapper class around the mmap functions is taken from the mozc project. The wrapper supports both *nix and Windows oses, although only the former support the shared objects described in the article. I changed the length type in the Mmap class to allow larger than 2GB files. The MmapFst itself is basically a cut n' paste job on the ConstFst, the main difference is the MmapFst uses the mmap wrapper instead of  array to access the state and arc information.

Given the Fst code we want to make the OpenFst library and existing OpenFst command line tools aware of of the MmapFst type. To achieve this we need the c++ code and the Makefile to construct a shared object. The registration code that is shown below and is very simple


The makefile below will compile the code to a shared object.

The final step is to make sure the shared object is on the LD_LIBRARY_PATH. We can then convert fsts to the mmap type and use it in all the command line tools. For example we can convert an Fst to MmapFst  type by:

fstconvert --fst_typo=mmap < some.fst > mmap.fst

and view fst information by:

fstinfo mmap.fst

fst type                                          mmap
arc type                                         standard
input symbol table                        none
.....
....

This entry was posted by Edobashira. Bookmark the permalink.

8 thoughts on “Adding a User Defined Fst to OpenFst”

  1. I know there has been some time since your last post to the openFST topic, but I was so glad, to find someone who actually provide some examples to follow and learn the structure of the fst-thingy.

    Can you recommend a way of learning the way of openFST? I need it more basic than it is provided at the openFST-website. There isn't much of explanation for me.

    Thanks in advance and greetings,
    Dree

  2. Thanks for the comment. For learning the code using the Windows port under Visual Studio is easiest as the the intellisense really helps out navigating the code and with the complicated parameters.
    Then working through the Mohri papers and the OpenFst SLT tutorial and trying to recreate the toy examples and projects.
    I have a new site www.lvcsr.com with an old FST tutorial in the publications sections and a few very short posts.
    If you have any OpenFst questions I normally answer posts on the OpenFst forum or you can ask me directly by email paul at edobashira.com.

  3. Thank YOU very much for answering my request on a topic thats nearly a year back. I will try to go the ways you pointed out. If I got questions, I now know to whom I can entrust them!
    Have a nice sunday,
    Dree

Leave a Reply

Note: only a member of this blog may post a comment.