Botok

State-of-the-art tokenizers for Tibetan language.

This is the documentation of our repository botok.

Features

  • Support various dialects.

  • Fully customizable to world list and adjustments rules.

  • Allows adjusting word list and rules with Adjustments component of the Dialect Pack.