scripts

Dataset parsing scripts and utilities.

Modules

pretokenize	Apply ITEM pretokenization to a given dataset.
shell	Load a given dataset and start a shell.
tokenize	Apply an ITEM tokenizer to a given dataset.