pretokenize

Apply ITEM pretokenization to a given dataset.

Functions

adapt_dataset_from_parquet(self, data, path, ...)

Classes

Pretokenizer([writer, executor, ...])

class undertale.datasets.scripts.pretokenize.Pretokenizer(writer: str = 'parquet', executor: str = 'local', logging_directory: str | None = None)

Bases: Dataset