tokenize

Apply an ITEM tokenizer to a given dataset.

Functions

adapt_dataset_from_parquet(self, data, path, ...)

Classes

Tokenizer(*args, tokenizer, **kwargs)

class undertale.datasets.scripts.tokenize.Tokenizer(*args, tokenizer: str, **kwargs)

Bases: Dataset