Functionalizer

A lossless pre-tokenization framework that decomposes orthographic word variants into combinations of opcodes and operands to avoid vocabulary bloat. It supports reversible transformations including case changes, diacritic modifications, and character repetitions, improving vocabulary utilization without discarding any original text information.