superword tokenization

A class of tokenization methods that relax conventional word-boundary constraints, allowing multi-word expressions spanning spaces to be merged so that the tokenizer can directly produce tokens containing spaces. Most implementations still employ greedy frequency-based merging.