tokenizer_config.json

A configuration file in the Hugging Face ecosystem that controls tokenizer behavior, defining how text is split into tokens, including pre-tokenizer and decoder settings.