Tokens are the basic unit of measurement for how large language models process text, roughly equivalent to 3/4 of a word in English or 1-2 characters in Chinese