TOON
A serialization format designed specifically for LLM context compression. By declaring fields in batches, it reduces the repetition of field names and thereby significantly lowers token consumption.
Timeline (last 90 days)
In a test with regular tabular data of 10 rows and 4 fields per row, encoded using the cl100k_base tokenizer, JSON was 323 tokens and 963 characters, while YAML was 221 tokens and 590 characters, with YAML ahead by 31.6%; the TOON format was 97 tokens and 235 characters, 70% fewer than JSON and 56.1% fewer than YAML.
For nested, non-tabular data (application configuration, a single object with sub-objects), the size of YAML and TOON is almost identical, because the TOON specification borrows YAML's indentation syntax to handle non-tabular structures; regular arrays of objects are TOON's home turf, where its advantage is obvious.
A practical engineering pattern is: keep the data actually flowing inside the system as JSON, convert only the text handed to the model into YAML or TOON, and then require the model to return in JSON format anything that needs to go back into code.
All Facts (3)
In a test with regular tabular data of 10 rows and 4 fields per row, encoded using the cl100k_base tokenizer, JSON was 323 tokens and 963 characters, while YAML was 221 tokens and 590 characters, with YAML ahead by 31.6%; the TOON format was 97 tokens and 235 characters, 70% fewer than JSON and 56.1% fewer than YAML.
50%UnverifiedFor nested, non-tabular data (application configuration, a single object with sub-objects), the size of YAML and TOON is almost identical, because the TOON specification borrows YAML's indentation syntax to handle non-tabular structures; regular arrays of objects are TOON's home turf, where its advantage is obvious.
50%UnverifiedA practical engineering pattern is: keep the data actually flowing inside the system as JSON, convert only the text handed to the model into YAML or TOON, and then require the model to return in JSON format anything that needs to go back into code.
50%