FIDEL
A reversible pre-tokenization scheme for the Ethiopian Abugida script that decomposes characters into consonant and vowel sequences and linearizes them, exposing the Semitic consonantal root structure.
Timeline (last 90 days)
FIDEL (FIdäl DEcomposition and Linearization) is a reversible, abugida-aware pre-tokenization scheme proposed in a paper by rss.
The core of FIDEL is to decompose each abugida character into two parts, a consonant and a vowel order, and then output them in linearized form while guaranteeing reversibility.
FIDEL extends the SPLINTER reduction method from abjad scripts to Ethiopian abugida scripts.
FIDEL was evaluated on three Ethiopian languages, with the corpus, architecture, budget and protocol held fixed in the experiments and tokenization as the only variable.
Experiments found that FIDEL reduction consistently improves news classification performance and also helps NER at larger model capacities.
More aggressive FIDEL reduction schemes cause sentiment analysis performance to drop, while extractive question answering performance remains unchanged.
A possible reason for the sentiment analysis performance drop is that sentiment analysis relies more on subtle lexical cues, which excessive decomposition destroys, whereas classification and NER rely more on roots and morphological structure and thus benefit from consonant exposure.
The FIDEL study covers only three Ethiopian languages, and its generalizability remains to be verified.
Pre-tokenization strategies need to be adjusted according to task type, and future work could explore adaptive reduction strength or task-aware tokenization schemes.
All Facts (9)
FIDEL (FIdäl DEcomposition and Linearization) is a reversible, abugida-aware pre-tokenization scheme proposed in a paper by rss.
50%UnverifiedThe core of FIDEL is to decompose each abugida character into two parts, a consonant and a vowel order, and then output them in linearized form while guaranteeing reversibility.
50%UnverifiedFIDEL extends the SPLINTER reduction method from abjad scripts to Ethiopian abugida scripts.
50%UnverifiedFIDEL was evaluated on three Ethiopian languages, with the corpus, architecture, budget and protocol held fixed in the experiments and tokenization as the only variable.
50%UnverifiedExperiments found that FIDEL reduction consistently improves news classification performance and also helps NER at larger model capacities.
50%UnverifiedMore aggressive FIDEL reduction schemes cause sentiment analysis performance to drop, while extractive question answering performance remains unchanged.
50%UnverifiedA possible reason for the sentiment analysis performance drop is that sentiment analysis relies more on subtle lexical cues, which excessive decomposition destroys, whereas classification and NER rely more on roots and morphological structure and thus benefit from consonant exposure.
50%UnverifiedThe FIDEL study covers only three Ethiopian languages, and its generalizability remains to be verified.
50%UnverifiedPre-tokenization strategies need to be adjusted according to task type, and future work could explore adaptive reduction strength or task-aware tokenization schemes.
50%