FIDEL: An Abugida-Aware Pre-Tokenization Scheme for Ethiopian Languages

FIDEL: reversible abugida-aware pre-tokenization boosts classification/NER but hurts sentiment on Ethiopian languages.
The paper proposes FIDEL, a reversible pre-tokenization scheme for Ethiopian abugida scripts that decomposes characters into consonants and vowel orders and linearizes them, extending the SPLINTER method. Controlled experiments on three Ethiopian languages show FIDEL reduction consistently improves news classification and helps NER with larger models, but more aggressive reduction degrades sentiment analysis while extractive QA remains unchanged. This reveals task-dependence of tokenization and the value of linguistic priors for low-resource, non-Latin-script languages.
Introduction
Subword tokenizers (such as BPE and WordPiece) assume that meaningful linguistic units can be recovered by learning boundaries, but this logic applies better to concatenative morphology and fits poorly with Semitic root-and-pattern systems. This is especially true for the abugida scripts used by Ethiopian languages: each character encodes a consonant-vowel unit, while the consonantal structure is hidden. A recent paper from rss proposes FIDEL (FIdäl DEcomposition and Linearization), a reversible, abugida-aware pre-tokenization scheme that attempts to inject linguistic priors at the tokenization level.

Background: When Tokenizers Meet Abugida Scripts
Mainstream subword tokenizers perform well on Latin-script languages because they can learn affix boundaries through statistical co-occurrence. However, the abugida scripts of Ethiopian languages fuse consonants and vowels into a single glyph—for example, one character simultaneously represents a syllable like /kə/. This makes it difficult for tokenizers to expose the consonantal root structure that is crucial in Semitic languages. The paper points out that this script characteristic invalidates the assumption of learning boundaries during tokenization, necessitating dedicated linguistically-aware preprocessing.
Abugida is a writing system between syllabic scripts and consonantal alphabets. Each basic character represents a consonant plus an inherent vowel (usually /ə/ or /a/), with other vowels indicated by diacritics or character variants. The Ge'ez script of Ethiopia and its derived scripts are typical representatives of abugidas. Unlike Latin letters, abugida scripts do not write vowels separately; instead, vowel information is fused into the consonantal glyph, so a single character carries both consonant and vowel information. This characteristic means that character sequences cannot directly reflect the linear structure of consonantal roots, posing a challenge for subword tokenizers that rely on character boundaries.
The FIDEL Approach: Decomposition and Linearization
The core of FIDEL is to decompose each abugida character into two parts—a consonant and a vowel order—and then output them in a linearized form. This scheme is reversible, meaning the original text can be reconstructed from the decomposed sequence. The research extends the SPLINTER reduction method, originally used for abjads (consonantal alphabets), to Ethiopian abugida scripts. SPLINTER was previously used for abjad scripts such as Arabic, reducing vocabulary sparsity by mapping consonant patterns. FIDEL was evaluated on three Ethiopian languages with strict experimental controls: corpus, architecture, budget, and protocol were fixed, with tokenization as the only variable.
SPLINTER is a pre-tokenization reduction method for consonantal alphabets (abjads), originally used for languages such as Arabic. Its core idea is to map vocabulary to consonantal skeletons, reducing vocabulary sparsity by merging word forms that share the same consonantal pattern. For example, in Arabic, many word forms share the same consonantal root but differ in vowel patterns; SPLINTER helps models capture morphological regularities by exposing consonantal structure. FIDEL extends this idea to abugida scripts but requires additional handling of vowel order information, because abugida characters themselves encode vowels. During decomposition, consonants and vowel orders must be separated and linearized while ensuring reversibility.
Experimental Design and Evaluation Tasks
The paper reports intrinsic tokenization statistics for BPE and UnigramLM at different vocabulary sizes and tests multiple downstream NLP tasks, including transfer scenarios. Specific tasks cover news classification, named entity recognition (NER), sentiment analysis, and extractive question answering. This controlled-variable experimental design allows the tokenization effect to be cleanly isolated, avoiding interference from other factors.
UnigramLM is a tokenization algorithm based on a unigram language model. Unlike BPE's greedy merging strategy, it starts from a large vocabulary and progressively prunes through a probabilistic model, selecting the subword set that maximizes corpus likelihood. UnigramLM typically produces tokenizations that better conform to linguistic statistical patterns and supports sampling from multiple tokenization candidates. In FIDEL's experiments, using both BPE and UnigramLM as base tokenizers tests whether the FIDEL pre-tokenization scheme is independent of the specific tokenization algorithm, thereby enhancing the generalizability of the conclusions.
Results: Classification and NER Benefit, Sentiment Analysis Declines
Experiments found that FIDEL reduction consistently improves news classification performance and also helps NER with larger model capacities. However, more aggressive reduction schemes lead to decreased sentiment analysis performance, while extractive question answering performance remains unchanged. The paper does not shy away from negative results, and this honesty is commendable. A possible reason is that sentiment analysis relies more on fine-grained lexical cues, and excessive decomposition destroys these signals; whereas classification and NER rely more on roots and morphological structure, thus benefiting from consonant exposure.
Implications and Limitations
The value of FIDEL lies in demonstrating the effectiveness of linguistic priors at the tokenization stage, especially for morphologically complex languages with special scripts. However, the scheme is not a panacea: the degradation in sentiment analysis suggests that pre-tokenization strategies need to be adjusted according to task type. Furthermore, the study covers only three Ethiopian languages, and generalizability remains to be verified. Future work could explore adaptive reduction strength or task-aware tokenization schemes.
Conclusion
For low-resource languages with non-Latin scripts, FIDEL provides a reversible, controllable path to tokenization optimization. It reminds us that tokenization is not a neutral preprocessing step but a design choice embedded with linguistic assumptions.
One-line Summary
FIDEL proposes a reversible abugida-aware pre-tokenization scheme that improves classification and NER but degrades sentiment analysis on Ethiopian languages.
Paragraph Summary
The paper proposes FIDEL, a reversible pre-tokenization scheme for Ethiopian abugida scripts that extends the SPLINTER method by decomposing characters into consonants and vowel orders and linearizing them. Controlled-variable experiments on three Ethiopian languages show that FIDEL reduction consistently improves news classification performance and also helps NER with larger model capacities, but more aggressive reduction leads to decreased sentiment analysis performance, while extractive question answering remains unchanged. This reveals the task-dependence of tokenization strategies and demonstrates the value of linguistic priors for low-resource, non-Latin-script languages.
Related articles

Anthropic's IPO Eve: Five Risks Behind the Trillion-Dollar Valuation and the Truth About Its Governance
Breaking down Anthropic's IPO prospectus: $184M daily revenue at a trillion-dollar valuation, the truth behind $104B in paper losses, a five-class share governance structure, unusual disclosures in 94 pages of risk factors, and how the lock-up schedule may affect retail investors.

FBI Warns: Hackers Weaponize AI Agent ARTEX to Attack 200+ Organizations
FBI warns that AI penetration framework ARTEX is being used to attack 200+ organizations, with CrowdStrike tracking the VaporPanda group. We break down ARTEX's agent orchestration architecture, the real attack chain, loss data, and the FBI's five defense recommendations.

FBI Warns: Free AI Hacking Tool Artex Spirals Out of Control, Breaches Cloud Systems in Three Hours
FBI warns that the free open-source AI penetration tool Artex is being abused by criminals and state-backed hackers to breach cloud systems in as little as three hours. This article analyzes real attack cases, overlooked side-door risks, and defense measures enterprises and individuals can take immediately.