[KongchangAI]
ConceptFIdäl DEcomposition and Linearization

FIDEL

A reversible pre-tokenization scheme for the Ethiopian Abugida script that decomposes characters into consonant and vowel sequences and linearizes them, exposing the Semitic consonantal root structure.

Timeline (last 90 days)

Nov 17

FIDEL (FIdäl DEcomposition and Linearization) is a reversible, abugida-aware pre-tokenization scheme proposed in a paper by rss.

Unverified50%
Nov 17

The core of FIDEL is to decompose each abugida character into two parts, a consonant and a vowel order, and then output them in linearized form while guaranteeing reversibility.

Unverified50%
Nov 17

FIDEL extends the SPLINTER reduction method from abjad scripts to Ethiopian abugida scripts.

Unverified50%
Nov 17

FIDEL was evaluated on three Ethiopian languages, with the corpus, architecture, budget and protocol held fixed in the experiments and tokenization as the only variable.

Unverified50%
Nov 17

Experiments found that FIDEL reduction consistently improves news classification performance and also helps NER at larger model capacities.

Unverified50%
Nov 17

More aggressive FIDEL reduction schemes cause sentiment analysis performance to drop, while extractive question answering performance remains unchanged.

Unverified50%
Nov 17

A possible reason for the sentiment analysis performance drop is that sentiment analysis relies more on subtle lexical cues, which excessive decomposition destroys, whereas classification and NER rely more on roots and morphological structure and thus benefit from consonant exposure.

Unverified50%
Nov 17

The FIDEL study covers only three Ethiopian languages, and its generalizability remains to be verified.

Unverified50%
Nov 17

Pre-tokenization strategies need to be adjusted according to task type, and future work could explore adaptive reduction strength or task-aware tokenization schemes.

Unverified50%

All Facts (9)

Unverified

FIDEL (FIdäl DEcomposition and Linearization) is a reversible, abugida-aware pre-tokenization scheme proposed in a paper by rss.

50%
Unverified

The core of FIDEL is to decompose each abugida character into two parts, a consonant and a vowel order, and then output them in linearized form while guaranteeing reversibility.

50%
Unverified

FIDEL extends the SPLINTER reduction method from abjad scripts to Ethiopian abugida scripts.

50%
Unverified

FIDEL was evaluated on three Ethiopian languages, with the corpus, architecture, budget and protocol held fixed in the experiments and tokenization as the only variable.

50%
Unverified

Experiments found that FIDEL reduction consistently improves news classification performance and also helps NER at larger model capacities.

50%
Unverified

More aggressive FIDEL reduction schemes cause sentiment analysis performance to drop, while extractive question answering performance remains unchanged.

50%
Unverified

A possible reason for the sentiment analysis performance drop is that sentiment analysis relies more on subtle lexical cues, which excessive decomposition destroys, whereas classification and NER rely more on roots and morphological structure and thus benefit from consonant exposure.

50%
Unverified

The FIDEL study covers only three Ethiopian languages, and its generalizability remains to be verified.

50%
Unverified

Pre-tokenization strategies need to be adjusted according to task type, and future work could explore adaptive reduction strength or task-aware tokenization schemes.

50%

Source Articles