Normalization

Adding normalization logic to the parser to map obvious synonyms (such as "at risk" and "At Risk") to the correct enum value can improve fault tolerance and avoid discarding the entire response.

The correct order for the defense process is: first do homoglyph mapping and Unicode normalization, then run filters and classifiers on the normalized text, while retaining the original text for display and logging.

Normalization should be done only once, at the boundary; if different parts of the application each normalize separately, the differences between them become new gaps.

Vectors are normalized when building the index, so that cosine similarity at runtime reduces to a single dot product.

The author proposed a two-step processing strategy: first apply auto contrast or normalization to the grayscale image, then binarize at 128, which effectively removes gray edges. After reprocessing, all 11 rejected files passed, and among the subsequent 300 assets there were zero edge-related rejections.

An embedding model typically consists of three parts: the tokenizer, the encoder weights, and the normalization step, and a change in any one part changes the entire mapping function, producing a new space with completely different semantic coordinates in the same dimensionality.

The cosine similarity between two embedding vectors is meaningful only when they are produced by the same mapping function (the same model); changing the model (a new checkpoint, a new tokenizer, or a change in normalization) produces embeddings with the same shape but a different space.

If features are not normalized, features with larger magnitudes will dominate the evaluation function, rendering the weights optimized by the genetic algorithm meaningless.