Indian Team Releases ZY-S1 Preview: A Hands-On Study of Fine-Tuning the 1.18B HRM-Text Model

Indian AI team ZYR AI recently released an experimental model, ZY-S1 Preview, on Hugging Face — a supervised fine-tuning attempt on the 1.18B-parameter HRM-Text model. Though small in scale, it offers a worthwhile case study for developers interested in the capabilities of small-parameter models.

Training Configuration: Starting from HRM-Text-1B

The technical approach behind ZY-S1 is straightforward: it takes Sapient AI's HRM-Text-1B (a PrefixLM architecture) as its base, performs 1,000 steps of supervised fine-tuning (SFT), followed by 120 steps of correction training. The training data comes from a filtered version of the MnemicAI/Ling-Coder-SFT-English-Clean dataset.

The model is released under the Apache-2.0 license with 1.183B parameters. From an engineering standpoint, this is a lightweight experiment rather than a production-grade model aiming for comprehensive capability coverage. The PrefixLM architecture has its own characteristics for text generation tasks, and ZY-S1's choice to perform SFT on top of it leans more toward validating the effectiveness of the fine-tuning pipeline.

PrefixLM (Prefix Language Model) is an architecture that sits between autoencoding models (like BERT) and autoregressive models (like GPT). Its core feature: the input sequence is divided into a "prefix" and a "suffix." Tokens within the prefix can attend to each other (similar to bidirectional attention), while tokens in the suffix can only see themselves and preceding tokens (similar to causal attention). This property makes it natural for "given context, generate continuation" tasks, commonly used in text generation, translation, and summarization. In implementation, it typically relies on token_type_ids to distinguish prefix from suffix and allows frameworks to dynamically construct attention masks. Compared to standard decoder-only models, PrefixLM requires extra handling of this input structure during fine-tuning, placing higher demands on the user's engineering details.

Early Evaluation: Improved Language Understanding, Math Reasoning Is the Weak Spot

The author published small-sample evaluation results on 25 questions, comparing the base model with ZY-S1:

• DROP (reading comprehension): EM unchanged from 4→4, F1 improved slightly from 18.24→19.76
• ARC-Challenge: from 76→80, a 4-percentage-point gain
• MMLU: from 48→52, also a 4-point improvement
• MATH-500: from 4→0, a decline rather than an improvement

These numbers convey several signals. The improvements on ARC-Challenge and MMLU suggest that fine-tuning provides some benefit for commonsense reasoning and multi-task understanding. But the drop to zero on MATH-500 is worth noting — the fine-tuning data is dominated by code and general English SFT content, so the math degradation may be related to data distribution shift. Of course, with only 25 questions, the sample size is very small, and these figures are better treated as directional references rather than strict conclusions.

DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark that requires a model to read an English passage and answer questions involving discrete reasoning (such as addition, subtraction, sorting, counting, and comparison). Unlike conventional QA tasks, DROP emphasizes multi-step logical operations on numbers and entities in the text rather than mere information extraction. Evaluation metrics include exact match (EM) and F1: EM requires the answer to be exactly identical, while F1 measures word-level overlap. It is often used in early small-model evaluations to observe subtle changes in reasoning ability, but under small-sample conditions it is easily affected by noise. ARC-Challenge is the difficult subset of the AI2 Reasoning Challenge, with questions mostly drawn from elementary school science that require combining commonsense with multi-step reasoning — a common benchmark for measuring model knowledge transfer and logical ability. MMLU covers multiple-choice questions across 57 subjects and is mainly used to assess a model's overall performance across diverse domains.

How to Run: Watch Out for trust_remote_code and token_type_ids

There is one key detail when using ZY-S1: loading the model requires setting trust_remote_code=True, and you must pass in token_type_ids — 1 for prompt tokens and 0 for generated tokens. This is not surprising for a PrefixLM architecture, but it's easy to overlook for developers accustomed to standard autoregressive models.

The author says full evaluations will be released soon and welcomes community feedback.

trust_remote_code=True is a security mechanism in the Hugging Face Transformers library. By default, from_pretrained only loads standard model classes built into the library; when a model author provides custom modeling code in the repository (such as a special architecture or a modified attention implementation), users must explicitly pass this parameter so the program will download and execute the .py files in the repository. This design is meant to remind users that remote code has the same privileges as a local Python script, and the source should be verified as trustworthy before loading. ZY-S1 requires enabling this parameter, indicating that its inference logic is not fully covered by the standard Transformer implementation and depends on the custom wrapper provided by the author.

What's the Value of Small-Model Fine-Tuning?

The significance of ZY-S1 Preview is not about topping leaderboards. At 1.18B, it cannot compete with larger models on complex reasoning tasks. But it demonstrates several points worth noting: attempts by a domestic Indian team at fine-tuning open-source models, the practical fine-tuning effects of the PrefixLM architecture, and the payoff boundaries of doing SFT on a small dataset.

For developers who want to reproduce it or continue experimenting on this basis, the Apache-2.0 license and publicly available training configuration lower the barrier. And the degradation on MATH-500 also reminds us: the selection and quality of fine-tuning data may be more critical than the number of training steps.

Once the full evaluation results are out, the true capability boundaries of ZY-S1 will become clearer.

Supervised fine-tuning (SFT) refers to continuing training a pretrained model with labeled "input-output" pairs so that it adapts to a specific task format or style. The benefits of SFT are highly dependent on data distribution: if the fine-tuning data is dominated by code and general English instructions, the model's performance in those areas will improve, but abilities unrelated to the fine-tuning objective (such as mathematical reasoning) may decline due to "catastrophic forgetting." In ZY-S1, MATH-500 dropping from 4 to 0 is a typical manifestation of this distribution shift. Catastrophic forgetting refers to a neural network rapidly losing previously learned knowledge when learning a new task, and it is especially pronounced in small models — because parameter capacity is limited and old and new knowledge compete fiercely. Mitigation strategies include mixing in original pretraining data, using parameter-efficient methods such as low-rank adaptation (LoRA), or controlling the learning rate and number of training steps.