EmsyAI
A 196.7-million-parameter language model implemented from scratch by an individual developer. It employs modern architectural techniques including GQA, SwiGLU, and RoPE, was trained on consumer-grade GPUs, and serves as a hands-on exploration of engineering practices for industrial-scale large model training.
Timeline (last 90 days)
EmsyAI (V4) is a language model with 196.7 million parameters, trained on a consumer-grade GPU
EmsyAI uses a GQA attention mechanism configured with 16 query heads and 4 KV heads
EmsyAI was trained on 1.96 billion tokens, with a context window of 4096 tokens, a hidden dimension of 1024, an FFN dimension of 2816, and a vocabulary of approximately 16k
EmsyAI was trained for 15,000 steps using FP16 mixed precision, with validation perplexity converging to 5.86
EmsyAI uses a 2.3 million parameter LoRA adapter for instruction fine-tuning on the CodeAlpaca dataset, achieving a score of 0.0% on the HumanEval code benchmark.
When the hidden dimension reaches 1024, attention logits occasionally spike dramatically, becoming a stability risk after scaling up.
In EmsyAI's BPE tokenizer, bytes 0–3 overlap with special control tokens (such as <|endoftext|>), which is a bug that silently degrades performance.
EmsyAI's final weights are exported in GGUF format and can be run in Ollama.
All Facts (8)
EmsyAI (V4) is a language model with 196.7 million parameters, trained on a consumer-grade GPU
50%UnverifiedEmsyAI uses a GQA attention mechanism configured with 16 query heads and 4 KV heads
50%UnverifiedEmsyAI was trained on 1.96 billion tokens, with a context window of 4096 tokens, a hidden dimension of 1024, an FFN dimension of 2816, and a vocabulary of approximately 16k
50%UnverifiedEmsyAI was trained for 15,000 steps using FP16 mixed precision, with validation perplexity converging to 5.86
50%UnverifiedEmsyAI uses a 2.3 million parameter LoRA adapter for instruction fine-tuning on the CodeAlpaca dataset, achieving a score of 0.0% on the HumanEval code benchmark.
50%UnverifiedWhen the hidden dimension reaches 1024, attention logits occasionally spike dramatically, becoming a stability risk after scaling up.
50%UnverifiedIn EmsyAI's BPE tokenizer, bytes 0–3 overlap with special control tokens (such as <|endoftext|>), which is a bug that silently degrades performance.
50%UnverifiedEmsyAI's final weights are exported in GGUF format and can be run in Ollama.
50%