IBM Granite 4.1 Open-Source Models: Hands-On SVG Generation Test Across 21 Quantized Versions

IBM Granite 4.1: An Apache 2.0 Open-Source Model Family

IBM recently released the Granite 4.1 series of large language models, available in three sizes — 3B, 8B, and 30B — all under the Apache 2.0 open-source license. Developers can freely use them in commercial projects, which makes them quite attractive in today's open-source model ecosystem.

Apache 2.0 is one of the most permissive licenses in the open-source software world, allowing users to freely use, modify, and distribute software — including for commercial purposes — without requiring derivative works to be open-sourced. By contrast, Meta's Llama series, often called "open-source," actually uses a custom license with additional restrictions for companies with over 700 million monthly active users; some Mistral models have also used licenses restricting commercial use. In today's LLM landscape, high-quality models truly licensed under Apache 2.0 are rare, giving Granite 4.1 a unique legal compliance advantage in enterprise scenarios — developers don't need to worry about license traps.

Granite team member Yousaf Shah provided a detailed overview of the series' training process on the Hugging Face blog. As a key part of IBM's AI foundation model strategy, Granite 4.1 continues the technical direction of previous versions while further optimizing performance and usability.

Unsloth Provides 21 GGUF Quantized Versions

Shortly after Granite 4.1's release, the well-known model optimization team Unsloth quickly released a collection of GGUF-format quantized versions of the 3B model (unsloth/granite-4.1-3b-GGUF), offering 21 different quantized model files ranging in size from 1.2GB to 6.34GB, totaling approximately 51.3GB.

GGUF (GPT-Generated Unified Format) is a model storage format designed by Georgi Gerganov, creator of the llama.cpp project, specifically optimized for CPU and hybrid CPU/GPU inference. It's one of the most popular formats for running LLMs locally, widely used in inference frameworks like llama.cpp and Ollama. Quantization is the technique of compressing model weights from high-precision floating-point numbers (like FP16, where each parameter occupies 2 bytes) to lower-precision representations (like 4-bit, where each parameter occupies only 0.5 bytes). Common quantization levels include Q2_K, Q4_K_M, Q5_K_M, Q8_0, etc., with higher numbers indicating higher precision. Different quantization levels make different trade-offs between model size and inference quality: larger quantized files preserve more of the model's original capabilities, while smaller files sacrifice some precision in exchange for lower hardware requirements.

The Unsloth team is renowned for their efficient model fine-tuning and quantization tools. Their 21 versions span the full quantization spectrum from extreme compression to near-lossless, allowing developers to choose flexibly based on their hardware constraints.

Pelican Riding a Bicycle: A Classic SVG Generation Test

When prominent developer Simon Willison saw these 21 quantized versions, he decided to run an experiment he'd long wanted to try: test different quantized versions of the same model with the same prompt and observe the differences in output quality.

Simon Willison is the co-creator of the Django web framework and the author of Datasette, a data exploration tool. In recent years, he has become one of the most active independent developers and commentators in the AI tools space. His blog (simonwillison.net) is widely considered one of the best sources for tracking LLM technology developments. His LLM command-line tool makes it easy to call various local and cloud-based models, which is the technical foundation that enabled him to quickly batch-test all 21 quantized versions.

His chosen prompt was the classic "Generate an SVG of a pelican riding a bicycle." SVG (Scalable Vector Graphics) is an XML-based vector graphics description language that defines images using text code for points, lines, curves, and shapes. Having LLMs generate SVG has become a popular testing method because it simultaneously challenges multiple capabilities: the model needs spatial reasoning to understand geometric forms and relative positions of objects, code generation ability to output syntactically correct SVG markup, and world knowledge to understand the visual characteristics of both pelicans and bicycles. This task originally spread organically among AI researchers on Twitter/X and gradually evolved into an informal community-consensus benchmark. Large-parameter models like Claude and GPT-4 can typically generate recognizable images, while smaller models often fall short.

Test Results: Different Quantized Versions Performed Almost Equally Poorly

Simon admitted that the results were far more boring than he expected. There was no obvious quality gradient between different quantized versions — they all performed quite poorly.

From the comparison images he published, SVG images generated by six different model sizes ranging from 1.67GB to 1.2GB were almost all jumbles of abstract geometric shapes. Interestingly, the smallest model actually produced the most bicycle-like pattern, while the largest model only produced a shape vaguely resembling a pelican.

The results themselves are quite illuminating:

1. A 3B parameter model is simply not up to the task of SVG generation, regardless of quantization precision — the ceiling of its fundamental capability is just too low
2. Quantization impacts different tasks unevenly — for tasks the model is already bad at, differences in quantization level are masked by the fundamental lack of capability
3. The spatial reasoning and code generation abilities required for SVG generation may need 8B or even 30B parameter-scale models to manifest effectively

A large language model's parameter count (where B in 3B, 8B, 30B stands for billion) directly determines the model's "capacity" — the complexity of knowledge and reasoning patterns it can encode and express. Research shows that many capabilities exhibit so-called "emergence": when parameter count is below a certain threshold, the model is almost completely unable to perform specific tasks; once it exceeds the threshold, capabilities suddenly appear. The emergence threshold for tasks like SVG generation that require precise spatial reasoning may be just above 3B parameters. This explains why quantization-level differences were almost invisible in this experiment — all versions were below the capability threshold, and the minor precision loss from quantization was negligible compared to the fundamental insufficiency of the base capabilities.

Practical Takeaways for Developers

While this small experiment might seem like just a fun test, it reveals several points worth noting when selecting models:

Base Model Size Matters More Than Quantization Level

In practical applications, choosing the right base model size is more critical than agonizing over quantization levels. If the task itself exceeds the model's capabilities, no amount of quantization precision will help. First confirm that the model's capabilities cover your requirements, then consider using quantization to reduce deployment costs.

The Collaboration Efficiency of the Open-Source Ecosystem Is Impressive

From IBM's model release to Unsloth providing quantized versions to community members running various tests, the entire process was completed within days. The Apache 2.0 license lowers the barrier to participation and accelerates this ecosystem cycle. For developers looking to deploy LLMs locally, this means usable quantized versions become available shortly after new models are released.

Choosing the Right Evaluation Benchmark Is Equally Important

Simon mentioned he plans to redo this experiment in the future with models that are better at drawing pelicans. This also reminds us that evaluating model capabilities requires choosing benchmark tasks with matching difficulty — tasks that are too easy can't differentiate between versions, while tasks that are too hard make all versions perform poorly, equally failing to yield meaningful conclusions. In statistics, this is known as the "ceiling effect" and "floor effect": when test difficulty severely mismatches the subject's ability, results cluster at one end of the scale and lose discriminative power. Ideal benchmark tasks should distribute the tested model's performance across a meaningful range, effectively revealing true differences between versions.

Summary

The release of Granite 4.1 adds a competitive option to the open-source LLM market, with the Apache 2.0 license giving it a clear advantage in commercial application scenarios. Meanwhile, this pelican SVG experiment serves as a lighthearted reminder that in AI model evaluation and selection, understanding the match between task requirements and model capabilities is often more important than chasing minute differences in quantization parameters.

Key Takeaways

• IBM released the Granite 4.1 open-source model series (3B/8B/30B) under the Apache 2.0 license
• Unsloth provided 21 GGUF quantized versions for the 3B model, ranging from 1.2GB to 6.34GB
• Simon Willison tested all 21 quantized versions with a "pelican riding a bicycle" SVG prompt, finding minimal quality differences and universally poor performance
• The experiment shows that the 3B parameter model has a capability ceiling for SVG generation tasks, with quantization-level impact masked by fundamental capability limitations
• Choosing the right base model size matters more than obsessing over quantization precision; evaluation requires appropriately matched benchmark tasks