Darwin-27B-ZTC-v5 Tops Decision Benchmark with Record 0.81 Accuracy

VIDRAFT's New Model Sets a Record on the Structured Decision Benchmark

Darwin-27B-ZTC-v5, the latest model from South Korean Pre-AGI AI startup VIDRAFT (비드래프트), has reclaimed the top spot on the official Hugging Face evaluation benchmark LocalLLaMA/typed-decisions with an accuracy score of 0.81. This is the highest score ever recorded on the leaderboard, and the result holds strong reference value for engineers building decision automation and AI agent systems.

The previous generation of Darwin-27B-ZTC-v5, Darwin-27B-ZTC, had already claimed the top spot with a score of 0.74. Version v5 achieves a +0.07 accuracy improvement on the same leaderboard. In this highly competitive benchmark space with tightly clustered scores, that gain is no small improvement.

What the Typed Decisions Benchmark Is

To appreciate what Darwin-27B-ZTC-v5 has accomplished, it helps to first understand the benchmark it was evaluated on. typed-decisions is a structured, probabilistic classification benchmark specifically designed to measure a model's ability to make calibrated decisions across multi-dimensional classification tasks — tasks drawn directly from real-world enterprise workflow scenarios.

The dataset's core specifications are as follows:

• License: Apache-2.0
• Size: 1K–10K samples
• Modality: Tabular + text
• Task type: Text classification (structured decisions)
• Language: English
• Subsets: agent_trace_observability, customer_service, invoice_processing, security_incidents, each containing 400 rows split into training (300) and test (100)

Each sample contains rich multi-label output fields: outcome, action, needs_review, risk, and urgency. Each label comes with a predicted label, confidence score, and a full probability distribution. This makes Typed Decisions a calibration-aware benchmark, not just an accuracy benchmark.

Calibration is a core concept in evaluating the quality of probabilistic predictions. Even if a model has high classification accuracy, if only 60% of its "90% confidence" predictions are actually correct, then the model is "overconfident" and its probability outputs cannot be trusted. Calibration-aware benchmarks are designed precisely to measure this kind of deviation: they check not only whether the model's highest-probability label is correct, but also whether the confidence it claims matches its actual accuracy rate. Common metrics include Expected Calibration Error (ECE) and the Brier score. In enterprise decision scenarios, calibration quality is often more important than raw accuracy — for example, in invoice routing or security incident classification, the system needs to decide based on confidence whether to "process automatically" or "escalate to a human," and miscalibrated probability outputs directly lead to incorrect automation decisions.

How the Model Is Evaluated

The Typed Decisions benchmark is designed to test whether a model can simultaneously classify multiple decision dimensions from structured workflow state — essentially simulating the judgment an intelligent agent needs when observing a running process and deciding how to respond.

Each evaluation sample contains three parts:

1. Workflow state description: A structured text passage describing what the agent or system is doing
2. Structured questions: A set of questions the model must answer, such as "What is the risk level?" or "Does this trace require human review?"
3. Gold labels: The standard answers, along with reasoning and factor explanations

Crucially, the model must not only output labels but also provide a probability distribution across label categories. For example, the needs_review__probability_true field captures the model's confidence about whether human intervention is needed — a far more granular signal than a simple binary yes/no.

This design simulates a System-1 workflow evaluation: rapid, structured judgment under real enterprise conditions (data erasure queues, security incident classification, invoice routing, customer service escalations, and agent trace observability).

System-1 workflow evaluation borrows the concept of "fast thinking" from cognitive psychology (originating from Daniel Kahneman's dual-system theory). System-1 refers to fast, intuitive, automatic judgment, in contrast to System-2, which is slow, deliberative thinking requiring reasoning. In enterprise AI agent scenarios, System-1-style evaluation means the model must make immediate, reliable classification judgments about structured workflow state within limited context and time — rather than performing multi-turn chain-of-thought reasoning. These tasks are closer to real needs in production environments: agent systems need to decide within milliseconds to seconds whether to escalate a ticket, flag a risk, or trigger human review. Therefore, Typed Decisions evaluates not the model's ability to "deliberate carefully," but the quality of its rapid judgment in structured, high-throughput decision flows.

Leaderboard Data and the Competitive Landscape

According to the public LocalLLaMA/typed-decisions leaderboard data:

Rank	Model	Accuracy Score
1	FINAL-Bench/Darwin-27B-ZTC-v5	0.81
2	FINAL-Bench/Darwin-27B-ZTC	0.74
2	empiriolabsai/aplomb-1	0.74
4	ProKope-AI/ProKope-421M-8K	0.72
5	ProKope-AI/ProKope-421M	0.71
6	adaptive-classifier/typed-decisions-minilm-l6-specialist	0.61

A few observations worth noting:

Darwin-27B-ZTC-v5 leads its closest competitor by a margin of +0.07. Given that two models in the second tier are tied at 0.74, this lead represents a significant gap in benchmark testing.

The benchmark is filtered to include base models only, excluding models that individually declare task-specific supervised fine-tuning. This means the leaderboard compares the models' native capabilities rather than performance after specialized tuning.

Competing models range in parameter size from 421 million to several billion. Darwin-27B-ZTC-v5 achieves its results with 27 billion parameters, which is particularly noteworthy in the context of parameter efficiency.

Base models refer to models that have undergone only large-scale pretraining, without supervised fine-tuning (SFT) for specific downstream tasks. The leaderboard's inclusion of only base models means it evaluates a model's "native" generalization and decision-making abilities, rather than results "gamed" through task-specific data tuning. This filtering mechanism is significant in benchmark evaluation: models that have undergone specialized fine-tuning often show large score gains on specific tasks, but those gains are difficult to transfer to new scenarios. By contrast, base model performance better reflects the true level of a model's architecture design, pretraining data quality, and general reasoning capability. For engineers who need to build multi-scenario decision systems, base model scores are more informative, because they indicate the performance floor a model may reach on tasks it has not been specifically trained for.

How to Test It Yourself

The LocalLLaMA/typed-decisions dataset is publicly available on Hugging Face and can be loaded directly through the Datasets library:

# Load the dataset via the Hugging Face Datasets library
pip install datasets

python -c "
from datasets import load_dataset
ds = load_dataset('LocalLLaMA/typed-decisions', 'customer_service')
print(ds['test'][0])
"

You can also explore it interactively via Data Studio on the dataset page, or run queries using the SQL Console built into the Hugging Face dataset viewer.

As for how to access the Darwin-27B-ZTC-v5 model itself and its availability, it's best to confirm directly through VIDRAFT's official channels. The leaderboard entry links to the evaluation results via the FINAL-Bench organization on Hugging Face, where model cards and availability updates can be viewed.

FAQ

Is Typed Decisions a calibration benchmark or an accuracy benchmark?

Both. Each prediction includes a full probability distribution for each label category (such as outcome__probabilities, risk__score, urgency__score), which means models are evaluated not only on whether their top label is correct but also on the calibration quality of their confidence. This is more demanding than a flat classification benchmark.

What enterprise use cases does this benchmark cover?

The four subsets map directly to common enterprise automation scenarios: customer service classification, invoice processing routing, security incident classification, and AI agent trace observability (determining whether an autonomous agent's action requires human review). These are all practical, high-stakes decision domains.

How does Darwin-27B-ZTC-v5 differ from its predecessor?

According to public leaderboard data, v5 achieves an accuracy score of 0.81 versus 0.74 for its predecessor, a gain of +0.07. Internal architecture or training differences have not been publicly disclosed.

One-line Summary

VIDRAFT's Darwin-27B-ZTC-v5 sets a new Typed Decisions benchmark record with 0.81 accuracy, up 0.07 from its predecessor.

Paragraph Summary

South Korean AI startup VIDRAFT's latest model, Darwin-27B-ZTC-v5, tops Hugging Face's typed-decisions benchmark with 0.81 accuracy, up 0.07 from its predecessor's 0.74, marking the highest score in the leaderboard's history. The benchmark is a calibration-aware structured decision evaluation requiring models to output multi-dimensional labels and full probability distributions for enterprise workflow states across customer service, invoice processing, security incidents, and agent trace observability. The leaderboard includes only base models, and Darwin-27B-ZTC-v5 leads competitors ranging from 421M to several billion parameters with its 27 billion parameters. The dataset is publicly available under Apache-2.0 and can be loaded directly via Hugging Face Datasets for testing, while model access must be confirmed through VIDRAFT's official channels.