Model Router Auto-Evaluation
An open-source model router evaluation tool from Microsoft that supports automated assessment of model routing solutions against any baseline model across three dimensions—quality, cost, and latency—and generates visual dashboards summarizing the results.
Timeline (last 90 days)
Model Router Auto-Evaluation is an open-source project that can automatically evaluate the performance of Foundry Model Router against any baseline model across three dimensions: quality, cost, and latency.
Official guidance recommends that the judge model should differ from the baseline model, to avoid a model scoring its own outputs and thereby favoring outputs that are closer to its own style.
The Model Router Auto-Evaluation project comes with a sample dataset of 10 prompts, which users can replace with a custom dataset.
The deployment names for the baseline model and judge model can be found under Build → Models in the Foundry portal, and custom deployment names require adding a corresponding pricing entry in the pricing section of default.yaml.
All Facts (4)
Model Router Auto-Evaluation is an open-source project that can automatically evaluate the performance of Foundry Model Router against any baseline model across three dimensions: quality, cost, and latency.
50%UnverifiedOfficial guidance recommends that the judge model should differ from the baseline model, to avoid a model scoring its own outputs and thereby favoring outputs that are closer to its own style.
50%UnverifiedThe Model Router Auto-Evaluation project comes with a sample dataset of 10 prompts, which users can replace with a custom dataset.
50%UnverifiedThe deployment names for the baseline model and judge model can be found under Build → Models in the Foundry portal, and custom deployment names require adding a corresponding pricing entry in the pricing section of default.yaml.
50%