ATLAS-Bench: A New Benchmark for Tool Use in Local Search Agents

Local search is a foundational capability for LLM agents supporting everyday decisions and long-term planning—it requires finding places that meet user needs given a specified time and location. However, existing benchmarks offer limited coverage of whether search results satisfy hard requirements, whether preferences are considered, and whether factual and experiential judgments are made based on place attributes and reviews. The recently released ATLAS-Bench attempts to fill this gap.

Why Local Search Needs a Dedicated Benchmark

Local search tasks appear simple but actually involve multiple layers of judgment: among the conditions users specify, some are hard constraints that must be satisfied (such as business hours, accessibility facilities), while others are merely preferences (such as "a bit quieter"); some are verifiable objective facts (such as whether parking is available), while others depend on subjective experiences in reviews (such as service attitude). A qualified agent must not only find places but also distinguish among these conditions, retrieve place attributes and reviews, provide evidence-backed recommendations, or clearly report "no place satisfies all hard requirements" when no solution exists.

Existing benchmarks rarely examine these dimensions simultaneously, making it difficult to accurately measure model performance in real-world scenarios. The starting point of ATLAS-Bench is precisely to separate "constraint satisfaction" and "evidence support" in evaluation.

Hard constraints and preferences correspond to two handling approaches for Constraint Satisfaction Problems (CSP) in local search. Hard constraints are non-negotiable boolean conditions, such as "currently open" or "wheelchair accessible"—if any condition is not satisfied, the recommendation is deemed invalid; preferences are tradeable soft conditions, such as "quiet environment" or "high rating," typically handled through scoring or ranking. The significance of distinguishing the two lies in the evaluation criteria: for hard constraints, the pass rate should be verified item by item, while for soft constraints, ranking quality is what matters. If a model mistakes preferences for hard constraints, it will over-filter and reduce recommendation coverage; conversely, if it treats hard constraints as preferences, it will give places that are actually unusable.

Benchmark Composition: 1,200 Queries Across Three Regions

ATLAS-Bench contains 1,200 queries spanning three regions. These queries are expanded from real search events on the platform and constructed using place attributes, reviews, and existing user profiles. Each query explicitly distinguishes between hard requirements and preferences, factual conditions and subjective judgments, and comes with supporting evidence and item-by-item checking criteria.

Query quality is verified through iterative assertion checks and cross-model review to avoid ambiguous or undecidable samples. This construction approach allows evaluation to go beyond whether the final recommendation "looks reasonable" and instead verify constraint satisfaction item by item.

ATLASBox: A Standardized Evaluation Environment

Paired with the benchmark is ATLASBox, a standardized evaluation environment with built-in search tools and a fixed points of interest (POI) knowledge base. Given a query, timestamp, and location, the agent must retrieve place attributes and reviews, recommend evidence-backed options, or report that no place satisfies all hard requirements.

Evaluation dimensions cover three aspects: answer quality, trajectory quality, and execution cost. The introduction of trajectory quality means that not only results but also the reasonableness of the agent's search process is examined, which is particularly important for diagnosing tool-using agents.

Trajectory quality refers to whether the series of intermediate actions an agent takes from receiving a query to giving an answer are reasonable, including tool call order, whether retrieval parameters are on-topic, whether ineffective searches are repeated, and whether the agent continues to verify when evidence is insufficient. Unlike end-to-end evaluation that only judges the final answer, trajectory evaluation can pinpoint failure points: whether relevant reviews were not retrieved, or were retrieved but not correctly used. Common approaches compare the agent's action sequence against reference trajectories or rule-based checklists, measuring metrics such as the proportion of effective retrievals and the number of redundant calls, thereby further distinguishing "wrong result" into "wrong process" and "wrong judgment."

Model Performance Across 14 Configurations

The study evaluated 14 configurations of 11 models. Results show that GPT-6-Luna achieved the highest constraint satisfaction rate among returned recommendations at 71.5%, with a factual accuracy of 95.6%. However, on the metric of "returning at least one acceptable recommendation for satisfiable queries," it lagged behind GLM-5.3—which scored 85.3% versus GPT-6-Luna's 67.3%.

This comparison reveals a core tension: a model can achieve high constraint satisfaction and factual accuracy on individual recommendations while simply not giving recommendations for some queries, leading to reduced overall usability. In other words, there is a tension between strictly satisfying individual requirements and providing usable recommendations across queries—precisely the real challenge facing local search agents.

Constraint satisfaction rate and recommendation coverage rate are two metrics that pull in opposite directions. Constraint satisfaction rate measures the proportion of returned recommendations that satisfy all hard requirements, with the denominator being the number of recommendations; recommendation coverage rate measures the proportion of satisfiable queries for which at least one acceptable recommendation is given, with the denominator being the number of queries. A model can raise the former by "only recommending when confident," but at the expense of the latter, manifesting as a large number of queries with no results. Therefore, the two must be interpreted jointly: high satisfaction rate with low coverage often means excessive conservatism, while high coverage with low satisfaction rate means hasty recommendations. The ideal state is maintaining high satisfaction rate while coverage approaches its upper limit.

Implications for Agent Development

The value of ATLAS-Bench lies in advancing the evaluation focus from "is the answer correct" to "are constraints satisfied, is evidence sufficient, is the process reasonable." For teams building local search applications, this means optimizing two things simultaneously: first, the ability to strictly verify hard constraints; second, avoiding the tendency toward excessive conservatism that leads to "better to say nothing than to make a mistake." The joint evaluation of answer quality, trajectory quality, and execution cost in the benchmark also provides quantifiable reference for trade-offs in tool-calling strategies.

One-line Summary

ATLAS-Bench evaluates local search agents' constraint satisfaction and evidence support capabilities using 1,200 queries and a standardized environment.

Paragraph Summary

ATLAS-Bench is a new benchmark for LLM agents' local search capabilities, containing 1,200 queries across three regions expanded from real search events, explicitly distinguishing hard requirements from preferences and factual conditions from subjective judgments, with supporting evidence and item-by-item checking criteria. The accompanying ATLASBox provides standardized search tools and a fixed POI knowledge base, evaluating agents across answer quality, trajectory quality, and execution cost. Testing 14 configurations of 11 models shows that GPT-6-Luna has the highest constraint satisfaction rate (71.5%) and 95.6% factual accuracy, but only 67.3% recommendation coverage on satisfiable queries—below GLM-5.3's 85.3%—revealing the tension between individual strictness and overall usability.