RouteEval: A Benchmark for Evaluating LLM Tool Calling in Geographic Route Generation

By Veejhay Roy, Roger Jin • 2025-12-17

A comprehensive evaluation of 13 state-of-the-art LLMs on geographic route generation tasks, revealing insights into tool-calling capabilities and accuracy trade-offs. Our benchmark assesses 9,498 total evaluations across 50 diverse prompts with 16 runs each.

Abstract

Large Language Models (LLMs) are increasingly being used in applications requiring interaction with external tools and APIs. However, systematic evaluation of their tool-calling capabilities in real-world geographic tasks remains limited. We present RouteEval, a benchmark designed to evaluate LLMs' ability to generate accurate running routes through tool calling. Route generation presents unique challenges beyond general tool use: it requires spatial reasoning, precise numerical constraint satisfaction, and real-world validation. Unlike mathematical calculation tasks that test numerical precision in isolation, route generation demands that models synthesize spatial knowledge, interpret user preferences, and produce physically feasible results validated against real-world constraints.

Our benchmark assesses 13 state-of-the-art models across 50 diverse prompts with 16 runs each (totaling 800 evaluations per model), measuring their capability to use a route generation tool effectively while adhering to distance constraints. We evaluate routes using Google Maps distance validation and introduce a novel accuracy metric that penalizes deviation from target distances. Our analysis of 9,498 completed evaluations reveals significant variation in model performance, with GPT-5 (High) achieving the highest average accuracy (0.650) but lower success rate (91.1%), while Claude-Sonnet-4 demonstrates perfect success rate (100.0%) with moderate accuracy (0.458). These results highlight the complex trade-offs between reliability and precision in LLM tool calling for location-based services.

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. Recent developments have extended these capabilities to include interaction with external tools and APIs through structured tool calling mechanisms [OpenAI, 2024; Anthropic, 2024; Google, 2024a]. This advancement enables LLMs to perform complex tasks that require accessing real-time data, performing calculations, or interacting with specialized services.

Running route generation represents a particularly challenging application domain for LLM tool calling that combines spatial reasoning, numerical precision, and practical constraint satisfaction. Unlike mathematical calculation benchmarks that test numerical accuracy in isolation, route generation requires models to: (1) understand and interpret natural language descriptions of user preferences and constraints, (2) synthesize geographic knowledge about locations, landmarks, and terrain, (3) generate physically feasible routes that respect distance targets while accommodating user preferences, and (4) utilize tool-calling APIs to produce structured, validable outputs. This combination of natural language understanding, spatial reasoning, numerical precision, and tool orchestration makes route generation an ideal testbed for evaluating LLM capabilities beyond purely text-based tasks.

The RouteEval Benchmark

RouteEval introduces a comprehensive benchmark designed to evaluate LLMs' ability to generate accurate running routes through tool calling. Our benchmark assesses 13 state-of-the-art models across 50 diverse prompts with 16 runs each (totaling 800 evaluations per model), measuring their capability to use a route generation tool effectively while adhering to distance constraints.

Key Contributions

  • Comprehensive Evaluation: 13 state-of-the-art LLMs evaluated on 50 diverse route generation prompts with 16 runs each
  • Novel Accuracy Metric: Measures deviation from target distances using Google Maps validation
  • Large-Scale Analysis: 9,498 completed evaluations (out of 10,400 planned) revealing distinct patterns in model reliability and precision
  • Real-World Validation: Routes validated using Google Maps distance measurements for accuracy
  • Success-Accuracy Trade-off Analysis: Systematic evaluation of the relationship between reliability and precision in tool-calling tasks

Methodology

Tool Calling Framework

Our evaluation framework uses the standardized tool calling mechanism provided by the OpenRouter API [OpenRouter, 2024], which supports multiple LLM providers through a unified interface. Models are presented with a generate_running_route tool that accepts:

  • estimated_distance: Target distance in miles (number)
  • route_type: Type of route (loop, out-and-back, or point-to-point)
  • waypoints: Array of location objects with address and description fields

The tool expects a JSON structure with waypoints containing both address and description fields, enabling models to specify geographic locations with contextual information.

Evaluation Process

Each evaluation follows a standardized process:

  1. Prompt Generation: User provides a prompt requesting a running route with specific constraints (distance, location, preferences)
  2. Tool Call Generation: LLM processes the prompt and generates a tool call with appropriate parameters
  3. Format Validation: System validates the tool call format and parameter types
  4. Route Execution: Valid tool calls are executed to generate waypoint sequences
  5. Distance Validation: Google Maps Directions API validates the actual walking distance of the generated route
  6. Accuracy Calculation: Accuracy score is calculated based on distance deviation from target

Accuracy Metric

We introduce a novel accuracy metric that penalizes deviation from target distances:

accuracy = max(0, 1 - (|actual_distance - target_distance| / target_distance))

This metric ranges from 0 to 1, where:
- 1.0: Perfect match to target distance (≤5% error)
- 0.8-0.99: High accuracy (within 20% of target)
- 0.0: Significant deviation (≥100% error)

The metric provides symmetric treatment of over- and under-estimation. For example, a 5-mile route request that generates a 6-mile route receives an accuracy score of 0.80, while a 4-mile route receives the same score.

Success Rate Measurement

Success rate measures the proportion of evaluations where the model successfully completed the tool-calling task and generated a valid route. Failures include:

  • Failure to use tool calling (returning plain text instead)
  • Invalid tool call format or parameters
  • API timeouts or errors
  • Invalid waypoint locations that cannot be geocoded

Success rate provides a complementary metric to accuracy, capturing model reliability independent of distance precision.

Models Evaluated

We evaluated 13 state-of-the-art LLMs representing diverse architectures and providers:

  • OpenAI: GPT-5 (High reasoning), GPT-4o-mini
  • Anthropic: Claude-Opus-4.1, Claude-Sonnet-4
  • Google: Gemini-2.5-Pro, Gemini-2.5-Flash
  • xAI: Grok-4
  • DeepSeek: DeepSeek-V3.1
  • Zhipu AI: GLM-4.5
  • Moonshot AI: Kimi-K2
  • Qwen: Qwen3-235B-A22B-2507, Qwen3-Coder
  • NousResearch: Hermes-4-70B

Model selection prioritized recent releases with documented tool-calling support and availability through the OpenRouter API.

Results

Our analysis of 9,498 completed evaluations (out of 10,400 planned: 50 prompts × 16 runs × 13 models) reveals significant variation in both success rate and accuracy across all evaluated models.

Overall Performance

Model Success Rate Avg Accuracy Perfect Rate High Accuracy Rate
GPT-5 (High) 91.1% 0.650 13.4% 49.0%
Grok-4 98.5% 0.555 6.3% 26.9%
Gemini-2.5-Pro 98.5% 0.524 5.8% 21.2%
Gemini-2.5-Flash 99.5% 0.520 6.9% 23.2%
DeepSeek-V3.1 92.6% 0.519 7.2% 26.0%
Claude-Opus-4.1 99.1% 0.477 2.9% 25.7%
GLM-4.5 78.5% 0.475 6.2% 23.4%
Claude-Sonnet-4 100.0% 0.458 4.9% 27.5%
Kimi-K2 88.8% 0.444 5.8% 22.5%
GPT-4o-mini 99.0% 0.430 4.3% 19.1%
Hermes-4-70B 91.6% 0.410 4.5% 17.4%
Qwen3-235B 77.9% 0.373 3.8% 20.4%
Qwen3-Coder 75.0% 0.315 2.7% 11.7%

Perfect Rate: Routes with accuracy ≥ 0.95 (≤5% distance error)
High Accuracy Rate: Routes with accuracy ≥ 0.80 (≤20% distance error)

Key Findings

  1. Top Accuracy: GPT-5 (High) achieves the highest average accuracy (0.650) but with lower success rate (91.1%). The gap between GPT-5 (High) at 0.650 and the second-place Grok-4 at 0.555 indicates a meaningful performance advantage, suggesting that increased reasoning capability significantly improves distance precision.

  2. Top Reliability: Claude-Sonnet-4 demonstrates perfect success rate (100.0%) with moderate accuracy (0.458), showing exceptional reliability in tool calling execution.

  3. Performance Variation: Average accuracy ranges from 0.315 to 0.650, indicating significant differences in distance precision across models. This 106% performance gap highlights substantial variation in tool-calling capabilities.

  4. Success-Accuracy Trade-off: Models with highest success rates do not necessarily achieve highest accuracy. Claude-Sonnet-4 achieves perfect reliability but ranks 8th in accuracy, while GPT-5 (High) leads in accuracy but has lower reliability.

  5. Perfect Route Generation: GPT-5 (High) generates perfect routes (accuracy = 1.0) in 13.4% of cases, nearly twice as often as the next best model (DeepSeek-V3.1 at 7.2%). Perfect routes remain relatively rare across all models, with rates ranging from 2.7% to 13.4%.

  6. High Accuracy Routes: Analysis of routes achieving accuracy ≥ 0.8 (within 20% of target distance) shows GPT-5 (High) succeeds in 49.0% of evaluations, significantly outperforming other models. Claude-Sonnet-4 achieves 27.5% high accuracy rate despite lower average accuracy, demonstrating consistency in producing usable routes.

Success Rate Analysis

Claude-Sonnet-4's perfect success rate (100.0%) demonstrates exceptional reliability in tool calling, while Qwen3-Coder's 75.0% success rate indicates challenges with consistent tool call execution. The variation in success rates (75.0% to 100.0%) suggests different levels of robustness in tool-calling implementations across models.

Accuracy Distribution

Models show distinct accuracy distribution patterns:
- GPT-5 (High): Highest concentration of high-accuracy routes (49.0% ≥ 0.80 accuracy)
- Claude-Sonnet-4: Most consistent performance with 100% success rate
- Qwen3-Coder: Lowest performance across all metrics, suggesting limitations in tool-calling capability

Implications

These findings have important implications for deploying LLMs in location-based services:

For Developers

  • Model Selection Matters: Performance varies significantly across models. Choose models based on whether reliability (high success rate) or precision (high accuracy) is more critical for your application.

  • Application-Specific Optimization:

  • For applications requiring high precision: GPT-5 (High) offers best accuracy (0.650) despite lower success rate
  • For applications requiring reliability: Claude-Sonnet-4 provides perfect success rate (100.0%) with acceptable accuracy

  • For balanced performance: Grok-4 and Gemini models offer strong combinations of both metrics

  • Validation is Essential: Even top-performing models achieve only 65% average accuracy, indicating that validation and fallback strategies are necessary for production systems.

  • Fallback Strategies Needed: Success rates below 100% require error handling and retry mechanisms in production deployments.

For Researchers

  • Tool Calling Optimization: Current tool-calling mechanisms may need refinement to better handle numerical constraints and real-world validation. The success-accuracy trade-off suggests opportunities for model-specific optimization.

  • Domain-Specific Evaluation: Generic benchmarks may not capture the nuances of domain-specific applications like route generation. Specialized evaluation frameworks are necessary for practical applications.

  • Reasoning Capability Impact: GPT-5 (High)'s superior performance suggests that enhanced reasoning capabilities significantly improve tool-calling precision, warranting further investigation into reasoning-enhanced tool calling.

Limitations

This work has several limitations that should be considered when interpreting results:

Scope Limitation

RouteEval focuses specifically on running route generation, not general geographic route generation. The benchmark design, prompts, and validation are tailored to pedestrian running scenarios. Results may not generalize to other route types such as driving routes, cycling routes, or multi-modal transportation.

Geographic Coverage

Our evaluation prompts focus primarily on US locations (28 prompts) with some international cities (10 prompts). Performance may vary for locations with different geographic characteristics, less detailed mapping data, or regions underrepresented in model training data.

Route Complexity

RouteEval focuses on relatively simple route generation tasks (2-13.1 miles, 3-8 waypoints). Performance on more complex routing scenarios with additional constraints (elevation profiles, surface type specifications, safety considerations, accessibility requirements) may differ significantly.

Single Tool Evaluation

Our benchmark evaluates a single tool calling scenario. Real-world applications often require multi-tool orchestration (e.g., geocoding services, elevation APIs, weather APIs), which may reveal different patterns in model capabilities.

Temporal Validity

Model capabilities evolve rapidly. Our evaluation represents a snapshot of model performance as of October 2025. Future model updates may significantly change these results.

Try It Yourself

You can try the route planning system yourself! The interactive route planner demonstrates real-time LLM tool calling for route generation using the same methodology described in this research.

Try the Route Planner →

Conclusion

RouteEval provides a foundation for understanding and improving LLM tool-calling capabilities in location-based services. Our benchmark reveals that while current models show promise, there is significant room for improvement in balancing reliability and precision for geographic applications.

Key takeaways:
- Significant Performance Variation: Accuracy ranges from 0.315 to 0.650, highlighting the importance of model selection
- Clear Trade-offs: Models demonstrate distinct strategies in balancing success rate versus accuracy
- Practical Implications: Even top-performing models require validation and error handling for production use
- Research Opportunities: Enhanced reasoning capabilities show promise for improving tool-calling precision

Future work should explore:
- Fine-tuning models specifically for geographic tasks
- Developing specialized tool-calling mechanisms for numerical constraints
- Expanding the benchmark to include more diverse geographic scenarios and route complexities
- Investigating multi-tool orchestration capabilities
- Exploring methods for improving both reliability and precision simultaneously

References

  1. Anthropic. Tool use (function calling), 2024. URL https://docs.anthropic.com/claude/docs/tool-use. Accessed: 2024-10-01.

  2. Google. Gemini API documentation, 2024a. URL https://ai.google.dev/docs. Accessed: 2024-10-01.

  3. OpenAI. Function calling, 2024. URL https://platform.openai.com/docs/guides/function-calling. Accessed: 2024-10-01.

  4. OpenRouter. OpenRouter API documentation, 2024. URL https://openrouter.ai/docs. Accessed: 2024-10-01.

  5. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

  6. Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). Gorilla: Large language model connected with massive APIs. arXiv preprint arXiv:2305.15334.

  7. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824-24837.

  8. Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., ... & Alon, U. (2023). WebArena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

For the complete research paper with detailed methodology, comprehensive results, and full analysis, see: RouteEval Research Paper

This research was conducted as part of the Polygence Project. For questions or collaboration opportunities, please contact usroyvj@gmail.com or roger@nousresearch.com.