Voice-Driven Geometric Interaction: LLM Semantic Parsing and Function Calling Architecture in Practice

Starting with a Real-World Use Case

A developer posted an insightful question on Reddit: how to use a specialized "math LLM" to translate natural language instructions into actual operations on geometric shapes, then execute them through a software SDK.

Here's what he had in mind: a user speaks or types a command like "take the concave shape," the system detects the keyword "concave" and automatically surfaces a slider for visual control over the effect's intensity. When the user follows up with "break it up," the slider appears again, letting the SDK determine the splitting granularity. He also wanted the system to automatically treat synonyms like "hills" and "dips" as equivalent to "concave."

This use case cuts to the heart of a core challenge in LLM application development: how to organically combine language understanding with deterministic mathematical computation and external tool invocation.

Clearing Up a Common Misconception

What "Math LLMs" Can and Can't Do

"Math LLMs" typically refer to models strengthened through mathematical reasoning training data — such as DeepSeek-Math, Qwen-Math, and tool-augmented GPT variants. These models owe their capabilities to specific training strategies: starting from general pretraining, they undergo continual pretraining on large-scale math corpora (arXiv papers, competition problem sets, textbook exercises), followed by supervised fine-tuning on Chain-of-Thought data, and reinforcement learning guided by a Process Reward Model (PRM). The key distinction between PRM and an Outcome Reward Model (ORM) is that PRM scores each step in the reasoning process rather than only the final answer — this significantly improves reliability in multi-step derivations. These models excel at problem-solving, symbolic reasoning, and constructing logical chains.

However, for the developer's use case, relying purely on a "math LLM" is actually the wrong approach. Computing dot products or determining shape concavity are deterministic geometric calculations that don't require an LLM to "infer" results. Even math-specialized models are constrained by the inherent limitations of token-based representation — floating-point numbers aren't continuously distributed in token space, making exact arithmetic inherently unreliable. Where LLMs are truly needed is in semantic parsing and intent routing — mapping natural language like "hills" and "dips" to specific geometric concepts and function calls.

The core takeaway: LLMs handle "listening" and "translating"; geometric computation goes to specialized libraries. Having an LLM perform precise floating-point arithmetic is actually the least reliable option.

Recommended Architecture: Function Calling Is the Real Key

Replace Direct Computation with Tool Invocation

The modern paradigm for this type of application is Function Calling (or Tool Use). GPT-4, Claude, Qwen, and other leading models all natively support this capability.

Function Calling was first formally introduced and standardized by OpenAI in 2023 for the GPT series. The core mechanism: developers describe callable external function signatures (function name, parameter types, description) to the model in JSON Schema format. During inference, the model determines when a tool call is needed and outputs a structured JSON invocation instruction conforming to the schema — rather than free text. Architecturally, Function Calling effectively transforms an LLM from a closed knowledge system into an open reasoning orchestrator, enabling collaboration with databases, APIs, local codebases, and any other external system.

The overall workflow looks like this:

1. Developers define tool functions in advance — e.g., detect_concavity(shape), calculate_dot_product(shape), split_shape(shape, ratio)
2. Function descriptions (name, parameters, purpose) are provided to the LLM as a schema
3. When the user inputs "take the concave shape," the LLM parses the intent and outputs a structured invocation: {"function": "detect_concavity", "args": {...}}
4. The application intercepts the call, hands it off to the SDK for actual geometric computation, and returns the result

This architecture ensures mathematical precision is guaranteed by trustworthy code, while the LLM stays focused on what it genuinely does well: semantic mapping.

Synonym Normalization and Slider Interaction

Synonym handling has two approaches:

• Lightweight approach: Declare mapping rules directly in the System Prompt — e.g., "'hills,' 'dips,' and 'concave' all refer to concave geometric features" — and let the LLM handle normalization naturally. The System Prompt holds the highest priority in the conversation context, used for defining the model's role, behavioral constraints, and domain knowledge injection. This falls under Prompt Engineering, has minimal development overhead, but relies on the model's instruction-following ability and may produce inconsistencies with low-parameter models or complex ambiguous scenarios.

• Rigorous approach: Combine word vectors or semantic similarity matching to standardize synonyms during preprocessing — suitable for production-scale deployment. Modern practice typically uses sentence embedding models like Sentence-BERT (SBERT) to map words or phrases to dense vectors, then computes cosine similarity for semantic proximity. For terrain/geometry synonym scenarios, domain-specific lexicons combined with vector retrieval double-verification can be applied. Approximate nearest-neighbor libraries like FAISS (Facebook AI Similarity Search) can match against large vocabulary sets in milliseconds, making them well-suited for real-time interactive scenarios requiring low-latency responses.

Dynamic slider triggering is a frontend interaction concern: when the intent returned by the LLM includes a flag indicating "user needs to specify a degree parameter," the application layer surfaces the slider UI accordingly, feeding the real-time slider value back as a parameter to the next function call — completing the feedback loop.

Layered Design and Technology Choices

Three-Layer Architecture, Each with Its Own Responsibility

Layer	Responsibility	Recommended Technology
Semantic Layer	Natural language parsing, intent routing, synonym normalization	General-purpose LLM with Function Calling support
Computation Layer	Geometric relationship evaluation, vector operations, precise floating-point calculation	Shapely (geometry), NumPy (vectors/dot products)
Interaction Layer	Dynamically rendering UI components (sliders, etc.) based on intent	Frontend framework + structured parameters from LLM

Shapely, used in the computation layer, is backed by the GEOS (Geometry Engine - Open Source) engine and fully implements the JTS (Java Topology Suite) topological computation specification, handling concavity analysis, shape simplification, spatial relationship evaluation, and more. For concavity detection, NumPy is commonly used to compute cross products of adjacent edge vectors — if the sign of the cross product changes across polygon vertices, those vertices are concave points. Both libraries share a key characteristic: deterministic computation implemented in C/C++ that guarantees identical output for identical input — something LLMs fundamentally cannot provide. Semantic understanding ability takes priority over mathematical problem-solving ability — this matters more when selecting a model than rankings on math benchmarks.

Model Selection Recommendations

• Commercial models: The GPT-4 series or Claude offer the most mature tool-calling capabilities, with high stability for complex instruction parsing — ideal for rapid product validation
• Open-source options: The Qwen2.5 series (including Math variants) and DeepSeek both provide comprehensive tool-calling support with local deployment capability — well-suited for privacy-sensitive or offline use cases

A Key Engineering Principle

Do not let the LLM directly output geometric computation results. Even the most mathematically capable models today can still make errors when handling precise floating-point operations and edge cases. The correct approach: have the LLM output "which function to call and what parameters to pass," returning full computational determinism to the code layer. This is the foundational principle for engineering that is robust, testable, and maintainable.

Summary

What this developer envisions is essentially a natural language–driven geometric operation system. Its technical core isn't about finding a sufficiently powerful "math LLM" — it's about designing a well-layered architecture:

• LLM handles semantic translation and intent routing
• Specialized SDK handles precise geometric computation
• Frontend handles dynamic interactive feedback

Understanding this clearly prevents over-investment in the wrong direction, and keeps limited engineering resources focused on the parts that genuinely require intelligence.