Spring Boot + LLM: Ship AI Features Without Building a Python Service

Title

Spring Boot + LLM: Ship AI Features Without Building a Python Service

Meta Description

Integrate LLM features directly into your existing Java API with Spring Boot and Spring AI to automate ticket classification. Covers structured output, enum guardrails, application-level deadlines, concurrency budgets, retry stacking pitfalls, and fallback strategies, with fault-injection test data and a pre-launch checklist.

Article Content

When many Java teams think about adding AI features, their first instinct is to spin up a separate Python microservice. But if the requirement is simply "call a model, get back structured data, and use it in an existing API," you can absolutely do this inside the Spring Boot service you've already deployed, monitored, and secured.

This article comes from a real hands-on experiment: using Spring Boot 4.1.1 + Spring AI 2.0.1 + Java 21, running a local OpenAI-compatible mock server, and taking a ticket auto-classification feature end to end. The focus isn't the demo — it's the "boring parts" you must handle before going live: typed output, call deadlines, concurrency budgets, and fallback plans.

Why Python shouldn't be the default choice

Spring AI and LangChain4j are both mature options on the JVM. The decision logic is simple: if you're already on Spring Boot and want auto-configuration, application.properties parameters, Micrometer observability, and a streaming ChatClient, choose Spring AI. If you're outside the Spring ecosystem (Quarkus, Micronaut, or plain Java), or you need a higher-level "AI services" interface-style Agent and a broader integration catalog, choose LangChain4j.

Both support OpenAI-compatible APIs, covering a wide range of hosted models and local model servers (like Ollama, vLLM). To be clear: for features like ticket classification — "call model → parse structured output → store or return" — keeping it in Spring Boot is reasonable. It's only worth introducing a Python service when you need to train/fine-tune models, run data science pipelines, or do in-process model inference.

Spring AI is the official Spring framework for AI integration. Its core abstractions are ChatClient (a streaming chat interface) and EmbeddingClient, seamlessly integrated with Spring Boot auto-configuration via starter dependencies. Model parameters can be written directly in application.properties and are automatically wired into Micrometer metrics and tracing. It has built-in support for OpenAI, Anthropic, Azure OpenAI, Ollama, and other providers, with especially complete adapters for OpenAI-compatible APIs. LangChain4j, meanwhile, is a community project inspired by LangChain. Beyond low-level model calls, it offers declarative interfaces like AiServices — just define a Java interface and the framework generates the implementation, handling prompt assembly and result parsing. Its integration catalog (vector stores, document loaders, etc.) is also broader. In short, Spring AI wins on native integration with the Spring ecosystem, while LangChain4j wins on higher-level abstractions and richer surrounding integrations. Both run on the JVM, so you don't need to switch languages just to call a model.

Core design: cage the uncertainty

Ask the model for just one small record

Define an LlmTriage record with fields: enum Category, enum Priority, summary, and a tags list. Every field is a promise the model must fulfill — the fewer the fields, the more controllable it is.

A key detail: keep the model-side type (LlmTriage) separate from the API return type (TicketTriage). In the author's first version, the API record with a source field was passed directly to .entity(...), and as a result Spring AI generated a JSON Schema listing source as required — forcing the model to guess whether it was a fallback source. The model shouldn't fill in things it doesn't know.

Using enums is the cheapest guardrail. The generated schema includes "enum": ["BILLING", "BUG", ...], so if the model fabricates "PAYMENTS", parsing fails outright rather than letting an unknown category leak into the database.

Structured Output means having the model return data conforming to a predefined JSON Schema rather than free text. Spring AI's .entity(Class<T>) reflects a JSON Schema from the target Java record and sends it along with the request. Providers that support this capability (like OpenAI's Structured Outputs) enforce constraints at the decoding stage, so the output conforms to the schema at the syntactic level. Java's record is an ideal carrier: immutable fields, concise, and each component naturally maps to a schema property. Enums play a key role here — they map to the schema's enum constraint, restricting the model's choices to a finite set and eliminating hallucinations like "fabricated categories" at the source. Note that fields generated as required in the schema are promises the model must fulfill, so the model-side type should contain only fields the model can actually decide. Any metadata populated by the application (like a source marker) must be stripped from that type.

Deadlines, budgets, and fallback

The service layer does four things:

• Input limits: ticket text must be 1 to 4000 characters; anything longer is rejected outright, protecting your token bill and latency.
• Prompt injection defense: ticket content goes into <ticket> tags via template parameters, and the system prompt states the rules. This doesn't fully prevent injection, but combined with enum output, the worst case is a misclassification rather than arbitrary text reaching the UI.
• Application-level deadline: use virtual threads to wrap the blocking call in a Future, and once triage.timeout (6 seconds in the example) hits, cancel and fall back — regardless of what's happening on the provider side.
• Concurrency budget: a Semaphore limits in-flight model calls (8 in the example). When full, it falls back immediately instead of queuing — protecting both the rate limit and the thread pool.

The fallback returns source: "fallback", telling the UI and metrics system this one needs manual classification. The feature degrades gracefully rather than causing an outage.

Virtual threads (a finalized Java 21 feature, JEP 444) are lightweight threads scheduled by the JVM. They're extremely cheap to create and automatically unmount the carrier thread when blocked, making them ideal for wrapping model calls that "wait on network I/O." Submit the blocking call to a virtual-thread executor and get a Future; the main thread can then apply an application-level deadline via future.get(timeout), and on timeout, cancel(true) triggers interruption and transitions to the fallback path. A Semaphore is an AQS-based counting synchronizer whose tryAcquire() is non-blocking and immediately returns a boolean — perfect for the "fail fast when the budget is full rather than queueing" semantics, avoiding request pileup that causes latency avalanches while protecting the downstream provider's rate limits. Together, these give the most uncertain link in the chain — the model call — the same predictability as an ordinary remote call: a cap, a timeout, and a safety net.

Fault-injection test results: two counterintuitive findings

The author used a mock server to force each failure mode. Actual results:

• Valid JSON → normal return, ~0.7s
• Prose instead of JSON → fallback, ~0.01s (parse exception)
• Model hangs for 8s → fallback, 6.0s (application deadline triggered)
• HTTP 500 → fallback, ~0.5s (upstream called 2 times)

Finding 1: Retries stack. With default configuration, a single failed user request produced 4 upstream model calls. Spring AI 2.0's OpenAI integration is built on the official openai-java SDK, which has its own retry setting (spring.ai.openai.chat.max-retries, default 3 in the configuration metadata). For a user-facing endpoint with a fallback, this means the user waits through three additional billed attempts. Setting it to 1 brought it down to 2 calls. Another detail: spring.ai.openai.chat.options.max-retries is also in the metadata, but in testing it had no effect — only the property without options worked. Before going live, count upstream calls in your provider's logs.

Finding 2: Keep your own deadline. The metadata also has spring.ai.openai.chat.timeout (default 60s), but setting it to 4s did not cut off a hanging call before the 6s application deadline. The author didn't dig deeper and recommends verifying in your own environment. Either way, the application-level deadline is something you control and can unit-test.

Retry stacking stems from retries happening at multiple independent layers: the openai-java SDK has exponential backoff retries for 429/5xx and connection errors (default 3), and if application code wraps another retry framework on top (like Spring Retry or Resilience4j), a single user click gets amplified into a multiplicative number of upstream calls. For endpoints with a fallback, retries often do more harm than good — the fallback already guarantees availability, so extra retries only lengthen the user's wait and increase billing. SDK timeouts and application deadlines have different semantics: the former applies to a single HTTP connect/read, the latter constrains the total duration of the whole business call. If the SDK timeout exceeds the application deadline, the application layer returns the fallback first while the underlying request may continue in the background — and could even be billed after cancellation. The relationship should be "SDK timeout ≤ application deadline," with the application layer as the source of truth, verified by unit tests.

Pre-launch checklist

• Output is a small record with enums; model-side type is separate from the API type
• Input length is bounded and passed as a template parameter, not concatenated into instructions
• max-tokens is set, and temperature is lowered for classification scenarios
• Application deadline is shorter than the HTTP client timeout and the user's patience
• Count retries end to end (SDK + your code + client)
• Concurrency is capped, and overflow goes to fallback
• Responses indicate whether they came from the model or the fallback, and that ratio is monitored
• You know which personal data leaves your system with the prompt, and the provider's data retention terms permit it

Version notes

Spring AI 2.0 targets Spring Boot 4.x; if you're still on Boot 3.x, use Spring AI 1.x. The ChatClient and .entity(...) APIs exist there too, but property names may differ. .entity(...) also supports reliability switches (validateSchema() retries with feedback on errors, useProviderStructuredOutput() sends the schema as a provider-level constraint). The author didn't test these on the mock, so they're not in the sample code. Every validation retry is another billed call. Also, if the same project pulls in springdoc, watch for swagger-annotations version conflicts (the author had to pin 2.2.55).

When to add a Python service

It's reasonable to spin up a separate Python service when you need to train or fine-tune models, need libraries that only exist in Python for in-process inference, or your ML team wants an independent release cadence. Even then, the structure on the Spring Boot side stays the same: a typed call with a deadline and fallback — only the URL changes.

For most business features that "call a model and get structured data," the answer is straightforward: you don't need a new language and a new deployment unit just to use AI.

One-line Summary

Implement ticket classification inside an existing service with Spring Boot + Spring AI, focusing on typed output, deadlines, and fallback.

Paragraph Summary

Based on a real experiment, this article demonstrates how to implement ticket auto-classification directly in a deployed Spring Boot 4.1.1 + Spring AI 2.0.1 + Java 21 service without building a separate Python microservice. It compares Spring AI and LangChain4j for selection, noting that only training/fine-tuning models or in-process inference justify Python. The core design cages model uncertainty: constrain output with a small enum-bearing record, separate model-side and API types, limit input length, implement an application-level deadline with virtual threads, cap concurrency with a semaphore, and unify fallback. Fault-injection tests reveal two counterintuitive findings — SDK retries stack with application retries (a single failed request produces 4 upstream calls by default), and SDK timeouts don't replace application deadlines. The article ends with a pre-launch checklist and version compatibility notes.