Spring AI Getting Started: Build Java AI Apps in 5 Steps

Why Java Developers Should Care About Spring AI

In an era where generative AI is sweeping through software development, Java ecosystem developers no longer need to envy Python's convenience. Spring AI provides a one-stop abstraction for interacting with large language models (LLMs) in Spring Boot applications, letting you quickly build AI features using familiar annotations and dependency injection. Whether you're connecting to a local Ollama model or calling cloud APIs, Spring AI dramatically lowers the integration barrier.

According to the practical sharing in the tech blog Spring AI: First Steps, you only need five steps to go from scratch to a complete capability ranging from "Hello World" to real-time streaming responses. This article is based on that tutorial and, combined with Spring AI's core design philosophy, maps out a clear learning path for you.

Preparation: Running a Local Model

Spring AI supports multiple model providers, but the most convenient way to get started is to run open-source models locally using Ollama. First, make sure the Ollama service is running and download the model you need (for example, qwen2.5:3b):

ollama pull qwen2.5:3b

Then configure the connection information in your Spring Boot application.properties:

spring.ai.ollama.chat.options.model=qwen2.5:3b
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.temperature=0.7

temperature controls the randomness of the output: 0.0 tends toward determinism, while 1.0 is more creative. You can replace it with any model that appears in ollama list, but the name must match exactly.

Ollama is an open-source local large model runtime tool that bundles model weight downloads, the inference engine, and an HTTP API together, allowing developers to run open-source models on a laptop without a GPU cluster or cloud service account. Its architecture is divided into two layers: the Ollama daemon is responsible for loading models and exposing a REST interface (default port 11434), while the CLI is used to pull, list, and run models. Spring AI's Ollama Starter essentially wraps this REST interface into Spring auto-configuration beans. Note that the inference speed of local models depends on VRAM size and quantization precision; a 3-billion-parameter model like qwen2.5:3b usually achieves acceptable response speeds on consumer-grade hardware, but there is a clear gap in reasoning capability compared with cloud-based large models. It's sufficient for getting-started validation, but production scenarios need to assess whether model capability matches requirements.

Step 1: AI Hello World—The Simplest Chat Endpoint

Create a REST controller, inject ChatClient.Builder, and build a ChatClient. The following code implements an endpoint that sends a fixed prompt to the model and returns a text response:

@RestController
public class AiController {
    private final ChatClient chatClient;

    public AiController(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder.build();
    }

    @GetMapping("/ai/hello")
    public String aiHello() {
        String prompt = "Say 'Hello, World!' and add one short, interesting fact about the Java language.";
        return chatClient.prompt()
                .user(prompt)
                .call()
                .content();
    }
}

Visit http://localhost:8080/ai/hello to see the model's reply. The first request may take 2–5 seconds because the model needs to be loaded into memory. If you encounter a 404 model-not-found error, check the model name in ollama list. .call().content() is a synchronous call; if you need a typewriter effect, .stream() will be introduced later.

This example demonstrates Spring AI's most basic API style: composing prompts, calling the model, and extracting content through a fluent builder API.

Step 2: Structured Output—Making the LLM Return Java Objects

In real business scenarios, we often need the model to output structured data rather than plain text. Spring AI supports directly deserializing JSON responses into Java Records. Define a Record to represent a routing intent:

public record RoutingIntent(
    String targetService,
    String action,
    String userId,
    int priority
) {}

Then add an endpoint in the controller and use the .entity() method to specify the target type:

@GetMapping("/ai/route")
public RoutingIntent routeRequest(@RequestParam String text) {
    String systemPrompt = """
        You are a smart API gateway. Analyze the user's request 
        and extract data for routing. 
        If a field is unknown, leave it empty (null or 0).
        """;

    return chatClient.prompt()
            .system(systemPrompt)
            .user(text)
            .call()
            .entity(RoutingIntent.class);
}

Request http://localhost:8080/ai/route?text=Disable notifications for user user-999, and Spring will automatically serialize the JSON returned by the model into a RoutingIntent object. This eliminates the tedious work of manually parsing JSON and lets the LLM's output flow directly into your business logic.

Structured output relies on the LLM's JSON mode or function-calling-style constrained decoding capability. When Spring AI calls .entity(Class), it converts the target Java type into a JSON Schema and constrains the model—through prompts or the model provider's native structured output parameters (such as OpenAI's response_format)—to produce only JSON that conforms to that Schema, which is then deserialized into an object by Jackson. This implies two engineering constraints: first, the model must support structured output or have sufficient instruction-following capability, as small-parameter models may return malformed JSON and cause deserialization exceptions; second, Spring AI performs retries and format repair by default, but for complex nested types it is still advisable to explicitly list field names and value ranges in the prompt. Regarding field optionality, using wrapper types in Java Records (such as Integer rather than int) correctly expresses null semantics, preventing the model from being forced to assign zero values when fields are missing.

Step 3: Function Calling—Letting the LLM Call Java Methods

Function Calling is a key capability of AI applications: the model can proactively request the execution of local code to obtain real-time data or perform operations. Spring AI simplifies this process with the @Tool annotation.

Create a SystemTools service whose methods are annotated with @Tool and whose parameters are described with @ToolParam:

@Service
public class SystemTools {
    @Tool(description = "Checks the current status and latency (in ms) of the specified microservice")
    public String getServiceStatus(@ToolParam(description = "Name of the service, e.g., 'payment', 'auth', 'database'") String serviceName) {
        // Simulate calling a monitoring system
        return switch (serviceName.toLowerCase()) {
            case "payment" -> "Status: OK, average latency 45ms.";
            case "auth" -> "Status: DEGRADED, latency 300ms, experiencing timeouts.";
            default -> "Status: Unknown service or no data available.";
        };
    }
}

In the controller, register the tool via ChatClient.Builder.defaultTools(systemTools):

public AiController(ChatClient.Builder chatClientBuilder, SystemTools systemTools) {
    this.chatClient = chatClientBuilder
            .defaultTools(systemTools)
            .build();
}

After that, when a user asks "What is the current status of the payment and auth services?", the model will recognize that it needs to call the getServiceStatus method, Spring AI will execute it and return the result, and the model will then generate a final answer based on real data. This avoids model "hallucinations" and ensures the accuracy of the information.

The mechanism of Function Calling is as follows: the application sends the names, descriptions, and parameter schemas of the available tools to the model together; the model does not execute code directly but returns a structured "call intent," which Spring AI executes locally via reflection on the corresponding Java method, then sends the execution result back to the model as a tool message for a second round of reasoning, ultimately generating a natural language answer. Therefore, the quality of the @Tool description and @ToolParam descriptions directly determines whether the model can select the right tool at the right time—this is essentially prompt engineering. From a security perspective, special attention is needed: the input parameters of tool methods come from model generation, so whitelist validation and permission control must be implemented to prevent the model from being induced to call high-risk operations; at the same time, timeouts should be set for tool execution to prevent external dependencies from blocking the entire conversation chain.

Step 4: Prompt Templates—Dynamically Generating Prompts

Hardcoded prompts are hard to maintain and reuse. Spring AI provides PromptTemplate, which supports Mustache-like variable placeholders. Create an endpoint that receives user input and renders the template:

@GetMapping("/ai/template")
public String generateWithTemplate(
        @RequestParam String userName,
        @RequestParam String serviceName,
        @RequestParam String action
) {
    String templateText = """
        You are a system administrator. User {userName} wants to perform the action '{action}' 
        on the {serviceName} service. 
        Provide a brief instruction on how to do this safely.
        """;

    PromptTemplate promptTemplate = new PromptTemplate(templateText);
    String finalPrompt = promptTemplate.render(Map.of(
            "userName", userName,
            "serviceName", serviceName,
            "action", action
    ));

    return chatClient.prompt()
            .user(finalPrompt)
            .call()
            .content();
}

Templates can be stored in separate files (.txt or .prompt), achieving separation from code and version management. This approach makes prompt engineering more systematic.

Step 5: Streaming Responses—Achieving a Typewriter Effect

For chat interfaces or voice assistants, waiting for a complete response harms the user experience. Streaming allows tokens to be returned one by one, and the frontend can render them in real time. Spring AI returns Flux<String>, which Spring Boot automatically converts to Server-Sent Events (SSE).

First, add the WebFlux dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

Then create a streaming endpoint:

@GetMapping(value = "/ai/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE + ";charset=UTF-8")
public Flux<String> streamResponse(@RequestParam String question) {
    return chatClient.prompt()
            .user(question)
            .stream()
            .content();
}

The accompanying frontend page (stream-test.html) uses fetch to read the stream and parse the SSE data chunk by chunk, dynamically updating the interface. The complete code is in the original article; the core logic is a reader.read() loop that decodes the byte stream, extracts the content of data: lines, and appends it to the DOM. Open http://localhost:8080/stream-test.html to experience the word-by-word output effect.

Server-Sent Events (SSE) is a one-way server push protocol based on HTTP, with the response header text/event-stream; the server continuously pushes events as text lines starting with data:, and the browser consumes them through EventSource or the streaming read interface of fetch. Compared with WebSocket, SSE natively supports reconnection, uses standard HTTP ports, and requires no extra protocol upgrade—making it very suitable for LLM scenarios of "one question, one answer, with the server continuously emitting tokens." Spring AI's .stream() returns Reactor's Flux<String>, and Spring MVC or WebFlux adapts it into SSE frames; spring-boot-starter-webflux is introduced to obtain Reactor and reactive codec support. There are two common pitfalls in practice: first, charset=UTF-8 needs to be explicitly declared in produces, otherwise Chinese characters may be garbled; second, reverse proxies (such as Nginx) buffer responses by default, and buffering must be disabled to actually see word-by-word output.

Summary and Next Steps

Through these five steps, you have mastered the core usage of Spring AI: basic chat, structured output, function calling, prompt templates, and streaming responses. These capabilities are enough to build many practical AI integration scenarios, such as intelligent customer service, dynamic routing, and system monitoring Q&A.

Spring AI is still evolving rapidly. Going forward, you can explore vector database integration (RAG), multimodal input, and more complex tool orchestration. For Java teams, Spring AI provides a low-friction path to adopting AI and is well worth deeper practice.

The practical steps in this article are based on the tech blog Spring AI: First Steps, and the code examples have been organized according to the logic of the original article.