Integrating CloudWeGo Eino with Custom OpenAI-Compatible APIs: A Practical Guide

When building LLM applications in the Go ecosystem, CloudWeGo's Eino framework is a choice worth considering. It provides chat models, tool calling, graph orchestration, and ReAct Agents, with a style similar to LangChain but tailored for Go. Its OpenAI component (github.com/cloudwego/eino-ext/components/model/openai) is a common entry point for connecting to compatible services such as vLLM, Ollama's /v1 endpoint, LiteLLM, or gateways. One developer, using eino v0.7.13 and eino-ext OpenAI component v0.1.13 on Go 1.27.2, ran detailed tests against a local mock server and recorded the actual HTTP request details. Based on this measured data, this article outlines the key configuration points and pitfalls when connecting to custom APIs.

Minimal Configuration: Three Fields and a Compilation Pitfall

After installing dependencies, the first go build may trigger a compilation error: sonic v1.14.1, which Eino depends on, fails on Go 1.27 with undefined: GoMapIterator. The fix is to manually upgrade sonic to the latest version and run go mod tidy.

Model initialization itself is concise, with the core being the BaseURL, APIKey, and Model fields. Note that the struct comments state BaseURL is for Azure, but in practice all non-Azure clients use it. Eino does not read OPENAI_API_KEY or OPENAI_BASE_URL from environment variables; they must be passed explicitly.

cm, err := openai.NewChatModel(ctx, &openai.ChatModelConfig{
    BaseURL: "https://your-gateway.com/v1",
    APIKey:  os.Getenv("YOUR_API_KEY"),
    Model:   "YOUR_MODEL_ID",
    Timeout: 60 * time.Second,
})

Testing confirms that the request path is strictly BaseURL + /chat/completions, and /v1 is not automatically appended. If BaseURL is only https://host, the path becomes /chat/completions, which most gateways will return 404 for. The request body is minimal by default, containing only messages and model, and does not send temperature or max_tokens. The Timeout field sets no timeout by default, so you must configure it manually or pass a custom HTTPClient.

Sampling Parameters Use Pointers; Zero Values Are Sent

Sampling parameters such as Temperature, MaxTokens, and TopP are pointer types. Setting Temperature: &zero will actually send "temperature":0 in the request body, whereas some Python wrappers treat zero values as unset and drop them. Additionally, you can override configuration for a single call via functions like model.WithTemperature, or even temporarily switch models with model.WithModel, without rebuilding the client.

Streaming calls via cm.Stream automatically add "stream":true and "stream_options":{"include_usage":true}. If a compatible service rejects the stream_options field, streaming calls will fail while Generate works normally. For reasoning models, reasoning_content in the response is mapped to out.ReasoningContent; inline <think> tags are preserved in out.Content, and the framework does not strip them.

No Automatic Retries; You Need Your Own Transport

This is the most surprising behavior. Tests show that for 401, 400, 429 (with Retry-After: 2), 500, or 503, Eino sends the request only once and never retries. Retry-After is ignored. For CLI tools this may be harmless, but for production services, a single 429 can cause a user-visible failure.

Error checking requires matching eino-ext's own *openai.APIError type, not the underlying go-openai branch's type, because the component performs a conversion.

var apiErr *openai.APIError
if errors.As(err, &apiErr) && apiErr.HTTPStatusCode == 429 {
    // handle rate limiting
}

A clean way to compensate for retries is to implement an http.RoundTripper that caches the request body and performs exponential backoff retries on 429 or 5xx. In testing, a mock service that returned two 429s before succeeding returned correctly after three requests (about two seconds apart); a permanent 503 returned an error after three retries; and 401 still made only one request. Note that http.Client.Timeout covers the total time of all retries, so leave ample margin.

http.RoundTripper is an interface in Go's standard library net/http package, defined in the Transport field of http.Client. Every HTTP request passes through the RoundTrip(*Request) (*Response, error) method before being sent and after receiving a response. A custom RoundTripper can intercept requests non-invasively to implement cross-cutting logic such as retries, logging, rate limiting, and adding auth headers. When implementing retries, note that http.Request.Body is a one-time-read io.ReadCloser that is consumed after the first send; before retrying, you must cache the original body and reconstruct an io.NopCloser. Also, retry only idempotent requests (such as GET) or clearly safe scenarios (such as LLM inference requests, which typically have no side effects), respect the Retry-After header for 429 responses, use exponential backoff with jitter for 5xx, and set a maximum retry count to avoid infinite loops.

ReAct Agent's MaxStep: Steps Are Not Equal to Tool Call Count

When creating a ReAct Agent, MaxStep is easy to misunderstand. Testing found that MaxStep counts the number of graph node executions, and each loop iteration includes one model node and one tool node. With the default MaxStep of 12, the agent actually made 7 model requests and 6 tool calls, then returned [GraphRunError] exceeds max steps. Setting MaxStep: 40 corresponds to 20 model requests and 20 tool calls. In other words, the number of tool calls is approximately MaxStep / 2.

Each request resends the full history; the last request of a 40-step run carried 39 messages, so context length grows quickly. You can check whether the limit is reached via errors.Is(err, compose.ErrExceedMaxSteps) and return partial results or a notice to the user.

Integration Checklist

• BaseURL must include /v1; Eino only appends /chat/completions.
• Pass APIKey explicitly; environment variables are not read.
• Set Timeout or provide a custom HTTPClient; there is no timeout by default.
• Implement retries yourself via a Transport, matching eino-ext's *openai.APIError for errors.
• Every streaming request includes stream_options.include_usage.
• Set MaxStep to roughly twice the desired number of tool rounds.

These details come from packet capture and verification of actual network requests. For Go developers using Eino to connect to self-hosted models or API gateways, they can help avoid common configuration pitfalls.

Background

Eino's compose package uses graphs to express execution flows: nodes represent computation units, and edges define dependencies and transitions between nodes. The ReAct Agent is internally compiled into a cyclic graph—the model node produces tool call intents, the tool node executes calls and fills in results, and conditional edges decide whether to return to the model node. MaxStep limits the cumulative number of node executions in this graph, not the repeat count of a single action. Therefore, a complete "think-act" loop typically consumes two steps (model node + tool node), which explains why tool call count is approximately MaxStep / 2. Understanding this graph execution model helps you set step limits correctly and identify which type of node caused the consumption when the limit is exceeded.