Storage SDK Agent: A Delivery Loop of Planning, Retrieval, Verification, and Repair

The Volcano Engine storage team is pushing large models from "code completion tools" to "deliverable engineering systems." The Agent they built for the object storage SDK scenario is not a script that generates code in one shot, but a pipeline that includes planning, retrieval, sandbox verification, and reflection-based repair. The value of this pipeline lies not in the number of roles, but in the fact that each stage can provide explainable, reviewable evidence for the gaps in the previous stage.

The Consistency Challenge of Multi-Language SDK Maintenance

Object storage provides SDKs for more than ten mainstream languages, including Java, Python, Go, C++, and Rust. Interfaces are continuously iterated, and file operations in file storage also involve complex business logic. Maintainers need to ensure that the same capability behaves consistently across different languages while conforming to each language's coding conventions. Once code and documentation diverge, the cost of understanding and troubleshooting ultimately falls on users.

The core difficulty here is the simultaneous existence of multiple consistencies. The engineering solution is to split the task into three dimensions: interface semantics, language implementation, and accompanying documentation, and to determine acceptance criteria for each, avoiding treating "the file has been generated" as delivery completion.

Why Template Tools Are Not Enough

The team investigated common interface generation tools, service-oriented definition languages, and internal templating tools based on interface descriptions. These tools excel at generating data structures and basic requests, but expose two pain points: fixed templates may lack the natural expression of the target language; and special business rules often require modifying global templates, with hard-to-control impact.

Templates still have advantages where structures are fixed. What this article supplements is business semantics and flexible adaptation capability. The tagging example of the object storage vector interface illustrates this well: the interface does not directly pass through the bucket name, but needs to combine the region, account identifier, and vector bucket name into a resource path parameter in a specific format. The template version directly places the bucket name into the request content, while the model implementation first constructs the resource identifier and then places it into the path of the tagging request. This shows that beyond the interface definition, there are business conventions that determine invocation behavior—one cannot just verify whether parameter names are complete, but must also check encoding positions and assembly rules.

The "service-oriented definition language" mentioned in the article usually refers to specifications such as OpenAPI and gRPC's Protocol Buffers used to describe interface contracts. They can uniformly describe request methods, parameter types, and return structures, and then use code generators to produce client skeletons for each language. Their limitation is that they describe the "shape" of the interface rather than its "semantics": business conventions such as how resource paths are concatenated and where authentication fields are placed are often outside the scope of the specification, so generated results are structurally correct but not necessarily behaviorally correct. In multi-language SDK scenarios, such specifications are suitable as a structural baseline, while language conventions and business rules still require additional knowledge sources and verification stages to fill in.

From Full Prompting to Separation of Planning and Execution

The architecture was not complete from the start. Initially, the team put the full documentation into the prompt, encountering window limits and hallucinations; then introduced a method of calling tools while reasoning to connect external knowledge, but complex tasks still lacked global arrangement; so they separated planning from execution, letting the planner first decompose tasks; further practice exposed that the code was not necessarily usable, so sandbox verification and reflection-based repair were added, and finally graph orchestration was used to organize these stages.

This evolution path shows: knowledge being accessible does not equal a complete process, and a complete process does not equal runnable output. Each added layer should clearly state which gap it fills.

The planner is responsible for understanding user intent and directing the task to a preset process, such as new interface generation or existing interface iteration, without writing specific code itself. During execution, it continuously monitors task status and context, receives feedback, and advances to the next stage. This division of labor prevents a single role from handling all details and losing the global picture. At the execution level, there are two choices: dynamically generating tasks is more flexible but may miss important nodes due to unstable model judgment; predefined workflows determine necessary steps in advance. The working mode of SDK delivery is relatively fixed, so the team chose predefined workflows—this is a scenario trade-off, not a form that all Agents should adopt.

Knowledge Base, Sandbox, and Static Evaluation Each Play Their Role

Infrastructure and memory are the common support for upper-layer execution. The knowledge base provides reliable context and high-quality example code for the model, and the sandbox provides an isolated environment to check whether the generated code can run and pass unit tests. The two solve different problems: knowledge helps decide "how it should be written," while the sandbox helps observe "how what is written actually behaves." Static evaluation supplements syntax, specification, and documentation requirements. It is important to distinguish these types of evidence—retrieving similar examples cannot replace running tests, and passing tests cannot prove documentation completeness.

The knowledge base handles scattered information such as design documents, historical code, and error correction records. The article presents a periodically running data pipeline: scripts scan specified sources, the model analyzes, extracts, and transforms raw documents, cleaned knowledge enters vector storage, and the upper layer retrieves on demand. This way business knowledge can continuously enter the execution context instead of being written into prompts only once.

In the retrieval chain, the team uses information extracted during the knowledge cleaning stage as metadata filtering conditions for vector retrieval, so that recall is not only based on semantic similarity but can also be filtered by business attributes. Two pieces of code that look similar may not be applicable to the same product or task; rash reuse will bring errors into new implementations. As for why they chose the object storage vector capability, the team mainly considered the access pattern of continuous writing, long-term storage, and on-demand retrieval; decoupling storage and compute and billing by storage volume and retrieval volume better fit this workload characteristic, and the native combination of file and vector capabilities also simplifies data management. This is a cost judgment based on specific workloads; when query frequency or latency targets differ, a new comparison is needed.

A sandbox is an isolated temporary runtime environment, usually configured with the target language's runtime, dependencies, and test framework, allowing generated code to compile, execute unit tests, or make simulated requests within it without affecting production systems. For an Agent, its value is turning "the code looks right" into "the code actually runs," outputting reviewable pass or fail results and logs. Note that test coverage in the sandbox is limited; passing tests only means the checked paths are correct and cannot replace static specification checks and documentation review. Conversely, static checks cannot detect path errors that only surface at runtime. Therefore, the three types of evidence answer different questions and should be used in parallel during acceptance.

Reflection Loop: Let Failure Evidence Drive Repair

The evaluation layer is responsible for static quality control, checking SDK code, example code in documentation, and documentation content specifications according to predefined rules, specifications, and static analysis tools. Such checks do not require actually running business requests and are suitable for early detection of low-cost fixable issues. But syntactically correct requests can still call wrong paths, and specification-complete documentation can still explain parameters incorrectly, so static evaluation is only one type of concrete evidence. Associating check results with file locations and violated requirements makes it easier to support precise repair in the next round.

When static checks find problems or dynamic verification such as unit tests fails, the reflection mechanism is activated: it reads failure logs, error messages, and evaluation reports, analyzes root causes and repair plans, and guides the executor's next round of modifications. The key is evidence transfer—if only "it failed" is given, the executor is prone to adjust by guessing; retaining the actual error and verification conditions provides a chance to fix the specific problem. After repair, relevant checks should be rerun to confirm the original failure disappears and observe whether new problems are introduced.

The article adopts two roles: executor and evaluator. The executor completes the task, and the evaluator reviews and scores according to predefined multi-dimensional standards and gives actionable improvement suggestions. The loop continues until the quality threshold is reached or the maximum number of iterations is hit. The 95-point threshold cited in the article is an example and does not mean every task should use this number. The value of independent roles lies in separation of duties, but if both rely on the same erroneous information, they may misjudge together. Therefore, evaluation standards should be connected as much as possible to actual code specifications and runtime results, and a loop limit should be retained.

The reflection mechanism here belongs to the "self-repair" Agent pattern: the model replans and modifies artifacts based on failure feedback rather than simply retrying generation. Its essential difference from one-shot generation is the introduction of external signals—compilation errors, test assertion failures, static check reports, etc.—as the basis for the next round of modification, constraining the iteration direction by evidence. There are two common failure modes: first, feedback information is too general, and the model repeatedly makes speculative modifications; second, the evaluator and executor share the same erroneous assumption, and the loop converges to a wrong result. Therefore, in practice, complete error logs are usually retained, the maximum number of loops is limited, and relevant checks are rerun after repair to confirm the original problem disappears and no regression is introduced.

Observability and Measurement: From Traces to Quality Gates

The SDK Agent has a long chain and many interactions, and scattered logs make it hard to reconstruct what a task experienced. The team built observability based on Log Service, collecting full-lifecycle information and linking it into execution traces, supporting long-context records, compatible with the OpenTelemetry protocol, and providing dashboard templates for stability monitoring, performance analysis, troubleshooting, and iteration. Recording a lot is only the foundation; it is also necessary to correlate the stages of the same task to see where a certain evaluation failure occurred and which artifact the subsequent repair targeted.

A task trace presented in the article brings three concrete findings: the first code generation failed because it did not pass unit tests, and the trace retained the error effect; the timeline shows that the code sandbox took most of the time, so the optimization direction is not only model generation speed; and it shows the process of triggering reflection after evaluation failure and completing code repair. These are observations from a single task and do not mean the bottleneck of all tasks is in the sandbox. A transferable method is to first distinguish waiting, execution, and verification along the trace, and then decide what to optimize.

To validate prompt adjustments and architectural upgrades, the team also built a standardized static automatic evaluation pipeline, set quality gates at stages such as code generation and documentation writing, and used datasets covering multiple scenarios to test results. This differs from the responsibility of the reflection loop in a single execution: reflection helps repair the current task, while unified evaluation observes whether a system change brings overall benefits.

OpenTelemetry is a cross-language observability data standard that unifies the collection and export formats of traces, metrics, and logs. In Agent scenarios, it can split a task into spans such as planning, retrieval, generation, verification, and reflection, recording the duration, input and output, and status of each stage, thereby reconstructing the complete execution trace. The advantage of adopting a standard protocol is that it can connect to existing monitoring backends and dashboards, avoiding building an isolated instrumentation system. However, trace data itself only describes what happened; to judge whether quality has improved, it still needs to be analyzed together with quality gate results and evaluation datasets to locate "slower" or "worse" to specific stages.

Effect Data and Interpretation Boundaries

The team reports that the quality scores of generated code and documentation are in the 90-95 range, and the average adoption rate of code requiring no manual modification is 93.84%. These two metrics should be distinguished: the score comes from the team's own evaluation system, and the adoption rate describes whether modifications were made during actual adoption; neither equals proof that no defects exist. The article does not provide the complete denominator, sample distribution, and long-term failure situation, so these numbers are suitable for illustrating the results the team observed and cannot directly predict the performance of another product.

In terms of delivery scale, the article reports that object storage has completed more than 100 interfaces and more than 180 accompanying documents, Log Service has completed more than 170 interfaces and fixed more than 80 existing interface issues, the manual intervention rate is below 10%, and more than 300 person-days have been saved in total. These results respectively include new additions, capability completion, and issue fixes, and cannot be simply added together as the same type of workload; saved person-days also depend on the estimation baseline of the original process.

The article also mentions a leading automaker project, claiming that with this capability the iteration cycle was compressed from monthly to weekly. This is a project case description lacking detailed work scope, baseline, and calculation method for external adjustments, and cannot be converted into a uniform multiple improvement. For engineering teams, it is more useful to simultaneously record delivery cycle, artifact quality, and reasons for manual intervention, and then combine traces, quality gates, and adoption records to understand whether the improvement comes from reducing repeated generation, shortening troubleshooting, or advancing multi-language and documentation more synchronously.

Transferable Implementation Suggestions

The main line of this delivery chain can be summarized as: the planner selects a preset process, the executor generates code and documentation with knowledge, static checks and the sandbox find problems, the evaluator gives specific feedback, the reflection mechanism drives repair, and traces and standardized datasets help the team continuously improve. For self-built SDK or code generation scenarios, one can start with an interface that has special parameter rules, run through the knowledge source, actual request, and acceptance conditions end to end, and then gradually add languages and task types, so that expansion is built on evidence that can be explained, reviewed, and compared.