SLO

Inference latency is typically decomposed into two core metrics: time to first token (TTFT) and subsequent token generation throughput.

SRE (Site Reliability Engineering) is an engineering practice proposed and popularized by Google, whose core idea is to solve operational problems using software engineering methods and to quantify system reliability as SLOs for continuous measurement.

SLI is a specific metric for measuring service health, SLO is an internally set target value, and SLA is an external commercial commitment that entails financial penalties when violated.

The industry typically uses SLO (Service Level Objective) and Error Budget to quantify impacts such as performance degradation, for example using '99.9% of requests return within 500ms' as a target.

An SLI is the actual measured value, an SLO is the internal target the team sets for the measured value, and the error budget is the room for failure allowed by 100% minus the SLO; the three form a closed loop.

Five steps for setting reliability goals for AI features: choose only one quality SLI per feature, version the eval set, set the SLO from a week of baseline data, tag AI-written PRs, and define explicit failure blocking rules.

For AI systems, attaching the SLO only to availability means model quality degradation will never exhaust the error budget, and thus will never trigger the brake.

SLOs (Service Level Objectives) and error budgets: mature platforms usually set quantitative targets for availability and use them to drive capacity planning and failure drills.

P99 latency is generally a key metric for assessing the true experience of production systems; Google's SRE framework calls it tail tolerance and recommends explicitly specifying P95 or P99 latency upper bounds in SLOs.

Existing P/D disaggregation systems typically employ fixed prefill/decode worker ratios combined with request routing for load distribution, which leads to latency SLO violations when workload demand ratios fluctuate.

SLO cannot be obtained once through administrative procedures, but must be continuously maintained through ongoing transparent information disclosure, community engagement, and responsiveness to local concerns.

Amazon's move to abandon NDAs is essentially a proactive management of the risk of SLO deficiency.

A Service Level Objective (SLO) is a quantified commitment to service quality, which in LLM inference scenarios typically encompasses both latency and availability dimensions.

The core task of benchmarking is to find the maximum sustainable throughput under SLO constraints, rather than the system's absolute peak throughput.

Goodput refers to the proportion or throughput of requests successfully completed under SLO constraints, treating requests that exceed the latency budget as failures and excluding them from effective output.

Agent Reliability introduces SLO and error-budget semantics, and provides SLO assertions that can be used in tests/CI.

NVIDIA recommends a top-down capacity planning approach: define SLOs, run benchmark tests, and calculate the total number of GPUs needed.

Error budgets are a management tool between SLOs and SLAs that transforms reliability from an absolute target into a flexibly allocatable resource.

The core decision logic for building a multimodal inference service is to first quantify the proportion of visual encoding in the overall inference overhead, then assess disaggregation benefits in combination with SLO requirements and load diversity.

The applicability of EPD disaggregation depends on three factors: the proportion of visual encoding overhead, the diversity of multimodal inputs, and the Service Level Objective (SLO).