LLM API

API billing typically distinguishes between input tokens and output tokens; in multi-turn conversations, the outputs from earlier turns become inputs in later turns and are billed again.

The request-response cycle handled by a traditional API gateway usually completes within 50 to 200 milliseconds, whereas an LLM API often needs to keep the connection open for 30 to 120 seconds.

LLM APIs are billed by token, with approximately 1–2 tokens per English word and approximately 1–2 tokens per Chinese character.

Since 2023, API prices for mainstream LLMs have fallen by more than 90%

调用第三方LLM API的AI产品数据流向通常经历「用户设备→产品服务器→LLM服务商」三个节点，每个节点都存在独立的安全风险面

For LLM APIs, retryable errors include 429, 5xx and timeouts, while non-retryable errors include 400 parameter errors and 401 authentication failures.

Free-tier LLM API limits are usually distributed across four levels: request quota, response size, model capability and usage tracking.

Learning Agentic AI should start with directly using LLM APIs, mastering parameter controls such as temperature and max tokens, structured prompt engineering, and function calling mechanisms.

When calling an LLM API, the marginal cost grows linearly with the number of requests, whereas the marginal cost per execution of locally compiled decision logic is close to zero, with the main cost shifting to one-time compilation and validation investment.

Prompt caching charges only one-tenth the price for input tokens that have already been processed, making it one of the highest-leverage cost optimizations on modern LLM APIs.

Calling LLM APIs in production requires adding retries and clear failure paths, for example exponential backoff retries for HTTP 429 status codes.

The author's canary costs about $0.4 per month in LLM calls, with 720 runs, each processing just 10 tokens with one cheapest model.

The LLM API availability probe sends a 10-token prompt "reply OK" to the cheapest model and requires the response to arrive within 30 seconds, capturing overall API outages, expired keys, and slow degradation.

The UplinkSwitch aggregation platform extracted 66 chat models from 11 developers with official list prices from the real-time catalog of its own API gateway; all 66 models are below official list prices on both input and output sides, with the largest price difference reaching 95%.

LLM APIs are priced separately for input and output tokens, and the per-token price does not change with the temperature and seed parameters.

LLM APIs rate-limit on two independent dimensions, requests per minute and tokens per minute; even with monthly budget to spare, an endpoint can still return 429 during traffic peaks.

The correct way to track AI agent costs is: inject a correlation ID at the API gateway, propagate these IDs through the orchestration framework, and intercept all LLM API calls, emitting standard OpenTelemetry spans with token metadata.

Traditional cloud FinOps tools focus on infrastructure utilization (CPU/memory) and do not parse token metrics from API payloads, causing API-based AI consumption to become a blind spot.

In LLM repair loops, the recommended typical circuit breaker parameters are: 5 consecutive failures trigger the break, a 30-60 second cooldown, and rate-limited probing during the half-open phase.

Pressing all LLM traffic for critical business onto a single model API makes annual cumulative downtime reaching the hour level almost the norm rather than the exception.