Real-World Validation Notes on Self-Hosted Open-Source Models: Colab, RunPod, and the API Route

A developer documented a three-phase experiment in a public repository, self-hosting open-source large models with Google Colab and RunPod, ultimately moving their day-to-day code and document review work to the API route. The value of this article lies not in showing a "successful deployment," but in candidly distinguishing "what was validated" from "what was not"—this rigor in engineering record-keeping is itself more noteworthy than the conclusion.

Three-Phase Experiment: From Connectivity Test to Midway Abandonment

The starting point was a Qwen Gradio app running on Colab, after which the author rented GPUs on RunPod and attempted to connect an open-weight model to Open WebUI. The entire exploration was divided into three phases.

Phase 1: Minimal Connectivity Test

Using an RTX PRO 4000 24GB to run Qwen2.5-1.5B-Instruct, deployed via vLLM. The validation result was just one thing: a single reply displayed in Open WebUI. An interesting detail is that the author asked in Japanese, but the model returned the English "Connected successfully." The measurement window from start to first-stop confirmation was 319.8 seconds, with an estimated cost of about $0.051.

This phase did not evaluate answer quality, instruction-following ability, or inference speed—it was simply a "does it connect" test.

vLLM is a high-throughput open-source LLM inference and serving engine, built around PagedAttention memory management, capable of serving models through an OpenAI-compatible /v1/chat/completions interface. This means any frontend that supports the OpenAI API (such as Open WebUI) only needs to point its base URL at vLLM to connect, without writing a separate adapter layer for each model. vLLM also supports extended parameters like chat_template_kwargs, tool calling (which must be explicitly enabled), and quantized weight loading, but it is sensitive to the format of startup arguments—once a quoted JSON value on the command line is rewritten by the platform layer, the process exits immediately. This is precisely the technical root of the restart-loop failure described below.

Phase 2: One Round-Trip with a 27B Model

Switching to an A100 SXM 80GB to run Qwen3.8-27B BF16, again serving an OpenAI-compatible interface through vLLM. This time the model followed instructions and replied in Japanese with "接続確認できました" (connection confirmed). The window from before deployment to after stopping, showing $0/hr, was 875.1 seconds, estimated at about $0.44 based on $1.80/hour.

Similarly, quality, speed, long context, and image input were all unvalidated. The author specifically cautions that Phase 1 and Phase 2 are both connectivity tests, and "one reply" cannot be generalized into any conclusion about quality.

Phase 3: An Attempt with Zero Output

The plan was to run the GGUF version of GLM-5.3-Flash on an H200 SXM 141GB using the llama.cpp server. The container image pull completed (digest matched), but the total model weight size was 108.7GB, and during download the rate dropped from about 9GB/min all the way down to about 1.6GB/min. When container disk usage reached about 69GB, the author aborted the run according to a pre-set stop condition.

This phase produced zero model outputs, and no A/B evaluation was performed. The estimated cost was about $2.04.

GGUF is a single-file model format introduced by the llama.cpp ecosystem, packaging weights, quantization parameters, tokenizer, and metadata together, supporting multiple quantization levels from Q4 to Q8, making it convenient to run on consumer-grade or single-GPU machines with limited VRAM. Its companion, llama-server, is a lightweight HTTP service that pulls weights directly from Hugging Face via the -hf parameter at startup, so /health keeps returning 502 instead of listening normally until the weight download completes—this error semantics is completely different from vLLM's. The cost of GGUF is that the file size of very large models (108.7GB in this case) is constrained by network bandwidth and container disk, making the download phase itself the main risk point, whereas BF16 safetensors weights are typically used with vLLM.

Operational Failures and Model Capability Are Two Different Things

The most valuable part of this record is that it strictly separates operational-level issues from the model's own capabilities. In the final retrospective, the author explicitly states: platform or safety restrictions should not be interpreted as model incompetence.

Can't Get the GPU Back After Stopping

In an early trial of Phase 1, the model was ready but no reply was confirmed. After stopping, the same Pod could not reacquire a GPU. The record shows that RunPod releases the GPU on Stop; if other users occupy the GPU on the original host, the stopped Pod cannot be resumed, and the container disk is also wiped. The countermeasure for the final trial was: set up the UI first, then only update the connection target once the new Pod comes up.

Quotes Stripped, Causing a Restart Loop

In Phase 2, the author put an argument with a JSON value into the startup command. RunPod stripped the quotes, vLLM exited due to the argument error, and the Pod entered a restart loop. /health returning 404 or 502 could not distinguish a restart loop from slow loading; the recorded lesson is: read the system and container logs within the first minute. The fix was to remove the argument and edit the Pod; since weights had not started pulling yet, only time was lost.

First Request Rejected

Open WebUI 0.12.2 defaults Function Calling to Native, which sends tools enabled with tool_choice: "auto", and vLLM rejects the request because the server has no tool-calling settings. Switching to Legacy and regenerating resolves it.

Function Calling refers to the model outputting a structured call request in its reply, with the client executing an external function and then passing the result back to the model. In the OpenAI protocol, available functions are declared via the tools array, and tool_choice: "auto" means the model decides whether to call. The problem is: in Native mode, the frontend attaches these fields by default, and if the inference server has not enabled --enable-auto-tool-choice and specified a --tool-call-parser, it treats requests with tools as invalid and returns 400. Legacy mode instead injects tool descriptions at the prompt level and has the frontend parse the text, requiring nothing special of the server. The two are not a difference in capability but an operational failure caused by inconsistent protocol contracts.

Conflict Between Reasoning Models and Small max_tokens

Qwen3.8 "thinks" first, and 64 tokens may be exhausted before the answer appears. After sending chat_template_kwargs = {"enable_thinking": false} as a custom parameter, only the answer is returned.

Reasoning models generate an internal chain of reasoning before giving the final answer, and these tokens are also counted against the output budget. If max_tokens is set too small (64 in this case), the budget is exhausted during the reasoning phase, and the returned content may be empty or only half a thought process, manifesting as "the model isn't responding." There are three common countermeasures: increase max_tokens, disable thinking via enable_thinking: false in the chat template, or separate the reasoning content and the final answer into different fields. Note that disabling thinking changes model behavior, so such comparisons must be strictly distinguished from "insufficient model capability"—it is a configuration-level issue.

Pitfalls of Template Defaults

RunPod's "vLLM Latest" template defaults to a guessable API key and assumes a persistent volume's HF_HOME. The author replaced the former with a Secret reference and moved the latter to container disk.

Download Speed Becomes the Main Risk

In Phase 3, /health returned 502 the entire time, but this is not an error signal—llama-server only starts listening after the -hf pull completes. The author tracked progress via container disk usage: about 16GB at 23:13, about 35GB at 23:15 (about 9GB/min), about 51GB at 23:18 (about 5.5GB/min), about 69GB at 23:29 (about 1.6GB/min). Stop condition 3 required the model to be ready within 25 minutes of startup, and the projection at the time was that the pull would complete around 23:54 and be ready around 23:58. The author issued the abort command at 23:31:12.

The record does not determine why the download slowed down, and the author refuses to speculate.

Work Not Visible on the GPU Bill

The extensive non-GPU costs listed in the retrospective are the part of self-hosting that is often underestimated:

• Preparation without GPU: pinning model registry versions, checking the first 8MB of the GGUF header, checking inference image implementation, the evaluation set, and a scorer validated with a mock server (the first version counted HTTP errors as refusals, fixed during validation).
• Completing UI setup before GPU startup: Function Calling, max_tokens, disabling thinking, and model ID configured in advance.
• Debugging time: of the 875 seconds in Phase 2, about 4 seconds (about $0.12) were spent on the argument error.
• Proof of shutdown: checking $0/hr after stopping, $0 spend rate after termination, Pod count, and the endpoint returning 404 in Phase 3.
• Reconciliation: Phase 3 showed a balance drop of $3.17 against an estimate of $2.04; the gap of about $1.10 may be billing delay, but it is unconfirmed.

The retrospective records "about two weeks" as the author's actual calendar and preparation investment, and explicitly states it is not wall-clock time, GPU hours, or benchmark time.

API or Self-Hosting: Five Questions to Ask Yourself

The author's eventual day-to-day direction is the MiMo-V2.6-Pro API, used for reviewing routine code, documents, and final deliverables. But the record makes the state very clear: this API is an unvalidated operational candidate—no account, no key, no calls, no results, and no comparison against self-hosting on the same workload. The author's choice today is a personal decision, not a measurement result.

The following is the author's general decision framework (not empirical conclusions):

1. How do you compare costs? Pods are billed hourly, APIs are billed per token, and unit prices cannot be compared directly. You need to compare based on assumed usage and check the vendor's current pricing page when deciding.
2. What happens to the data? Whether code or documents can be sent to an external API must be resolved before sending. Check the vendor's official page for data retention and usage practices, along with applicable contracts, permissions, and regions. APIs are not inherently private, and self-hosting is not inherently secure.
3. How much will you use? Pod costs grow with time, API costs grow with tokens, and which is cheaper depends on usage volume.
4. How much operational time can you invest? The non-GPU work listed in the previous section is all your own job when self-hosting.
5. How much change exposure do you want to bear? API prices and behavior may change on the vendor's schedule; with self-hosting, you have to keep up with model and serving-stack updates yourself.

Scenarios Where Self-Hosting Is Still Worthwhile

The retrospective lists several situations where self-hosting may be suitable: reproducibility with pinned versions, control over weights, custom requirements, privacy architecture, and sustained high utilization. The author adds personal inferences including:

• Needing specific weights or specific quantization. The Phase 3 record notes that tracks A and B share quantization and size but differ in weights, enabling a fair comparison. Comparisons that fix quantization and size and vary only weights are hard to set up on a single-vendor API.
• Inputs cannot leave your environment. Self-hosting is a strong candidate, but you still need to validate how your own environment handles data.
• Wanting to adjust the serving stack itself. Server settings and startup arguments can only be changed when you run it yourself.
• High and stable usage rate. If utilization is consistently high, the cost picture may change, but this still needs to be measured separately.

The author concludes by emphasizing: RunPod was usable for them, and Phases 1 and 2 each got one reply; ending Phase 3 does not mean RunPod is unusable.

What This Record Teaches About Engineering Practice

The lessons recorded in the repository include: check how startup command arguments are split locally, and move arguments with JSON values out of the startup command; read container logs in the first minute; complete all UI setup before GPU startup; treat large weight pulls as a main risk, consider pre-pulling to a network volume using a cheap CPU Pod in the same region, use a parallel downloader, and verify SHA-256; split the readiness stop condition into pull and load phases; prepare standby GPUs and their quotas in advance; define proof of shutdown; inject API keys via Secret references; and report operational failures separately from model capability.

This record does not claim any cross-phase total cost, any quality or speed benchmark for any model, any result for any GLM model, the cause of the Phase 3 download slowdown, nor any measurement of the MiMo-V2.6-Pro API's behavior, quality, latency, or cost. In self-hosting content awash with "one-click deployment" narratives, this honest accounting of boundaries is actually more valuable as a reference.

One-line Summary

A developer's three-phase self-hosting experiment with open-source models yielded only two replies, ultimately pivoting to an unvalidated API.

Paragraph Summary

The article documents a developer's three-phase experiment self-hosting open-source large models with Google Colab and RunPod: Phase 1 and Phase 2 each obtained one reply via vLLM on an RTX PRO 4000 and an A100 respectively, while Phase 3 was voluntarily aborted during the download of 108.7GB of GLM GGUF weights on an H200 due to a sharp drop in download speed, with zero model output. The author deliberately separates operational failures (GPU release, quote stripping, Function Calling conflicts, reasoning token exhaustion) from model capability, and lists numerous hidden costs beyond the GPU bill, such as preparation, debugging, proof of shutdown, and reconciliation. Day-to-day review eventually moved to the MiMo-V2.6-Pro API, but that API's status is unvalidated, and the choice is a personal decision rather than a measurement result.