EmbeddingGemma 2 on Cloud Run: int8 CPU vs. NVIDIA L4 Showdown

When deploying an open-source text embedding model to the cloud, should you pick CPU or GPU? The author of the EmbeddingGemma 2 project ran a complete experiment on Google Cloud Run: the same model, three deployment shapes, with everything from request latency to cost per million documents calculated using real billing data.

Why It's Worth Running Your Own Embedding Model

Managed decision APIs are packaging up the work embedding models have done for years and selling it. TypeSafe's Jev launched on September 15, 2026, and can perform typed Q&A on a piece of text: pick one label from a list, place text on a scale, or return the probability that a statement holds. It's priced at $0.042 per million input tokens, accessible via waitlist, OpenRouter, and Vercel's AI Gateway.

Retrieving passages by query and tagging text by the closest description are exactly what embedding models excel at. EmbeddingGemma 2 is an open model that can run inside your own Google Cloud project, billed at Cloud Run rates, with all text staying within the project boundary. This article does the math on that path using Jev's pricing units, and measures its speed and deployment cost.

What EmbeddingGemma 2 Is

EmbeddingGemma 2 turns text into 768-dimensional vectors, bringing semantically similar text closer together in vector space. It ships with vision and audio encoders, but this project throws them away and loads only the text tower.

The vectors are Matryoshka vectors: the first 128, 256, or 512 of the 768 dimensions are themselves usable embeddings, so a search index can trade a bit of quality for smaller storage. The model also accepts prompt names—use SearchQuery for questions and Document for passages to be retrieved.

Matryoshka Representation Learning is a training technique that keeps a single embedding model usable at different truncated dimensions, named after Russian nesting dolls. During training, the model is required to make the first several dimensions (e.g., 128, 256, 512) each constitute a valid, independently usable embedding, rather than only the full 768 dimensions being meaningful. The benefit is that a search index can use lower dimensions to gain smaller storage and faster nearest-neighbor retrieval without retraining or maintaining multiple models. The cost is that the lower the truncation, the more noticeably retrieval quality degrades, so a trade-off must be made between recall and storage cost.

Three Services, One Dockerfile

Service	Weights	Instance	Billing
emb-cpu-repack	Text-only, int8 linear layers, int4 token table, 203 MiB	1 vCPU, 2 GiB	Per request
emb-cpu-original	Text-only, float32	1 vCPU, 4 GiB	Per request
emb-l4-original	Text-only, bf16	4 vCPU, 16 GiB, 1 L4	Per instance

The repack is built by repack/repack_text_w8a8_emb4.py in the repo: every linear layer of the text model is stored as int8 with one scale per output channel, and the 262,144-row token table uses int4. The int8 linear layers run on PyTorch's fbgemm kernels, which are CPU kernels, so the L4 service serves the original bf16 weights. 4 vCPU plus 16 GiB is the minimum instance spec required for Cloud Run to attach an L4.

Key Deployment Details

The CPU service uses --cpu-throttling to choose per-request billing—CPU is only allocated and billed while a request is in flight; --min-instances 0 lets the service scale to zero so idle time costs nothing; --concurrency 1 makes a single vCPU handle only one request at a time. The container can see all of the host's cores, while the Cloud Run quota grants only one vCPU, so TORCH_THREADS=1 is mandatory—otherwise PyTorch will spawn threads based on the host core count.

The L4 service in turn needs --no-gpu-zonal-redundancy, giving up failover capacity in a second availability zone and dropping the rate from $0.0002909 per L4-second to $0.0001867. There's also an easy trap to fall into: Cloud Run reserves paths ending in z, so the conventional /healthz never reaches the container and the frontend returns a 404 directly, which is why the health check route was changed to /health.

GPU quota is partitioned by region. The image was placed in us-central1, but the same deployment was rejected there—the region had no non-redundant L4 quota. The author switched to us-east4 to complete the deployment, and Cloud Run's CPU, memory, and L4 rates are identical across both locations in the billing catalog. Before deploying, check the quota with gcloud beta quotas info describe NvidiaL4GpuAllocNoZonalRedundancyPerProjectRegion.

Per-request billing and per-instance billing are two fundamentally different cost models on Cloud Run. Per-request billing (with --cpu-throttling) allocates and settles CPU only during request processing, incurring almost no compute cost while the instance is idle, making it suitable for sparse, bursty traffic. Per-instance billing charges for every second from instance startup to shutdown, regardless of whether requests come in, but buys continuous compute and more stable latency. GPU instances currently only support per-instance billing, which is the root cause of the L4 option's costs climbing sharply at low busy ratios.

Accuracy Verification: How Much Does int8 Quantization Cost You

This is the most worthwhile part of the whole project to examine closely. All three services have to pass the same set of nine checks: health status, vector length and unit norm at four Matryoshka dimensions, the 128-dimensional vector must be the first 128 numbers of the 768-dimensional one, invalid input is rejected, and relevant passages rank ahead of two irrelevant passages. All pass.

Per-text cosine similarity between the repack and the float32 original ranges from 0.9888 to 0.9978; the L4's bf16 is nearly identical to the original float32, with cosine reaching 0.9999 to 1.0000. But a cosine near 0.99 only says the vector directions agree—the real test is retrieval. The author evaluated on SciFact (300 queries against 5,183 scientific abstracts, 768 dimensions):

Weights	nDCG@10	recall@10	Top 10 shared with float32
float32 original	0.854	0.944	100%
int4 token table only	0.850	0.939	92.7%
repack (int8 + int4)	0.836	0.926	85.5%

The repack retains 97.9% of the original's nDCG@10. The accuracy loss from quantization is clear and controllable, which is the precondition for daring to use it in production retrieval.

nDCG@10 (normalized discounted cumulative gain) is a standard metric for measuring retrieval ranking quality: it gives higher weight to results ranked near the top and discounts relevant documents that appear lower down, finally normalizing to a range between 0 and 1. recall@10, by contrast, only checks whether the top 10 results contain the relevant document, regardless of order. Together they characterize retrieval performance fairly comprehensively—nDCG reflects ranking quality, and recall reflects whether anything was missed. In this article, the repack retains 97.9% of the original's nDCG@10, indicating little loss in ranking quality, but the shared top 10 is only 85.5%, indicating that the ranking of some specific documents has shifted, with limited impact on retrieval where ranking is the ultimate goal.

Performance and Cost Measurements

The author ran three benchmarks per variant, each with a new revision (i.e., a new instance), recording the server-reported latency. The median results are as follows:

Variant	Query latency	Docs/sec	Cost per million queries	Cost per million documents
cpu-original	251 ms	0.9	$8.93	$39.05
cpu-repack	121 ms	1.1	$3.91	$25.87
l4-original	56 ms	333	$16.27	$0.87

On the same 1 vCPU, the repack's query latency is only half of float32's, its document throughput is 1.29× higher, and it uses half the memory. The L4's document throughput is 297× that of the repack.

Where the Cost Curves Cross

A per-request CPU service costs nothing when idle. The L4 is billed per instance, so every second an instance is online burns money whether busy or not—$1.0465 per hour, roughly ten times a 1 vCPU instance. So the real comparison comes down to two numbers: how much faster the L4 is, and how much of the time it's busy.

Cost per million documents as a function of busy ratio:

Busy ratio	L4 per million docs	cpu-repack per million docs
100%	$0.87	$25.87
50%	$1.75	$25.87
10%	$8.73	$25.87
1%	$87.28	$25.87

The break-even point is 3.4%: as long as the L4 is doing work for more than 3.4% of its online time, it's cheaper per document than the repack.

Converted to the per-million-input-tokens unit Jev uses (documents of 141 tokens): cpu-original is $0.2770, cpu-repack is $0.1835, and the L4 at full load is only $0.0062, or $0.0619 at 10% busy, while Jev's list price is $0.0420. In other words, once the L4 is busy for more than 14.7% of its online time it matches Jev's unit price, and at full load Jev's price is 6.8× higher. The CPU repack, meanwhile, is 4.4× Jev's unit price—but it costs nothing when idle.

Cold Starts: The CPU's Achilles' Heel

--min-instances 0 means the first request after idling has to spin up an instance on the spot. The measured numbers are pretty stark: the L4's first request took 17.3 seconds total (4.8 seconds of model loading); the float32 CPU service 48 seconds (15.5 seconds of model loading); the repack 58 to 84 seconds.

Model loading accounts for only a small part of that—the bulk is instance scheduling, image pulling, and container startup. The L4 image is the largest (4092 MB compressed), yet its cold start is the fastest, which suggests the CPU service is slow elsewhere. For scenarios where this latency is unacceptable, you can use --min-instances 1 to keep it warm, at the cost of idle instances being billed as usual.

Which One to Choose

The conclusion depends on the shape of the workload.

For search queries arriving sporadically during the day, choose CPU repack. It has the lowest cost per query ($3.91/million), costs nothing between requests, doesn't need GPU quota, and retains 97.9% of the original's retrieval score.

For batch-indexing a corpus, choose L4. It embeds 297× faster, costs only $0.87 per million documents when busy, and beats the repack as long as it's working for more than 3.4% of its online time. Just let it scale to zero when done.

The float32 CPU service, meanwhile, is the reference baseline against the other two and not a production choice in itself.

A Few Easily Overlooked Engineering Details

bf16 vectors must be normalized in float32. Sentence Transformers' normalize_embeddings=True normalizes at the model's own precision. On the L4 the model runs bf16, so vectors that should be unit length have norms of only 0.9975 at 128 dimensions and 1.0015 at 512 dimensions, and using the dot product as cosine carries the same error. The author's fix was to take the raw vectors and renormalize them in float32; after the change, all three services print norms of 1.0000.

The repack's vectors drift with the host. Its int8 kernels depend on the host CPU, and the same check gives results from 0.986 to 0.998 on different Cloud Run hosts. Harmless for retrieval, but worth knowing.

The L4 was not tested for concurrency. The author explicitly states that the benchmark was single-request serial, with one document length (141 tokens), three runs per variant, and cold start measured a single time. The comparison with Jev uses only its list price, and an accuracy comparison between the two on the same batch of data is left for later.

The value of this project isn't in giving an absolute "CPU good or GPU good," but in laying out the variables of the cost model: billing mode (per request vs. per instance), busy ratio, the accuracy loss from quantization, and the hidden cost of cold starts. Plug these numbers in when choosing, and the answer will emerge on its own.

The difference between bf16 and float32 in vector normalization stems from the accumulation of floating-point rounding. Normalization requires dividing a vector by its L2 norm so its length is 1; but bf16 has only an 8-bit mantissa, far fewer than float32's 23 bits, so the error when summing squares is larger, causing the computed norm to deviate from 1 (0.9975 at 128 dimensions and 1.0015 at 512 dimensions in the article). Using the bf16 vector's dot product directly as cosine similarity would carry this systematic bias. The solution is to take the raw bf16 vector, convert it to float32, and renormalize, bringing the norm back to 1.0000 and thus ensuring similarity is comparable across services and precisions.

One-line Summary

The same embedding model deployed on Cloud Run as int8 CPU versus NVIDIA L4, where the cost-latency trade-off depends on workload shape.

Paragraph Summary

The EmbeddingGemma 2 project compared three deployments of the same model on Google Cloud Run: an int8-quantized CPU service billed per request, a float32 CPU baseline, and an NVIDIA L4 billed per instance. The int8+int4 quantization retains 97.9% of the original's SciFact nDCG@10, with per-text cosine of 0.9888-0.9978. The L4's document throughput is 297× the CPU repack's, costing only $0.87 per million documents at full load, and it's cheaper than the CPU as long as it's busy more than 3.4% of its online time. The CPU repack costs $3.91 per million queries and nothing when idle, at the cost of 58-84 second cold starts. The conclusion is to choose by workload: CPU repack for sporadic daytime queries, L4 for batch indexing.