Holdfast: Using Gemma to Explain Webhook Failures, With Code as the Safety Net

Title

Holdfast: Using Gemma to Explain Webhook Failures, With Code as the Safety Net

Meta Description

Holdfast is a webhook safety net: persist before acknowledging, verify signatures, park failures, and use Gemma 3 4B to generate failure explanations from both customer and developer perspectives—while retries, deduplication, and redelivery are all decided by Go code.

Article Content

Everyone who has built integrations has run into lost webhook messages. When Stripe or GitHub sends an event, if your app happens to be down, slow to respond, or a field has just been renamed, that message often vanishes silently. You usually find out two days later when a customer calls asking: why haven't three payments landed?

Holdfast, submitted by developer Tanay for the Hacktoberfest Open Source AI Challenge, aims to solve exactly this problem. He brought the plain but absolute principle of message queues to webhooks—a message is not considered acknowledged until it has been safely stored.

Core Design: Code Decides, the Model Explains

Holdfast sits between providers (Stripe, GitHub) and your application and does five things: persist before acknowledging, verify signatures and quarantine forged or replayed messages, retry patiently or park safely, explain what went wrong in two voices, and safely redeliver messages rather than sending duplicates.

The most notable design constraint of this project can be summed up in one sentence: the model never decides where a message goes. Retries, parking, deduplication, and redelivery are all plain, test-covered Go code. Gemma 3 4B is responsible for only two things: picking one of eight fixed causes (constrained by a JSON schema) and writing the explanation text. After it writes, the code checks it word by word.

Customer-facing explanations that contain terms like "HTTP", "payload", or "endpoint", or that promise actions no one will perform, are rejected and replaced with wording from built-in rules. Titles are always generated by code. This boundary is drawn clearly, and it serves as a reminder to any developer thinking about putting an LLM into a critical path: let the model do what it's good at, and don't let it make decisions.

JSON schema constraints refer to passing a structured output specification (such as field names, types, enum value ranges, and length limits) along with the request when calling a large language model, requiring the model to produce only JSON that conforms to that specification. It turns "free-text generation" into "constrained fill-in-the-blank", allowing downstream code to safely parse and validate the output. This is usually paired with verbatim post-validation: model output is first parsed against the schema, then business code checks the semantics (for example, whether banned words appear or whether it promises actions that weren't performed), and if any step fails, it falls back to text generated by the rule engine. This two-layer structure of "schema + code checks" is a common approach when putting an LLM into a critical path, because the model itself does not guarantee compliance—the real safety net must be provided by deterministic code.

A Bug That Taught the Author a Lesson

In the first version, in a "field renamed" scenario, Gemma got the cause right, but the customer summary said "the notification has been safely saved and is being reviewed." In reality, no one was reviewing anything. Worse, the developer explanation got the direction backwards—it said the app "now needs customer_id", when in fact Stripe had changed and the app had not.

The author's response was not to write a better prompt, but to move more work into code:

• Detect renames in code. A shape-diff walks both JSON bodies and finds that data.object.customer_id disappeared and data.object.customer.id appeared—that is a directional rename. The model receives "the sender now sends X instead of Y; the app has not changed" rather than a raw list of paths.
• Summaries must have a fixed structure: what happened to this payment (who, how much), the cause in plain language, and what happens next.
• The promise check rejects phrases like "is being reviewed", "will be resent", or "our team is...".
• Developer fix suggestions must point out the moved field, otherwise it falls back to the rule-generated version.

After the fix, the output for the same scenario was: "The sender now uses data.object.customer.id instead of data.object.customer_id. Please modify the app to accept data.object.customer.id." The direction was right, and no actions were invented.

Model Selection by Measurement, Not Gut Feeling

The author ran an actual comparison between Gemma 3 1B and 4B. In the rename scenario, 1B wrote promises like "the app will resend the parked event" and also told the sender to change—both errors were caught by the current checks and fell back to the rule-based answer. 4B took about 16 seconds on a laptop GPU and could accurately state the customer, the amount, and the specific field.

Deployed on Render's two-core CPU, the quality was the same, but one explanation took 92 seconds; after a round of optimization, two live measurements were 48 seconds and 83 seconds. This also explains a product-level design decision: the page instantly shows the built-in rule-based answer first, then replaces it once Gemma has written its version. The latency doesn't disappear—it is simply hidden.

Ollama is a tool for running open-weight models locally, providing an HTTP interface on port 11434 by default, and is often used to run models like Gemma and Llama on your own servers. llama.cpp is the inference engine commonly used underneath it, responsible for loading weights and managing KV cache and context memory. The LLAMA_ARG_* environment variables mentioned in the article are used to limit context length, batch size, and cache size, preventing the engine from allocating memory based on host resources rather than container quotas. In containerized deployments, if llama.cpp is not explicitly limited, it may read the host's total memory and core count, retain an overly large prompt cache, and eventually trigger an OOM. Understanding this layer helps assess the real cost and stability boundaries of "small local models" in the cloud.

Three Pitfalls Hit on Launch Day

On the first day of deployment, all Gemma calls failed with connection refused, pointing to holdfast-ollama:10000. Ollama was running fine on port 11434, but Render's service-to-service address used the default port. The rule engine automatically took over, and the site remained available the whole time—exactly the intended fallback path. Adding one line, PORT=11434, to render.yaml fixed it.

There were two more problems that same afternoon. One was out-of-memory: llama.cpp inside the container saw the entire host machine (60GB+ of memory, 16 cores) rather than the 4GB plan quota, so it retained a prompt cache that could grow to 8GB and a repacked copy of the weights. Three LLAMA_ARG_* settings brought it back under 4GB.

The second was runaway responses. Some diagnostics output hundreds of tokens and still couldn't close the JSON. The author reproduced it on a laptop CPU with exactly the same settings as Render: in 3 out of 5 runs, Gemma tried to end a string with curly quotes (”}); JSON treated them as ordinary characters, so the model couldn't stop and started repeating itself. Another trigger was Go writing JSON fields in alphabetical order (cause, fix, summary), which led the model to stuff the entire story into the developer explanation and leave the shop owner with a single line. The fixes included adding length limits to each field, having code truncate at stray curly quotes, checking that the summary ends like a sentence, and reordering fields to cause, summary, fix. After the fix, it went from 2 usable out of 5 to 4 usable out of 4, taking 16–40 seconds.

These three incidents share one thing in common: no matter what went wrong with Gemma, visitors still got the correct, plain answer from the rule engine. Gemma makes explanations better, but it was never the link that saves a payment.

Security Details Were Not Skipped

Live mode lets users enter the URL that Holdfast forwards to, which naturally introduces SSRF risk—without protection, anyone could point the target at holdfast-ollama:11434 or a cloud metadata address and then read the response on the event page. Holdfast checks the target address once when saving and again on every dial, so a hostname that only later resolves to a private IP will still be rejected. In addition, because providers deliver at least once, the redelivery plan sends each event only once in arrival order, and already-delivered events and those with failed signatures are never resent.

SSRF (Server-Side Request Forgery) refers to an attacker inducing a server to make requests to internal addresses, thereby accessing services that should not be exposed, such as the model service holdfast-ollama:11434 on the same private network or the cloud provider metadata address 169.254.169.254. Validating only the URL string submitted by the user is not enough, because a domain may resolve to a private IP only after it is saved, so the resolution result must be checked again at the time of the actual dial. DNS rebinding exploits exactly this time gap: the first resolution returns a public address and passes validation, then it is changed to resolve to an internal address. Double validation plus blocking private network ranges is the basic protection for forwarding services.

The First User and Its Honesty

The author's first user was his own small app, Tasklet, a little tool that turns GitHub issues into to-dos. The actual process: after connecting the repository, the GitHub test ping was acknowledged within 0.27 seconds; opening issue #1, a to-do appeared in Tasklet within 4 seconds; after switching Tasklet to maintenance mode (returning 503) and opening issue #2, Holdfast retried 5 times within 34 seconds (waiting 2, 4, 8, and 16 seconds respectively) and then parked it safely, with Gemma explaining the cause in 40 seconds; after turning off maintenance mode and clicking "redeliver all parked messages", the to-do for issue #2 appeared exactly once, and nothing was lost.

The author explicitly states that "no one paid", and that the first user was himself and his own app, so he makes no claim of sales. He also lists what isn't finished: Stripe Connect requires a platform account, which he doesn't have, so Stripe can only be configured manually—paste an address into Stripe's webhook settings and enter the signing secret into Holdfast, and the website says so honestly.

What Open Models Mean Here

Payment webhooks carry customer names, amounts, and addresses. Using open-weight models on the same private network means this data doesn't flow to any AI company, and freelancers can honestly tell clients that. A small shop might generate hundreds of events a day, and a small model on one server can explain every failure at a fixed cost. More practically, when Gemma promised a nonexistent human review, the author could see exactly what it received, change what the code told it, and constrain output with a JSON schema—that's engineering, not waiting for a vendor to update the model.

The model is just a configuration option (HF_MODEL). Whether to choose 1B or 4B is decided by measured data.

If you also maintain integrations for a few small clients, this project offers an approach worth borrowing: put the LLM in the explanation layer, keep decision-making in code, and prepare a usable fallback for the day the model fails.

One-line Summary

Holdfast uses an architecture of "persist before acknowledging" plus code-driven decisions and model-generated explanations to keep webhooks from losing messages.

Paragraph Summary

Holdfast is an entry in the Hacktoberfest Open Source AI Challenge that brings the message-queue principle "a message is not acknowledged until it has been safely stored" to webhooks. It sits between Stripe, GitHub, and the application, persisting before acknowledging, verifying signatures, retrying or parking, and safely redelivering. Its core constraint is that the model never decides where a message goes: retries, parking, deduplication, and redelivery are all handled by test-covered Go code, and Gemma 3 4B only selects from eight fixed causes and writes explanations, with output doubly constrained by a JSON schema and word-by-word code checks—text containing jargon or invented promises is replaced with fallback wording. The author also documents a field-rename bug, the measured comparison of 1B vs. 4B, three launch-day incidents involving ports, OOM, and runaway curly quotes along with their fixes, and security details such as double SSRF validation.