Redaction Library Leaks Private Key: A Victory for Adversarial Testing

Title

Redaction Library Leaks Private Key: A Victory for Adversarial Testing

Meta Description

Redaction library llm-stream-guardrails leaked a PEM private key by trusting the wrong end marker. The author shares how adversarial property-based testing uncovered four vulnerabilities, all fixed in 0.7.9.

Article Content

A zero-dependency redaction library, llm-stream-guardrails, leaked a PEM private key in streaming output—even though its test suite was all green. After fixing the issue, the author publicly disclosed the vulnerability details (GHSA-777v-7mvx-mxgp) and shared four lessons from the experience.

The Vulnerability Itself: Trusting the Wrong End Marker

The library's job is to strip secrets and PII from LLM output streams before they reach the screen or logs. It identifies PEM private keys by the BEGIN PRIVATE KEY marker and masks the entire block, but it determines the end by checking for any labeled END line.

The problem lies here: when a private key's end marker is mistakenly written as END CERTIFICATE, the library thinks the region has safely closed, so it holds nothing and only masks the header. The base64-encoded private key body flows through in plaintext. And that leaked body can be reconstructed into a usable private key.

The author emphasizes that such incorrect markers are not rare—concatenated PEM bundles, truncated captures, or a model outputting a private key as a certificate can all produce this result.

PEM (Privacy-Enhanced Mail) is a common encoding format for keys and certificates. Its structure wraps a base64-encoded body between two boundary lines: -----BEGIN <label>----- and -----END <label>-----. The PEM "label" is not just decoration—it identifies the block type (e.g., PRIVATE KEY, CERTIFICATE, RSA PRIVATE KEY). Parsers typically rely on label matching to determine the complete extent of a block. If the end label does not match the start label, a strict parser will reject the block, but a lenient parser may consider the block closed merely because an END appears. The core of this vulnerability is that the parser mistakenly treats any END line as the end of the current block, rather than requiring an exact label match. In practice, concatenated PEMs, copy-paste errors, or label errors during model generation can all cause this mismatch. Therefore, secure parsing must bind the start and end labels, not rely on arbitrary END lines.

Why the Tests Didn't Catch It

The test suite asserted the behavior the code was designed for: correctly formatted key blocks get masked. And it did that. But no test asserted that leaks do not occur on malformed blocks. Because no one writes tests for inputs they didn't think of.

The author's solution is not to add handwritten cases, but to move from example-based testing to adversarially testing properties: for any input, can a value the policy considers should be masked reach the output? Then actively search for inputs that break this property.

Property-based testing differs from traditional example-based testing: the latter asserts outputs for specific inputs, while the former defines general properties (invariants) that apply to all inputs and automatically generates a large number of random or boundary inputs to try to falsify these properties. In redaction scenarios, a key property is "any value identified by policy as a secret must not appear in the output in any form." Example tests can only verify the legitimate inputs the designer anticipated and cannot cover malformed, truncated, or adversarial inputs; property-based testing actively searches for counterexamples through generators, uncovering leak paths the designer never thought of. Fuzzing is a common implementation of property-based testing, exploring the state space by continuously mutating inputs. The author mentions "about 320,000 fuzz test chunks," exactly using fuzzing to generate a large number of streaming chunks to verify the invariant, thereby exposing vulnerabilities missed by example tests.

Three Other Issues Found in the Same Round of Adversarial Testing

Once you start chasing properties rather than examples, more issues surface.

IBAN after a quote. A valid IBAN immediately following a JSON quote was masked when filtering the whole string, but was released in fragments during streaming—streaming output was inconsistent with whole-string output, and the library promised this would never happen.

Real token treated as a placeholder. The heuristic that skips documentation placeholders triggers whenever a value contains four consecutive identical characters, allowing a real credential that happened to contain four consecutive characters to pass through.

Unicode lookalikes. Full-width digits or invisible separators in values bypassed the ASCII detector.

Each one is a single-line property violation, and each one had a green test suite.

IBAN (International Bank Account Number) is a standard account identifier for cross-border bank transfers, typically containing a country code, check digits, and bank account details, up to 34 characters long. In structured data such as JSON, an IBAN often appears as a string value and may be wrapped in quotes. During streaming, data arrives in multiple chunks sequentially; if the redaction logic only runs on the complete string, chunk boundaries can split the IBAN, causing some fragments to leak unmasked. Unicode lookalike characters (e.g., full-width digit "１" vs. ASCII "1") are visually similar but have different code points; ASCII-based detectors ignore full-width digits or invisible separators (such as zero-width spaces), allowing real sensitive values containing these characters to bypass detection. Both types of issues are property violations caused by "streaming chunks" and "encoding variants," and are adversarial inputs that redaction libraries must defend against in real streaming environments.

Release and Reflection

All four vulnerabilities were fixed in version 0.7.9 (npm install llm-stream-guardrails@^0.7.9), each with a regression test that fails on the old version. The invariant that streaming output matches whole-string output now holds across about 320,000 fuzz test chunks, and the leaked private key body no longer reaches the output. The security advisory is public: GHSA-777v-7mvx-mxgp.

The author concludes: passing tests only proves the code does what you asked; it knows nothing about what you didn't ask. For a redaction tool—for anything whose job is to "hide"—that gap is the entire product.

One-line Summary

A redaction library leaked a private key by trusting any END marker; the author shares four security lessons.

Paragraph Summary

This article describes a vulnerability where a zero-dependency redaction library leaked a PEM private key in streaming output. The library holds a private key block based on the BEGIN PRIVATE KEY marker, but its end detection relies solely on any labeled END line. When the end marker was mistakenly written as END CERTIFICATE, the library assumed the block had closed, masked only the header, and the base64 private key body leaked in plaintext. The test suite was all green because it only tested correctly formatted key blocks and never asserted that malformed blocks do not leak. The author turned to adversarial property-based testing and found three more vulnerabilities: streaming vs. whole-string inconsistency causing an IBAN after a quote to leak, a placeholder heuristic misjudging a real token, and Unicode lookalikes bypassing the ASCII detector. All vulnerabilities were fixed in version 0.7.9 and verified with about 320,000 fuzz test chunks. The lesson: passing tests only proves the code does what you asked; it knows nothing about what you didn't ask.