Byte-Level Auditing: How Tawfiki AI Independently Judges Closed-Source Governance Engines

Most AI governance frameworks share a hidden flaw: they let the system under test report its own behavior, and then grade those reports. The system says it passed, and the test framework believes it—more a confirmation ritual than a real audit. The Tawfiki Sovereign Security Audit Harness, which sparked discussion on Reddit, tries to sidestep this problem with a set of hard constraints: the engine takes no part in judging its own verdict. Bytes go in, bytes come out, and the determination is made independently by pre-registered deterministic logic.

Why Closed-Source Governance Engines Need "Distrustful" Auditing

When the subject under audit is a closed-source AI governance engine, the auditor faces a threefold dilemma: the source can't be seen, the system's self-reports can't be trusted, and conventional test suites happen to hand the pass/fail decision right back to the very system that needs to be verified.

Tawfiki's approach strips adjudication power away from the engine entirely. According to its published methodology, the entire audit framework (B001–B005) is open source, any party can run it under witness, and the engine itself never participates in scoring. This means that even if the engine's internal logic is completely invisible, the audit conclusion can still stand independently.

Preregistration originates from the methodology of experimental psychology and clinical medicine: publish hypotheses, metrics, and thresholds before the study begins, to prevent adjusting standards after seeing the data (i.e., "moving the goalposts"). In AI auditing, preregistration hash-commits the decision rules, classifier parameters, and pass thresholds before execution, so audit results can only be judged against established criteria and the scoring logic cannot be modified based on observed outputs. The irreversibility of hash commitment ensures that once rules are submitted they cannot be quietly changed, while the witness holding the original hash provides after-the-fact verifiability. This mechanism turns auditing from "after-the-fact explanation" into "ex-ante commitment," and is the foundation for reproducible, manipulation-resistant audits.

How Output Bytes Are Scored

If the engine doesn't participate in scoring, who scores the output? Tawfiki distinguishes two types of scoring mechanisms.

Binary suites (B001, B002) derive their verdicts from input bytes and output bytes, compared against pre-registered external rule sets using deterministic comparators, without introducing a second LLM or semantic judge. Behavioral suites (B003, B004, B005, DOS) use committed classifiers—triplet similarity and keyword-set comparators—whose parameters are recorded as hashes in the metadata before the run begins.

The key point is that any classifier output must have a committed hash record to be accepted. The scoring problem is not offloaded to another model; classifier parameters are fixed, committed, and verifiable before execution.

Deterministic Comparator refers to decision logic that does not rely on probabilistic models or semantic judgment and always produces the same output for the same input. Unlike introducing a second LLM as a judge, deterministic comparators are typically based on exact matching, regular expressions, rule sets, or fixed algorithms, and their behavior is fully determined by input bytes and pre-registered rules, unaffected by model randomness, prompt phrasing, or context drift. In an audit context, this means the adjudication process itself can be independently reproduced: anyone holding the same inputs, rules, and comparator code can reach the same conclusion, thereby eliminating the recursive trust problem of "the scorer itself needs to be scored."

Measured Results of the Five Behavioral Suites

Each suite runs against thresholds pre-registered in prereg.json, committed before any test began.

B001 - AI Security Audit: covers five attack categories—prompt injection, role confusion, context override, system prompt extraction, and policy bypass—with 1,000 adversarial prompts in total, judged against a pre-registered external rule set. Result: 1,000/1,000 all blocked.

B002 - Novel Adversarial Resistance Test (NART): generates 600 novel attack vectors live on each run and hash-commits them before execution to counter corpus memorization. Generator-schema memorization is listed as a disclosed residual risk, and the primary defense is the live witness probe protocol described below. Result: 600/600 all blocked.

B003 - Swarm Consistency: governance seeds are tested across independent agent instances. Two-phase judgment—any violation in any instance fails that seed; only violation-free swarms proceed to the cross-instance divergence gate. Result: 15/15 seeds passed, 0% propagation rate.

B004 - State Persistence: across multi-turn conversation sequences, governance posture drift is measured by posture signature comparison rather than raw text similarity—the latter measures prompt-following ability, not governance ability. Result: 25/25 turns, 0% drift rate.

B005 - Axiomatic Intelligence: each category of contradiction is tested across multiple instances. A combination of refusal plus answer counts as a violation; pattern-matched workarounds cannot pass. Result: 8/8, zero false negatives.

Two more suites complete the record: SVA (G≠V verification under adversarial pressure, passed with signed receipt) and DOS (semantic re-review, 84/100, re-review pending, with the sub-100 score honestly disclosed).

Three Design Decisions Worth Scrutinizing

The commitment mechanism for adjudication logic. All rule sets, classifier weights, comparators, thresholds, and epsilon are hash-committed to prereg.json before the run. The goalposts cannot be moved after seeing results. A sealed chain that binds only outputs but not adjudication logic can be attacked by the operator—this framework closes that path.

Controls that score the scorer. Before any suite runs, the framework audits itself first: bytes with known violations must receive the highest detection score, known-safe bytes must score zero, corrupted bytes must be caught, and an empty-pattern engine must produce nothing. If any control fails, the entire run is voided.

VOID as a first-class outcome. Control failures, chain breaks, or sandbox escapes are not counted as FAIL; instead, the entire run is voided. FAIL means the engine genuinely failed at governance; VOID means the audit execution itself was invalid. Conflating the two would let a bad audit masquerade as an engine failure.

Sealed Chain and Execution Chain are the technical basis for audit integrity. The sealed chain binds each execution's inputs, outputs, verdicts, and metadata in sequence via a hash chain or signed log, so any tampering with historical records causes chain verification to fail. The execution chain records the complete event sequence within a single run, including Q&A pairs from live probes and classifier invocation records. Witnesses use --verify mode to replay the hash chain and confirm the records have not been tampered with, without any engine involvement. This design shifts trust from "the engine's self-report" to "cryptographically verifiable record integrity," making audit conclusions independent of the subject under audit.

The Witness Protocol Under a Proprietary Model

Because the engine binary is proprietary, third-party verification takes the form of witnessed live execution: the witness receives harness.py and the prereg template before the session; the witness provides or co-generates the audit salt at the ceremony; the corpus is deterministically generated from the salt in front of the witness; during execution, the witness supplies arbitrary live probe prompts—this is the primary anti-replay mechanism, because a replay rig cannot answer entirely novel arbitrary prompts it has never seen. The engine answers the live probes, and the responses are sealed into the execution chain. The witness takes away run.jsonl, prereg.json, harness.py, and the corpus, and independently verifies the chain in --verify mode—all without the engine.

Clarifying the 2,000x Efficiency Claim

This figure comes from a separate, independent test framework—the Tawfiki Efficiency Engine Audit (TEE). The 2,000x is measured in FLOPs (floating-point operations consumed per governed inference output), not hardware FLOPS (throughput), and reflects architectural workload reduction across four domains (legal, financial, medical, and 1MB-scale conversation), sealed with SHA-256. The per-domain range is 4,901x to 17,689x, with a verification bound of 2,000x. Air-gapped environment, seven runs, median timing.

The efficiency framework uses incompressible controls (Shannon entropy lower bound) for fraud detection: if the engine can compress high-entropy random input, that run is voided. By Shannon's source coding theorem, real compression of truly random input is impossible, so passing this control is treated as fraud. A003 memory efficiency remains pending—data was collected during A002, but dedicated per-run repeated measurement is needed to reach peer-review-grade sealing, and this is explicitly listed as future work rather than a current result.

Methodology Public, Engine Closed

The framework's stance is clear: methods public, engine proprietary, witnessed runs available on request. The publisher actively invites outside scrutiny—if the audit framework has design flaws, public exposure is itself the way to find them. For closed-source AI governance, a field where trust is hard to establish, taking adjudication power out of the subject's hands at least gives "passing" a verifiable meaning.

One-line Summary

Tawfiki's open-source audit framework strips adjudication power from closed-source governance engines, using pre-registered deterministic rules and witnessed execution to achieve verifiable audits.

Paragraph Summary

Addressing the audit dilemma of closed-source AI governance engines, the Tawfiki Sovereign Security Audit Harness proposes a framework in which the engine takes no part in its own adjudication. Core mechanisms include: all rule sets, classifier parameters, and thresholds are hash-committed to prereg.json before the run; binary suites use deterministic comparators against external rule sets, while behavioral suites use hash-committed classifiers; controls audit the framework itself first, and any failure voids the entire run (VOID), strictly distinguished from engine governance failure (FAIL). Five behavioral suites cover adversarial prompts, novel attacks, swarm consistency, state persistence, and contradiction-category tests, reporting all-blocked or all-passed results. Proprietary engines are verified through witnessed live execution, with witnesses supplying arbitrary live probe prompts as an anti-replay mechanism. Efficiency claims come from an independent framework measured in FLOPs, and memory efficiency results remain pending.