Meta JiTTests: Using LLMs to Write "Designed-to-Fail" Tests to Intercept Risky Code

Meta has LLMs generate expected-to-fail tests before code merge, already intercepting 74 severe privacy incidents.
Meta has introduced large models into software testing with JiTTests, letting LLMs generate "expected-to-fail" test cases for code changes (diffs) about to be merged, exposing risks before code lands rather than testing after the fact. The system is in production and caught 74 privacy-related severe incidents in H1, with a 60% true-positive rate reported by engineers. This shows real value in high-risk scenarios, but false positives remain significant, making it better as an auxiliary signal than an absolute gate. Meta chose privacy defects as the entry point because they are costly and hidden. The overall approach reflects a pragmatic engineering paradigm of narrowing the scenario and using expected failure to gain an earlier exposure window.
The usual logic of software testing is to write code first and then add tests. Meta has flipped this process on its head: before risky code is merged, it has a large model first write a batch of tests that are "expected to fail." This system, called JiTTests (Just-in-Time Tests), has moved out of the experimental stage and into production.

From "Testing After the Fact" to "Defending Before Merge"
JiTTests has a clear purpose: it does not aim to generate tests that pass, but instead generates test cases expected to fail for code changes (diffs) that are about to be merged. If the code does introduce a problem, the tests will fail as expected, exposing the risk before merge; if the approach is correct, this batch of tests should pass, yet it ends in failure, thereby forming a barrier.
The key to this "reverse design" lies in the choice of timing. Traditional unit tests often lag behind code submission, and defects may only be discovered after they have already entered the main branch. JiTTests moves the checkpoint forward to before the code lands, letting the LLM judge from the semantics of the diff where problems are likely to occur and then write targeted tests.
Production Data: Privacy Defect Interception and True-Positive Rate
According to MarkTechPost, Meta has already put JiTTests into production use. In the first half of the year, this system caught 74 privacy-related severe incidents (SEVs) before code merge, and the true-positive rate reported by engineers reached 60%.
These two numbers deserve to be viewed separately. The 74 privacy SEVs show that the system does play a role in real high-risk scenarios; privacy issues are often subtle and severe in consequence, so being able to stop some of them before merge is valuable. The 60% true-positive rate points to the other side: a considerable proportion of alerts are still false positives, and engineers need to spend time distinguishing them. For a system that automatically generates tests, this ratio suggests it is better suited as an auxiliary signal than as an absolute gate.

True-Positive Rate is a core metric for evaluating the accuracy of defect detection systems. It refers to the proportion of all alerts reported by the system that actually correspond to real problems. For example, if the system issues 100 alerts and 60 of them are confirmed as real defects, the true-positive rate is 60%. Its complement is the false-positive rate (false alarm rate), i.e., the proportion of normal code mistakenly judged as problematic. In automated testing scenarios, a high true-positive rate means engineers spend less time sifting through noise. For LLM-generated tests, a 60% true-positive rate is a medium-to-high level—enough to prove the system has practical value, but not yet enough to serve as a hard gate blocking merges. It is better used as an auxiliary signal, combined with human judgment or a secondary screening mechanism.
SEV (Severity) is the internal incident grading system used by large tech companies, usually divided by scope of impact, degree of data loss, and user visibility. The higher the SEV level, the more serious the incident, often triggering cross-team emergency response and post-mortem review. The "privacy-related SEV" mentioned in the article specifically refers to severe incidents caused by data processing, permission control, or information leakage. Such incidents involve not only technical fixes but may also involve legal compliance and user trust, which is why Meta lists them as a priority interception target.
Why Privacy Defects
It is not surprising that Meta chose privacy as the entry point. Privacy violations are costly and subject to strict review, and many problems hide in data flows and permission boundaries, where manual review easily misses them. LLMs have an advantage in understanding code semantics and cross-file context, and can identify patterns such as "this change may cause sensitive data to flow somewhere it should not go," then generate targeted tests.
This also explains why JiTTests emphasizes "designed to fail": tests for privacy defects are hard to measure with conventional coverage metrics. Actively constructing cases that will fail is closer to real risk than pursuing a test pass rate.
Code diff is the format in version control systems that describes the changes between two commits, usually marking additions, deletions, and modifications at the line level. In the CI (continuous integration) process, automated checks are usually triggered based on the diff before code merge. JiTTests uses the diff as input, letting the LLM analyze the semantics of the change, judge which modifications may introduce risk, and then generate targeted test cases. Compared with analyzing the entire codebase, focusing on the diff greatly narrows the context the model needs to process, which both reduces inference cost and makes the generated tests closer to the actual impact scope of this change. This is also the technical prerequisite that enables JiTTests to be embedded in the pre-merge process and achieve "just-in-time" interception.
Trade-offs in Engineering Implementation
Integrating LLM-generated tests into the CI process requires solving several practical problems. False positives erode engineer trust; a 60% true-positive rate means about two out of every five alerts are noise, and over the long term, screening and feedback mechanisms are needed. The stability of generated tests is equally important—if the tests themselves are flaky, they become a new maintenance burden.
Meta's approach provides a reference paradigm: do not treat the LLM as a universal test generator, but focus on the narrow scenario of "high-risk diffs," using expected failure to gain an earlier exposure window. This narrowing-of-scope approach is more pragmatic than pursuing general coverage.
The CI (continuous integration) process is an engineering practice in which teams automatically build, test, and merge code changes into the main branch. A typical pipeline includes static checks, unit tests, integration tests, and other stages; failure at any stage blocks code merge. Integrating LLM-generated tests into CI means that every code submission triggers model inference and produces test cases, which imposes requirements on latency, cost, and stability: excessive inference time slows development, and unstable test results cause frequent false alarms in the pipeline. Therefore, JiTTests chooses to trigger only in the narrow scenario of "high-risk diffs" rather than running on all submissions, a trade-off between coverage and engineering feasibility.
Implications for AI Programming Tools
The significance of JiTTests goes beyond Meta internally. It demonstrates a differentiated use of LLMs in software engineering: not replacing humans in writing code, but acting as a "nitpicker" at the checkpoint of code merge. Compared with productivity tools such as code completion, this defensive use places higher demands on reliability and better reflects the model's depth of understanding of code semantics.
For teams introducing AI programming assistants, the JiTTests approach is worth borrowing: use the model in the highest-risk stages, accept a certain false-positive rate, and rely on process design rather than model perfection as the fallback.
One-line Summary
Meta has LLMs generate expected-to-fail tests before code merge, already intercepting 74 severe privacy incidents.
Paragraph Summary
Meta has introduced large models into the software testing process with the JiTTests system, letting LLMs generate "expected-to-fail" test cases for code changes (diffs) about to be merged, exposing risks before the code lands rather than testing after the fact. The system has entered production, catching 74 privacy-related severe incidents in the first half of the year, with engineers reporting a 60% true-positive rate. These figures show the system is indeed valuable in high-risk scenarios, but false positives still account for a considerable proportion, making it better suited as an auxiliary signal than an absolute gate. Meta chose privacy defects as the entry point because they are costly and highly hidden. The overall approach reflects a pragmatic engineering paradigm of narrowing the scenario and using expected failure to gain an earlier exposure window.
Related articles

Gemini 4 Neon Hands-On: Value King or Confident Catastrophe?
Gemini 4 Neon review: 1M-token context, four thinking levels, 201 files in one KingBench4 pass at one-tenth Fable 5.2's cost—but it hallucinates fake data, leaves 7 symlink escapes, and bricks keyboards.

Google Gemini 4 Neon Hands-On: 6 3D Works That Impress and Fail
Hands-on with Google Gemini 4 Neon on a 3060 laptop: 6 3D works tested. Sunshower Fishing Harbor and mechanical keyboard impress; soft-body physics and faces fall short.

Gemini 4 Neo Web Novel Writing Test: Faint AI Flavor, Textured Dialogue
Bilibili creator's first-look review of Gemini 4 Neo for Chinese web novel writing: testing a Three Kingdoms transmigration opening with faint AI flavor, emotionally textured dialogue, and strong crisis building, plus comparison of 4A/4H/4C models and full workflow demos.