Jev on NASA Kepler Signals: 72.5% Accuracy with Zero Fine-Tuning

An AI Agent Test That Goes Off the Beaten Path

Most AI agent demos stick to familiar territory — task routing, tool calling, and the like. One developer on Reddit decided to throw a system called Jev into completely uncharted waters: astronomical data analysis, using it to process historical observation signals from NASA's Kepler Space Telescope.

The core question was straightforward: given 8,000+ Kepler Objects of Interest (KOIs), could Jev determine — based solely on measurement data describing each signal — which ones were confirmed planet candidates and which were false positives? The key experimental design detail: the true labels from NASA's archive were hidden until after each prediction was saved, ruling out any form of data leakage.

The Kepler Space Telescope is an exoplanet-hunting satellite launched by NASA in 2009. Its core method is Transit Photometry: when a planet passes in front of a star, it causes a periodic, slight dimming of the star's brightness (typically less than 1%). The telescope continuously monitored the light curves of roughly 150,000 stars to detect this signal. Kepler Objects of Interest (KOIs) are objects whose light curves show planet-like transit features, flagged as candidate targets. Due to cosmic noise, binary star systems, instrument errors, and other factors, many KOIs ultimately turn out to be false positives — only a relatively small fraction are confirmed as planets. This gives the KOI dataset an inherently class-imbalanced character, and each object comes with dozens of measured parameters such as orbital period, transit depth, and stellar radius ratio, making it an ideal structured-data scenario for testing AI reasoning.

The Results: What Does 72.5% Accuracy Actually Mean?

According to the data the author published, Jev's performance across all 8,054 decisions was:

• Overall accuracy: 72.5%, consistent with NASA archive labels
• The simple 3-rule baseline scored only 64.4% by comparison
• For confirmed planet identification, it successfully recovered 1,999 / 2,731 (roughly 73%)
• For false positive detection, it caught 81.7% of false positive samples

The most noteworthy point: the entire process involved no fine-tuning whatsoever, and no examples from this dataset were used. In other words, Jev completed this batch of astronomical signal classification under fully zero-shot conditions.

The 8-percentage-point lead over the baseline may not seem dramatic, but consider that baseline rules are typically heuristics hand-crafted by astronomers based on physical intuition. A general-purpose system surpassing them consistently with zero domain priors suggests its reasoning over structured measurement data has meaningful transfer value.

What the Confusion Matrix Reveals About Bias

From the complete setup, request logs, and confusion matrix the author published in a GitHub Gist, it's clear that Jev is noticeably stronger at false positive identification (81.7%) than at planet recovery (roughly 73%). This asymmetric performance means the system leans "conservative" in its judgments, tending to classify borderline signals as false positives.

For a task like exoplanet searching, this bias cuts both ways. On one hand, reducing false positives lowers the cost of expensive ground-based telescope follow-up verification. On the other hand, missing genuine planet candidates (roughly a 27% miss rate) means potentially overlooking valuable targets. Any real scientific deployment would require calibrating the tradeoff between recall and precision based on the research objective.

The Value — and Limits — of This Kind of Test

The significance of this experiment isn't really the 72.5% number itself, but what it validates more broadly: can a system designed for agent/routing scenarios transfer its reasoning ability to scientific data outside its training distribution?

Based on publicly available information, a few things about this test deserve a sober look:

• Single-source sample: this is a personal experiment published by one Reddit developer, with no third-party replication yet
• Weak baseline: the 3-rule baseline is not a mainstream machine learning classifier in astronomy (e.g., random forests typically achieve 90%+ on KOI classification), so the comparison mostly shows "better than naive rules" rather than "state-of-the-art performance"
• Limited technical disclosure: Jev's technical details are sparsely described in the original post — readers need to dig into the Gist documentation to understand how requests were constructed

The author has open-sourced the full experimental configuration, request content, and confusion matrix on Gist — a commendable level of reproducible transparency. For developers interested in exploring how far general-purpose reasoning systems can go in specialized scientific domains, this is a useful reference point.

In the field of astronomical planet candidate classification, mainstream machine learning methods have matured considerably. Random Forest classifiers tuned on KOI datasets routinely exceed 90% accuracy; deep learning methods (such as convolutional neural networks processing raw light curve time series) reach above 96% in some published papers. NASA's official Autovetter tool and the ExoFOP platform also integrate multiple statistical and ML pipelines for candidate vetting. Compared to these purpose-trained models, Jev's 72.5% sits at a notably lower level — but that's precisely the premise of this experiment: it tests whether a general-purpose reasoning system never optimized for astronomical tasks can perform zero-shot classification from measurement descriptions alone, not whether it can compete with domain-specific models on accuracy. The two are solving problems at different levels and shouldn't be directly compared.

Summary

Pushing an AI system into domains it "shouldn't be good at" often reveals more about its true capability boundaries than benchmark leaderboard performance. Jev's zero-fine-tuning test on NASA Kepler data — 72.5% accuracy and an 8-point lead over a simple baseline — demonstrates the potential for general reasoning to transfer to scientific data, while also exposing a weakness in planet recall. It's not a conclusion ready for scientific production use, but it provides an interesting, concrete case study for the ongoing discussion of how far general-purpose agents can really go.