Should AI Tutors Give Answers Directly? I Ran a SQL Experiment
A 6-person experiment found that AI tutors giving hints before answers make learners persist in fixing errors after the tutor disappears, but can't prove causation.
Developer Vikrant Negi ran a 6-person controlled experiment comparing two AI tutor strategies: Arm A forces attempts before hints, Arm B gives answers directly. In an unassisted test a week later, Arm A got all 36 problems correct, while Arm B only got 25. But both groups were identical on first-attempt correctness—the entire gap came from error-fixing behavior. The experiment had multiple flaws including lack of supervision and problem ordering, making causal claims impossible. Nevertheless, the researcher pre-registered ten check criteria and honestly reported two failures, making this 'no victory claimed' project a methodological exemplar for AI education product design.
The AI Tutor's Dilemma: Helping or Hurting?
When a learner gets stuck and clicks the "Help" button, what should an AI tutor give them?
Most tools just hand over the answer. ChatGPT does this. Copilot does this. It feels efficient — the learner can move on.
But a large-scale study (Bastani et al., PNAS 2025) sounds an alarm for everyone building these tools: students who practiced math with AI giving direct answers actually performed worse on later exams than students with no AI assistance at all. Once the AI was taken away, they did even worse.
Developer Vikrant Negi, in his capstone project for the 100x Engineers AI course, decided to answer a more specific question through experimentation: If a tutor forces you to try first and gives hints before answers, will you do better a week later when the tutor is gone?
Experiment Design: Two "Help" Strategies Face Off
Negi built a single-page SQL tutor app. Users read a problem described in natural language, write a SQL query, and click Run. The page tells you if you're correct. When you're stuck, you click "Help."
The Help button's behavior is the entire core of the experiment. He built two versions — called "arms" in experimental terminology:
| When you press Help | Arm A: Try First | Arm B: Direct Answer |
|---|---|---|
| You haven't tried anything yet | Refuses: "Run it once first" | Shows the answer |
| After one attempt | LLM-generated hint | Shows the answer |
| After two more attempts | Shows the answer | Shows the answer |
Arm B is the baseline — it behaves like most AI tools today. Everything else is identical between the two versions: same page, same 16 practice problems, same checking mechanism. The only difference is one configuration parameter.
A judge at Demo Day called this experiment an "arm wrestling match," and the name stuck.
About the term "arm": In a controlled experiment, an arm refers to one of the experimental conditions or treatment groups that participants are assigned to. Each arm corresponds to an independent intervention, with all arms held constant on other conditions — only the variable under study differs.
For example, in a clinical trial, the new drug group, placebo group, and no-treatment control group are three different arms. Differences between arms must be strictly confined to the target variable, otherwise results cannot be attributed to that variable.
Arm A and Arm B's names ("Try First" and "Direct Answer") are just labels the researcher chose for convenience — their real identities are "experimental group" and "control group." The final comparison is the difference between the two arms on the same dependent variable (here, post-test performance without assistance).
Testing Method: Focus on Performance "After the Tutor Disappears"
Six participants (mostly developer colleagues) were split into two groups of three. Grouping was based on SQL proficiency demonstrated in two prior rounds of testing, ensuring both groups had similar skill levels.
The testing process had three steps:
- Practice phase: Complete 16 SQL problems on
GROUP BYandHAVINGwith tutor assistance - Wait 5 to 7 days
- Final test: 12 new problems testing the same skills, with no help button
Step three is the key. The researcher doesn't care how people perform with a tutor — he cares what they can still do after the tutor is gone.
Before seeing any results, Negi wrote down his criteria for interpreting the results in advance, including ten checks — if these checks failed, the results wouldn't be trustworthy. This practice later proved crucial.
Why wait 5–7 days? This isn't arbitrary. The "spacing effect" in educational psychology shows that after a period of forgetting following learning, retrieving information actually strengthens memory traces. But the waiting here serves a second purpose: letting short-term memory from the practice phase fade, thereby testing whether learners truly internalized transferable skills rather than just remembering answers they'd just seen.
If testing happened on the same day as practice, Arm B might score well simply from "having just seen similar solutions," masking the difference between the two strategies. After a week's wait, anyone who hasn't truly learned will be exposed. This also explains why the researcher's most important metric is "what can you still do a week after the tutor disappears" — it's closer to real-world independent problem-solving.
Results Revealed: A Stunning Gap and an Unexpected Truth
Each group of 3 people answered 12 problems, so each group had 36 answers.
| Out of 36 answers | Arm A: Try First | Arm B: Direct Answer |
|---|---|---|
| Correct (after any number of attempts) | 36 | 25 |
| Correct on the first attempt | 17 | 17 |
| Problems opened at least once | 36 | 29 |
| Fixed after first error | 19/19 | 8/12 |
Notice the second row. On first attempts, the two groups were identical: 17 correct answers each.
This means the tutor didn't make people better at writing correct queries from memory. The entire gap came from what people did after answering incorrectly:
- Arm A opened every single problem. Arm B left 7 blank.
- Arm A fixed all 19 first errors. Arm B only fixed 8 out of 12.
This differed from the researcher's prediction. He originally expected people to remember more, but what he saw was people persisting and fixing errors.
Why does the "fixed after first error" metric matter? In programming education, a learner's ability to debug errors is often more important than getting it right the first time. Error-fixing behavior reflects metacognitive ability — recognizing you're wrong, locating the cause, adjusting strategy, and trying again.
Arm A's 19/19 full recovery means every time they wrote something wrong initially, they at least tried again and eventually got it right; Arm B's 8/12 means nearly one-third of first errors were simply abandoned. This difference echoes research on "fixed mindset vs. growth mindset": treating errors as signals to be solved rather than evidence of failure is key to continuous improvement. An AI tutor that gives answers directly actually deprives learners of the opportunity to practice this "error recovery loop."
Why Not Claim Victory: A Rigorous Scientific Attitude
This is the part Negi is most proud of — even though it looks like a failure.
Two of the ten checks failed:
The behavioral difference between groups wasn't as large as expected. Arm A's core idea is "try before getting help," but this difference only matters if Arm B people don't try first. However, they did try — Arm B averaged 0.83 attempts before clicking Help. So Arm A didn't create a habit that Arm B lacked.
The test was too easy for Arm A. Arm A scored 100%. When one group gets a perfect score, you can't tell how far ahead it actually is.
Additionally, two unplanned factors distorted the results:
Problem ordering issue. The 12 problems were arranged from easiest to hardest. Three people stopped early, thus all missing the hardest problems. The largest gap between groups — 6 answers on the hardest skill — was actually just 6 blank answers.
Not everyone used the help feature. One Arm A participant never pressed Help during practice. So he never experienced the "try first" tutor mechanism at all, yet still scored 12/12. If you only count the four people who actually used the help feature, the gap shrinks from 11 answers to 3.
Taken together: results lean toward "try first" being better, but you can't claim that "try first" caused this result.
Therefore the conclusion the researcher can defensibly make is narrower: Making learners try first appears to change their behavior after answering incorrectly, but doesn't change their ability to get it right on the first attempt.
What does "pre-registered check criteria" mean? This is a core practice in the open science movement, where researchers publicly write down before collecting data how they will analyze it, what results would support the hypothesis, and what results would count as failure. Negi's ten pre-listed checks are a simplified version of this approach.
Pre-registration prevents two common biases: first, "post-hoc rationalization" — cherry-picking favorable explanations after seeing results; second, "p-hacking" — repeatedly trying different statistical methods until a significant result appears. When a researcher can openly say "two checks failed," their conclusions actually carry more credibility than reports claiming "all metrics improved." For product experiments, writing down check criteria is equally useful: it forces you to distinguish between "what I want to see" and "what the data actually supports."
Missing Supervision: An Overlooked Key Variable
At Demo Day, a judge asked a critical question: "Were practice and testing done on-site? Were you watching?"
The answer was no. Negi sent everyone a link, and they completed it on their own schedule.
This detail changes how results should be interpreted. When people work unsupervised, you can't know what other tools they used — a ChatGPT tab is just Cmd+Tab away.
And it affected the two groups unequally:
- Arm B people had no reason to leave the page — the answer was one click away.
- Arm A people were blocked before attempting. That's exactly the moment you'd paste the problem into ChatGPT.
If Arm A people did this, they skipped the "struggle" process the tutor was designed to create. This would make Arm A's tutor effect weaker than it should be, shrinking the gap between groups.
The final test was also unsupervised. If people looked things up during the test too, both groups' scores would be higher, shrinking the gap again.
So the unsupervised environment may have made results look weaker than they actually are, not stronger. This is actually a good story — it just didn't make it onto the slides.
Lesson: If a detail about how testing was conducted would change how others interpret your results, put it on the slides. If you omit it, people will assume standard practice, and yours might not be.
If Starting Over: Five Improvements
Negi listed five things he'd do differently next time:
- Shuffle problem order for each person. This way, stopping early wouldn't always hit the hardest problems.
- Use harder test problems so no group can score perfectly.
- Make "correct on first attempt" the primary metric. This is the number that wouldn't change in his new prediction, making it the best evidence he was wrong.
- Track who actually used the feature. People who never pressed Help never experienced the tutor.
- Conduct testing on-site, or at least log when people switch tabs or paste content.
Core Takeaways: Three Principles for AI Education Product Design
Although this experiment had a small sample size (only 6 people), its methodological value far exceeds its statistical significance:
Big gaps can come from places you didn't expect. 36 vs. 25 looks like better memory, but the first row of data reveals the truth — the gap came from error-fixing behavior, not knowledge retention.
Write your check criteria before seeing results. This is the only reason the researcher could openly say "two checks failed" instead of making excuses after the fact.
Changing your prediction isn't failure. His prediction shifted from "people remember more" to "people recover better." This change itself is the project's most useful output.
For anyone building AI education tools or any product that needs to decide "how much should AI help," this experiment sends a clear signal: giving answers directly may satisfy users in the moment, but may cultivate dependence rather than capability. And to measure whether an AI tutor is truly effective, you should look at what learners can still do after it's gone.
According to Negi, the next article in the series will cover how he built this tutor system — making the LLM never decide whether to provide help. The decision is made by ordinary code; the LLM only generates the hint text. It's precisely this rule that makes the experiment inspectable.
You can try both versions yourself:
- Arm A (Try First): https://tranquil-starlight-f0129e.netlify.app/index.html?arm=A
- Arm B (Direct Answer): https://tranquil-starlight-f0129e.netlify.app/index.html?arm=B
Related articles
Bilibili's Hit Dify Tutorial: A Practical Guide to Building 20+ AI Apps from Scratch
A distilled guide to Bilibili's top-rated Dify tutorial: Windows + Docker deployment, MySQL connection, five app types, model integration strategy, and publishing.
Agent Skills Beginner's Guide: From Concepts to Multi-Agent Collaboration in Practice
A comprehensive guide to Agent Skills: core concepts, components, the importance of Description, standardization, beginner learning paths, and multi-agent collaboration.
One Neuron Detects Both Cats and Cars? We Reproduced It and Found a New Explanation
The cat-car neuron reinterpreted: researcher Cammarata argues it distinguishes pointy from round, not cats from cars. We reproduced the experiment and found the new explanation holds on most inputs—but cars are the exception. The interpretability debate continues on language models.