Building an AI Math Tutor That Photographs, Marks, and Explains Mistakes in a Day

Most AI math tools aim to solve problems for you. A developer spent one day building the opposite: Somo (Swahili for "lesson") — a math tutor that takes a photo of your handwritten work, uses AI to find the first incorrect line, circles it in red like a teacher, and explains it in English, Swahili, or Spanish.

Why someone would build this

The author grew up in Tanzania. Local primary schools teach in Swahili, but upon entering secondary school all subjects switch to English — even the NECTA national exam is taken in a language you're still learning. Many students are fine at math; they get stuck on language. After moving to New York, he saw the same struggle in English learners.

This shaped Somo's problem statement: instead of having AI solve problems directly, can AI use the language the student thinks in to show them why they went wrong? The author built a prototype in a day to test the idea.

Five-step flow: from photo to real exam practice

Somo's usage flow breaks into five clear actions:

1. Photograph: Take a picture of the problem and your handwritten work.
2. Diagnose: Gemini recognizes the handwriting and returns structured JSON containing the transcribed text, the concept tested, the first incorrect line, and the reason for the error. The UI circles that line in red.
3. Confirm: Somo first shows "what I read," and the student confirms or corrects it before any explanation begins.
4. Teach: Stream a short lesson covering the concept, where it went wrong, a real-life analogy (market scales, subway fares), an example with different numbers (to avoid giving away the answer), and the English terminology used in exams.
5. Check: Answer a problem. Get it wrong and you get a simpler explanation and a new analogy; get it right and you move on to real past exam questions.

In AI engineering, reliability is the bulk of the work

The author's core takeaway: prompts are the easy part; the real work is everything around them.

Self-verification with a human backstop

The system makes a second independent model call to re-solve the problem without seeing the diagnostic result. When the two answers disagree, the student sees "I'm not sure I read that correctly."

But verification has blind spots. A printed problem the author tested lost its formatting, and both model calls read it the same wrong way — consistently. That's exactly why the "human confirmation" step exists: the student has the paper; the model doesn't.

Failover is a feature, not a patch

Gemini's free quota ran out mid-development. The author funneled all calls into one function: try Gemini, retry once on 503/429, then degrade to Claude, and added a circuit breaker — skip Gemini for 60 seconds after a quota error. Diagnostic time dropped from ~116 seconds (waiting on an overloaded model) to ~10 seconds.

Circuit Breaker comes from distributed systems. Its core idea: when a dependency fails repeatedly, actively "break" calls to it for a while instead of letting every request hit the same failure. Somo's implementation: after Gemini reports a quota or overload error, skip it for 60 seconds and hand requests to Claude. The benefit isn't just saving retries — it converts uncontrollable vendor flakiness into predictable response times. No request can get stuck waiting on an overloaded model, and overall time drops from ~116 seconds to ~10 seconds. Circuit breakers are typically paired with fallback: during the open circuit, a backup provider keeps things running, then traffic returns to the primary path. It's a standard pattern for LLM apps handling quota fluctuations and rate limits.

Labels must be honest

The pipeline once labeled a British practice paper as "real past exam questions." Now every question is clearly tagged as past exam, web practice, or Somo-generated, and off-topic questions are rejected.

On three handwritten problems the author tested, Somo correctly found the error line every time: a division-by-2 mistake, a missing negative sign when moving terms, and multiplying both sides of an equation by different numbers. The author notes: no real students have used it, and all test problems were written by himself.

Tech stack: three components, each with a role

The backend is FastAPI with Server-Sent Events for streaming; the frontend is a single-file mobile interface styled like a school exercise book: grid paper, red margin line, teacher's notes in handwriting font.

Gemini: the key to structured output

gemini-flash-latest handles the multimodal heavy lifting. The key is structured output — no free-text parsing; diagnostic results are validated directly against a Pydantic schema. Because error_step is a real field, the UI knows which line to circle. Gemini also handles streamed explanations and generating check questions.

Gemma 4 (gemma-4-26b-a4b-it, via the same Gemini API) handles high-frequency, low-risk tasks: grading and judging whether two answers are equivalent ("x = 4" vs "4"). It's fast and cheap, ideal for checks that run on every answer.

Structured Output means the model is constrained to return results in a predefined data schema rather than free-form text. The developer defines a schema with Pydantic, and Gemini enforces the output to that structure via function calling or response schema parameters — field types and required fields are validated on the model side. For an app like Somo, this determines programmability: only when the diagnostic result contains an explicit integer field like error_step can the frontend know which line to circle in red. With natural language, you'd need a fragile parsing layer that breaks whenever the model's wording changes. Structured output also makes downstream validation, caching, and retries deterministic — a key step for bringing multimodal models into production pipelines.

Nimble: turning "more practice" into "real exam practice"

When a lesson starts, a background task has Nimble search the live web for past exam papers on that concept, prioritizing official sites. US students get NY Regents question banks; when official sites yield nothing, it searches the open web.

Nimble found the right documents: the Tanzania Examinations Council's official math papers and examiner reports from 2017 to 2023. But PDF results only returned ~300–700 character summaries with no actual questions. The fix: treat Nimble as a discovery tool. When it hits short PDF results, Somo downloads the file, uses pypdf to extract up to 20,000 characters (processing five papers in parallel), then has Gemini pick a question that appears verbatim in the source and tests the same concept. NECTA's "Candidate Item Response Analysis" reports turned out to be a goldmine because they quote every exam question word for word.

The first real result came from a matrix word problem on the NECTA CSEE 2023 basic math paper, displayed with the exam stamp and source link. Results are cached to disk by fuzzy topic match, so each topic is searched only once.

Sentry: catching a bug logs can't see

All calls to Gemini, Gemma, Claude, and Nimble are Sentry spans, and background tasks are their own transactions. Every Gemini-to-Claude degradation also throws a warning issue, turning the issue list into a timeline of "when and why we switched providers."

The real win came from a bug exposed by the trace waterfall: during a past-exam search, a NECTA server took 3.6 minutes to send a single PDF. The code had a 12-second timeout, but httpx times per chunk, so a slow-drip server never triggers it. The fix was a hard deadline on the whole download, cutting worst case from ~4 minutes to ~30 seconds. The author admits this bug was invisible in logs. Sentry is configured with send_default_pii=False, so student photos and prompt text are never sent.

httpx timeouts are per-chunk, which is counterintuitive. By default, it times each chunk as it reads the response body, not the entire download. That means if a server drips data very slowly — each chunk just under the timeout threshold — a request can drag on for minutes without triggering a timeout. This kind of issue is nearly invisible in normal logs because every read "succeeds" with no error, just abnormally long total time. The fix is to explicitly set an overall timeout for the whole download (or use httpx.Timeout with separate read/write/connect plus an outer deadline), cutting worst case from ~4 minutes to ~30 seconds. This also shows the value of a trace waterfall: it breaks slow requests into spans so you can see where time is spent.

The real experience of a one-day hackathon

The project was born at a one-day MLH hackathon. The author had no team and no idea. He spent the morning brainstorming, settled on education because of his background, and spent the rest building.

Three things stood out:

• Building with coding agents: He started with Gemini CLI, hit a usage limit, and switched to Claude. The workflow changed — time went into design decisions and hands-on testing with handwritten problems, less into boilerplate.
• Sponsor talks are genuinely valuable: He hadn't planned to use Nimble but decided to integrate it after hearing the talk, and it became the feature he's most proud of.
• Real-world friction: Venue wifi kept his phone from connecting to his laptop, Gemini quota ran out, government servers were painfully slow — each one forced a better design.

The deadline was 5 PM. He froze features at 3:45, then wrote the README, recorded a demo, and submitted in the final minutes.

Three things worth borrowing

Somo's value isn't "yet another AI solver" — it's showing a different product path: don't do the problem for the student; locate the error and explain why. This positioning matters especially in education settings with high language barriers.

At the engineering level, three things are reusable: first, multi-model self-verification plus human confirmation is more effective than just improving prompt quality; second, circuit-breaker plus fallback failover design turns uncontrollable vendor fluctuations into predictable response times; third, observability can't rely on logs alone — traces catch hidden issues like httpx per-chunk timeouts.

Of course, the project is still a prototype: no real students involved, all test samples written by the author, and multilingual teaching effectiveness not yet validated by educational experiments. The author's emphasis on "honest labels" and "human confirmation" precisely illustrates the restraint AI education tools need most at this stage.

Background notes

PDF text extraction differs from web scraping. PDFs store drawing instructions by page and font position internally, not semantic HTML. Libraries like pypdf parse content streams and reassemble extractable text by page, but scanned documents, complex layouts, or encrypted files may yield nothing or garbled output. Somo's strategy is to use Nimble for discovery first, then download the original PDF itself, extract up to 20,000 characters with pypdf, process multiple papers in parallel, and finally have Gemini pick questions matching the concept from the extracted text. This decouples "search service" from "content retrieval": search finds the right document, extraction gets the body text, and verbatim matching ensures questions aren't rewritten or truncated.

One-line Summary

Somo uses AI to find the first error in a student's handwritten work and explain it in their native language, showing a different product path for educational AI.

Paragraph Summary

Somo is a math tutor app that inverts mainstream solvers: photograph your handwritten work, and AI identifies and circles the first incorrect line, then explains why in English, Swahili, or Spanish. The author saw the same struggle in Tanzania and among English learners in New York — students are fine at math but stuck on language — and built a prototype in a day. The flow has five steps: photograph, diagnose, confirm, teach, check, with human confirmation as a backstop for model reading blind spots. Engineering details include multi-model self-verification, circuit-breaker plus fallback cutting diagnostic time from ~116s to ~10s, Nimble searching real exams with pypdf extraction, and Sentry traces catching an httpx per-chunk timeout bug. The project is still a prototype: no real students, all test problems written by the author.