Penmentor: Using Claude to Grade Handwritten Math Homework with Teacher-Like Precision

When AI Grading Meets Handwriting

In the Show HN section of Hacker News, a project called Penmentor has drawn attention. It claims to use the Claude model to grade and annotate users' handwritten math homework like a teacher. Although the current post is not very popular (9 points, no comments yet), the idea itself touches on a long-standing pain point: how can digital learning tools truly understand the thinking process on paper?

Traditional online education platforms usually rely on multiple-choice questions or formula editor input, but handwritten problem-solving—especially mathematical derivation—is the most natural way for students to express themselves. Penmentor attempts to bridge this gap: users upload handwritten math content, and Claude not only recognizes the text but also points out errors in the steps, provides annotations, and simulates feedback from a real teacher.

Technical Implementation: Claude's Multimodal Capabilities

Judging from public information, the core of Penmentor is calling Claude's vision model. The Claude series models have image understanding capabilities and can parse handwritten formulas, symbols, and diagrams. Unlike simple OCR, grading math homework requires understanding the logical chain—for example, whether algebraic transformations are equivalent, and whether calculus steps are rigorous.

In the project demo, Claude can directly add annotation boxes and text on handwritten images. This "marking" method is close to the effect of a teacher circling and drawing with a red pen. However, the accuracy of handwriting recognition, the ability to parse complex formulas (such as integral signs and matrices), and the rigor of grading still need more practical testing to verify. At present, the project seems to be in an early stage, and its functional boundaries and limitations are not yet clear.

The Difference Between OCR and Multimodal Understanding

OCR (Optical Character Recognition) mainly solves the problem of "converting text in an image into editable text." It outputs a character sequence and does not care about semantics. Grading handwritten math homework, however, requires multimodal understanding: the model must not only recognize "∫ x² dx" but also know the variable of integration, the limits, and the integrand, and judge whether the next transformation is equivalent. Vision-language models (such as Claude's visual capabilities) usually encode images into feature vectors and then feed them together with text instructions into a language model for reasoning, so they can perform the combined task of "recognition + reasoning." But such models are sensitive to the spatial layout of symbols, superscripts and subscripts, and fraction structures. In handwritten scenarios, they can easily misread cursive symbols as other characters, which is also one of the main bottlenecks for the accuracy of applications like Penmentor.

Potential Application Scenarios and Value

For students, Penmentor provides a way to get immediate feedback—step-level guidance without waiting for a teacher to grade. For teachers, it can serve as an auxiliary tool to quickly check common errors in a large number of assignments and save time on repetitive work. In remote learning or self-study scenarios, this kind of tool is especially useful.

In addition, the project also demonstrates another possibility for AI in education: not replacing teachers, but enhancing interactivity in the practice phase. Grading and annotation are key links in the teaching loop. If AI can reliably complete this part of the work, it will significantly improve learning efficiency.

"Formative Assessment" in the Teaching Loop

In education, assessment is often divided into "summative assessment" (such as final exams) and "formative assessment" (such as classroom exercises and homework grading). The core value of formative assessment lies in timely feedback: the sooner students receive correction after making a mistake, the easier it is for them to correct misunderstandings. The feedback cycle of traditional homework grading is usually measured in days, while AI grading tools can compress the cycle to minutes. The "red-pen annotation" simulated by Penmentor is a typical form of formative assessment—it not only points out right and wrong but also marks the specific step where the error occurs. If such tools can be combined with error notebooks and knowledge point graphs, they can also help learners locate weak areas, extending from a single grading session to long-term learning path planning.

Challenges and Open Questions

Although the idea is appealing, Penmentor faces several practical challenges. The first is the robustness of handwriting recognition: different people's handwriting varies greatly, and mathematical symbols are easy to confuse (such as "1" and "l", "∂" and "d"). The second is grading accuracy: AI may misjudge correct steps as wrong, or ignore subtle logical gaps. For advanced mathematics, verifying the equivalence of steps requires strong symbolic reasoning capabilities, and current vision models still have limitations in this regard.

In addition, privacy issues cannot be ignored—handwritten homework may contain personal information, and uploading it to the cloud for processing requires a clear privacy policy. The project has not yet disclosed how data is processed. Finally, the commercialization path and pricing strategy are also unknown.

The Limitations of Symbolic Reasoning and Vision Models

Verifying the equivalence of mathematical steps essentially requires symbolic reasoning capabilities, such as judging whether two algebraic expressions are identically equal, whether integration by substitution is valid, and whether limit operations satisfy the conditions. Current vision-language models mainly rely on pattern matching and language priors and do not have strict symbolic computation engines. They may "appear" to give reasonable annotations but make mistakes where precise derivation is required. This phenomenon is often called "hallucination." A common engineering mitigation is to let the vision model only handle "recognizing and transcribing" handwritten content, then pass the transcription result to a symbolic computation system (such as SymPy or Mathematica) for equivalence verification, and finally have the language model organize the feedback language. This hybrid architecture of "vision model + symbolic engine" may be a key path to improving grading reliability.

Conclusion

Penmentor is a small experiment worth watching. It attempts to apply Claude's visual understanding capabilities to the niche scenario of handwritten math grading. Although current information is limited and technical maturity remains to be verified, it points to a clear direction: AI is moving from "answering questions" to "grading the process." If the team can solve the problems of recognition accuracy and grading reliability, such tools may become daily assistants for learners. Developers interested in AI education applications and Claude's multimodal capabilities should keep an eye on it.