MedScribe AI: How an Open-Source AI Agent Generates Radiology Reports

In regions where radiologists are scarce, the wait for a chest X-ray report can be measured in days. MedScribe AI, built by a six-person team in Pakistan during a 48-hour hackathon, aims to compress that process with open-source AI agents—going from image upload to a complete clinical report in just seconds.

The Pain Point: Prediction Is Not a Report

The radiology reporting workflow is slow, costly, and inaccessible, a problem especially acute in low-resource regions like Pakistan. Local radiologists are severely understaffed, and large numbers of routine chest X-rays pile up.

A more critical issue is that most diagnostic AI tools stop at producing a raw prediction label (e.g., "pneumonia: positive"). Clinicians who receive this result still have to write the report themselves, determine priority, and communicate with patients. There is a gap between prediction and an actionable report, and MedScribe AI targets exactly this gap.

A Three-Stage Pipeline Architecture

MedScribe AI chains computer vision and generative AI into a complete clinical reporting production line, divided into three independently replaceable stages.

Stage One: Vision Model Identifies Lesions

The underlying system uses a ResNet18 model trained on the Kaggle pediatric chest X-ray dataset, with the help of TorchXRayVision, an open-source medical imaging foundation model. According to figures published by the project team, the model achieves 90.5% accuracy, 98.4% precision, and a 93.2% F1 score.

The model also integrates a Grad-CAM explainability module that highlights the specific regions on the X-ray influencing the prediction. This is critical for clinical trust—doctors see not a black-box conclusion but an evidence-based localization of the lesion.

TorchXRayVision is an open-source deep learning library for chest X-rays that brings together pretrained model weights from several large public datasets (such as NIH ChestX-ray14, CheXpert, MIMIC-CXR, and others). Its value lies in this: annotated medical imaging data is extremely expensive to obtain, and ordinary teams struggle to train a reliable model from scratch, whereas TorchXRayVision provides a baseline that can be directly fine-tuned or used for inference, significantly lowering the barrier to development. It also unifies the label systems and preprocessing pipelines across different datasets, making cross-dataset transfer more feasible.

Grad-CAM (Gradient-weighted Class Activation Mapping) is an explainability technique that computes the gradient of a target class score with respect to the feature maps of a convolutional layer, producing a heatmap that highlights the image regions the model focuses on most when making that prediction. In medical settings, Grad-CAM lets doctors verify whether the model's basis falls on the actual lesion location rather than artifacts or irrelevant structures, thereby judging whether the prediction is trustworthy. It does not alter the model itself; it serves only as a post-hoc explanation tool attached to an already trained convolutional network.

Stage Two: Four Agents Collaborate to Generate Reports

This is the most "agentic" part of the entire system. Built on the LangGraph framework, four AI agents collaborate in sequence:

Agent	Responsibility
Vision	Runs the CV model, returns findings and confidence
Reasoning	LLM decides which tools to call (ReAct loop)
Reporting	Generates a structured clinical report
Escalation	Flags critical cases and triggers immediate attention

Among them, the Reasoning agent employs a ReAct loop, dynamically deciding subsequent tool calls based on visual findings rather than following fixed rules. This design gives the system a degree of adaptability when facing different combinations of abnormalities.

LangGraph is an agent orchestration framework introduced by the LangChain team that models multi-agent workflows as directed graphs: nodes represent computational steps or agents, and edges define how state is passed between nodes. Compared with linear chain-style calls, the graph structure supports branching, loops, and conditional jumps, making it suitable for scenarios that need to dynamically decide the next step based on intermediate results. In MedScribe AI, the serial collaboration of four agents and the looped calls of the Reasoning agent can be expressed directly with graph nodes and edges, and each agent can be independently replaced or debugged.

ReAct (Reasoning + Acting) is a paradigm that lets a large language model alternate between reasoning and tool calls. The model first outputs a line of thought, decides which tool to call and with what parameters, observes the tool's returned result, and then continues reasoning, looping until it reaches a final answer. Compared with a hardcoded rule-based workflow, ReAct allows the system to dynamically choose what to look up next and whether escalation is warranted based on the specific findings returned by the vision model, giving it a degree of adaptability to different combinations of abnormalities.

Stage Three: Bilingual Interactive Interface

The application layer is built on Streamlit. After a user uploads a chest X-ray, the system returns a complete report within seconds, providing two versions simultaneously: a professional clinical report for doctors, and a plain-language summary for patients.

What is particularly noteworthy is the bilingual output—English and Urdu. For local patients in Pakistan, understanding their own diagnosis in their native language directly affects the quality of doctor-patient communication. The report can also be exported as a PDF, including the original X-ray and the Grad-CAM heatmap.

Why Stick with the Open-Source Path

MedScribe AI's technology choices fall almost entirely within the open-source ecosystem, and this is no accident.

The inference layer uses openai/gpt-oss-120b running via Groq, achieving fast and free inference. More importantly, patient X-ray data is not sent to any proprietary server—for medical scenarios, keeping data within local control is a hard requirement for privacy compliance.

The remaining components likewise follow open-source principles: TorchXRayVision provides pretrained weights for medical imaging, LangGraph provides a customizable agent orchestration framework, and PyTorch + torchvision ensure reproducible training. As the project team puts it: "Open-source AI is not just a technical choice—for healthcare in underserved regions, it is the only option that makes the economics work."

Business Model: Can Clinics Afford It?

The project team offers clear paid use cases:

• Private clinics in Pakistan and South Asia without on-staff radiologists
• Telemedicine platforms that need automated preliminary reports
• Hospital chains handling large volumes of routine chest X-rays

The pricing logic is straightforward—clinics pay $50 per month for AI preliminary triage, far less than the cost of hiring a part-time radiologist. This price gap means real demand space in low-resource markets.

48-Hour Hackathon Output and Limitations

MedScribe AI was completed within 48 hours by six Pakistani members at the Pak Angels Generative & Agentic AI Hackathon (Cohort 11). As a hackathon project, its level of completeness is commendable: from model training and agent orchestration to the front-end interface and PDF export, it covers the full chain.

But its limitations must also be acknowledged. The pediatric chest X-ray dataset used by the project is limited in scale, and the model's generalization ability has not yet been validated on adult imaging or multi-center data; the accuracy and compliance of the clinical reports have also not been systematically evaluated in a real medical environment. From hackathon demo to clinical deployment, multiple hurdles remain in between, including regulatory approval, clinical validation, and liability definition.

Nevertheless, the approach demonstrated by MedScribe AI—using an open-source agent framework to upgrade "prediction" into an "actionable report," and localizing it for low-resource scenarios—offers a pragmatic reference path for landing medical AI.

One-line Summary

A Pakistan team built an open-source AI agent in a 48-hour hackathon, upgrading chest X-ray prediction into bilingual clinical reports.

Paragraph Summary

MedScribe AI is a medical AI project built by a six-person team from Pakistan in a 48-hour hackathon, aimed at solving the pain points of scarce radiologists and backlogs of chest X-ray reports in low-resource regions. It upgrades diagnostic AI from a single prediction label to a complete actionable report, using a three-stage pipeline: a vision model based on ResNet18 and TorchXRayVision (90.5% accuracy) to identify lesions and generate Grad-CAM heatmaps; four LangGraph-based agents (Vision, Reasoning, Reporting, Escalation) collaborating to generate structured reports; and a Streamlit interface providing doctor and patient versions with bilingual English and Urdu output. The system is fully open-source, keeps patient data local, and targets clinics in South Asia without on-staff radiologists at around $50 per month. The project is relatively complete, but its dataset size, generalization ability, and clinical validation still have clear limitations.