Frames of Truth: Using Open-Source AI to Uncover the Stories Behind Photos

You walk past a building, a street, or a monument every day without ever wondering who lived there or what happened in that spot. Take a photo, and all that context remains invisible. Frames of Truth, built by developer Vincenzo De Sanctis for the Hacktoberfest open-source AI challenge, aims to change that.

It's an Android app that reads GPS coordinates from a photo's EXIF metadata, then calls open-source AI models to tell the story of where the photo was taken — not what's in the frame, but the story of the place where you pressed the shutter.

GPS Coordinates Are the Starting Point, Not the Destination

Every smartphone photo carries hidden metadata: the time it was taken, the device model, and (if location is enabled) where it was shot. That last piece of information is a GPS coordinate — it records where the camera was, not necessarily what the lens was pointed at.

The distinction matters. If you're standing in your backyard photographing a distant hillside, the GPS logs your backyard, not the hill. The photo carries your perspective, not the hill's coordinates.

Most apps ignore this — they slap a generic address on the photo and call it done. Frames of Truth treats the GPS coordinate as the beginning of an investigation. After loading a photo, the app: reads the GPS coordinates from EXIF, runs reverse geocoding (Photon, Nominatim), finds nearby points of interest (Geoapify, Overpass), pulls historical context from Wikipedia and Wikidata, sends the photo to a vision model for image description, and finally uses a large language model to generate a story broken into three sections: "Place," "Image," and "Curiosity."

For deeper exploration, a second button generates a more thorough investigation report with six sections: location, surroundings, Wikipedia & Wikidata, recent news (via GDELT), photo metadata, and a set of investigative hypotheses. The interface supports 7 languages: Italian, English, French, German, Spanish, Portuguese, and Dutch.

Who It's Built For

The developer positions the app as an "anti-doomscrolling" tool. You have to go outside to use it — walk somewhere, point your camera at something, and bring back a real photo of a real place. The screen is the shortest part of the experience; the place itself is the point. It's for people who walk past things every day without asking questions, for travelers who want to skip opening ten browser tabs, and for anyone who wants a reason to explore their own neighborhood.

Reverse geocoding is the process of converting latitude/longitude coordinates into a human-readable address — the reverse of ordinary geocoding (address → coordinates). Photon and Nominatim are both open-source reverse geocoding services built on OpenStreetMap data: Nominatim is the official reference implementation from OpenStreetMap, while Photon was developed by Germany's Komoot team with optimizations for search speed. Geoapify and the Overpass API are used for points of interest (POI) discovery — Overpass is OpenStreetMap's read-only query interface, enabling nearby feature lookups by type (restaurants, monuments, historic buildings, etc.) and radius. GDELT (Global Database of Events, Language, and Tone) is an open dataset that monitors global news media in real time across more than 100 languages; the app uses it to add recent news context to a location. All of these services are built on open data and require no commercial licensing, consistent with the project's overall open-source philosophy.

The Tech Stack: Python Running on Android

Frames of Truth uses an unusual combination of technologies. The mobile layer is built in Kotlin with Python integration via Chaquopy; the backend is Python (Flask) running as a local HTTP server on 127.0.0.1:5000; and the frontend is a WebView loading local HTML/CSS/JS pages.

This architecture may seem roundabout, but it lets the developer reuse all Python logic — geocoding, POI discovery, Wikipedia queries, LLM calls — without rewriting any of it in Kotlin. EXIF and HEIC handling relies on Drew Noakes' MetadataExtractor library.

The entire project is open-source under the MIT license, with source code hosted on GitHub (github.com/vincenzodesanctis/frames-of-truth). An APK is available on the Releases page, supporting Android 7.0 and above. On first launch, users enter their own API keys via the settings menu (both Groq and Geoapify offer free tiers).

Three Hard Problems

Three technical challenges during development are worth documenting.

HEIC support: Modern Android phones and iPhones save photos as .heic by default, and Python's PIL library can't read this format natively. The solution was to detect HEIC files at the Kotlin layer by reading the ftyp magic number, extract EXIF metadata before any conversion, then convert the image to JPEG for the vision model. This way, GPS coordinates are preserved even as the file format changes.

GPS scrubbing on Android 10+: Starting with Android 10, the system strips GPS data from EXIF for any file read via a content URI — unless the app requests the ACCESS_MEDIA_LOCATION permission. Without it, every photo comes back with lat: null; with it, coordinates are intact. The developer admitted this took several days to figure out, and also requires calling MediaStore.setRequireOriginal() on the URI.

User-managed API keys: Rather than hardcoding his own keys, the developer built an in-app settings menu supporting both manual entry and import from a .txt file. Keys are stored in SharedPreferences, bridged to the Python layer via a JavaScript Interface, and never leave the device.

Chaquopy is an SDK that embeds a Python interpreter inside an Android app, letting developers call Python functions directly from Kotlin/Java code and vice versa. It integrates via a Gradle plugin and packages the CPython interpreter and required Python packages into the APK at build time. The tradeoff is a larger APK size (typically tens of MB extra), and some Python libraries with C extensions require precompiled Android ABI versions to work. For projects reusing an existing Python ecosystem (requests, Pillow, Flask, etc.), however, this is far more economical than rewriting logic from scratch in Kotlin. The WebView-loading-local-HTML/CSS/JS architecture also allows the frontend to iterate independently of native UI components — particularly efficient during rapid prototyping.

HEIC (High Efficiency Image Container) is an image container format built on the HEVC video coding standard, developed under Apple and Nokia leadership. It became the default capture format on iPhone starting with iOS 11 (2017), and some Android manufacturers have adopted it as well. Compared to JPEG, HEIC achieves roughly half the file size at equivalent quality — at the cost of compatibility, since many older libraries and third-party services don't support it natively. A "magic number" refers to a fixed byte sequence at the start of a file used to identify its format without relying on the file extension. HEIC files contain the ASCII string ftyp at offset 4, followed by a brand identifier (e.g., heic, heix). Reading the byte stream directly in Kotlin to detect the magic number is more reliable than relying on file extensions, especially when users select files from a gallery where extensions may be missing or incorrect.

Why Open-Source Weight Models

The entire AI pipeline is built on open-weight models: Qwen 3.8 on Groq for visual analysis, GPT-OSS on Groq for story generation, Llama 3.3 on OpenRouter as a narrative fallback, and Gemini's REST API as a vision fallback.

The developer emphasizes that choosing open-weight models wasn't an afterthought — it was a design constraint from the start. He wanted the ability to switch providers at any time without rewriting the app, avoiding lock-in to any single vendor. In his view, open-source innovation delivers several concrete benefits:

• No vendor lock-in: If Groq changes its pricing or shuts down its free tier tomorrow, he can switch to OpenRouter, Together AI, or a self-hosted model without changing any application code — just one API call.
• Portability: Because the AI pipeline is provider-agnostic, the app can eventually run small local models on-device. Closed-source APIs can't offer this — you can't run GPT-4 on a phone, but you can run Llama 3.2 or Gemma 2.
• Transparency: Anyone can read the pipeline end to end — which model processes the photo, what prompts are sent, how responses are structured. No black boxes.
• Community-driven infrastructure: The project depends on OpenStreetMap, Wikipedia, Wikidata, and GDELT — all maintained by global volunteers.
• Cost control for indie developers: A solo developer can't afford to pay per token for a hobby project. Free tiers from Groq and OpenRouter make this kind of experimentation viable.

Building on closed-source vision APIs, by contrast, would mean being locked into pricing that can change at any time, rate limits that fail at the worst moments, terms of service that often prohibit certain use cases, and model choices that can be deprecated without notice.

Architectural Room for Offline Use

The challenge brief mentioned "open-weight models that work without a signal." Frames of Truth still requires an internet connection for AI inference, but the architecture is designed to be portable to on-device models. The AI pipeline is decoupled from providers, so switching from Groq to a local model requires changing one function call; the vision and narrative layers are independent, making it possible to run a small vision model on-device while keeping the narrative online (or vice versa); the Python backend runs entirely on-device via Chaquopy with no server required; and the geocoding layer can be swapped for pre-downloaded OpenStreetMap data for full offline use.

The developer's take: open-weight models are evolving fast. Today they need cloud GPUs; tomorrow they'll run on your phone. Frames of Truth is ready for both futures.

On-device inference refers to running an AI model directly on a device's CPU/GPU/NPU rather than sending data to a cloud server. Recent Qualcomm Snapdragon 8-series, MediaTek Dimensity 9000-series, and Apple A-series chips all include dedicated neural processing units (NPUs) capable of running quantized models in the 1B–7B parameter range at acceptable speeds. MLC LLM, Android ports of llama.cpp, and Google's MediaPipe LLM Inference API are currently available on-device inference frameworks. The 1B/3B variants of Llama 3.2 and the 2B variant of Gemma 2 are among the open-source vision/language models that run well on mid-to-high-end phones today. Because Frames of Truth wraps AI calls in isolated functions and runs its backend in Python, replacing a cloud API call with local llama.cpp bindings would theoretically require changing only a few lines of code.

What One Indie Project Says About Open-Source AI

This project uses no AI agent tooling — it was built by hand-testing layer by layer on a real Android device, debugged iteration by iteration via Logcat.

In his retrospective, the developer wrote that open-source AI isn't just a cheaper substitute for closed-source models — it's a different way of building software. When you can see the model, switch providers, and read the entire pipeline, you actually own your product. You won't wake up one day to find that the service you depend on has changed its terms, raised prices tenfold, or shut down its free tier. For an indie project, that control is everything.

His favorite part? The app makes him want to go outside and test it. Every time he takes a photo, he learns something new about a place he thought he already knew. In his view, "touching grass" doesn't mean abandoning technology — it means using technology to reconnect with the world. Done right, open-source AI is a bridge, not a wall.