AI Map Feature Alignment in Practice: A Hybrid CV + VLM Architecture Breakdown

Background: The Cross-Version Map Feature Alignment Problem

A developer building a D&D virtual tabletop (VTT) application raised a representative computer vision challenge: how to reliably transfer map symbols — doors, secret doors, stairs, and similar features — between two different versions of the same scene's map.

The setup is this: the developer typically has two maps. One is a reference map with annotated symbols for doors, secret doors, ladders, and staircases. The other is a live-play battle map, the version players actually use. The problem is that these two maps are rarely pixel-for-pixel identical — they may be different artistic renditions of the same location, with slightly different scales, crops, textures, or even subtle geometric differences.

What the developer wants to do is essentially: "This secret door is at this position on the reference map — where does it land on the battle map?" And to do this for every relevant feature. This is a classic image registration problem, but the difficulty spikes sharply because the two maps are "artistically redrawn versions" rather than transformed copies of the same raster image.

Attempts with Multimodal LLMs — and Their Limits

The developer has been relying primarily on multimodal AI models like ChatGPT, Codex, and Claude Code to tackle this. The rough workflow looks like:

1. Feed both the reference map and battle map to the AI
2. Identify known features on the reference map (usually with coordinates)
3. Ask the model to locate corresponding architectural landmarks on the battle map
4. Infer the transformation between the two maps based on those landmarks
5. Transform door/symbol coordinates from the reference map to the battle map
6. Manual visual inspection, iterating as needed

The developer also tried breaking it into stages — first having the system reason from recognizable landmarks like walls, room corners, and corridors, then placing symbols relative to those features.

Why Results Are Inconsistent

This approach works reasonably well when the two maps are very similar, but falls apart otherwise — sometimes badly. The system might identify roughly the right region yet place an individual feature dozens of pixels or even several in-game meters off.

Secret doors are the hardest case. On the reference map, a secret door is often represented by a subtle cartographic symbol blended into the wall. On the battle map, that symbol may not exist at all — the real task is identifying the corresponding wall segment. AI models frequently identify the right room but then:

• Place the secret door on the wrong wall
• Place it on the right wall but with significant positional drift
• Confuse nearby architectural features for the target landmark
• Apply a transformation that works in one part of the map but drifts severely in another

This exposes a fundamental issue: a single global affine transform (scale/rotation) is often insufficient, because maps can contain irregular geometry with local deformations that a single set of parameters can't capture.

Getting Back to Basics: This Is an Image Registration Problem

The developer admits to having no computer vision background, and has been treating multimodal models as tools capable of "visual reasoning" and deriving correspondences. But they're starting to realize this should probably be framed as a more traditional image registration and feature matching problem, with AI playing only one part of the pipeline.

From a technical standpoint, that's exactly right. Asking a VLM (vision-language model) to directly output pixel-precise coordinates is fundamentally asking it to work in a way it isn't built for. Large models excel at semantic understanding and coarse-grained localization, but geometric precision is not their strong suit. This is the core reason why no amount of prompt tweaking produces stable results.

Traditional CV Tools Worth Bringing In

For cross-style image registration problems like this, the mature computer vision toolchain includes:

• Keypoint detection and feature matching: classic methods like SIFT and ORB, or deep learning approaches like SuperPoint
• Homography and affine transform estimation: for establishing a geometric mapping between the two images
• RANSAC: for filtering out incorrect matches and robustly fitting transform parameters
• SuperGlue / LightGlue: graph neural network-based feature matchers that are more robust than traditional descriptors across style and viewpoint changes
• Semantic segmentation: to explicitly extract structural information like walls and rooms
• Non-rigid or piecewise image registration: to handle local deformations and inconsistent wall geometry

For the case of "two maps depicting the same underlying floor plan but in different art styles," traditional intensity- or texture-based registration fails because it assumes visual similarity between images. Matching based on structural semantics — wall lines, room outlines — is far more reliable here.

Recommended Approach: CV for Geometric Precision, VLM for Semantic Understanding

After analyzing the problem holistically, the most sensible technical direction is a hybrid CV + VLM architecture where each component does what it's actually good at.

Step 1: Semantic Structure Extraction

Start by using a segmentation model (such as SAM or a purpose-trained wall segmentation network) to extract walls, rooms, and corridors from both maps as structured vector or mask representations. VLMs can assist here by helping identify "what's a door vs. what's a wall," abstracting away messy art styles into a consistent topological structure.

Step 2: Structure-Based Piecewise Registration

Don't aim for a single global transform. Given the irregular geometry of maps, use a piecewise or non-rigid registration strategy:

• Start with coarse room-level matching to establish room-to-room correspondences
• Estimate local transforms within each room or region independently
• Apply RANSAC to filter bad correspondences and avoid the "locally correct, globally drifted" failure mode

Step 3: Feature Transfer and Symbol Localization

For features like secret doors that exist only on the reference map, the right approach is: don't try to match the symbol itself — match the wall segment it belongs to. First determine which wall segment the secret door belongs to on the reference map, and where it sits relative to that segment (e.g., "30% along the wall from corner A to corner B"). Then find the corresponding wall segment on the battle map and position the feature at the same relative proportion. This allows reliable placement even when the battle map has no secret door symbol of its own.

Step 4: Manual Anchor Points as a Safety Net

In practice, letting users manually mark a small number of anchor points (e.g., a few obvious room corners) dramatically improves registration accuracy. This is an engineering tradeoff with very low cost and very high return — when full automation isn't realistic, a semi-automated workflow is often the optimal solution.

Lessons for Hybrid AI + CV Projects

This case has practical takeaways for any developer trying to apply AI to specialized vision problems.

LLMs are not universal tools. Multimodal models shine at semantic understanding, coarse localization, and assistive reasoning — but for tasks requiring geometric precision and sub-pixel coordinate output, dedicated CV algorithms remain irreplaceable. Decomposing the problem into "what AI is good at" and "what traditional algorithms are good at" is almost always more effective than endlessly tuning prompts.

Frame the problem in standard terminology first. The developer's eventual realization that this is an image registration problem was itself a key step toward solving it. Before writing code or tuning models, classifying your problem using the field's standard terminology helps you quickly surface mature toolchains, relevant papers, and open-source projects.

For cross-style registration scenarios like D&D maps, tools worth evaluating as starting points include LoFTR and LightGlue for feature matching, MicroSAM for segmentation, and mature non-rigid registration libraries from the medical imaging world such as SimpleITK and SimpleElastix.