inkGround: Benchmarking Fine-Grained Visual Grounding of Cursive Handwriting in Vision-Language Models

Handwriting Trajectory Recovery: From Static Images to Dynamic Ink

Human handwriting is produced as an ordered sequence of pen-tip movements, but offline images retain only the final ink. Handwriting Trajectory Recovery aims to reconstruct ordered pen-tip trajectory points and their pen-up/pen-down states from static handwriting images. This task has practical value in signature verification, forensic handwriting analysis, educational assessment, and digital archiving.

Existing image-trajectory paired datasets are typically captured by digitizing tablets under controlled conditions, limiting both writer coverage and stylistic diversity. More critically, there is currently no benchmark to measure the visual grounding capability of general-purpose vision-language models (VLMs) on the stroke trajectory recovery task. To fill this gap, researchers propose the inkGround benchmark.

Offline vs. online handwriting is a fundamental distinction in handwriting analysis. Online handwriting is recorded in real time by digitizing devices during writing, capturing—beyond glyph shapes—the temporal coordinates, pressure, and velocity of the pen tip, and thus naturally carries a trajectory sequence. Offline handwriting retains only the static image after writing is complete (e.g., scans or photographs), with the temporal dimension entirely lost. Handwriting trajectory recovery addresses precisely the inverse problem of "reconstructing online from offline"—the spatial adjacency of ink in an image is not equivalent to the actual writing order; stroke overlaps, cursive connections, and pen-lift positions must all be inferred from visual cues, making this task inherently an underdetermined generation problem with sequence constraints.

Construction and Features of the inkGround Benchmark

inkGround is a stroke trajectory recovery benchmark for full-page, camera-captured handwriting images. It contains 537 page images, approximately 18,700 words, and about 163,000 strokes. Each page image is paired with reference trajectories generated by a pretrained offline-to-online model and validated through OCR consistency filtering and human raters.

Since these reference trajectories are pseudo-labels representing only one possible pen-tip trajectory, the researchers also evaluated VLMs on the IAM-OnDB dataset, which provides genuinely recorded pen-tip trajectories. This dual-dataset design enhances evaluation reliability: inkGround covers full-page handwriting in natural settings, while IAM-OnDB provides real trajectory references under controlled conditions.

Task Formulation: Text-Conditioned Point Grounding

inkGround defines the task as text-conditioned point grounding: given a page image and a target word, the model must locate that word and output its ordered stroke points. This formulation tightly couples visual grounding with trajectory generation, requiring the model to possess both fine-grained spatial understanding and sequence modeling capabilities.

Text-conditioned point grounding is a point-level variant of the visual grounding task. Traditional visual grounding provides a referring text and requires the model to output a bounding box for the target object in the image; point grounding refines the output to a set of coordinate points, no longer merely answering "where" but requiring a point-by-point characterization of the target's spatial structure. In this task, the text condition is a specific word—the model must both locate the word in the sense of text recognition and reconstruct that word's ink as an ordered 2D point sequence—thereby simultaneously testing fine-grained spatial perception, character segmentation, and sequence generation, which is precisely where general-purpose VLMs tend to fall short.

Evaluation Metrics and Model Lineup

Outputs are scored across multiple dimensions:

• Trajectory fidelity: DTW (Dynamic Time Warping) and Chamfer distance measure how closely generated trajectories match reference trajectories.
• Readability: the Accurate Rate and Correct Rate of rendered strokes.
• Style similarity: evaluates stylistic consistency between generated trajectories and reference handwriting.

The researchers benchmarked four open-source VLMs and three frontier closed-source VLMs using a shared context prompt. Closed-source models—especially Gemini-3.1-Pro—produce clear, readable trajectories close to the reference trajectories; this ranking also holds on the IAM-OnDB dataset.

DTW (Dynamic Time Warping) measures similarity between two point sequences that may differ in length and exhibit temporal misalignment or local stretching. It uses dynamic programming to find an optimal alignment path that minimizes cumulative point-pair distances, allowing one sequence to be nonlinearly "stretched" along the time axis to match another. This fits trajectory evaluation well: even when a generated trajectory matches the overall shape of the reference, differing writing tempo or segmentation will severely distort point-wise Euclidean distances, whereas DTW provides a more reasonable fidelity measure after alignment. Complementary to this, Chamfer distance ignores point order and computes the average distance from each point in one set to its nearest point in the other, focusing on spatial shape proximity. Together, they evaluate recovered trajectories from two perspectives: sequence alignment and spatial distribution.

Key Findings and Industry Implications

The evaluation results show that frontier closed-source models clearly outperform open-source models on the cursive handwriting trajectory recovery task. Gemini-3.1-Pro stands out in trajectory fidelity and readability, producing stroke trajectories close to reference level. This advantage is reproduced on the real trajectory data of IAM-OnDB, indicating that the ranking is not merely an artifact of pseudo-label characteristics.

For developers, inkGround provides a reproducible evaluation framework for comparing different VLMs' capabilities on handwriting visual grounding tasks. For researchers, the benchmark reveals gaps in current general-purpose VLMs' fine-grained handwriting understanding and points to directions for future model improvement: enhancing spatial grounding precision for cursive handwriting and stroke sequence modeling capability.

As vision-language models penetrate more vertical scenarios, fine-grained benchmarks like inkGround that target specific capability dimensions will become increasingly important. They not only measure whether a model "can understand" but further ask "how finely it can see and how accurately it can reconstruct."

Pseudo-labels refer to annotations that are not directly collected by humans or real devices but automatically generated by existing models and then filtered and validated. In inkGround, reference trajectories come from a pretrained offline-to-online model; since the same static image can theoretically correspond to multiple plausible writing orders, a single pseudo-label represents only one possible pen-tip trajectory rather than the unique ground truth. This is precisely why the researchers introduced IAM-OnDB as a supplement: the latter provides genuinely recorded pen-tip trajectories, allowing real data to verify whether model rankings obtained on pseudo-labels are robust. Understanding this property of pseudo-labels helps correctly interpret what "close to reference trajectories" specifically means in the benchmark—it measures agreement with one plausible trajectory, not absolute correctness.

One-line Summary

inkGround is the first benchmark to systematically evaluate general-purpose VLMs' ability to recover pen-tip trajectories from static handwriting images.

Paragraph Summary

The article introduces the handwriting trajectory recovery task—reconstructing ordered pen-tip trajectory points and pen-up/down states from static handwriting images—and notes that existing datasets have limited coverage and that no benchmark measures general-purpose VLMs' visual grounding capability. To address this, researchers propose inkGround, containing 537 camera-captured full-page handwriting images, approximately 18,700 words, and 163,000 strokes, with reference trajectories generated by an offline-to-online model and validated through OCR consistency filtering and human verification, alongside cross-validation on IAM-OnDB which provides real trajectories. The task is defined as text-conditioned point grounding, scored across multiple dimensions including DTW, Chamfer distance, readability, and style similarity. Testing four open-source and three closed-source VLMs reveals that closed-source models—especially Gemini-3.1-Pro—perform notably well, with consistent rankings on real trajectory data, exposing gaps in general-purpose VLMs' fine-grained handwriting understanding.