Solving Cumulative Text Drift: A Practical Guide to Anchor-Based Alignment for Historical Manuscripts and Digital Transcriptions

When building training data for Vision-Language Models (VLMs), historical handwritten documents are one of the toughest challenges. Vision-Language Models are a core research direction in multimodal AI, with representative models including OpenAI's GPT-4V, Google's Gemini, and the open-source community's LLaVA. The core capability of these models lies in simultaneously understanding visual information and natural language text, establishing semantic connections between the two. To train high-quality VLMs, one of the most critical pieces of infrastructure is precisely aligned image-text paired datasets—the model needs to "see" an image while "reading" text that precisely corresponds to it. If the image-text correspondence is off, the model will learn incorrect cross-modal mappings, fundamentally damaging its comprehension abilities. Historical handwritten documents, with their non-standardized page layouts and complex physical forms, represent one of the most challenging material categories for constructing such training data.

Recently, a developer named LegacyDataLabs shared a major technical upgrade to their historical Swedish handwriting data pipeline on Reddit—successfully extending processing capabilities to Selma Lagerlöf's 1891 handwritten manuscripts. This represents not only a leap in engineering scale but also reveals a widespread yet often overlooked challenge in cross-modal dataset construction: Cumulative Text Drift.

Why Manuscript Alignment Breaks Down: The Causes of Cumulative Text Drift

The project had previously completed processing of Hjalmar Söderberg's work Doktor Glas relatively smoothly. But when the pipeline faced Gösta Berlings saga—a 382-page, 126,886-word epic—the problems became fully exposed.

The core contradiction is this: the digitized ePub transcription text is a continuous, linear text stream, while the historical manuscript is a chaotic, non-linear physical artifact. ePub (Electronic Publication), as an open e-book standard format, is designed around "reflowable layout"—text content automatically rearranges based on the reading device's screen size, without preserving the original print edition's physical pagination information. This means text extracted from an ePub is a pure text stream stripped of all physical layout information—page break positions, margin whitespace, handwritten annotation areas, and other spatial information from the original manuscript simply don't exist in the ePub. When you try to mechanically map such a linear text stream onto the original manuscript, standard linear word distribution algorithms quickly fail.

Specifically, the factors causing drift include:

• Crossed-out paragraphs: Text struck through in the author's manuscript often doesn't exist in the digital transcription, causing image-text misalignment.
• Structural whitespace: Blank areas on manuscript pages cannot be filled with text.
• Archival inserts: Metadata pages inserted during library archiving (e.g., an explanatory card inserted at page 5) directly compress or stretch page boundaries.

The author notes that errors are cumulative. By page 50, text drift had become so severe that the transcription content was several pages away from the corresponding high-resolution image, completely destroying the ground-truth accuracy essential for VLM training. Ground-truth is a fundamental concept in machine learning, referring to annotation data deemed "absolutely correct," serving as the baseline reference for model training and evaluation. In handwriting recognition tasks, ground-truth is typically human-verified text transcription that precisely corresponds to each image. If the ground-truth itself is misaligned, the model will train under erroneous signals—not only failing to learn correct patterns but systematically acquiring false associations. For multimodal models that depend on precise image-text correspondence, this misalignment is fatal.

The Solution: Anchor-Based Synchronization Mechanism

Facing this challenge, the brute-force approach would be manual page-by-page proofreading, but for a nearly 400-page manuscript, this is cost-prohibitive. The author chose a more elegant engineering path—building an Anchor-Based Synchronization Layer in Python.

Core Logic of Anchor Calibration

The approach draws from the "periodic correction" concept in control theory. Control theory is an engineering discipline studying the behavior and regulation of dynamic systems, with its core problem being how to keep a system's actual output continuously approaching the desired target. In classical feedback control frameworks, "periodic correction" is a common strategy: the system doesn't need precision adjustments at every moment; instead, it measures actual deviation at preset checkpoints and applies corrective force to bring the system back on track. Periodic position corrections in GPS navigation and waypoint calibration in inertial navigation systems are typical applications of this principle.

The anchor synchronization mechanism here is highly analogous—rather than letting the algorithm drift freely, several known, human-verified fixed checkpoints are established throughout the corpus, forcing the alignment engine to recalibrate at these points. The author manually mapped and verified 8 different checkpoint anchors across the 126,886-word corpus, ranging from the famous opening sentence on the pulpit at page 13 all the way to the closing sentence outside the graveyard wall at page 379. These anchors function like positioning stakes along a railway, segmenting the infinite possibility of drift into manageable, controlled intervals—achieving global alignment robustness at relatively low manual cost.

Historical Spelling Normalization

When processing late 19th-century handwritten materials, historical spelling variants must also be addressed. In the evolution of Swedish, the late 19th to early 20th century was an active period of spelling reform. Sweden officially implemented a major spelling reform in 1906, replacing 'f' with 'v' in many words (e.g., hvad→vad, öfver→över), simplifying 'dt' to 't' (e.g., godt→gott), and standardizing certain vowel spellings. This means there are systematic differences between the spelling in Lagerlöf's 1891 manuscript and the contemporary standard spelling used in the modern ePub digital transcription. Without normalization, the same word in the manuscript and transcription would be treated by the algorithm as two completely different tokens, causing alignment matching failures.

The script first tokenizes the clean text, then applies a custom normalization layer to handle these historical spelling differences, such as öfver/över and sof/sov (old/new spellings of the same words). Building an effective normalization mapping table requires deep knowledge of the specific language's historical orthographic evolution.

After normalization, the alignment engine is forced to dynamically recalibrate text boundaries between two fixed checkpoints. This means that even if deleted paragraphs or inserted pages appear within a given interval, the error remains confined to that interval and doesn't propagate beyond the next anchor—this is precisely the key to solving "cumulative" drift.

Dataset Output and Storage Structure

After processing through this pipeline, the final output is a perfectly synchronized dataset: every high-resolution page image is surgically mapped to its corresponding text segment, stored in a production-ready .jsonl structure. JSONL (JSON Lines) is a lightweight data serialization format where each line is an independent, complete JSON object, separated by newline characters. Compared to traditional CSV or single JSON files, JSONL format has significant advantages in machine learning data engineering: it natively supports streaming reads (no need to load the entire file into memory), facilitates parallel processing and incremental appending, and can express nested complex data structures (e.g., a single sample containing image paths, multi-level text annotations, and metadata simultaneously). Hugging Face's Datasets library natively supports loading and processing JSONL format, making it one of the de facto standards for publishing multimodal datasets in the open-source community.

The author has updated public samples on Hugging Face (LegacyDataLabs homepage). As a noteworthy detail, the updated sample_dataset.jsonl presents layout data from both Söderberg and Lagerlöf side by side, allowing users to directly inspect the schema and observe how the data transitions between two different historical handwriting styles. This design is particularly friendly to researchers looking to evaluate dataset generalization capabilities.

Data Security: Compute-to-Data Sandbox Architecture

Historical manuscript datasets are extremely valuable, and balancing open training access with anti-scraping protection is a practical challenge for commercialization. The author continues the Compute-to-Data (CtD) framework approach from earlier work. Compute-to-Data is a data governance paradigm first systematically proposed and promoted by Web3 data marketplace projects such as Ocean Protocol. Its core philosophy inverts the traditional "send data to algorithms" workflow into "send algorithms to data"—data always remains in the owner-controlled environment, and computation tasks enter the isolated space where the data resides as code or containers, outputting only computation results (such as model weights) rather than raw data.

The full-scale production datasets (including Doktor Glas and Gösta Berlings saga) are placed in air-gapped environments. Air-gapping is the highest level of physical isolation in information security, meaning the system is completely disconnected from any external network. Enterprise clients deploy training scripts directly into isolated Docker containers:

• Containers are launched with --network none, cutting all network connections;
• Datasets are mounted as read-only volumes (ro).

This "data stays put, algorithms come in" model ensures models can access high-quality ground-truth data for training while physically preventing data from being copied or leaked. This architecture holds significant importance for commercializing copyright-protected cultural heritage data, medical data, and other sensitive assets, as it solves the "usable but invisible" challenge at the technical architecture level—a mainstream approach for commercializing sensitive data assets today.

Universal Insights for Multimodal Alignment

The value of this project extends beyond the Swedish literary dataset itself. It reveals a broadly underestimated problem in multimodal data engineering: when alignment between two modalities is achieved through continuous mapping, any local structural differences will trigger global cumulative errors.

The anchor synchronization approach—using a small number of high-confidence, human-verified points to constrain unbounded drift into segmented, controllable errors—has strong generalizability. Whether for OCR corpora, subtitle alignment, or document layout analysis, this "human-machine collaboration with periodic correction" pattern is worth adopting. In subtitle alignment scenarios, the audio timeline and text subtitles similarly suffer from cumulative drift caused by speech rate variations, silent segments, or audio edits; in document layout analysis, the misalignment between scanned physical layouts (multi-column, tables, figure captions) and extracted linear text streams is equally problematic. The anchor synchronization strategy provides a low-cost, highly robust universal solution for these scenarios.

The author also mentioned at the end that a third manuscript will be introduced soon, and solicited community input on spelling normalization experiences for turn-of-the-century handwritten materials. For developers working on historical document digitization and VLM training, this is a practical case worth following closely.

Key Takeaways