Wieszcz-XIX: A 3.1-Billion-Word Historical Polish Corpus and Temporally Anchored Language Models

A Language Sample Buried Under OCR Dust

Historical Polish is well-documented as a language, but machine-readable annotated corpora covering it are extremely scarce. For the period from 1800 to 1918, only about one million words have been compiled into machine-processable form — the vast majority of texts remain buried under OCR (optical character recognition) output of highly variable quality. This means researchers hoping to apply modern NLP methods to Polish from this historical period have very little annotated data to work with.

The Wieszcz-XIX project was created specifically to address this gap. The research releases a corpus containing 6.75 billion tokens (approximately 3.1 billion words) across 294,369 documents, drawn primarily from Polish-language periodicals published between 1800 and 1918. This is more than three orders of magnitude larger than previously annotated corpora from the same period, providing unprecedented raw material for historical and computational linguistics research.

OCR technology converts printed text into machine-readable characters, but for historical documents its accuracy is constrained by paper degradation, typeface variation (19th-century Polish texts frequently used Gothic and cursive scripts), and scan quality. OCR errors in historical corpora often follow systematic patterns — specific character pair confusions (e.g., "rn" misread as "m") appear frequently, introducing substantial noise at the word-form level. For NLP models, low-quality OCR text poses a dual challenge: it causes vocabulary bloat (multiple misspelled variants of the same word are treated as distinct tokens) and causes character-level models to learn spurious patterns. This is why historical corpus projects universally face a fundamental tension between data scale and data quality, and why the cost of annotation makes large-scale manual correction nearly infeasible.

How the Corpus Was Built

The research team sourced data from Wolne Lektury and Internet Archive, two public-domain Polish document platforms. The raw text was passed through a complete processing pipeline: filtering low-quality content, removing near-duplicate documents, auditing and removing modern text that leaked in from after 1918, and performing document-level segmentation.

Notably, the team did not shy away from the data's flaws — they quantified them. They measured three categories of issues: OCR corruption (using a false-positive lower bound as a baseline), near-identical duplicate content (which was removed), and post-boundary text leakage. Content from after 1918 was excluded from the training corpus, with residual leakage compressed to just 0.04%–0.38% of bytes — and that residual was already present in the source transcriptions. The publicly released corpus is document-for-document identical to the one used for training, ensuring reproducibility.

In manually verified samples, the character error rate was only 0.68% when the source text was clearly legible — but approximately 45% of sampled passages were entirely uncorrectable due to source quality issues. This kind of candid disclosure about data quality is uncommon among many large corpus projects today.

A Ladder of Temporally Anchored Models Trained from Scratch

Building on this corpus, the research team trained a series of decoder-only architecture models from scratch, with parameter counts ranging from 47 million to 349 million, forming a scale "ladder." The core objective was to measure temporal boundedness — whether the models genuinely "inhabit" the historical period they were trained on, rather than being contaminated by modern language patterns.

The experiments compared these historical models against two modern Polish base models (one significantly larger than the historical models), and a revealing crossover phenomenon emerged: the 349M-parameter model outperformed both modern models, and the 107M-parameter model outperformed its size-matched counterpart, on specific metrics. Concretely, for modern models, processing post-1918 vocabulary costs approximately 3.1 fewer bits-per-byte compared to processing historical-period vocabulary — a gap that does not exist for the historical models; conversely, the historical models require fewer bits to process period-appropriate vocabulary.

When shown historical texts, models trained in Wieszcz-XIX preserved the original spelling conventions, while modern counterpart models could only partially do so. This demonstrates that temporally anchored models trained from scratch genuinely capture the distinctive linguistic character of their target era.

---

Bits-per-byte (bpb) is an information-theoretic metric measuring a language model's "surprise" at its input — essentially a measure of how efficiently the model compresses the text: lower values indicate more accurate, more "familiar" predictions. It is conceptually similar to perplexity but normalized per byte, making it more suitable for fair comparisons across languages and encodings. In the temporal boundedness experiments, modern models show lower bpb (more familiarity) on post-1918 text and higher bpb (less familiarity) on historical text, while the historical models show the opposite pattern — this crossover validates the models' linguistic era membership. A decoder-only architecture means the model includes only the Transformer's decoder component, where each token prediction attends only to prior context, making it suitable for language modeling and generation tasks; the GPT series uses this architecture.

The Trade-off Between Scale and Data Repetition

The research also yields a finding with practical implications for training: the gains from increasing model parameters are approximately twice those of performing a second pass over the training data. In other words, in historical corpus scenarios with limited data, expanding model capacity is more cost-effective than repeatedly cycling through the same data. This provides actionable resource-allocation guidance for similar low-resource or historical language modeling projects.

The team has publicly released the corpus, code, and model weights, making the entire study highly reproducible and reusable.

---

A "second pass" over data (also called epoch reuse) refers to training a model on the same dataset a second time when the total data volume is fixed. When data is plentiful, increasing data diversity typically outperforms repeating already-seen data; but in scarce-data scenarios like historical corpora, researchers often must choose between data reuse and model scaling. The finding that "parameter scaling yields roughly twice the benefit of data reuse" aligns with recent large-model scaling laws research: when data is the bottleneck, compute is better invested in model capacity than in additional training epochs. This conclusion has direct budget implications for low-resource and historical language modeling projects.

A Necessary Ethical Caution

The paper explicitly includes a content warning: because the training data is drawn directly from real historical publications, the models reproduce the biases of that era, including antisemitic language. This warning highlights an inescapable ethical tension in historical corpus modeling — the price of fidelity to history is that the model faithfully reproduces the uncomfortable discourse of that period. How to strike a balance between scholarly value and potential harm is a challenge this kind of research must confront directly.

Implications for Historical Language Modeling

The significance of Wieszcz-XIX extends beyond a new corpus or a set of models — it demonstrates a methodology for historical language modeling: starting from noisy OCR sources, constructing reproducible training sets through rigorous deduplication and temporal leakage auditing, and using temporal boundedness as a quantitative metric to verify whether a model is genuinely grounded in its target period. For other languages and historical periods that lack modern annotated resources, this framework offers clear and transferable guidance.