Wieszcz-XIX

The paper provides a content warning: because the training data is drawn from real publications of the historical period, the model reproduces the biases of that era, including antisemitic language.

The Wieszcz-XIX corpus is more than three orders of magnitude larger than the annotated corpora available for the same period.

The data sources for Wieszcz-XIX are the Polish public domain literature platform Wolne Lektury and the Internet Archive.

The processing pipeline includes filtering low-quality content, removing near-duplicate documents, eliminating modern text leaked from after 1918, and segmenting at the document level.

Residual text from after 1918 was reduced to only 0.04% to 0.38% of byte count, and these residuals were already present in the transcription source texts.

In manually proofread samples, the character error rate was only 0.68% when the text was clearly legible, but approximately 45% of sampled passages were completely uncorrectable due to source quality issues.

The research team trained a series of decoder-only architecture models from scratch, with parameter scales ranging from 47 million to 349 million.

For modern models, processing post-1918 vocabulary consumes approximately 3.1 more bits per byte than processing historical vocabulary, whereas this gap does not exist for historical models.

Models trained on Wieszcz-XIX are able to preserve the original spelling conventions when presenting historical texts, while modern baseline models can only partially do so.

The gains from increasing model parameters are approximately twice those from performing a second pass over the data.

The team has made the corpus, code, and model weights publicly available.

Wieszcz-XIX released a corpus containing 6.75 billion tokens (approximately 3.1 billion words) covering 294,369 documents, with content drawn primarily from Polish-language periodicals published between 1800 and 1918.