3B Model Reads Japanese Invoices, Validation Boosts Accuracy to 86.8%

Title

3B Model Reads Japanese Invoices, Validation Boosts Accuracy to 86.8%

Meta Description

Qwen2.5-VL-3B fine-tuned with QLoRA extracts Japanese qualified invoices. Combined with check digits, registry lookups, and five tax arithmetic validations, it achieves an 86.8% auto-approval rate on the synthetic test set, with zero verifiable field errors among approved invoices. Vertical writing remains a weak point, and the limitations of synthetic data are clear.

Article Content

Japanese qualified invoices (適格請求書) must include a registration number, amounts and taxes per tax rate. These fields come with "symbolic ground truth"—the registration number has a check digit, the corporate number can be looked up in public databases, and the tax amount is an arithmetic result—making extraction results automatically verifiable.

Developer raihan-js fine-tuned Qwen2.5-VL-3B-Instruct with a QLoRA adapter on 3,000 synthetic invoices, then used five cumulative validations to decide whether to auto-approve. On the synthetic test set, 83.0% of invoices were completely correct, 86.8% were auto-approved, and none of the approved invoices contained field errors that validation could catch.

The Problem: Instead of Asking "How Accurate Is the Model," Ask "How Many Errors Slipped Through"

Traditional OCR or VLM extraction evaluations only look at field accuracy, but what really matters in the invoice scenario is: how many errors were intercepted, and how many slipped into downstream processes.

Japan's qualified invoice system happens to provide self-checking conditions: the registration number format is fixed (T plus 13 digits), the corporate number contains a check digit, and the tax amount can be derived purely arithmetically at 10%/8% tax rates. This means an extractor can self-verify against symbolic ground truth rather than relying solely on probabilistic confidence.

The author's focus therefore shifts to: how many erroneous extractions validation can catch, and how many errors can bypass validation?

Symbolic ground truth refers to "objective answers" that don't depend on manual re-annotation and can be mechanically derived through rules or databases. In this article's scenario, the registration number (T + 13 digits) of a Japanese qualified invoice has a check digit, the corporate number can be found in the official registry, and the tax amount is obtained by multiplying the tax base by 10% or 8%—these are not subjective judgments but results that can be independently recomputed by a program. Traditional OCR/VLM evaluation often compares model output against manual annotations, which can themselves contain errors and are costly; with symbolic ground truth, extraction results can be cross-validated against rules and databases, enabling automated "self-scoring," and making the question of "whether errors were intercepted"—which truly affects downstream processes—quantifiable.

The Approach: Three Extractors + Five Cumulative Validations

Three extractors output the same JSON structure:

• OCR + Rules: hand-written rules on PaddleOCR text boxes
• Zero-shot: Qwen2.5-VL-3B-Instruct direct inference
• QLoRA fine-tuned: the same model plus adapter, trained for 1 epoch on 3,000 synthetic invoices, taking 4 hours 57 minutes on an RTX 3060

The five cumulative validations are, in order: schema structure, check digit, registration database lookup (must be valid on the invoice date), issuer name matching against the registry, and tax amount arithmetic under three rounding conventions (切り捨て/四捨五入/切り上げ). If all pass, the invoice is auto-approved; if any fails, it goes to manual review with the failure reason attached.

QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method proposed in 2023. It freezes all original weights on a 4-bit quantized base model and injects only a small number of trainable low-rank matrices (LoRA adapters) into attention layers and other positions. Compared to full-parameter fine-tuning, trainable parameters typically account for only a few thousandths of the original model, greatly reducing VRAM requirements, so training can be completed on a single consumer-grade GPU. In this article, the 3-billion-parameter Qwen2.5-VL was trained on an RTX 3060 (12GB VRAM) for only about 5 hours on 3,000 samples for 1 epoch—exactly the feasibility that QLoRA brings; moreover, since the base weights are not rewritten, the adapter is only a few dozen MB in size, making it easy to release and reuse separately. The trade-off is that quantization introduces slight precision loss, and downstream task performance highly depends on the capability boundaries of the base model itself.

Data and the Pitfalls Unearthed by Three Tests

The dataset deliberately excludes real companies: all issuers are fictional, names are composed of parts, registration numbers are generated using real check digit rules, and numbers already present in the official registry snapshot are excluded. The registry used for validation is also synthetic data, released together with the dataset. There are 4,100 invoices in total, covering 6 layouts (including vertical writing 縦書き), seals (角印), Reiwa and Western calendar dates, full-width digits, and scan noise. Each rendered page is checked against gold labels, and issuers do not overlap across different data splits.

The author says testing "earned back its cost," uncovering three issues:

1. About one-fifth of "clean" invoices had dates earlier than the issuer's registration date, and the validator correctly determined them invalid
2. Artificially injected 1-yen tax errors may be difficult to distinguish from different rounding conventions and are essentially undetectable, so they must be excluded from the "injected errors" set
3. The "corrupt one digit" generator sometimes swaps 0 and 9, and the check digit cannot see this error—because its weighted sum is taken modulo 9

Check digit is a redundant digit at the end of a code used to detect input errors, obtained by taking a weighted sum of the preceding digits and then applying a modulus. The Japanese corporate number (National Tax Agency registration number) uses 13 digits, with the last digit calculated from the first 12 digits using fixed weights. Its characteristic is that when a single digit is miswritten or two adjacent digits are swapped, in the vast majority of cases the check digit will not match and thus be identified; but there are design blind spots—for example, the weighted sum modulo 9 mentioned in this article means that after swapping 0 and 9, the checksum remains unchanged, and such errors cannot be caught by the check digit itself. Understanding the coverage and blind spots of check digits is a prerequisite for assessing whether "auto-approval" is reliable.

Results: Auto-Approval Rate Reaches 86.8% After Fine-Tuning

Test Set (600)	Full-Invoice Exact	Auto-Approved	Errors in Covered Fields Among Approved
OCR + Rules	37.8%	55.0%	0 out of 292 (upper bound 1.3%)
Zero-shot	5.3%	8.1%	3 out of 43 (7.0%)
QLoRA	83.0%	86.8%	0 out of 461 (upper bound 0.8%)

"Covered fields" refer to fields where validation can catch errors: registration number, issuer name, tax base, totals, and line item figures. This definition was fixed before scoring the fine-tuned model. Without validation, the fine-tuned model had covered-field errors in 12.8% of invoices. Of 69 artificially injected document issues (wrong check digit, unregistered number, deregistered issuer, someone else's number, tax amount error), all 69 were sent to manual review, and 95.7% were caught by the corresponding validation.

Where Validation Cannot See

Among invoices approved by the fine-tuned model, 4.6% still contained errors, all in fields no one verifies: 10 wrong invoice numbers, 10 wrong dates, and 1 line item. The reason is straightforward—the registry knows who the issuer is, not who the customer is; even if the date is wrong, it can pass as long as the issuer is within a valid window.

The OCR baseline is worse: 32% of approved invoices had typos in line item descriptions (e.g., 「費」 read as 「费」), which no arithmetic can detect. Validation makes covered fields safe to automate; other fields require separate controls.

Layout Weak Point: Vertical Writing

Horizontal layouts perform excellently: 97.3% exact on the test set and 99.0% on unseen horizontal layouts. Vertical writing is the weak point: 40.0% on the test set, mainly due to Latin text printed sideways (registration number 62.7%, date 82.7%), while table sections are actually normal; on unseen fully vertical layouts, tables are rotated and only 4%.

The learning curve shows that more data cannot solve this problem: 8.7% with zero training samples, 84.7% at 800 samples, and 89.3% at both 2,400 and 3,000 samples.

Vertical writing (縦書き) is the traditional Japanese writing style arranged top-to-bottom and right-to-left, widely used in official documents, forms, and formal paperwork. For vision-language models, the difficulty of vertical writing lies in: models are mostly pretrained on horizontal corpora, and reading order, text box aspect ratios, and the sideways rotation of Latin characters (such as the "T" in registration numbers and Arabic numerals) in vertical writing are all inconsistent with the training distribution. This explains why in this article up to 60% of vertical layout failures are concentrated in registration number and date fields, while table content is normal due to its regular structure; it also suggests that simply adding more training samples of the same vertical layout will be hard to break through, and diverse vertical layout structures or specialized preprocessing are needed.

Hidden Risks Masked by the Synthetic Registry

On a real registry (2.58 million companies, all passing check digits), the author corrupted one digit in each of 300,000 registered numbers: check digits caught 95.6%, lookup caught 4.1%, and 0.29% landed on another real company—because registered numbers exhibit clustering. Only comparing issuer names can catch these. The synthetic registry does not have clustering characteristics, so it underestimates this risk; name validation is the fallback.

Limitations and Next Steps

• Synthetic invoices are cleaner and more regular than real invoices: 6 layouts, 18 products, two tax rates, corporations only. No guarantees are made about real invoices.
• Sole proprietors are not covered: there are no public check digit rules, and individual registration data is personal information and was not loaded.
• Single model, single run, single random seed.
• The base model Qwen2.5-VL-3B-Instruct uses the Qwen Research License, so the adapter is limited to non-commercial research use.

The code, 69 tests, dataset generator, and all result files are in the repository; the dataset and adapter have been released on Hugging Face. bash scripts/gpu_chain.sh can rerun zero-shot, training, and fine-tuned evaluation, and resume from checkpoints after a crash.

The author's planned next steps include: testing on real invoices for which usage rights are held; adding recipient validation from customer master data; adopting a larger base model with a more permissive license; and giving vertical layouts more than one training structure.

One-line Summary

QLoRA fine-tuning Qwen2.5-VL-3B to extract Japanese qualified invoices, with five validations achieving 83% full-invoice exact rate and 86.8% auto-approval rate.

Paragraph Summary

This article evaluates three Japanese qualified invoice information extractors (OCR+rules, zero-shot VLM, QLoRA fine-tuned VLM). The core idea is to use the invoice's own symbolic ground truth (registration number check digit, corporate registry, tax amount arithmetic) to perform five cumulative validations on extraction results, transforming "how accurate is the model" into "how many errors slipped through." The 3-billion-parameter model fine-tuned on 3,000 synthetic invoices achieved an 83.0% full-invoice exact rate and 86.8% auto-approval rate on the test set, with zero verifiable field errors among approved invoices; without validation, that model had field errors in 12.8% of invoices. Limitations are that validation cannot cover invoice numbers, dates, and line item descriptions, so approved invoices still contain 4.6% unvalidated errors; vertical writing performs notably poorly (40.0%); and the synthetic registry underestimates the risk of "corrupting one digit hitting another real company" (0.29% in the real registry).