Does Financial Speech ASR Need Continual Learning Regularization? An Empirical Study of EWC and mini-KFAC

An Overturned Intuition: Task Complementarity

When we deploy speech recognition systems in the real world and continuously update them with new data, a classic concern emerges — catastrophic forgetting. As models learn new knowledge, they often erase previously acquired capabilities. Earnings call transcription is a textbook example of this continual learning (CL) dilemma: an ASR system must continuously absorb new recordings while risking forgetting previously learned financial speech patterns.

Yet this study reaches the opposite conclusion. Researchers adapted a general-domain Whisper-small model to the Earnings22 benchmark dataset and found that across all evaluated methods, performance on in-domain financial speech held-out sets consistently improved during adaptation — rather than degrading. In other words, the anticipated forgetting never materialized.

The central explanation for this phenomenon is task complementarity. When a new task does not conflict with existing knowledge — or even reinforces it — the foundational premise of continual learning no longer holds. In such cases, imposing regularization constraints becomes counterproductive.

Whisper is a large-scale multilingual speech recognition model released by OpenAI in 2022. It uses an encoder-decoder Transformer architecture and was weakly supervised on approximately 680,000 hours of multilingual audio from the internet. Whisper-small is a mid-sized variant with roughly 244 million parameters — computationally accessible while retaining strong general-purpose recognition capability, making it a common starting point for domain-specific fine-tuning. Earnings22 is a benchmark dataset designed specifically for evaluating financial-domain speech recognition, consisting of real earnings call recordings from publicly listed companies. It features dense industry terminology, numerical expressions, and non-native accents, with word error rates (WER) typically higher than general-domain settings, making it a standard testbed for measuring ASR generalization in specialized speech scenarios.

Why Regularization Can Backfire

One of the dominant approaches in continual learning is regularization-based methods, the most representative being Elastic Weight Consolidation (EWC). The idea is to estimate each parameter's importance to old tasks and penalize large deviations from those parameter values during new task training, thereby protecting previously learned knowledge.

This mechanism carries an implicit assumption: new and old tasks compete, and unconstrained updates will cause forgetting. However, in the specific context of earnings call speech, general-domain and financial-domain speech are highly correlated at the level of acoustic and linguistic features. The adaptation process itself constitutes positive transfer. Applying EWC to "lock" parameters near their old optima in this scenario actively hinders the model from absorbing domain-specific financial patterns, degrading overall performance.

The study consequently offers an important methodological recommendation: before applying any continual learning constraints, first assess whether genuine complementarity or conflict exists between tasks. Blindly applying regularization may solve a problem that doesn't exist while undermining real gains.

The mathematical core of EWC is the Fisher Information Matrix (FIM). The FIM measures how much each model parameter contributes to the curvature of the loss function — intuitively, higher curvature means a parameter is more sensitive to the old task's predictions and more costly to change. EWC uses this importance as a penalty coefficient added to the new task's loss, taking the form $L_{new} + \lambda \sum_i F_i (\theta_i - \theta_{i}^{})^2$, where $\theta^{}$ is a snapshot of parameters after old task training and $\lambda$ controls the degree of protection. The problem is that this penalty term fundamentally assumes the old-task optimum conflicts with the new-task optimum — once the two actually point in the same direction, the penalty is not only redundant but artificially introduces resistance in parameter space, impeding convergence along the correct gradient direction.

When Regularization Is Needed: mini-KFAC Outperforms Diagonal EWC

Of course, task complementarity is not a universal law. In scenarios where forgetting is a genuine risk and regularization is warranted, the choice of method becomes critical. The study compared two approaches:

Limitations of Diagonal EWC

Standard EWC typically uses a diagonal Fisher information matrix to approximate parameter importance. This approach is computationally cheap but assumes parameters are mutually independent, ignoring inter-parameter correlations. For the highly coupled structure of neural networks, this approximation is often too coarse, leading to inaccurate importance estimates and unstable optimization.

The mini-KFAC Improvement

The study introduces a layer-wise mini-Kronecker-Factored Approximate Curvature (mini-KFAC) approximation. KFAC uses Kronecker product decomposition to approximate the block structure of the Fisher information matrix, capturing second-order curvature information among parameters within each layer — a much closer approximation to the true parameter importance distribution compared to the diagonal approach.

Experimental results show that in scenarios requiring regularization, mini-KFAC EWC achieves:

• Lower word error rate (WER)
• More stable optimization

This means that when continual learning constraints are unavoidable, replacing the coarse diagonal approximation with a more precise curvature estimate is a worthwhile priority.

Kronecker-Factored Approximate Curvature (KFAC) was originally proposed by Martens and Grosse in 2015 to accelerate natural gradient optimization. Its core idea is to approximately factorize each layer's Fisher information matrix as the Kronecker product of two smaller matrices: $F_l \approx A_{l-1} \otimes G_l$, where $A_{l-1}$ is the second-order statistics of the layer's input activations and $G_l$ is the second-order statistics of the output gradients. This factorization reduces the originally $O(p^2)$ matrix operations to two independent smaller-matrix computations, making block-wise second-order information tractable at real network scales. "mini-KFAC" is a further simplification of full KFAC that reduces the sample count used for per-layer statistical estimation, controlling memory and compute overhead while achieving a more practical balance between accuracy and efficiency — particularly well-suited for updating per-layer importance estimates in continual learning scenarios.

Practical Engineering Implications

The value of this study lies not just in incremental metric improvements, but in its reflection on continual learning methodology. Many engineering practices "default" to enabling regularization to prevent forgetting — but the findings here serve as a reminder:

1. Forgetting is not inevitable. When incremental data belongs to a coherent domain aligned with the model's existing capabilities, adaptation is more likely to be positive reinforcement than destructive interference.
2. Regularization has costs. It can limit a model's ability to fully learn from new data — a poor trade-off when tasks are complementary.
3. Method selection matters. When regularization is genuinely needed, second-order approximations like mini-KFAC generally outperform simple diagonal approaches.

For practitioners maintaining ASR systems in vertical domains like financial speech transcription, this means the first step in designing model update pipelines should not be reaching directly for forgetting-prevention mechanisms. Instead, quantitatively assess the relationship between old and new tasks first. Only after confirming a genuine conflict should you introduce appropriate regularization — that is the path to a well-calibrated balance between stability and performance.

Summary

Using earnings call transcription as a real-world testbed, this study challenges the reflexive assumption in continual learning that forgetting must always be defended against. The core finding is: task complementarity should be assessed before constraints are applied. When adapting from general-domain to financial speech, the adaptation yields synergistic gains — regularization is counterproductive. And when regularization is genuinely required, mini-KFAC EWC achieves lower WER and more stable optimization compared to conventional diagonal EWC. Together, these findings provide a pragmatic decision framework for continuously updating domain-specific ASR systems.