New Breakthrough in Discrete Distribution Estimation: Minimax Risk under Hockey-stick Divergence

First minimax optimal rates for discrete distribution estimation under hockey-stick divergence, revealing polynomial and exponential phase transitions.
This study is the first to systematically investigate minimax estimation of discrete distributions under hockey-stick divergence. The authors consider estimating an unknown discrete distribution from i.i.d. samples and minimizing the worst-case expected hockey-stick divergence, depending on alphabet size k, sample size n, and divergence parameter α. They derive non-asymptotic upper bounds via an add-constant estimator and matching lower bounds using a Bayesian risk argument, proving no estimator can do better. The bounds show two distinct behaviors: polynomial decay when n is sufficiently large relative to k, and exponential growth when n is small. This provides tools for theoretical analysis in differential privacy and clarifies the relationship between sample complexity and parameters.
Estimating discrete distributions is a fundamental task in statistical learning and information theory, but achieving optimal accuracy under different divergence measures has always been a challenge. A recent study is the first to systematically investigate minimax estimation of discrete distributions under hockey-stick divergence, providing non-asymptotic upper and lower bounds and revealing a transition from polynomial to exponential regimes as parameters vary.

Background and Problem Setup
Discrete distribution estimation is widely used in language modeling, bioinformatics, and privacy protection. Traditional research has mostly focused on KL divergence or total variation distance, while hockey-stick divergence, as a type of f-divergence, is particularly important in differential privacy and hypothesis testing. The divergence is defined as: for two distributions P and Q, D_α(P||Q) = ∫ |dP/dQ - α|_+ dQ, where α is a parameter. Although hockey-stick divergence frequently appears in theoretical computer science, the statistical properties of distribution estimation under this divergence have been rarely studied.
This study fills this gap: the authors consider estimating an unknown discrete distribution from independent and identically distributed samples and minimizing the worst-case expected hockey-stick divergence. The problem depends on three key parameters: alphabet size k, number of samples n, and divergence parameter α.
f-divergence is a class of measures of difference between distributions defined by a convex function f, with the general form D_f(P||Q) = ∫ f(dP/dQ) dQ. Common divergences such as KL divergence (f(t)=t log t), total variation distance (f(t)=|t-1|/2), and χ² divergence (f(t)=(t-1)²) all belong to this family. f-divergences have good properties such as non-negativity and joint convexity, but the difficulty of statistical estimation can vary greatly under different f, so research on specific f has independent value.
Add-constant estimator is one of the most classic strategies in discrete distribution estimation: add a constant c (often 1/2 or 1) to the observed count of each symbol, then normalize by dividing by n+ck. When c=1/2, it is the famous Laplace smoothing or Krichevsky–Trofimov estimator. Its advantages are simplicity, no parameter tuning, and achieving minimax optimal rates under measures such as KL divergence. However, the optimal choice of constant c depends on the specific form of the target divergence, and one of the key contributions of this paper is determining the appropriate constant for the asymmetry of hockey-stick divergence.
Main Results: Upper and Lower Bounds
The authors obtain non-asymptotic upper bounds using an add-constant estimator, i.e., adding a constant to the count of each symbol and then normalizing. This simple estimator achieves the optimal rate under the polynomial regime. Meanwhile, using a Bayesian risk argument, the authors derive lower bounds, proving that no estimator can surpass this lower bound. Together, the upper and lower bounds show that as parameters vary, the minimax risk exhibits two distinct behaviors: polynomial and exponential.
Specifically, when the sample size n is sufficiently large relative to the alphabet size k, the risk decays at a polynomial rate; when n is small, the risk grows exponentially. This phase transition phenomenon is closely related to the properties of hockey-stick divergence, as this divergence is very sensitive to differences in the tails of distributions.
Technical Contributions and Proof Ideas
In the upper bound analysis, the choice of constant in the add-constant estimator is crucial. The authors carefully balance bias and variance and adjust for the asymmetry of hockey-stick divergence. For the lower bound, they construct a Bayesian prior such that the posterior risk of any estimator is at least as large as the Bayesian risk under the constructed prior, thereby obtaining a minimax lower bound. The proof uses tools such as convexity and the Le Cam method.
A highlight of this work is that the upper and lower bounds match within constant factors, thus characterizing the exact order of the minimax risk. Moreover, the results apply to all α>0, including the case α=1 where it reduces to a multiple of total variation distance.
The Le Cam method is a classic tool for proving minimax lower bounds. Its core idea is to transform the minimax risk lower bound into a hypothesis testing problem between a pair of indistinguishable distributions. Specifically, by constructing two prior distributions (or two specific distributions) such that any estimator has large average risk over these two distributions, a lower bound is obtained. This method is particularly suitable for non-asymptotic settings because it does not rely on asymptotic normal approximations and can provide explicit bounds valid for any sample size n. This paper combines it with Bayesian risk arguments, adapted to the characteristics of hockey-stick divergence.
Practical Significance and Future Directions
This research provides theoretical tools for privacy-preserving data analysis. For example, in differential privacy, hockey-stick divergence is often used to define privacy loss. Precise estimation error bounds help design more efficient privacy mechanisms. Future work may include extensions to other f-divergences, consideration of adaptive sampling, or study of high-dimensional structures.
Although this study is theoretical, it provides practitioners with clear guidance on sample complexity: under hockey-stick divergence, how the required sample size varies with alphabet size and accuracy requirements. For researchers engaged in distribution estimation or privacy protection, this is a fundamental work worth attention.
In summary, this study is the first to reveal the minimax rate of discrete distribution estimation under hockey-stick divergence, with matching upper and lower bounds and a phase transition phenomenon, laying a theoretical foundation for future research.
In differential privacy, privacy loss is typically characterized by hockey-stick divergence (also called δ-approximate max divergence): a randomized mechanism M satisfies (ε,δ)-differential privacy if and only if for any adjacent datasets, the hockey-stick divergence (with parameter α=e^ε) of their output distributions does not exceed δ. Therefore, accurately estimating hockey-stick divergence helps empirically audit privacy guarantees from data or design mechanisms with better privacy budget allocation. The minimax rates given in this paper are directly related to how accurately privacy loss can be assessed for a given sample size, providing a theoretical basis for privacy auditing.
Related articles

Gemini 4 Hydrogen Released: A 2.5x Faster Everyday AI Model
Google releases Gemini 4 Hydrogen, claiming 2.5x the speed of Gemini 3.8 Flash, with four-tier thinking control, a million-token context, and Orchestrator multi-model collaboration. API pricing is $1.50 per million input tokens.

Gemini 4 Hydrogen Hands-On: Google's Fastest and Cheapest AI Model
Hands-on with Gemini 4 Hydrogen: the fastest, cheapest model in the Gemini 4 family. We break down the Carbon/Argon/Hydrogen split, API pricing traps, free-tier limits, and the cost trade-offs of four thinking levels—with a real bug-debugging demo.

Gemini 4 Hydrogen vs Sonnet Comparison Test: Uses Only 1% of Weekly Quota, Performance on Par
Hands-on test: Gemini 4 Hydrogen vs Sonnet on the same CRUD task—Hydrogen used only 1% weekly quota with comparable performance and better UI details, while Anthropic's model used 7%. Developer says daily tasks don't require expensive models.