Graduate Student Seminar: Convergence of Expected Suboptimality and Asymptotic Distribution of Adam-type Optimizers

13596
A group of students attend a seminar

Graduate Student Seminar: Convergence of Expected Suboptimality and Asymptotic Distribution of Adam-type Optimizers

Yi Luo, PhD Student in Statistics and Data Science at Washington University in St. Louis
Many statistical estimators are computed by stochastic optimization, making the statistical properties of the optimization output central to their use. For Adam-type updates, the current gradient enters both the momentum buffer and the second-moment state used to normalize it.  We establish convergence rates and asymptotic normality while retaining this dependence and deriving the required trajectory control. The general result uses a fixed first-moment decay rate $\beta_1\in[0,1)$, a second-moment decay rate $\beta_{2,t}=1-1/t$, and base stepsizes $\alpha_t=\alpha t^{-\gamma}$ with $1/2<\gamma<1$. Under global strong convexity and smoothness of the population risk and a uniform conditional sub-exponential bound on martingale-difference gradient noise, we derive a path wise envelope and unconditional cumulative and pointwise expected excess-risk bounds of orders $T^{1-\gamma}$ and $t^{-\gamma}$, respectively, up to logarithmic factors, without assuming bounded iterates or bounded stochastic gradients. With additional local Hessian and conditional covariance regularity, these estimates yield asymptotic linearity of the full Polyak--Ruppert (PR) average at the population minimizer, with the classical sandwich covariance in fixed dimension.  Independent-sample covariance estimation provides confidence intervals for fixed linear functionals.
 
A separate diagonal quadratic model with zero momentum and bounded i.i.d.\ noise variables shows why current-step normalization matters for centering. For $\beta_{2,t}=1-t^{-\kappa}$, $0<\kappa\le1$, we identify an explicit third-moment correction.  It is negligible at the root-$T$ scale for $\kappa>1/2$; when its coefficient is nonzero, it shifts the Gaussian center at $\kappa=1/2$ and determines a slower convergence scale below the threshold.  Lagged normalization has a centered limit under the sufficient condition $\gamma+\kappa>1$. Fixed second-moment decay rates are outside these results. Numerical experiments illustrate the convergence and distributional conclusions.