Generalization error of min-norm interpolators in transfer learning

Abstract: Min-norm interpolators naturally emerge as implicit regularized limits of modern machine learning algorithms. Recently, their out-of-distribution risk was studied when test samples are unavailable during training. However, in many applications, a limited amount of test data is typically available during training. The properties of min-norm interpolation in this setting are not well understood. In this talk, I will present a characterization of the bias and variance of pooled min-L2-norm interpolation under covariate and model shifts. I will demonstrate that the pooled interpolator captures both early fusion and a form of intermediate fusion. Our results have several implications. Under model shift, adding data always hurts prediction when the signal-to-noise ratio is low. However, for higher signal-to-noise ratios, transfer learning helps as long as the shift-to-signal ratio lies below a threshold that I will define. I will also present data-driven methods to determine (i) when the pooled interpolator outperforms the target-based interpolator, and (ii) the optimal number of target samples that minimizes the generalization error. Our results further show that under covariate shift, if the source sample size is small relative to the dimension, heterogeneity between domains improves the risk. Time permitting, I will introduce a novel anisotropic local law that helps achieve some of these characterizations and may be of independent interest in random matrix theory. 

Bio: Pragya Sur is an Assistant Professor of Statistics at Harvard University, specializing in high-dimensional statistics and statistical machine learning. Her research is funded by an NSF DMS Award, the Eric and Wendy Schmidt Fund for Strategic Innovation, a William F. Milton Fund Award, and a Dean’s Competitive Fund for Promising Scholarship. She currently serves as an Associate Editor for Statistical Science. Between ’22-’24, she led the IMS New Researchers Group. In ’23, she was named an International Strategy Forum (ISF) Fellow, a program that recognizes rising leaders aged 25 – 35 from Africa, Asia, North America, and Europe. In ’21, she was an invited speaker at the National Academies’ symposium on Mathematical Challenges for Machine Learning and Artificial Intelligence and was also an invited long-term participant at the Simons Institute for the Theory of Computing, UC Berkeley. She earned her Ph.D. in Statistics from Stanford University in ’19, where she received the Theodore W. Anderson Theory of Statistics Dissertation Award (’19) and the Ric Weiland Graduate Fellowship (’17).