June 16, 2021


Dedicated Forum to help removing adware, malware, spyware, ransomware, trojans, viruses and more!

Curse of Dimensionality in Unconstrained Private Convex ERM. (arXiv:2105.13637v1 [cs.LG])

We consider the lower bounds of differentially private empirical risk
minimization for general convex functions in this paper. For convex generalized
linear models (GLMs), the well-known tight bound of DP-ERM in the constrained
case is $tilde{Theta}(frac{sqrt{p}}{epsilon n})$, while recently,
cite{sstt21} find the tight bound of DP-ERM in the unconstrained case is
$tilde{Theta}(frac{sqrt{text{rank}}}{epsilon n})$ where $p$ is the
dimension, $n$ is the sample size and $text{rank}$ is the rank of the feature
matrix of the GLM objective function. As $text{rank}leq min{n,p}$, a
natural and important question arises that whether we can evade the curse of
dimensionality for over-parameterized models where $nll p$, for more general
convex functions beyond GLM. We answer this question negatively by giving the
first and tight lower bound of unconstrained private ERM for the general convex
function, matching the current upper bound
$tilde{O}(frac{sqrt{p}}{nepsilon})$ for unconstrained private ERM. We also
give an $Omega(frac{p}{nepsilon})$ lower bound for unconstrained pure-DP ERM
which recovers the result in the constrained case.