Poster

Hyperparameter Tuning with Renyi Differential Privacy

Nicolas Papernot · Thomas Steinke

Keywords: [ differential privacy ]

award Outstanding Paper
[ Abstract ]
[ Visit Poster at Spot H1 in Virtual World ] [ OpenReview
Mon 25 Apr 10:30 a.m. PDT — 12:30 p.m. PDT
 
Oral presentation: Oral 1: Learning in the wild, Reinforcement learning
Mon 25 Apr 5 p.m. PDT — 6:30 p.m. PDT

Abstract:

For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood. However, few studies have reasoned about the privacy leakage resulting from the multiple training runs needed to fine tune the value of the training algorithm’s hyperparameters. In this work, we first illustrate how simply setting hyperparameters based on non-private training runs can leak private information. Motivated by this observation, we then provide privacy guarantees for hyperparameter search procedures within the framework of Renyi Differential Privacy. Our results improve and extend the work of Liu and Talwar (STOC 2019). Our analysis supports our previous observation that tuning hyperparameters does indeed leak private information, but we prove that, under certain assumptions, this leakage is modest, as long as each candidate training run needed to select hyperparameters is itself differentially private.

Chat is not available.