Insights into the Role of the Initialisation and Curriculum through Parity Targets

Title

Insights into the Role of the Initialisation and Curriculum through Parity Targets

Speaker

Elisabetta Cornacchia - INRIA Paris, Argo project-team

Abstract

Parity functions are a common benchmark for testing learning algorithms. Recent works have shown that neural networks with regular architectures trained via gradient descent can efficiently learn low-degree parities, but struggle with intermediate-degree parities (1 ≪ k ≪ d, where d is the ambient dimension and k is the parity degree). The learnability of high-degree parities (k=d − O(1)) remains unsettled. While learnable in the Statistical Query (SQ) setting, we find that, for regular neural networks, the initialization of the weights is crucial: Rademacher initialization enables efficient learning, but even slight perturbations make them hard to learn. We then use the parity benchmark to demonstrate benefits of curriculum learning, where simpler samples are presented before more complex ones. When training on a mixture of sparse and dense inputs, shallow networks can efficiently learn parities if trained on sparse examples first, whereas if trained on unordered samples they require extra steps to succeed. Finally, we discuss how some of these findings extend to other Boolean functions.

Bio

Elisabetta Cornacchia is a postdoctoral researcher in the Argo project-team at INRIA Paris and the Department of Computer Science at École Normale Supérieure (DI ENS). She earned her PhD in 2023 at EPFL, where she was supervised by Professor E. Abbé. From 2023 to 2024, she held a postdoctoral position at MIT under the supervision of Professor E. Mossel.

When

Monday December 2nd, 14:30

Where

Room 322, UniGe DIBRIS/DIMA, Via Dodecaneso 35