In our paper to appear at this year's NeurIPS, we consider a supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer neural network with random i.i.d inputs. We prove a formula for the generalization error achieved by l2 regularized classifiers that minimize a convex loss. We also design an optimal loss and regularizer that provably leads to Bayes-optimal generalization error.