We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first.In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. Wealso demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data it-self plays an important role in determining the degree of memorization. Read More

#assurance, #neural-networks

The roles played by learning and memorization represent an important topic in deep learning research. Recent work on this subject has shown that the optimization behavior of DNNs trained on shuffled labels is qualitatively different from DNNs trained with real labels. Here, we propose a novel permutation approach that can differentiate memorization from learning in deep neural networks (DNNs) trained as usual (i.e., us-ing the real labels to guide the learning, rather than shuffled labels). The evaluation of weather the DNN has learned and/or memorized, happens in a separate step where we compare the predictive performance of a shallow classifier trained with the features learned by the DNN, against multiple instances of the same classifier, trained on the same input, but using shuffled labels as outputs. By evaluating these shallow classifiers in validation sets that share structure with the training set, we are able to tell apart learning from memorization. Application of our permutation approach to multilayer perceptrons and convolutional neural networks trained on image data corroborated many findings from other groups. Most importantly,our illustrations also uncovered interesting dynamic patterns about how DNNs memorize over increasing numbers of training epochs, and support the surprising result that DNNs are still able to learn, rather than only memorize, when trained with pure Gaussian noise as input. Read More

#assurance, #neural-networks

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family, or to the regularization techniques used during training.Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice.Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a ran-dom labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.We interpret our experimental findings by comparison with traditional models. Read More

#assurance, #neural-networks

Deep Neural Networks (DNNs) have been widely applied invarious recognition tasks. However, recently DNNs have been shown tobe vulnerable against adversarial examples, which can mislead DNNs tomake arbitrary incorrect predictions. While adversarial examples are wellstudied in classification tasks, other learning problems may have differ-ent properties. For instance, semantic segmentation requires additionalcomponents such as dilated convolutions and multiscale processing. Inthis paper, we aim to characterize adversarial examples based on spatialcontext information in semantic segmentation. We observe that spatialconsistency information can be potentially leveraged to detect adversar-ial examples robustly even when a strong adaptive attacker has accessto the model and detection strategies. We also show that adversarialexamples based on attacks considered within the paper barely transferamong models, even though transferability is common in classification.Our observations shed new light on developing adversarial attacks anddefenses to better understand the vulnerabilities of DNNs. Read More

#assurance, #neural-networks

Recent studies have highlighted adversarial examples as a ubiquitous threat to dif-ferent neural network models and many downstream applications. Nonetheless,as unique data properties have inspired distinct and powerful learning principles,this paper aims to explore their potentials towards mitigating adversarial inputs.In particular, our results reveal the importance of using the temporal dependencyin audio data to gain discriminate power against adversarial examples. Tested onthe automatic speech recognition (ASR) tasks and three recent audio adversarialattacks, we find that (i) input transformation developed from image adversarial de-fense provides limited robustness improvement and is subtle to advanced attacks;(ii) temporal dependency can be exploited to gain discriminative power againstaudio adversarial examples and is resistant to adaptive attacks considered in ourexperiments. Our results not only show promising means of improving the robust-ness of ASR systems, but also offer novel insights in exploiting domain-specificdata properties to mitigate negative effects of adversarial examples. Read More

#assurance, #neural-networks

Deep neural networks (DNNs) have been foundto be vulnerable to adversarial examples resultingfrom adding small-magnitude perturbations to in-puts. Such adversarial examples can mislead DNNsto produce adversary-selected results. Different at-tack strategies have been proposed to generate ad-versarial examples, but how to produce them withhigh perceptual quality and more efficiently re-quires more research efforts. In this paper, wepropose AdvGAN to generate adversarial exam-ples with generative adversarial networks (GANs),which can learn and approximate the distributionof original instances. For AdvGAN, once the gen-erator is trained, it can generate perturbations effi-ciently for any instance, so as to potentially acceler-ate adversarial training as defenses. We apply Adv-GAN in both semi-whitebox and black-box attacksettings. In semi-whitebox attacks, there is no needto access the original target model after the gener-ator is trained, in contrast to traditional white-boxattacks. In black-box attacks, we dynamically traina distilled model for the black-box model and op-timize the generator accordingly. Adversarial ex-amples generated by AdvGAN on different targetmodels have high attack success rate under state-of-the-art defenses compared to other attacks. Ourattack has placed the first with 92.76% accuracy ona public MNIST black-box attack challenge. Read More

#assurance, #neural-networks