In the last few years, rapid development of deep learning method has boosted the performance of face recognition systems. However, face recognition still suffers from a diverse variation of face images, especially for the problem of face identification. The high expense of labelling data makes it hard to get massive face data with accurate identification information. In real-world applications, the collected data are mixed with severe label noise, which significantly degrades the generalization ability of deep learning models. In this paper, to alleviate the impact of the label noise, we propose a robust deep face recognition (RDFR) method by automatic outlier removal. The noisy faces are automatically recognized and removed, which can boost the performance of the learned deep models. Experiments on large-scale face datasets LFW, CCFD, and COX show that RDFR can effectively remove the label noise and improve the face recognition performance.