Abstract

The intrinsic dimensionality (ID) of multi-dimensional data collections is one of their most fundamental characteristics. Estimates of ID provide an important notion of the complexity of the data, which, in turn, is crucial to selecting the right approach and designing effective machine learning models. There is a wide range of applications for ID estimation, from widely used dimensionality reduction to adversarial attacks, outlier detection and search indices to more theoretical fields like similarity search, discriminability, graph construction, and extreme value theory. However, the notions provided by ID estimations of the data stop at the threshold when designing machine or deep learning models, providing little to no insight when selecting model hyper-parameter configurations. In this work, we explored the idea of using a relation between the intrinsic and extrinsic dimensionality of an image manifold to provide an intuition for selecting an appropriate dropout rate to regularize neural networks in the context of image classification problems. We studied the characteristics of several ID estimators and applied them to image datasets and introduced a new formula to compute values for the dropout rate dependent on ID estimations. We empirically studied the effects of using this new rate by analyzing its effects in the training of several state-of-the-art image classification models on benchmark datasets. We showed that using this technique can consistently improve the performance of several well-established models.