Login | Register

Using Intrinsic Dimensionality to Improve Dropout Regularization


Using Intrinsic Dimensionality to Improve Dropout Regularization

Fernandez Cruz, Javier (2021) Using Intrinsic Dimensionality to Improve Dropout Regularization. Masters thesis, Concordia University.

[thumbnail of Fernandez_MCompSc_F2021.pdf]
Text (application/pdf)
Fernandez_MCompSc_F2021.pdf - Accepted Version
Available under License Spectrum Terms of Access.


The intrinsic dimensionality (ID) of multi-dimensional data collections is one of their most fundamental characteristics. Estimates of ID provide an important notion of the complexity of the data, which, in turn, is crucial to selecting the right approach and designing effective machine learning models. There is a wide range of applications for ID estimation, from widely used dimensionality reduction to adversarial attacks, outlier detection and search indices to more theoretical fields like similarity search, discriminability, graph construction, and extreme value theory. However, the notions provided by ID estimations of the data stop at the threshold when designing machine or deep learning models, providing little to no insight when selecting model hyper-parameter configurations. In this work, we explored the idea of using a relation between the intrinsic and extrinsic dimensionality of an image manifold to provide an intuition for selecting an appropriate dropout rate to regularize neural networks in the context of image classification problems. We studied the characteristics of several ID estimators and applied them to image datasets and introduced a new formula to compute values for the dropout rate dependent on ID estimations. We empirically studied the effects of using this new rate by analyzing its effects in the training of several state-of-the-art image classification models on benchmark datasets. We showed that using this technique can consistently improve the performance of several well-established models.

Divisions:Concordia University > Gina Cody School of Engineering and Computer Science > Computer Science and Software Engineering
Item Type:Thesis (Masters)
Authors:Fernandez Cruz, Javier
Institution:Concordia University
Degree Name:M. Comp. Sc.
Program:Computer Science
Date:8 July 2021
Thesis Supervisor(s):Fevens, Thomas
ID Code:988539
Deposited By: Javier Fernandez Cruz
Deposited On:29 Nov 2021 16:44
Last Modified:29 Nov 2021 16:44
All items in Spectrum are protected by copyright, with all rights reserved. The use of items is governed by Spectrum's terms of access.

Repository Staff Only: item control page

Downloads per month over past year

Research related to the current document (at the CORE website)
- Research related to the current document (at the CORE website)
Back to top Back to top