Abstract

Machine learning is a technology that builds predictive models from data, allowing generalization to unseen cases. At the core of every learning problem lies an optimization challenge, and solving these problems reliably is crucial to resolving the obstacles surrounding machine learning. Primarily, conventional optimization algorithms employed for training machine learning frequently are often ill-suited for various applications. Concerted efforts are needed to refine and optimize various components of machine learning training. This thesis explores fundamental optimization algorithms across various machine learning applications. By enhancing optimization schemes, including optimizers and model compression techniques, the resilience and effectiveness of machine learning applications can be improved.

The first segment of the thesis introduces an innovative low-rank matrix factorization scheme aimed at enhancing the scalability of machine learning. Gaussian Process Regression is used as the machine learning
model to scale with low rank matrix factorization in this section of the thesis due to its lightweight nature, which enables the incremental updating of model parameters online prior to prediction. A nonconvex
formulation of a low-rank matrix factorization (SRLSMF) with convex formulation of a low-rank matrix factorization initialization (ℓ1-SRLSMF), is advocated to scale Gaussian Process Regression (GPR). Thus, by
employing convex nonconvex low rank matrix factorization to scale a given the Gaussian Process Regression model, the model can avoid local minima and converge to a solution with smaller recovery residuals.
Also, the running time of convex nonconvex low rank matrix factorization is expected to be smaller than that of applying nonconvex low rank matrix factorization alone under the same stopping criterion. To the best of our knowledge, the machine learning method proposed in this thesis is the first to exploit nonconvex formulation of a low-rank matrix factorization (SRLSMF) with convex formulation of a low-rank matrix factorization initialization (ℓ1-SRLSMF) to scale machine learning in machine learning domain. Recognizing the cost-prohibitive nature of standard eigen decomposition for online Gaussian Process Regression covariance update, we implement incremental eigen decomposition within the ℓ1-SRLSMF and SRLSMF Gaussian Process Regression methodologies. Finally, an illustration of the potential applications in suturing, knot-tying and needle passing task using kinematic dataset is provided.

In the latter part of the thesis, a novel adaptive stochastic gradient descent (ASGD) method, which leverages the non-uniform p-norm concept to train machine learning is presented. The proposed ASGD assigns distinct categories of coordinate values with varying base learning rates, thereby enabling the training of machine learning models. Additionally, theoretical guarantees for the efficacy of the proposed ASGD method in convex and nonconvex settings is discussed. The ability of the proposed ASGD approach to detect suturing gestures within the remote surgical gesture recognition task is discussed.

Finally, potential avenues for future research are outlined.