Abstract

The design of handwriting recognition systems has been widely investigated in pattern recognition and machine learning literature. It was first attempted to enhance the system's performance by improving the recognition rate to reach $100\%$ which has not achieved yet. Despite the low misclassification error rate, there are still some misclassified test samples. This imposes a very high cost on the whole recognition system. The cost has to be reduced as much as possible which consequently leads to the consideration of reject option to prevent the recognition system from classifying test samples with high prediction uncertainty.

The main contribution of this thesis is to propose a novel multistage recognition system that is capable of producing true prediction probability outputs and then reject test samples accordingly. An argument is supported that principally formulated probabilistic classifiers are the best reliable candidates to be utilized in the consideration of reject option. The implementation of reject option based on either non-probabilistic classifier's output score or conversion to probability measures is prone to mistake when compared to an accurate prediction probability output.

The Convolutional Neural Network (CNN) is utilized as the automatic feature extractor that can properly harness the spatial correlation of the input raw handwritten images and extract a feature vector with strong discriminative properties. The SVM is used as a powerful classifier to accurately deal with the issue of big data sets. The authentic intuition of extracting the most informative training samples by using the distinguished support vector set from the SVM is also proposed.

The Gaussian process classifier (GPC) in the Bayesian nonparametric modeling framework is introduced as the core element of the whole recognition system that can reliably provide an accurate estimate of the posterior probability of the class membership. Experiments under various inference methods, likelihood functions, covariance functions, and learning approaches are conducted in the hope of finding the best model configuration and parameterization. The models are evaluated on two popular handwritten numeral data sets known as MNIST and CENPARMI. The best GPC model in this multistage framework on MNIST can reach $100\%$ reliability rate with the lowest rejection rate of $1.48\%$, the best result achieved in the field.

Another inherently probabilistic classifier, known as relevance vector machine (RVM), is also investigated. The RVM is formulated through the sparse Bayesian linear modeling to classification problems and it produces reliable prediction probability outputs. However, In comparison of the GPC with RVM, this argument is experimentally supported that the sparsity is not capable of improving the rejection performance on the data sets.