Abstract

This thesis introduces two types of adaptable human–computer interaction methods in single-subject and multi-subject unsupervised environments.

In a single-subject environment, a single shot multi-box detector (SSD) is used to detect the hand's regions of interest (RoI) in the input frame.
The RoI detected by the SSD model is fed to a convolutional neural network (CNN) to classify the right-hand gesture.
In a crowded environment, a region-based convolutional neural network (R-CNN) detects subjects in the frame and the RoI of their faces. These regions of interest are then fed to a facial recognition module to find the main subject in the frame.
After the main subject is found, the R-CNN model feeds the right-hand RoI of the main subject to the CNN to classify the right-hand gesture.
In both single-subject and multi-subject environments, a fixed number of gestures is used to describe specific commands from the right-hand gestures to the vehicle (take off, land, hover, etc.).

A smart glove on the left hand is used for more precise vehicular control.
A motion-processing unit (MPU) and a set of four flex sensors are used in the smart glove to produce discrete and continuous signals based on the bending value of each finger and the roll angle of the left hand.
The generated discrete signals with the flex sensors are fed to a support vector machine (SVM) to classify the left-hand gesture, and the generated continuous signals with the flex sensors and the MPU module are used to determine some characteristics of the gesture command such as throttle and angle of the vehicle.

Three simultaneous validation layers have been implemented, including (1) Human-Based Validation, where the main subject can reject the robot’s behavior by sending a command through the smart glove, (2) Classification Validation, where the main subject can retrain the classifier classes at will, and (3) System Validation, which is responsible for finding the error's source when the main subject performs a specific command through the smart glove.

The proposed algorithm is applied to groups consisting of one, two, three, and four subjects including the main subject to validate and investigate the behavior of the algorithm in various desirable and undesirable situations.
The proposed algorithm is implemented on an Nvidia Jetson AGX Xavier GPU.