Abstract

Increasingly complex mobile applications require more accurate methods of automated GUI testing.
Traditional testing frameworks relying on fixed delays or pixel-based image comparison methods
have a lot of limitations. Most of the time, these methods misclassify GUI rendering states, which
leads to false positives. This thesis proposes a new approach to these issues by the inference of the
rendering state of GUIs. Drawing on large-scale pre-trained image classification models like Vision
Mamba, it enables the accurate classification between rendered GUIs. It does this through a fine-
tuning process of a large model. It also involves more sophisticated model-training techniques to
ensure that the best is obtained. Instead, this system architecture’s semantic examination of GUI
elements goes deep into more meaningful matches of context and visual information well beyond
anything at the level of pixels.
The efficiency and accuracy of GUI testing will increase significantly with the proposed approach.
That is different from fixed throttles that introduce unnecessary delays: it guarantees automated
tests execute on fully rendered GUIs, which, in turn, reduces false positives. With this enhancement,
development teams will save time and resources. Deep learning-based classification introduces a
dynamic system that changes according to various GUI rendering scenarios; therefore, it is also
more robust than what is already available.
The contributions of this thesis are three-fold: it proposes a new deep learning-based approach
for inferring GUI rendering states, develops a high-quality dataset to support fine-tuning models,
and performs an in-depth comparative analysis of large image classification models for GUI testing.