Abstract

Rapid advancements in artificial intelligence algorithms have sharpened the focus on street signs due to their prevalence. This research was driven by beneficial applications of end-to-end systems to humans, municipal agencies, and automobiles. However, the variation of materials, shapes, colors, and fonts in some signs, such as shop signboards, have presented complicated challenges to AI-based systems to detect and classify them. Previous studies built classification models by considering the whole storefront. Their classification results were negatively impacted by the inclusion of other components within the storefront. This research focuses on shop signboards as they are much more consistent.

The main objective of this research is to detect and classify shop signboards based on deep learning and machine learning techniques. To achieve that, data acquisition was necessary for models training purposes. Therefore, the Shop Signboard ShoS dataset was collected from Google street images. A total of 10k store signboards were captured within 7500 images. All the collected images were fully annotated and made available for the public for several research purposes.

Then, the Shop Signboard Detection and Classification Framework SSDCF was designed and built to tackle most of the existing challenges. Three main components were fully implemented and evaluated: signboard detector, text extractor, and shop classifier to classify commercial stores based on the textual information. For signboard detector, two models were trained and tested utilizing the ShoS dataset. Findings surpassed the performance of YOLOv3 without any color preparation. For text extractor, the evaluation of Google Vision OCR showed better results even with the existence of influential factors, such as stylized fonts and skewed images. For shop classifier, out of the two trained and tested classifiers, SVM showed great performance even with classes that have some difficulty factors. The performance of the classifier had been enhanced by 4\% approximately after adding the augmented data which was generated by the Random Deletion method and a novel Thesauruses-inspired method named \textit{OCR-Thesauruses}. Each component has been trained and tested individually at first. Then, the full end-to-end framework was implemented and evaluated using the SVT public dataset, and the outcome reached an F1-score=89\%. The classification performance was also compared with human performance based on the texts extracted from the signs. Human subjects were provided with textual information only and were not exposed to shop sing images. The results showed that our classifier exceeded human performance by about 15\% due to the prior knowledge the classifier learned from all text data during training.

Finally, the results of the second component of our framework, the text extractor, were statistically analyzed to check the impact of typeface styles used in shop signboards on the recognition rates. The findings showed a significant association between the typeface style and the recognition rate. So, it is recommended to use ''Serif" and ''Sanserif" styles over ''Script" and ''Decorative" in designing shop signboards. If using stylized fonts is a must, it is advised to add keywords that distinguish a store class from another using a better typeface design, such as ''Serif" or ''Sanserif" styles.