Abstract

Artificial intelligence, and especially deep learning, have had a major spotlight shined on them in recent years. Our goal is to mathematically and computationally explore some of its potential limits - specifically, those related to identity effects and generalization. Identity effects refer to the ability to recognize specific patterns, such as checking whether two sounds are identical, something human brains are very adept at handling from a very young age. It has been observed that depending on how the input data is ``encoded”, that is what type of vectors differentiate the pair of objects to be classified a deep neural net might fail to generalize identity effects outside the training set. Namely, given that matching objects are valid pairs, and non-matching objects are not, some types of encodings will allow the network to generalize to inputs it has never seen before, and some will not. Previous research shows that the existence of a specific orthogonal transformation on the encodings can predict impossibility of generalization. We explore mathematically and then test numerically whether approximate orthogonality leads to predictable loosening of the impossibility, eventually finding that approximate orthogonality leads to approximate impossibility. In particular, adding random noise to traditional encodings such as the ``one-hot'' encoding can help neural networks generalize outside the training set.