View article

Deep learning architectures have enabled unprecedented advances in a wide range of artificial intelligence-related applications. The empirical success of these architectures has posed fundamental riddles regarding their operation in the front lines of modern theoretical machine learning research. Related theoretical efforts can be broadly divided into (i) explaining the observed success of deep learning architectures and (ii) harnessing these insights for improving their operation. In this chapter, we outline a tensor analysis-based contribution to understanding and improving the expressivity of prominent deep learning architecture classes. We detail a successful proof methodology which includes analyzing grid tensors of the functions realized by deep learning architecture classes, which was applied for convolutional, recurrent, and self-attention networks. The rank of an architecture's grid tensor is used for bounding …