Layerfold: A Python Library to Reduce the Depth of Neural Networks

Abstract

Large-scale models are the backbone of Computer Vision and Natural Language Processing, their generalizability allows for transfer learning and deployment in different scenarios. Their large size, however, means that reducing their computational and memory demands remains challenging.Recent research proposes to achieve ``layer collapse'', a condition where multiple layers can be combined due to the collapse of non-linearities to linear operators. While this is an important discovery, most studies remain theoretical, often replacing non-linearities with simple identity functions and not providing a real implementation of the more compact architecture.Our contribution is LayerFold, a library that studies and concretely implements collapsed layers merging. We address typical cases, from fully connected to convolutional layers, discussing constraints and prospective challenges. Our tests on edge devices reveal that merely reducing network depth doesn't always result in faster computation, even when GPU-equipped. This work raises important warnings and opens the door to further advancements in efficient model deployment.

Keywords: Deep learning, Layer collapse, Depth compression, Pruning