Layerfold: A Python Library to Reduce the Depth of Neural Networks

28 Pages Posted: 12 Oct 2024

See all articles by Giommaria Pilo

Giommaria Pilo

Télécom Paris

Nour Hezbri

Télécom Paris

André Pereira e Ferreira

Télécom Paris

Victor Quétu

Télécom Paris

Enzo Tartaglione

Télécom Paris

Abstract

Large-scale models are the backbone of Computer Vision and Natural Language Processing, their generalizability allows for transfer learning and deployment in different scenarios. Their large size, however, means that reducing their computational and memory demands remains challenging.Recent research proposes to achieve ``layer collapse'', a condition where multiple layers can be combined due to the collapse of non-linearities to linear operators. While this is an important discovery, most studies remain theoretical, often replacing non-linearities with simple identity functions and not providing a real implementation of the more compact architecture.Our contribution is LayerFold, a library that studies and concretely implements collapsed layers merging. We address typical cases, from fully connected to convolutional layers, discussing constraints and prospective challenges. Our tests on edge devices reveal that merely reducing network depth doesn't always result in faster computation, even when GPU-equipped. This work raises important warnings and opens the door to further advancements in efficient model deployment.

Keywords: Deep learning, Layer collapse, Depth compression, Pruning

Suggested Citation

Pilo, Giommaria and Hezbri, Nour and Pereira e Ferreira, André and Quétu, Victor and Tartaglione, Enzo, Layerfold: A Python Library to Reduce the Depth of Neural Networks. Available at SSRN: https://ssrn.com/abstract=4985344 or http://dx.doi.org/10.2139/ssrn.4985344

Giommaria Pilo (Contact Author)

Télécom Paris ( email )

19 Place Marguerite Perey
Palaiseau, 91120
France

Nour Hezbri

Télécom Paris ( email )

19 Place Marguerite Perey
Palaiseau, 91120
France

André Pereira e Ferreira

Télécom Paris ( email )

19 Place Marguerite Perey
Palaiseau, 91120
France

Victor Quétu

Télécom Paris ( email )

19 Place Marguerite Perey
Palaiseau, 91120
France

Enzo Tartaglione

Télécom Paris ( email )

19 Place Marguerite Perey
Palaiseau, 91120
France

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
36
Abstract Views
192
PlumX Metrics