Accelerating Neural ODEs Using Model Order Reduction

Tutkimustuotos: ArtikkeliScientificvertaisarvioitu

18 Lataukset (Pure)

Abstrakti

Embedding nonlinear dynamical systems into artificial neural networks is a powerful new formalism for machine learning. By parameterizing ordinary differential equations (ODEs) as neural network layers, these Neural ODEs are memory-efficient to train, process time series naturally, and incorporate knowledge of physical systems into deep learning (DL) models. However, the practical applications of Neural ODEs are limited due to long inference times because the outputs of the embedded ODE layers are computed numerically with differential equation solvers that can be computationally demanding. Here, we show that mathematical model order reduction (MOR) methods can be used for compressing and accelerating Neural ODEs by accurately simulating the continuous nonlinear dynamics in low-dimensional subspaces. We implement our novel compression method by developing Neural ODEs that integrate the necessary subspace-projection and interpolation operations as layers of the neural network. We validate our approach by comparing it to neuron pruning and singular value decomposition (SVD)-based weight truncation methods from the literature in image and time-series classification tasks. The methods are evaluated by acceleration versus accuracy when adjusting the level of compression. On this spectrum, we achieve a favorable balance over existing methods by using MOR when compressing a convolutional Neural ODE. In compressing a recurrent Neural ODE, SVD-based weight truncation yields good performance. Based on our results, our integration of MOR with Neural ODEs can facilitate efficient, dynamical system-driven DL in resource-constrained applications.
AlkuperäiskieliEnglanti
Artikkeli3175757
Sivut519-531
Sivumäärä13
JulkaisuIEEE Transactions on Neural Networks and Learning Systems
Vuosikerta35
Numero1
Varhainen verkossa julkaisun päivämäärä2022
DOI - pysyväislinkit
TilaJulkaistu - 2024
OKM-julkaisutyyppiA1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä

Julkaisufoorumi-taso

  • Jufo-taso 3

Sormenjälki

Sukella tutkimusaiheisiin 'Accelerating Neural ODEs Using Model Order Reduction'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä