In-Person Poster presentation / poster accept

Liquid Structural State-Space Models

Ramin Hasani · Mathias Lechner · Johnson (Tsun-Hsuan) Wang · Makram Chahine · Alexander Amini · Daniela Rus

MH1-2-3-4 #41

Keywords: [ Deep Learning and representational learning ] [ time series. memory ] [ state-space models ] [ liquid neural networks ] [ recurrent neural networks ]

[ Abstract ]
[ OpenReview
Wed 3 May 2:30 a.m. PDT — 4:30 a.m. PDT


A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on an extensive series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structured SSM, such as S4, is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structured state-space model, dubbed Liquid-S4, improves generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32\% on the Long-Range Arena benchmark. On the full raw Speech Command recognition dataset, Liquid-S4 achieves 96.78\% accuracy with a 30\% reduction in parameter counts compared to S4. The additional gain in performance is the direct result of the Liquid-S4's kernel structure that takes into account the similarities of the input sequence samples during training and inference.

Chat is not available.