Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP

Name: Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Description: Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

Sasha Rush

Sasha Rush surveys linear RNNs and state space models as replacements for attention in transformer-style models, in a talk for MLSys. It covers the foundational preliminaries and predates Mamba.

HN commenters recommend it for building intuition about state space models and the tradeoffs between transformer and SSM layers.

3 Hacker News mentions · 2024–2024

Topics

Machine learning & deep learning
LLMs & generative AI

Watch on YouTube

Explore more videos