Were RNNs All We Needed? - Revisiting the Power of Minimal Recurrent Architectures
0:00 Introduction and Hypothesis: The paper questions the necessity of complex recurrent architectures like S4 and Mamba, proposing that simpler RNNs might be just as effective when implemented correctly.
2:00 Limitations of Transformers: Transformers suffer from quadratic memory and compute requirements with respect to sequence length, limiting their ability to handle long sequences. RNNs, on the other hand, have constant memory requirements.
3:43 RNN Drawbacks: Backpropagation Through Time (BPTT): Traditional RNNs require BPTT, which hinders parallel training and can lead to gradient degradation.
5:28 State-Space Models and Parallelism: Newer state-space models like S4 and Mamba achieve parallelism by removing the dependency of the input computation on the past hidden state.
9:06 Minimal RNNs (minLSTM and minGRU): The paper introduces minimal versions of LSTMs and GRUs that remove hidden state dependencies in input, forget, and update gates, enabling parallel training.
12:54 Parallel Scan for Efficient Computation: These minimal RNNs can be implemented using parallel scan, a fast algorithm for prefix sum computations, allowing for significant speed improvements.
14:56 Trade-offs and Power: While minimal RNNs might be less powerful than traditional RNNs in a single layer, stacking multiple layers can reintroduce past dependency and improve performance.
19:55 Weak Experimental Evidence: The paper's experimental results are limited to simple tasks like selective copying and reinforcement learning benchmarks, where the advantages of complex RNNs might not be fully apparent.
23:28 Task Suitability: The findings suggest that minimal RNNs might be well-suited for tasks with inherent locality, such as character-level language modeling, where the need for complex long-range dependencies is less pronounced.
26:45 Potential for Scalability: Despite the limited experimental evidence, the paper's hypothesis remains valid. Scaling up minimal RNNs could potentially achieve performance comparable to complex state-space models on more challenging tasks.