-
Memory Attention
Replacing attention's value projection with a token-indexed memory table
-
Attention Zoo: Summary of SSMs and Transformers
An interactive guide to modern sequence models, explore architectures and recurrences across the linear-softmax landscape.
-
Raven (Part-2)
Architecture and Results
-
Raven (Part-1)
Memory as a set of Slots
-
On the Legacy of Linear Transformers in Positional Embeddings 📍
Duality of Forget Gates and Position Embeddings in Sequence Modeling