In-context Recall Benchmarks Table

Model	# Params (M)	(Rec / Conv) Mem. (M elts)	Recall tasks			NIAH-1						NIAH-2						NIAH-3
Model	# Params (M)	(Rec / Conv) Mem. (M elts)	SWDE	FDA	SQuAD	1K	2K	4K	8K	16K	32K	1K	2K	4K	8K	16K	32K	1K	2K	4K	8K	16K	32K
Transformer
w. RoPE	340	\(\boldsymbol{\infty}\) / 0.0	42.3	34.5	22.1	100	100	0.0	0.0	0.0	0.0	100	100	0.0	0.0	0.0	0.0	71.6	47.6	0.0	0.0	0.0	0.0
w. Gate (FoX)	376	\(\boldsymbol{\infty}\) / 0.0	52.5	64.3	30.1	100	100	32.2	8.0	4.2	0.0	100	100	100	24.0	11.6	3.2	95.4	85.6	64.2	11.6	7.2	0.0
SSM
GLA	475	12.5 / 0.4	29.0	11.4	30.3	74.6	25.1	8.2	2.2	0.0	0.0	91.2	37.2	21.4	3.6	0.0	0.0	84.2	57.1	20.8	10.2	2.3	0.0
GSA	399	12.5 / 0.0	23.8	14.5	24.9	99.2	97.1	90.0	67.4	29.6	11.0	96.6	98.8	28.0	5.1	1.0	0.0	60.0	30.1	13.5	1.0	0.0	0.0
GDN	475	12.5 / 0.4	29.5	8.3	31.3	99.2	100	99.8	92.0	41.8	22.1	99.2	92.0	43.6	17.8	6.2	4.0	92.6	80.6	37.8	5.2	6.8	2.5
Mamba-2	382	12.5 / 0.4	25.7	14.9	31.9	99.2	95.6	52.2	12.8	5.4	2.8	99.8	98.0	68.2	15.4	4.4	3.8	53.4	53.6	17.4	1.8	2.2	3.2
SWA	374	12.5 / 0.0	10.0	14.4	29.7	29.8	11.0	6.2	3.4	1.2	0.0	36.2	14.4	10.2	3.8	3.2	0.0	26.2	9.2	7.4	1.4	1.8	0.0
Raven	424	12.5 / 0.0	34.1	22.7	35.4	99.8	100	99.8	99.8	99.4	91.4	98.8	98.0	98.8	81.6	23.0	8.8	76.8	43.6	13.4	1.0	0.0	0.0
Transformer
w. RoPE	693	\(\boldsymbol{\infty}\) / 0.0	58.9	63.6	41.3	92.4	90.8	0.0	0.0	0.0	0.0	99.6	99.2	0.0	0.0	0.0	0.0	33.4	9.0	0.0	0.0	0.0	0.0
w. Gate (FoX)	694	\(\boldsymbol{\infty}\) / 0.0	64.9	80.0	41.5	100	100	99.0	28.6	12.4	0.0	100	100	67.4	9.0	7.8	0.0	83.2	63.8	0.2	0.4	0.2	0.0
SSM
GLA	892	16.5 / 0.4	50.1	35.9	39.2	100	94.2	29.4	3.6	0.0	0.0	100	89.8	25.8	3.8	3.0	0.0	94.6	51.2	3.2	1.8	1.6	0.0
GSA	750	16.5 / 0.0	42.8	28.9	33.6	100	100	99.8	97.6	64.0	0.0	99.4	90.0	29.8	3.6	2.4	0.0	79.0	16.6	2.4	0.2	0.0	0.0
GDN	892	16.5 / 0.4	46.0	31.9	35.9	100	100	100	100	94.8	45.2	100	95.4	35.2	14.2	9.2	5.8	79.4	65.4	10.6	6.4	4.2	2.0
Mamba-2	711	16.5 / 0.6	42.7	21.3	34.6	99.8	63.4	13.6	1.8	0.8	0.6	97.4	49.2	24.2	6.6	4.2	3.2	89.8	17.4	14.6	1.4	2.4	0.6
SWA	693	16.5 / 0.0	14.9	12.2	28.8	11.0	6.2	3.4	1.2	0.6	0.0	14.4	6.4	5.0	1.2	2.4	0.0	15.2	4.8	1.4	2.4	3.4	0.0
Raven	792	16.5 / 0.0	41.7	24.1	39.9	99.8	100	99.8	99.2	98.2	91.0	100	99.4	90.8	35.2	6.8	3.4	90.4	5.2	0.6	0.0	0.0	0.0