Benchmarks
Needle 3 through the shipped CQ2 binary, confidence gate on · baselines at f16 under vLLM · DeepSeek V4 Flash via its cloud API
Mobile Actions
exact call · 961 rows · accuracy (%)
25
50
75
100
DeepSeek V4 Flash
cloud
88.4
Needle3-20L-121M
121M
86.0
LFM2.5 1.2B
1.2B
82.4
Needle3-16L-98M
98M
80.7
Qwen3.5 0.8B
800M
76.0
LFM2.5 350M
350M
72.8
LFM2.5 230M
230M
69.3
FunctionGemma 270M
270M
65.1
Needle 2
45M
63.5
Apple FM
3.0B
57.6
Needle3-8L-52M
52M
36.8
Needle3-4L-29M
29M
11.7
DroidCall
exact calls in order · 200 rows · accuracy (%)
25
50
75
100
DeepSeek V4 Flash
cloud
60.5
Needle3-20L-121M
121M
47.0
Needle3-16L-98M
98M
40.0
Needle3-8L-52M
52M
36.5
LFM2.5 1.2B
1.2B
35.5
LFM2.5 350M
350M
32.5
Qwen3.5 0.8B
800M
28.0
Needle3-4L-29M
29M
21.0
Needle 2
45M
17.0
FunctionGemma 270M
270M
16.5
LFM2.5 230M
230M
11.5
BFCL v4
AST match · 3,641 rows · accuracy (%)
25
50
75
100
DeepSeek V4 Flash
cloud
77.2
LFM2.5 1.2B
1.2B
62.0
LFM2.5 350M
350M
59.1
Qwen3.5 0.8B
800M
56.8
Needle3-20L-121M
121M
50.2
FunctionGemma 270M
270M
46.6
LFM2.5 230M
230M
46.3
Needle3-16L-98M
98M
41.3
Needle3-8L-52M
52M
28.2
Needle3-4L-29M
29M
19.5
DSTC8
field F1 · 1,813 turns · F1 (%)
25
50
75
100
DeepSeek V4 Flash
cloud
80.0
LFM2.5 230M
230M
53.0
Qwen3.5 0.8B
800M
49.0
LFM2.5 1.2B
1.2B
48.0
Needle3-20L-121M
121M
40.7
Needle3-16L-98M
98M
28.5
FunctionGemma 270M
270M
27.0
LFM2.5 350M
350M
20.0
Needle3-8L-52M
52M
15.3
Needle3-4L-29M
29M
6.9
SNIPS gold
field F1 · 700 rows · F1 (%)
25
50
75
100
DeepSeek V4 Flash
cloud
69.4
LFM2.5 1.2B
1.2B
43.0
Qwen3.5 0.8B
800M
35.0
LFM2.5 350M
350M
34.0
Needle3-20L-121M
121M
30.2
FunctionGemma 270M
270M
29.0
LFM2.5 230M
230M
27.0
Needle3-16L-98M
98M
23.5
Needle3-8L-52M
52M
16.6
Needle3-4L-29M
29M
7.7
SNIPS 7-way
field F1 · 700 rows · F1 (%)
25
50
75
100
DeepSeek V4 Flash
cloud
66.7
LFM2.5 1.2B
1.2B
38.0
Qwen3.5 0.8B
800M
34.0
LFM2.5 350M
350M
29.0
Needle3-20L-121M
121M
24.7
LFM2.5 230M
230M
22.0
Needle3-16L-98M
98M
19.2
FunctionGemma 270M
270M
14.0
Needle3-8L-52M
52M
10.1
Needle3-4L-29M
29M
4.3