Back to Devoured - May 13, 2026
DEVOURED
Cactus Needle (GitHub Repo)

Cactus Needle (GitHub Repo)

AI opensourcepython GitHub
Cactus Compute distilled Gemini 3.1 into a 26M-parameter model that outperforms Qwen-0.6B at function calling and runs 6,000 tokens/sec on MacBooks.
What: Cactus Needle is a 26M-parameter Simple Attention Network distilled from Gemini 3.1, trained on 16 TPU v6e for 200B tokens over 27 hours. Runs at 6,000 tokens/sec prefill and 1,200 decode, with fully open weights on GitHub. Beats FunctionGemma-270m, Qwen-0.6B, Granite-350m, and LFM2.5-350m on single-shot function calling.
Why it matters: Shows distillation from frontier models into sub-30M specialists is production-ready, enabling a shift from cloud APIs to local-first AI on consumer devices.
Deep dive
  • 26M-parameter Simple Attention Network distilled from Gemini 3.1 using encoder-decoder architecture (12 encoder, 8 decoder layers)
  • Encoder uses self-attention with GQA and RoPE but no feedforward network; decoder adds cross-attention and tool calling head
  • Pretrained on 200B tokens over 27 hours on 16 TPU v6e, then post-trained on 2B tokens of function call data in 45 minutes
  • Runs at 6,000 tokens/sec prefill and 1,200 decode on Cactus hardware; weights fully open on GitHub
  • Beats FunctionGemma-270m, Qwen-0.6B, Granite-350m, and LFM2.5-350m on single-shot function calling for personal AI
  • Includes web playground UI for testing and finetuning on custom tools at the click of a button
  • Designed for consumer devices (phones, watches, glasses) prioritizing local inference over conversational scope
  • Architecture uses ZCRMSNorm, gated residuals, tied embeddings, d=512, 8 heads/4 KV heads, 8192 BPE vocab
  • CLI supports finetuning on custom JSONL data, data generation via Gemini, evaluation, and TPU management
  • Authors acknowledge small models can be "finicky" and recommend testing on own tools before deployment
Decoder
  • Simple Attention Network: Novel architecture class designed for efficient inference on consumer devices, using encoder-decoder structure without traditional feedforward networks in the encoder
  • GQA (Grouped Query Attention): Memory-efficient attention mechanism that groups multiple query heads to share key-value pairs, reducing KV cache size
  • RoPE (Rotary Position Embeddings): Position encoding technique that applies rotation matrices to queries and keys, enabling better length extrapolation
  • ZCRMSNorm: Zero-centered Root Mean Square normalization, a normalization layer variant
Original article

Needle

We distilled Gemini 3.1 into a 26m parameter "Simple Attention Network" that you can even finetune locally on your Mac/PC. In production, Needle runs on Cactus at 6000 toks/sec prefill and 1200 decode speed. Weights are fully open on Cactus-Compute/needle, as well as the dataset generation.

d=512, 8H/4KV, BPE=8192
                                  ┌──────────────┐
                                  │  Tool Call   │
                                  └──────┬───────┘
                                        ┌┴──────────┐
                                        │  Softmax  │
                                        └─────┬─────┘
                                        ┌─────┴─────┐
                                        │ Linear (T)│  ← tied
                                        └─────┬─────┘
                                        ┌─────┴─────┐
                                        │ ZCRMSNorm │
                                        └─────┬─────┘
                                     ┌────────┴────────┐
                                     │ Decoder x 8     │
                                     │┌───────────────┐│
                                     ││ ZCRMSNorm     ││
                                     ││ Masked Self   ││
                                     ││ Attn + RoPE   ││
                                     ││ Gated Residual││
                                     │├───────────────┤│
  ┌──────────────┐                   ││ ZCRMSNorm     ││
  │ Encoder x 12 │──────────────────────▶Cross Attn   ││
  │              │                   ││ Gated Residual││
  │ ┌──────────┐ │                   │└───────────────┘│
  │ │ZCRMSNorm │ │                   └────────┬────────┘
  │ │Self Attn │ │                      ┌─────┴─────┐
  │ │ GQA+RoPE │ │                      │ Embedding │  ← shared
  │ │Gated Res │ │                      └─────┬─────┘
  │ │          │ │                    ┌───────┴───────-┐
  │ │ (no FFN) │ │                    │[EOS]<tool_call>│
  │ └──────────┘ │                    │ + answer       │
  │              │                    └───────────────-┘
  └──────┬───────┘
         │
    ┌────┴──────┐
    │ Embedding │
    └────┬──────┘
         │
    ┌────┴──────┐
    │   Text    │
    │  query    │
    └───────────┘
  • Pretrained on 16 TPU v6e for 200B tokens (27hrs).
  • Post-trained on 2B tokens of single-shot function call dataset (45mins).

Needle is an experimental run for Simple Attention Networks, geared at redefining tiny AI for consumer devies (phones, watches, glasses...). So while it beats FunctionGemma-270m, Qwen-0.6B, Graninte-350m, LFM2.5-350m on single-shot function call for personal AI, Those model are have more scope/capacity and excel in conversational settings. Also, small models can be finicky. Please use the UI in the next section to test on your own tools, and finetune accordingly, at the click of a button.

Quickstart

git clone https://github.com/cactus-compute/needle.git
cd needle && source ./setup
needle playground

Opens a web UI at http://127.0.0.1:7860 where you can test and finetune on your own tools. Weights are auto-downloaded.

Usage (Python)

from needle import SimpleAttentionNetwork, load_checkpoint, generate, get_tokenizer

params, config = load_checkpoint("checkpoints/needle.pkl")
model = SimpleAttentionNetwork(config)
tokenizer = get_tokenizer()

result = generate(
    model, params, tokenizer,
    query="What's the weather in San Francisco?",
    tools='[{"name":"get_weather","parameters":{"location":"string"}}]',
    stream=False,
)
print(result)
# [{"name":"get_weather","arguments":{"location":"San Francisco"}}]

Finetuning

# Playground (generates data via Gemini, trains, evaluates, bundles result)
needle playground

# CLI (auto-downloads weights if not local)
needle finetune data.jsonl

CLI

needle playground                  Test and finetune via web UI
needle finetune <data.jsonl>       Finetune on your own data
needle run --query "..." --tools   Single inference
needle train                       Full training run
needle pretrain                    Pretrain on PleIAs/SYNTH
needle eval --checkpoint <path>    Evaluate a checkpoint
needle tokenize                    Tokenize dataset
needle generate-data               Synthesize training data via Gemini
needle tpu <action>                TPU management (see docs/tpu.md)
@misc{ndubuaku2026needle,
  title={Needle},
  author={Henry Ndubuaku, Jakub Mroz,  Karen Mosoyan, Roman Shemet, Parkirat Sandhu, Satyajit Kumar, Noah Cylich, Justin H. Lee},
  year={2026},
  url={https://github.com/cactus-compute/needle}
}
Explore all 50 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi