Imagine running a 20-billion-parameter language model on your phone, generating text at 120 tokens per second—faster than most people can read. That’s the claim behind Maple-Preview, a new model from DeepGrove AI, showcased on Hacker News. The trick? A combination of two cutting-edge techniques: ternary quantization and a Mixture-of-Experts architecture.
For years, on-device AI has been limited to small models—typically 1 to 7 billion parameters—because phones have limited memory and compute. Maple-Preview’s approach could change that, offering a path to larger, more capable models that run locally, preserving privacy and enabling offline use. But does it deliver on quality, or is it just a clever demo? Let’s break down what makes this model tick and what it means for the future of on-device AI.
The Core Innovation: Ternary Weights
Most language models store their weights as 16-bit or 8-bit floating-point numbers. Maple-Preview uses ternary weights, meaning each weight is constrained to one of three values: -1, 0, or +1. This is a dramatic simplification. Instead of needing 16 bits to store each weight, you only need about 1.58 bits (since log₂(3) ≈ 1.58). That’s a reduction of nearly 90% in memory footprint.
Think of it like storing a photograph in black and white instead of full color—you lose some nuance, but the file is much smaller. For neural networks, this trade-off can be surprisingly small in practice, thanks to research like Microsoft’s BitNet, which showed ternary models can approach the quality of full-precision models, especially for smaller sizes.
The benefit on a phone is huge. A 20B-parameter model with ternary weights takes up roughly 4 GB of storage, but with MoE, the actual memory footprint can be less, making it fit within the 8 GB RAM of recent iPhones.
The Architecture: Mixture of Experts
A Mixture-of-Experts (MoE) model contains many specialized sub-networks, or “experts,” but only a small fraction are activated for any given input. Maple-Preview has 20 billion total parameters, but for each token it processes, it might only use, say, 2 to 4 billion active parameters. This is like having a team of 20 specialists, but only calling on the few most relevant for each question—saving time and compute.
MoE is not new; it’s used in models like Mixtral and DeepSeek. But combining it with ternary quantization is a novel twist that pushes the efficiency envelope further.
Performance: 120 Tokens per Second
The headline number—120 tokens per second—is impressive. To put it in context, a typical on-device model like Llama 3.2 3B runs at maybe 50-70 tokens per second on a high-end phone. Maple-Preview is nearly twice as fast, despite having far more total parameters.
This speed likely comes from the ternary weights, which allow for faster matrix multiplications on the phone’s GPU or Neural Engine. However, it’s important to note that the 120 tok/s figure is likely for short prompts on a recent Pro model. Real-world performance with long contexts or multitasking may vary.
Quality Concerns: Is 20B Actually 20B?
Here’s where skeptics raise an eyebrow. With MoE, only a fraction of parameters are active per token. So the effective capacity of Maple-Preview might be closer to a 3-5B dense model. The “20B” headline can be misleading if you interpret it as equivalent to a dense 20B model.
Moreover, ternary quantization historically degrades quality. While BitNet has shown promise, the trade-off is real. The HN community will be eager to see benchmarks like MMLU or perplexity scores. Without those, it’s hard to assess whether Maple-Preview is genuinely useful or just a tech demo.
Why This Matters: On-Device AI’s Next Step
Apple, Google, and Qualcomm have all been pushing on-device AI for privacy and offline use. But their models are tiny compared to cloud-based giants like GPT-4. Maple-Preview’s approach could narrow that gap, allowing phones to run more capable models without sending data to the cloud.
Imagine using a language model for coding assistance, summarizing emails, or even a chatbot that works on an airplane. That’s the promise of on-device AI. Maple-Preview shows a viable path to larger models on consumer hardware.
The Hacker News Reception
The Show HN post garnered 120 points and 34 comments—moderate interest. Discussions likely revolved around the feasibility of the performance claims, the quality of the model, and comparisons to existing on-device models. The community is right to be curious; this is a significant engineering achievement, but it needs rigorous validation.
What’s Next?
DeepGrove AI has released Maple-Preview as a “preview,” suggesting they’re seeking feedback. Whether they’ll open-source the model or release detailed benchmarks remains to be seen. If they can demonstrate quality that holds up, this could be a step toward a new generation of on-device AI.
For now, Maple-Preview is a fascinating proof of concept that combines two powerful techniques to achieve something that seemed impossible a few years ago: running a 20-billion-parameter model on a phone at blazing speed. The question is whether it can move from impressive demo to practical tool.
Maple-Preview is a bold experiment that pushes the boundaries of what’s possible on mobile hardware. By combining ternary weights with a Mixture-of-Experts architecture, DeepGrove AI has achieved remarkable speed and memory efficiency. While quality remains an open question, this preview offers a glimpse into a future where powerful AI runs entirely on your device, protecting your privacy and working without an internet connection. It’s a development worth watching, and we’ll likely see more innovations in this space soon.
Summary
- Maple-Preview is a 20B-parameter MoE model that runs at 120 tokens per second on an iPhone, a significant speed milestone.
- It uses ternary weights (values -1, 0, +1), reducing memory footprint to ~1.58 bits per weight, enabling the model to fit in phone RAM.
- The MoE architecture means only a fraction of parameters are active per token, so effective capacity is lower than a dense 20B model.
- Quality concerns exist due to ternary quantization, but prior research like BitNet suggests the trade-off can be acceptable.
- This demo highlights the potential for larger, more capable on-device AI, with benefits for privacy and offline use.
FAQ
Q: What is ternary quantization?
A: It’s a technique that stores neural network weights as one of three values: -1, 0, or +1, instead of high-precision numbers. This drastically reduces memory usage and speeds up computation, at a small cost to model quality.
Q: How does a 20B parameter model fit on a phone?
A: Two reasons: ternary weights use about 1.58 bits per weight, so 20B weights take roughly 4GB. Also, it’s a Mixture-of-Experts model, so only a small subset of experts is active for each token, reducing the active memory footprint.
Q: Is Maple-Preview open-source?
A: The announcement is a “Show HN” preview, but it’s unclear if the model weights or code are publicly available. Check DeepGrove’s website or the HN thread for details.
Q: How does 120 tok/s compare to other on-device models?
A: Smaller models like Llama 3.2 3B typically run at 50-70 tok/s on high-end phones. Maple-Preview claims nearly double that speed, which is impressive.
Q: What are the practical uses?
A: Possible uses include offline chatbots, text summarization, coding assistance, and other tasks that require language understanding without sending data to the cloud.