Perplexity has built a new local artificial intelligence engine for Apple silicon, and the company says it can run a large Qwen model considerably faster than Apple’s MLX-LM software. Named Lily, the engine will eventually be released as open source, although Perplexity has not announced when the code will become available.
Lily powers the local side of Perplexity’s Hybrid Compute system for Mac. The feature divides work between cloud models and an AI model running directly on the computer. Cloud models can handle demanding research and reasoning, while the local model works with private files and applications that users may not want to send elsewhere.
That arrangement only works well if the local model can keep pace. Waiting several seconds for every response would make Hybrid Compute feel like two disconnected products. Perplexity therefore built an inference engine specifically for Apple silicon instead of relying entirely on an existing general-purpose framework.
Apple offers MLX, an open-source machine-learning framework designed for Macs powered by its M-series processors. MLX-LM sits on top of that framework and provides tools for loading and running large language models. It supports numerous models and architectures, which makes it useful to developers who want flexibility.
Lily takes a narrower approach. Perplexity designed the engine specifically for Apple silicon and Qwen3.6-35B-A3B, a mixture-of-experts model containing 35 billion parameters. The model activates only about 3 billion parameters for each token, reducing the amount of computation required.
Perplexity’s engine uses a Rust runtime and custom Metal kernels to execute operations tailored to the Qwen model. Neither PyTorch nor MLX sits in the execution path. Lily also includes an OpenAI-compatible chat completion API, which should make it easier to integrate with software already designed around that interface.
According to Perplexity’s testing, Lily averaged 23 percent higher prompt-processing throughput and 35 percent higher token-generation throughput than MLX-LM. The company tested both engines on a MacBook Pro equipped with an M5 Max processor, a 40-core GPU, and 128GB of unified memory.
With a 4K-token prompt and 4K-token context, Lily processed the prompt at 5,749.9 tokens per second and generated output at 186.6 tokens per second. MLX-LM reached 4,737.5 prompt tokens per second and generated 140.9 tokens per second under the same conditions.
Across context lengths ranging from 256 to 128K tokens, Lily’s advantage ranged from 12 to 42 percent for prompt processing and from 31 to 37 percent for token generation. Perplexity says its engine came out ahead at every length it tested.
Those results sound impressive, but this is not a simple case of Perplexity building a replacement for MLX. MLX-LM must accommodate many models, while Lily can organize nearly every part of its runtime around the structure of one particular Qwen model. That narrower focus gives Perplexity optimization opportunities that a broader framework cannot always pursue without sacrificing compatibility.
Perplexity kept more of the workload on the GPU, reduced synchronization between the CPU and GPU, and avoided writing some temporary data to unified memory. It also adjusted how Lily handles attention as the context grows. At 128K tokens, one of those attention changes improved decoding performance by 40.2 percent compared with a version of Lily that did not use it.
Interestingly, not every commonly used AI technique produced a benefit. Speculative decoding, which asks a smaller model to propose tokens for the larger model to verify, made Lily’s single-request decoding workload 18 percent slower. Perplexity found that the additional work and irregular routing between the model’s experts outweighed the potential savings.
The Qwen model would ordinarily require around 70GB to store its weights using bfloat16. Four-bit quantization shrinks that checkpoint to 19.4GB, allowing it to remain in the Mac’s unified memory. It is still a demanding model, however, and Perplexity demonstrated it on one of Apple’s most powerful and expensive laptop configurations.
The results also come directly from Perplexity. Independent developers cannot reproduce the findings until the company publishes Lily’s code, so the numbers should be treated as promising internal benchmarks rather than settled fact.
Still, Lily shows why Apple silicon has become an attractive platform for local AI. Unified memory allows a Mac to hold relatively large models, while software designed around the hardware can reduce data movement and keep more work on the GPU.
Lily could become useful beyond Perplexity Computer if support expands to additional models and M-series processors. For now, though, developers have Perplexity’s benchmarks and technical explanation rather than code they can independently test.
Support independent tech journalism
NERDS.xyz is independently owned and operated. If you enjoy my coverage of Linux, AI, hardware, cybersecurity, and tech culture, consider supporting the site on Ko-fi.
Support NERDS.xyz