Apple Brings Frontier-Class AI to the Desktop: Running Near-Trillion Parameter Models Locally on Mac Studio with MLX

Apple has announced a major milestone for local artificial intelligence development, demonstrating that powerful frontier-class models—including the massive, nearly trillion-parameter Muse Spark Open—can now be run, tested, and fine-tuned locally on a cluster of Mac Studio desktops. Driven by the latest release of the open-source MLX array framework, researchers and developers can leverage the unified memory architecture of Apple silicon to experiment with cutting-edge AI in a private, secure environment right from their desks.

The breakthrough comes as the latest iteration of MLX introduces native support for sophisticated distributed computing techniques, hardware-accelerated matrix operations, and high-speed multi-node communication. By combining the capabilities of the newly released M5 Ultra chip with advanced parallelization schemes and the high-throughput connectivity of Remote Direct Memory Access (RDMA) over Thunderbolt, Apple’s hardware and software ecosystem is bridging the gap between local workstations and enterprise-grade data center workloads.

At the heart of this local AI capability is the MLX framework, designed specifically to exploit the unified memory architecture found in Apple silicon. This design allows the system memory to be shared seamlessly between the CPU and GPU, giving developers unprecedented headroom to load massive language models into RAM without encountering the traditional bottlenecks of discrete graphics cards. In previous technical disclosures, Apple highlighted the foundational strengths of the M5 architecture, particularly its specialized GPU Neural Accelerators. These dedicated execution units are engineered to handle heavy matrix multiplication operations natively, serving as the critical engine behind the performance leaps observed in the newest generation of Mac Studio systems.

To showcase the practical limits of this hardware, Apple’s engineering team benchmarked a diverse lineup of open-weight large language models across multiple generations of Mac Studio hardware, specifically comparing the newly released M5 Ultra against its predecessor, the M3 Ultra. The evaluation encompassed three distinct frontier-class architectures spanning a wide spectrum of sizes. These included the newly minted Muse Spark Open, a massive mixture-of-experts model boasting 816 billion total parameters; Mistral Medium 3.5, a 128-billion parameter dense architecture; and Qwen 3.8, a highly capable 27-billion parameter dense model.

While dense architectures like Qwen 3.8 and Mistral Medium 3.5 keep all of their parameters active during inference, mixture-of-experts models like Muse Spark Open deploy a clever efficiency strategy. By keeping only a fraction of its total experts active at any given time—in this case, 42 billion active parameters out of 816 billion—Muse Spark Open manages to curtail computational demands during inference. To evaluate these models efficiently, Apple utilized varying precision formats, running Qwen 3.8 in native BF16, Mistral Medium 3.5 in native FP8, and Muse Spark Open in 4-bit NVFP4 precision to optimize its memory footprint.

Evaluating these systems requires looking closely at three primary performance indicators across various context sizes: time to first token, generation throughput measured in tokens per second, and peak power consumption measured in watts. For everyday developer workflows, particularly those involving complex agentic coding applications that necessitate extensive context windows, performance at both 8,192 and 65,536 tokens is critical.

When examining single-machine deployments, both Qwen 3.8 and Mistral Medium 3.5 fit comfortably within the 512GB of unified memory available on a fully configured Mac Studio. Under high-load scenarios with large context sizes of 65,536 tokens, these models consume roughly 68GB and 158GB of RAM respectively. Comparing their execution on an M3 Ultra versus an M5 Ultra reveals striking performance divergences driven by the underlying hardware upgrades.

Because the initial generation of the response, represented by the time to first token, is heavily compute-bound, the GPU Neural Accelerators in the M5 Ultra display a massive advantage. The specialized hardware support for rapid matrix multiplication yields significant speedups, delivering time-to-first-token results up to 4.6 times faster than the M3 Ultra baseline. This dramatic reduction in latency establishes the M5 Ultra-equipped Mac Studio as an exceptional desktop choice for handling dense, context-heavy reasoning tasks.

Conversely, generation throughput, or decode speed, is fundamentally constrained by memory bandwidth. Here, the M5 Ultra chip introduces a substantial upgrade, delivering 1.2 terabytes per second of unified memory bandwidth, an increase over the 819 gigabytes per second found in the M3 Ultra. This enhanced bandwidth translates directly into faster generation speeds, with the M5 Ultra reporting a 1.6-fold increase in tokens generated per second for supported models.

Despite this formidable computational output, the Mac Studio maintains a remarkably modest energy profile. Peak power consumption during intensive inference workloads sits at approximately 200 watts for the M3 Ultra system and roughly 300 watts for the M5 Ultra system. This energy usage represents a fraction of the power consumed by common household appliances, ensuring that high-performance AI development can be conducted safely and efficiently within standard home or office environments without specialized cooling or electrical infrastructure.

However, operating truly massive frontier-class models like Muse Spark Open requires scaling beyond a single workstation. Even when compressed into a 4-bit NVFP4 quantized format, Muse Spark Open requires approximately 493 gigabytes of memory just for the model weights, which swells further when accounting for the intermediate computational memory needed during active inference. Running such a model demands a distributed approach across multiple Mac Studio units.

To address this, the latest version of MLX integrates native support for distributed multi-node execution, relying on low-latency communication primitives enabled by the RDMA over Thunderbolt driver introduced in recent macOS releases. By interconnecting multiple Mac Studio systems using dual Thunderbolt cables per link to effectively double available bandwidth, developers can implement two primary parallelization schemes: pipeline parallelization and tensor parallelization.

Pipeline parallelization divides a model across its depth, storing sequential layers on separate machines. For instance, half of the model’s layers are executed on a primary Mac Studio, which then passes its intermediate outputs to a secondary Mac Studio to process the remaining layers. Because the layers are still evaluated sequentially, decode speeds under pipeline parallelization remain comparable to single-machine performance. However, for massive context sizes, pipeline parallelization actually achieves faster prefill speeds—up to 1.8 times faster than a single node—by chunking large contexts and overlapping computational cycles across the networked machines.

For even greater performance scalability, tensor parallelization distributes the model across its width, splitting individual layers among multiple computers so that every machine processes a fraction of each layer simultaneously. While tensor parallelization significantly enhances compute speed and generation throughput, it requires extremely low-latency communication infrastructure due to the constant exchange of small data packets across nodes. The introduction of the high-speed Thunderbolt RDMA backend in MLX, combined with optimized synchronization features, ensures that multi-node setups can overcome communication overhead and achieve high efficiency.

Benchmarking data demonstrates that scaling Muse Spark Open across a cluster of four M5 Ultra-powered Mac Studios allows the system to generate over 36 tokens per second at an 8,192 token context size, operating well above average human reading speeds. Similar scaling applied to dense models like Qwen 3.8 pushes decode speeds past 50 tokens per second when distributed across a four-node tensor parallel configuration.

This local infrastructure directly supports the burgeoning ecosystem of agentic AI applications. Developers are increasingly adopting autonomous coding tools and local development agents that rely on persistent server architectures and expansive context windows. By combining the MLX server utilities with multi-node launch configurations, developers can spin up local API endpoints that serve massive frontier models directly to third-party development environments and agentic frameworks.

The convergence of these hardware and software advancements over the past year marks a definitive shift in how advanced machine learning research and application development can be conducted. By uniting the raw performance of Apple silicon, specialized GPU neural acceleration, high-bandwidth unified memory, and efficient multi-node networking via Thunderbolt RDMA, developers now possess the capability to run and fine-tune near-trillion parameter artificial intelligence models locally, privately, and at scale.