Apple Releases Updated MLX Framework Enabling Local Execution of Frontier-Class AI Models on Mac Studio

Apple has released a major update to MLX, its open-source array framework designed for Apple silicon, empowering AI researchers, developers, and engineers to run massive, frontier-class artificial intelligence models locally in a private environment. Utilizing the unified memory architecture intrinsic to Apple silicon, the latest version of the framework allows users to perform inference, fine-tuning, and advanced experimentation directly on their desktop hardware. The update specifically targets the capabilities of the M5 architecture, including its dedicated GPU Neural Accelerators and high-bandwidth unified memory, bridging the gap between massive cloud-based clusters and local desktop setups.

Demonstrating the capabilities of the new framework, developers can now run the newly released Muse Spark Open model—a massive architecture boasting nearly a trillion parameters—right from an office desk. Operating on a cluster of four Mac Studio computers, the Muse Spark Open model generates over 36 tokens per second, a performance velocity well above average human reading speed. By scaling out from a single Mac Studio equipped with up to 512GB of unified memory to a multi-machine configuration utilizing RDMA over Thunderbolt drivers, local hardware can now support an expansive array of frontier-class language models.

The framework update was evaluated across multiple open-weight large language models using the MLX LM package, comparing performance between the recently introduced M5 Ultra chip and the previous-generation M3 Ultra. The M5 generation introduces a GPU Neural Accelerator capable of dedicated matrix multiplication operations, a foundational requirement for accelerating machine learning workloads. Performance benchmarks assessed three distinct models spanning varying scales: the 816-billion-parameter Muse Spark Open, the 128-billion-parameter Mistral Medium 3.5, and the 27-billion-parameter Qwen 3.8.

Architectural differences dictate how these models process information on local hardware. Muse Spark Open utilizes a mixture of experts (MoE) design, activating only a fraction of its total parameters during inference to reduce computational overhead, and is evaluated in NVFP4 precision. In contrast, Mistral Medium 3.5 and Qwen 3.8 employ dense architectures where every parameter remains active during inference, evaluated in their native floating-point precisions of FP8 and BF16, respectively. Evaluations measured three primary operational metrics across various context sizes: time to first token, generation throughput, and peak power usage.

The local execution of large language models on Mac Studio has grown increasingly popular for agentic workflows, such as agentic coding applications that necessitate extensive context windows. Consequently, performance metrics were examined at both 8,192 and 65,536 token context sizes. Smaller dense models like Qwen 3.8 and Mistral Medium 3.5 fit comfortably within the 512GB memory limits of a single Mac Studio, occupying up to 68GB and 158GB of RAM respectively under the maximum 65,536-token context configuration.

Comparing single-node performance between the M3 Ultra and M5 Ultra highlights the distinct computational advantages of the newer hardware generation. Time to first token performance is fundamentally compute-bound, a domain where the GPU Neural Accelerators in the M5 Ultra excel. Dedicated hardware support for rapid matrix multiplication translates directly into substantial speedups, delivering up to 4.6 times faster time-to-first-token performance compared to the M3 Ultra baseline. This efficiency establishes the M5 Ultra-powered Mac Studio as a preferred desktop solution for handling tasks characterized by exceptionally large context windows.

Conversely, token generation speed is bound by memory bandwidth. The M5 Ultra chip delivers 1.2 terabytes per second of unified memory bandwidth, marking a significant increase from the 819 gigabytes per second offered by the M3 Ultra. This hardware enhancement yields a direct 1.6 times speedup in token generation throughput. Despite these performance gains, the Mac Studio maintains high energy efficiency, registering peak power consumption figures of approximately 200 watts for the M3 Ultra configuration and 300 watts for the M5 Ultra setup. This modest power draw enables deployment in standard home or office environments without specialized electrical infrastructure.

While dense models fit on a single workstation, ultra-large frontier models like Muse Spark Open exceed the memory capacity of an individual machine. Running Muse Spark Open in its native BF16 precision would demand more than 1.6 terabytes of RAM for model parameters alone. Although 4-bit NVFP4 quantization reduces the memory footprint to approximately 493 gigabytes, auxiliary memory required for intermediate computations prevents the model from running on a single computer, necessitating distributed parallelization across multiple Mac Studio units.

To accommodate models exceeding single-node memory capacities, MLX supports distributed configurations that rely on high-speed inter-machine communication. macOS features a low-latency RDMA driver over Thunderbolt, which the MLX framework leverages via communication collectives known as jaccl. By connecting multiple Mac Studio units with dual Thunderbolt cables to double link bandwidth, developers can implement two primary parallelization strategies: pipeline parallelization and tensor parallelization.

Pipeline parallelization distributes a model across its depth, storing sequential layers across different computers. For instance, the first Mac Studio executes the initial layers of a model and passes the intermediate outputs to a second machine to complete the remaining layers. This method requires minimal code modifications and yields decode speeds comparable to a single machine because layers are still evaluated sequentially. Furthermore, during large context processing, pipeline parallelization can surpass single-node performance for prefill tasks by segmenting large contexts into smaller chunks and overlapping computations across the connected nodes.

Tensor parallelization distributes the width of the model across multiple computers rather than its depth, allowing every machine to hold a fraction of each layer so that per-layer computations happen concurrently. While tensor parallelization improves inference speed when implemented efficiently, it introduces architectural complexity and requires frequent, small-scale communication operations for every token and layer. Because of this communication frequency, low-latency technologies such as Thunderbolt RDMA are essential for achieving performance gains.

Using the mlx.launch utility included with the framework, users can coordinate multi-node jobs across a Thunderbolt mesh network. Benchmarks indicate that dense architectures scale effectively across multiple machines using tensor parallelization. Although communication overhead can occasionally impact prefill times at massive context sizes due to normalization steps in mixture-of-experts models, adding more machines scales decode throughput significantly, allowing configurations of four M5 Ultra systems to sustain high generation speeds.

The intersection of these hardware and software developments makes it feasible to operate sophisticated agentic AI systems locally. Developers can initialize a server using the mlx_lm.server command paired with multi-node launch utilities, exposing an OpenAI-compatible local endpoint. This local server can then be integrated directly with agentic development environments, enabling complex, automated coding workflows within a secure, private computing environment.

The culmination of these engineering efforts reflects a broader push toward local AI execution on Apple silicon. By combining specialized hardware features like the M5 GPU Neural Accelerators, high-bandwidth unified memory, low-latency RDMA over Thunderbolt connectivity, and optimized software frameworks for mixture-of-experts models, developers can run models exceeding 800 billion parameters locally with high responsiveness and energy efficiency.