This analysis details the construction and performance benchmarking of a high-density, low-power cluster built from four Apple Mac Studio units, utilizing M3 Ultra silicon to run very large language models (LLMs) locally. The primary performance scaling improvements stem from the integration of Remote Direct Memory Access (RDMA) over Thunderbolt, newly introduced in the macOS 26.2 beta, and the deployment of the Exo 1.0 clustering software. Comparative testing across high-parameter count models (Llama 3.3, Kimmy K2, Deepseek v3.1) demonstrates that the low-latency Thunderbolt interconnect dramatically outperforms standard 10 Gigabit Ethernet, achieving up to a 3.25x speed increase in inference throughput for dense models like Llama 3.3 (reaching 15.5 tokens per second across four nodes). While scaling efficiency is noted to be less linear for Mixture of Experts (MoE) architectures, the cluster offers a compelling cost and power efficiency advantage (600W total consumption for the dense Llama model) when compared to high-end dedicated GPU solutions like the Nvidia H200. Current limitations include the beta nature of the RDMA features and incomplete software compatibility with external AI development agents.
Summary of Mac Studio AI Cluster Performance
0:44 Cluster Configuration: The system comprises four Mac Studio units featuring M3 Ultra chips, providing an aggregated 1.5 terabytes (TB) of unified memory for large model execution. Half the units have 512 GB RAM, and the other half possess 256 GB RAM.
1:15 Enabling Technologies: Performance scaling is attributed to two key technologies: (1) Apple's stealth release of RDMA over Thunderbolt in the recent macOS 26.2 beta (compatible with M3 Ultra/M4 Pro or better chips), and (2) the public release of Exo 1.0, the software utilized to orchestrate the Mac Studio cluster.
1:17 Efficiency and Cost: The assembled cluster offers capabilities comparable to significantly more expensive solutions while consuming low power (under 600W total load for the dense model test) and occupying a unique price/capability class.
5:10 Single-Node Baseline (Llama 3.3): The 70 billion parameter Llama 3.3 model (FP16 weight, 150 GB) achieved an initial response time of approximately 2 seconds and an inference rate of 5 tokens per second (t/s) on a single Mac Studio.
6:30 Ethernet Bottleneck: Clustering the Llama 3.3 model across two and four Macs using 10 Gigabit Ethernet networking did not increase the inference rate beyond 5 t/s. This performance stall was identified as being caused by high latency inherent in standard Ethernet, acting as a critical bottleneck for distributed calculation steps.
8:51 RDMA Latency Reduction: Implementing RDMA over Thunderbolt 5 directly connecting the Macs is claimed to reduce inter-node latency by 99%.
9:50 Llama 3.3 Performance via RDMA (4 Nodes): Switching to RDMA with tensor sharding for the Llama 3.3 model across four Macs increased throughput to 15.5 t/s, representing a 3.25x speed increase over the single-node baseline. Time-to-first-token (TTFT) improved to just over 1 second.
10:26 Kimmy K2 Performance (MoE Scaling): The larger Kimmy K2 instruct model (4-bit quantization, 540 GB) was tested. Pipelining over two machines yielded 25 t/s. Scaling to all four machines increased throughput to 35 t/s. TTFT improved from 6.5 seconds down to 1.5 seconds.
14:09 Scaling Dynamics: Performance scaling is significantly more efficient for dense models (like Llama) compared to Mixture of Experts (MoE) models (like Kimmy K2 or Deepseek v3.1), as MoE models require more numerous, smaller calculations that exacerbate networking overhead in the current software iteration.
14:51 Deepseek v3.1 Performance (MoE): The 671 billion parameter Deepseek v3.1 (8-bit quant, 37B active parameters) scaled poorly, moving from 20 t/s on two machines to only 24 t/s on four machines. TTFT, however, improved notably from 3.5 seconds to under 2 seconds.
17:24 Power Draw Comparison: The four-Mac cluster drew approximately 600 watts when running the dense Llama 3.3 model. This is less power consumption than a single Nvidia H200 GPU (estimated at $30,000 USD), which, while faster (approx. 50 t/s on a half-size FP8 model), requires significantly more power.
17:49 Stability and Software Maturity: The setup is utilizing highly beta features for CPU/GPU synchronization, leading to occasional instability and crashes.
16:21 Software Ecosystem: While Exo provides an OpenAI-compatible API, full integration with advanced AI coding agents (e.g., OpenCode) and standard development tools is currently limited, necessitating interaction primarily through basic software environments (e.g., Jan).