Building a Super Powerful AI Cluster with Mac Studios: An Exploration with XO Labs
0:03 Goal: The aim is to create a powerful AI cluster using five Mac Studios to run the "baddest AI models."
0:11 XO Labs: The project utilizes XO Labs, a new and beta platform for AI clustering, which supports various hardware including Raspberry Pis and gaming PCs with high-end GPUs.
0:24 Mac Studios: Mac Studios are used for this video as the primary hardware for AI clustering, driven by curiosity.
0:46 Why AI Clusters: The purpose is to run AI models locally, avoiding reliance on large companies and keeping data private.
0:54 Parameter Explanation: The video clarifies the concept of parameters in AI models (B = billion), explaining that more parameters generally lead to better reasoning and factual accuracy.
0:54 VRAM Requirements: Different AI models have different VRAM requirements. LAMA 3.2 3B requires 4GB of VRAM, LAMA 3 14B requires 40GB, and LAMA 3. 405B requires 1.4TB.
0:30 Quantization: Quantization reduces model size for smaller GPUs but can sacrifice accuracy.
0:40 Unified Memory Architecture: M-series Macs utilize a unified memory architecture, where system RAM can be used for GPU tasks, offering advantages over traditional systems with separate VRAM and system memory.
0:50 Cluster Setup: Five M2 Ultra Mac Studios with 64GB RAM each are connected via a 10 Gigabit Ethernet switch, noted to be a potential bottleneck.
0:09 Networking Bottleneck: The 10 Gigabit Ethernet connection is identified as the biggest bottleneck, compared to faster networking solutions.
0:10 How XO Works: XO Labs distributes the AI model across each Mac, with each Mac running part of the job, necessitating efficient communication.
0:11 Thunderbolt Connection: A Thunderbolt hub is used as a bridge to attempt to improve networking speeds, though performance is tested against Ethernet.
0:11 Installation: The video briefly covers the installation process, including Python 3.12 and MLX installation.
0:14 XO GUI: XO Labs provides a GUI for interaction, model selection, and testing.
0:15 Downloading models: Initially, model downloads appear to be happening on each individual machine, which is inefficient.
0:16 Benchmarking: Initial tests show that clustering with five Macs results in 29 tokens per second, compared to 117 tokens per second with just one, indicating a network bottleneck.
0:16 Performance: Running the same model over the thunderbolt connection saw negligible improvements.
0:19 Thunderbolt Test: Tests are conducted using a Thunderbolt hub as a networking bridge.
0:21 70B Model: The LAMA 3.3 70B model is tested with both Thunderbolt and 10 Gigabit Ethernet connections.
0:21 Performance: The 70B model had 15 tokens a second on one host.
0:21 Slow downloads: Model download times are long and painful. Feature request to allow local hosting of models for faster distribution.
0:24 Testing: The video experiments to compare Thunderbolt vs. Ethernet
0:24 Issue: It seems that the setup isn't using the Thunderbolt bridge.
0:26 Deep Seek R One: The most ambitious test involves Deep Seek R One, a massive model.
0:26 Download Problem: It attempts to download an almost 200 gig model on each host, prompting high RAM usage and the use of swap space, which is undesirable for performance reasons.
0:29 Inference Engine: A special command and inference engine were used, but filling the RAM across the board leads to slowdown and timing out of the system.
0:30 Performance: The Deep Seek model ran at 0.6 tokens/second. The network bottleneck and no GPU usage led to troubleshooting efforts.
0:30 Alama: Alama is tested because it makes it very simple to download models on one machine.
0:30 Russdesk: Used because the AI model requires running Linux.
0:33 Fabric Testing: Tests of the Fabric API are run.
0:34 Summary: In summary, the video highlights both the potential and the challenges of using Mac Studios for AI clustering, emphasizing the need for better networking solutions to fully realize their capabilities.