Critics questioned the JAX benchmark and choice of C.
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
Next will be writing the inference stack in C for simultaneous high-speed RL across a large block of GB300s.
(We do use a little C++ tbh, but not much)
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
Yes.
It’s not that we’ve discovered some magic bullet, but rather that JAX, or at least the open source version of it, is mostly optimized for small to medium-sized training runs on Google TPUs, whereas we need to massive training runs on Nvidia GPUs.
Pipeline parallelism is essential and crushes fully-sharded data parallelism at scale.
And C will compile to the most efficient binary short of assembly. Maybe we will do a little assembly too.
There’s a ton of headroom to better optimize AI training and inference.
Note, I am posted this to encourage those who enjoy getting incredible performance out of hardware to join SpaceX
Next will be writing the inference stack in C for simultaneous high-speed RL across a large block of GB300s.
(We do use a little C++ tbh, but not much)
Critics questioned the JAX benchmark and choice of C.
SpaceX has almost finished writing V1.0 of an in-house AI training stack in C that exact-maps to 220k GB300s with 800G NICs, making heavy use of pipeline parallelism and getting as close to bare metal as possible.
The potential speed improvement vs JAX for large training runs is over an order of magnitude.
Positive users praise SpaceX's custom C-based AI training stack for 220k GPUs as deep engineering that could deliver major speed gains, while negative users direct personal insults at Elon Musk.
