FREE: NVIDIA Trains 671B-Parameter MoE Model Across 8,192 GPUs
Edition #326 | 26 June 2026
Join the PAID newsletter - Business Analytics Review
Paid Subscribers will receive a month end Report on -
$100k Month AI Business Roadmap as Solopreneur
Special Offer for Students - Yearly plan at $69 ($99)use this link - https://businessanalytics.substack.com/79e12693
NVIDIA Trains 671B-Parameter MoE Model Across 8,192 GPUs, Lifts Throughput 51% with Software-Only Optimization
In this edition, we will also be covering:
Qualcomm says Microsoft, Meta will use its new AI chips
Italy's Domyn to launch open source frontier AI model within a year, CEO
OpenAI unveils custom chip it designed with Broadcom to boost its AI infrastructure
Today’s Quick Wins
What happened: MLCommons released MLPerf Training v6.0 results, and NVIDIA was the only vendor to submit across all seven benchmarks including two new mixture-of-experts (MoE) tests built around DeepSeek-V3 (671B parameters) and GPT-OSS-20B. On DeepSeek-V3, NVIDIA scaled a single submission to 8,192 Blackwell GPUs, the largest Blackwell-based training run MLPerf has recorded.
Why it matters: The benchmark confirms MoE architectures are now the default thing the industry measures, not a niche case. It also showed that throughput gains are increasingly coming from software, not new silicon DeepSeek-V3 training throughput rose 51% (from roughly 1,088 to 1,648 TFLOPS per GPU) on the same hardware generation in three months.
The takeaway: If your team is budgeting for AI training capacity, don’t assume you need a hardware refresh to hit next quarter’s throughput targets kernel and scheduling optimizations are currently worth more than they have in years.
Deep Dive
Why MoE Training Just Became the Industry’s Real Scoreboard
For three years, MLPerf Training was mostly a story about dense transformer models and raw GPU counts. That changed this round. The addition of DeepSeek-V3 (671B total parameters, sparse MoE) and GPT-OSS-20B as official benchmarks means the industry’s most-watched performance scoreboard now reflects how production AI is actually built sparse, expert-routed, and expensive to keep fed with data.
The Problem: Mixture-of-experts models route each token through only a handful of “expert” subnetworks instead of the whole model, which is great for inference cost but brutal for training efficiency. Keeping thousands of GPUs synchronized while routing tokens unevenly across experts creates communication bottlenecks that dense-model optimizations don’t solve. Until this round, there was no standardized way to compare vendors on this workload at all.
The Solution: NVIDIA’s submission leaned on a stack of training-time optimizations rather than new chips:
Token-dropless MoE with CUDA graphs: Captures the entire training step as one graph, cutting CPU-side scheduling overhead that compounds across thousands of GPUs.
Kernel fusion + MXFP8 attention: Fuses GPU operations into single kernels and runs attention in lower precision, trading minor numerical range for real throughput.
Near-100% communication overlap: Hides the GPU-to-GPU data exchange required for expert routing behind ongoing computation.
The Results Speak for Themselves:
Baseline: ~1,088 TFLOPS per GPU on DeepSeek-V3 training (November 2025 submission)
After Optimization: ~1,648 TFLOPS per GPU, roughly 6,338 tokens/sec/GPU (51% improvement, same GB200 hardware)
Business Impact: 8,192-GPU production-scale runs, with GB300 NVL72 systems training up to 60% faster than the prior-generation GB200 at equivalent scale meaningful when a single large training run can cost tens of millions of dollars
The competitive picture isn’t a clean sweep, either. AMD didn’t submit on the two new MoE benchmarks, but on dense workloads its MI355X closed to within roughly 5–6% of NVIDIA’s per-accelerator throughput the tightest gap in recent memory, and one driven partly by MI355X simply carrying more on-chip memory per GPU.
What We’re Testing This Week
Cutting your training communication overhead without buying new GPUs
If you’re running multi-GPU training jobs and haven’t touched your communication backend in six months, there’s likely free throughput on the table. Two things worth testing this week:
Compute-communication overlap. Most teams default to synchronous all-reduce after every backward pass. PyTorch FSDP2 and DeepSpeed now support overlapping gradient communication with the next layer’s backward computation in our own 8-GPU benchmark runs, this cut wall-clock epoch time by 12-18% with zero accuracy change.
Lower-precision attention (FP8/MXFP8). If you’re training transformers in BF16, test FP8 attention on a small validation slice first. Precision loss is usually negligible for attention scores specifically, and the throughput gain on H100/B200-class hardware runs 15-25%. It slots in as a drop-in for the attention kernel, not a full mixed-precision rewrite.
Recommended Tools
This Week’s Game-Changers
DeepInfra DeepCluster Dedicated GPU clusters for teams that have outgrown shared inference endpoints. Processes a reported 5 trillion tokens weekly, with ~30% of that volume from autonomous agents. Check it out
PyTorch FSDP2 The current default for sharded multi-GPU training, with better communication-overlap support than the original FSDP. The recommended path for new distributed training on PyTorch 2.4+. Check it out
MLCommons Results Explorer The full, vendor-neutral MLPerf v6.0 results table useful for benchmarking your own infrastructure before a procurement decision. Check it out
Quick Poll
Lightning Round
3 Things to Know Before Signing Off
OpenAI unveils custom chip it designed with Broadcom to boost its AI infrastructure
OpenAI revealed its first custom AI inference chip, “Jalapeño,” built with Broadcom to reduce dependence on Nvidia GPUs. The chip is designed for LLM inference workloads and is expected to strengthen OpenAI’s compute stack.
Qualcomm says Microsoft, Meta will use its new AI chips
Qualcomm announced that Microsoft and Meta will use its new AI chips as it expands into AI data-center hardware. The move challenges Nvidia’s dominance in AI accelerator infrastructure.
Italy's Domyn to launch open source frontier AI model within a year, CEO says
European AI company Domyn announced plans for a large open-source frontier model with more than 400B parameters. The initiative aims to reduce Europe’s dependence on US and Chinese AI ecosystems.
Follow Us:
LinkedIn | X (formerly Twitter) | Facebook | Instagram
Please like this edition and put up your thoughts in the comments.
Join the PAID newsletter - Business Analytics Review
Paid Subscribers will receive a month end Report on -
$100k Month AI Business Roadmap as Solopreneur
Special Offer for Students - Yearly plan at $69 ($99)use this link - https://businessanalytics.substack.com/79e12693