mxfp8, mxfp4, nvfp4 formats and applications in PyTorch

mxfp8, mxfp4, nvfp4 formats and applications in PyTorch - Vasily Kuznetsov & Driss Guessous, Meta The introduction of MicroScaling formats (MX) by the Open Compute Project has led to native hardware support by major hardware vendors. This talk will explore the end-to-end workflow of utilizing MX formats on modern hardware, highlighting the performance benefits and challenges associated with these new dtypes. Key topics include: Smaller Bit Widths: Understanding the impact of smaller bit widths on precision and throughput, referencing the OCP MicroScaling Formats Spec. And PyTorch’s plan to tackle increasing frequency of lower precision formats. Low Precision Training and Inference: Discussing the challenges posed by the reduced precision of these new dtypes and how to utilize them for training and inference.