Breaking Heterogeneity Barriers: Unified Cloud-to-Robot AI System SW Stack for...

Breaking Heterogeneity Barriers: Unified Cloud-to-Robot AI System SW Stack for Embodied Intelligence - Yonghua Lin, BAAI Embodied AI demands unprecedented efficiency: brain-planning models (VLM) evolve in the cloud while sensing-action models (VLA) run on resource-constrained robots. This dual-stack paradigm introduces critical challenges including compute fragmentation (requiring 1,000+ GPU clusters for VLMs vs ≤10ms response for VLAs), hardware heterogeneity across cloud and edge devices, and latency cliffs in end-to-end execution pipeline. Multi-robot scenarios further exacerbate these challenges, often exceeding human-level response standards (≤100ms) and causing system instability. In this talk, we present FlagOS - our PyTorch-native unified AI system software stack that seamlessly extends PyTorch’s capabilities to embodied AI. Through its parallel framework FlagScale, unified communication library FlagCX, and underlying Triton-based operator library FlagGEM with AI compiler FlagTree, FlagOS provides a PyTorch-based solution for Embodied AI. We’ll demonstrate how this stack powers innovative embodied AI solutions like RoboBrain and RoboOS across diverse robot hardware platforms, effectively mitigating the challenges posed by hardware heterogeneity.