Scaling OpenSearch: Next-Generation Shard Allo...

Scaling OpenSearch: Next-Generation Shard Allocation for High-Performance Clusters - Rishab Nahata, Arpit Bandejiya & Himshikha Gupta, In large-scale OpenSearch clusters, shard allocation has long been a significant bottleneck, often leading to extended waiting times and potential API timeouts. This session will showcase enhancements that makes reroute iterations time-bound, resulting in up to 90% improvement in allocation performance. We’ll dive deep into the technical journey of how we transformed cluster operations from taking several minutes to seconds. The session will walk you through our innovative approach to optimising allocators and allocation deciders. Through real-world examples and performance metrics, we’ll see the dramatic impact these improvements have on cluster operations. Key highlights include the architecture behind time-bound reroute iterations, the challenges we overcame during implementation, and the remarkable performance gains achieved in production environments. We’ll share actual case studies demonstrating how these optimisations have helped cluster management for large clusters. Whether you’re managing a growing OpenSearch cluster or architecting a new large-scale cluster, you’ll leave this session with practical insights into achieving faster and more efficient shard allocations.