Repeatable Benchmarking: An Exploration of OpenSearch Vs Elasticsearch

Repeatable Benchmarking: An Exploration of OpenSearch Vs Elasticsearch - Evan Downing, Trail of Bits Ever wonder how OpenSearch and Elasticsearch compare in real-world performance scenarios? This talk presents findings from Trail of Bits’ comprehensive benchmarking assessment of both engines across multiple workloads. We’ll dive into surprising performance differences where OpenSearch v2.17 achieved 1500x faster execution on certain date histogram operations, while Elasticsearch v8.15 was faster at text queries. Using OpenSearch Benchmark, we measured 90th percentile service times across Big5, NYC Taxis, PMC, NOAA, and Vectorsearch workloads, identifying where each engine excels and where each needs improvement. We’ll explore the statistical validation of our results, performance trajectory across versions, and recommendations for optimization opportunities. We’ll reveal performance inconsistencies, outliers, and configuration experiments that impacted results. Today, our work continues on newer releases of both platforms, with preliminary findings suggesting improved performance for both engines. Engineers from both communities will gain valuable insights into operational strengths and areas for improvement in their preferred search engine.