Pushing the Limits of Big Data in the Browser

A radical new take on how to architect large data applications entirely in JavaScript – no python required. This talk covers how I built the open-source libraries Hyparquet and HighTable for loading Apache Parquet files client-side in the browser.

00:00 - Introduction & Motivation 00:44 - The Problem with Current Data Tools 02:12 - Time to First Data: Why Speed Matters 03:05 - Traditional Backend Architectures & Their Drawbacks 04:22 - Burn the Backend: Embracing Simplicity 05:13 - OpenAI vs Anthropic: Different Approaches to Data Processing 06:35 - Building Frontend-First Apps Without a Backend 07:47 - Local-First Apps: Beyond Privacy and Sync 08:45 - JSCAD: Fully Browser-Based 3D CAD 10:10 - Cloud-Native File Formats: GeoTIFF & Parquet 11:29 - Querying Parquet Files Directly in the Browser 12:20 - Benchmarking Parquet Loading Methods 13:29 - Making JavaScript Fast for Data Engineering 14:49 - WASM Optimizations for Decompression 16:48 - Building a High-Performance Data Viewer 17:46 - Live Demo: Hyperparam Data Viewer 19:18 - What’s Next: Apache Iceberg in the Browser 20:46 - The Case for Better JavaScript Data Tools 21:45 - Rethinking the Need for Backends 22:35 - The Future of Cloud-Native Data Formats 22:53 - Join the Project & Final Thoughts