I'm building a browser-based log viewer. The backend holds up to 2M log entries in a ring buffer. Logs can be 1 to 100 lines (may be more) each (stack traces, multi-line output). The frontend fetches ~200 entries at a time via WebSocket (By design can modify). Users need to scroll through the entire buffer smoothly, including tailing live logs at 20-40K/sec.
What I've tried
Sliding window with React DOM: Fetch more rows when user scrolls to edges. Programmatic scroll changes trigger scroll events, causing feedback loops and jumps.
Virtual sentinel: Tall placeholder div with only visible rows rendered. Works for fixed-height rows, but breaks when rows vary in height since you need to know heights before fetching the data.
Flatten multi-line logs into single-line rows: Fixes the height problem but multiplies row count per update, causing jank and messy index mapping.
xterm.js: Smooth rendering via canvas, but only supports appending. Can't load older entries when user scrolls to the top without losing scroll position.
Core problem
Virtual scrolling needs row heights upfront. With variable-height rows and remote data, you can't know heights until you fetch and render. Every workaround introduces a new issue.
How do production log viewers (Datadog, Grafana, DevTools) solve this? Any libraries or patterns that handle variable-height virtual scroll with on-demand remote data?
Stack: React, Node.js backend, Socket.IO.