Context:
I'm designing a bare-metal Cassandra 5.0.7 cluster and sizing nodes for hot data storage. I have not yet purchased the hardware, so I'm still open to feedback on the specs. Before committing, I want to understand how much the density ceiling has shifted with modern Cassandra.
The most referenced Stack Overflow thread I found on this topic is from 2015 (Cassandra 2.x):
The answers there point out real problems, bootstrapping takes a long time, repairs are heavy, and streaming can be fragile. But those points were made before a lot of important changes were made to Cassandra, especially:
- Zero Copy Streaming (4.0): dramatically faster node streaming
- Incremental repair improvements (4.0): production-ready and much lighter on resources
- UnifiedCompactionStrategy / UCS (5.0): better SSTable size control, parallel compaction sharding, hot-swappable config
- Trie Memtables + BTI SSTable format (5.0): reduced GC pressure, better write throughput and tail latency
My planned setup (hardware not yet purchased):
- Apache Cassandra 5.0.7
- Bare metal, NVMe SSDs
- 32 cores / 64 GB RAM per node (planned, feedback welcome)
- Hot data
- Replication factor 3
- Target density: ~16-20 TB/node
My questions:
1. With Cassandra 5.x improvements (especially UCS + ZCS + incremental repair), has the practical ceiling for hot data per node shifted significantly compared to the 2015 recommendations?
2. The old thread frequently cited 1-5 TB as a safe range. Does that guidance still apply to Cassandra 5.x, or has it been meaningfully pushed up?
3. Has anyone run clusters at 10 TB+ per node on Cassandra 5.x with hot data and what latency requirements? What were the main operational pain points?
4. Does 32 cores / 64 GB RAM seem reasonable for this density and workload, or would you size differently, particularly given the compaction and repair concurrency needs at 16-20 TB/node?
5. What monitoring signals should I watch to detect when I'm approaching the density limit for a given workload (e.g., compaction backlog growth, repair duration trends)?
I'm aware of Jon Haddad's 2025 blog series targeting 20 TB/node (rustyrazorblade.com), but that seems aimed at cold or mixed workloads. I'm looking for real-world feedback, if possible specifically on hot data at high density.
Any production experience or benchmarks would be highly appreciated.