Scaling and benchmarking a critical message bus using a new indexing strategy

50 points by eatonphil a day ago on hackernews | 4 comments

Linear scans break at scale. Partitioning by prefix and using pooled 1024-entry blocks is the right move to prevent 2 GB worst-case index bloat.

: hansvm | 16 hours ago
Yes, they do, but the extent to which that matters varies greatly. Accuracy/time tradeoffs exist (and are moderately common at $WORK right now). If you provably can't do better than a linear scan (and benefit from the increased accuracy from doing so at a business level), you might as well lean into it and choose a dead-simple, CPU-friendly algorithm for your problem.

rtpg | 16 hours ago

This is pretty funny to see, because every financial firm interview process I've seen involves some variant of solving a bunch of stuff with min heaps.

Never before has a set of engineers been more primed to solve a problem

jgalt212 | 9 hours ago

To me, buses seem like a single point of failure your app / service would be do best to avoid depending on. I won't argue you shouldn't publish everything to the bus, but I will argue you should minimize the number of subscribers.