▶ YouTube Summaries

• YT :: https://www.youtube.com/watch?v=x6WqD4qHxSc
• Original title :: Scaling Slack and GitHub with Maude Lemaire

Gergely interviews Maude Lemaire (author of "Refactoring at Scale"): eight years on Slack's backend performance team, database scaling at GitHub, and time-to-first-token work at Cursor. Themes: team-sharded architectures breaking under enterprise customers, migrating hot tables safely, load testing with realistic websocket traffic, scatter queries hidden behind ActiveRecord, why LLMs help little with performance investigation, and rotating engineers between product and product-platform teams.

Slack enterprise growing pains (2016+)

• Everything was sharded by team; a big team got a bigger database, until the biggest instance AWS sells was not enough. Websocket servers were also assigned per company.
• Enterprise Grid modelled a big company as many small "teams" plus an enterprise row; cross-team DMs and company-wide channels all hit the single enterprise shard.
• Channel membership lived in three tables (DMs, public channels, private channels + group DMs), so sidebar and mention-badge queries were big unions.
• Event floods: power users (e.g. an IBM admin in ~30 workspaces) got every shared message once per workspace - no dedup - and their clients kept dying. Distinct from thundering herds.
• Thundering herd example: CEO posts "Friday off" to a 100k channel; everyone clicks the notification, reads history and adds custom emoji, each reaction fanning out to 100k sockets. Mitigation: prioritize websocket message types and drop emoji under load; cap of 23 distinct custom emoji per message.

Migrating the membership tables

• No ORM; ~400 call sites with inline SQL. Step one: regex them out and copy each query verbatim into a thin, clearly named data-access layer - no optimization, just one choke point.
• Heavy feature-flagging, double writes, backfills, shadow reads with diffs tracked in warehouse dashboards.
• War story: flipped internal Slack to read from the new table, saw a discrepancy, habitually nuked and re-backfilled - with the flag still on. Everyone at Slack had a blank sidebar for ~30 minutes (during the after-work gathering, so few noticed).

Load testing (Koi Pond)

• Start from how the product actually falls over; product judgment decides what to simulate and what to drop.
• A 500k-user customer (likely Amazon), pulled forward by COVID, needed 500,000 live websocket connections - API-only load tests missed the real bottlenecks.
• Headless Chrome per user in Kubernetes: $0.37 per instance per 5-hour test, ~$150k per run, half the time spent restarting crashed pods.
• Replacement: lightweight Go processes, each holding a websocket and replaying API + edge-cache request patterns. Lower fidelity, cheap and repeatable.
• A recorder slash command captured every API call and websocket message of a real flow, turned into scripts for the fake clients; PMs were encouraged to record their own flows for automated testing.
• Data model later moved to Vitess with channel as the sharding key.

GitHub (2024)

• Still largely in own data centers, running out of space; mandate: move to Azure and "do less" work per request.
• Only 16 Vitess shards for all pull requests and issues, because scatter queries (no sharding key) hit every shard - more shards would make them worse.
• ActiveRecord hides SQL entirely: a five-line function with no visible query; nobody sees the cost of a new line of code. Domain extraction to Go services (starting with stars) was attempted and later abandoned.
• The repo home page ran 100+ MySQL queries across at least three clusters; ActiveRecord caches only identical queries; nearly every count is computed live. Maude's view: show "99+" and stop counting.

AI at work (Cursor time-to-first-token)

• LLMs helped almost nothing with performance investigation; they speed up implementation once the fix is known.
• They instrument badly in async code and write tests that merely confirm the code - she uses mutation-testing skills to force real edge-case coverage.
• Traces had millions of spans and sampling was added in scattered places; sample in one place only.
• She had an LLM produce a sequence diagram of the TTFT path, printed it on blueprint paper and annotated it by hand while reading code.
• Opus proposed parallelizing four independent tasks - useless, since one dominated latency and ~1% of users triggered the others. Real wins were mundane, like longer Redis TTLs.
• Verifies everything manually: platform work must cover the whole foundation, while AI tends to bolt on "an extra bathroom past the foundation" instead of questioning assumptions.

Product vs product-platform

• Good platform engineers come from product engineers who solved the same problem several times and want to solve it once.
• Platform teams drift from product reality; fix with rotations of product engineers onto platform teams (like embedded SRE).
• With product teams shipping AI-generated code fast, platform guardrails matter more than ever - "the spaghetti now wants out of the plate".