▶ YouTube Summaries

• YT :: https://www.youtube.com/watch?v=a0sZV0qIbI0
• Original title :: Building resilient systems with Sam Newman

Gergely Orosz talks to Sam Newman, author of Building Microservices and the new Building Resilient Distributed Systems. Sam calls microservices an architecture of last resort, justified mainly by organizational autonomy and independent deployability. He walks through his three rules of distributed systems (it takes time, the other side may be gone, resources are finite), observability, idempotency keys versus fingerprints, thundering herds, fail open versus fail closed as a business decision, and David Woods' four dimensions of resilience. On AI he is cautious: hedge vendors and models, use LLMs to produce deterministic code, beware cognitive debt and cognitive surrender, and keep humans owning a modular architecture while AI works inside the modules.

Background: Thoughtworks and the rise of microservices

• Path :: Acorn Electron and ZX Spectrum in the UK, software engineering degree with a sandwich year at an Alstom unit (software for turbines and the European Space Agency). A Fortran 77 codebase had a 99,999 break condition on every loop; he replaced them all with a regex refactoring after two days with an O'Reilly regex book, "the closest I've felt like a god".
• Thoughtworks :: Joined at under 400 people (about 4,000 at his departure), an XP and London-school TDD shop. Built early build pipelines around 2004, first tech lead was Dave Farley, interviewed Jez Humble as his backfill, and spent a year and a half at Google teaching test automation. Early on a constant churn of reinvention (he built DBDeploy, a database refactoring tool, and two CruiseControl replacements).
• Programming is not typing :: Martin Fowler's line, and the core of every argument with clients against pairing. Peter Naur's "program as a theory in the minds of developers": code is a side effect of people building a system together, which is why AI does not change the collaborative nature of the work.
• Origin of the term :: James Lewis pitched "micro apps" at an architectural symposium in the Lake District around 2011-12; someone said "more like services". It grew out of fine-grained SOA (the "gorilla SOA" movement against SOAP). Sam came at it from continuous delivery: if you want to change things often, the integration of what you change is the impediment.
• Real definition :: An opinionated style of SOA. First opinion, and the one that matters: services are independently deployable, one service at a time. Second, softer: boundaries follow the business domain rather than technical layers.
• Uber example :: Gergely recalls many services created because of a mandate that anything new must be its own service, not because of design. Years later teams were merging them by domain.
• Last resort :: Microservices push you into aggressive distribution of state, so you need a good reason. The best reason is organizational autonomy for large organizations with lots of developers and middle management; microservices are a means to that end, not magic. People adopt them and expect other benefits to appear.
• Dotcoms versus enterprise :: Tech company domains are simple and evolve together with the software. Enterprise domains encode decades of paper-based processes, which is why DDD (2004) was about modelling existing non-digital processes. AI-native startups versus incumbents may repeat this: incumbents will retrofit and their corporate DNA will crush much of it.

Specs versus code as source of truth

• Production is truth :: Quoting Charity Majors. Everything else (code, specs) is a definition of what the system should do, and there is an ongoing debate about where that definition lives.
• Spectrum of spec-driven development :: Birgitta Boeckeler's article on Martin Fowler's site: spec-first (write code, discard spec), spec-anchored (keep the spec and update it), and companies that never look at the code. Tools: Kiro, Spec Kit, Tessl. Open question whether the world is ready to treat the spec as the source of truth.
• Tacit knowledge :: Chris Ford of Thoughtworks calls it the biggest problem of AI-driven delivery. Code review surfaced "this doesn't feel right" instincts. To work at the spec level you must unpack and write down what you know, and defining what good looks like is a precondition.
• Shared mental model breaks :: People "pairing" with an AI assistant instead of a human go off in separate directions, creating context switching and losing the big picture. Pairing also spread context (bus factor); without code review that sharing is lost.
• Human review dropping :: He is fine with a respected firm that dropped human code review to spend the time on design docs, effects and observability; not fine if nobody does the thinking instead.
• Prerequisites :: Several speakers at an O'Reilly event on spec-driven delivery said it does not work unless you can define what good looks like and verify it. Teams feed observability and OTel data back into the agent.

The book: three rules, observability, resilience trade-offs

• Why the book :: A rubber ring for people who find themselves building a distributed system; starts with simple foundations (chapter two observability, three timeouts), grows into resilience engineering, with real incidents. He first met resilience engineering via an impenetrable David Woods paper around 2015-16 and spent years digesting it. Honesty about mistakes and public post-incident reviews (Cloudflare is the model, fast and self-critical, versus marketing-style writeups) are how the industry improves; DORA now forces some disclosure.
• Rule 1 :: You cannot send information instantaneously. Physics, plus BGP having a bad day.
• Rule 2 :: Sometimes the thing you want to talk to is not there: a SAN with SSDs on fire, rabbits eating a network cable in an office duct.
• Rule 3 :: Resource pools are not infinite (rule contributed by Ben Cole). Most outages come down to some resource saturated somewhere, often triggered by a retry storm. Remember these before CAP, PACELC or Byzantine consensus.
• Observability :: A property of a system. A single process gives high-quality signals for free (debugger, thread dump); distributed systems need extra work, and it is much easier from the start than retrofitted. Treat raw signals as a stream of structured events and build up traces (spans sharing a trace ID) and then SLIs and SLOs. You cannot choose a timeout without knowing the latency distribution. Vendor-agnostic OpenTelemetry helps. SLAs are what you are sued over; only meeting the SLA leaves customers unhappy.
• How resilient do you need to be :: Resilience is a degree, not binary; it needs a threat-model-like north star. Monzo runs a stripped-down standby bank, rewritten from scratch on a different cloud, at about a tenth of the cost. Zoom after the AWS US-East-1 outage likely decided multi-region was not worth the cost, and that is a legitimate decision as long as it matches what users expect. Technologists must be able to explain trade-offs (cost, latency, build time) to business stakeholders. A resilient system is one that keeps changing; one that does not is brittle.

Idempotency, thundering herds, fail open or closed

• Idempotent operations :: Applying them repeatedly has the effect once (lift button, delete). Payments: when no response arrives you cannot tell whether the request was processed, so retries need to be safe.
• Idempotency keys :: Client generates a unique ID (UUID), server remembers processed keys. Used by Stripe, PayPal and AWS APIs. Needs support on both sides, hard to retrofit, so use it from the start for anything that needs it.
• Fingerprints :: Server-side hash of the request fields, a retrofit option. Gives false positives on legitimate repeats (Danish payment terminals rejecting a second identical 40 kroner payment within minutes; launching 20 identical EC2 instances in a loop), so keep them for a limited time. Best used together with keys: same key with changed parameters can be rejected as invalid.
• Thundering herds :: Any mismatch of requests to resources. Kinds: denial of service (use a CDN or Cloudflare), self-inflicted (cache restart causing misses that overload the origin, so persist cache snapshots or keep services off until the cache refills), retry storms (Square's 2017 Multipass outage: 500 hard-coded immediate retries against Redis), and legitimate success (Twitter, healthcare.gov launching in 36 states at once; fix with constrained signups or phased launch). Scaling out is not a defence against DDoS: it becomes a denial of wallet.
• Fail open or fail closed :: Same technical problem, different business answer. E-commerce can sell an item it is unsure it has (back-order or refund); concert tickets for a big tour cannot. Uber failed open on uncertain payments during the growth phase and failed closed when the focus moved to profitability. The decision is per feature and even per value (a 10 euro ride versus 1,000). CAP and PACELC come down to "I cannot talk to that part, what should we do", which starts as a conversation with the product owner.

Resilience engineering: four dimensions

Based on David Woods:

• Robustness :: Absorb known perturbations (Kubernetes replacing a failed pod). Most engineers stop here. Robustness measures add complexity and new failure modes, which is not solvable, only to be aware of; it is also why people skip multi-region or multi-cloud.
• Rebound :: Time to detect and recover (DR, chaos monkey, on-call drills). Requires humility that robustness may have failed, and regular practice because systems change. Shout-out to Uptime Labs for incident training.
• Graceful extensibility :: Dealing with surprise, which is about humans. Command-and-control organizations with narrow job descriptions do it badly. Drills help, such as Google SRE's wheel of misfortune. Meta's BGP and DNS outage, where physical access depended on their own domain, is a lesson to check whether your physical security depends on your own systems.
• Sustained adaptability :: Continual learning and adapting, including when there is no incident: read others' incident reports (the VOID repository). The last two dimensions are people and culture, and psychological safety is required; the last three chapters of the book cover this.

AI: resilience, cognitive debt, modular architecture

• Macro versus micro :: AI helps at the tactical level (correlation analysis, incident.io-style post-incident support). The macro economics worry him: analysts say the big labs need about 2.6 trillion in revenue by 2030 to justify capex while the whole software market is about 1.4 trillion. He cannot fix that, but can hedge.
• Hedging :: Be multi-vendor and multi-model, use open-weight options, take fallback routes via Google, Amazon or Microsoft because the labs' own availability is poor, break monolithic agents into workflows and replace parts with deterministic code. Charity's point: distributed systems were never deterministic anyway.
• What LLMs are :: No concept of causality ("why did the LLM delete my database"); not world models. Guardrails are not the long-term answer, and we overestimate them because they seem smart. Prefer having the agent call a script that is known to work over reimplementing it. He uses AI for research (NotebookLM, clicking every link to check) but does not use it for writing.
• Cognitive debt :: Margaret-Anne Storey's term: erosion of shared mental model and critical thinking as people use AI in different directions, analogous to technical debt.
• Cognitive surrender :: LGTM on anything the AI produces: AI writing documents no one reads, AI reading them. Oncologist example: an LLM that flags scans you might have missed makes you better, one that hands you results to sign off turns the expert into an approver. AI was meant to free us from drudgery, but evidence shows more context switching and longer hours.
• Complexity :: He hoped the platform space would see a complexity collapse; instead AI is added on top of old complexity.
• Modular architecture :: Humans define modules and the connections between them (mental model, easy to change gaps); AI goes "hog wild" inside the boundaries. Shopify's modular monolith as an example. Not one-size-fits-all: try a software factory (specify interface, verify) for low-risk modules with good tests, SLOs and cheap rewrites, keep collaborative work on critical ones.
• Starting a factory :: Begin with spec-driven development, then decide how to verify output along three axes (Kevin Henney's reframing of non-functional requirements): development, operational and business requirements. If you cannot say what good looks like, do specs but not a factory. It may be model-driven development all over again.

Learning architecture and where to find Sam

• Books :: Software Architecture Fundamentals (Neal Ford, Mark Richards), Vlad Khononov's Balanced Coupling (and his DDD book, plus skills for Claude), Parnas' 1971-72 papers on information hiding. George Fairbanks' "architecturally evident coding style".
• Modularize today :: Even in an existing codebase, start drawing boundaries; with decent tests and statically typed languages or modern LLM-friendly Python and Ruby there is little excuse. Code should represent the mental model, since we communicate through code and maybe specs.
• Where :: samnewman.io (training, consulting, talks), the Modern Software Engineering YouTube channel run by Dave Farley, and the book on shelves and in digital form on the O'Reilly platform.