▶ YouTube Summaries

← All summaries
fknight#▶ Watch

Why AI gains vary: habits of Amazon's 10x teams

2026-11-09 Mon ⏱ 18 min fknight

Forrest Knight asks why developers with the same models and harnesses see wildly different gains from AI, from under 2x to thousands of PRs a month. The central story is an Amazon experiment from 2025: 50 ordinary teams working on existing systems, all given the same AI tool (Q / Kiro). Half came in under 3x, the median was about 4.5x, and some teams exceeded 10x. The difference was not money or model but changed ways of working: five habits (write everything down, accept a slowdown first, do not babysit agents, spec before code, invest in tests and pipeline). All of this happened before Opus 4.5, so current potential is likely higher. A short Grok Bot sponsor segment sits in the middle.

The question

  • Claim :: for two years developers have heard that AI makes them 10x. On side projects the author sees gains well beyond 10x, but side projects are not representative.
  • Observation :: friends at AI labs, a freshly funded Silicon Valley startup, banks, defense and healthcare report very different results with the same models and harnesses. Some get at most 2x; Lauren Tan shipped about 2,000 good PRs in a month to a real production codebase.
  • Other extreme :: Anthropic rebuilt its source control because Git could not handle roughly 30,000 merges a day.
  • Not about money :: labs have effectively unlimited compute and strong incentive, but throwing tokens at the problem does not produce these results. Other factors matter more.
  • Spread in the research :: studies and reports show everything from -19% output to 2x and above (METR, Stanford, Faros AI and others).

Sponsor segment: Grok Bot

  • Product :: agent bot with connections (Cloudflare, Vercel, Google Sheets, PostHog, X, Convex, GitHub); engineering tasks run on Cursor cloud agents and can be controlled from phone, laptop or desktop.
  • Demo :: an engineering bot reads the PRs of the author's Jetty repo, summarises them and ranks priority; the three most urgent PRs merge cleanly and are merged.

Amazon rollout: from one team to 50

  • Bedrock team :: needed a new inference data plane, estimated at 30 people over 18 months. Six engineers with Kiro built it in two and a half months. Caveat: the six were among the best in the company (two distinguished engineers), already expert in distributed systems and LLM architecture.
  • Prime Video :: six different engineers, 10-day sprint with Q. The project estimate dropped from 90 weeks to 24 weeks. Caveat: a senior engineer spent three weeks beforehand writing small, detailed, well-scoped tasks.
  • Amazon Stores test :: 50 teams on existing systems, junior to senior mix, same AI tools (Q), throughout 2025. Measured what reached production, not commit counts.
  • Result :: about half under 3x, median about 4.5x, some teams over 10x.
  • Why Amazon :: it is the only larger established company publishing such data; small startups with three developers also ship at these rates, but are not the example people ask for.

What the slow teams did

  • Under-3x teams :: sprinkled the tool on top of the existing way of working and kept all previous habits, changing nothing but plugging Q in. Still gained up to 3x.
  • Top teams :: fundamentally changed how they worked. Amazon interviewed them and called the takeaways habits, not hacks, because the way of working has to change permanently.

Habit 1: write everything down

  • Move knowledge from Slack, heads and code-review comments into skills and steering files, so the AI has the context.
  • Every agent mistake is treated as missing context: ask what was not told, instead of cursing at the agent. The human takes responsibility.
  • As models improved through 2025, the teams removed rules. Fewer rules are needed now; too many rules and too much context can blow out the context window. Skill collections and pstack remain worth a look, but this is a personal observation, not a rule.
  • Today the agent can be connected directly to code-review comments and Slack channels, so manual copying into skill files is less necessary, as long as nothing lives only in a head.

Habit 2: get slower first

  • Good teams saw productivity drop at the start while building the process.
  • Work done in that phase: MCP servers so the agent can reach everything it needs, better error messages so the model sees what broke, restructuring the codebase so an agent can navigate it.
  • Some teams changed language, for example from Python or JavaScript to TypeScript or Rust, so type checkers and compilers push back on the agent instead of the human. The author promises a follow-up video with a caveat on this.
  • Possible future video: how to restructure a pre-AI codebase for the AI era.

Habit 3: do not babysit the agents

  • Replying to a single agent every 30 seconds makes the human the bottleneck; 4-6x is unreachable that way.
  • Frontier teams wrote only 1-2% of code by hand and ran agents looping for hours, several in parallel. Agents keep going, check their own work, and report back once everything compiles, runs and passes tests.
  • Looping for hours was rare in 2025 (Ralph Loop came in 2026), so the result is more notable.
  • Same idea as Lauren Tan's pstack: first build one agent that is trusted, then stack agents in parallel.

Habit 4: write the spec before the code

  • Equivalent to plan mode (the author rarely uses it now, but the idea holds). A vague prompt followed by arguing over the implementation is the worst path.
  • No need for a huge spec for every task, which makes things worse. Prompts must be specific, even if shorter than before.
  • Amazon engineers have long practised spec-driven development, and Kiro builds it in.
  • Cheaper too: iterating on a plain-English document costs less than reviewing and rewriting generated diffs.

Habit 5: build the test and verification pipeline

  • Tests, linters, integration, performance and security tests. The author urges open-source repositories that accept external contributions to build the same.
  • ROI is higher than ever: the pipeline lets the agent keep looping beyond compiler and type checks, fixing whatever the pipeline finds.
  • Services are mocked locally on the laptop, giving a fast feedback loop with deterministic responses and no cloud dependencies. Faster loops mean more loops, and more loops mean better output.

Conclusion

  • Gap :: all teams had the same tools. Under-3x teams did not write down context, did not build the pipeline and stayed in the loop themselves.
  • Timing :: the experiment ran before Opus 4.5 (released 2025-11-24), with models like Claude 3.7, 4.0 and 4.1, which often made 20 unwanted changes for a small refactor. Models, harnesses and connected tooling have improved enormously since.
  • 2026 plan :: Amazon is scaling from 50 teams to about 2,000.
  • Takeaway :: 10x developers exist, but are invisible to anyone relying on the model alone with minimal effort. Working alone and working in a team are different systems, and codebase structure, task structure and process must be adapted. A further layer, AI agents, is left for later.

Sources mentioned

  • Lauren Tan, tweet on shipping about 2,500 PRs in a month (X, 2026-09-21) and pstack part 1
  • Clare Liguori, From AI-Assisted to AI-Native: Building a Frontier Development Team (AI Engineer World's Fair)
  • Anthropic, Redesigning source control at Anthropic
  • METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
  • Yegor Denisov-Blanch, How to Quantify AI ROI in Software Engineering (Stanford)
  • Faros AI, AI Engineering Report 2026
  • AWS, How frontier teams are reinventing AI-native development; Kiro (https://kiro.dev)
  • Linear, How teams build, Edition 01