/
.webp)
Generated by Master Biographer | Source for LinkedIn Content
There is one rule at Amazon that overrides everything else.
Not "move fast." Not "be customer obsessed." Not any of the fourteen leadership principles.
The rule is older and more visceral than any of those: the shopping cart must never fail.
Not "show a friendly error message." Not "degrade gracefully." Never fail. Customers must be able to add items to their cart whether a disk is failing, a network is flapping, a data center is underwater, or a server has caught fire. The shopping cart is the moment a customer goes from browsing to buying. That moment cannot be interrupted. Not once. Not by anything.
This sounds like a product requirement. It is actually a philosophical declaration that rewrites thirty years of database orthodoxy.
Because the traditional answer to "what do you do when nodes disagree?" is: stop. Wait for consensus. Protect data integrity above everything. Reject the write if you cannot guarantee it.
Amazon's answer is different. Amazon's answer is: write anyway. Accept the inconsistency. Merge the conflict later. The customer keeps shopping.
That single requirement — deceptively simple, technically insane — is why DynamoDB exists.
And it built something far stranger than the engineers who built it expected.
2004. A company at war with its own database.
Amazon in 2004 is not the empire you know today. It's a fast-growing e-commerce company that has bet its entire infrastructure on Oracle, the enterprise database giant. Oracle is the safe choice. The respectable choice. Every serious company runs Oracle.
But "serious" is a problem when your traffic doubles every year.
Oracle's enterprise edition cannot scale horizontally. It was built for vertical scale — bigger machines, more RAM, faster disks. But Amazon's workload doesn't care about bigger machines. It cares about more nodes, more regions, more availability zones, more redundancy. Oracle was designed for a world where uptime was a matter of engineering. Amazon needed a world where uptime was a law of physics.
Werner Vogels, Amazon's CTO since 2005, later describes what the engineers find when they actually look at their own database usage data. About 70 percent of Amazon's database operations are pure key-value lookups — find a customer, get their cart, update an order. Primary-key queries, single rows returned, no joins. Another 20 percent are range queries returning a set of rows. That leaves 10 percent — just 10 percent — of operations that actually use the JOIN functionality that sits at the heart of Oracle's architecture. They are running a Ferrari to fetch the mail.
The conclusion Vogels draws is uncomfortable: "The days of the one-size-fits-all monolithic database are behind us."
The engineers start building something new. They call it Dynamo.
The Failure That Couldn't Be Tolerated
Inside Amazon, the engineers keep running into the same problem: when nodes fail — and in a distributed system, nodes always fail — the shopping cart needs to make a decision. Does it reject the write and tell the customer they can't add an item? Or does it write anyway, accept a possible inconsistency, and sort it out later?
Traditional database thinking says: reject the write. Protect data integrity above all else. Show the customer an error. They'll understand.
Amazon's business says: never reject the write. Never. The customer must be able to add the item. Always.
Vogels frames the operational reality in terms that leave no room for ambiguity: "Even the slightest outage has significant financial consequences and impacts customer trust." This is not a CTO platitude. It is a design constraint that every engineering decision flows through. The system that fails to write a shopping cart item has cost Amazon real money and real goodwill. The system that writes an inconsistent item and resolves it later has cost nothing.
That asymmetry — where a failed write is worse than an inconsistent write — is the founding insight of Dynamo.
The CAP Theorem, Made Concrete
In 2000, computer scientist Eric Brewer had articulated the CAP theorem: a distributed system can provide at most two of three guarantees — Consistency, Availability, and Partition Tolerance. In a world where network partitions are inevitable, you must choose between consistency and availability.
Academics had discussed this as theory. Amazon turned it into operational policy.
Dynamo chooses availability. Explicitly. Deliberately. The system is designed to continue accepting writes even when nodes cannot communicate, even when replicas disagree, even when the network is fragmented. The engineers document this choice with unusual honesty in the eventual paper: "Dynamo sacrifices consistency under certain failure scenarios in order to maintain availability."
This is not a compromise. It is a conviction.
The mechanism is eventual consistency — a term that Werner Vogels would later define precisely: "a specific form of weak consistency; the storage system guarantees that if no new updates are made to the object, eventually all accesses will return the last updated value." Eventually. Not immediately. Eventually.
The system handles the shopping cart scenario like this: if two devices add items to the same cart simultaneously, Dynamo preserves both writes as separate versions, using a data structure called a vector clock to track causality. When the customer reads the cart back, the application merges both versions. The customer gets all the items. Nothing is lost. The "conflict" is resolved at read time, by the application, not the database. And an "add to cart" operation "can never be forgotten or rejected."
For a shopping cart, this is not just acceptable — it is ideal. For a bank account, it would be catastrophic. The engineers know this. The system is designed for the former.
Nine Names on a Paper
Starting around 2004, the Dynamo team builds the system. By 2007, it is in production, handling shopping carts and session management across Amazon's infrastructure. Production metrics are striking: 99.9995 percent successful response rate. Zero documented data loss events. Successful operation across server failures, datacenter outages, and network partitions.
In October 2007, nine Amazon engineers publish their work at the ACM Symposium on Operating Systems Principles (SOSP) — one of the most selective venues in computer science. The paper is titled: "Dynamo: Amazon's Highly Available Key-value Store."
The nine authors are: Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Vamsi Kambhampati, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels.
Pay attention to the fifth name. It will matter enormously.
The paper is written with an unusual combination of humility and precision. It documents not just the architecture but the reasoning — why each design decision was made, what tradeoffs were accepted, what the team learned in production. The configurable replication parameters (N for replication factor, R for minimum reads, W for minimum writes) are explained in terms of the actual business decisions they encode. In production, Amazon typically runs (3, 2, 2) — three replicas, two required for reads, two required for writes. Change those numbers and you change the tradeoff between latency, durability, and availability.
Why does Amazon publish? The paper's own language is revealing: Dynamo "is not directly exposed externally as a web service," but "Amazon's technologies are used to power parts of our Amazon Web Services." Publishing is partly scientific contribution, partly competitive signal, partly recruiting lever. Amazon is showing the industry it has solved problems that no one else had even framed correctly — and inviting the best distributed systems engineers in the world to come work on them.
The Dynamo paper becomes one of the most cited documents in distributed systems history. And the system it describes remains unavailable to anyone outside Amazon.
The Deliberate Gap
Between the 2007 paper and the 2012 product launch, there are five years.
Those five years are not confusion or delay. They are a deliberate experiment.
In November 2007 — the same month the Dynamo paper is published — Amazon launches SimpleDB, the first managed NoSQL database service on AWS. SimpleDB is a test: will developers want a managed key-value store as a cloud service? Will they pay for it?
The answer is yes, but with caveats. SimpleDB has hard limits: 10 GB per domain, unpredictable latency because it indexes everything, confusing "Machine Hours" pricing. Developers adopt it but complain constantly.
What happens next reveals something fundamental about developer psychology. Despite Dynamo's technical superiority over SimpleDB — better performance, better durability guarantees, more sophisticated conflict resolution — developers inside Amazon who have the option of using either system choose SimpleDB and S3. They choose the managed, simpler system over the more powerful one.
Vogels later describes this with cold clarity: "Nobody wanted to learn how to do that if they didn't have to. Ultimately, developers wanted a service."
They "voted with their feet."
That observation drives the design of DynamoDB. The question is not "how do we open-source Dynamo?" It is "how do we turn Dynamo's architecture into a service that developers will actually use?" The answer takes five years because the engineering challenge is not the database. It is the operational layer — the automated partitioning, the replication management, the capacity provisioning, the zero-downtime scaling. You are not building a database. You are building a database that runs itself.
The Launch
DynamoDB launches on January 18, 2012. Not at a conference. Not with a keynote. Just a blog post from Werner Vogels, a product page, and a pricing sheet.
The announcement is precise: DynamoDB is "the best parts of the original Dynamo design (incremental scalability, predictable high performance) combined with the best parts of SimpleDB (ease of administration, consistency, and a table-based data model)." It is, explicitly, a synthesis. Not Dynamo. Not SimpleDB. Something new that learned from both.
But the synthesis is not just a product decision. It is also an architectural one — and this is where the DynamoDB story gets genuinely strange.
The DynamoDB that launches in 2012 is not the Dynamo system from the 2007 paper. It shares the name and some of the philosophy. But under the hood, it has rebuilt the core.
The original Dynamo used a peer-to-peer gossip protocol for membership and replication — nodes discover each other, share state, converge. This is elegant for an internal system with trusted engineers at the wheel. For a managed cloud service with customers writing arbitrary access patterns, it is a liability.
DynamoDB instead uses Multi-Paxos for replication — a leader-based consensus protocol where a single leader replica handles all writes and strongly-consistent reads. The shift is significant: instead of eventual consistency everywhere, DynamoDB offers a choice. Eventually-consistent reads are cheaper and faster. Strongly-consistent reads — available at double the cost — route to the leader and return the latest committed value. The same database, two consistency tiers, developer's choice.
This is a product insight as much as a technical one. Developers from relational backgrounds do not trust eventual consistency. They understand what they are giving up. Offering a strongly-consistent path, even at a cost, makes DynamoDB acceptable to a far wider audience.
The Provisioned Throughput Innovation
The pricing model at launch is unlike anything that has existed before it. Instead of charging for compute hours or node-hours or disk, DynamoDB introduces Provisioned Throughput: you specify exactly how many read and write operations per second you need, and you pay for that capacity whether you use it or not.
A Read Capacity Unit (RCU) represents one strongly-consistent read of up to 4KB per second. A Write Capacity Unit (WCU) represents one write of up to 1KB per second. At launch: $0.01 per hour for 10 WCUs, $0.01 per hour for 50 RCUs. Storage at $1 per GB per month.
This makes performance predictable in a way that no previous database service could offer. There are no noisy neighbors stealing your IOPS. Your table performs exactly as specified. The single-digit millisecond latency promise is not marketing — it is an engineering commitment backed by SSDs (unusual in 2012) and synchronous replication across multiple Availability Zones. The data is durable before the confirmation arrives.
The NoSQL Validation
DynamoDB's launch in 2012 matters beyond Amazon. The NoSQL movement at this point is real but fragmented — MongoDB, Cassandra, CouchDB, Redis, Riak, each startup and team betting on a different system. There is no enterprise consensus.
Amazon's entry changes the political landscape. The largest internet retailer in the world, a company whose database requirements are not hypothetical, is publicly saying: for the majority of our use cases, NoSQL is the right architecture. That is an endorsement no academic paper or startup could provide. Within a year, DynamoDB prices drop 85 percent. That is the classic AWS playbook: build the infrastructure, drive down costs as scale grows, make it impossible for anyone to compete on price.
Prime Day and the Religion of Predictability
DynamoDB becomes the backbone of Amazon's own operations at a scale that is almost incomprehensible.
During Amazon Prime Day 2017, DynamoDB handles 12.9 million requests per second. Not peak. Sustained.
By Prime Day 2021, that number has grown to 89.2 million requests per second across a 66-hour window. Alexa, Amazon.com, Amazon fulfillment centers, all running on DynamoDB simultaneously. Trillions of API calls. Single-digit millisecond latency throughout.
But between 2012 and 2021, the engineering team learns several hard lessons that no blog post would ever fully describe. The 2022 USENIX paper is more honest than any press release.
The Hot Partition Problem
The original Dynamo paper assumed, with some justification, that keys would be distributed roughly uniformly. Consistent hashing guarantees an even distribution of key space across partitions, so if your key distribution is random, your load is evenly distributed.
Real-world workloads are not random.
Real-world workloads have hot keys — viral items, trending products, celebrity accounts, whatever is generating disproportionate traffic at any given moment. When DynamoDB detects that a partition is too large (exceeding 10 GB) or too busy, it splits it — divides the partition and its provisioned throughput in half.
The problem emerges after the split. If the traffic hitting the partition is non-uniform — concentrated on one portion of the key range — then splitting the throughput equally means the hot portion now has less throughput than it had before. The system has made the hot partition worse. Throttling occurs even when the table's total provisioned throughput is technically sufficient.
This is a fundamental design gap between the theory (uniform distribution) and the reality (concentrated traffic). For early DynamoDB customers with uneven workloads — game leaderboards, social media activity feeds, anything with viral dynamics — this created unexplained throttling that was genuinely hard to diagnose. The key was "hot," but the table capacity looked fine in aggregate.
Amazon's solution, developed over years, is Adaptive Capacity — the system automatically shifts throughput allocation to match actual demand within a partition, eliminating the rigid per-partition throughput limits that created the problem. This feature, launched in 2019 and made automatic for all tables, is the product of years of production pain. It is the kind of lesson you can only learn at scale.
The Metadata Cache Catastrophe
The 2022 paper also documents an existential risk that most customers never knew existed.
DynamoDB uses a metadata service to route client requests to the correct storage nodes. To avoid hammering the metadata service on every request, the system caches routing information. This cache achieves a 99.75 percent hit rate — by any engineering standard, excellent.
But that 99.75 percent number conceals a catastrophic failure mode.
During a cold start — when the cache is empty, either from a deployment or after a large-scale failure — every single request misses the cache and hits the metadata service directly. With millions of requests per second as steady-state load, a cold cache means the metadata service faces a 400x traffic amplification. Four hundred times its normal load, arriving simultaneously, immediately.
That is not a degraded state. That is a kill shot. If the metadata service goes down under this load, DynamoDB cannot route any requests. The entire service fails.
The solution is MemDS, a distributed in-memory caching system that maintains constant background load on the metadata service — even when requests are being served from cache. This trades some steady-state overhead for the elimination of the metastability trap. The cache can still empty; but when it does, the metadata service has been continuously warmed and can absorb the load. Cold start becomes a managed transition rather than a catastrophic event.
The team publishes this in 2022. It is, quietly, a confession that between 2012 and some point in the late 2010s, DynamoDB had a failure mode that could theoretically bring down the entire service at worst possible moment — a major outage that triggers a cascading restart.
The Pricing Wars
DynamoDB's Provisioned Capacity model becomes famous in two ways: it makes certain workloads radically cheap, and it makes certain other workloads brutally expensive in ways developers did not expect.
The problem is the unit model. An RCU reads 4 KB strongly consistent. A WCU writes 1 KB. If your items are larger, you consume more units. If your traffic is spiky, you either over-provision (wasting money) or under-provision (getting throttled during spikes). Predicting your workload accurately enough to provision correctly requires a level of operational rigor that most startup teams do not have.
Amazon's answer, launched in November 2018, is DynamoDB On-Demand — a pay-per-request model that requires no capacity planning. For unpredictable workloads, this is salvation. For high-volume steady-state workloads, it can be five times more expensive than provisioned capacity.
The system now has two pricing modes, both with edge cases, both requiring careful workload modeling to choose correctly. The horror stories accumulate on Hacker News: the startup whose DynamoDB bill triples when a feature goes viral and hits WCU limits; the company that miscalculates the free tier and faces hundreds of dollars in unexpected charges in month one.
ACID transactions, added in 2018, add another complexity layer. Transactional writes consume two WCUs per item (the write, plus a copy for conflict detection). Transactional reads consume two RCUs. The safety is real. The cost is real too.
The Cassandra Connection: One Author, Two Databases
Here is the story the industry rarely tells cleanly.
Avinash Lakshman co-authors the Dynamo paper in 2007. He is one of the nine engineers whose work defines the NoSQL movement.
Shortly after, Lakshman leaves Amazon and joins Facebook. At Facebook, he faces a structurally similar problem: how do you store the inbox search index for hundreds of millions of users, with no single point of failure, across commodity hardware?
The answer he builds with Facebook engineer Prashant Malik borrows directly from the Dynamo architecture — consistent hashing, gossip protocol, eventual consistency — and adds ideas from Google's Bigtable paper: column families, wide rows, a structured data model. They build it, according to lore, initially at a Facebook hackathon.
They call it Cassandra, after the Greek prophetess cursed to speak truth but never be believed.
Facebook open-sources Cassandra in July 2008 — less than a year after the Dynamo paper. Apache Cassandra becomes a project in 2009. It goes on to power Netflix, Apple, Instagram, Uber, Discord, and thousands of other companies. Lakshman then co-founds Riptano, later renamed DataStax, to commercialize Cassandra — turning his Amazon-era insight into a billion-dollar commercial database company.
Separately, Basho Technologies builds Riak — another Dynamo-inspired distributed key-value store, explicitly citing the Dynamo paper as its architectural foundation. Riak even preserves Dynamo's configurable N, R, W parameters in its API.
The DNA runs in a straight line: the shopping cart problem at Amazon → the Dynamo paper → Avinash Lakshman at Facebook → Cassandra → DataStax. And simultaneously: the Dynamo paper → Basho → Riak.
Amazon published a research paper describing an internal tool. That paper seeded an entire industry of competitors. It is one of the more unusual acts of accidental generosity in tech history — a company describing, in precise technical detail, exactly how to build a system that would compete with them.
The Single-Table Design: Counterintuition as a Feature
Perhaps the strangest thing about DynamoDB is what it demands of developers: you must unlearn everything you know about database design.
In a relational database, you normalize. You create separate tables for customers, orders, products. You JOIN them at query time. The schema reflects the domain model. The query planner figures out how to assemble the data.
DynamoDB cannot JOIN. Not "doesn't recommend it" — structurally cannot. The architecture prohibits it because JOINs do not scale linearly. They scale multiplicatively. At DynamoDB's target scale, they would be catastrophic.
The answer DynamoDB forces is single-table design: store all your application entities in one table, organized by access pattern. A customer and their orders live in the same table, under the same partition key, retrievable in a single request. You design the data structure to match the query, not the domain model. The schema is the query plan, baked in at write time.
This is deeply counterintuitive. Relational database training tells you that mixing entity types in one table is a sin. DynamoDB says it's the architecture. The learning curve is steep enough that it has spawned an entire sub-industry of DynamoDB consultants, courses, and books. Alex DeBrie's "The DynamoDB Book" exists because the gap between "how databases work" and "how DynamoDB works" is large enough to build a business in.
That gap is, in a way, the whole point. DynamoDB is not a general-purpose database. It is a specific philosophy about what a database should be — one that sacrificed generality for the relentless, aggressive demand of a shopping cart that must never fail.
1. The DynamoDB of 2012 shares a name with the 2007 Dynamo paper but rebuilt the core architecture from scratch.
This is the detail that confuses engineers who read the paper expecting to understand the product. The 2007 Dynamo used peer-to-peer gossip protocols and lacked a leader — every node was equal, replication was eventual, conflicts were resolved by the application. The 2012 DynamoDB uses Multi-Paxos with a leader replica that handles all writes. The 2022 USENIX paper states this directly: DynamoDB "shared most of the name of the previous Dynamo system but little of its architecture." The paper is the founding philosophy. The product is a different engineering implementation of that philosophy.
2. The Dynamo paper was published the same month Amazon launched SimpleDB — and the timing was deliberate.
November 2007: the Dynamo SOSP paper drops. Also November 2007: Amazon launches SimpleDB, its first public managed NoSQL service. This is a signal strategy — Amazon reveals what it built internally while opening a commercial alternative that is less capable but fully managed. Dynamo itself would remain internal for five more years. The academic paper and the commercial product launch on the same day. The paper educates the world; the product captures the market.
3. DynamoDB had a theoretical failure mode where a cold cache could generate 400x load amplification on its core metadata service.
The 99.75 percent cache hit rate for DynamoDB's routing metadata looked like an engineering achievement. It was hiding a catastrophic bimodal behavior: when the cache emptied (deployment, failure, restart), every single request became a metadata lookup simultaneously. At production request rates, this creates a 400x traffic spike to the metadata backend — a load that could cascade into a full service outage. The solution, MemDS, was built specifically to maintain constant background traffic to the metadata service so cold-cache events never arrive on a cold backend. The 2022 paper documents this. No outage post-mortem ever described it publicly.
4. The same engineer who co-authored the Dynamo paper then built Cassandra at Facebook — and later commercialized it as DataStax.
Avinash Lakshman is fifth on the 2007 Dynamo paper. He left Amazon, joined Facebook, and co-created Cassandra in 2008 — explicitly inheriting Dynamo's consistent hashing and gossip protocol. He then co-founded Riptano (later DataStax) to commercialize Cassandra. One engineer: Amazon's Dynamo, Facebook's Cassandra, and DataStax's commercial database empire. The two most cited distributed systems papers in NoSQL history share a common author. The NoSQL revolution traces, in a meaningful line, to one person's repeated application of the same insight.
5. DynamoDB added ACID transactions in 2018 — six years after launching as an explicitly non-transactional database, and a full decade after the Dynamo paper's anti-transaction philosophy.
The 2007 Dynamo paper was built on the conviction that strong consistency was a liability — that systems willing to accept inconsistency would always outperform and outscale systems that demanded it. DynamoDB launched in 2012 without transactions. Six years later, Amazon added TransactWriteItems and TransactGetItems — atomic, all-or-nothing operations across multiple items. The system designed to avoid consistency guarantees now supports the consistency guarantee at the core of every relational database. The cost is real (two RCUs per read, two WCUs per write in a transaction). But the capability exists. Enterprise customers demanded it, and Amazon delivered it. The eventually-consistent system learned to be eventually consistent with ACID.
Werner Vogels, in the original Dynamo paper, 2007:
"Reliability and scalability of a system is dependent on how its application state is managed."
That sentence sounds like a design principle. It is actually a political argument. It is saying: the database team should not get to decide what "correct" means. The application team should. The business should.
The shopping cart must never fail.
Everything else is a consequence.
Sources: Amazon Dynamo SOSP 2007 paper (DeCandia et al.); allthingsdistributed.com — Werner Vogels, "Amazon's Dynamo" (Oct 2007), "Eventually Consistent" (Dec 2008), "Amazon DynamoDB" (Jan 2012); "Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service" — USENIX ATC 2022; brooker.co.za, "DynamoDB's architecture" (Jul 2022); Facebook Engineering, Cassandra open-source announcement (Jul 2008); AWS DynamoDB product documentation and pricing history; Werner Vogels, "A Decade of Dynamo" (Oct 2017).
