Canva, the design platform, has shared details of a new architecture designed to handle session revocation for its hundreds of millions of active sessions. The key innovation is using Amazon S3 as a scalable and durable datastore for revocation data, avoiding the cost and complexity of a networked database lookup for most authentication requests.
Canva's system stores encrypted session information in browser cookies. This allows application gateways to authenticate requests without contacting a central datastore, which is a common technique for scaling to large user bases. However, managing revoked sessions and changes to permissions in near real-time presented a significant challenge.
Previously, Canva kept 12 hours of revocation data in memory, while session refreshes continued to check MySQL. As the platform grew, hundreds of gateway instances could each request more than a million revocations from MySQL during deployments, creating a coordinated and disruptive database load.
The S3-Based Solution
To solve this, Canva chose Amazon S3 over a traditional datastore like Redis. This avoided the operational complexity of maintaining another datastore, while leveraging S3's durability and availability.
The 12-hour revocation window is divided into 30-minute S3 objects. Gateways download these objects as needed. Each revocation is represented as a 16-byte binary record containing a principal and timestamp. The sorted arrays enable direct in-memory searches, reducing the cache footprint by 87.5% . Gateways use conditional GETs to download changed chunks and discard data older than 12 hours.
Asynchronous workers are responsible for scanning for new revocations, merging them into the latest chunk, and uploading the result. Conditional PUTs provide optimistic concurrency control when workers update the same object. ZooKeeper leader election reduces conflicts but is not required for correctness.
A Coordination Primitive
Sharing the article, a software engineer highlighted the use of Amazon S3 as more than an object store. By using immutable objects and conditional writes, Canva effectively used S3 as a coordination primitive, a role traditionally filled by a dedicated distributed coordination service.
The design also addresses recovery and deployment. A gateway can reconstruct its local revocation state by downloading the relevant S3 chunks, rather than requiring a database to rebuild the cache. Canva reports that the worker can process more than 2,000 revocations per second, exceeding its expected requirements, while even a chunk containing one million revocations represents about 16 megabytes of binary data.
Tradeoffs and Benefits
The approach sparked discussion online, with one developer questioning why Canva didn't just use a refresh token scheme with short-lived access tokens. The response argued that keeping revocation data in memory provided better tradeoffs, as frequent token refreshes would increase database load and make availability dependent on the database during refresh operations.
Following the migration, Canva reduced its session revocation database to two read replicas for redundancy and improved deployment speed. Database load became more predictable, scaling with revocation write throughput and overall site traffic rather than the number of gateway instances loading the cache. Canva said testing multiple implementations on real infrastructure helped validate that the selected design could meet its scalability requirements.