Test Your Auth Flow Without Production (Sponsored)

Authentication is often the least-tested part of an app. Live environments need network access and real credentials, while mocks miss the failures that break production.

@workos/emulate runs the WorkOS API locally for development and automated testing. Seed users, organizations, RBAC roles, and SSO connections, then test full AuthKit login flows, signed webhooks, token refresh, and error handling without touching production. Responses and event shapes come from the WorkOS OpenAPI spec, so your tests exercise the same surface your app uses in production. Run it locally or in CI with npm, Homebrew, Docker, or a standalone binary.

Run WorkOS locally

A customer making a card payment expects it to get approved or declined almost instantly. This single requirement shapes everything about how a payment is built, because the entire processing of the payment has to finish within this short window.

We recently spoke with Ben Cane, a Distinguished Engineer at American Express, to understand how their platform handles it.

To understand how things work, let us picture what happens when a transaction arrives at American Express. It is routed into one of several independent processing units and moves through a chain of microservices. However, partway through, one of those services begins to fail, and the customer at the checkout terminal is still waiting.

Just retrying the transaction inside the failing unit carries obvious risk. Also, moving the transaction to another server is an even harder problem. This is because the second unit would need to know how far the first transaction went. Sharing such knowledge creates a dependency that can turn two independent units into a fragile system.

The American Express engineering team handles this with a cell-based architecture that helps keep the impact of a disruption to a minimum. In our chat with the engineering team of American Express, we learned that most of the engineering effort goes into keeping that isolation intact under pressure. In this article, we will try to understand how the transaction runs through such a cell-based architecture and how the payments are processed even when some services are failing.

Core Payments Ecosystem

In a typical payment flow, American Express sits in the middle of a chain. A merchant holds a relationship with an acquiring bank that sends the transaction to American Express, and American Express delivers it to the card issuer that holds the account balance. This middle hop is the core payment network’s job.

See the diagram below that shows this setup:

In 2018, American Express began modernizing this platform onto cloud-native infrastructure. Immediately, it was clear that one assumption had to change. While the older systems ran on hardware engineered to stay running, the cloud infrastructure behaved differently. Servers could disappear for reasons outside any one team’s control. You have to assume failures completely outside your control will happen more often, and build the application around that.

Two familiar patterns were considered:

• Event-driven processing fit poorly. The workload needs to run at scale with low latency and a real-time response, leaving little room for delay. Asynchronous work exists, but the answer itself still has to arrive fast.

• A monolith is where most teams start, but it fits the scaling requirements poorly.

The design American Express engineering team ultimately chose was driven, in part, by decades of internal thinking. Ben pointed out that back in the SOA era, long before microservices existed as a concept, his teams were already trying to reduce the impact of a single failure on a specific system. The patterns were present even before the more formal term of cell-based architecture was coined.

Cell Boundaries

In a cell-based architecture, a cell is a complete, self-sufficient copy of the payment processing stack. In other words, everything required to process a transaction lives inside one boundary. This includes microservices, databases, DNS, and supporting infrastructure.

There are five key properties of a cell:

• It deploys independently and processes payments on its own.

• It owns its microservices, databases, and other components.

• It forms a single failure domain, so problems inside stay inside.

• It can be pulled out of rotation for maintenance or during an incident, with the rest of the platform carrying on.

• It does not have synchronous cross-cell dependencies in the critical path.

Reference data still replicates across cells, observability data can aggregate across cells, and router instances communicate with each other across cells. What stays out of the critical path is any blocking call between cells during transaction processing.