Hook
Three timestamps. 17:16. 17:25. 17:36 — UTC+8, same calendar day, no date carried in the feed. Nine minutes between the first and second status-page update. Eleven between the second and the third. Inside that window, GitHub's API, Issues, Pages, Pull Requests and Actions all degraded at once. Twenty minutes, if you accept the frame the status page sold you.
I was mid-push. A Groth16 constraint refactor, forty-one files, a rebase I had spent the morning untangling. git push stalled on the TLS handshake and then returned remote: Internal Server Error — which is not a Git error. Git does not have HTTP status codes. It is an HTTP status wearing a Git costume, and my terminal has a reflex for that string: it means the fault is four layers up, inside somebody else's authorization service, and nothing I type will change the outcome.
What I did next, three million other developers did next. I opened the status page. I read the words "degraded performance." I did arithmetic on nine-minute update intervals. And then I noticed that the five affected services map, almost one-to-one, onto the critical path of every smart contract deployment pipeline currently running in production. Not a SaaS incident. A supply-chain event that happened to be short.
Context: What GitHub Actually Is to This Industry
Start with the inventory, because the inventory is the argument.
Bitcoin Core develops in the open at one repository. Proposals become BIPs, BIPs become pull requests, pull requests become ACK threads, ACK threads become merged consensus changes. Package relay, cluster mempool, assume-utxo refinements — every one of those arguments was settled inside an issue tracker that was, for twenty minutes, returning errors.
Ethereum's execution layer is nominally diverse: geth, nethermind, besu, erigon, reth. Client diversity is a headline talking point in every post-mortem about every consensus incident. The diversity lives in the binaries. The coordination — release tags, security advisories, coordinated disclosure windows — lives in one forge.
Then there is the dependency layer, which is where I actually live. Foundry pulls dependencies as git submodules. Hardhat resolves through npm, and a meaningful share of those tarballs are published from GitHub-hosted sources. OpenZeppelin Contracts, the Solidity compiler, the entire audit tooling stack — Slither, Echidna, Medusa — resolves through the same domain. An audit deliverable is a commit hash plus a reproducible test suite. The hash only means something if the remote resolves.
And then GitHub Actions, which is the thing I want to spend most of this article on. Actions is the de facto CI plane for smart contract development. It runs the Foundry suite. It runs the fork tests. It holds the deployer private key as an encrypted secret and, on merge to main, it signs and broadcasts deployment transactions. That last sentence is the one that should have appeared in an incident report and did not.
One calibration note before the analysis. The underlying item here is four factual data points transcribed from a single official status page. No independent reporting. No root-cause analysis. Twelve hours after an event like this, that is all anyone has, and I am not going to pretend otherwise. Confidence calibration is not hedging; it is the difference between an analysis and a horoscope. Where I reason past the published facts, I will say so.
The published failure chain, though, is a gift: "collaboration system database replication delay, leading to authorization interface errors, leading to a site-wide increase in error rates." Three-stage causality in one sentence of status prose. Status pages normally say "elevated error rates, investigating." This one names a datastore, a replication lag, and a downstream dependency direction. That is a fingerprint, and it is worth reading carefully.
The Chain, Mechanically
Replication lag is not a mysterious condition. It is the ordinary consequence of a topology that writes to one node and reads from others.
Take Postgres streaming replication as the canonical case, because it is the most common shape at this scale. A primary accepts writes and ships WAL to hot standbys. Standbys replay. Between the commit on the primary and the replay on a standby there is a window — the replay lag — and that window is not constant. It widens under write bursts, under long-running read queries that block replay, under network jitter, under a replica that was just restarted and is catching up, and under replication-slot pressure when a consumer falls behind. Synchronous commit with remote_apply bounds the window at the cost of write latency. Asynchronous replication buys throughput and sells you the window. Most large platforms run a mixed configuration, and most of them are honest about it in a design doc and silent about it in a status channel.
Now place authorization on the far side of that window. Any token model that requires a lookup against a replicated store inherits the freshness of whichever replica answers the query. Short-lived tokens validated by signature alone do not. Opaque tokens validated by introspection, or by a database lookup, or by a cache that was itself populated from a replica read, do.
A total outage is honest. A partial outage is a liar. This part deserves more attention than it received. When a service is fully down, every client learns the same fact immediately, and every client backs off. When authorization returns intermittent 401s against credentials that are in fact valid, each client believes something different. Some retry with the same token. Some refresh. Some surface a "session expired" screen to a human who then re-authenticates, which is a write, which lands on the primary, which widens the lag, which produces more 401s. That is a positive feedback loop with a human inside it. It is structurally the same loop as a bank run driven by a stale oracle price: the system has a truth, the readers cannot reach it, and read errors compound into writes that make the truth harder to obtain.
The second mechanical point is amplification. Authorization is not a leaf service. It sits in the call path of nearly everything — the API, the web front end, webhook delivery, and the Actions scheduler that must verify a workflow's permissions before it hands a job to a runner. A leaf failing subtracts one feature. A hub failing multiplies against every dependency that synchronously asks it a question. Five services reported as degraded. Underneath, the call graph is a star, and the hub was the center of it.
Blast Radius Nobody Reported
The status page listed five services. That list is a floor, not a ceiling, and the gap between the two is where the actual damage sits.
Consider what depends on those five and never appears on any status page. Registry pulls, where container images live in a GitHub-hosted registry. Dependency resolution for ecosystems whose proxies re-fetch from GitHub. Foundry's git-submodule installs mid-build, which fail with a git-level error and then get retried by a shell script with no jitter. Webhook deliveries to every alerting channel, which means the incident also degraded some teams' ability to be alerted about the incident. Code scanning, secret scanning, advisory databases.
None of that is a claim about what happened inside this specific twenty-minute window. It is a claim about the shape of the dependency graph, and the shape does not change between incidents. I have run builds that failed because a transitive dependency's tarball host degraded, and the failure surfaced as an unrelated type error eleven minutes into a compile. Cascading failures rarely look like the thing that caused them.
What follows is the piece I want to formalize, because I have not seen it written down anywhere, and it carries the highest consequence.
CI as an Unaudited Transaction Submitter
Here is the standard deploy job. A workflow triggers on merge to main. The runner checks out the repo, runs the test suite, then executes a deploy script. That script instantiates a signer from an encrypted secret, constructs a transaction, signs it, broadcasts it to an RPC endpoint, and polls for the receipt in a loop with a timeout.
That loop is not idempotent. Nothing about it is idempotent.
When the authorization layer starts throwing intermittent errors, the workflow's own API calls begin failing — fetching artifacts, reading the deploy manifest, reporting status. Depending on how the job was written, the step retries. Depending on how the runner was configured, the job may be re-queued onto a different runner instance. That second runner reads the same secret, constructs the same transaction, signs with the same key, and broadcasts.
Now there are two signed transactions from one account. Ethereum's nonce scheme resolves this in exactly one of two ways. If both use the same nonce, one executes and the other is rejected as a duplicate — or replaces it, if the fee bump clears the replacement threshold, which is a race with a semi-deterministic outcome and an indeterminate fee. If the retry logic derived a fresh nonce from pending or latest at a moment when the first transaction had not propagated to the node it queried, you get two distinct nonces and two executions.
Ethereum's nonce is a weak idempotency key: it guarantees single inclusion per nonce, not single execution of your intended business logic. If the contract is a counter, you incremented it twice. If the function is a two-step ownership transfer, you may have moved ownership to an intermediate address and then re-executed into a broken state. If the function appends to an array, you now hold duplicate entries and a storage layout that no longer matches the ABI your indexer assumes.
I learned where this class of bug lives during the 2020 DeFi summer, auditing liquidity-mining contracts nobody else was reading. The bug I found was a reentrancy in a reward distributor: a function that read a shared state variable, made an external call, and wrote back based on the stale read. The retry storm above is the same shape one layer up. The authorization service is the external call. The re-entrant invocation is the re-queued job. The shared state variable is the chain account's nonce, held by an actor that cannot see the other actor.
Cost: during the 2021 mint congestion I did the arithmetic on ERC-721 versus ERC-721A and found the batched path saving users roughly $45 per transaction at peak. Invert it, and a duplicated mainnet deploy cost real money. Today, with quiet blockspace, a duplicated transaction costs a couple of dollars. The dollar figure is not the risk. The risk is a multisig where two competing proposals sit at the same nonce, one executes, the other's collected signatures become dead weight, and the signers who had not yet approved must now reconstruct which hash is live. That is a governance failure produced by a CI retry, and it will never appear in an incident report, because it will look like a human mistake.
Shared Storage Layout, Five Facades
Strip the naming away and the published chain describes a storage-layer coupling. Five services that present as five products all read from one replicated datastore, and one shared authorization service mediates access for all of them. In Solidity terms: five contracts sharing a storage layout, with one modifier enforcing access control for all five.
That is a legitimate architecture. It is also a monolith with extra network hops and a more expensive failure mode. The microservice boundary buys independent deployability. It does not buy an independent blast radius if the boundary is a facade over one datastore and one auth service. Service boundaries are only as real as the data boundaries underneath them. I have watched the same mistake in protocol design: teams split a monolith into three contracts, keep one shared oracle read and one shared owner, then act surprised when a single price-feed incident takes out all three legs simultaneously. Composability is leverage. Leverage cuts in both directions.
The status-page behavior deserves a fair hearing before I criticize it. Nine-to-eleven-minute update intervals sit inside the usual enterprise band. The first update went out fast. What it lacked was an actionable description — "degraded performance" is a placeholder, and the diagnostic content arrived in the second or third revision. That sequencing matters more than the cadence. A status page's job is not to prove the on-call engineer is awake. It is to give downstream operators enough information to decide whether to wait or to execute their fallback, and "degraded" does not answer that question.
The historical pattern is worth flagging with an explicit confidence penalty. If you pull the public postmortems from large developer platforms over the last five years, the recurring nouns are orchestrator, configuration change, and shared datastore. Those three account for the largest share of multi-service cascades in the public record. My read on this event points at the same class. Confidence: medium. I am reasoning from the shape of the disclosed chain, not from a root cause I have not seen.
The Staleness Invariant
I spent six months in 2022 reverse-engineering oracle manipulation paths in algorithmic stablecoins, after watching a designed system eat itself. The finding that survived all my own second-guessing: decentralizing the reporter set does not decentralize the freshness of the feed. You can have thirty independent reporters and still liquidate against a median computed over a ninety-second window. Latency is the invariant. Node count is marketing.
Same shape here. GitHub's redundancy is real — multiple regions, multiple replicas, a runner fleet large enough to need its own scheduler. None of it bounds the staleness of an authorization read. What failed was not a shortage of copies. What failed was a freshness window propagating into a binary decision: valid or invalid, proceed or reject. Once a stale read crosses into a branch condition, it stops being a data-quality problem and becomes a control-flow problem.
The generalization is worth carrying into every oracle-dependent design. Your redundancy budget should be spent on freshness guarantees, not on additional copies of the same stale number. A perps funding rate, a liquidation threshold, an LTV calculation — all sit downstream of an assumption about how old the price might be, and that assumption is almost never written into the code with a check attached. Chainlink's decentralization story is about node count. Node count has never stopped a depeg from being liquidated four blocks late.
Availability Arithmetic and the Measurement Boundary
Run the numbers, because the numbers are the argument.
If an enterprise SLA promises 99.9% monthly availability, the error budget is about 43 minutes per 30-day month. A single twenty-minute cross-service event consumes roughly 46% of that budget if it is counted as full unavailability. Most of these events are not counted that way. They are classified as degradation, and degradation is either excluded from credit calculations or weighted by a formula that lives in a contract appendix nobody reads at signing time.
The measurement boundary is the product. A CI pipeline that authenticates every step against a failing authorization service has an availability of zero for the duration, while the service itself may report 99.9% on its own dashboard. That is not fraud. It is a definitional gap, and enterprise procurement signs across it daily. I have been on the receiving end of that gap as a user and on the defending end as an operator. Both sides prefer it to the alternative, because a rigorous definition of availability would make the contracts longer and the invoices larger. An SLA is a promise that both parties agree not to audit.
And twenty minutes is the status-page frame, not the event. Add the tail: merge queues draining, requeued workflow matrices running at reduced concurrency, webhook replays, replication catch-up, and every downstream retry policy quietly accumulating jitter-free backoff timers. My estimate for a twenty-minute event with a large Actions matrix is a two-to-six hour recovery tail. Confidence: medium. Basis is queue-depth reasoning, not telemetry, because the telemetry is not public.
Contrarian: Git Solved This, and Then We Undid It
The comfortable reading of this event is that a company failed to keep pace with its own scale. The uncomfortable reading is that the underlying protocol solved this problem in 2005, and this industry spent fifteen years unwinding the solution.
Git is a distributed version control system. Every clone is a complete replica with full history. You can commit, branch, diff, and merge with the network cable unplugged. It was designed by a person with documented contempt for centralized hosting, explicitly so that the loss of any single server would be an inconvenience rather than a stoppage.
Then we rebuilt the centralization on top of it, one reasonable local decision at a time. Pull-request-gated review, so nothing merges without an API call. Actions-only CI, so nothing ships without a hosted runner. Hosted container registries, so nothing deploys without a hosted pull. Automated dependency bots, so nothing gets patched unless a hosted service notices. Server-side branch protection, so nothing lands without a hosted policy engine approving. Each of those decisions was correct in isolation, defended competently in a design review, and adopted universally. Collectively they reintroduced the exact dependency the protocol was built to eliminate.
We adopted a distributed protocol and then bolted a centralized authorization layer onto the top of it, and called the result best practice. If you want the same pattern in another domain: hash power concentrating into a handful of pools while everyone recites the decentralization of the node count. We accept the convenient layer and pretend the redundant layer beneath it is doing work it is not being asked to do.
There is a second blind spot, and it is a security one rather than an availability one. Intermittent 401s against valid credentials teach a population of users a reflex: refresh the token, re-authenticate, paste the credential again. That reflex is exactly what a phishing kit requests. An adversary who can induce auth-layer flakiness — or time a campaign to coincide with it — inherits a user base trained to re-authenticate on demand and progressively unable to distinguish a legitimate prompt from an illegitimate one. Nobody puts this in a threat model because it presents as a UX consideration. It is not. Availability engineering and security engineering are one discipline with two vocabularies. Code does not lie, but it often forgets to breathe.
Worth noting what the transparency actually bought us. A single sentence of status-page prose disclosed more about internal architecture — a named datastore, a named failure mode, a named dependency direction — than any marketing artifact ever has. If that level of disclosure were standard, root-cause analysis would be a formality rather than a research project. Optimism's RetroPGF is the only public-goods funding mechanism I have seen that reliably pays for defensive infrastructure instead of paying for announcements, and the reason it matters here is that most of the redundancy which would have softened this event is unglamorous maintenance work that nobody funds voluntarily.
Takeaway
The RCA will point at either a replication topology change or a bad configuration push. Configuration changes account for the plurality of large-platform cascades in the public record, and the nouns in the disclosed chain lean that way. Confidence: medium. When it lands, read the section describing what the authorization layer does when the datastore beneath it is stale, and read it as a design document rather than an apology.
What to monitor is not GitHub's uptime percentage. It is whether teams move deployment authority off a shared third-party identity plane. Self-hosted runners. Signed commits. Mirrored bare repositories on infrastructure you control. Deploy keys scoped to a single repository with a command restriction. All documented, all cheap, all almost universally unadopted, because the hosted path is three lines shorter in a YAML file and nobody gets promoted for writing the four-hundred-word runbook.
Twenty minutes cost this industry nothing measurable today. Ask what the twenty-hour version costs. A consensus client release delayed across a hardfork boundary. An emergency upgrade window missed because the pipeline that executes it is returning 401s. A bridge's operational runbook executed by hand, from memory, under time pressure, by someone reading a status page that says "degraded."
Gas wars are just ego masquerading as utility. Uptime claims, it turns out, are a status page masquerading as architecture.
