HyperEVM data delayed
Major
2026-11-14 18:40 UTC · 11.9 h · HyperEVM · Upstream data provider · Resolved
A third-party trace provider for HyperEVM returned errors, so HyperEVM blocks were delivered up to ~12 hours late. Nothing was lost. Shorter provider stalls on the same network: 11/11 (37 min), 11/13 (5 min), 11/14 (31 min) and 11/15 (15 min).
FixFailover to alternate trace providers; adding trace-provider redundancy is in progress.
Polygon block delivery stalled
Major
2026-11-05 14:01 UTC · 69 min · Polygon · Upstream data provider · Resolved
An upstream RPC provider failed to follow a Polygon hard fork, which stopped Polygon ingestion for about 70 minutes. Blocks resumed and were delivered in order.
FixMoved Polygon to a second provider.
Ethereum beacon-chain data delayed
Minor
2026-11-04 17:05 UTC · 4.9 h · Ethereum Beacon Chain · Upstream data provider · Resolved
The beacon-API provider hung, which stalled the Ethereum beacon-chain feed for about 5 hours. Execution-layer Ethereum was unaffected.
FixSwitched the beacon feed to another provider.
Some pipeline deliveries delayed
Major
2026-11-02 23:48 UTC · 21 min · All customers · Our platform · Resolved
A repeat of the 11/01 memory throttle, before the fix shipped. Deliveries kept flowing, but about 1,400 jobs across 8 customers ran up to 21 minutes late. None were lost.
FixSame as 11/01; the raised memory limits shipped the next morning.
Pipeline deliveries delayed for all customers
Major
2026-11-01 11:25 UTC · 2.5 h · All customers · Our platform · Resolved
The pipeline worker hit its memory ceiling and kernel reclaim throttling stalled it. Deliveries to every customer stopped from 11:25 to 13:55 (they had been queueing late since 10:40). About 65,000 deliveries were late by up to 2h20m. None were lost.
FixRaised the worker memory limits and added a phone-paging alert for any queue stall that affects three or more customers.
Pipeline deliveries delayed for all customers
Major
2026-10-31 13:12 UTC · 20 min · All customers · Our platform · Resolved
A short precursor of the 11/01 stall, with the same memory-throttle signature.
FixSame as 11/01.
Pipeline deliveries delayed for some customers
Minor
2026-10-06 16:28 UTC · 5.3 h · Some customers · Our platform · Resolved
During the migration, a high-volume pipeline monopolised the new engine's job queue, so deliveries for customers already on the new engine were held about 5 hours behind. Customers still on the legacy engine were unaffected. Nothing was lost.
FixFair, live-first job claiming and a continuous worker pool, which drains about 3.5× faster.
Intermittent API errors
Minor
2026-10-05 00:00 UTC · 46.8 h · All customers · Our platform · Resolved
A request to an unknown network could crash an API worker. While the worker restarted, about 1 in 8 requests on some routes returned 502.
FixFixed the error handler so a bad request returns 404 instead of crashing the worker.
Elevated delivery latency across chains
Minor
2026-09-20 14:54 UTC · 10.8 h · Multiple networks · Our platform · Resolved
A new Polkadot Asset Hub transaction format broke decoding, and the error path exhausted worker memory. Delivery latency on other chains rose from seconds to a few minutes. Nothing was lost.
FixIsolated undecodable transactions so a decode error no longer recycles the connection.
Platform-wide delivery disruption
Major
2026-02-16 14:00 UTC · 2.0 h · All customers · Our platform · Resolved
A data disk was unmounted on one ingestion host and two cluster processes errored. Deliveries to almost every customer crawled for about 2 hours across 14:00–19:00, with catch-up bursts in between. Nothing was lost.
FixDisk remounted and processes restarted. Superseded by the later single-box rebuild with host-level disk and memory alerts.
Platform-wide outage (February instability)
Major
2026-02-05 05:00 UTC · 3.5 h · All customers · Our platform · Resolved
Part of a rough stretch from 2/04 to 2/16 in which the old ingestion stack fell behind or went down several times. On 2/05 the ingestion cluster stopped processing blocks on every network and no customer received deliveries for 3–4 hours (hourly-resolution record). Shorter 'behind real time' delays on 2/04, 2/07 and 2/12 are counted as degraded.
FixCluster restarted. The pacemaker cluster was later rebuilt (oscar v2), and per-network stall and host alerts were added.
Platform-wide delivery disruption
Major
2026-01-15 09:00 UTC · 60 min · All customers · Our platform · Resolved
Deliveries to most customers fell below a quarter of normal for up to an hour (hourly-resolution record). The root cause was not recorded.
FixNot recorded.
Platform-wide ingestion and delivery outage
Major
2026-01-13 08:05 UTC · 3.9 h · All customers · Our platform · Resolved
Every ingestion cluster node stopped reporting at 08:05. Block processing halted on all networks, and deliveries stopped for all but a trickle of one or two customers for about 4 hours.
FixCluster restarted; same follow-ups as 2026-02-05.
Platform-wide delivery outage
Major
2025-06-12 18:00 UTC · 110 min · All customers · Our platform · Resolved
Deliveries to every customer stopped for up to 2 hours (hourly-resolution record). The root cause was not recorded.
FixNot recorded.
How we measure and classify
Data API. Probed every minute from Cloudflare's network, outside our infrastructure. Earlier history comes from
our own once-a-minute probe of the same endpoint and, since 2026-10-12, from request logs: a minute is down when at least half of
requests returned a server error or none were served.
Pipeline delivery. Whether the shared platform is moving data for every customer. It is down when no
customer receives a delivery for 5 minutes or more, or when our workers stop or stop receiving blocks. A failed write to a customer's own
endpoint is not downtime (we attempted delivery). Problems affecting only some customers are listed but not counted, and delays from
upstream data providers appear under block ingestion. History before launch comes from our delivery logs, at minute resolution.
Block ingestion. Two timings per network, checked every minute: how old the newest processed block is
(are we keeping up with the chain?) and how long the oldest unprocessed block has been waiting (how deep is the backlog?). Either over the
network's delivery target (5 min by default) is degraded; over its stall limit (1 h by default, longer for slow chains like Bitcoin) is an
outage. A handful of stuck blocks on an otherwise current network is shown as delayed blocks, not an outage. Uptime is counted in
network-minutes across core networks. History comes from our per-network stall and backlog alerts, which go back to July 2025, with
backlog sizes converted to time using each network's measured block time.
Severity. Major: data lost, the whole platform stalled for 10 min or more, a core network down for an hour or more,
or one customer's deliveries failing for an hour or more. Minor: shorter core-network outages (15–60 min), a supported network down for an hour or more,
or partial errors for 15 min or more. Shorter events and anything confined to testnets are tracked internally but not listed here.
What counts. Platform-wide failures, ours or a data provider's, count against uptime. Incidents affecting only some
customers are listed but don't count as platform downtime. A network that itself halts is
listed but not counted, because no data exists to deliver. Minutes we could not observe are excluded rather than counted as up. Times are UTC.