Skip to content
Emberwire OTA firmware delivery

Platform status

All systems operational.

Last evaluated 24 September 2026 at 09:40 UTC. Delivery health is measured from what devices actually managed to do, not from a probe in the same rack as the node.

14 / 14delivery nodes taking traffic
99.97 %delivery availability, 30 days
4waves currently running
0open incidents

Components

Component State 30 days 90 days What it does
Control-plane APIoperational 99.98 %99.96 % Everything under https://api.{{DOMAIN}}/v1.
Image ingestoperational 100.00 %99.99 % Chunked upload, deduplication, delta precomputation.
Signing serviceoperational 100.00 %100.00 % Manifest signing for customers who delegate their release key.
Wave scheduleroperational 99.99 %99.97 % Cohort advancement, soak timing, halt evaluation.
Delivery networkoperational 99.97 %99.95 % 14 nodes. Measured as completed transfers over attempted.
Device check-inoperational 99.99 %99.98 % The protocol every enrolled device speaks on its interval.
Webhooksoperational 99.94 %99.90 % Outbound event delivery. Never affects wave behaviour.
Dashboardoperational 99.99 %99.98 % A client of the same API you use.
Audit exportoperational 100.00 %99.99 % Signed JSONL generation for the retention window.

Delivery network — last 30 days

26 Aug → 24 Sep 2026

Orange: the Warsaw cache-tier fault on 8 September. Grey: the following day, while devices that had re-fetched blocks from Amsterdam drained back to their usual node. No device installed a bad image on either day — the digest check is exactly what it is for.

Waves running now

Aggregated across customers, product lines anonymised to their identifiers.

Release Product line Cohort Offered Confirmed Rolled back State
rel_2026_09_22_cpl_router_ax3 4 of 5 · 25 % 314,900313,74441 advancing
rel_2026_09_23_apl_meter_g4 2 of 5 · 1 % 4,7824,7603 soaking · 2 h 11 m
rel_2026_09_19_bpl_pos_t7 5 of 5 · 100 % 37,41037,20912 completing
rel_2026_09_24_apl_sensor_e2 1 of 5 · canary 2,1402,1310 soaking · 3 h 48 m
4 waves—— 359,232357,84456 —

Rolled-back devices are running their previous build and are working. Across the four waves the rollback rate is 0.016 % — well under the 0.5 % halt threshold.

Node state

NodeCityState Egress nowCache hit Median first byte
fra-aFrankfurtserving7.4 Gbit/s99.1 %18 ms
fra-bFrankfurtserving6.1 Gbit/s98.8 %21 ms
ams-aAmsterdamserving5.2 Gbit/s98.4 %24 ms
lon-aLondonserving4.6 Gbit/s98.9 %27 ms
waw-aWarsawserving3.1 Gbit/s97.6 %31 ms
sto-aStockholmserving1.9 Gbit/s98.2 %34 ms
mad-aMadridserving2.4 Gbit/s98.0 %29 ms
mil-aMilanserving2.2 Gbit/s97.9 %33 ms
iad-aAshburnserving5.8 Gbit/s99.3 %22 ms
dfw-aDallasserving3.9 Gbit/s99.0 %26 ms
yyz-aTorontoserving1.7 Gbit/s97.4 %35 ms
gru-aSão Pauloserving2.0 Gbit/s96.8 %48 ms
sin-aSingaporeserving2.8 Gbit/s98.1 %38 ms
nrt-aTokyoserving1.5 Gbit/s97.2 %41 ms
14 nodes——50.6 Gbit/s98.3 %—

Egress now is a one-minute average. It is high at the moment because the router release is in its 25 % cohort — see the wave table above. Full inventory on the delivery network page.

Scheduled maintenance

When (UTC)WhatExpected impact
1 Oct 2026 · 01:00–03:00 Cache tier replacement at fra-b. Node drains beforehand and stops appearing in candidate lists. No transfer is interrupted; European waves may run slightly slower.
8 Oct 2026 · 02:00–02:20 Control-plane database version upgrade. API read-only for up to 20 minutes. Running waves continue; wave control calls return E_INTERNAL and are safe to retry.
22 Oct 2026 · 01:00–04:00 Two additional delivery nodes enter production. None expected. They will appear in the node inventory on the day they take traffic.

Incident history

Everything in 2026 that degraded a customer-visible behaviour, with what actually caused it.

8 September 2026 · 19:12–21:04 UTC degraded · 1 h 52 m

Corrupt blocks served from waw-a

A failing storage device in the Warsaw cache tier returned blocks that did not match their digests. Agents detected the mismatch, discarded the data without writing it, and re-fetched from ams-a. Roughly 2,400 devices saw a slower transfer and 340 reported E_DIGEST at least once. No device staged or installed a corrupt image.

Fix: the node was drained at 19:31 once the pattern was clear, the storage device was replaced, and the cache was rebuilt from origin. We have since added a per-node bad-block rate to the health check so a node removes itself before three devices have to notice. Shipped in platform 2026.09.1.

14 July 2026 · 06:40–09:46 UTC partial · 3 h 06 m

Wave scheduler held at cohort boundaries

A control-plane database failover left the scheduler unable to persist cohort transitions. It did what it is designed to do in that situation and held, rather than advancing without a durable record. Eleven waves across four customers did not advance for the duration; two releases finished roughly three hours later than planned.

No device received an unintended build, and no halt failed to fire — halting does not depend on the same write path. Fix: the scheduler now keeps a local intent log and replays it after a failover, which reduces the hold to the failover time itself.

2 May 2026 · 11:05–16:25 UTC degraded · 5 h 20 m

Webhook delivery backlog

A customer endpoint began answering with a 90-second delay before returning 200. Our retry queue treated those as successes and filled its worker pool, delaying webhook delivery for every customer by up to 40 minutes at the peak.

Platform behaviour was unaffected: halts, cohort advancement and device delivery do not depend on webhooks. Fix: per-destination worker isolation and a 10-second response timeout.

11 February 2026 · 22:18–23:06 UTC degraded · 48 m

Control-plane latency above 2 s

An unbounded query behind the device-search endpoint was triggered by a selector matching several million devices. API latency rose across the board; the dashboard was effectively unusable for the duration.

Fix: the selector evaluator now refuses an unbounded search with E_SELECTOR and requires a cursor. Device check-ins were never affected — they are served by a separate path, on purpose.

Notifications

Get told when something changes here

One message per incident, one per scheduled maintenance, nothing else. Status notifications are separate from your account and can go to a shared operations address.

This page is static: submitting shows a confirmation and sends nothing.