Back to overview
Resolved

Voice calls interruption (upstream datacenter network fault)

Jul 24, 2026 at 12:48pm UTC
Affected services
Calls

Resolved
Jul 24, 2026 at 1:47pm UTC

Post-mortem, 24 July 2026. Voice calls were interrupted for approximately 11 minutes (14:36 to 14:48 CEST). Inbound, outbound and web calls routed through our primary voice node were affected. The dashboard, API, data storage and documentation were not affected, and no customer data was lost or exposed.

What happened. Our infrastructure provider, Hetzner, had a network fault in their Nuremberg datacenter that isolated our primary voice node. Our platform automatically detected the failure and began failing over to a secondary node in a separate location. One step in that automated failover, moving our public network address to the secondary node, did not complete on its own: it relies on the provider's management API, which was itself degraded by the same fault. Our team completed that step manually, which restored service. The provider resolved the underlying fault shortly afterwards.

What we have changed. We have already rewritten that failover step so it completes reliably on its own, without manual intervention, even when the provider's systems are degraded, and we validated the change the same day with a live failover. We are additionally building a second, independent failover path that does not rely on the provider's systems at all.

We hold ourselves to a high reliability standard, and an interruption of this length is more than we accept. We are sorry for the disruption.

Created
Jul 24, 2026 at 12:48pm UTC

At 14:36 CEST our primary voice node in Nuremberg became unreachable due to a network fault at our datacenter provider (Hetzner, nbg1 region). Inbound and web calls routed through this node were affected. We failed over to our secondary voice node in a separate location and calls have been restored. Any call in progress during the short window may have dropped; no customer data was affected, as account and call data are stored separately and were not impacted. We are now serving from the secondary node and will keep monitoring while the provider resolves the upstream fault.</message>
<parameter name="starts_at">2026-07-24T12:36:00Z