Incidents | Voicedock

Voice calls interruption (upstream datacenter network fault)

Fri, 24 Jul 2026 13:47:17 -0000

Post-mortem, 24 July 2026. Voice calls were interrupted for approximately 11 minutes (14:36 to 14:48 CEST). Inbound, outbound and web calls routed through our primary voice node were affected. The dashboard, API, data storage and documentation were not affected, and no customer data was lost or exposed. What happened. Our infrastructure provider, Hetzner, had a network fault in their Nuremberg datacenter that isolated our primary voice node. Our platform automatically detected the failure and began failing over to a secondary node in a separate location. One step in that automated failover, moving our public network address to the secondary node, did not complete on its own: it relies on the provider's management API, which was itself degraded by the same fault. Our team completed that step manually, which restored service. The provider resolved the underlying fault shortly afterwards. What we have changed. We have already rewritten that failover step so it completes reliably on its own, without manual intervention, even when the provider's systems are degraded, and we validated the change the same day with a live failover. We are additionally building a second, independent failover path that does not rely on the provider's systems at all. We hold ourselves to a high reliability standard, and an interruption of this length is more than we accept. We are sorry for the disruption.

Calls recovered

Fri, 24 Jul 2026 13:03:39 +0000

Calls recovered

Calls went down

Fri, 24 Jul 2026 12:59:53 +0000

Calls went down

Voice calls interruption (upstream datacenter network fault)

Fri, 24 Jul 2026 12:48:45 -0000

At 14:36 CEST our primary voice node in Nuremberg became unreachable due to a network fault at our datacenter provider (Hetzner, nbg1 region). Inbound and web calls routed through this node were affected. We failed over to our secondary voice node in a separate location and calls have been restored. Any call in progress during the short window may have dropped; no customer data was affected, as account and call data are stored separately and were not impacted. We are now serving from the secondary node and will keep monitoring while the provider resolves the upstream fault. 2026-07-24T12:36:00Z

Calls recovered

Fri, 24 Jul 2026 12:45:12 +0000

Calls recovered

Calls went down

Fri, 24 Jul 2026 12:36:51 +0000

Calls went down

API and database unavailable

Mon, 29 Jun 2026 11:06:47 -0000

**Post-mortem — API & database outage, 29 June 2026** **Summary.** Between approximately 10:25 and 11:00 UTC (~35 minutes), our API, database and call handling were unavailable. A network fault in our underlying hosting infrastructure isolated the host that runs our primary database. Service has been fully restored and the database recovered cleanly, with no data loss. **Timeline (UTC).** - 10:25 — Monitoring detected the database and API becoming unreachable. - 10:27–10:40 — Confirmed the primary database host was isolated at the network layer: the server itself stayed healthy but was cut off from the network (the hosting provider's own management console could not reach it either). - ~10:57 — The host was recovered by our infrastructure provider. - 11:00 — Database, API and calls fully restored; we verified the database recovered cleanly. **Root cause.** A network switch fault in our hosting infrastructure isolated the single host that runs our primary database — the machine kept running, but was cut off from the network and the internet. The same class of fault has affected us once before. Because our primary database currently runs on a single host, recovery depended on the provider restoring connectivity rather than on us failing over to a standby. **Impact.** API requests failed, dashboard data was unavailable, and inbound and outbound calls could not be established (call configuration is resolved from the database). No data was lost. **What we're doing about it.** The redundancy and failover work that lets us switch to a standby database in a separate location — independent of any single host or network segment — is in active development and being prioritized. Database backups are already replicated off-site. We'll keep posting updates as these safeguards go live. We're sorry for the disruption. If this outage affected your integration and you'd like specifics, reach us at support@flireo.com.

API and database unavailable

Mon, 29 Jun 2026 11:02:36 -0000

Resolved: service has been fully restored. The underlying infrastructure fault recovered around 11:00 UTC and our database, API and calls are operating normally again. We verified the database recovered cleanly. Total impact was approximately 10:25–11:00 UTC. We recognize this is the second occurrence of this failure mode, and the redundancy/failover work that prevents it is being prioritized. Thank you for your patience.

Api recovered

Mon, 29 Jun 2026 10:59:38 +0000

Api recovered

Databases recovered

Mon, 29 Jun 2026 10:57:58 +0000

Databases recovered

API and database unavailable

Mon, 29 Jun 2026 10:51:25 -0000

Identified: a fault in our underlying infrastructure has isolated our database — the same root cause as our previous outage. The redundancy/failover that prevents this is in active development but not yet live. We're actively monitoring recovery and will update here as soon as service is restored. We apologize for the disruption.

API and database unavailable

Mon, 29 Jun 2026 10:51:16 -0000

We're investigating an outage affecting our API and database. Inbound and outbound calls, the API, and dashboard data may be unavailable or degraded. We'll post updates here as we learn more.

Api went down

Mon, 29 Jun 2026 10:27:08 +0000

Api went down

Databases went down

Mon, 29 Jun 2026 10:25:44 +0000

Databases went down

Service disruption — upstream network incident (recurrence)

Thu, 18 Jun 2026 00:35:00 -0000

Resolved. Our upstream provider restarted the affected systems and connectivity was restored at 00:31 UTC. All services have been operational since. Post-mortem: between 22:55 and 00:31 UTC a recurring network fault at our cloud infrastructure provider isolated the database node that powers VoiceDock, making the API, dashboard and voice calls unavailable for roughly 1 hour and 36 minutes. This was the second occurrence of the same upstream fault within 24 hours. The node itself stayed healthy throughout and no data was lost; the outage was a loss of network connectivity, resolved once the provider restored the affected segment. A single dependency should not be able to take the service down. We are accelerating two improvements already underway: relocating our backups to storage independent of the affected infrastructure, and removing the single point of failure in our data layer through geographic redundancy. We apologise for the disruption and are monitoring closely.

Databases recovered

Thu, 18 Jun 2026 00:28:50 +0000

Databases recovered

Api recovered

Thu, 18 Jun 2026 00:28:29 +0000

Api recovered

Service disruption — upstream network incident (recurrence)

Wed, 17 Jun 2026 23:25:00 -0000

The cause has been identified as a network fault at our upstream cloud infrastructure provider, the same fault that affected us earlier today. The provider's technicians are working on restoring the affected network segment. Our database node itself remained healthy but was unreachable at the network level, and no data has been lost.

Api went down

Wed, 17 Jun 2026 22:57:08 +0000

Api went down

Service disruption — upstream network incident (recurrence)

Wed, 17 Jun 2026 22:57:00 -0000

We are investigating a renewed loss of connectivity to our database and API, affecting voice calls and the dashboard. This appears to be a recurrence of the earlier upstream network incident at our cloud infrastructure provider. We are working to restore service as quickly as possible.

Databases went down

Wed, 17 Jun 2026 22:55:36 +0000

Databases went down

Service disruption — upstream network incident

Wed, 17 Jun 2026 12:43:36 -0000

**Post-mortem — service disruption on 17 June 2026** **Summary.** VoiceDock experienced a ~45-minute disruption caused by a network fault at our cloud infrastructure provider's datacenter. Inbound and outbound calls and the dashboard were unavailable during the window. No data was lost or corrupted. **Impact (≈11:18–12:04 UTC).** Calls could not be configured and the dashboard could not load data, because both depend on our database host, which was isolated at the network level. The host itself stayed healthy — only its datacenter network path failed. Call recordings (stored separately) were unaffected. **Timeline (UTC).** - 10:54 — last call processed normally. - 11:18 — automated monitoring detected the API and database becoming unreachable; on-call was alerted within ~1 minute. - ~11:30 — investigation confirmed the host was fully unreachable at the network level; cause traced to the provider. - 11:48 — status page updated with the identified cause. - ~12:03 — the provider restored the network segment and the host returned automatically. - 12:08 — all services verified healthy; incident resolved. **Root cause.** A network switch ("leaf") fault in our provider's datacenter isolated the server hosting our database and platform at the network level — an upstream provider incident outside our direct control. **What went well.** Sub-minute detection and alerting, customer communication via this status page during the window, and nightly-validated backups (restore and secret decryption both verified). **What we're improving.** 1. **Off-site backups** — moving database backups to independent, off-provider storage so recovery never depends on the affected host. 2. **Redundancy** — evaluating a standby in a second location and a managed, multi-region database option to remove this single-provider, single-location failure mode. We apologise for the disruption and thank you for your patience.

Service disruption — upstream network incident

Wed, 17 Jun 2026 12:08:57 -0000

Connectivity has been fully restored. The upstream network incident at our cloud provider has been resolved, and all VoiceDock services — inbound and outbound calls and the dashboard — are operating normally again. We are continuing to monitor to confirm stability. Thank you for your patience, and we apologise for the disruption.

Databases recovered

Wed, 17 Jun 2026 12:06:41 +0000

Databases recovered

Api recovered

Wed, 17 Jun 2026 12:05:48 +0000

Api recovered

Service disruption — upstream network incident

Wed, 17 Jun 2026 11:48:16 -0000

We have identified the cause of the current disruption. A network fault at our cloud infrastructure provider's datacenter has interrupted connectivity to part of the VoiceDock platform. This is an upstream incident outside our direct control; the provider's engineers are actively working on a resolution. During this period, inbound and outbound calls and the dashboard may be unavailable. No data has been lost or is at risk — the disruption is limited to network connectivity. We are monitoring closely and will post a further update as soon as service is restored or we have a clearer timeline. We apologise for the inconvenience.

Databases went down

Wed, 17 Jun 2026 11:19:36 +0000

Databases went down

Api went down

Wed, 17 Jun 2026 11:18:03 +0000

Api went down

Api recovered

Wed, 10 Jun 2026 10:37:01 +0000

Api recovered

Api went down

Wed, 10 Jun 2026 10:35:43 +0000

Api went down

Databases recovered

Wed, 10 Jun 2026 10:33:04 +0000

Databases recovered

Databases went down

Wed, 10 Jun 2026 10:27:50 +0000

Databases went down

Api recovered

Wed, 10 Jun 2026 10:05:40 +0000

Api recovered

Api went down

Wed, 10 Jun 2026 09:59:44 +0000

Api went down

Docs recovered

Wed, 06 May 2026 05:46:12 +0000

Docs recovered

Docs went down

Wed, 06 May 2026 00:01:09 +0000

Docs went down