Outage report for August 31st, 2019

On the morning of August 31st, we were notified by our network monitoring systems that all Pushover services became unavailable at 7:22 CDT. This included our API, website, desktop client, and dispatch of any queued messages. A message announcing the outage was posted to our Twitter account @PushoverApp.

We were notified soon after by our server/network provider that there was a datacenter-wide power outage affecting all customers of the datacenter. Unfortunately an ETA was not available from the provider during this outage.

At 9:29 CDT, power was restored to the datacenter, but the abrupt shutdown of our servers left their database replication in a failed state. This was resolved and all Pushover services were made available by 9:59 CDT.

While Pushover’s infrastructure is largely redundant and can usually survive a partial outage without affecting users, this large-scale outage affected all of our servers at once. We will be looking into what measures can be taken to further enhance our redundancy to avoid future outages of this type.

For more updates, we encourage users to follow our Twitter account @PushoverApp.