Intermittent connectivity issues on December 21st, 2015

On December 21st, 2015 at approximately 03:15 CST, Pushover’s web services began experiencing intermittent connectivity and latency problems that persisted until 06:40. These issues have been resolved and Pushover’s website, API, and Desktop client should be responding normally for all users. If you continue to experience problems reaching any of our services, please contact us.

Background: during the morning of Friday, December 18th, 2015, the upstream network provider for Pushover began experiencing intermittent packet loss along networks reaching some users. At this time, Pushover was switched over to its secondary server configuration, which exists on separate hardware at a different datacenter with a different upstream provider. This transition was performed seamlessly and all traffic was being routed to its backup network by the afternoon.

However, on the morning of Monday, December 21st, 2015, some users began experiencing high latency or failed connections to Pushover’s website and API. It was determined that these failures were caused by certain parts of the server software reaching built-in resource limitations. These limits had not been raised on the secondary server to the level that they were on the primary server that had been processing most traffic up until Friday. The limits were raised on the secondary server and it began processing all requests normally by 06:40.