October 24-25 DDoS Attack

On October 20th, Pushover’s website began receiving a Distributed Denial of Service (DDoS) attack from thousands of different IP addresses. The rate of attack was not substantial and was easily identifiable, so it did not cause much of a service disruption. This attack was limited to our website (dashboard) and did not impact our API or message sending.

Once the IP addresses were automatically blocked throughout the 20th and 21st, the attackers moved on to more substantial attacks on Pushover’s API servers beginning on October 24th. The rate of these attacks were substantial enough that Pushover’s network infrastructure provider began automatically blocking much of the traffic which unfortunately impacted legitimate access to our API.

Throughout the 24th and 25th, our API and website were sporadically unavailable due to these attacks, at one point causing one our servers to become unresponsive. Upon restarting the server, its replicated database became out of sync, causing a very small number of users, devices, and applications registered during a small window on the 24th to be lost as it was re-synchronized.

In the afternoon of the 25th, we moved our services behind a DDoS mitigation company to reduce the impact on our infrastructure and restore reliable connectivity to our users. During this move, our e-mail gateway services for @api.pushover.net and @pomail.net services were temporarily unreachable for around two hours. As these e-mails were likely queued up on remote e-mail servers, once our services were restored, these messages flowed into our e-mail gateways and notifications were processed as quickly as possible.

At no point during these attacks were our servers compromised or breached. These attacks were packet floods and DNS amplification attacks aimed at our servers with the intent to make them unreachable and/or cause monetary loss.

Technical note: A side-affect of moving behind the DDoS mitigation service is that TLS (SSL) negotiation against api.pushover.net is now being handled by the mitigation company so we are not currently able to support legacy TLS/SSL ciphers which may be needed by older servers, which we had been able to accommodate before.

We are continuing to monitor these attacks and will make necessary mitigations as needed.

Update 10/26 05:45 CDT: After a quiet period of service stability, the DDoS attacks resumed early this morning and Pushover’s network provider completely disabled (null routed) access to api.pushover.net’s IP addresses, making our API unreachable for about 2 hours. Service has been restored at 05:20 CDT.