October 24-25 DDoS Attack

On October 20th, Pushover’s website began receiving a Distributed Denial of Service (DDoS) attack from thousands of different IP addresses. The rate of attack was not substantial and was easily identifiable, so it did not cause much of a service disruption. This attack was limited to our website (dashboard) and did not impact our API or message sending.

Once the IP addresses were automatically blocked throughout the 20th and 21st, the attackers moved on to more substantial attacks on Pushover’s API servers beginning on October 24th. The rate of these attacks were substantial enough that Pushover’s network infrastructure provider began automatically blocking much of the traffic which unfortunately impacted legitimate access to our API.

Throughout the 24th and 25th, our API and website were sporadically unavailable due to these attacks, at one point causing one our servers to become unresponsive. Upon restarting the server, its replicated database became out of sync, causing a very small number of users, devices, and applications registered during a small window on the 24th to be lost as it was re-synchronized.

In the afternoon of the 25th, we moved our services behind a DDoS mitigation company to reduce the impact on our infrastructure and restore reliable connectivity to our users. During this move, our e-mail gateway services for @api.pushover.net and @pomail.net services were temporarily unreachable for around two hours. As these e-mails were likely queued up on remote e-mail servers, once our services were restored, these messages flowed into our e-mail gateways and notifications were processed as quickly as possible.

At no point during these attacks were our servers compromised or breached. These attacks were packet floods and DNS amplification attacks aimed at our servers with the intent to make them unreachable and/or cause monetary loss.

Technical note: A side-affect of moving behind the DDoS mitigation service is that TLS (SSL) negotiation against api.pushover.net is now being handled by the mitigation company so we are not currently able to support legacy TLS/SSL ciphers which may be needed by older servers, which we had been able to accommodate before.

We are continuing to monitor these attacks and will make necessary mitigations as needed.

Update 10/26 05:45 CDT: After a quiet period of service stability, the DDoS attacks resumed early this morning and Pushover’s network provider completely disabled (null routed) access to api.pushover.net’s IP addresses, making our API unreachable for about 2 hours. Service has been restored at 05:20 CDT.

Verizon Wireless Issue (Resolved)

On October 10th, we became aware of an issue with our iOS and Android apps reporting errors reaching our API, only affecting users on Verizon Wireless.

This issue does not affect receiving push notifications through Pushover, as those are relayed through Apple’s and Google’s servers, though downloading of notification image attachments and app icons is done through our API, which cannot be reached on Verizon. This would also affect reaching our website through the app or a web browser while using Verizon wireless for connectivity.

Upon gathering more reports, the issue appears to be DNS related, and Verizon is actively blocking the entire pushover.net domain for reasons unknown. Attempts to contact Verizon’s technical support and security team have been unhelpful.

We are encouraging Verizon customers to contact Verizon and report the problem so they are made aware of it.

Update 10/13: After receiving some DNS troubleshooting data from a helpful Pushover user on Verizon, it appears Verizon is using a 3rd party DNS filtering system which has erroneously flagged the pushover.net domain as being associated with malware (possibly due to the nature of how our API and callback mechanisms work). After contacting the company that is providing this DNS information, we are awaiting their delisting of our domain.

Update 10/13 17:00 CDT: We have released a new version of our Android app with a temporary workaround that tries a fallback domain if the pushover.net domain cannot be resolved. We are working on releasing a similar workaround for iOS.

Update 10/14 16:10 CDT: We have been notified by the 3rd party DNS filtering company that they have investigated our domain and found that it never had any malware, and have removed our domain from their blacklist. We are working to confirm that Verizon customers can now reach our domain.

Update 10/14 20:32 CDT: We have confirmed that Verizon Wireless customers are now able to reach our pushover.net domain.

Outage report for August 31st, 2019

On the morning of August 31st, we were notified by our network monitoring systems that all Pushover services became unavailable at 7:22 CDT. This included our API, website, desktop client, and dispatch of any queued messages. A message announcing the outage was posted to our Twitter account @PushoverApp.

We were notified soon after by our server/network provider that there was a datacenter-wide power outage affecting all customers of the datacenter. Unfortunately an ETA was not available from the provider during this outage.

At 9:29 CDT, power was restored to the datacenter, but the abrupt shutdown of our servers left their database replication in a failed state. This was resolved and all Pushover services were made available by 9:59 CDT.

While Pushover’s infrastructure is largely redundant and can usually survive a partial outage without affecting users, this large-scale outage affected all of our servers at once. We will be looking into what measures can be taken to further enhance our redundancy to avoid future outages of this type.

For more updates, we encourage users to follow our Twitter account @PushoverApp.

Animated GIFs and monospace text

Pushover 3.4 has been released today for Android, iPhone/iPad, and Desktop, and brings full support for animated GIFs and monospace text support.

Many Pushover users rely on our e-mail gateway to receive Pushover notifications from e-mails sent out by devices that don’t directly support Pushover, such as security cameras. After adding image support, we received a number of requests from users to support animated GIFs that are sent out from these security cameras, showing a few seconds of activity instead of just a static snapshot.

Animated GIFs can be sent directly to our API as well, as detailed in our API documentation.

In addition to animated GIFs, version 3.4 brings support for monospace text in messages (but not notifications at this time, due to limitations on mobile platforms). This text is shown in a monospace font that preserves spaces and tabs, which can be useful for sending a formatted message with columns. These messages can be sent to our API with the monospace parameter set to 1.

Pushover 3.4 is available now for Android, iPhone/iPad, and Desktops/Browsers.