Verizon Wireless Issue (Resolved)

On October 10th, we became aware of an issue with our iOS and Android apps reporting errors reaching our API, only affecting users on Verizon Wireless.

This issue does not affect receiving push notifications through Pushover, as those are relayed through Apple’s and Google’s servers, though downloading of notification image attachments and app icons is done through our API, which cannot be reached on Verizon. This would also affect reaching our website through the app or a web browser while using Verizon wireless for connectivity.

Upon gathering more reports, the issue appears to be DNS related, and Verizon is actively blocking the entire pushover.net domain for reasons unknown. Attempts to contact Verizon’s technical support and security team have been unhelpful.

We are encouraging Verizon customers to contact Verizon and report the problem so they are made aware of it.

Update 10/13: After receiving some DNS troubleshooting data from a helpful Pushover user on Verizon, it appears Verizon is using a 3rd party DNS filtering system which has erroneously flagged the pushover.net domain as being associated with malware (possibly due to the nature of how our API and callback mechanisms work). After contacting the company that is providing this DNS information, we are awaiting their delisting of our domain.

Update 10/13 17:00 CDT: We have released a new version of our Android app with a temporary workaround that tries a fallback domain if the pushover.net domain cannot be resolved. We are working on releasing a similar workaround for iOS.

Update 10/14 16:10 CDT: We have been notified by the 3rd party DNS filtering company that they have investigated our domain and found that it never had any malware, and have removed our domain from their blacklist. We are working to confirm that Verizon customers can now reach our domain.

Update 10/14 20:32 CDT: We have confirmed that Verizon Wireless customers are now able to reach our pushover.net domain.

Outage report for August 31st, 2019

On the morning of August 31st, we were notified by our network monitoring systems that all Pushover services became unavailable at 7:22 CDT. This included our API, website, desktop client, and dispatch of any queued messages. A message announcing the outage was posted to our Twitter account @PushoverApp.

We were notified soon after by our server/network provider that there was a datacenter-wide power outage affecting all customers of the datacenter. Unfortunately an ETA was not available from the provider during this outage.

At 9:29 CDT, power was restored to the datacenter, but the abrupt shutdown of our servers left their database replication in a failed state. This was resolved and all Pushover services were made available by 9:59 CDT.

While Pushover’s infrastructure is largely redundant and can usually survive a partial outage without affecting users, this large-scale outage affected all of our servers at once. We will be looking into what measures can be taken to further enhance our redundancy to avoid future outages of this type.

For more updates, we encourage users to follow our Twitter account @PushoverApp.

Animated GIFs and monospace text

Pushover 3.4 has been released today for Android, iPhone/iPad, and Desktop, and brings full support for animated GIFs and monospace text support.

Many Pushover users rely on our e-mail gateway to receive Pushover notifications from e-mails sent out by devices that don’t directly support Pushover, such as security cameras. After adding image support, we received a number of requests from users to support animated GIFs that are sent out from these security cameras, showing a few seconds of activity instead of just a static snapshot.

Animated GIFs can be sent directly to our API as well, as detailed in our API documentation.

In addition to animated GIFs, version 3.4 brings support for monospace text in messages (but not notifications at this time, due to limitations on mobile platforms). This text is shown in a monospace font that preserves spaces and tabs, which can be useful for sending a formatted message with columns. These messages can be sent to our API with the monospace parameter set to 1.

Pushover 3.4 is available now for Android, iPhone/iPad, and Desktops/Browsers.