iOS Critical Alerts

A new version of our iPhone/iPad app is available today that adds support for Apple’s Critial Alerts.

Added in iOS 12, Critical Alerts are special notifications that can bypass the device’s mute switch and Do Not Disturb settings to generate audible alerts in emergency situations. This functionality is only available to applications that have applied to and been approved by Apple. We’ve been applying since the day Critical Alerts were announced at Apple’s developer conference in 2018 and Pushover has finally been approved.

Pushover for iOS now supports generating Critical Alerts for your most important messages, such as security camera alerts. Our Android application has had similar functionality since 2013 by optionally playing notification sounds as alarms, bypassing the device’s mute switch.

Critical Alerts are not enabled by default, and can be enabled in the Pushover app’s settings menu. The first time the “Critical Alerts for high-priority” option is enabled, you’ll need to approve a dialog from iOS allowing Pushover to generate Critical Alerts:

Since Critical Alerts bypass your device’s mute switch, they do not play sounds at your device’s current volume setting. You’ll need to choose a volume at which to play Critlcal Alerts from Pushover, and these go to eleven.

Quick Do Not Disturb Mode

Today we’re releasing new versions of our Android, iPhone/iPad, and Desktop apps with a quick Do Not Disturb button. When pressed, you can quickly choose a time period in minutes, hours, or days, and all notifications for your devices will be silenced during that period. Once the time period is up, your notifications will resume automatically.

We’ve heard from many users that use Pushover to receive alerts from network monitors that when things break, they often break loudly with many alerts coming in a short amount of time. This new Do Not Disturb functionality can quickly silence new notifications while you work on resolving the problem. Once the Pushover app is opened, it will display all messages received during your Do Not Disturb period as usual.

For Android users, we’ve also added this button to our home-screen widget. The default time period used on the widget is the last time period selected inside the app.

For those needing Do Not Disturb on a regular schedule (such as overnight or on weekends), our Quiet Hours functionality continues to be available to set custom schedules.

October 24-25 DDoS Attack

On October 20th, Pushover’s website began receiving a Distributed Denial of Service (DDoS) attack from thousands of different IP addresses. The rate of attack was not substantial and was easily identifiable, so it did not cause much of a service disruption. This attack was limited to our website (dashboard) and did not impact our API or message sending.

Once the IP addresses were automatically blocked throughout the 20th and 21st, the attackers moved on to more substantial attacks on Pushover’s API servers beginning on October 24th. The rate of these attacks were substantial enough that Pushover’s network infrastructure provider began automatically blocking much of the traffic which unfortunately impacted legitimate access to our API.

Throughout the 24th and 25th, our API and website were sporadically unavailable due to these attacks, at one point causing one our servers to become unresponsive. Upon restarting the server, its replicated database became out of sync, causing a very small number of users, devices, and applications registered during a small window on the 24th to be lost as it was re-synchronized.

In the afternoon of the 25th, we moved our services behind a DDoS mitigation company to reduce the impact on our infrastructure and restore reliable connectivity to our users. During this move, our e-mail gateway services for @api.pushover.net and @pomail.net services were temporarily unreachable for around two hours. As these e-mails were likely queued up on remote e-mail servers, once our services were restored, these messages flowed into our e-mail gateways and notifications were processed as quickly as possible.

At no point during these attacks were our servers compromised or breached. These attacks were packet floods and DNS amplification attacks aimed at our servers with the intent to make them unreachable and/or cause monetary loss.

Technical note: A side-affect of moving behind the DDoS mitigation service is that TLS (SSL) negotiation against api.pushover.net is now being handled by the mitigation company so we are not currently able to support legacy TLS/SSL ciphers which may be needed by older servers, which we had been able to accommodate before.

We are continuing to monitor these attacks and will make necessary mitigations as needed.

Update 10/26 05:45 CDT: After a quiet period of service stability, the DDoS attacks resumed early this morning and Pushover’s network provider completely disabled (null routed) access to api.pushover.net’s IP addresses, making our API unreachable for about 2 hours. Service has been restored at 05:20 CDT.

Verizon Wireless Issue (Resolved)

On October 10th, we became aware of an issue with our iOS and Android apps reporting errors reaching our API, only affecting users on Verizon Wireless.

This issue does not affect receiving push notifications through Pushover, as those are relayed through Apple’s and Google’s servers, though downloading of notification image attachments and app icons is done through our API, which cannot be reached on Verizon. This would also affect reaching our website through the app or a web browser while using Verizon wireless for connectivity.

Upon gathering more reports, the issue appears to be DNS related, and Verizon is actively blocking the entire pushover.net domain for reasons unknown. Attempts to contact Verizon’s technical support and security team have been unhelpful.

We are encouraging Verizon customers to contact Verizon and report the problem so they are made aware of it.

Update 10/13: After receiving some DNS troubleshooting data from a helpful Pushover user on Verizon, it appears Verizon is using a 3rd party DNS filtering system which has erroneously flagged the pushover.net domain as being associated with malware (possibly due to the nature of how our API and callback mechanisms work). After contacting the company that is providing this DNS information, we are awaiting their delisting of our domain.

Update 10/13 17:00 CDT: We have released a new version of our Android app with a temporary workaround that tries a fallback domain if the pushover.net domain cannot be resolved. We are working on releasing a similar workaround for iOS.

Update 10/14 16:10 CDT: We have been notified by the 3rd party DNS filtering company that they have investigated our domain and found that it never had any malware, and have removed our domain from their blacklist. We are working to confirm that Verizon customers can now reach our domain.

Update 10/14 20:32 CDT: We have confirmed that Verizon Wireless customers are now able to reach our pushover.net domain.

Outage report for August 31st, 2019

On the morning of August 31st, we were notified by our network monitoring systems that all Pushover services became unavailable at 7:22 CDT. This included our API, website, desktop client, and dispatch of any queued messages. A message announcing the outage was posted to our Twitter account @PushoverApp.

We were notified soon after by our server/network provider that there was a datacenter-wide power outage affecting all customers of the datacenter. Unfortunately an ETA was not available from the provider during this outage.

At 9:29 CDT, power was restored to the datacenter, but the abrupt shutdown of our servers left their database replication in a failed state. This was resolved and all Pushover services were made available by 9:59 CDT.

While Pushover’s infrastructure is largely redundant and can usually survive a partial outage without affecting users, this large-scale outage affected all of our servers at once. We will be looking into what measures can be taken to further enhance our redundancy to avoid future outages of this type.

For more updates, we encourage users to follow our Twitter account @PushoverApp.