I just got an email from Ahmed at Zenprise about the BES outages today in Europe. Today, a Zenprise customer in the UK got an alert from Zenprise at 13:32 that the RIM SRP network went down. That network looked to be back up and running around 15:00. One of Zenprise’s US customers supports users in Europe and received an alert at roughly the same time, but service for them was restored at 14:13.
Zenprise helps BES administrators cut down support tickets by using automation. This system was well demonstrated today.
One of the automated diagnostics that their product runs before triggering an alert is to run a network protocol to port 3101 to test RIM connectivity (that’s the port the BES server talks to the RIM network on). It looks like one of the advertised IP addresses of the RIM network went down, and the traffic was rerouted to the secondary IP address. The propagation of the DNS changes may have taken some time, which is why some customers saw service restore faster than others.
Thanks again to Zenprise for the information.