Click to enlarge the image
BoxTone have provided CIO with the details as to what happened during the BlackBerry outage. According to BoxTone, the problems began between 3PM and 4PM EST. BlackBerry Messenger and BIS internet browsing outages were quickly reported around the web.
Between 6:30 and 7:00 PM, the problem extended to BES email, preventing the delivery of emails for anyone on a BES. BoxTone began noticing a large increase in messages pending, and generated a warning alert before their customers flooded them with calls. BoxTone also placed all affected BES and Carriers in a Critical state on their customers’ Operations Dashboards (depicted by the red dots next to each BES and carrier). The steady growth in Pending Messages beginning around 6:45 continued until the issue was resolved early this morning. From their monitoring data, it appears that BES were able to communicate with the RIM NOC throughout the outage; however, the NOC was unable to deliver messages.
At about 12:09AM, BoxTone detected a brief disconnect in teh SRP connection of each BES to the NOC; it appears RIM reset the NOC SRP connection to complete their fixes. Following this reset, delivery of BES mail resumed.
By 2:45 AM or earlier, BoxTone detected that most customers’ email and services were back to normal and the backlog of pending emails had been delivered. BoxTone generated notifications informing their users that their service levels had returned to normal and updated the status of the BES and carriers to Normal.
Overall, this outage seems to have lasted around 7 hours. Thankfully, it wasn’t during peak hours, or this could have seriously impacted the working day. I hope RIM does a much better job of fixing these outages because 7 hours is way too long.