RIM Co-CEOs, Jim Balsillie and Mike Lazaridis, Talk About Outage

13 Comments

Today at 10AM EST, RIM’s Co-CEOs, Jim Balsillie and Mike Lazaridis, as well as David Yach, the CTO of RIM, were on a call to talk about the recent BlackBerry outage and what happened. It’s important to note that RIM has had great track record with uptime and while this particular outage has been much more serious than any other we can remember, the real problem was the timing. After all the punishment RIM has taken in the media, this outage couldn’t have come at a worse time. More about the conference call after the jump.

Mike Lazaridis started off with an apology and said they’ll be sorting out the issues soon and begin on restoring confidence with customers. You can watch the above video of Mike Lazaridis giving an apology and talking about the issue. Mike also took the time to explain what caused the outage and it was described as follows:

On Monday, a dual-redundant, high capacity core switch failed. This caused failures in the EMEA. This EMEA failure caused a cascade-failure in the system and a backup switch didn’t work. This meant that there was an incredible amount of data and RIM has been working to “drain the pool of data” so to speak. The reason for the spread to Latin America is that much of the infrastructure there is serviced by EMEA infrastructure.

Now, the focus is:

1. Working with vendors to prevent future failures.
2. Auditing the infrastructure to better understand why it took so long to bring the network back online.
3. Working to restore trust.

The question and answer session talked about reparations but it’s a little too early to talk about that. There’s also software license agreements with BES users and RIM will be dealing with those customers moving forward.

  • http://caspan.com Caspan

    What I love is people that bitch and bitch and bitch about something they pay $5ish a month to RIM for. then they want a reimbursement for the outage. well they were down for 3 days. So here is some simple math:

    3 days divided by 31 days in the month gets you 9.6% of the month there was an outage. So now take your $5 you paid and figure out what you paid for that you didn’t get….. $0.48 so you lost out on a whole 50 cents and people are asking for $50 credits to phones and crap. I get that its the principle and not the money but even myself as an IT professional that uses my BlackBerry day in and day out for my job had no issues dealing with the outage. 

    Put things in perspective people and relax we all survived not being able to BBM and needing to check web mail to see if anyone tagged you in a Facebook photo!

  • http://caspan.com Caspan

    What I love is people that bitch and bitch and bitch about something they pay $5ish a month to RIM for. then they want a reimbursement for the outage. well they were down for 3 days. So here is some simple math:

    3 days divided by 31 days in the month gets you 9.6% of the month there was an outage. So now take your $5 you paid and figure out what you paid for that you didn’t get….. $0.48 so you lost out on a whole 50 cents and people are asking for $50 credits to phones and crap. I get that its the principle and not the money but even myself as an IT professional that uses my BlackBerry day in and day out for my job had no issues dealing with the outage. 

    Put things in perspective people and relax we all survived not being able to BBM and needing to check web mail to see if anyone tagged you in a Facebook photo!

  • http://about.me/kylemcinnes Kyle McInnes

    I barely noticed the outage. Sure, I like using BBM but when I notice the “delivered” status not appearing I know something is up. So I SMS or go to a laptop/desktop to email. No big deal. I’m sure it was a pain for some people but overall it’s probably just really bad timing and the media is looking for more reasons to say “RIP RIM”. 

  • http://caspan.com Caspan

    Exactly not a great time to have it happen, what I have learned is that the media has no knowledge, smart people don’t write for papers, good writes write for papers and they find other people that are smart to build their story on. When it comes to RIM instead of talking to someone smart and in the know they just rebroadcast what everyone else is feeling.

  • http://twitter.com/mProductive mProductive™

    Hear, hear…there seems to be an amazing disconnect between expectation and the harsh but simple truths about life. Frustrating Yes but considering other problems in the world I would offer these simple words of encouragement and perspective, oh and they’re free ‘Suck it up princess’ …ok, now I’ll go have a little lie down.

  • http://twitter.com/mProductive mProductive™

    Hear, hear…there seems to be an amazing disconnect between expectation and the harsh but simple truths about life. Frustrating Yes but considering other problems in the world I would offer these simple words of encouragement and perspective, oh and they’re free ‘Suck it up princess’ …ok, now I’ll go have a little lie down.

  • http://twitter.com/mProductive mProductive™

    Hear, hear…there seems to be an amazing disconnect between expectation and the harsh but simple truths about life. Frustrating Yes but considering other problems in the world I would offer these simple words of encouragement and perspective, oh and they’re free ‘Suck it up princess’ …ok, now I’ll go have a little lie down.

  • http://caspan.com Caspan

    LoL nice….

  • http://twitter.com/Hardened Hardened

    There is a LOT of arrogance between die-hards on the net. 

    What many fail to realize is beyond the consumer there are corporations that CANNOT active BB’s for new hires, or migration for existing users that maybe travelling that same day an activation is required (travelling in MANY cases overseas). Some will require critical emails on the device. Webmail is used to ensure that Exchange completed the activation – yet no laptop for the corporate office where the user starts at …. fills out papers and orientation with HR then is on a flight that same day (8hrs later) to overseas sights! 

    Today … for the fist time in 12yrs while troubleshooting BB I could NOT complete even an HRT on a HUP for a taxation client – we’re full swing into taxation prep for the quarter. I felt incredibly embarrased to give him a 9900 for calls only with all the previous data from his older 9700. To make matters worse is I had thought since he’s had his SIM for 3yrs that simply switching the SIM on Rogers would resolve. NOT! Switching the newly activated SIM to my 9700 worked for PIN-2-PIN and Corp Emails – in the BNIB opened 9900 nadda. Wiped  tested FAILED> Error “Connected to RIM: Abort!” seriously!??!!! Rogers …. could not find the PIN yet when entered received “Invalid PIN” this is a carrier provided 9900 (Rogers) from our authorized vendor. Case opened with Rogers and RIM. I believe this NOC issue has MORE issues than what RIM CEO’s and CIO are letting on – I’m not the ONLY one experiencing this EXACT issue!!

    NOC is …. sorry WAS RIM’s greatest strength … its now their achilles heel. Here in Canada we don’t pay $5/mth for data like some punk-ass puny …. we pay $35-75/mth on corporate and even avg $45/mth for personal services … for 500MB/data. These are FACTS: Rogers pricing quoted!

    The lame argument of push vs “real-push” is a joke from 4yrs and beyond … push is push as long as it works in real-time and can be actioned upon. 

    MANY die-hards assume those on the net complaining are 14yr olds missing their FB/Twitter updates or consumer emails. EVEN if they where the majority … so WHAT?! They have a RIGHT to complain … their locked into a carrier contract enforced and supported by RIM. RIM has always advertised immediacy of internet information and made a killing doing so … staked a name in the market and in history doing so. When the string is pulled, especially for the youths today that grew up on internet immediacy it really hits home. Youths today care more for their mobile smartphones than many of us adults do and very much on a personal level. 

    Many adults use this to … meet their kids for pickup to travel home, for emergency and safety communications – think of how many girls are sexually assaulted per day and recovering each week all paranoid and relying on security of immediate communication?!!! Think of how many corporations require BB for emergency communications: BNSF saved the life of a railway worker 5yrs ago when his leg was cut-off by a train and he was able to BBM corp contacts a quick auto-text phrase in less than 30seconds, call his wife for 1min before passing out. 911 crew arrived in less than 10mins. He lost his leg … he has the company of friends and family still and his children STILL have father!

    THINK beyond your own arrogance about how a global communications outtage affects others. Please!

  • http://caspan.com Caspan

    Okay I will try to answer half of what you just wrote, why only half? Because you could not put a sentence together or a thought, you kind of threw up your thoughts on your message.

    You said “What many fail to realize is beyond the consumer there are corporations that CANNOT active BB’s for new hires”
    Okay what is so important for a brand new hire that they cannot live without email on a mobile device for 3 days? They are brand new employee how busy can they possibly be?

    You said “or migration for existing users that maybe travelling that same day an activation is required (travelling in MANY cases overseas)”
    Why are you trying to migrate a user while they are traveling? Are you asking for problems? Is it so important that person gets a new phone in the middle of an outage?

    You said “Some will require critical emails on the device.“

    If they are so critical then why has your company policy setup in a way that if a user does not have 1 device working that their entire job comes to a halt? That is bad IT and business practices; if you have put all your eggs in one basket then whose fault is it when the handle breaks? The basket maker or the person that put all their eggs in 1 basket and their career depend on those eggs? I think you have to not blame RIM for this but yourself and your practices

    You said “Webmail is used to ensure that Exchange completed the activation”
    No web mail is so that users can access their email from any web device in the world! They can use any computer to log in and check for critical emails you mentioned before this….

    You said “yet no laptop for the corporate office where the user starts at …. fills out papers and orientation with HR then is on a flight that same day (8hrs later) to overseas sights!”

    Again who runs this company that they hire people 2 hours before they need them? I get it happens that last minute hires happen but poor planning on their part does not constitute an emergency on your part.

    You said “Today … for the fist time in 12yrs while troubleshooting BB I could NOT complete even an HRT on a HUP for a taxation client – we’re full swing into taxation prep for the quarter”

    I have no idea what you are talking about here sorry

    You said “I felt incredibly embarrased to give him a 9900 for calls only with all the previous data from his older 9700”

    Why did you feel embarrassed? Did you cause the outage? Crap happens they will live for 3 days, they will trust me!

    You said “To make matters worse is I had thought since he’s had his SIM for 3yrs that simply switching the SIM on Rogers would resolve. NOT! Switching the newly activated SIM to my 9700 worked for PIN-2-PIN and Corp Emails – in the BNIB opened 9900 nadda. Wiped tested FAILED> Error “Connected to RIM: Abort!” seriously!??!!! Rogers …. could not find the PIN yet when entered received “Invalid PIN” this is a carrier provided 9900 (Rogers) from our authorized vendor. Case opened with Rogers and RIM. I believe this NOC issue has MORE issues than what RIM CEO’s and CIO are letting on – I’m not the ONLY one experiencing this EXACT issue!!”

    I have no clue what your problem is here this was one of those brain pukes I was talking about earlier.

    You said “NOC is …. sorry WAS RIM’s greatest strength … its now their achilles heel. Here in Canada we don’t pay $5/mth for data like some punk-ass puny …. we pay $35-75/mth on corporate and even avg $45/mth for personal services … for 500MB/data. These are FACTS: Rogers pricing quoted!“

    Yes you are correct I am also on Rogers and we pay $45 a month for 1 GB BES package. That whole $45 is not just money for RIM, only about about $5-$10 of that goes to RIM, so let’s not play with numbers here to make a point. IF you had an iPhone you would be paying that much for data as well but it would be a 6GB package for the same price because they are data hogs.

    You said “The lame argument of push vs “real-push” is a joke from 4yrs and beyond … push is push as long as it works in real-time and can be actioned upon”

    I never said anything about push email sorry wrong conversation”

    You said: “MANY die-hards assume those on the net complaining are 14yr olds missing their FB/Twitter updates or consumer emails. EVEN if they where the majority … so WHAT?! They have a RIGHT to complain … their locked into a carrier contract enforced and supported by RIM. RIM has always advertised immediacy of internet information and made a killing doing so … staked a name in the market and in history doing so. When the string is pulled, especially for the youths today that grew up on internet immediacy it really hits home. Youths today care more for their mobile smartphones than many of us adults do and very much on a personal level. “

    So give them a ball and tell them to go and play outside, they will survive. They are not locked into a contract they didn’t have a choice to read and agree to. You make it seem like it their contract was forced on them, they wanted a cheap phone so decided to lock into a 3 year agreement. So because they are unintelligent they get to complain? No shut up and reap what you sow, no one forced their hand they decided. If you take RIMs outage and take it over your before mentioned 12 years how much uptime is that? Like 99.999 that’s 5 9s of uptime, tell me who else has that out there? No one!

    You said “Many adults use this to … meet their kids for pickup to travel home, for emergency and safety communications -”

    Use the damn phone and pick it up and call them. Are you lazy or a bad parent? you act like BBM is the sole method of communication for the entire planet, like there are no other alternatives. It’s like some girl tried to BBM her mom to be picked up and she does not see it go through for 5 minutes so she just sits there for a month. No she calls here mom and says “hey mom tried to BBM you but it didn’t go through, can you pick me up now?” that’s how we can get a message to one another we talk… you know that ancient method of communication….

    You said “think of how many girls are sexually assaulted per day and recovering each week all paranoid and relying on security of immediate communication?!!!”

    So are you telling me hundreds or even one girl got rapped because BBM was down? Are you fing kidding me you are one of those people that use horrible thought situations to make a simple thing seem like 9/11 could happen again? They have a freaking phone in their hand how instant of a communication tool do you want. Do you think they have time to BBM someone they are about to get sexually assaulted!!!!!!! And they wait for an answer without any other order of recourse?

    You said “Think of how many corporations require BB for emergency communications: BNSF saved the life of a railway worker 5yrs ago when his leg was cut-off by a train and he was able to BBM corp contacts a quick auto-text phrase in less than 30seconds, call his wife for 1min before passing out. 911 crew arrived in less than 10mins. He lost his leg … he has the company of friends and family still and his children STILL have father!”

    They have a father because he was smart and called someone! BBM had nothing to do with him surviving. If he was dumb enough to have a phone in his hand and his first thought was to BBM someone then there is something wrong with the world. He has in his had a phone that will put him in direct communication with 911 when in 10 seconds he can say what would take him 30 seconds to type

    You said “THINK beyond your own arrogance about how a global communications outtage affects others. Please!”

    I am starting to think I am the only rational person left on earth after this message. Every single one of your points had 0 merit I am not trying to be an ass hole here but I will defend my opinion that if I hear people bitching, I will say exactly what someone else sad on here “suck it up princess”

    Yes it sucked it was down but life goes on……. And tell your company they need an new BES administrator, HR and corporate IT policy because they FAILED not RIM.

  • http://twitter.com/Hardened Hardened

    Hi Caspan … I’ll try to be brief and hope you’re able to track my reply in relevance. 

    1 I work for a gold mining company … we’re global. Many exchange servers are not global but based to cover regions as per best practices and design. Access control in AD is a bit split for where the user will be based out of. I can create accounts in various regions, however its best another IT rep in another location – where a new user will be based – creates the account, prepares imaging a laptop while I create the mailbox in my location (since our BES is centralized and reaches the mailbox in under 600ms). Using webmail “I” can logon to the users account, thus allowing BES to verify the account for BB Wireless Activation (or web desktop activations). 

    Many times new employees are to have an orientation at the head office then off overseas to their home office. This is where their laptop will be prepped, and due to non-worldwide support for head office PC hardware (stupid IT decision employed by a director I’ll completely agree), the user leaves with luggage and BB – unless roaming is not part of inter-carrier agreement and thus a user will arrive at destination to be picked-up to be taken to the office. If a BB is with the user then the user will arrive at the airport and arrange transportation (calls or PIN or BB). The type of business this is not ALL airports are 4 season hotel equipped and have limo’s waiting. Many of these new hires ARE very important and will be taking their responsibility role almost immediately with a liaison during transition – some are translators. Emails are very critical, and have a better affect of related reference with others in TO/CC to a specified person vs 1 call to another’s phone and the other is not ready. Many times you cannot phone on a plane, but the email will be queued and sent within 10mins of leaving the plane/airport. For those users that are travelling with a BB email communication has more effectiveness to book transportation in advance when a call to 1 person cannot be live and a Voicemessage is left. An email can reach multiple people – live or in a queue – where each can collaborate amongst each other and meet a targeted deadline or arrival while the traveller cannot be reached nor communicate during that flight. 

    definitions: “Today … for the fist time in 12yrs while troubleshooting BB I could NOT complete even an HRT on a HUP for a taxation client – we’re full swing into taxation prep for the quarter”
    HRT = Host Routing Table (automatically engaged upon seating battery or done manually). 
    HUP = Hardware UPgrade – you’re on Rogers so I figured you’d hear this a few times over the years.

    My “brain fart” above was that while troubleshooting I needed to contact Rogers tier 2 blackberry support. They can logon to a RIM portal for their carrier (Rogers) and search for valid BB PINs … this brand new unit didn’t exist for it to allow for a BES activation, let alone a BIS activation. The device simply didn’t exist – something I’ve never come across and is different than a device being reported lost/stolen. I’m pretty sure this is related to RIM’s NOC router outrage. 

    I don’t believe that you really understand nor or open enough to understand that not ALL corporations in various industries operate the same way; nor should they. 

    Regarding Outages … RIM has had many unplanned in the last 12 years … many of the posters and BB users know this. The uptime is great but its direction in support has taken a nose dive over the last few months. I can name 1 comparison that has faired better, actually 2: Domino and Exchange as an infrastructure they both have better uptime than RIM since RIM requires SQL Server to connect to either. During my 12yrs working as support – RIM allowed an SRP to be duplicated (shocking I know and if I dig hard enough I can find the RIM ticket and the SRP if I really wanted to). RIM is not as flawless as you claim it to be … I’ve had the pleasure to work in varying industries (mostly financial) to see the variances of issues. 

    HR does a great job, along with IT … this issue has nothing to do with them. the failure I mentioned is completely in context and tone and the break was at RIM. The troubleshooting was done, whether I dumped my thoughts without structure (long day: 15hrs) or just hard to read, it was done and in full. I see nobody arguing my troubleshooting. Thus its not IT failing. 

    Lastly, Users usually drop, damage or take their BB for an occasional swim or a device is lost/stolen. Users are not static for their career in 1 region only. Many will travel back to another office or the head office and a BB can easily be prepared for their arrival – nothing wrong or hard to understand or infer there. users are by nature mobile and an infrastructure should support that for mobile devices. 

    Yes I get a phone call can be made. Kids should be using calls more, along with their parents yet MANY schools in Canada disallow kids to use their phones in class or have their phones ring in class. The benefit for a BB is to have different vibration notifications differentiated from PIN/BBM vs Email/Calls – making it easy for a student to quickly leave class check something important or know that its something to check just after last class and meet at front, the nearest mall or take the public transit to the doctors office and be picked up there later. Again think outside what you understand to the be norm. 1 way alone doesn’t suit all. I agree maybe I need to bend my thinking as well; maybe a lot more here considering the attendees. We’ve agreed on many posts on this and a few other blogs over the past year so I know we’re both experienced in the field of BB’s. 

    The main point is that this outage couldn’t have happened at the worse time for RIM and it’s really important enough to RIM for Lazaridis to post a personal video message – something neither CEO has ever done in past outages. I recall the last major outage lasting 3days if I’m correct and a press statement was given. This was hard hit and important enough for RIM to address it with a top tier corporate face. That is key – offering applications for $100 or whatever value may appease most but it only helps consumers mostly; some corporate users are restrictive of which apps users can install based on industry and as such many of those free apps the end user may value cannot be partaken by them. 

  • http://caspan.com Caspan

    Maybe I am a hard ass Hardened but sometimes I feel like we whine too much you know.. people just want to blame someone else because their job got harder. I was born raised on a farm where there was you your fault his fault it was all your fault if you didn’t get it done! You had to survive for yourself and no one was there to hold your hand when things got tuff. If something happened then you had to figure it out or get you would go broke not planting a crop or feed the cows or anything. If a job sucked you couldn’t just hope someone else would do it you did it. I just think that so many people think now a days that they are just always owed something and how quick it is to say “wasn’t my fault” when you really dig down it was to some extent.

    I get that not all business work the same but who really is at fault here? As I read your response I here you being mad at RIM because you or your business has made bad decision that if 1 piece of technology fails your company comes to a standstill. That is not good business practice, and was my whole point, and the more you describe it the more I can tell you exactly the problem. Yes RIM went down which caused the problem but your origination has no plan B, it is not a hard thing to realize that when you rely on a 3rd party provider for a service and it is mission critical to your business that you have a plan b. That is like needing a mission critical server for the company and it goes down because 1 hard drive fails. Yes you could blame the hard drive manufacturer and you could even prove it was their fault 100%, that does not negate the fact that you know, without this being up it would cause disruptions to your entire business. SO you as a smart IT person tell management “hey I need a spare couple of drives so I can RAID this machine” this way if there is a failure we can keep limping along till we replace the failed drive. What it sounds like to me is you are using RIM as a scape goat for not having a good fail over plan in case of emergency or outage. This is just business 101 if you rely on a 3rd party vendor that is critical to your day to day function!

    As for your mention of Domino and Exchange these are products not services, so their uptime is dependent on you not Microsoft or Lotus

    If there are rules that prevent kids from using their phones in the class room you think that BBMing is then okay? They should be learning not texting, the only reason a kids need to use a phone at school is for contact and that can be made after classes at lunch or after school. If it is an emergency then call the school. Again your point here is nonsense and before you say I just don’t understand I do I have 2 kids in school to whom I tell leave your phone in your locker, or school bag turned off, school is not for texting it’s for learning.

  • Maxmz02

    Don’t blame CEO, they want RIM win.
     
    RIM has strange culture and self distruct political environment.
     
    In RIM if a new hired person figure out major problem and introduce efficient approach, both manager and his buddy group member will proof their wrong approach works. just like someone point out driving a car is right way, pushing a car is wrong way, then both manager and his buddy group member will hate you, and proof that 3 person can also move the car by pushing it. cheating email will be sent to some vice president, saying like: see, the car moving, pushing a car is a natural part of the process, in order to deny new hired contribution of introducing skill of drive a car, they have to deny merit of driving a car.
     
    It is very strange company culture and strange company political environment, it promote stealing and cheating skill. RIM’s management may be a typical instance in MBA course.
     
    This culture deny or steal hardworking team members’ contribution/innovation, generate strange political environment, destroy RIM.
     
    So don’t blame CEO, some of their VPs and VPs’ expert generate terrible culture and self destruct political environment.