In spite of all the effort spent on disaster recovery and redundancy it is amazing how fragile IT systems can be. The latest disaster is T-Moblie sidekick, built on the Microsoft acquired Danger client/server platform.
If T-Mobile was smart they’d offer free account transfers to Google Android or RIM smartphones for anyone who wants to dump their sidekick device. How you handle the outages to survive is the sign of maturity. One example of handling a crisis situation is Johnson and Johnson’s Tylenol crisis.
"The PR industry has an important role to play in helping companies identify and manage risks that could damage their reputation." Nick Purdom of PR Week
THE TYLENOL CRISIS, 1982
Johnson and Johnson survived based on its credo defined in 1943. We’ll see if T-Mobile, Sidekick or Microsoft Danger will survive.
The Danger platform is a client/server model.
A powerful client-server architecture
IP-based communications allow you to develop powerful web
services, real-time information, and networked applications.
Guaranteed delivery of data
Powerful HTTP library
Device-to-device communications
Asynchronous network communications
REST, XML/RPC and SOAP
All application data is backed up to the Danger Service
Encryption and authentication are managed by Danger
It will be interesting to know if we will ever know the full story. One speculation was the data loss was caused by an attempted upgrade to the storage array without backing up first.
In the Danger case, it appears from initial speculation that the data was lost because they attempted to upgrade a storage array without backing it up first. Here is a case of smart and rational people who do this for a living at one of the best companies in the world, and they didn't even bother making a backup ¿ so what hope do we have? Relying on the cloud as a backup didn't work, because somebody forgot to backup the backup.
T-Mobile has started to manage the PR nightmare.
Sidekick customers, during this service disruption, please DO NOT remove your battery, reset your Sidekick, or allow it to lose power.
Updated: 10/10/2009 12:35 PM PDT
T-MOBILE AND MICROSOFT/DANGER STATUS UPDATE ON SIDEKICK DATA DISRUPTION
Dear valued T-Mobile Sidekick customers:
T-Mobile and the Sidekick data services provider, Danger, a subsidiary of Microsoft, are reaching out to express our apologies regarding the recent Sidekick data service disruption.
We appreciate your patience as Microsoft/Danger continues to work on maintaining platform stability, and restoring all services for our Sidekick customers.
Regrettably, based on Microsoft/Danger's latest recovery assessment of their systems, we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger. That said, our teams continue to work around-the-clock in hopes of discovering some way to recover this information. However, the likelihood of a successful outcome is extremely low. As such, we wanted to share this news with you and offer some tips and suggestions to help you rebuild your personal content. You can find these tips in our Sidekick Contacts FAQ. We encourage you to visit the Forums on a regular basis to access the latest updates as well as FAQs regarding this service disruption.
In addition, we plan to communicate with you on Monday (Oct. 12) the status of the remaining issues caused by the service disruption, including the data recovery efforts and the Download Catalog restoration which we are continuing to resolve. We also will communicate any additional tips or suggestions that may help in restoring your content.
We recognize the magnitude of this inconvenience. Our primary efforts have been focused on restoring our customers' personal content. We also are considering additional measures for those of you who have lost your content to help reinforce how valuable you are as a T-Mobile customer.
We continue to advise customers to NOT reset their device by removing the battery or letting their battery drain completely, as any personal content that currently resides on your device will be lost.
Once again, T-Mobile and Microsoft/Danger regret any and all inconvenience this matter has caused.