Once upon a time about twenty years ago, our then-current Internet Service Provider (ISP) was purchased. All servers & web domains were transferred from the previous provider to the new owners in a poorly executed “Big Bang” migration – everyone got moved over a single weekend.
Twenty days later, the smoke still hadn’t cleared. Our web site was crippled, and e-mail service was sporadic. Most of the customers acquired by the purchase either found a new provider or were looking by the third week.
This is the equivalent of buying an expensive ice sculpture and leaving it in the summer sun… the money invested melts away as your disillusioned customers rush toward the exits.
There are two interesting aspects to this fiasco.
- The technical debacle – A tale of arrogance and ignorance, trying to convert tens of thousands of domains all at once with what appeared to be little planning, prototyping, or testing; changing the help ticket application and moving the support telephone lines on the same day they migrated all customer domains. The outcome was sadly predictable: the migration created a huge smoking crater where customer web sites used to be. As customers realized they had problems and tried to contact the provider they discovered the help ticket and telephone systems were both offline. This exacerbated the second part of the disaster…
- Botched communication – Before and during the crisis.
I want to focus on the troubled communication aspects of this tragedy to see what can be learned and applied to YOUR next disaster to make it less disastrous.
The vendor notified customers via e-mail a month before the move that the migration was going to occur and encouraged users to back up their sites and data. They also cautioned users that any changes made after backups were taken on March 15th would be lost when systems were restored on April 15th.
Setting aside that a 30-day delay between backup and restore is criminally poor service in the 21st century, I received the e-mail warnings prior to the conversion only because my personal e-mail account is registered as our domain administrator. Several thousand other customers never got this message. Some because they only check their administrator e-mail when they are actively engaged in site administration. Others apparently have non-technical types monitor the admin accounts who received the message but didn’t understand its significance. Bottom line: many businesses lost 30 days of critical customer data – not just web pages, but databases with purchase and payment information.
Take away points:
- Keep people informed about what you plan to do
- Assure they are getting the message
- Check to see if they understand the message
At the time of the conversion, the vendor established a web page to communicate migration status. Good idea.
Unfortunately, the page was hard to find. Bad idea. People who identified problems days after the conversion and tried to enter trouble tickets discovered what looked like a “secret” site that seemed to be hiding the fact that many of their problems were already known and still unresolved.
The migration status web page was updated every few hours on the first day, then once per shift for a day or so, then once per day, then not at all. The last entry was April 22nd, a week after the conversion and two weeks before I wrote this essay. The first entry triumphantly proclaimed the success of the migration noting a few “minor issues”. Later updates acknowledged more serious problems, and asked customers to be patient. Still later entries chided users for some of the ways that they had coded web pages and applications, telling them “if they had done things ‘correctly’ many of the problems would not have occurred”. Starting and then stopping communication while problems persisted? Terrible Idea.
Take away points:
- Set up and publicize a central source of status information
- Make sure that status is easy to find and available to everyone who might care.
- Keep status current, even if you have no new information. Better a message every two hours saying “no change” than silence
- Establish a gatekeeper or editor for all broadcast communication that is responsible for assuring that content is balanced and not defensive. Explaining what is happening and why can be helpful. Blaming your victims for your mistakes? A truly AWFUL idea.
I logged five trouble tickets and never received a reply. On the rare occasions when the support phone number worked, I left voice messages, but I never received a call back. Five days after the migration, a notice was posted to the status page reporting that all problems were known and asking customers to please stop submitting trouble tickets.
Take away points:
- Positive acknowledgment of all incoming communication is essential. One reason for the flood of trouble tickets that overwhelmed the provider’s support staff was people reporting the same problem several times because they had not received a response. Customers were doubly frustrated because they felt their issues were not being heard or addressed.
- Precision is vital if you must broadcast a response. It is arrogant to say (and dismaying to hear), “We know about all problems”. Much better to say, “We are aware of problems X, Y and Z and will notify you when they have been resolved”
The best communication rule I’ve heard is: “The burden of communication lies with the party that has the most to lose.” Everyone had a lot to lose in a crisis situation. The ISP likely failed as a consequence of their poor migration (karma). Customers’ operations were disrupted and many lost business, data and money. Dozens of small operations were mortally wounded as a consequence and had to cease operation.
Clearly, better planning and testing could have prevented much of this disaster. Better communication might have mitigated some of the consequences. Informed customers might have been better prepared and more patient once they understood the nature of the problems and could see progress addressing them. Instead, most gave up in frustration.
Article by Payson Hall, PMP®, consulting project manager for Catalysis Group, Inc. in Sacramento. Payson is bringing a series of project management classes to Impact Foundry this fall, beginning with Project management for Nonprofits: Reading the Tea Leaves on August 15th.
