This last week was an intense week of face-to-face discussions talking to some really smart people who are working on innovative solutions. For example, my meeting with Smooth-Stone was quick and compressed, running at speeds of idea transfer that builds future relationships of information exchange. When I blogged about meeting Smooth-Stone there is nothing I really wrote about that isn’t public information other than I now know 4 Smooth-Stone executives.
I keep the GreenM3 blog constrained to either my own thoughts or public accessible information, always asking people for permission to blog about specific areas. This way I can have intense knowledge sharing conversations and people know I won’t blog about conversations without their permission. As one executive who I met on the phone last week, then met in person twice, and have taken him on wild ride of ideas introducing him to people who can help him execute green solutions said, “you know 95% of what you talk about, people can’t follow and you lose them.”
Part of the reason why I write this blog is to slow down, simplify ideas, discuss publicly accessible concepts - thinking about when is the right time to discuss ideas with a broader audience. One of my long time technology friends who I am glad to start discussing his ideas is John Farmer now that he has his own blog at http://farmhead.blogspot.com/.
Here is a bit of background on John.
I've always been fascinated by machinery, whether software, hardware, or large organizations. I'm currently an Engineering Director at Adobe Systems and work on SaaS-related technology.
Interests
John’s interest and I overlap a lot, including both having black belts which is not relevant for physical fighting, but more as we have gotten older in how you fight battles and win vs. the competition in organizations and the industry.
John’s recent posts on Incident Management reminds me of a green approach to the problem. How do you be the most efficient and effective in resolving the problem?
Tips for Handling Events, Incidents, Outages, and Maintenance
I get a lot of questions from new service teams about what they should do to prevent downtime but very few people ask for advice on how to handle an incident. This is a bit like asking a boxer for the best way to avoid getting in the ring. It’s not a question of “if” you’re going to be in the ring but “when”. There’s an old saying – the more you bleed in the gym, the less you bleed in the ring and that definitely applies to incident management as well.
John has taken the time to write three posts on Incident Management.
Having sat in on more war rooms than I’d like to remember, I thought it might be handy to write down some of the things that my team has found useful over the years. I think every service organization should have a standard approach towards three specific activities:
1. Tips for Handling Service Incidents (just one service)
2. Tips for Handling Service Outages (multiple services affected)
3. Tips for Handling System Maintenance
Here is one my favorite Tips.
Get your head straight
First, stay calm. The worst thing you could do is cause a major outage, destroy some data, or make the existing problem worse in a panic. Simple problems can easily become large complicated problems after a few bad decisions made in haste. Take a breath before continuing. This is especially important with a page at 3AM or if a panicky client is in your office. Tell the client you’ll handle the problem and run through your normal procedure.
John closes with good advice that is grounded in years of martial arts practice and ways to handle the stress of combat.
I hope these posts help you with your handling of incidents, outages, and maintenance. Success here is mostly about being prepared, being calm, good communication, and practice, practice, practice. If you think your service is bullet-proof and you won’t need the practice – you’re wrong :-)
I’ll be reading John’s blog post on a regular basis, and referencing posts that I think are relevant to a green data center approach. On my last trip, I was able to squeeze a 1/2 hour meeting with John before I flew from SJC to SEA.
In the airport, I was able to shake hands with three of the Smooth-Stone executives I met 8 hrs earlier and were flying back to Austin. The Smooth-Stone CEO was on the flight to SEA, and we were able to discuss more ideas when we landed and I hitched a ride instead of taking the bus back to Redmond. This last week was intensely interconnected.