James Hamilton has a good post on the next point of server differentiation being efficiency at very high temperature.

Next Point of Server Differentiation: Effiiciency at Very High Temprature

High data center temperatures is the next frontier for server competition (see pages 16 through 22 of my Data Center Efficiency Best Practices talk:http://mvdirona.com/jrh/TalksAndPapers/JamesHamilton_Google2009.pdf and 32C (90F) in the Data Center). At higher temperatures the difference between good and sloppy mechanical designs are much more pronounced and need to be a purchasing criteria.

The infrastructure efficiency gains of running at higher temperatures are obvious. In a typical data center 1/3 of the power arriving at the property line is consumed by cooling systems. Large operational expenses can be avoided by raising the temperature set point. In most climates raising data center set points to the 95F range will allow a facility to move to a pure air-side economizer configuration eliminating 10% to 15% of the overall capital expense with the later number being the more typical.

James gives 3 downsides to higher temperature.

These savings are substantial and exciting. But, there are potential downsides: 1) increased server mortality, 2) higher semi-conductor leakage current at higher temperatures, 3) increased air movement costs driven by higher fan speeds at higher temperatures. The former, increased server mortality, has very little data behind it. I’ve seen some studies that confirm higher failure rates at higher temperature and I’ve seen some that actually show the opposite. For all servers there clearly is some maximum temperature beyond which failure rates will increase rapidly. What’s unclear is what that temperature point actually is.

In my early career at HP I worked as a reliability engineer and stress tested equipment in extreme cold and heat, analyzing failures. This problem reminds me also of one of the lessons I learned working in distribution logistics at HP and Apple, it is cost prohibitive to design the 99.9999% packaging to ship things, and you need to strike the right balance dependent on what you are shipping and its value.

Intel, AMD, and disk driver vendors will discus their energy efficiency, but just like packaging design, thermal efficiency is not sexy and what people think about for energy efficiency.

The complexity of this is huge.

We also know that the knee of the curve where failures start to get more common is heavily influenced by the server components chosen and the mechanical design. Designs that cool more effectively, will operate without negative impact at higher temperatures. We could try to understand all details of each server and try to build a failure prediction model for different temperatures but this task is complicated by the diversity of servers and components and the near complete lack of data at higher temperatures.

And, here is where James totally gets it.  He says we need models.

We also know that the knee of the curve where failures start to get more common is heavily influenced by the server components chosen and the mechanical design. Designs that cool more effectively, will operate without negative impact at higher temperatures. We could try to understand all details of each server and try to build a failure prediction model for different temperatures but this task is complicated by the diversity of servers and components and the near complete lack of data at higher temperatures.

So, not being able to build a model, I chose to lean on a different technique that I’ve come to prefer: incent the server OEMs to produce the models themselves. If we ask the server OEMs to warrant the equipment at the planned operating temperature, we’re giving the modeling problem to the folks that have both the knowledge and the skills to model the problem faithfully and, much more importantly, they have ability to change designs if they aren’t fairing well in the field. The technique of transferring the problem to the party most capable of solving it and financially incenting them to solve it will bring success.

My belief is that this approach of transferring the risk, failure modeling, and field result tracking to the server vendor will control point 1 above (increased server mortality rate). We also know that the Telecom world has been operating at 40C (104F) for years (see NEBS)so clearly equipment can be designed to operate correctly at these temperatures and last longer than current servers are used. This issue looks manageable.

How do you solve this problem?

One smart guy dev guy, Ade Miller.had a good answer which I hope he’ll blog about soon is calculating PUE for a desktop and server. 

So is it more like a equipment PUE vs. data center PUE.

If server vendors started to publish their equipment PUE. What is the IT load of the motherboard, what is the overhead for the power supply and fans?  Would we be looking to buy the best PUE servers?

For you who get power supplies, fan, and analog devices, this will make a lot of sense. Oh yeh, I also was program manager on the Macintosh II power supplies, and learned a lot from a great development team.

John Boyd is an innovator in fighter pilot techniques. I found some of his work based on looking for control theory and modeling.

Fast Company has an article about John Boyd’s work that is shorter than the book about his life which I am also reading.

Why is this interesting?

It's all about rapid assessment and adaptation to a complex and rapidly changing environment that you can't control.

This is the challenge to building a green data center.  The issues are constantly changing, and there is much beyond your control.  What you can do is assess the situation and adapt.

The model helps unify the approach.  An example used is Toyota’s organizational approach.

Bower and Hout's classic example -- and one that Boyd also studied -- was Toyota, which designed its organization to speed information, decisions, and materials through four interrelated cycles: product development, ordering, plant scheduling, and production. Self-organized, multifunctional teams at Toyota, they observed, developed products and manufacturing processes in response to demand, turning out new models in just three years compared with Detroit's cycle of four or five.

Systems like Toyota's worked so well, Boyd argued, because of schwerpunkt, a German term meaning organizational focus. Schwerpunkt, Boyd wrote, "represents a unifying medium that provides a directed way to tie initiative of many subordinate actions with superior intent as a basis to diminish friction and compress time." That is, employees decide and act locally, but they are guided by a keen understanding of the bigger picture.

Good models allow you to see things you didn’t know where there.

A modeling approach has the potential to create a convergence of ideas  to drive new innovation, and better education of the team.

John Boyd achieved this in fighter pilot techniques.

Boyd theorized that large organizations such as corporations, governments, or militaries possessed a hierarchy of OODA loops at tactical, grand-tactical (operational art), and strategic levels. In addition, he stated that most effective organizations have a highly decentralized chain of command that utilizes objective-driven orders, or directive control, rather than method-driven orders in order to harness the mental capacity and creative abilities of individual commanders at each level. In 2003, this power to the edge concept took the form of a DOD publication "Power to the Edge: Command...Control...in the Information Age" by Dr. David S. Alberts and Richard E. Hayes. Boyd argued that such a structure creates a flexible "organic whole" that is quicker to adapt to rapidly changing situations. He noted, however, that any such highly decentralized organization would necessitate a high degree of mutual trust and a common outlook that came from prior shared experiences. Headquarters needs to know that the troops are perfectly capable of forming a good plan for taking a specific objective, and the troops need to know that Headquarters does not direct them to achieve certain objectives without good reason.

Trust is a key ingredient to these ideas.  Modeling can be used to promote trust.

Note: OODA loop is described here.