As the density and power usage in the datacenter continues to rise, so does the heat. When planning the design of the datacenter, it is important to understand the workload and the expected increase in power in order to effectively deploy cooling technologies. Without this, equipment and cooling are not properly aligned and money is wasted.
Hot aisles, cold aisles, containment, the correct management of racks and when to deploy HVAC are all key considerations in heat management. The biggest challenge in cooling the datacenter is designing a solution that is flexible and which doesn’t create its own hot spots that cannot be easily cooled.
Cannon Technologies has decades of experience in managing complex cooling issues within the datacenter and Mark Hirst, T4 product manager explains how to plan and deploy effective cooling.
The numbers tell the story
In late 2011, DatacenterDynamics released their annual energy usage survey for datacenters. It made for stark reading.
1. Global datacenter power usage in 2011 was 31GW. This is equivalent to the power consumption of several European nations.
2. 31GW is approximately 2% of global power usage in 2011.
3. Power requirements for 2012 was projected to grow by 19%.
4. 58% of racks consume 5kW, 28% consume 5-10kW and the rest consume more than 10kW per rack.
5. 40% of participants stated that increasing energy costs will have a strong impact on datacenter operations going forward.
If these numbers come as a shock, they should be considered against several other factors that will impact the cost of running datacenters.
1. Environmental or carbon taxes are on the increase and datacenters are seen as a prime target by regulators.
2. As a result of the Fukushima nuclear disaster, several European countries are planning on reducing and even eliminating nuclear as a power generation source. This will create a shortage in supply and drive up power costs.
3. Around 40% of all power usage in the datacenter is to remove heat and could be considered waste.
4. The move to cloud will only shift Capital Expenditure (CAPEX) out of the budget. Power is Operational Expenditure (OPEX) and will be added to the cost of using the cloud thus driving OPEX at a faster rate than CAPEX is likely to come down.
Design for cool
Removing heat effectively is all about the design of the cooling systems. There are several parts to an effective cooling system:
1. The design of the datacenter.
2. Choosing the right technology.
3. Effective use of in rack equipment to monitor heat and Computational Fluid Dynamics (CFD) to predict future problems.
A major part of any efficient design is the datacenter itself. The challenge is whether to build a new datacenter, refurbish existing premises or retrofit cooling solutions. Each has the ability to deliver savings with a new build and refurbishment likely to deliver the greatest savings. Retrofitting can also deliver significant savings on OPEX, especially if reliability is part of the calculation.
Building a new datacenter provides an opportunity to adopt the latest industry practices on cooling and take advantage of new approaches. Two of these approaches are free air cooling and splitting the datacenter into low, medium and high power rooms.
In 2010, HP opened a free air cooling datacenter in Wynyard, County Durham, UK. In 2011, HP claimed it had only run the chiller units for 15 days resulting in an unspecified saving on power.
Refurbishing an existing datacenter can deliver savings by allowing easy access to rerun power and change out all the existing cooling equipment.
In 2011 IBM undertook more than 200 energy-efficiency upgrades across its global datacenter estate. The types of upgrades included blocking cable and rack openings, rebalancing air flow and shutting down, upgrading and re-provisioning air flow from computer room air conditioning (CRAC) units. The bottom line was a reduction of energy use of more than 33,700MWh which translates into savings of approximately $3.8m.
Retrofitting a datacenter can be a tricky task. Moving equipment to create hot aisles and deploying containment equipment can have an immediate impact on costs.
Kingston University was experiencing problems in its datacenter. IT operations manager, Bill Lowe admits ‘As the University has grown, so too has the amount of equipment housed in its datacenter. As new racks and cabinets have been added, the amount of heat generated started to cause issues with reliability and we realised that the only way to deal with it was to install an effective cooling system. Using Cannon Technologies Aisle Cocoon solution means that we will make a return on investment in less than a year.’
Choosing the right technology
Reducing the heat in the datacenter is not just about adding cooling. It needs to start with the choice of the equipment in the racks, how the environment is managed and then what cooling is appropriate.
1. Select energy efficient hardware.
Power supplies inside servers and storage arrays should all be at least 85% efficient when under 50% load. This will reduce the heat generated and save on basic power consumption.
2. Raise input temperature.
ASHRAE guidelines suggest 27oC as a sustainable temperature which is acceptable to vendors without risking warranties. Successive generations of hardware are capable of running at higher input temperatures, so consider raising this beyond 27oC where possible.
3. Workload awareness.
Understand the workload running on the hardware. High energy workloads such as data analysis or high performance computing (HPC) will generate more heat than email servers and file and print operations. Mixed workloads cause heat fluctuations across a rack so balancing workload types will enable a consistent temperature to be maintained making it easier to remove excess heat.
4. Liquid cooling.
Liquid cooling includes any rack that uses water or any gas in its liquid state. Industry standards body ASHRAE has recently begun to talk openly about the benefits of liquid cooling for racks where very high levels of heat are generated. This can be very hard to retrofit to existing environments due to the problems of bringing the liquid to the rack.
5. Hot/cold aisle containment.
This is the traditional way to remove heat from the datacenter. Missing blanking plates allow hot air to filter back into the cold aisle reducing efficiency. Poorly fitted doors on racks and the containment zone allow hot and cold air to mix. Forced air brings other challenges such as missing and broken tiles allowing hot air into the floor while too much pressure, prevents air going up through the tiles.
6. Use of chimney vents
This can be easily retrofitted even in small environments. Using fans, the chimney pulls the hot air off the rack and vents it away reducing the need for additional cooling.
Computer Room Air Conditioning (CRAC) has been the dominant way of cooling datacenters for decades. It can be extremely efficient although that depends on where you locate the units and how you architect the datacenter in order to take advantage of the airflow.
One danger with poorly placed CRAC units, as identified by The Green Grid, is the problem of multiple CRAC units trying to control humidity if air is returned at different temperatures. The solution is to network the CRAC units and do coordinated humidity control.
Effective placement of CRAC units is a challenge. When placed at right angles to the equipment, their efficiency drops away over time causing hot spots and driving the need for secondary cooling.
In high density datacenters, ASHRAE and The Green Grid see Within Row Cooling (WIRC) cooling, to get the cooling right to the sources of the heat, as imperative. WIRC also allows for cooling to be ramped up and down to keep an even temperature across the hardware and balance cooling to workload.
If the problem is not multiple aisles, just a single row of racks, use open door containment with WIRC, or alternatively liquid based cooling. This is where the doors between racks are open, allowing air to flow across the racks but not back out into the aisle. Place the cooling units in the middle of the row and then graduate the equipment in the racks with the highest heat closest to the WIRC.
For blade servers and HPC, consider in rack cooling. This solution works best where there are workload optimisation tools that provide accurate data about increases in power load so that as the power load rises, the cooling can be increased synchronously.
New approaches to CRAC are extending its life and improving efficiency. Dell uses a sub floor pressure sensor to control how much air is delivered by the CRAC units. This is a very flexible and highly responsive way to deliver just the right amount of cold air and keep a balanced temperature.
Dell claims that it is also very power efficient. In tests, setting the subfloor pressure to zero effectively eliminated leaks and while it created a small increase in the power used by the server fans, it heavily reduced the power used by the CRAC units. Dell states that this was a 4:1 reduction. Dell has not yet delivered figures from the field to prove this saving but it does look promising.
8. Workload driven heat zoning
Low, medium and high power and heat zones allow cooling to be effectively targeted based on compatible workloads. An example of this is the BladeRoom System where the datacenter is partitioned up by density and power load.
Effective monitoring of the datacenter is critical. For many organisations, this is something that is split across multiple teams and that makes it hard to identify problems at source and deal with them early. When it comes to managing heat, early intervention is a major cost saving.
There are three elements here that need to be considered:
1. In rack monitoring.
2. Workload planning and monitoring.
3. Predictive technologies.
All three of these systems need to be properly integrated to reduce costs from cooling.
In Rack Monitoring
This should be done by sensors at multiple locations in the rack; front, back and at four different heights. It will provide a three dimensional view of input and output temperatures and quickly identify if heat layering or heat banding is occurring.
As well as heat sensors inside the rack, it is important to place sensors around the room where the equipment is located. This will show if there are any issues such as hot or cold spots occurring as a result of air leakage or where the air flow through the room has been compromised. This can often occur due to poor discipline in the datacenter where boxes are left on air tiles or where equipment has been moved without an understanding of the cooling flow inside the datacenter.
Most datacenter management suites, such as CannonGuard provide temperature sensors along with their CCTV, door security, fire alarm and other environmental monitoring.
Workload Planning and Monitoring
Integration of the workload planning and monitoring into the cooling management solutions should be a priority for all datacenter managers. The increasing use of automation and virtualisation means that workloads are often being moved around the datacenter to maximise the utilisation of hardware.
VMware, HP and Microsoft have all begun to import data from Data Center Information Management (DCIM) tools into their systems management products. Using the DCIM data to drive the automation systems will help balance cooling and workload.
Computational Fluid Dynamics (CFD) and heat maps provide a way of understanding where the heat is in a datacenter and what will happen when workloads increase and more heat is generated. By mapping the flow of air it is possible to see where cooling could be compromised under given conditions.
Companies such as Digital Reality Trust, use CFD not only in the design of a datacenter, but as part of the daily management tools. This allows them to see how heat is flowing through the datacenter and move hardware and workloads if required.
There is much that can be done to reduce the cost of cooling inside the datacenter. With power costs continuing to climb, those datacenters that reduce their power costs and are the most effective at taking heat out of the datacenter will enjoy a competitive advantage.