In my previous article, we have discussed the various cooling strategies such as room, row, rack & hybrid cooling. One of the question that you may have left behind is, which one is best for your scenario. While the answer is completely depending on your data center infrastructure designs. Let us compare room, row and rack based cooling to provides guidance on when to use each type for most next generation data centers.

Every data center air conditioning system has two key functions: to provide the bulk cooling capacity, and to distribute the air to the IT loads. The first function of providing bulk cooling capacity is the same for all cooling architectures, namely, that the bulk cooling capacity of the air conditioning system in kilowatts must exhaust the total power load (kW) of the IT equipment. The various technologies to provide this function are the same whether the cooling system is designed at the room, row, or rack level. The major difference between cooling architectures lies in how they perform the second critical function, distribution of air to the loads. Unlike power distribution, where flow is constrained to wires and clearly visible as part of the design, airflow is only crudely constrained by the room design and the actual air flow is not visible in implementation and varies considerably between different installations. Controlling the airflow is the main objective of the different cooling system design approaches.

To make effective decisions regarding the choice between room, row, or rack-based cooling for new data centers or upgrades, it is essential to relate the performance characteristics of the cooling methods to practical issues that affect the design and operation of real data centers. So let us compare these cooling methods against various criterias commonly identified by data center users such as;

• First Cost
• Electrical Efficiency
• Water piping or other piping near IT equipment
• Cooling unit location
• Redundancy
• Heat removal method
• Agility
• System availability
• Life Cycle cost(TCO)
• Serviceability

First Cost

Most data center managers are concerned with the first cost of different cooling methods. An analysis is done to show how the first cost varies for the three different chilled water cooling methods as a function of rack power densities. Here we can see an illustration of the results for a data center based on these assumptions.

Below picture shows a simulation of “first costs” (capital costs for cooling units, piping, chillers, installation and containment) associated with room-based, row-based, and rack-based cooling systems.  The simulation assumes 480 kW of total IT load and looks at first costs associated with different rack densities (3 kW to 20 kW).  First costs for all systems decline as rack density increases thanks to a shrinking data center footprint – e.g., fewer racks, raised floors, and piping are needed.  First costs for row-based cooling are slightly higher than for room-based, as more cooling units and more piping are needed. Rack-based first costs drop dramatically as server density increases (e.g., racks fill up with servers), but they are still higher than row and room-based cooling.

Room-based cooling has the lowest first cost because it has fewer cooling units and less piping. The cost decreases slightly as rack power density increases because, for the same data center capacity, the model assumes a smaller data center footprint as density increases. As a result, less raised floor and piping is required, hence the lower first cost. Note that the room-based electrical efficiency will be worse as the rack power density increases. HAC (hot aisle containment) increases rack power density for both cooling methods, and greatly reduces cooling system power consumption, although the first cost increases slightly due to the cost of containment. 

Row-based cooling has a slightly higher first cost than room-based because the row-based has relatively more cooling units and more piping. The cost decreases as the rack power density increases for the same reason as room-based cooling except that the number of cooling units will also decrease with increasing density.

 not only reduces row-based cooling power consumption, but also reduces the first cost as less cooling units are required. 

The first cost for rack-based cooling is quite higher than room-based and row-based cooling at lower rack power densities. This is because the increase in the number of cooling units which increases capital cost of units and piping for the lower densities. For example, for the 3 kW per rack scenario the row-based cooling has a total of 48 cooling units, but the rack-based cooling will increase that to 160 units. Also the rack-based cooling requires front and rear containment for the rack and cooling unit, which adds extra first cost to the system. As density increases, the first cost improves dramatically because the number of cooling units will be reduced in order to optimize the first cost. So, the rack-based cooling is more economical for high rack power densities.

Electrical Efficiency

Electrical costs are becoming a larger fraction of total operating costs, due to increasing electric rates, the increase in electrical power required by the servers, and the increase of power density. While the dependency of electrical costs on electric rates and server power is well understood, the affect of power density on electrical costs is not generally considered.

This illustration shows the effect of power density on annual electrical costs for three different chilled water cooling methods using the same assumptions. Below picture shows annual cooling costs (electricity costs) for the same simulation depicted in above picture. The simulation assumes 480 kW of total IT load and looks at first costs associated with different rack densities (3 kW to 20 kW).

The electrical costs for room-based cooling without HAC are highest because the room-based cooling needs to move more air over larger distances and the CRAH units need to consume power to stir or mix the air within the room to prevent hotspots. This electrical cost is reduced by using HAC due to the separation of airflows. As density increases, energy costs decrease slightly due to the shorter pipe lengths and the associated decrease in pump power consumption.

The electrical costs for a row-based cooling are consistently lower than room-based cooling because the CRAH units are closely coupled to the load, and sized to the load. Unnecessary airflow is avoided, which can save more than 50% fan power consumption compared with room-based cooling. The electrical cost will increase as rack power density increases, because the number of cooling units will be reduced, and more airflow and water flow is required for each cooling unit to achieve the required capacity to maintain the temperatures. The higher operating speed of the fan reduces the effective savings which can be achieved with variable speed fans. In this case, adding redundant units will actually lower the energy consumption but will result in higher first cost. In addition the higher water flow required to maintain capacity consumes additional energy.

The electrical costs for rack-based cooling are higher at low densities due to the increase in the number of cooling units which requires more power consumption to move air and water. Even with variable speed fans, the increase in cooling units at lower densities limits energy savings due to minimum fan speeds. At lower densities, the minimum fan speed provides more airflow than is required. In addition, more piping is required to push the water through. As rack power density increases, energy costs decrease. But, for high densities, the cost will start to increase because each rack has one cooling unit, and as the densities increase more airflow is required from each of the cooling units, the CRAH fans will approach the maximum operating speed which reduce the effective savings that can be achieved from the variable speed fans. Furthermore, the higher water flow required to maintain capacity also consumes additional energy.

Piping Near IT Equipment

Research shows that users are concerned with water or refrigerant piping co-located with IT equipment due to the possibility of fluid leaking onto IT equipment, and the associated downtime and/or damage. 

High-density data centers with multiple air conditioners generally use a chilled water cooling system and this trend is expected to continue due to environmental and cost concerns. Although refrigerants that have less possibility of damaging IT equipment exist, they are a more costly alternative to water. Concerns regarding availability and the drive toward higher densities have lead to the introduction of pumped refrigerant systems within the data center environment. These systems are typically composed of a heat exchanger and a pump which isolate the cooling medium in the data center from the chilled water and allows for oil-less refrigeration to reduce contamination in the event of a leak. However, the system could also isolate other cooling liquids such as glycol.

Cooling Unit Location

The location of an air conditioning unit can have a dramatic effect on the system performance. In the case of rack-based cooling, this problem of performance predictability is completely eliminated since the exact location of the air conditioner to the target load is determined. The benefit is that the cooling performance can be completely characterized in advance. If a phased deployment is part of the system design, the location of future air conditioning units requires little planning or forethought, being automatically deployed with each rack. 

Row-based cooling depends on simple design rules to locate air conditioners. The quantity and locations of row-based air conditioners are determined by rules that have been established through simulation and testing. Naturally this includes ensuring that the air conditioners are sufficiently sized to the row density specification. In addition, there are other rules, such as avoiding row end locations, which maximize the performance and capacity of the system. During future deployments, some location flexibility is retained up until the time of deployment. Average or peak-to-average rack power density of the row can be used to establish the quantity and locations of air conditioners in a just-in-time process. Row-based cooling is the most flexible compared to the rack-based approach, has a smaller footprint, and lower cost. 

In the case of room-based cooling without containment, efficiency is greatly dependent on the location of the cooling units. For example, the most effective locations may not be feasible, due to physical room constraints including doorways, windows, ramps, and inaccessibility of piping. The result is typically a sub-optimal design even when considerable amounts of engineering are applied. In addition, the logistics of installing room-based air conditioners typically require that they be placed into the room in advance, comprehending all future IT deployment phases. Since the exact layout of future IT phases may not be known, the locations of the air conditioners are often grossly ineffective. This is why containment is so critical for modern room-based cooling designs. Containment allows much more flexibility in the placement of cooling units. Contained room-based cooling also permits the additional option of locating the CRAH units outside of the data center.

Redundancy

Redundancy is necessary in cooling systems to permit maintenance of live systems and to ensure the survival of the data center mission if an air conditioning device fails. Power systems often use dual path feeds to IT systems to assure redundancy. This is because the power cords and connections themselves represent a potential single point of failure. In the case of cooling, N+1 design is common instead of dual path approaches because the common air distribution paths, being simply open air around the rack, have a very low probability of failure. The idea here is that if the system requires four CRAH units, the addition of a fifth unit to the system will allow any one of the units to fail and the total cooling load will be satisfied. Hence the name “N+1” redundancy. For higher power densities, this simple concept of redundancy breaks down. Let’s discuss how redundancy is provided differently for the three cooling methods. 

Redundancy for Rack Based Cooling

For rack-based cooling, there is no sharing of cooling between racks, and no common distribution path for air. Therefore, the only way to achieve redundancy is to provide an N+X or 2N dual path CRAH system for each rack, essentially at least two CRAH systems per rack. This is a severe penalty when compared with the alternative approaches. However, for isolated high density racks, this is very effective as the redundancy is completely determined, predictable, and independent of any other CRAH systems.

Redundancy for Row Based Cooling

Row-based cooling provides redundancy at the row level. This requires an additional or N+1 CRAH unit for each row. Even though the row CRAH units are smaller and less expensive than room units, this is a significant penalty at light loads of 1-2 kW per rack. However, for higher density, this penalty is eliminated and the N+1 approach is sustained up to 25 kW per rack. This is a major advantage when compared with either room or rack-based designs, which both trend to 2N at higher densities. The ability to deliver redundancy in high density situations with fewer additional CRAH units is a key benefit of the row-based cooling and provides it a significant total cost of ownership (TCO) advantage. 

Redundancy for Room Based Cooling

For room-based cooling, the room itself is a common air supply path to all the IT loads. In principle, this allows redundancy to be provided by introducing a single additional CRAH, independent of the size of the room. This is the case for uncontained room-based cooling at very low densities, and gives this approach a cost advantage at low densities. However, in uncontained room-based cooling at higher densities, the ability of a particular CRAH to make up for the loss of another is strongly affected by room geometry. For example, the air distribution pattern of a specific CRAH cannot be replaced by a backup CRAH unit that is remotely located from the failed unit. The result is that the number of additional CRAH units that are required to establish redundancy increases from the single additional unit required at low densities to a doubling of CRAH units at densities greater than 10 kW per rack. This is not the case for a room-based cooling that uses containment because the supply and return air paths are separated. 

Heat Removal Method

The special issues that we just discussed are influenced by the heat removal method. Direct expansion computer room air conditioners (CRAC) used to cool the data center operate differently than chilled water (CRAH) units. Using CRAC units in this way will affect their efficiency, humidification, and redundant operation. A design analysis must be done to comprehend the operation and controls of the specified cooling solution in a particular project. 

Agility, Availability, Total cost of ownership, Serviceability, and Manageability

We will see how to make an effective choice among cooling methods by comparing them against other criterias that are important in data centers, including agility, systems availability, total cost of ownership, serviceability, and manageability. Let us draw the advantage and disadvantage in a table,

If you would like to compare more of the advantages and disadvantages of this criterias, I would strongly suggest you to have a look at this article by APC(Page 9).

As a summary of above table, we can say the below,

• Rack-based cooling is the most flexible, fastest to implement, and achieves extreme density, but with additional expense. 
• Row-based cooling provides many of the flexibility, speed, and density advantages of the rack-based approach, but with less cost. 
• Room-based cooling allows for quick changes to the cooling distribution pattern by reconfiguring the floor tiles. Cooling redundancy is shared across all racks in the data center with low densities. This method offers cost and simplicity advantages. 

The conventional legacy approach to data center cooling using room-oriented architecture has technical and practical limitations in next generation data centers. The need of next generation data centers to adapt to changing requirements, to reliably support high and variable power density, and to reduce electrical power consumption and other operating costs have directly led to the development of row and rack-oriented cooling architectures. These two architectures are more successful at addressing these needs, particularly at operating densities of 3 kW per rack or greater. The legacy room-oriented approach has served the industry well, and remains an effective and practical alternative for lower density installations and those applications where IT technology changes are minimal.

Row and rack-oriented cooling architecture provides the flexibility, predictability, scalability, reduced electrical power consumption, reduced TCO, and optimum availability that next-generations data centers require.

It is expected that many data centers will utilize a mixture of the three cooling architectures. Rack-oriented cooling will find application in situations where extreme densities, high granularity of deployment, or unstructured layout are the key drivers. Room-oriented cooling will remain an effective approach for low density applications and applications where change is infrequent. For most users with newer high density server technologies, row-oriented cooling will provide the best balance of high predictability, high power density, and adaptability, at the best overall TCO.

Knowledge Credits: Energy University by Schneider Electric

Have a comment or points to be reviewed? Let us grow together. Feel free to comment.