Designing, Building, and Populating a 10-Megawatt Datacenter
Doug Hughes, D.E. Shaw Research, LLC
Doug Hughes from D.E. Shaw Research, LLC (Deshawresearch.com) a firm that develops novel algorithms and machine architectures for high-speed molecular dynamics simulations of proteins and other biological macromolecules, presented on building his company’s new data center and many of the choices, gotchas, and tips he found along the way.
Hughes went on to show the benefits and differences between wet and dry cooling systems. The main benefit of a wet cooling system is efficiency. However, water pipes take of quite a bit space and pipes can leak. Electronics and water do not mix well. In addition, humidity control with a wet system can be problematic. Lastly, if your environment freezes in the winter consideration must be made to ensure cooling water pipes do not freeze during maintenance periods. Dry systems on the over hand, can control humidity well using waste heat and can be placed almost anywhere with no major structural pipes required. However, dry systems are less efficient.
In a data center there are two competing factors. First human comfort, no one wants to work in a frigid environment, nor a blistering hot one. Second is creating the highest delta-t or difference in temperature difference in temperature at the inlet and outlet. For example a high delta-t would result in the coldest possible “cold aisle”, and hottest possible “hot aisle.” The best cooling design is a compromise between human comfort and delta-t.
Humidification is used in a data center to moderate static electricity. Various sources of humification are available such as plant steam, a steam canister, an infrared system, or an ultrasonic system. A general recommendation of 40-60% humidification is the industry standard but Hughes believes this tolerance is a bit tight.
Various economizing techniques are available to decrease the cost of cooling a data center. On the air side, venting waste heat directly outside can save a lot of cost. In addition, humidification of cooler air is more efficient as cooler air’s dew point is lower. On the water side, a heat exchanger placed outdoors can increase savings especially in colder regions.
Hughes recommends a few things when using wet cooling system. First disable humidification on all but one of your cooling units, or use a purpose specific humidifier. The other option is to ensure precise calibration of all cooling units. Hughes recommended this because the cooling units will fight to humidify and dehumidify. Next, keep an eye out for changing conditions in your environment such as increased load on servers, increase or decreased number of servers, and change in outside temperatures. Next, disable reheat. Reheating is the process of dehumidification where the system chills the air down to the dew point, and then reheats it. It is much more efficient to have one unit bypass warm air from outside.
A number of consideration must be made for the flooring a data center. Generally two choices are available, cement/epoxy floors or raised floors. A cement/epoxy floor has a high weight load, but is bad for chilled water cooling as pipes are usually run under the floor. A raised floor is more expensive, but allows room for chilled water piping and cables. The a raised floor becomes more expensive when high load capacities are needed or as the height of the floor increases. In addition, when using a raised floor with chilled water cooling a consideration must be made that leaking water pipes and power cables do not mix well.
Hughes presented various fire suppression techniques and technologies available for data centers. Pre-action system is a water based system with sprinklers. However the pipes of the system are filled with air during normal operation. When smoke is detected a pump is a turned on and floods the pipes. Water is only released when the individual sprinkler heads reaches a certain temperature. The biggest problem with using a water based system is cleanup. In addition to cleanup, water is not efficient at putting out interior fires or fires inside contained areas as it takes a little while for the water to reach inside the contained area. The next type of fire suppression is a dry agent based system. The benefits of a dry system is minimal downtime as there’s no dry-out period, and interior fires are extinguished quickly. However some systems require room sealing, and have a corrosion potential. A new system named Aero-K is potassium based and is safe for humans and hardware.
A number of suggestions were presented on power efficiency. High voltages and fewer voltage conversions equal better efficiency. Typically each power conversion made about 1% to 2% of the energy is lost to waste. The use of 3-phase allows for 173% more power to be carried in a power line than a single phase system. When buying equipment insist on high efficiency and correctly sized power supplied from vendors with power factor correction. Lastly, only use as much redundancy as required.
Another consideration when building a data center is the density required, or how many kilowatts per rack. Hughes noted that while blade servers offer cabling advantages and space savings, their typical power requirements per compute unit is about the same as new servers. In general, as density increases cooling and power requirements increase too.
Lastly during Q&A the suggestion was made by an audience member to ensure that your sump pumps and drip pan pumps are on protected power. A question of flywheel use in Hughes’ data center was brought up. Flywheels are used as an alternative to uninterruptible power supplies, but only offer a short duration of power, but are typically designed to allow enough time for generators to turn on. Flywheels are a less toxic, require less maintenance, and are more efficient than uninterruptible power supplies.
Beyond VDI: Why Thin Client Computing and Virtual Desktop Infrastructures Aren’t Cutting It
Monica Lam, MokaFive and Stanford University
Monica Lam presented on the current issues and myths of Virtual Desktop Infrastructure (VDI) and MokaFive’s LivePC product. MokaFive can be found at mokafive.com where the player portion of LivePC is available for free.
Thin-client computing does not reduce the cost of hardware because of a few reasons. First is the reduced cost of a PC today, and the similar cost of thin-client hardware. Monika gave the example of a suitable PC costing $499 and thin-client hardware costing $300 plus $60 a year. In addition, employer’s can depend on employees use of their own computers. Another reason is when moving desktop virtualization into a data center additional cost of data center operation is incurred like having to provide cooling and power. If the virtualization is running at the end-point often passive cooling can be used. Lastly, when designing the systems for servers in a data center to support virtualization you must provision for the “Super Bowl effect” or the theoretical event when all your users login and use their virtualized desktops at the same time.
Centralized management does not have to lead to a bad user experience. Regarding VDI, where the virtual machine’s are run on a central server, introduces a few factors that lead to a bad user experience. First the overhead of VDI, running multiple virtual machines on a single server causing resources are being shared among many users. Next, the user is running on a remote display, so all display information has to be sent and received across whatever network connection the user is on. Often this leads to very slow interaction performance. In addition, 3D graphics or other graphic intensive applications are very difficult to interact with over a remote desktop.
MokaFive’s LivePC product attempts to solve the problems of VDI by offering a centralized management interface that allows administrators the ability to create, update, and publish virtual machines. The virtual machines are published to a HTTP server and made available for downloading by the client. The client portion of the product lives on the user’s machine and downloads and runs the virtual machine. Virtual machines created by LivePC maintain two virtual hard drives. The first drive is managed by the administrator remotely, and the second is used to store any local changes to the virtual machine. LivePC will automatically pull differential updates to the first virtual hard drive as its updated by the administrator. In addition, LivePC allows the user to revert back to the original virtual machine undoing any changes to the operating system.
During the Q&A session the use case of users needing shared access to large pools of data was asked. Monica noted this is the one application where VDI is quite useful. Another question was asked how to backup local changes in the LivePC product. Monica responded that MokaFive decided to let users decide how to backup local changes as most enterprises have their own backup solutions in place. A security based question asking does LivePC prevent the host OS from screen capturing the guest OS. Monika said MokaFive treats as any data that is displayed on the screen is gone and not securable. She then noted that the only real solution to this problem is a trusted computing environment. The last question of the session was if MokaFive offered a bare metal install of their LivePC product. Monika answered that MokaFive initially developed a bare metal installation before going to using the virtual machine player model and it is still available.