Get an immersive understanding of the top data centers of multinational companies

If you trace their roots, the 21st century data center has its roots in the computer rooms that housed mainframes in the 1960s and 1970s. These large data centers were sturdy and rugged, built to house complex and expensive IT infrastructure that required sophisticated power and cooling systems. The modern data center originated in the 1990s, when many companies installed rows of servers in-house (often replacing the mainframes in the computer rooms that had previously been there). Then came off-site colocation data centers, where multiple companies placed their servers in various locations: dedicated rooms, lockable cabinets, or racks. Finally, we have today's dedicated data center buildings, complete with processing, storage, and connectivity systems, usually offering some combination of colocation, managed hosting, and cloud services (public, private, and/or hybrid).

Some of these data centers are truly behemoths. The world's largest data center in terms of available power and cooling capacity is Switch Communications Group's SuperNap-7 in Las Vegas, a colocation facility that covers 407,000 square feet (37,811 square meters, or 3.78 hectares). As cloud computing drives the need for high-density colocation, The Switch announced plans to expand data center space to 2.2 million square feet (204,387 square meters, or 20.4 hectares), starting with SuperNAP-8, which opened in May 2013, and breaking ground on SuperNAP-9 in 2014.

[[126464]]

[[126465]]

The Switch's SuperNAP facility in Las Vegas is planning to expand, bringing the total data center space to 2.2 million square feet (204,387 square meters). Image source: The Switch Communications Group

Today, "data center users" include large IT infrastructure and service providers such as Amazon and Google, as well as any enterprise user or consumer connected to the Internet. However, for the purposes of this article, the typical user is an IT administrator in an enterprise with an internal data center - whose data center runs both traditional enterprise applications and virtualized workloads that already use some cloud services (most likely SaaS applications) and is considering the next step: expand internal capacity, move more workloads to the public cloud, or adopt a hybrid strategy?

Data center usage is currently undergoing a seismic shift due to the growing use of “cloud” (i.e. outsourced and external) infrastructure, services and applications. According to Cisco, by 2016, nearly two-thirds (64%) of all data center traffic will be handled in cloud data centers rather than in “traditional” (i.e. on-premises) data centers. In contrast, in 2011, 61% of traffic was handled in traditional data centers and only 39% in cloud data centers.

Data center design and construction

The typical data center is an industrial building that provides space for IT equipment and all the necessary power distribution, cooling, cabling, fire protection, and physical security systems. Data centers are usually built in areas where power and land costs are low, but where there is enough manpower to staff the data center (unless it is a so-called "lights-out" data center, which is managed remotely and requires little on-site staff.)

The data center industry alliance Uptime Institute classifies data centers according to a four-tier system based on workloads, and the Telecommunications Industry Association (TIA) certifies data centers (TIA-942 standard). The standard specifies the architectural, security, mechanism and telecommunications requirements needed to achieve a given level of availability for each tier: Tier 1 is 99.671% (no more than 28.8 hours of downtime per year); Tier 2 is 99.741% (no more than 22 hours of downtime per year); Tier 3 is 99.982% (no more than 1.6 hours of downtime per year); Tier 4 is 99.995% (no more than 0.4 hours of downtime per year).

Another widely cited number related to data centers is PUE, or power usage effectiveness. Pioneered by the Green Grid, PUE is the ratio of total power used in a data center to that used by IT equipment. The ideal PUE ratio is 1.0, with actual values ​​ranging from over 2.5 to around 1.1 (see chart below). Given the scale of its data center operations, it’s no surprise that Google leads the way in power usage efficiency: The search giant claimed a “combined” PUE of 1.12 across all its data centers as early as the fourth quarter of 2012 (it’s called “combined” because Google accounts for all sources of power spending and measures it year-round). Another way to express PUE is to use its inverse, known as data center infrastructure efficiency (DCiE). Google scored 0.893, or 89.3%, on this metric.

To understand the data center sector, Uptime Institute launched an annual survey in 2011. Its 2012 Data Center Industry Survey, conducted in March/April 2012, surveyed 1,100 data center managers, IT administrators and senior executives from North America (50%), Europe (23%), Asia (14%) and Central/South America (7%).

The survey's key findings include: 30% of respondents expect data center capacity to run out in 2012; most plan to keep existing data centers running by consolidating servers and upgrading data center infrastructure. Compared to the 2011 survey, 10% fewer respondents plan to build new data centers in 2012, while 10% more plan to move more workloads to the cloud.

The survey found that the main drivers for cloud adoption include cost reduction (27%), scalability (23%), customer/user demand (13%), and speed of deployment (13%). The main factors hindering cloud adoption are security concerns (64%), compliance/regulatory issues (27%), cost (24%), and lack of in-house cloud management expertise (20%).

Uptime Institute also found that organizations measured PUE more accurately in 2012 than in 2011, and generally reported a decrease in PUE, which was in the range of 1.8-1.89. Power-saving strategies mainly include: hot aisle/cold aisle containment and raised floor inlet temperature (finding the ideal compromise between IT equipment fan power and cooling energy).

The Uptime Institute's 2012 Data Center Industry Survey reported that PUE (power usage effectiveness) values ​​from its 1,100 respondents varied widely. Only 6% of respondents reported a PUE below 1.3. Image source: Uptime Institute

Other trends noted in the survey include: increased interest in prefabricated modular data centers or components (9% have deployed, 8% plan to deploy, and 41% are considering deploying); and the beginnings of deploying data center infrastructure management (DCIM) tools (more on these trends below).

How much data is there?

How much data is flowing through the world's data centers, big and small, and what are the trends? Cisco's annual Global Cloud Index (GCI) has answered this question since 2011. In the 2012 report, Cisco used network data from the 10 largest enterprises and Internet data centers to predict that by 2016, global data center traffic will reach 554 exabytes (EB) per month. By the way, 1 exabyte = 1000 petabytes (PB) = 1000000 terabytes (TB). In annual terms, this has soared from 1.8 zettabytes (ZB) in 2011 to 6.6 ZB, a compound annual growth rate of 31%:

Estimated data center traffic growth by 2016 (1 zettabyte = 1,000 exabytes). Image and data source: Cisco Global Cloud Index

Where does all this data come from? Cisco breaks down data center traffic (both enterprise and consumer) into three broad categories: traffic that remains inside the data center, traffic that travels between data centers, and traffic that travels from the data center to the user over the Internet or IP WAN. According to Cisco, the majority of traffic (76%) is traffic that remains inside the data center.

Only 17% of data center traffic is estimated to be transmitted from the data center to the user. Image and data source: Cisco Global Cloud Index

If we consider server workloads, the trend towards the cloud becomes clear: Cisco estimates that of the 180.6 million workloads installed in 2016, 62% were in cloud data centers and 38% were in traditional data centers; in contrast, in 2011, 30% were in cloud data centers and 70% were in traditional data centers, out of a total of 71.1 million workloads.

Projected server workload positions by 2016 (server workload is defined as the tasks a server performs to run applications and support multiple users). Image and data source: Cisco Global Cloud Index.

The average number of workloads per physical server in cloud data centers is expected to increase from 4.2 in 2011 to 8.5 in 2016; as for less efficient servers in traditional data centers, the estimated and predicted numbers are 1.5 and 2 in 2011 and 2016, respectively.

Cisco also predicts a "transition of power" in the regional distribution of cloud-based workloads: In 2011, North America was far ahead, accounting for 37.8% of the 21.4 million workloads, and Asia Pacific accounted for 31.3%; by 2016, Asia Pacific is expected to account for 36.2% of the 112.1 million workloads, while North America will fall behind with 26.4%.

One thing is clear: for the foreseeable future, organizations will use a mix of in-house and outsourced IT infrastructure; and a fundamental trend is the increase in the use of services located in public cloud data centers, especially in the Asia-Pacific region.

Rethinking the data center

So how can you design a more efficient, scalable, flexible and secure data center? Specifically, how can IT service providers reduce build costs, get closer to a PUE of 1.0, increase computing power per watt, reduce latency for users and reduce carbon emissions? And first, how can they manage these complex systems more effectively? Here are a few trends that will impact the data center of the 21st century (also known as Data Center 2.0).

Microserver

For certain kinds of high-volume, low-computing-power workloads, such as serving web pages, executing search-engine queries or parallel data-processing tasks, a new class of servers called microservers could take up less data-center space and consume less power than traditional Xeon- or Opteron-based enterprise servers.

HP is one of the leaders in the emerging microserver space, and it has already launched Project Moonshot. The first Moonshot server was announced in November 2011, an ARM-based system-on-chip (SoC) platform called Redstone. HP installed it as a demonstration system in its Discovery Lab. According to HP, the Redstone development server platform is based on the existing ProLiant SL6500 chassis, and its server tray can accommodate both computing boxes and storage boxes, providing four times the density of traditional ProLiant servers (each tray can accommodate up to 72 computing nodes, that is, each 4U chassis can accommodate 288 computing nodes), while consuming only one-tenth of the power.

HP's first Gemini servers are part of the company's Project Moonshot and are based on Intel's low-power Atom S1200 (Centerton) processors. Image credit: HP

In June 2012, HP changed course and announced that the first batch of mass-produced Moonshot microservers (code-named Gemini) would use compute boxes based on Intel's "Centerton" Atom processors, specifically the Atom S1200 series with a nominal TDP (thermal design power) of 6.1 watts to 8.5 watts. The Gemini servers were released as early as the second quarter of 2013. However, ARM definitely has a high position in the Moonshot blueprint: HP's Pathfinder partner program is designed to help manufacturers develop for the Moonshot platform, and the latest member is Texas Instruments and its ARM-based (Cortex-A15 MPCore) Keystone II SoC. However, it is still unknown whether the Keystone II-based compute box will appear in a future generation of Gemini servers.

Of course, other vendors are also involved, especially Dell and its ARM-based Copper servers. Earlier this year, AMD acquired microserver vendor SeaMicro to strengthen its data center server solutions business. SeaMicro's Atom-enabled SM15000 servers have already been certified by Cloudera for CDH4, a widely deployed Apache Hadoop distribution for big data analytics applications.

While microservers are a trend worth watching, they are not suitable for all types of workloads and are unlikely to revolutionize the data center anytime soon. For example, research firm IHS iSuppli predicts that by the end of 2016, only 1.2 million microservers will be shipped -- just 10% of the total server market.

Solid-state storage

Fast, low-power, shock-resistant, and drop-resistant solid-state storage devices are common in client devices, such as flash memory in smartphones and solid-state drives in tablets and ultrabooks. SSD arrays are also increasingly finding their way into data centers, where their performance and power efficiency are particularly attractive. However, as with microservers, solid-state storage is not universally suitable for this environment.

The main factor holding back mass enterprise adoption of SSDs is price; while prices are falling, they are still significantly higher than traditional hard drive-based solutions. Another potential issue with SSDs is limited "write endurance" - after exceeding a maximum number of P/E (program/erase) cycles, non-volatile NAND flash blocks fail. Intel has addressed this with its HET (High Endurance Technology), which combines hardware and firmware improvements to enhance the endurance of enterprise-class SSDs. HET includes "background data refresh," which moves data during periods of inactivity to avoid reading areas of high activity.

An SSD array responds to database requests much faster than a hard disk-based array (left); SSDs also operate much cooler (right). Image credit: Intel

In Intel's testing (see above), first in a control environment and then in a production data center (running a security compliance database), SSD arrays delivered the following benefits over comparable HDD-based enterprise-class (15,000 RPM) storage systems: Up to 5x better performance for random disk I/O tasks; Reduced read/write latency (up to 10x/7x, respectively) and reduced maximum latency (up to 8x); Faster switching from idle to active; No longer seek time when disks are full; 50% less power consumption, plus one-third less heat. Intel also said the higher initial cost of the SSD solution (3x higher) was worth it because of the following benefits: Less time IT staff spend dealing with tricky I/O queue depths; No backlog in logging monitoring data (eliminating potential compliance issues); Reduced latency in patching systems; Guaranteed performance and capacity to handle 3 to 5 years of workloads; Easier installation than traditional SAN or NAS solutions.

Enterprise-class SSD storage is available not only in SATA-based arrays (such as EMC's new XtremeIO product), but also in PCI cards from many vendors, including Fusion-io and LSI. X-IO takes a nontraditional approach. The company specializes in hybrid storage systems that combine SSDs and regular hard disks in a single storage pool; its firmware (Continuous Adaptive Data Placement) only puts data on the SSDs when there is a significant performance improvement. According to Mike Wills, CEO of RTW Hosting, replacing a traditional SAN with a hybrid array from a vendor such as X-IO can provide 15 to 20 times the performance, as well as a 60% increase in power efficiency. Although it needs to be optimized in terms of both read/write performance and storage capacity, a tiered hybrid solution may be the way to go.

Software Defined Data Center

Server virtualization and storage virtualization are two mature concepts, but in order to achieve maximum data center efficiency and flexibility, the third major part of the IT system: the network probably needs to undergo a transformation similar to the separation of the control layer from the physical hardware. Software-defined networking (SDN) is a new field, and major network vendors are still coming up with their own solutions: VMware, a subsidiary of EMC, acquired SDN professional company Nicira as early as July 2012, which was a major move in this regard.

The Open Network Foundation (ONF) is the organization that develops and promotes SDN standards, and its main tool is the OpenFlow protocol. In April 2012, ONF's founding member Google revealed details of the large-scale implementation of OpenFlow on its data center backbone network, which carries the search giant's internal traffic (Google's other Internet-facing backbone network carries user traffic).

Google's implementation of OpenFlow on its internal backbone network uses the search giant's own network switching equipment. Image credit: Google

Google's benefits from SDN deployment include: shorter deployment time, the ability to use software to simulate backbone networks for testing, easy upgrades, greatly improved network utilization (close to 100%, while the industry average utilization is only 30% to 40%), and generally high stability. In response to its strategy in servers, Google built its own network switches for this project - if Google network switches are widely adopted, this will make traditional data center network equipment manufacturers (especially Cisco) uneasy.

Modular Data Center

Traditional, purpose-built, “brick and mortar” data centers are expensive and time-consuming to build (and to expand once capacity is reached). Prefabricated modular data centers, often packed into standard shipping containers (which contain servers, networking, and cooling equipment), allow companies to quickly add the capacity they need now, and then quickly and cost-effectively deploy more modules as needs grow.

The Uptime Institute survey mentioned above specifically noted that respondents have a strong interest in prefabricated modular data centers. However, this trend is clearly in its early stages: 52% of respondents said they are still considering traditional construction methods for future data center expansion, and 19% are considering prefabricated components that use little to no traditional construction methods; at the same time, 21% of respondents said that fully containerized data centers are an option worth paying attention to.

Only 11% of respondents claimed to be "extremely confident" in the maturity of the modular data center market, but 71% agreed that "meeting the changing needs of successive generations of IT equipment" is more important than "a long lifespan of data center infrastructure."

HP claims its modular POD 240a data center can hold up to 42 racks of industry-standard equipment and has an upfront cost that is a quarter of that of a similar traditional data center and can be deployed in just 12 weeks.

For those interested, there are a number of vendors competing to offer modular data centers, including AST, Cisco, Colt, Dell, HP, IBM, IO and Toshiba, among others.

Green Data Center

Traditional data centers consume a lot of electricity and other resources (such as cooling water). As data center capacity is bound to grow, enterprises and service providers are under pressure to do a good job of environmental protection and reduce the resource usage of their data centers.

The Green Grid-inspired PUE metric is now widely used and cited in an effort to improve power efficiency, with state-of-the-art data centers now achieving a PUE of around 1.1 – that’s a DCiE (Data Center Infrastructure Efficiency) of 90.9%.

Several trends mentioned earlier, such as the transition from traditional on-premises data centers (low server utilization) to modern cloud data centers (high server utilization), the growing use of low-power microservers and solid-state (or hybrid) storage systems, and power-efficient modular data centers such as HP's POD 240a (called "EcoPOD"), will help create more power-efficient data centers. Another initiative, Facebook's Open Compute Project, which will promote simplified open source server and other IT infrastructure designs, may also help improve data center power efficiency in the future, but it is still early days for this initiative.

[[126466]]

Verne Global's modular data center in Ireland uses 100% green electricity and uses outdoor air rather than refrigeration equipment to cool IT equipment. Image source: ZDNet

When it comes to getting environmental certification, the location of the data center is a key factor to consider. For example, ZDNet visited a modular data center built in Ireland a few years ago, and the operator Verne Global obtained a long-term, low-cost, 100% environmentally friendly (renewable hydro/geothermal power) electricity supply for it. Ireland's climate is also suitable for free cooling-using outdoor air to cool IT equipment instead of installing power-consuming refrigeration equipment.

Some data center cooling solutions are quite bizarre. For example, Intel tried submerging servers in barrels of mineral oil. This infographic summarizes Intel's other more conventional design ideas:

Data Center Infrastructure Management (DCIM)

To run IT systems efficiently and make informed decisions about future data center capacity, you need current and forecasted data on a range of application workloads in your company, their capacity requirements, and the comparative costs of running them on-premises, in the public cloud, or as a hybrid solution. Otherwise, you risk over-provisioning and wasting money, or under-provisioning and experiencing performance bottlenecks and outages. Both situations are undesirable; for IT administrators, both have job-threatening consequences.

Tools that can help you collect and interpret this data fall into the broad category of data center infrastructure management (DCIM). This is a new but rapidly growing industry that includes a variety of solutions, some of which are dedicated to managing the physical infrastructure of the data center (IT hardware, power and cooling systems, etc.), and some more specialized solutions that not only provide an overview of physical, virtual and cloud infrastructure, but also provide predictive analysis capabilities and "what-if" scenario planning. This latter category of solutions is sometimes called DCPM (data center predictive modeling).

Forrester Research's 2012 DCIM Solutions Market Overview lists eight core functions: inventory and discovery, maintenance and change control, data collection, comprehensive monitoring and display dashboards, alerts, control, trend analysis, and the ability to design future solutions that can be deployed. Emerging functions listed by Forrester include: power planning and capacity based on actual use rather than rated use; modeling that recognizes usage; network capacity modeling and integration of DCIM with conventional IT management tools.

Forrester lists three categories of DCIM vendors: data center facilities and infrastructure vendors (such as Emerson); IT management vendors (such as CA Technologies); and system hardware vendors (such as HP).

Sentilla's data center performance management software provides basic DCIM capabilities, with the added bonus of predictive analytics and "what-if" scenario planning. This example shows the comparative costs of various server upgrade strategies. Image credit: Sentilla

Cloud platforms and open standards

For most companies, the path to cloud computing begins in an internal data center, which typically runs a mix of traditional, siloed enterprise applications and virtualized "private cloud" workloads, often based on VMware's proprietary vCloud technology.

When it comes to leveraging public cloud services, whether it's "cloud bursting" (surges in management load) in a hybrid cloud architecture or as a service for a completely new deployment, vendor lock-in issues arise. No one wants to give a large portion of their IT infrastructure to one vendor's cloud platform, only to find out they can't migrate to another vendor's cloud when they need to. This is the driving force behind open source cloud platforms.

The most famous example is OpenStack, an open source software framework based on Linux that is backed by the OpenStack Foundation, which promotes and governs it. OpenStack was launched in 2010 by hosting provider Rackspace and NASA, and now has broad support from the industry, including industry giants such as Cisco, Dell, HP and IBM.

Other well-known open source cloud platforms include: Apache CloudStack (originally developed by Citrix, whose implementation is CloudPlatform), Eucalyptus, OpenNebula, and DTMF's OVF 2.0.

OpenStack, in particular, has received a lot of attention as a major competitor to VMware, which runs public and hybrid clouds through its service provider partners. Despite the frequent “cloud wars,” it’s worth noting that the largest public cloud platforms: Amazon Web Services, Google Compute Engine, and Microsoft Windows Azure are all highly proprietary in nature.

An open, fully compatible set of cloud software platforms and APIs promises to remove barriers to wider adoption of cloud technology, allowing more companies to reap its benefits: on-demand self-service, rapid scalability, and transparent pricing.

Conclusion

Data centers are complex and expensive facilities, and running them efficiently requires multiple skills in IT infrastructure management, energy management, and building management. It is no wonder that, as we have seen, many companies are increasingly outsourcing these tasks.

Whether you maintain a traditional on-premises data center, outsource IT systems to a public cloud, or adopt a hybrid strategy depends on the mix of workloads involved. If you're migrating existing on-premises workloads to the cloud, or expect to handle a surge in load, a hybrid cloud solution may be a good fit. On the other hand, if you're deploying a completely new workload, it may be more appropriate to go all-in on the cloud from the start. However, if you're concerned about entrusting sensitive or mission-critical business processes and data to a cloud environment (for compliance or reliability reasons, for example), you may want to keep them in your own on-premises data center and keep them under tight control.

Whether in-house or outsourced, the 21st century data center will improve its efficiency and flexibility through a combination of virtualization and consolidation throughout the IT architecture (servers, storage and networking), increased use of low-power hardware (such as microservers and solid-state storage drives), modular data center construction, green power and cooling technologies, and DCIM software that orchestrates data center management and models future capacity expansion scenarios.

If you use any kind of outsourcing, remember that the cloud is not some magical environment that runs smoothly all the time. Do the necessary research: Check the service provider's service-level agreement (SLA), find out where your data will reside, how easy it will be to move it, find out what security and backup provisions the provider offers, and always be aware of any potential costs.