48V AC direct supply technology is the future development direction of data centers

With the large-scale construction of data centers, users are more concerned about energy costs and efficiency. The future development direction of data center power supply technology must be the direct supply of mains electricity. While saving the investment and site occupation of traditional UPS equipment and reducing the initial cost, it also reduces the later operating cost by reducing the number of conversion levels and improving power supply efficiency. Moreover, the efficiency improvement mentioned here does not only refer to the high efficiency of the power supply path from the grid side to the input side of the IT equipment, but also the high efficiency and green environmental protection of the entire energy path from the primary energy side to the CPU, etc. Although the PUE value in the traditional concept may increase, the unit computing energy consumption is reduced. As shown in Figure 1, the overall development trend of data center power supply technology in the future is from high-voltage/centralized/AC large UPS to low-voltage/distributed/DC small UPS, from the centralized lead-acid battery at the computer room level to the distributed small (lithium) battery embedded in the IT cabinet or even the server, and from the polluting fossil energy to the environmentally friendly green energy.

The core of data center power supply uninterruptible technology lies in uninterruptible power supply and its battery technology, so different battery connection positions also determine different power supply architectures. At present, the mainstream backup battery voltages in the industry range from 400 volts for UPS to 380V, 240 V and 48V for DC power supply, and even 12V for batteries embedded in IT equipment. There are even medium-voltage UPS using flywheels or small UPS with lower voltages such as 5V, etc., which are not listed here because of their seldom application. Figure 2 shows the main technical solutions for data center power supply in the industry at present, starting with the traditional UPS with centralized 400 volts lead-acid batteries, followed by the 240V high-voltage DC technology for standard servers that do not require customization and 240V batteries directly connected to the output bus, followed by the 48V DC or 380V high-voltage DC battery direct connection technology for servers that use customized 48V or 380V input power, and finally the 12V battery direct connection server motherboard input solution of Google and others. The closer the battery is to the end server motherboard or CPU, the more decentralized the power supply system is, and the corresponding IT system is also more distributed; the closer the battery is to the end, the higher the degree of customization of the power supply system, and the more difficult it is to carry out on a large scale for ordinary users; the closer the battery is to the end, the higher the control and management level of IT power supply and battery is required. ***, the closer the battery is to the end, the fewer the conversion levels on the power supply path from the power grid to the CPU, which brings higher conversion efficiency, but the transmission loss on the low-voltage side may increase. Therefore, comparing centralized and distributed, high voltage or low voltage, AC or DC, choosing different power supply architectures will greatly affect the reliability of the power supply system, power supply efficiency, construction cost, etc., as well as the maturity of technology and ecology, and application flexibility, etc.

In addition, with the development of battery technology and the introduction of green energy such as wind power, solar energy, and fuel cells in data centers, data centers are facing more opportunities and challenges. In a sense, the uninterruptible power supply technology of data centers will ultimately depend on the development of battery technology. The innovation of batteries will also bring changes to the power supply architecture of data centers. For example, the development of traditional low-density lead-acid batteries to high-energy-density lithium battery technology will likely move the backup battery from the battery room to the IT cabinet, or even to the inside of IT equipment. Similarly, the development of battery technology has made it possible to store energy from volatile green energy such as wind power and solar energy, and will also change the traditional data center power supply mode from a single power grid. These further discussions will not be expanded here. This article will focus on the mainstream technologies used in the industry. Through a rough analysis, it will be a good start so that everyone can have a more in-depth discussion of the development direction of future data center power supply technology.

Data center granularity level classification

Before we start this article, we can also consider the application scenarios of UPS from the top to the bottom of the data center. A typical data center can be divided into different granularities such as the entire campus, a single building, a computer room module, a micromodule, a cabinet, and a server. Since the scale of the entire campus and a single building is too large, there is basically no UPS that can cover the entire level, so it will not be discussed here. The typical approach at the computer room module level is to use the traditional centralized split cabinet-level UPS, the typical approach at the micromodule level is to use a semi-distributed integrated cabinet-level UPS, the typical approach at the whole cabinet level is to use a distributed DPS power frame-level UPS, and the typical approach at the server level is to use a component-level embedded battery UPS approach, as shown in Figure 3 (a) to (d) respectively. Similarly, under these four application architectures, the placement of the battery is also very different, from the centralized power battery room at the computer room module level, to the semi-distributed battery cabinet at the micromodule level, to the distributed battery BBU at the whole cabinet level, and finally to the component-level lithium battery pack at the server level. At different levels, the construction scale of both UPS and batteries ranges from large to small, from centralized installation to decentralized placement, from one-time investment to phased investment, and the impact of failures ranges from the entire computer room module to a single server, etc.

From the previous analysis, the use of distributed UPS can bring some benefits, such as phased construction and investment while growing, which can reduce the one-time investment of initial funds; and often many data centers have low load rates, or it takes a long time to fully load the server, so the low efficiency of centralized UPS under long-term low load can be effectively reduced by using distributed small UPS; in addition, the distributed small UPS invested and constructed in phases can flexibly match future technology upgrades or business changes, or be configured on demand according to different reliability levels, bringing more flexibility, etc. Therefore, distributed will bring some benefits over centralized, but is the more distributed the better?

The answer is obviously no. Let's analyze it from the perspective of the ecological or industrial chain level. First, the traditional UPS architecture in Figure 3 (a) has been developed for decades, and its technical maturity and upstream and downstream industrial chains are very mature, making it extremely difficult to implement. Secondly, the micro-module UPS in Figure 3 (b) can be an integrated modular UPS or an integrated 240V high-voltage DC, or even a 48V communication power supply system. Since it does not require customization and modification of existing IT equipment such as servers, it is relatively easy to implement. Next is the power supply frame + battery BBU solution at the whole cabinet level in Figure 3 (c). Due to the technical accumulation and industrial chain support of foreign OCP or Google, Microsoft whole cabinets or domestic Scorpio whole cabinets, as well as the large-scale application needs of Internet users, it is currently being implemented or being prepared for implementation, but there are certain difficulties. The last is the motherboard direct-mounted battery pack solution at the IT equipment level. Since the battery is embedded in the server, etc., IT equipment needs to be fully customized. In addition, the technical maturity of the direct-mounted battery pack solution for IT equipment is not yet sufficient, and it involves many upstream and downstream manufacturers and products of different models, so it is extremely difficult to implement. #p#

Why is 48V power supply architecture an important next step?

Earlier, we introduced the development trend and application scenarios of UPS in data centers. We believe that the development of traditional UPS architecture at the large computer room level in the first stage and semi-distributed UPS at the micromodule level in the second stage are relatively clear. However, these architectures all have more or less UPS in front of the server (whether it is AC UPS, 240V/380V high-voltage DC, or AC power + UPS, etc.), and the efficiency and cost are not good enough. Therefore, the direct supply of AC power, or the use of AC power to directly power IT equipment such as servers, and to provide protection against power failure should be the next development direction of data center power supply technology. Coupled with the rapid development of the whole cabinet technology, we believe that the two will be organically combined, that is, the use of AC power supply frame discussed above, plus the battery BBU solution with built-in power failure protection in the frame will be the next development direction.

Now the question is, if the whole cabinet adopts the direct supply technology of AC power, should its DC bus voltage adopt the existing traditional 12V bus architecture, or can we plan a 48V innovative architecture? We will compare the two technical routes from the following seven or eight aspects.

1. Comparison based on the future growth of cabinet power density

With the development of data center technology and the continuous improvement of IT equipment computing power, the power density of a single cabinet in a data center has increased from several kilowatts in the past to nearly ten kilowatts now, and cabinets of more than ten kilowatts have already appeared. Therefore, the power density of a single cabinet in the future is likely to reach more than 20 to 30 kilowatts. Therefore, in this case, if the entire cabinet continues to use a 12V busbar, the current will be as high as two or three thousand amperes. Whether it is the voltage drop, loss, structure, heat dissipation, price, etc. of the copper busbar, it will be a big challenge; on the contrary, if a 48V busbar architecture is used, based on the very mature 48V communication power supply market experience, these challenges of the previous 12V are not a problem, and the power expansion capability is expected to be upgraded to a single cabinet density of 30-40KW, which can achieve long-term planning. Therefore, the 48V architecture has great potential in increasing the power density of a single cabinet.

2. Comparison from the perspective of busbar loss

As estimated above, if the bus current reaches two to three thousand amperes under the 12V architecture, then the bus loss can reach more than 4 or 5 kilowatts even if the contact resistance or line impedance is as small as 1 milliohm. Even if multiple power supply frames or multiple busbars are used to share the current and reduce losses, the overall cost, space occupancy, copper consumption, and energy consumption are very large. On the contrary, if a 48V busbar architecture is used, the current is reduced to one-fourth of the original, and the busbar loss is reduced to one-sixteenth of the original, which can greatly reduce the busbar loss and heating risk.

3. Compare from the voltage tolerance range

We know that the input voltage range of a 12V powered server motherboard is usually 5% positive and negative. If the architecture of 12V bus + 12V battery BBU is adopted, it is basically difficult to reach this voltage tolerance range. Especially under a large current that may be as high as two or three thousand amperes, the voltage drop under an impedance of 1 milliohm is as high as two or three volts, and the voltage drop of the copper bar far exceeds this 5% voltage range of about 0.6V. If multiple power supply frames and multiple buses are used to share the current control voltage drop, then in addition to the cost, space and other issues mentioned above, an additional DC/DC voltage regulator circuit is required to ensure the voltage reduction caused by the discharge of the battery BBU. The total cost of this DC/DC voltage regulator circuit will be very high, which will also make the system overall uneconomical. Under the 48V bus architecture, the input voltage range allowed by the motherboard is usually very wide, 36V-58V, and the bus voltage drop is no longer a problem. In addition, the battery BBU can be directly connected to the 48V bus, and there is no need for a DC/DC voltage regulator circuit, so it becomes very easy to use the 48V bus architecture to allow a voltage tolerance range.

4. Comparison from the perspective of battery backup time

As mentioned earlier, since the voltage tolerance range is only 5% under the 12V bus architecture, the discharge cut-off voltage range of the battery connected to the bus will be very narrow. Even if a DC/DC voltage regulator circuit is added, the allowable battery discharge time will be shorter because the power supply undervoltage protection circuit will be triggered quickly. There is also the risk of heat dissipation under 12V low voltage and high current, so it is difficult to have a long battery backup time using a 12V bus architecture. If a 48V bus architecture is used, since there is a wide voltage tolerance range and the battery can be directly connected to the 48V output bus, the battery backup time can be longer, effectively ensuring the safety and reliability of the system. These features have been fully used and verified in the communications industry.

5. Compare from the cost perspective

In terms of cost, the differences in four parts are mainly considered. We compare them from the perspective of the entire power supply system. The first is the power supply frame. Since 48V power supply is widely used in the communications industry, it is low-cost and the cost per watt is much lower than that of 12V server power supply. Moreover, the total cost difference caused by the need for multiple power supply frames under the 12V architecture and the need for only a single frame under the 48V architecture is not taken into account. The second is the battery BBU. As mentioned earlier, the battery often needs a voltage-regulated DC/DC power supply under the 12V BBU architecture, while the 48V architecture does not require configuration. The 48V architecture also has advantages for this part of the battery BBU. Next is the power transmission and distribution of the busbar. Since the scale and number of copper bars are greatly reduced under the 48V architecture, and only a single frame is required, this part of the 48V architecture also has great cost advantages. The last is the comparison of the VRM power supply on the motherboard. Under the 48V architecture, the server motherboard uses a large number of BMP board brick power supplies to power the CPU, etc., and the cost is not high. Under the 12V architecture, multiple VRM power supplies are also required to power the CPU, and there is not much difference at this level. Therefore, from the previous comparison, the 48V architecture has great advantages.

6. Comparison from the perspective of overall efficiency

In terms of total efficiency, we are still comparing the entire power supply path from the power grid to the CPU. First, in the section from the mains to the busbar, the 48V communication power supply can now easily achieve a high efficiency of 97%, while the 12V server power supply mostly stays at around 94%; secondly, in terms of bus losses, since the transmission and contact impedance losses under the 48V architecture are only one-sixteenth of those under the 12V architecture, including the transmission and contact impedance losses on the busbar section and the server motherboard. This does not take into account the fact that the 48V architecture has less DC/DC losses than the 12V architecture during the charging and discharging process; finally, from the busbar to the CPU, the 48V to 1.3V BMP brick power supply is also more efficient and has less losses than the 12V to 1.3V VRM power supply. The 48V architecture improves the total efficiency by more than 5% compared to the 12V architecture, so the 48V architecture has a great advantage in terms of total efficiency from the mains to the CPU.

7. Compare with other equipment in the computer room such as switches

We know that in addition to servers, there are many other devices such as networks inside the data center, and these relatively small number of devices are usually difficult to customize. And we know that many network devices themselves have the option of 48V power supply, that is, many other IT equipment in the computer room itself supports 48V power input, and only need to be configured with a power module that supports 48V input. If the 12V power supply frame + battery BBU power supply architecture is adopted, there is basically no IT equipment that directly supports 12V input. Then it is not realistic to customize 12V power supply specifically for this part of the computer room IT equipment, and in the case that the entire computer room uses direct power supply from the mains, it is also very complicated and uneconomical to configure a large UPS specifically for a small number of network devices. Therefore, the use of 48V power supply frame + battery BBU architecture has a very large development advantage over the 12V architecture. #p#

How to develop 48V power supply architecture

After hundreds of years of development, the communications industry has now adopted a very standard and universal 48V power supply architecture. The industry chain and ecosystem are very mature, so that most network devices can directly support 48V DC power supply. Even the data center industry has tried to adopt a 48V power supply architecture before, but the traditional approach is to learn from UPS and put the 48V power supply and battery in the power room and battery room of the data center. The problem with this approach is that traditional server equipment often lacks a power supply that directly supports 48V input, and the power cable investment, transmission line voltage drop and transmission cable loss caused by 48V power supply at low voltage and long distance transmission, ultimately the traditional 48V power supply technology has not been developed in the data center industry. However, with the development of technologies such as whole cabinets, and the pursuit of low cost and high efficiency, direct power supply technology, high-power cabinets and lithium batteries, the use of a distributed 48V power supply architecture close to the cabinet or even built into the cabinet will be an important direction for future development.

We have introduced the many benefits of 48V power supply architecture over 12V power supply architecture. Now let's continue to explore how to carry out this work. After all, traditional data centers all use 12V power supply architecture, and switching to 48V power supply architecture will definitely bring many challenges. To address these challenges, we continue to conduct in-depth analysis from the power supply frame system, battery BBU technology, and the power supply replacement of 48V input to CPU path on the server motherboard.

1. 48V power frame system

Due to the widespread operation of 48V power supply technology in the communications industry, the cost of communications power supply is currently very low, and the efficiency can reach an ultra-high efficiency of 97%. In addition to decades of 48V power supply operation experience and mature battery management technology, the accumulation of these communications industry can be directly copied to the 48V power supply architecture of the data center industry. In general, 48V power supply technology is a mature technology with low cost, high efficiency, good management, and easy operation and maintenance. For example, the low-cost single-module 3KW standard 50A communications power supply widely used in the industry is used as a 48V power plug-in frame. Using 2+1 configuration for low-power cabinets within 6KW, 5+1 configuration for medium and large power cabinets of 15KW, and 8+1 configuration for ultra-high power cabinets of 24KW can achieve three-phase balance and only occupy about 2U of cabinet space. The universal power plug-in frame can also be used to configure the number of modules according to actual power requirements. The energy-saving sleep function can be used to achieve a high efficiency of more than 96% in the full load range, which is more than 15% higher than the total efficiency of traditional UPS plus 12V server power supply.

2. 48V lithium battery technology

After the development of battery energy storage, electric vehicles, consumer electronics and other industries, lithium battery technology has grown rapidly in the past few years. The safety has also been greatly improved, and the current price has also been greatly reduced. Considering its excellent discharge capacity, high temperature resistance, simple maintenance, high energy density and light weight. It is a research hotspot in many industries and is very suitable for the application scenarios of future data centers. It can be used to replace traditional lead-acid batteries. Considering its long service life, the cost is similar to that of lead-acid batteries from the perspective of ROI. 48V lithium battery BBU technology has been widely studied and applied in the communications industry and electric vehicle battery modules. It is now relatively mature and has accumulated a lot of industry, technology, and operation data. If a 2U to 3U high 48V battery BBU is configured next to the 48V power supply frame of the whole cabinet (+ WeChat to follow the network world), it can provide a stable 48V power supply for the whole cabinet and provide 5 to 15 minutes of battery backup power, which can bring great innovation to the power supply architecture of the data center. There is no need for traditional UPS investment, power room and battery room space, and there is no longer a lot of loss wasted on UPS and inefficient 12V power supplies. Cabinet-level growth and investment can be achieved, 48V power modules can be configured on demand, automatic energy-saving hibernation can be achieved, and power modules and battery BBU can be hot-swapped and replaced like a hard drive, ultimately achieving low-cost, high-efficiency, easy-to-maintain and manageable power supply for IDC.

3. From the BMP and VRM power supply level

Let's move on to the server motherboard power supply level. Traditional server motherboards are powered by 12V, and require auxiliary outputs such as 12Vstandby for management and control. Then a bunch of VRM power supplies on the server motherboard convert the 12V input into low-voltage 5V, 3.3V, 1.3V and other different outputs to power different units such as the hard disk, memory and CPU. The overall cost is actually not low and the power supply takes up more space on the motherboard.

In addition, these VRM power supplies are usually inefficient and waste a lot of energy. We know that the traditional PUE measurement standard does not actually take into account the energy loss caused by the power supply on the server motherboard. If these VRM power supplies are only 50% efficient, even if the PUE at the computer room level is 1.0, half of the energy will be wasted in the final stage. Therefore, the VRM power supply of the traditional 12V architecture is not optimized enough and has a lot of room for improvement. If the existing server motherboard can be changed to adopt an innovative 48V input power supply architecture, then what really needs to be done is to replace these VRM power modules with the industry's very common and mature 48V input BMP on-board power modules. Directly reduce the 48V input to 5V, 3.3V, and 1.3V, etc., and these BMP on-board modules are usually more efficient than traditional VRM power supplies. The large-scale application price may also have advantages, and there is not much impact on the existing server motherboard. Because these BMP on-board power modules can be installed on the motherboard in the form of 1/4 and 1/8 brick modules, using patch or welding processes, there is no need for too much power debugging and motherboard layout changes, it is very simple to use, and it is highly replicable and versatile.

Through the comparison and analysis of the previous sections, the distributed 48V power frame and battery BBU architecture is not difficult to implement from the perspective of technology maturity and industrial chain, but the benefits it brings are huge. At present, companies such as Google and Facebook have or are doing research and use in this area. We believe that it will be the next important development direction of the power supply architecture of the data center industry at home and abroad.

Another possible technical route

As we mentioned earlier, the development of UPS technology depends largely on the level of battery technology, which has led to different technical routes such as flywheel energy storage UPS, traditional lead-acid battery UPS, and distributed lithium battery UPS (mobile phones or laptops that use a lot of lithium batteries are some good application cases). Especially with the continuous development of battery technology in the future, there will definitely be new battery technologies to innovate and replace existing UPS.

But as of today, the low-cost lithium batteries widely used in the consumer electronics industry may bring technological innovation to traditional UPS, or at least accelerate the pace of improvement of traditional UPS. For example, the mobile phone batteries and laptop batteries mentioned above are some good examples. We know that even if we leave the power grid, mobile phones or laptops can continue to work until the next charge. UPS in data centers can continue to power the server in the event of a power outage until the diesel generator starts to continue to power the data center. Therefore, according to this idea, if we also install lithium batteries on IT equipment like mobile phones or laptops, we can continue to make calls and surf the Internet like mobile phones, and continue to work like laptops in the event of a short power outage. In the event of a long power outage, the diesel generator can be used to ensure continuous power supply, which can ensure the continuous operation of the business. So why do we still need traditional UPS?

Installing batteries inside servers is not a new idea. Google used it nearly 10 years ago, but at that time, lithium battery technology was not as mature and low-cost as it is today. At that time, Google used traditional small lead-acid battery modules and placed them at the tail of the server for power-off protection, achieving 99.9% power supply efficiency on the data center side. However, the series of challenges brought by lead-acid batteries, such as large size, temperature sensitivity, leakage risk, and poor discharge capacity, may only be solved by Google itself, and even Google may no longer use this technology today. Because now there is lithium battery technology with lower cost, high energy density, large current discharge capacity, good high temperature characteristics, and maintenance-free, it can completely replace lead-acid batteries for power-off protection inside servers. If the performance requirements are not very high, you can even use low-cost disposable lithium batteries to match the life cycle of the server, and only need to ensure continuous power supply for three or four years. Decommission and scrap directly with the server, buy as many as you need, and use direct power supply from the mains without UPS and maintenance, just like today's mobile phones and laptops. For example, we can see today that the price of lithium batteries in Xiaomi power banks has become surprisingly low (not necessarily applicable to data center scenarios). If a battery worth tens or hundreds of RMB is embedded in a server, the increase in the overall cost of the server is almost negligible, but the benefits brought to the data center are enormous.

Based on the previous ideas, if IT equipment adopts the embedded lithium battery solution, where should the battery be installed? Or how can it be carried out more easily? Naturally, we will think of two solutions: the battery can be installed outside the server motherboard and installed on the server motherboard. For the first solution, if the battery is installed outside the server motherboard, then using a battery module with the same package as the server power supply will be the simplest approach. Because the server is usually equipped with two power modules, if one of the power modules is replaced with a battery module of the same appearance and the same PIN pin as the original power module is used, it is equivalent to the battery module and the power module being directly connected in parallel, similar to power redundancy backup. Under normal circumstances, the mains power is used to directly supply power to the server motherboard and charge the battery module. Under abnormal circumstances, the backup battery immediately assumes the entire server motherboard load until the diesel generator starts to continue to supply power to the server motherboard from the power module and recharge the battery. Under this solution, there is basically no need to make any modifications to the existing server, just replace the battery module, which is the simplest to carry out, as shown in Figure 11.

If the second solution of installing the battery on the server motherboard is adopted, some adjustments need to be made inside the server. The battery can be installed in a free space outside the server motherboard, or it can be installed directly on the server motherboard. The specific choice can be flexibly made according to the actual situation, but it is recommended to adopt a hot-swappable replacement method. The figure below is a battery-built-in server prototype displayed by Intel at IDF a few years ago. The battery is directly installed on the 12V input of the server, away from the server's heating area, and supports plug-in installation and replacement in the form of battery modules.

[[125603]]

Conclusion

The power supply technology of data centers depends largely on the development level of battery technology. Based on the review of various power supply technologies in data centers, this article believes that the use of distributed power supply will become an important direction for future development. In addition, the use of 48V power supply architecture in the future will have more advantages than the 12V power supply architecture, and it is not ruled out that there will be the development direction of embedded battery technology for IT equipment in the future. However, the embedded battery solution for IT equipment will be more difficult to implement because it involves the customization and transformation of various equipment manufacturers, various models of servers and network equipment in the data center. In addition, each server needs to be equipped with a server power supply and an embedded battery pack, and the overall cost and overall efficiency are not superior. The innovative architecture of 48V power supply frame plus battery BBU is based on the current popular whole cabinet technology. It is low-cost, high-efficiency, easy to maintain, manageable, and can be quickly and massively delivered to the business, reducing the number of power supplies for distributed servers and improving overall energy efficiency. It has great advantages. Therefore, we believe that the overall better 48V power supply plus BBU architecture will be an important direction for the development of power supply technology in data centers in the future.