Artificial Intelligence and High Computing Systems
Two themes: the split of GPUs into AI and HPC processors and the use of artificial intelligence (AI) in HPC. Two weaker semi-themes at SC25 were quantum computing going mainstream and power and cooling. A
AMD announced that the next generation of their processors/accelerators will have two versions: The AMD MI455X GPU for AI and the AMD MI430X processor for double-precision floating point (float64 or FP64)-oriented code.
Data centers with the MI430X would more likely run HPL for the TOP500, which doesn’t mean the MI455X can’t run HPL. The MI455X data centers might choose not run it, either because the chip is designed for something else, so it might not perform as well as the MI430X, or the users of the ML455X have no interest in the HPL benchmarks.
Note that you are now seeing the same GPU product line bifurcate, with one GPU focused on AI and another on HPC.
The AI market has gotten big enough that it can demand its own accelerator, and the GPU manufacturers can make money from them.
Equally interesting is that the HPC market remains large enough for targeted GPUs – and the manufacturers can still make money from them.
AI in HPC
HPC is looking at AI as a new technology that can be used to help solve problems. One way AI can help HPC is the use of multiprecision (i.e., a mixture of precisions used in the same code to solve a problem as efficiently as possible).
Multiprecision
Multiprecision can reap great benefits by reducing the amount of data to be stored, because lower precision is used, which can improve performance. Who wouldn't like this one-two punch of less data to store and faster computation?
However, you must be sure your code produces the “correct” answer with these changes. A key aspect is that you don’t have to get the same answer as higher precision, but you must get an answer that satisfies the original problem and has an acceptable error.
For example, if the answer to a problem is supposed to be 10.84591045, is 10.84510439 acceptable? They are different answers, but is the second one correct? That decision needs to be made on the basis of criteria by the user or developer.
In a general sense, AI-oriented GPUs use less FP64 and more of other precisions (e.g., single-precision floating point (float32 or FP32), half-precision floating point (float16 or FP16), 8-bit-precision floating point (bfloat16 or BF16), etc.) to compute their answers.
AI doesn't necessarily need lots of FP64 computing power, but it can use lots of lower precision for computations. The Ozaki scheme allows you to use these lower precision compute units to achieve higher precision results, including FP64.
You can even use 8-bit integer (INT8) tensor cores for double-precision general matrix multiplication (DGEMM) computations that surpass the performance of FP64 units, and there are discussions about more emulations for DGEMM functions.
A great example of an algorithm that has been adapted to use lower precision units is HPL-MxP (a mixed-precision HPL benchmark), as well as work on other common algorithms such as fast Fourier transforms (FFTs) or eigenvalue computations.
AI for Surrogate Modeling
The use of AI, particularly large language models (LLMs), for surrogate models is an increasing area of research.
The use of surrogates is not a new concept.
The concept is: Rather than run a great number of full-fidelity simulations requiring enormous amounts of computational power and time, you run a select subset of the simulations and then create some sort of model that fits these points. You then use the model to find solutions for specific problems.
For example, if you are interested in MDO solutions of aircraft, you would run a set of high-fidelity simulations with techniques from design of experiments, Latin hypercube sampling, or Kriging methods. Next, you create a response surface from these data points and use it as a substitute for a real set of simulations – but with a huge speedup.
With LLMs, the approach is the same, but the response surface is replaced with an LLM model. Once trained, you then run inference to get the simulation result expected at that data point. More recently, this model is now called a digital twin. Examples of this approach were presented at SC25 by NVIDIA (https://blogs.nvidia.com/blog/apollo-open-models/.)
An interesting prospect for a developer is to push the model to produce results outside the range of tested parameters in the model.
You might get answers that aren’t correct or are nonsensical, but then again, you can get those types of answers even if your variable values are inside the parameter range. In fact, you might find something new, interesting, and innovative by going outside the parameter ranges. However, be sure to check these models to make sure they still satisfy reality, including running full simulations.
Quantum Computing, Power, and Cooling
Quantum computing seemed more like research than a product, despite the claims of vendors. Experts projected that in three to five years, quantum computing will be mainstream, and users will be able to schedule jobs on quantum computers just like any other node in the system.
Power and Cooling
Like it or not, power and cooling are now a fundamental part of system design. Before, you had CPUs, memory, networking, and storage. Then, accelerators, primarily GPUs, were added to the mix. Now, as more power is used in CPUs and GPUs and as the number of GPUs in the system increase, power and cooling are added to the design variables. Planning how you will power a system or build a system within a power limit is just as important as the network design.
Power sources are increasingly becoming a big question. Data centers are putting up wind turbines and solar arrays as fast they can be built, but it is still not enough. Data center builders are turning to nuclear energy for power, even spending the money to reopen previously closed nuclear power plants. Small nuclear reactors (SMRs) are being developed that take much less time to install and are mostly self-contained, and there is talk of re-using decommissioned nuclear power plants from navy submarines and ships for power generation because they are no longer used to power naval vessels.
Other sources of power such as large fossil fuel-powered generators are being used for data centers. Some companies are developing and are now offering jet engines to provide power. Aircraft manufacturer Boom Supersonic is developing natural gas turbines for power generation. These engines spend a great deal of their time running at full power, so they are designed for these operating conditions.
Equal to the importance of power is cooling. Do you have enough air-cooling capacity for your system design, or do you need to add more? Does cooling become a limiting factor for system design? Many system manufacturers are turning to liquid cooling their nodes. You must carefully plan for liquid cooling, just like air cooling, but with the addition of monitoring the system for leaks.
Companies are looking at underwater data centers for cooling and even data centers in space. Both have promises as well as challenges. Perhaps more conventionally, companies are also looking to use waste heat from data centers for various tasks, including power generation.
Reference: https://www.admin-magazine.com/Articles/SC25-Highlights/?utm_source=AU
Comments
Post a Comment