China Sunway Supercomputer Team Claims 174 Trillion Parameter AI Model

A team of researchers from China used the Sunway supercomputer to train an AI model with 174 trillion parameters called ‘bagualu,’ which means “alchemist’s pot.” The AI parameters are comparable to the number of synapses (1000 trillion in the human brain). However, human synapses and AI parameters are not equivalent. In 2020, Microsoft trained a…
China Sunway Supercomputer Team Claims 174 Trillion Parameter AI Model


A team of researchers from China used the Sunway supercomputer to train an AI model with 174 trillion parameters called ‘bagualu,’ which means “alchemist’s pot.” The AI parameters are comparable to the number of synapses (1000 trillion in the human brain). However, human synapses and AI parameters are not equivalent.

In 2020, Microsoft trained a natural language model using 17 billion parameters.

In2021, Google announced an AI model trained with 1.6 trillion parameters.

The Sunway supercomputer has a speed of a billion operations per second, or 5.3 floating-point operations per second (exaflops). It is using FP32 single precision operations. According to the researchers, it has 37 million CPU cores — four times as many as Frontier — and nine petabytes of memory. They also claim the 96,000 semi-independent computer systems, called nodes, resemble the power of a human brain. Communications between these nodes take place at a speed of more than 23 petabytes per second.

Here is a research paper describing the pre-training of the BaGuaLu model.

BaGuaLu: Targeting Brain Scale Pretrained Models with over 37 Million Cores

Abstract


Large-scale pretrained AI models have shown state-of-theart accuracy in a series of important applications. As the size of pretrained AI models grows dramatically each year in an effort to achieve higher accuracy, training such models requires massive computing and memory capabilities, which accelerates the convergence of AI and HPC. However, there are still gaps in deploying AI applications on HPC systems, which need application and system co-design based on specific hardware features.

To this end, this paper proposes BaGuaLu, the first work targeting training brain scale models on an entire exascale supercomputer, the New Generation Sunway Supercomputer. By combining hardware-specific intra-node optimization and hybrid parallel strategies, BaGuaLu enables decent performance and scalability on unprecedentedly large models.

The evaluation shows that BaGuaLu can train 14.5-trillion parameter models with a performance of over 1 EFLOPS


using mixed-precision and has the capability to train 174-trillion-parameter models, which rivals the number of synapses in a human brain.

Read More

Total
0
Shares
Leave a Reply

Your email address will not be published.

Related Posts
DARPA AI Contest to Assess Critical Minerals
Read More

DARPA AI Contest to Assess Critical Minerals

DARPA has partnered with the U.S. Geological Survey (USGS) to explore the potential for machine learning and artificial intelligence tools and techniques to accelerate critical mineral assessments. The goal is to significantly speed up the assessment of the nation’s critical mineral resources by automating key steps in the process. Assessments can quantify potential mineral sources…
Nuclear War Analysis
Read More

Nuclear War Analysis

The current nuclear arsenal will not kill all humans and the pattern of nuclear explosions for a nuclear war between the largest nuclear powers will not destroy civilization, let alone kill all people or even half of all people. The greatest risks from a total nuclear war are from fire and starvation and not from…
BYD and Tesla in Real Battle for Top Electric Car Company
Read More

BYD and Tesla in Real Battle for Top Electric Car Company

BYD is aiming for 280,000 monthly deliveries of battery electric (BEV) and hybrid electric cars (PHEV) by the end of the year. BYD sold and delivered 174,915 hybrids and BEV in August. In August, BYD had 82,678 BEVs which was nearly triple the 30,382 units in the same month in 2021. Plug-in hybrid vehicles were…
Lucid Motors Delivers 125 Cars in 2021
Read More

Lucid Motors Delivers 125 Cars in 2021

Lucid Motors had a reduced 2021 delivery target of 520 cars but they delivered only 125 cars. Another 175 cars have been delivered so far in 2022. Most of the cars have been recalled due to safety issues with the suspension. Lucid Motors might burn through its $6 billion in cash in two years. Lucid…