AI core shortage, Nvidia sold out

Author: Sun Pengyue Editor: Gale

Source: Zinc Finance

Image source: Generated by Unbounded AI

On August 8, the world computer graphics conference SIGGRAPH, the most important round table conference in the global computer industry, was officially held.

NVIDIA founder and CEO Jensen Huang attended and brought a new generation of NVIDIA super AI chip GH200. Jensen Huang is very confident about his new flagship product, calling the GH200 "the fastest memory in the world."

**In today's AI market, Nvidia is called "the center that runs the entire AI world." **Whether it is OpenAI or Google, META, Baidu, Tencent, Alibaba, all generative AIs rely heavily on Nvidia's AI chips for training.

Moreover, according to media reports, the total market demand for the Nvidia AI chip H100 in August 2023 may be around 432,000 pieces, and the current price of an H100 chip on Ebay has even reached 45,000 US dollars, equivalent to more than 300,000 RMB. Yuan.

There are more than 400,000 chip gaps, with a unit price of 45,000 US dollars, and the total price can easily reach millions of dollars.

Nvidia is experiencing a market wave that is crazier than the "mining era".

AI chip, one is hard to find

The so-called AI chips are actually graphics processing units (GPUs), whose main role is to help run the countless calculations involved in training and deploying artificial intelligence algorithms.

**That is to say, all kinds of intelligent performance of generative AI come from the stacking of countless GPUs. The more chips used, the smarter the generative AI gets. **

OpenAI is tight-lipped about the details of GPT-4 training, but according to media speculation, GPT-4 needs at least 8192 H100 chips, at a price of $2 per hour, pre-training can be completed in about 55 days, and the cost is about $21.5 million (150 million RMB).

According to Microsoft executives, the AI supercomputer that provides computing power support for ChatGPT is a large-scale top-level supercomputer that Microsoft invested 1 billion U.S. dollars in 2019, equipped with tens of thousands of Nvidia A100 GPUs, and more than 60 data centers. In total, hundreds of thousands of Nvidia GPUs are deployed in the center.

The AI chip required by ChatGPT is not fixed, but incrementally increased. The smarter ChatGPT is, the price behind it is that more and more computing power is required. According to Morgan Stanley's prediction, GPT-5 will need to use about 25,000 GPUs, which is about three times that of GPT-4.

**If you want to meet the needs of a series of AI products such as OpenAI and Google, then it is equivalent to a company of Nvidia providing cores for AI products all over the world, which is a great test for Nvidia's production capacity. **

NVIDIA H100

Although Nvidia is producing AI chips at full capacity, according to media reports, the large-scale H100 cluster capacity of small and large cloud providers is about to run out, and the "severe shortage problem" of H100 will continue until at least the end of 2024.

At present, Nvidia's chips for the AI market are mainly divided into two types: H100 and A100. H100 is the flagship product. In terms of technical details, H100 is about 3.5 times faster than A100 in 16-bit reasoning speed, and 16-bit training speed is about 2.3 times faster. times.

Whether it is H100 or A100, they are all co-produced by TSMC, which limits the production of H100. According to some media, it takes about half a year for each H100 to go from production to delivery, and the production efficiency is very slow.

Nvidia has stated that they will increase the supply capacity of AI chips in the second half of 2023, but did not provide any quantitative information.

Many companies and buyers are calling for Nvidia to increase the production of fabs, not only to cooperate with TSMC, but to hand over more orders to Samsung and Intel.

** Faster training speed **

**If there is no way to increase production capacity, then the best solution is to launch chips with higher performance to win by quality. **

As a result, Nvidia began to frequently launch new GPUs to improve AI training capabilities. First, in March this year, Nvidia released four AI chips, H100 NVL GPU, L4 Tensor Core GPU, L40 GPU, and NVIDIA Grace Hopper, to meet the growing computing power demands of generative AIs.

The previous generation has not yet been mass-produced and launched. Nvidia released the upgraded version of H100, GH200, by Huang Renxun at the SIGGRAPH World Conference on Computer Graphics on August 8.

It is understood that the new GH200 Grace Hopper Superchip is based on a 72-core Grace CPU, equipped with 480GB ECC LPDDR5X memory and GH100 computing GPU, with 141GB HBM3E memory, uses six 24GB stacks, and uses a 6144-bit memory interface.

NVIDIA GH200

The biggest black technology of GH200 is that as the world's first chip equipped with HBM3e memory, it can increase its local GPU memory by 50%. And this is also a "specific upgrade" specifically for the artificial intelligence market, because top-level generative AI is often huge in size but limited in memory capacity.

According to public information, HBM3e memory is the fifth-generation high-bandwidth memory of SK Hynix. It is a new type of high-bandwidth memory technology that can provide higher data transmission rates in a smaller space. It has a capacity of 141GB and a bandwidth of 5TB per second, which can reach 1.7 times and 1.55 times that of H100 respectively.

Since its release in July, SK Hynix has become the darling of the GPU market, ahead of direct rivals Intel's Optane DC and Samsung's Z-NAND flash chips.

It is worth mentioning that SK Hynix has always been one of Nvidia's partners. Starting from HBM3 memory, most of Nvidia's products use SK Hynix products. However, SK Hynix has been worrying about the production capacity of the memory required for AI chips, and Nvidia has asked SK Hynix to increase production capacity more than once.

When a large family with dystocia meets another large family with dystocia, people can't help but worry about the production capacity of GH200.

NVIDIA officially stated that compared with the current generation product H100, GH200 has 3.5 times higher memory capacity and 3 times higher bandwidth; and, HBM3e memory will enable the next-generation GH200 to run AI models 3.5 times faster than the current model.

**The speed of running the AI model is 3.5 times faster than that of H100. Does it mean that 1 GH200 is equivalent to 3.5 H100? Everything has to be learned through practice. **

But for now, what is certain is that, as the largest supplier in the AI market, Nvidia has further consolidated its leading position and widened the gap with AMD and Intel.

NVIDIA Rivals

Faced with a gap of 430,000 AI chips, no company is unmoved. In particular, Nvidia's biggest competitors, AMD and Intel, will not let them monopolize the entire market.

On June 14 this year, AMD Chairman and CEO Su Zifeng intensively released a variety of new AI software and hardware products, including the AI chip designed for large language models, MI300X. Officially launched a positive challenge to Nvidia in the AI market.

In terms of hardware parameters, AMD MI300X has as many as 13 small chips, containing a total of 146 billion transistors, and is equipped with 128GB of HBM3 memory. Its HBM density is 2.4 times that of Nvidia H100, and its bandwidth is 1.6 times that of Nvidia H100, which means that the processing speed of generative AI can be accelerated.

But unfortunately, this flagship AI chip is not in stock, but it is expected to be fully mass-produced in the Q4 quarter of 2023.

Another competitor, Intel, acquired artificial intelligence chip maker HABANA Labs for about $2 billion in 2019, entering the AI chip market.

In August of this year, on Intel's most recent earnings call, Intel CEO Pat Gelsinger said that Intel is developing a next-generation Falcon Shores AI supercomputing chip, tentatively named Falcon Shores 2, which is expected to be released in 2026.

In addition to Falcon Shores 2, Intel also launched the AI chip Gaudi2, which has already started selling, while Gaudi3 is under development.

It's just a pity that the Gaudi2 chip specification is not high, and it is difficult to challenge Nvidia H100 and A100.

AMD MI300X

** In addition to foreign semiconductor giants flexing their muscles and starting the "chip competition", domestic semiconductor companies have also started research and development of AI chips. **

Among them, the Kunlun core AI accelerator card RG800, Tianshu Zhixin's Tiangai 100 accelerator card, and Suiyuan Technology's second-generation training product Yunsui T20/T21 all indicate that they can support large-scale model training.

In this battle for chips that uses computing power as the standard and AI large models as the battlefield, Nvidia, as one of the biggest winners in the AI market, has demonstrated its strength in chip design and market share.

However, although domestic AI chips are slightly behind, the pace of research and development and market expansion has never stopped, and the future is worth looking forward to.