In 2023, generative artificial intelligence will set off a global craze, and the competition for large models will become increasingly fierce. In 2024, artificial intelligence will further drive improvements in chip computing power, storage capacity (storage performance) and energy efficiency, promote innovation in semiconductor architecture and advanced packaging, and bring new market growth.

01 Chip architecture moves towards diversification

The emergence of ChatGPT has broadened the market space for AI chips, and the demand for large-scale AI model training has surged. Therefore, high-computing power chips have become the main driving force for the current recovery of the semiconductor industry chain. In the AI wave, Nvidia's data center business in 2023 achieved a year-on-year growth of 217% with its A100, H100 and other GPU (graphics processing unit) products. As of March 6, 2024, its market value exceeded US$210 million.

As the current mainstream processor for AI computing, GPU itself has powerful parallel computing capabilities. However, in the market verification in recent years, it has also exposed problems such as high cost, long delivery cycle and high power consumption. On the one hand, NVIDIA is working hard to shorten the delivery cycle; on the other hand, various companies are innovating chip architecture in order to optimize the power consumption and cost of AI processors.

Therefore, chips such as ASICs that are suitable for specific scenarios have begun to attract the attention of cloud service manufacturers such as Google and Microsoft.

Google has been developing TPU (Tensor Processor) specifically for machine learning since 2016, and uses it as the computing power base of AlphaGo. TPU uses low-precision calculations to ensure deep learning effects while reducing power consumption and improving computing efficiency. The TPU v5p version released by Google in January this year is 2.8 times more efficient in large model training than v4. It is reported that this series of chips will also be used in the training of Google's Gemini AI large model.

On February 19, Groq, a startup founded by engineers from the former Google TPU core R&D team, also opened the experience entrance for its own product LPU (Language Processor). In terms of architecture, Groq's LPU uses TSP (Tensor Flow Processor) to accelerate complex workloads in artificial intelligence, machine learning, and high-performance computing. A Groq spokesperson said that the processor’s reasoning capability is 10 times that of NVIDIA H100.

In addition, as AI penetrates from the cloud to the terminal, many manufacturers believe that NPU (neural network processing unit) is a more suitable technical route for AI computing. Qualcomm's AI PC chip X Elite and Intel Core Ultra processors both integrate NPU to improve AI performance on the computer side.

Qualcomm AI engine integrates CPU, GPU, NPU and other processors 

(Image source: Qualcomm)

The multi-point blossoming of architecture not only reflects the choices of major enterprises between general-purpose chips and special-purpose chips, but also means that suppliers of more chip categories and their upstream and downstream companies have the opportunity to share the dividends of the AI era.

"In the past few years, GPU has remained the main choice for AI computing due to its complete development ecosystem. However, as model parameters continue to increase, chip requirements for computing energy efficiency have increased accordingly. Specialized processors are not suitable for certain AI application scenarios. The advantages will be very obvious. Taking into account various factors such as application scenarios and costs, the selection of hardware chips for AI computing in the future will coexist with multiple technical routes." Jia Tianyu, a researcher at the School of Integrated Circuits at Peking University, told a reporter from China Electronics News .

AI performance drives advanced packaging

While upstream chip design companies are embracing the "tremendous wealth" brought by AI, downstream packaging technology has also gained room for growth.

“Generative AI models require millions or hundreds of millions of parameters for reasoning, which puts forward higher requirements for chip processing speed, capacity, and bandwidth. This will promote advanced packaging technology represented by chiplets. Further development will bring about ecological changes in the packaging industry." Jing Cui, chairman of China Electronics Equipment Group Co., Ltd., told a reporter from China Electronics News.

The computing power of a chip is positively related to the number of transistors. Due to the slowdown of Moore's Law, the area of chips has shrunk to close to the physical limit. Even if manufacturers such as TSMC, Intel IFS, and Samsung Electronics have announced 3nm and more advanced processes, they are still facing the challenge of improving product yields. Therefore, advanced packaging technology and new packaging methods such as SIP (system-in-package) will be an effective way to continue Moore's Law.

Among them, Chiplet technology can be described as the "jewel in the palm".

On the one hand, Chiplets are popular among chip design companies. With its high flexibility, Chiplet not only realizes the stacking of different architectures and different process materials, but also eliminates the need for complex manufacturing processes, making it more cost-effective for design companies. According to institutional statistics, nearly 30% of high-performance CPUs and GPUs use chiplet designs, including computing chip suppliers such as NVIDIA and AMD.

On the other hand, the popularity of Chiplets has also promoted the continuous technological innovation of manufacturing and packaging and testing companies. Intel has joined forces with multiple companies to establish the UCIe (Universal Chip Interconnect Channel) standard for computing block communication within the chip, and has launched packaging technologies such as EMIB and Foveros to seamlessly integrate chips into packages.

TSMC previously collaborated with Xilinx to develop CoWoS packaging technology. Based on chiplets, CoWoS achieves multi-chip packaging, high-density interconnection and reduced power consumption through interconnection silicon interposers. As GPUs take center stage in AI, TSMC's CoWoS production capacity is quickly running out of capacity. At TSMC’s financial report meeting on January 18, President Wei Zhejia said that the demand for advanced packaging of AI chips is very strong, and the current production capacity is still unable to cope with the strong demand from customers, and the shortage of supply may continue until 2025.

Storage original manufacturers compete in production capacity and packaging technology

Computing power is an important support for training large models, while storage performance is closely related to the inference efficiency of large models. In the context of the aggregation of large models and the gradual implementation of AI applications, reasoning capabilities are emphasized by more and more chip and cloud service manufacturers. Therefore, multiple HBMs (high-bandwidth memory) are configured in GPU products to weaken the memory wall effect of the chip in AI computing and further reduce latency.

HBM's competition will become more intense in 2024.

Micron CEO Sanjay Mehrotra revealed at the financial report meeting at the end of 2023 that AI chips have strong demand for HBM, and Micron's HBM production capacity in 2024 is expected to be completely sold out - HBM3E, which will be mass-produced in early 2024, is expected to create hundreds of millions of dollars for Micron. of revenue.

Generally speaking, the higher the number of stacked layers of a memory chip, the stronger its performance, but the more obvious problems such as heat generation and yield rate will be. Therefore, similar to computing power chips, advanced packaging and related technologies have also become an important means to improve the performance of memory chips.

In addition to TSV (Through Silicon Via), a common packaging technology in HBM, Samsung Electronics strives to eliminate the thickness of NCF (Non-Conductive Film) material between stacked layers. It is reported that Samsung Electronics' 12-layer stacked HBM3E uses hot-pressed non-conductive film technology to compress the chip gap to a minimum of 7 microns, making the 12-layer stack consistent with the height of previous 8-layer stacked products.

SK Hynix has developed its own MR-MUF (Match Reflow Molding Underfill) technology, which is different from laying thin film materials on each layer of chips. This technology improves process efficiency and heat dissipation by injecting protective materials between stacked chips. Sun Haorong, vice president of SK Hynix, said: "In order to realize different artificial intelligence applications, the characteristics of artificial intelligence memory should also be diversified. Our goal is to respond to these changes with various advanced packaging technologies."

Material and architectural innovations enable low power consumption

In addition to improving computing power and storage capacity by adjusting the architecture and advanced packaging technology, AI chips also need to consider power consumption factors.

On the one hand, in data centers, the gradual increase in power consumption of AI servers will give rise to new solutions. It is understood that the power consumption of Nvidia H100 has reached 700W, and the power consumption of the B100 that will be launched later will increase by another 40%, which drives the existing cooling technology to be further optimized. NVIDIA CEO Jen-Hsun Huang previously revealed that the company's next-generation products will use liquid cooling solutions. Dell COO Jeff Clarke also said that "the engineering team is ready for (NVIDIA) this new product to bring high power consumption to the GPU." cooling solutions".

Since the development of AI requires a large amount of computing power, the demand for electricity has also surged. At this time, wide bandgap semiconductors and energy storage will also play a role. "The operation of large-scale computing infrastructure requires higher power and higher energy efficiency power electronic equipment to support it. This will be a new growth point for the wide bandgap semiconductor market such as silicon carbide and gallium nitride." Shenzhen Basic Semiconductor Co., Ltd. General manager He Weiwei told a reporter from China Electronics News, "In addition, the development of AI technology in the future will be highly dependent on energy, especially the advancement of photovoltaic and energy storage technology, which is also closely related to the semiconductor industry."

On the other hand, AI is experiencing penetration from the cloud to the end, and the end side pays more attention to the need for low power consumption. In-memory computing will likely become a new paradigm for AI computing at the edge.

"Compared with the large computing power GPU in the cloud, terminal intelligent computing chips pursue low power consumption and low cost while ensuring performance. New technologies such as storage and computing have the advantage of low power consumption and are expected to be used in a wide range of edge intelligence in the future. play a role in calculations." Jia Tianyu said.

Currently, in-memory computing has been used in intelligent terminal devices such as vision and health, and is expected to be applied in edge side, autonomous driving and even data centers in the future. According to an institutional report, the market size of in-memory computing will be nearly US$17.7 billion in 2023 and will reach US$52.6 billion in 2030, with a compound annual growth rate of 16.8%.

In-memory computing market size forecast (Image source: SNS Insider)

For in-memory computing, market prospects and technical challenges coexist. Wang Shaodi, founder and CEO of Zhicun Technology, told a reporter from China Electronics News: "Take a large AI model running on a mobile phone as an example. Now the 16GB LPDDR5 in the mobile phone has exceeded 70 US dollars, and the bandwidth is 70GB/s (compared to a cloud server bandwidth of nearly 1TB/s) is not easy to increase in a short period of time, and the expansion of bandwidth will inevitably lead to an increase in power consumption. Compared with traditional architecture AI chips, in-memory computing has greater energy efficiency in terms of cost, capacity, bandwidth and power consumption. Advantages, although it is still very challenging to meet the model computing power requirements on the edge side in a short period of time and achieve good application effects, it is a very worthwhile thing to do.”

View more at EASELINK