Source: Content compiled from Nikkei

Amazon has stopped developing its Inferentia AI chip and is instead focusing on semiconductors for training AI models, an area the company believes offers a better way to compete with AI chip leader Nvidia.

Since announcing its entry into the AI chip space in 2018, the company's cloud division, Amazon Web Services (AWS), has developed custom chips for inference prediction, Inferentia, and for AI model training, Trainium.

Amazon's cloud service has made both product lines available to AI companies. But AWS computing director Rahul Kulkarni told Nikkei and other news outlets on Wednesday that "the product lines will converge."

"We will focus on Trainium, providing both inference and training performance," Kulkarni said.

AWS will continue to support the current product line, including the Inferentia2 chip released in 2022, "but we will not actively develop any further features on Inferentia," Kulkarni said.

Trainium has greater memory capacity, supports a wider range of data formats, and also contains mechanisms for fast computation and communication when using multiple servers to process large amounts of data simultaneously.

Inferentia focuses on performing fewer computations at a lower cost. As generative AI becomes more advanced, both training and inference require more computation. As a result, the benefits of using different chips are fading.

"By offering Trainium as a single, unified product, we can achieve the same level of optimization and cost-effectiveness," Kulkarni said.

On Tuesday, at its annual tech event in Las Vegas, Amazon announced that the Trainium 2 chip is now available. The company also unveiled plans to release the third-generation Trainium 3 chip in the second half of 2025.

According to Amazon, Trainium 3 will be manufactured using an advanced 3-nanometer chip manufacturing process and will have twice the computing performance of Trainium 2.

Amazon's goal is to break Nvidia's dominance in the AI chip market, which currently accounts for about 90% of the world's market share. Amazon has been heavily promoting Trainium 2 to cloud customers, emphasizing that the chip costs less to run than Nvidia's products.

On Tuesday, Apple said it would use Trainium 2 chips for its own AI development. Anthropic, a US AI startup that is receiving an $8 billion investment from Amazon, is acquiring hundreds of thousands of Trainium2 chips to assist in the development of its generative AI models.

AWS's AI chip: What's different?

In two small rooms in an unmarked office building in Austin, Texas, a handful of Amazon employees have designed two microchips for training and accelerating generative AI. These custom chips, Inferentia and Trainium, offer AWS customers an alternative to training large language models on Nvidia as GPUs become increasingly difficult and expensive to procure.

"The world wants more chips for generative AI, whether they're GPUs or chips designed by Amazon itself," Amazon Web Services said. "I think we're better positioned than any other company on the planet to provide the capacity that our customers collectively need."

While AWS isn't making rapid progress with large models, Amazon's custom silicon could give it an edge in generative AI in the long run.

"I think the real difference is the technical capabilities they have," he said. "Because guess what? Microsoft doesn't have Trainium or Inferentia," he said.

AWS quietly began producing custom silicon as early as 2013, with a specialized piece of hardware called Nitro. Now, it's the highest-volume AWS chip. Amazon told CNBC that there is at least one such chip in every AWS server, and the total number of servers currently in use is more than 20 million.

In 2015, Amazon acquired Israeli chip startup Annapurna Labs. Then in 2018, Amazon launched Graviton, an Arm-based server chip that competes with x86 CPUs from giants such as AMD and Intel.

"Arm probably has about 10% of total server sales, and a lot of that will come from Amazon. So on the CPU side, they're doing pretty well," said Stacy Rasgon, senior analyst at Bernstein Research.

Also, in 2018, Amazon launched a chip focused on AI. Two years ago, Google announced the launch of its first tensor processor unit (TPU). Microsoft has yet to announce the Athena AI chip it has been working on, which it reportedly developed in partnership with AMD.

CNBC visited Amazon's chip lab in Austin, Texas, where Trainium and Inferentia are being developed and tested. Vice president of product Matt Wood explained what the two chips are for.

"There are these two different phases of machine learning. So you train machine learning models and then you do inference on those trained models," Wood said. "Trainium offers about a 50% improvement in price/performance over any other way to train machine learning models on AWS."

Trainium first became available in 2021, following the release of Inferentia, now in its second generation, in 2019.

Inferentia allows customers to "provide very low-cost, high-throughput, low-latency machine learning inference, which is all the predictions when you input a prompt into a generative AI model, all of which is processed in the generative AI model and gives a response," Wood said.

For now, though, Nvidia's GPUs still dominate when it comes to training models. In July, AWS launched new AI acceleration hardware with the Nvidia H100.  "Nvidia chips have a huge software ecosystem, and over the last 15 years, it has built an ecosystem around the chips that no other company has," Rasgon said. "The biggest winner in AI right now is Nvidia."

View more at EASELINK