What does "NPU" stand for? What can it do? What you need to know about this emerging technology.

Over the past year, there has been a lot of talk about neural processing units (NPUs). While NPUs have been available in smartphones for a few years, Intel, AMD, and more recently Microsoft have all launched AI-enabled consumer laptops and PCs equipped with NPUs.

NPUs are closely associated with the concept of AI PCs, and they are increasingly being used in chips produced by major hardware manufacturers such as AMD, Apple, Intel, and Qualcomm. NPUs have started to appear more and more in laptops since Microsoft launched its Copilot+ AI PC product earlier this year.

01 What does NPU do?

What an NPU does is act as a hardware accelerator for AI. Hardware acceleration is the use of specialized silicon to manage specific tasks, much like a head chef delegates different tasks to sous chefs so they can work together to prepare a meal on time. An NPU doesn't replace your CPU or GPU; instead, it's designed to complement the strengths of your CPU and GPU, handling workloads like edge AI so that your CPU and GPU can retain processing time for tasks they excel at.

GPUs are hardware accelerators designed specifically for rendering graphics, but have enough underlying flexibility to also be well suited for AI or certain types of scientific computing. It's long been the case that if you had an AI workload you wanted to work on, you wanted to use one or more high-performance [probably Nvidia?] GPUs to do the actual number crunching. Some companies are working on building specialized hardware accelerators specifically for AI, like Google's TPU, because the added graphics capabilities with the "G" in "GPU" aren't useful in a card designed purely for AI processing.

02 Workload is everything

Hardware acceleration is most useful in repetitive tasks that don't involve a lot of conditional branching, especially when the data volume is large. For example, rendering 3D graphics requires the computer to manage a constant stream of countless particles and polygons. It's a bandwidth-intensive task, but actually computes (mostly) trigonometric functions. Computer graphics, physics and astronomy calculations, and large language models (LLMs) like those that power modern AI chatbots are a few examples of workloads that are ideal for hardware acceleration.

There are two types of AI workloads: training and inference. Training happens almost entirely on GPUs. Nvidia has dominated both markets, leveraging its nearly two decades of investment in CUDA and its leadership in discrete GPUs, though AMD has fallen a distant second. Large-scale training happens at datacenter scale, as do inference workloads that run while you're communicating with cloud-based services like ChatGPT.

NPUs (and the AI PCs that connect to them) operate at a much smaller scale. They can complement the integrated GPUs in microprocessors from your favorite CPU vendor, providing additional flexibility for future AI workloads and a performance boost over waiting for the cloud.

03 How does the NPU work?

In general, NPUs rely on a highly parallel design to quickly perform repetitive tasks. In contrast, CPUs are generalists. This difference is reflected in the logical and physical architecture of NPUs. CPUs have one or more cores that access a small shared memory cache, while NPUs have multiple subunits, each with its own micro-cache. NPUs are suited for high-throughput and highly parallel workloads such as neural networks and machine learning.

NPUs, neural networks, and neuromorphic systems such as Intel's Loihi platform all share a common design goal: to emulate certain aspects of the brain's information processing.

Each device manufacturer that brings an NPU to market has its own microarchitecture specific to its product. Most manufacturers also release software development tools to work with their NPUs. For example, AMD offers the Ryzen AI software stack, while Intel continues to improve its ongoing open source deep learning software toolkit OpenVINO.

04 NPU and Edge Intelligence

Most NPUs are installed in consumer-oriented devices such as laptops and PCs. For example, Qualcomm's Hexagon DSP adds NPU acceleration to its Snapdragon processors for smartphones, tablets, wearables, advanced driver assistance systems, and the Internet of Things. The Apple ecosystem uses its Neural Engine NPU in its A-series and M-series chips for iPhones, iPads, and iMacs. In addition, some PCs and laptops are designated as Copilot+, which means they can run Microsoft's Copilot AI on the onboard NPU. However, some server-side or cloud-based systems also use NPUs. Google's Tensor Processing Units are NPU accelerators designed for high-performance machine learning in data centers.

One reason for the rise of NPUs is the growing importance of edge intelligence. There is already a growing need to collate data between sensor networks, mobile devices (such as phones and laptops), and the Internet of Things. At the same time, cloud-based services are subject to infrastructure latency. Local processing does not necessarily have to do everything in the cloud. This can be an advantage in terms of speed and security.

The question of whether you need an NPU is almost a red herring. Silicon Valley giants like Intel, AMD, and Apple have already invested in the technology. Whether you have a specific use for an NPU or not, the next time you build or buy a PC, there's a good chance the chip you choose will have an NPU. By the end of 2026, analysts expect 100% of U.S. enterprise PC purchases to have one or more NPUs embedded in the chip. In other words, don't worry about not being able to buy a system with an NPU. They'll find you.

View more at EASELINK