Interesting, I had high hopes for the NPU, given the snapdragon Elite GPU is rather weak.I call BS on all that AI processing stuff. As far as I can tell, somebody got their marketing message mixed up early on, and now the entire industry is stuck with a misleading measure:It's all about the neural processing and the TOPS...I'm still waiting for a satisfactory answer to that question.
These TPUs or NPUs or whatever you want to call them, were originally introduced in phones as a power-saving measure. Their TOPs are 8-bit integer processing operations per second. GPUs don't typically report that, as it's not interesting for games. But they report FLOPs instead, which is 32-bit floating point operations per second. They can typically swap out each 32-bit float for four 8-bit integers. As such, a measly Nvidia 3060 can easily reach those 50 TOPs required for the Copilot+ branding. Thus the TOPs themselves are not impressive. What is impressive, is that a TPU can do 50 TOPs on a smartphone power budget!
Thus these NPUs enable, for instance, transcribing audio continuously in the background, or recognizing text in a continuous stream of screenshots. These were never before feasible without burning your battery.
But the marketing instead focuses on LLMs, which they call "AI". The NPUs, however, are not practical for LLMs. LLMs are limited by memory, not compute. At least at the moment, a good LLM for conversation or image generation is hundreds of gigabytes in size. Woefully too big for any mobile device, NPU notwithstanding. Microsoft is shipping a few "small" SLMs with their Copilot hardware, but they are so underwhelming as to be useless in practice.
Thus, for the time being, NPUs are a power-saving measure for running small machine learning tasks continuously in the background. "AI" things are computed in the cloud.
Last edited:
