Apple M4 NPU/GPU Goal posts

The talk has started that Apple will announce the M4 processor at WWDC 2024. It seems Apple is now on a yearly cadence for its processors.

The industry, reacting to AI, seems to be converging on a 40 TOPs performance level for NPUs capable of running specially developed LLM models locally. Intels latest laptop chips are around the 10 mark and Qualcomms Snapdragon Elite is at 45 currently and shipping in laptops later in 2024.

Apples M3 is at 18 (their own FP16 number). So, one can imagine the M4 besides switching node types is going to need at least a 2x on the NPU. It’s important to realize that they need 2x just to match Qualcomm today, not QualComms CPU next year. Intel is behind but says they will also have a 40 TOP laptop processor in 2025. So, if Apple wants to keep up with Intel in 2025 then 40 is the answer but I would argue keeping up with Intel is not the target they should be going after… QualComm (on recent form) may at be 60–80 TOPs in 2025.

However, as far as I can see, current NPUs cannot run the whole LLM, they run a portion and the CPU shoulders the rest. This complicates running models on an NPU, an LLM developer needs to figure out what to run on the NPU and what to run on the CPU. It should be possible for Intel/Microsoft/Apple to do this for flavors/architectures of LLMs automatically but they do not right now so it falls on LLM developers. LLM developers are choosing to ignore NPUs as a result and do the easy thing, run it on the GPU which can run everything. Mobile vendors seem likely to…