David Patterson: Challenges and Research Directions for LLM Inference Hardware

Summary

[Submitted on 8 Jan 2026 (v1), last revised 14 Jan 2026 (this version, v2)] Title:Challenges and Research Directions for Large Language Model Inference Hardware View a PDF of the paper titled Challenges and Research Directions for Large Language Model Inference Hardware, by Xiaoyu Ma and David Patterson View PDF Abstract:Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI trends, the primary challenges are memory and interconnect rather than compute. To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication. While our focus is datacenter AI, we also review their applicability for mobile devices. Submission history From: Xiaoyu Ma [view email] [v1] Thu, 8 Jan 2026 15:52:11 UTC (832 KB) [v2] Wed, 14 Jan 2026 20:37:46 UTC (983 KB)