GATHER: A Gated-Attention Accelerator for Efficient LLM Inference

Large Language Models (LLMs) have become piv-otal, yet their auto-regressive inference suffers from significant memory bandwidth bottlenecks, hindering performance and energy efficiency. In this paper, we propose GATHER, a novel hardware accelerator architecture specifically designed for efficient generative AI inference. GATHER introduces two key contributions: (1) A token-stream processor that natively han-dles variable-length sequences, completely eliminating padding-related overhead. (2) A specialized Gated-Gather Engine that tackles the attention bottleneck by tightly coupling Top-K attention score selection with a dedicated address gather unit. This engine identifies the most salient tokens and issues optimized, batched memory requests to DRAM, drastically reducing off-chip traffic. Evaluation results show that our proposed architecture outperforms a single NVIDIA A100 GPU on GPT-2 and Llama-3-8B in terms of throughput and energy efficiency.