vLLM项目在AMD GPU平台上成功实现了投机解码技术1。这一方法通过轻量级草稿组件提出候选token，再由目标模型在单次验证中从左到右逐一接受或拒绝这些token，从而在单轮推理中提交多个token，显著提升大语言模型的推理吞吐量1。
研究涵盖了5种草稿方法的对比测试1。其中原生MTP与Gemma 4 MTP采用逐序生成方式；EAGLE-3从目标模型隐层状态自回归草稿；DFlash实现并行预测整个token块；DSpark在DFlash基础上融合轻量级序列头与置信度选择机制1。实验在Gemma-4-26B、Gemma-4-31B、Qwen3系列、Kimi-K2.5、MiniMax-M3等多个模型上进行，运行环境为AMD Instinct MI300X和MI355X GPU1。
性能提升方面，Gemma-4-26B-A4B-it模型在MATH500数据集上使用DFlash方法达到2.87倍吞吐量提升，Gemma 4 MTP在GSM8K达到2.74倍；Kimi-K2.5模型在DFlash方法下达到2.68倍1。研究发现num_speculative_tokens参数对性能影响显著，最优值通常在N=3-7范围内，具体取值因模型、数据集和草稿方法而异1。草稿token的接受率是决定性能的关键指标，后续位置的接受率下降可能导致吞吐量增长平台化或下降1。
Speculative decoding has been implemented in vLLM on AMD GPUs, offering significant performance improvements for large language model inference 1. The technique operates through a two-stage process where a lightweight drafting component proposes multiple candidate tokens, which are then verified by the target model in a single pass, enabling the acceptance or rejection of multiple tokens from left to right 1. This approach substantially increases throughput compared to conventional sequential token generation.
The implementation encompasses five distinct drafting methods, each with different architectural approaches 1. Native MTP and Gemma 4 MTP generate candidate tokens sequentially, while EAGLE-3 derives autoregressive drafts from the target model's hidden layer states 1. DFlash enables parallel prediction of entire token blocks, and DSpark builds upon DFlash by incorporating lightweight sequence heads and confidence-based selection mechanisms 1. Testing across multiple models including Gemma-4-26B, Gemma-4-31B, Qwen3 series, Kimi-K2.5, and MiniMax-M3 on AMD Instinct MI300X and MI355X GPUs demonstrated substantial gains 1. Throughput improvements reached 2.87× for DFlash on Gemma-4-26B-A4B-it using the MATH500 dataset, 2.74× for Gemma 4 MTP on GSM8K, and 2.68× for DFlash on Kimi-K2.5 1.
Performance optimization depends critically on the num_speculative_tokens parameter, with optimal values typically ranging between 3 and 7 depending on the model, dataset, and drafting method 1. The acceptance rate of drafted tokens emerges as a key performance indicator, as declining acceptance rates in later positions can lead to throughput plateau or degradation 1. The evaluation used Ubuntu 22.04, ROCm 7.2, vLLM 0.23.1, and PyTorch 2.11.0 1.
评论
还没有评论，欢迎留下第一条。