标签： FP8

从 DeepSeek-V3 开始，blockwise FP8 量化逐渐进入大家的视野，DeepGEMM 的开源更是扩大了这种量化方式的影响。但无论是 DeepGEMM 还是 CUTLASS，目前都只在 Hopper (SM_90) 或以后的架构上支持 blockwise FP8 量化。

Hopper 之前的 Ada Lovelace (SM_89) 一代的卡，比如 L40S、L20、40x0，虽然也支持 FP8 计算，但目前大部分实现都是通过 Python 写的 Triton Kernel 实现 blockwise FP8 量化计算。

支持 SM_89 上 FP8 Blockwise 量化的 CUDA Kernel

为了优化 blockwise FP8 量化在 SM_89 系列 GPU 上的计算性能，我基于 CUTLASS 实现了支持 blockwise FP8 量化的 CUDA Kernel，源代码在：https://github.com/NVIDIA/cutlass/pull/2328 。

这个 Kernel 不仅支持 SM_89 之后 GPU 上的 FP8 blockwise 量化，还支持 SM_80 之后 GPU 上的 INT8 blockwise 量化。使用方法可以参考 examples/85_ada_ampere_gemm_with_blockwise_scaling 下的用例。

在矩阵计算规模较小时，这个 Kernel 在 L40S 卡上的计算性能甚至可以超过 H800 卡上的 warp specialized + tma 实现。在矩阵计算规模较大时，例如 M=4096、N=7168、K=2048 时，在 L40S 上的计算吞吐是 260 TFLOPS，对比 DeepGEMM 的 1215 TFLOPS (https://github.com/deepseek-ai/DeepGEMM/pull/86)，大约是 1 : 4.67，而 L40S 对比 H800 的 FP8 算力是 733 TFLOPS : 3958 TFLOPS，大约是 1 : 5.39。可以看到这个 kernel 的性能还算过得去。

$./examples/85_ada_ampere_gemm_with_blockwise_scaling/85a_ada_fp8_gemm_with_groupwise_scaling_cute --m=4096 --n=7168 --k=2048
  Problem Size: 4096x7168x2048x1
  Tile shape (M, N, K): _64, _128, _128
  ScaleGranularityM: 1 (ScaleMsPerTile: 64)
  ScaleGranularityN: 128 (ScaleNsPerTile: 1)
  Running... 
  Result MSE: 4.56599e-06, MRE: 15.6246, greatest error: 0.0286751
  Disposition: Passed
  Avg runtime: 0.462145 ms
  GFLOPS: 260220

SGLang 集成

我在自己的 SGLang 分支的 sgl-kernel 中集成了这个 Kernel，在 L40S 卡上对比 SGLang 已有的 Triton 实现性能有大幅提升，最高提升幅度可达 60%。以下是一个例子，benchmark 性能的单位是 GB/s。

deepseek-ai/DeepSeek-V3 N=24576 K=7168: 
Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /workspace/git/sglang/python/sglang/srt/layers/quantization/configs/N=24576,K=7168,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128, 128].json
fp8 blockwise scaled matmul:
    batch_size     sgl-kernel  sglang triton
0          1.0     927.460912     905.494708
1          8.0    7562.373485    7500.555847
2         16.0   15083.308993   15001.111694
3         32.0   29920.694988   29920.694988
4         64.0   59195.447003   59357.498347
5        128.0  112069.375528  114995.465571
6        256.0  208242.387702  166830.544152
7        512.0  268966.505252  185836.556807
8       1024.0  286227.561402  187468.003955
9       2048.0  300380.325376  187069.874028
10      4096.0  297778.251151  189459.651356

分支源代码在 https://github.com/solrex/sglang/tree/sm89-fp8 ，需要自行编译 sgl-kernel，benchmark 脚本是 sgl-kernel/benchmark/bench_fp8_blockwise_gemm.py，感兴趣的朋友可以自行验证。

Qwen3 FP8 模型

因为 Qwen3 的 FP8 版本参数也是采用的 blockwise 量化，可以使用 Qwen3 量化模型对比新的 CUDA Kernel 和 Triton Kernel 的端到端性能。在 L40S 上，使用我实现的 CUDA Kernel 比原 Triton Kernel，可以提升 Qwen3-8B-FP8 模型吞吐 12% 左右。

# Triton Kernel
============ Serving Benchmark Result ============
Backend:                                 sglang-oai
Traffic request rate:                    64.0      
Max reqeuest concurrency:                64        
Successful requests:                     512       
Benchmark duration (s):                  139.73    
Total input tokens:                      262144    
Total generated tokens:                  262144    
Total generated tokens (retokenized):    262140    
Request throughput (req/s):              3.66      
Input token throughput (tok/s):          1876.12   
Output token throughput (tok/s):         1876.12   
Total token throughput (tok/s):          3752.25   
Concurrency:                             63.73     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   17390.95  
Median E2E Latency (ms):                 17425.19  
---------------Time to First Token----------------
Mean TTFT (ms):                          1887.69   
Median TTFT (ms):                        1911.72   
P99 TTFT (ms):                           3277.92   
---------------Inter-Token Latency----------------
Mean ITL (ms):                           30.59     
Median ITL (ms):                         28.11     
P95 ITL (ms):                            31.14     
P99 ITL (ms):                            31.97     
Max ITL (ms):                            3035.59   
==================================================

# CUDA Kernel
============ Serving Benchmark Result ============
Backend:                                 sglang-oai
Traffic request rate:                    64.0      
Max reqeuest concurrency:                64        
Successful requests:                     512       
Benchmark duration (s):                  123.98    
Total input tokens:                      262144    
Total generated tokens:                  262144    
Total generated tokens (retokenized):    262144    
Request throughput (req/s):              4.13      
Input token throughput (tok/s):          2114.39   
Output token throughput (tok/s):         2114.39   
Total token throughput (tok/s):          4228.78   
Concurrency:                             63.66     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   15416.44  
Median E2E Latency (ms):                 15475.84  
---------------Time to First Token----------------
Mean TTFT (ms):                          1407.69   
Median TTFT (ms):                        1370.36   
P99 TTFT (ms):                           2305.31   
---------------Inter-Token Latency----------------
Mean ITL (ms):                           27.68     
Median ITL (ms):                         25.81     
P95 ITL (ms):                            28.79     
P99 ITL (ms):                            29.88     
Max ITL (ms):                            2201.85   
==================================================

DeepSeek V3/R1 模型

基于我之前的工作《在 32 张 L40S/L20 上运行 DeepSeek-R1/V3 原版 FP8 模型》，在 32 张 L40S 上，单请求下，新 Kernel 的 otps 可以从 33 提升到 40 左右；压力下大概提升 5% 左右，下面是压测数据。

# Triton Kernel
============ Serving Benchmark Result ============
Backend:                                 sglang-oai
Traffic request rate:                    128.0     
Max reqeuest concurrency:                128       
Successful requests:                     256       
Benchmark duration (s):                  98.63     
Total input tokens:                      51200     
Total generated tokens:                  51200     
Total generated tokens (retokenized):    50971     
Request throughput (req/s):              2.60      
Input token throughput (tok/s):          519.11    
Output token throughput (tok/s):         519.11    
Total token throughput (tok/s):          1038.22   
Concurrency:                             127.26    
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   49032.00  
Median E2E Latency (ms):                 49199.08  
---------------Time to First Token----------------
Mean TTFT (ms):                          8294.87   
Median TTFT (ms):                        8306.07   
P99 TTFT (ms):                           15599.09  
---------------Inter-Token Latency----------------
Mean ITL (ms):                           205.47    
Median ITL (ms):                         183.05    
P95 ITL (ms):                            187.48    
P99 ITL (ms):                            243.63    
Max ITL (ms):                            11274.90  
==================================================

# CUDA Kernel
============ Serving Benchmark Result ============
Backend:                                 sglang-oai
Traffic request rate:                    128.0     
Max request concurrency:                 128       
Successful requests:                     256       
Benchmark duration (s):                  93.56     
Total input tokens:                      51200     
Total generated tokens:                  51200     
Total generated tokens (retokenized):    50980     
Request throughput (req/s):              2.74      
Input token throughput (tok/s):          547.24    
Output token throughput (tok/s):         547.24    
Total token throughput (tok/s):          1094.48   
Concurrency:                             127.28    
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   46516.28  
Median E2E Latency (ms):                 46715.28  
---------------Time to First Token----------------
Mean TTFT (ms):                          8396.97   
Median TTFT (ms):                        9343.49   
P99 TTFT (ms):                           13201.02  
---------------Inter-Token Latency----------------
Mean ITL (ms):                           192.20    
Median ITL (ms):                         172.65    
P95 ITL (ms):                            180.43    
P99 ITL (ms):                            217.66    
Max ITL (ms):                            10399.15  
==================================================

验证方法

使用我的 SGLang sm89-fp8 分支，编译安装 sgl-kernel，安装 sglang。或者也可以下载我编译好的 wheel 直接安装。

git clone -b sm89-fp8 https://github.com/solrex/sglang.git
cd sglang/sgl-kernel
make build                   # 编译安装本地 sgl-kernel，耗时较久
cd ..
pip install -e "python[all]" # 安装本地 sglang

在启动 SGLang 服务时，可以通过是否设置 SGLANG_SUPPORT_CUTLASS_BLOCK_FP8 环境变量，控制使用 CUDA Kernel 还是 Triton Kernel。启用 CUDA Kernel 的命令形如：

SGLANG_SUPPORT_CUTLASS_BLOCK_FP8=1 python3 -m sglang.launch_server --model-path /workspace/Qwen3-8B-FP8/ --port 8080 --tp 1 --host 0.0.0.0 --allow-auto-truncate --enable-torch-compile --torch-compile-max-bs 64 --cuda-graph-max-bs 64

可以通过启动日志中是否有 "Using default W8A8 Block FP8 kernel config." 类似日志来判断使用的 Kernel 是哪个。当下面这条日志存在时，代表使用的是 Triton Kernel：

[2025-06-03 18:20:24] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /workspace/git/sglang/python/sglang/srt/layers/
quantization/configs/N=6144,K=4096,device_name=NVIDIA_L40S,dtype=fp8_w8a8,block_shape=[128, 128].json

压测使用的脚本是：

python3 -m sglang.bench_serving --backend sglang-oai --dataset-path /workspace/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --random-range-ratio 1 --random-input-len 512 --random-output-len 512  --request-rate 64 --num-prompt 512 --max-concurrency 64 --host 127.0.0.1 --port 8080

在实现这个 kernel 时我遇到和解决了很多问题，现在也不敢说没有 bug。如果你在验证的时候发现任何问题，请不吝留言指教。