Is DeepGEMM directly applicable to backward in training?

[YouJiacheng]

bwd of GEMM is two GEMMs, but I wonder if I need to take some special care of the range of gradients?

[YouJiacheng]

Oh, so we must write a quantization kernel that produces the correct lhs_scales and rhs_scales.

[zhipeng93]

We need a fp8 gemm with 128x1 LHS scaling and 1x128 RHS scaling?

[LyricZhao]

We provide this library mainly for inference. So this library only supports DGRAD, not WGRAD.

In my understanding, WGRAD support needs more than a GEMM kernel, but also some utility fused kernels (e.g. transposing, fused with casting, fused with SwiGLU, fused with MoE layout). We want this library to be clean, so we didn't open-source them.

We may later release the WGRAD kernel, we will discuss about it internally :)

[YouJiacheng]

Thank you Chenggang!
Yup I forgot that WGRAD needs to transpose matrices.