📚200+ Tensor/CUDA Cores Kernels, ⚡️flash-attn-mma, ⚡️hgemm with WMMA, MMA and CuTe (98%~100% TFLOPS of cuBLAS/FA2 🎉🎉).

Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.

Optimizing SGEMM kernel functions on NVIDIA GPUs to a close-to-cuBLAS performance.

Several optimization methods of half-precision general matrix vector multiplication (HGEMV) using CUDA core.

FP64 equivalent GEMM via Int8 Tensor Cores using the Ozaki scheme

PyTorch half precision gemm lib w/ fused optional bias + optional relu/gelu

The simplest but fast implementation of matrix multiplication in CUDA.

Code for testing the native float16 matrix multiplication performance on Tesla P100 and V100 GPU based on cublasHgemm

code for benchmarking GPU performance based on cublasSgemm and cublasHgemm

Some common CUDA kernel implementations (Not the fastest).

Use tensor core to calculate back-to-back HGEMM (half-precision general matrix multiplication) with MMA PTX instruction.

Fast SGEMM emulation on Tensor Cores

CUDA optimization samples including sgemm, reduce... To be continued.

CUDA kernel functions

CUDA Kernels on V100

Fast Convoluion Implementation via CUDA

My attempt of making a GEMM kernel...

HPC course for Grad 3/4th 2021

Simple Tensorcore GEMM in ThunderKittens