GitHub Trending repo Repositories Mar 25 7 views

0xSero/turboquant: TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration

by 0xSero

TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration

Read Original
API LLM

Metadata

Stars
218
Forks
31
Language
Python
Watchers
218
Open Issues
4
License
GPL-3.0

Related