Home › LLM Tools › tiny-vllm

tiny-vllm

by jmaczan · github.com/jmaczan/tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

Stars
★ 944
Forks
67
Language
C++
Category
LLM Tools
License
Apache-2.0
aiattentionbatchingcoursecppcudahpcinferencellmllm-inferencepagedattentiontiny-vllmc++

View on GitHub ↗

Featured in