Minimalist local LLM proxy
Hi all! I made this small proxy when I built a small GPU cluster at home and wanted to share it with friends while keeping speed limits and token accounting. I also put a lot of effort in making it work as quick as possible, and IMO the result is worth sharing! It is not trying to be LiteLLM: in fact, I wanted to make it opposite, as small as possible, with a small amount of dependencies and without cloud connections by default. Will be happy to hear your feedback :) PS also installable with pip install smol-llm-proxy