NixOS module that runs your LLM inference servers and routes between them

Links
1

Hi there, I created LLMhop, a tiny OpenAI-compatible router to serve multiple LLM inference servers on the same machine, complete with a NixOS module to set it up.

It allows you to declare models and it provisions isolated llama.cpp, sglang, and/or vLLM workers (all three can run side by side), each on a loopback port with its route auto-registered. The router runs hardened under DynamicUser, GPU containers run rootless via Podman/quadlet, and no worker ever runs as root. Secrets go in via LoadCredential and ${file:...} references, never the world-readable store.

services.llmhop = {
  enable = true;
  llama-cpp.models."qwen3-8b".settings.hf-repo = "unsloth/Qwen3-8B-GGUF:UD-Q4_K_XL";
  vllm.models."llama-3-8b".model = "meta-llama/Meta-Llama-3-8B-Instruct";
};

I wrote up the design in more detail in a blog post.

2 Likes
2

Made something similar to this a while back: caniko/infernix: Nix flake for managing large language models (LLMs) - Codeberg.org

Use it daily