LlamaNexus

LlamaNexus is a Go proxy that sits between Open WebUI (and other OpenAI/Ollama-style clients, like the VS Code extension or WebUI) and llama.cpp's llama-server. It translates Ollama-style and OpenAI-style API calls into requests llama-server understands, while handling model downloads from Hugging Face, context-size overrides, and model unloading along the way.

It runs as a single Go binary (llamanexus) with four subcommands: serve, run, pull, and worker.

Why it exists

The primary goal is distributed inference across multiple machines using llama.cpp's RPC backend. Worker machines run the same Docker image started with the worker command, exposing their GPU or CPU resources to the primary server via ggml-rpc-server. The primary server's RPC backend distributes model layer computations across all connected workers, letting inference scale beyond a single machine's memory and compute.

Beyond distributed inference, LlamaNexus bridges two API dialects: Open WebUI speaks the Ollama API natively, while llama-server speaks an OpenAI-compatible API and runs in router mode, managing multiple GGUF models loaded from a local cache directory. LlamaNexus presents an Ollama-compatible (and OpenAI-compatible) HTTP API and forwards requests to the router underneath, translating request/response shapes as needed. It also takes over model downloading from Hugging Face Hub directly, so download progress can be reported back to the client's UI in real time.

Architecture

Open WebUI  --(Ollama / OpenAI API)-->  LlamaNexus  --(OpenAI-compatible API)-->  llama-server (router mode)
                                             |
                                             +--> hf_progress_download.py (Hugging Face downloads)
                                             +--> router.preset.ini (per-model context size overrides)

• LlamaNexus listens on the Ollama-standard port (11434 by default) and exposes both /ollama/api/* and /openai/v1/* routes.
• llama-server runs in router mode (--models-dir + --models-preset), auto-discovering GGUF files from the Hugging Face cache and managing per-model child processes, loading and unloading them on demand.
• hf_progress_download.py is a small Python sidecar that downloads models via huggingface_hub directly (bypassing the hf CLI, which doesn't expose usable progress when piped) and reports real byte-level progress as NDJSON lines on stdout.
• router.preset.ini is an INI file LlamaNexus reads and writes to override ctx-size (and potentially other launch args) on a per-model basis, since llama.cpp allocates the KV cache at model-load time and can't resize it live.

Companion apps

• LlamaNexus for VS Code chats directly with LlamaNexus from inside the editor, no Open WebUI needed.
• LlamaNexus WebUI is a standalone browser chat client with the same core experience.

Both use the same OpenAI-compatible endpoints documented in the API reference, plus a few dedicated endpoints (/api/set-ctx-size, /api/eject-model, /api/remove-model) for context-size changes, model ejection, and model removal from their respective UIs.