LlamaNexus

API reference

Endpoints

LlamaNexus exposes both /ollama/api/* and /openai/v1/* routes, forwarding to llama-server's single OpenAI-compatible router underneath.

Endpoint Purpose
POST /openai/v1/chat/completions Streaming SSE chat completion, with tool-calling support
GET /openai/v1/models Model list - real context size, trained context, load status (see below)
GET /status Connection check, plus current RPC worker mode/roster
POST /ollama/api/pull Download a not-yet-present model, streaming progress
POST /api/set-ctx-size Change a model's ctx-size and restart to apply it
POST /api/eject-model Unload a model from the server (keep the file on disk)
POST /api/remove-model Permanently delete a downloaded model
POST /api/rpc/mode, /api/rpc/workers, /api/rpc/workers/remove Toggle single-node/RPC mode, add/remove RPC workers

How model identifiers work

Models are addressed as repo:tag, e.g. unsloth/Qwen3.5-4B-GGUF:Q4_K_M. LlamaNexus scans the Hugging Face cache and infers a short quant-tag identifier from each cached file's name (Q4_K_M, Q8_0, Q5_K_M, Q4_0, or the filename with its .gguf extension stripped if none of those match).

Both /ollama/api/chat and /openai/v1/chat/completions normalize the request's model field (strips a trailing .gguf/:latest) before forwarding, so a client can request a model by its exact repo:filename.gguf form and it'll still match the router's registered id.

When pulling a model by a short tag that doesn't exactly match a real filename (e.g. Q4_K_M when the real file is model-name-Q4_K_M.gguf), LlamaNexus resolves the real filename against the repo's file listing before downloading. A successful pull restarts llama-server's router and waits (up to 60s) for it to come back up before the request completes.

GET /openai/v1/models detail

Each model entry includes:

Context size overrides

Open WebUI's per-model num_ctx setting (and the VS Code extension's /ctx-size command) work around llama.cpp allocating its KV cache at model-load time:

  1. LlamaNexus writes the new ctx-size into that model's section in router.preset.ini.
  2. Restarts the llama-server router process (it only parses --models-preset once, at its own startup).
  3. Waits briefly for the router to come back up before the triggering chat request proceeds.

The first message after changing context size pays a short restart cost (typically a few seconds); subsequent messages are unaffected until the value is changed again. A [*] global section in the preset file holds the baseline default for any model without a custom override.

Don't pass --ctx-size as a serve-level argument - it takes precedence over the preset file's per-model values and will silently defeat this mechanism.

POST /api/set-ctx-size

{ "model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M", "ctx_size": 8192 }

Writes the preset, restarts llama-server, waits for the router to come back up, and responds once it has:

{ "ok": true, "model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M", "ctx_size": 8192 }

Model unloading

Open WebUI's "eject" action and Ollama's keep_alive: 0 convention are both detected on /ollama/api/chat and /ollama/api/generate and forwarded to llama-server's router /models/unload endpoint, rather than treated as a real generation request.

POST /api/eject-model

{ "model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M" }

Forwards to llama-server's router /models/unload and responds with { "ok": true, "model": "..." } once done.

Model removal

POST /api/remove-model

Permanently deletes a downloaded model - unlike /api/eject-model (which only unloads it from memory), this removes it from disk entirely.

{ "model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M" }
  1. Best-effort unloads the model first, so a currently-loaded model isn't left served from a file that's about to disappear.
  2. Deletes every GGUF file backing that model from the Hugging Face cache - both the snapshot symlink(s) and the blob(s) they point to. A sharded quant has every shard removed together.
  3. Removes the model's section from router.preset.ini, if it had a ctx-size override.
  4. Restarts llama-server and waits for the router to come back up before responding.
{ "ok": true, "model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M", "deleted_files": 1, "freed_bytes": 18350698496, "removed_preset_section": true }

Responds 404 if the model isn't found in the cache.

Known quirks