Distributed inference

LlamaNexus supports two modes for connecting worker nodes to the primary server over llama.cpp's RPC backend.

Manual mode

Pass --rpc to serve with a comma-separated list of worker addresses. LlamaNexus starts llama-server immediately with those workers and does not listen for heartbeats.

command: serve -- --rpc 192.168.0.120:50052,192.168.0.110:50052

Auto-discovery mode

Pass --discovery to both serve and worker. Workers broadcast a UDP heartbeat once per second; the serve node listens, discovers workers automatically, and passes them to llama-server as --rpc arguments.

# On each worker machine
command: worker --discovery

# On the primary server
command: serve --discovery

On startup, serve waits 8 seconds (configurable with --discovery-wait) to collect heartbeats, then starts llama-server with all discovered workers. After that the watcher keeps running in the background: if a new worker appears or an existing one stops sending heartbeats for 5 seconds, llama-server is automatically restarted with the updated worker list.

Discovery flags

Flag	Default	Purpose
`--discovery`	`false`	Enable auto-discovery (use on both `serve` and `worker`)
`--discovery-port`	`50051`	UDP port used for heartbeat packets
`--discovery-wait`	`8s`	How long `serve` waits for heartbeats before starting `llama-server`
`--advertise-addr`	(auto-detected)	IP or `host:port` to advertise in heartbeats; overrides auto-detection

Docker networking requirement

Discovery uses UDP broadcast (255.255.255.255), which requires containers to be on the host network so packets reach the physical LAN interface rather than staying inside a Docker bridge. Add network_mode: host to both the serve and worker services in your Compose file:

services:
  llamanexus:
    network_mode: host

services:
  llamanexus-worker:
    network_mode: host

Advertise address override

When running inside Docker, the auto-detected IP may resolve to a container bridge address (e.g. 172.19.0.x) instead of the host's LAN IP. Override it explicitly:

command: worker --discovery --advertise-addr 192.168.0.120

Or via environment variable:

environment:
  - LLAMANEXUS_ADVERTISE_ADDR=192.168.0.120

The port is appended automatically if you supply only an IP. Priority order: --advertise-addr flag → LLAMANEXUS_ADVERTISE_ADDR env var → auto-detection.

Portable worker binaries

Worker images are built for a portable baseline CPU instruction set, so a binary built on one machine doesn't SIGILL on an older worker CPU - GPU offload is unaffected.