Usage

Serve Mode

Run a model directly with llama-server and expose an OpenAI-compatible API:

# Serve a model with API proxy on port 49222
./build.sh serve --model /path/to/model.gguf --api-port 49222

# Serve with a settings profile
./build.sh serve --model model.gguf --profile qwen

# Serve with API key authentication (Bearer token)
./build.sh serve --model model.gguf --api-port 49222 --api-key secret

# Serve with API proxy and WebSocket dashboard
./build.sh serve --model model.gguf --api-port 49222 --ws-enable

# Serve with custom dashboard port and auth
./build.sh serve --model model.gguf --api-port 49222 --ws-enable --ws-port 8081 --ws-auth mykey

# Serve with a custom backend binary path
./build.sh serve --model model.gguf --backend-binary /path/to/custom/llama-server

# Serve bound to a specific network interface
./build.sh serve --model model.gguf --host 0.0.0.0

# Redirect logs to a file (useful for systemd)
./build.sh serve --model model.gguf --log-file /var/log/llm-manager/model.log

# Combine options
# Serve with API proxy and WebSocket dashboard on a specific host
./build.sh serve --model model.gguf --api-port 49222 --ws-enable --host 192.168.1.100

# Redirect logs to a file (useful for systemd)
./build.sh serve --model model.gguf --log-file /var/log/llm-manager/model.log

# Combine options
./build.sh serve --model model.gguf --api-port 49222 --ws-enable --host 0.0.0.0 --backend-binary /opt/rocm/bin/llama-server --log-file /var/log/llm-manager/model.log

The serve command automatically resolves the llama-server binary from the backend-specific directory (~/.local/share/llm-manager/bin/llama-server-{cpu,vulkan,rocm}-{version}/) and sets LD_LIBRARY_PATH for shared libraries. If the binary is not found, it downloads it from the llama.cpp GitHub releases. Use --backend-binary to specify a custom binary path, --host to override the network bind address for both the API proxy and WebSocket servers (default is from config), and --log-file to redirect logs to a file instead of stdout.

Model Management

Listing Models

The Models panel shows all .gguf files found in your models directories (recursively). The display name is the relative path from the models directory.

• f — Filter local models by name (case-insensitive substring match)
• Esc — Clear active filter and return to full list

Loading and Unloading

• l or Enter — Load selected model
• u — Unload model from server
• Ctrl+D — Delete model (with confirmation)

When a model is loaded, its state changes to Loaded showing the port and PID. You can load multiple models when using Router mode.

Deleting Models

Pressing Ctrl+D prompts for confirmation before moving the model file and its YAML config to ~/.config/llm-manager/unused/. Both can be restored later.

Search

Search mode lets you browse and download GGUF models from HuggingFace:

Multi-word Search

Type space-separated words (e.g. qwen opus) to search with AND logic — all words must match the model name. Matching words are highlighted in cyan in the results list.

GGUF File Browser

When viewing GGUF files for a model:

Download Panel

When one or more files are downloading, the Download panel appears at the bottom of the screen, showing progress, speed (MiB/s), ETA, and status for each download. Before downloading, the app checks available disk space and warns if insufficient. Cancelled downloads automatically remove the temporary file.

Status indicators: Downloading (yellow), Paused (white), Complete (green), Cancelled (red), Error (red).

Loading Models

When you load a model, the application:

1. Resolves the llama-server binary for the selected backend (CPU/Vulkan/ROCm)
2. Spawns the server with the current settings
3. Loads the model via the server’s /models/load API
4. Polls the server’s /metrics and /health endpoints for status
5. Displays a progress bar showing loading phases

Loading Phases

The progress bar tracks:

• Server starting (8%) — llama.cpp binary is launched
• Loading model (7%) — weights file is being read
• Loading metadata (7%) — GGUF metadata is parsed
• Loading tensors (70%) — tensors are loaded and offloaded to GPU
• Server listening (8%) — HTTP server is ready
• Complete — model is ready for inference

During tensor loading, the progress bar shows offloaded layers (e.g., 16/32) parsed from llama.cpp’s log output.

Settings

Server Settings

Note: The Server Settings panel is hidden when a server is already running. Press F2 to toggle Server Settings only when no server is active.

LLM Settings

The LLM Settings panel has 32 standard fields, 16 expert fields (revealed with Ctrl+X), and 19 ultra fields (hidden even in expert mode), for a total of 67 fields. Arrow keys adjust values; +/- for coarse changes, Left/Right for fine. Toggle fields respond to e or Ctrl+E.

Loading

GPU Offload

Evaluation

Sampling

Repetition Control

Yarn RoPE

Tags

Backend

Expert Mode

Press Ctrl+X to toggle expert mode, which reveals 16 additional parameters:

Loading (expert): NUMA (None/Distribute/Isolate/Numactl)

GPU (expert): Cache Type K (toggle), Cache Type V (toggle), Main GPU, Fit, Active Experts (toggle)

Sampling (expert): Mirostat (Off/1/2), Mirostat LR, Mirostat Ent, Ignore EOS (toggle)

Repetition (expert): Presence Penalty (toggle, -2.0 to 2.0), Frequency Penalty (toggle, -2.0 to 2.0)

Speculative (expert): MTP (toggle), Spec Type, Spec Draft N Max

These fields follow the same navigation and editing rules as standard fields. Arrow keys adjust values, Enter enters direct edit mode, and dirty fields are highlighted in yellow.

Ultra Fields

19 ultra fields are hidden even in expert mode. They include: Typical P, Mirostat, Mirostat LR, Mirostat Ent, Ignore EOS, Samplers, DRY Multiplier, DRY Base, DRY Allowed Length, DRY Penalty Last N, Threads Batch, UBatch Size, Keep, Split Mode, Tensor Split, Main GPU, Fit, Embedding, RPC. These require direct config file editing or profile application.

Cache Type K/V options: F32, F16, BF16, Q8_0, Q5_0, Q5_1, Q4_0, Q4_1, Iq4Nl

Changing Values

Use Left/Right to adjust numeric fields by 1, or Up/Down for larger steps. Toggle fields respond to e or Ctrl+E. Dirty (changed) fields have the name in red and a trailing *. The status bar shows *unsaved* when settings are dirty.

Saving Settings

• Ctrl+S — Save settings for the selected model
• Ctrl+R — Reset to defaults
• e / Ctrl+E — Toggle enabled/disabled (for Keep in memory, Flash Attention, KV Cache Offload, Cache Type K/V, Fit, Unified KV, Max Tokens, Presence/Frequency Penalty, Max Concurrent Pred, MTP, Ignore EOS, Yarn RoPE, Active Experts)
• Ctrl+X — Toggle expert mode (reveals 16 additional parameters)

Dirty (changed) fields are highlighted with red names and a trailing *.

Keyboard Shortcuts

Log Panel

The Log panel displays live output from the llama.cpp server with level-based coloring.

Log Modes

Press f in the Log panel to toggle between modes. The current mode is shown in the panel title. Expand the log to fullscreen with Enter; collapse with Esc.

RPC Workers

RPC Workers enable distributed inference across multiple machines. Each worker has a name, IP address, and port (default: 50052).

Open the RPC Workers manager from the Server Settings panel. Within the manager:

WebSocket Dashboard

The WebSocket Dashboard provides a real-time visualization of model metrics in any web browser. Access it at http://localhost:49223 (default port).

Configuration

Open the Server Settings panel, navigate to Dashboard, and press Enter to configure:

When an auth key is set, clients must include it as a URL parameter: http://localhost:49223?auth=<key>. With TLS enabled, the URL uses https://.

Dashboard Display

The dashboard shows real-time metrics (TPS, prompt TPS, latency, context, VRAM, RAM, CPU) and current inference settings (backend, threads, temperature, sampling parameters, etc.) alongside the full server command line.

Benchmark Tuning

Benchmark Tuning auto-tunes model parameters for optimal performance. Access it by setting the Server Mode to BenchTune.

Two modes are available:

• RuntimeOnly — Single server, params sent in request body (no server restarts)
• Full — New server spawned for each parameter combination

Tunable parameters: temperature (0.4–1.0), top_p (0.8–1.0), top_k (40–50), repeat_penalty (1.0–1.2), flash_attn (0/1), threads (4–16), batch_size (512–2048), expert_count (1–4), context_length, spec_type (speculative decoding type), draft_tokens.

Results can be exported as Markdown table, JSON, YAML, or HTML report with summary cards, winner section, impact analysis, and Chart.js charts. Navigate between results with p (previous) and n (next).

System Prompt Presets

Named system prompts for different use cases. Built-in presets: General, Coder, Thinker, Mathematician. User presets are stored as YAML files in ~/.config/llm-manager/presets/<name>.yaml.

Open the System Prompt Presets panel and manage presets:

GPU Layers Cycling

In the LLM Settings panel, the GPU Layers field cycles through three modes with arrow keys:

Arrow keys cycle: Auto → 1 → 2 → … → N → All → Auto. Pressing Enter from a specific number opens an edit buffer for direct input. The -ngl flag is only added for Specific and All modes.

Tags

Per-model tags can be edited in the LLM Settings panel. The Tags field opens an edit modal where you can add, remove, or modify tags associated with the model. Tags are stored in the per-model YAML config.

MTP (Multi-Token Prediction)

MTP is an experimental feature that uses a draft model to predict multiple tokens in parallel, improving inference speed. When a model with MTP architecture is selected, the app automatically detects it and enables the --draft-mtp flag. The number of draft tokens is read from the GGUF metadata and displayed in the Model Info panel.

GGUF Metadata

The Model Info panel shows parsed GGUF metadata including: architecture, layers, hidden size, context length, attention heads, KV heads, domain, capabilities, quantization, parameters (e.g., “7B”, “405B”), tokenizer type, vocabulary size, and max context for VRAM. Metadata is parsed once and cached (debounced by file mtime).

Active Model Metrics

The Active Model panel shows real-time metrics:

The panel also shows benchmarking state with progress bar and current parameter display when running BenchTune.

Backend Selection

Multiple backends are supported via the llama.cpp server:

Each backend has its own independently configurable llama.cpp version. Switching versions is instant — no re-download.

Server Modes

VRAM Estimate

The app computes a detailed VRAM estimate based on model size, GPU layers, KV cache, activation overhead, and fixed overhead. The formula accounts for GQA ratio, FlashAttention (0.5× KV cache reduction), unified KV cache, KV cache quantization bytes, activation overhead (8× multiplier), and fixed overhead (3.8% of max VRAM or 500 MiB fallback). The estimate is shown in the LLM Settings title (e.g., “VRAM ~= 8.2 GB”).

Confirmation Dialogs

The app uses confirmation dialogs for destructive actions:

• Exit — warns about loaded models
• Delete — confirms irreversible deletion
• Reset — confirms resetting all LLM settings
• Unload — confirms unloading a model via API
• DeleteBackend — confirms deleting a backend binary version from disk

Mouse Support

Mouse interactions are supported: clicking on panels to focus them, and scrolling in the log panel, README panel, settings, profiles, and presets panels.

Panel Resize

The horizontal split between left panels (Models + Info) and right panels (Settings/README) can be resized:

The current split percentage is shown in the status bar (e.g., │ 55%). While actively resizing, the indicator shows │ 55% ← resize →.

CmdLine Overlay

Press Ctrl+K to view the full command line that would be executed to start the llama.cpp server. This shows the binary path, model path, and all parameters.

Press e in the overlay to export the command to /tmp/test_llamaserver.sh.

Server Status

The status bar shows the current server status at the top:

• Running: ● 9090 Normal (green dot with port and mode)
• Stopped: ○ Server (gray)

Press Ctrl+Alt+K to kill the running llama-server. When stopped, all loaded models are reset to Available state.

Profiles

Profiles are named presets of LLM settings. Built-in profiles include Qwen, Gemma, Llama, Mistral, and Phi. User profiles are stored as YAML files in ~/.config/llm-manager/profiles/<name>.yaml.

• p — Apply a profile to current settings
• Ctrl+S — Save current settings as a new profile (in the Profiles panel)
• Ctrl+D — Delete a user-defined profile (moved to unused_profiles/)