Configuration

Directory Layout

llm-manager uses XDG directories for config and data:

~/.config/llm-manager/          # Config directory
├── config.yaml                 # Global settings
├── models/                     # Per-model YAML configs
│   └── qwen2.5-7b.yaml
├── profiles/                   # Per-profile YAML configs
│   └── my-profile.yaml
├── presets/                    # Per-preset YAML configs
│   └── custom-preset.yaml
├── unused/                     # Deleted model configs
├── unused_profiles/            # Deleted profiles
└── unused_presets/             # Deleted presets

~/.local/share/llm-manager/     # Data directory
├── models/                     # GGUF model files
│   └── qwen2.5-7b.Q4_K_M.gguf
└── bin/                        # llama-server binaries
    └── llama-server-cpu-...

Per-model configs are named <model_name>.yaml where model_name is the GGUF filename without the .gguf extension. Deleted configs are moved to unused/ subdirectories (recoverable).

Config File

The main config file is ~/.config/llm-manager/config.yaml. It is created automatically on first run with sensible defaults.

models_dirs:
  - ~/.local/share/llm-manager/models
llama_server: llama-server
default:
  context_length: 131072
  threads: <physical cores>
  threads_batch: 8
  batch_size: 512
  temperature: 0.8
  # ... more settings

You can specify a custom config path with --config:

cargo run -- --config /path/to/config.yaml

Default Parameters

Advanced Parameters

These can be configured via the LLM Settings panel, per-model config files, or directly in config.yaml.

Profiles

Profiles are named presets of settings. The built-in profiles are:

All profiles also set context_length: 131072, top_p: 0.95, max_tokens: 4096, and uniform_cache: true. Qwen, Qwen-MoE, Qwen-Coding, Gemma, Llama, and Mistral also set jinja: true (Phi does not).

User-defined profiles are stored as individual YAML files in ~/.config/llm-manager/profiles/<name>.yaml. Built-in profiles are auto-merged on load.

System Prompt Presets

System prompt presets define the initial system prompt. Built-in presets:

The default preset is Coder. User-defined presets are stored as individual YAML files in ~/.config/llm-manager/presets/<name>.yaml. Built-in presets are auto-merged on load.

Backend Binaries

llama-server binaries are stored in ~/.local/share/llm-manager/bin/ with versioned directories:

~/.local/share/llm-manager/bin/
├── llama-server-cpu-{version}/llama-server
├── llama-server-vulkan-{version}/llama-server
├── llama-server-rocm-{version}/llama-server
├── llama-server-rocm-lemonade-{version}/llama-server
└── llama-server-cuda-{version}/llama-server

Binaries are downloaded from specialized repositories on first use:

• CPU, Vulkan, ROCm (Native): Fetched from ggml-org/llama.cpp
• ROCm Lemonade: Fetched from lemonade-sdk/llamacpp-rocm (ZIP, auto-detects GFX architecture like gfx1100)
• CUDA (NVIDIA): Fetched from ai-dock/llama.cpp-cuda (CUDA 12.8 builds)

Switching versions is instant — no re-download.

Per-backend Version Config

llama_cpp_version_cpu: null
llama_cpp_version_vulkan: null
llama_cpp_version_rocm: null
llama_cpp_version_rocm_lemonade: null
llama_cpp_version_cuda: null

Platform-specific backend variants (e.g. CpuArm64, CpuWindows, CpuMacosArm64) are handled through the Backend enum and platform field, not through separate version config keys. Each backend has its own independently configurable version.

Setting to null uses the latest release. Specific versions can be set via the version picker in LLM Settings. These selections are automatically persisted to your configuration and remembered across restarts.

Asset Names

Assets are selected based on the detected platform. Linux examples:

• CPU (x64): llama-{tag}-bin-ubuntu-x64.tar.gz
• CPU (ARM64): llama-{tag}-bin-ubuntu-arm64.tar.gz
• Vulkan: llama-{tag}-bin-ubuntu-vulkan-x64.tar.gz
• ROCm: llama-{tag}-bin-ubuntu-rocm-7.2-x64.tar.gz
• ROCm Lemonade: llama-{tag}-ubuntu-rocm-{gfx}-x64.zip (auto-detects GPU architecture)
• CUDA: llama.cpp-{tag}-cuda-12.8-amd64.tar.gz

Windows assets use *.zip (e.g. llama-{tag}-bin-win-cpu-x64.zip). macOS assets use llama-{tag}-bin-macos-arm64.tar.gz or llama-{tag}-bin-macos-x64.tar.gz.

Serve Mode

You can start a model directly from the command line without the TUI:

./build.sh serve --model /path/to/model.gguf

Options

Note: --threads, --context, and --gpu-layers are not CLI flags. They are configured via config.yaml (default section) or per-model override files.

API Proxy

The API proxy forwards requests to the llama.cpp server and provides OpenAI-compatible and Anthropic-compatible endpoints. It supports SSE (Server-Sent Events) streaming for chat completions and other streaming endpoints. CORS is enabled with dynamic origin validation — only requests from localhost, 127.0.0.1, or the configured bind host are allowed. External websites are blocked. When --api-key is set, all requests require Authorization: Bearer <key>.

API Endpoints

The API proxy explicitly handles the following endpoints, while all other paths are automatically proxied to the llama-server instance:

The following endpoints are forwarded to llama-server (llama-server built-in endpoints, not explicitly handled by llm-manager):

Model Overrides

Settings can be saved per-model. Overrides are stored as individual YAML files in ~/.config/llm-manager/models/<name>.yaml (where name is the GGUF filename without .gguf). When a model is loaded, its override settings are merged into the defaults. Deleted configs are moved to ~/.config/llm-manager/unused/ for recovery.

RPC Workers

You can manage a list of remote llama-rpc-server nodes for distributed inference. These are stored in the rpc_workers list in the config:

rpc_workers:
  - selected: true
    name: "Worker 1"
    ip: "192.168.1.10"
    port: 50052

Workers can be managed via the RPC Workers window in the Server Settings panel. Selected workers are combined into the --rpc flag when starting the server.