# local-llm

Local LLM setup on the Mac mini: [Ollama](https://ollama.com) serving models,
[opencode](https://opencode.ai) using them, and [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
generating images. Open WebUI (https://ai.quineglobal.com, on the Tart VM, see
petersweb-infra) uses Ollama and ComfyUI over Tart's bridge (`192.168.64.1`).

```sh
./setup.sh
```

does everything, and is safe to re-run:

- installs `ollama/com.ollama.server.plist` as the LaunchAgent that runs
  `ollama serve` (reloaded only when it changed, since that restarts Ollama);
- builds each model in `ollama/models/` (pulling its base model if missing);
- installs ComfyUI in `~/ComfyUI` (pinned tag + ComfyUI-GGUF), downloads the
  models in `comfyui/models.txt`, and installs `comfyui/com.comfyui.server.plist`;
- symlinks `~/.config/opencode/opencode.json` to `opencode/opencode.json`, so
  edits made either way end up here.

Needs `brew install ollama opencode` (setup.sh installs `uv` itself).

## Models

| Model | From | Used for |
|-------|------|----------|
| `qwen3-coder:30b-128k` | `qwen3-coder:30b` | opencode's default (128k context) |
| `qwen3:32b-32k` | `qwen3:32b` | opencode, as an option |
| `inspector` | `qwen3:32b` | 16k-context variant |

Each derived model mostly exists to set `num_ctx`. Ollama's OpenAI-compatible
API, which opencode talks to, ignores any context size sent with a request, so
without it a long agent prompt is silently cut to Ollama's small default
context. A Modelfile's `# name:` line is the name it's created under.

`qwen3:32b` is a thinking model, and in opencode it often ends its turn after
thinking, without making the tool call it planned. `qwen3-coder` was trained
for agentic tool use and doesn't think first, so it doesn't have that
problem, and it's much faster: a mixture of experts with ~3B active parameters.

## Server settings

From the LaunchAgent: listens on all interfaces (`0.0.0.0:11434`), keeps models
loaded for 30 minutes after last use (`OLLAMA_KEEP_ALIVE=30m`, so ComfyUI has
room on the 64 GB machine), flash attention on. Logs go to `/tmp/ollama.log`.

## ComfyUI

LaunchAgent `com.comfyui.server`, port 8188, logs in `/tmp/comfyui.log`. It has
no authentication, so it listens only on `127.0.0.1` and Tart's bridge
`192.168.64.1` (it can't bind that until a VM is running; launchd retries).

| Model | Files | Workflow |
|-------|-------|----------|
| Qwen-Image 2512 (Q4_K_M GGUF) | `diffusion_models/`, GGUF Qwen2.5-VL text encoder, `qwen_image_vae` | `comfyui/workflows/qwen-image.json` |
| Stable Diffusion XL base 1.0 | `checkpoints/sd_xl_base_1.0.safetensors` | `comfyui/workflows/sdxl.json` |

The workflows are in ComfyUI's API format, as Open WebUI needs them (Admin
Panel → Settings → Images). Qwen-Image is quantized because bf16/fp8 weights
don't fit next to Ollama's models, and MPS can't run fp8.