84e8326e61
ComfyUI serves image generation to Open WebUI on the Tart VM (ai.quineglobal.com). Models unload when idle on both sides so they share the 64 GB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
61 lines
2.8 KiB
Markdown
61 lines
2.8 KiB
Markdown
# local-llm
|
|
|
|
Local LLM setup on the Mac mini: [Ollama](https://ollama.com) serving models,
|
|
[opencode](https://opencode.ai) using them, and [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
|
|
generating images. Open WebUI (https://ai.quineglobal.com, on the Tart VM, see
|
|
petersweb-infra) uses Ollama and ComfyUI over Tart's bridge (`192.168.64.1`).
|
|
|
|
```sh
|
|
./setup.sh
|
|
```
|
|
|
|
does everything, and is safe to re-run:
|
|
|
|
- installs `ollama/com.ollama.server.plist` as the LaunchAgent that runs
|
|
`ollama serve` (reloaded only when it changed, since that restarts Ollama);
|
|
- builds each model in `ollama/models/` (pulling its base model if missing);
|
|
- installs ComfyUI in `~/ComfyUI` (pinned tag + ComfyUI-GGUF), downloads the
|
|
models in `comfyui/models.txt`, and installs `comfyui/com.comfyui.server.plist`;
|
|
- symlinks `~/.config/opencode/opencode.json` to `opencode/opencode.json`, so
|
|
edits made either way end up here.
|
|
|
|
Needs `brew install ollama opencode` (setup.sh installs `uv` itself).
|
|
|
|
## Models
|
|
|
|
| Model | From | Used for |
|
|
|-------|------|----------|
|
|
| `qwen3-coder:30b-128k` | `qwen3-coder:30b` | opencode's default (128k context) |
|
|
| `qwen3:32b-32k` | `qwen3:32b` | opencode, as an option |
|
|
| `inspector` | `qwen3:32b` | 16k-context variant |
|
|
|
|
Each derived model mostly exists to set `num_ctx`. Ollama's OpenAI-compatible
|
|
API, which opencode talks to, ignores any context size sent with a request, so
|
|
without it a long agent prompt is silently cut to Ollama's small default
|
|
context. A Modelfile's `# name:` line is the name it's created under.
|
|
|
|
`qwen3:32b` is a thinking model, and in opencode it often ends its turn after
|
|
thinking, without making the tool call it planned. `qwen3-coder` was trained
|
|
for agentic tool use and doesn't think first, so it doesn't have that
|
|
problem, and it's much faster: a mixture of experts with ~3B active parameters.
|
|
|
|
## Server settings
|
|
|
|
From the LaunchAgent: listens on all interfaces (`0.0.0.0:11434`), keeps models
|
|
loaded for 30 minutes after last use (`OLLAMA_KEEP_ALIVE=30m`, so ComfyUI has
|
|
room on the 64 GB machine), flash attention on. Logs go to `/tmp/ollama.log`.
|
|
|
|
## ComfyUI
|
|
|
|
LaunchAgent `com.comfyui.server`, port 8188, logs in `/tmp/comfyui.log`. It has
|
|
no authentication, so it listens only on `127.0.0.1` and Tart's bridge
|
|
`192.168.64.1` (it can't bind that until a VM is running; launchd retries).
|
|
|
|
| Model | Files | Workflow |
|
|
|-------|-------|----------|
|
|
| Qwen-Image 2512 (Q4_K_M GGUF) | `diffusion_models/`, GGUF Qwen2.5-VL text encoder, `qwen_image_vae` | `comfyui/workflows/qwen-image.json` |
|
|
| Stable Diffusion XL base 1.0 | `checkpoints/sd_xl_base_1.0.safetensors` | `comfyui/workflows/sdxl.json` |
|
|
|
|
The workflows are in ComfyUI's API format, as Open WebUI needs them (Admin
|
|
Panel → Settings → Images). Qwen-Image is quantized because bf16/fp8 weights
|
|
don't fit next to Ollama's models, and MPS can't run fp8.
|