local-llm/README.md

local-llm

Local LLM setup on the Mac mini: Ollama serving models, opencode using them, and ComfyUI generating images. Open WebUI (https://ai.quineglobal.com, on the Tart VM, see petersweb-infra) uses Ollama and ComfyUI over Tart's bridge (192.168.64.1).

./setup.sh

does everything, and is safe to re-run:

• installs ollama/com.ollama.server.plist as the LaunchAgent that runs ollama serve (reloaded only when it changed, since that restarts Ollama);
• builds each model in ollama/models/ (pulling its base model if missing);
• installs ComfyUI in ~/ComfyUI (pinned tag + ComfyUI-GGUF), downloads the models in comfyui/models.txt, and installs comfyui/com.comfyui.server.plist;
• symlinks ~/.config/opencode/opencode.json to opencode/opencode.json, so edits made either way end up here.

Needs brew install ollama opencode (setup.sh installs uv itself).

Models

Model	From	Used for
qwen3-coder:30b-128k	qwen3-coder:30b	opencode's default (128k context)
qwen3:32b-32k	qwen3:32b	opencode, as an option
inspector	qwen3:32b	16k-context variant

Each derived model mostly exists to set num_ctx. Ollama's OpenAI-compatible API, which opencode talks to, ignores any context size sent with a request, so without it a long agent prompt is silently cut to Ollama's small default context. A Modelfile's # name: line is the name it's created under.

qwen3:32b is a thinking model, and in opencode it often ends its turn after thinking, without making the tool call it planned. qwen3-coder was trained for agentic tool use and doesn't think first, so it doesn't have that problem, and it's much faster: a mixture of experts with ~3B active parameters.

Server settings

From the LaunchAgent: listens on all interfaces (0.0.0.0:11434), keeps models loaded for 30 minutes after last use (OLLAMA_KEEP_ALIVE=30m, so ComfyUI has room on the 64 GB machine), flash attention on. Logs go to /tmp/ollama.log.

ComfyUI

LaunchAgent com.comfyui.server, port 8188, logs in /tmp/comfyui.log. It has no authentication, so it listens only on 127.0.0.1 and Tart's bridge 192.168.64.1 (it can't bind that until a VM is running; launchd retries).

Model	Files	Workflow
Qwen-Image 2512 (Q4_K_M GGUF)	diffusion_models/, GGUF Qwen2.5-VL text encoder, qwen_image_vae	comfyui/workflows/qwen-image.json
Stable Diffusion XL base 1.0	checkpoints/sd_xl_base_1.0.safetensors	comfyui/workflows/sdxl.json

The workflows are in ComfyUI's API format, as Open WebUI needs them (Admin Panel → Settings → Images). Qwen-Image is quantized because bf16/fp8 weights don't fit next to Ollama's models, and MPS can't run fp8.