Run local models on the GPU

Create a Hoy VM with GPU acceleration, install the guest package in it with one command, run llama.cpp on your Mac's GPU, check it with hoy-gpu doctor, and know the limits.

Experimental. A local model that runs inside a Hoy VM can use your Mac's GPU. It is off by default and set per VM. It works for llama.cpp today; other programs get the GPU only in the cases listed under Limits.

Cua signs and notarizes the guest package. You install it inside the VM with one command, for your user only, with no administrator password.

What you need#

  • Hoy 0.17.0 or later, installed from the Hoy download with hoy doctor showing no failed check. The GPU renderer is part of the release.
  • A macOS image and a VM with enough memory for the model. A 12 GiB VM ran an 8 billion parameter model at 4 bits (4.6 GiB of weights) on a 24 GB Mac.
  • Your download token, the one in the install command on the Hoy download page. The package comes from the same place as Hoy itself.

1. Create a VM with GPU acceleration#

hoy create llm --image macos-26.5.2 --gpu-acceleration auto --memory 12G
hoy start llm --wait ready

Use the image name that hoy images shows. auto is the setting to use: each program gets the best GPU path that works for it. hoy doctor has a gpu-acceleration line that shows what the Mac offers, and hoy inspect llm shows the state of the VM's GPU renderer.

You can switch a stopped VM with hoy set llm --gpu-acceleration auto|off.

2. Install the guest package in the VM#

hoy ssh llm

Then, inside the VM:

curl -fsSL https://hoy.cua.ai/install.sh | HOY_TOKEN=<your token> bash -s -- --gpu-guest

The command downloads the package, checks its SHA-256 and that every program in it is signed by Cua's Developer ID (team YCK386LBJ7), installs it in ~/.local/hoy-gpu, and runs hoy-gpu doctor. Run it again to update. Add the tools to your PATH:

export PATH="$HOME/.local/hoy-gpu/bin:$PATH"

The package holds llama.cpp (llama-server, llama-cli, llama-bench, llama-completion) and the hoy-gpu launcher. Its licenses are in ~/.local/hoy-gpu/THIRD_PARTY_NOTICES.txt.

If the VM has no internet access, download the package on your Mac instead, with the same command and HOY_DOWNLOAD_ONLY=1 HOY_OUT=<a folder> in front of sh. It saves hoy-gpu-guest-<version>-arm64.tar.gz and its .sha256 there, checked. Copy the file into the VM with scp, then run tar xzf on it and sh hoy-gpu/install.sh.

3. Check it#

hoy-gpu doctor

hoy-gpu doctor lists the GPU paths this VM has and says why a path is missing. It exits 0 when at least one path works. hoy-gpu doctor llama-server says which path that program would get and why.

4. Run a model#

Download a model in GGUF format, then start the server:

curl -L -o model.gguf <URL of a .gguf file>
llama-server -m model.gguf -ngl 99 --port 8080

-ngl 99 puts every layer on the GPU. In another shell in the VM:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hello"}],"max_tokens":32}'

llama-bench -m model.gguf -ngl 99 measures speed. To see which path a run uses, start it with HOY_GPU_EXPLAIN=1. To choose one, use HOY_GPU_PATH=renderer or HOY_GPU_PATH=shim; an explicit choice never falls back to the other path.

How it works#

Two paths exist, and auto picks per program. For llama.cpp it picks the renderer for models under 3 GiB and for VMs without a display, and the shim for larger models that fit in the VM's free memory:

  • Renderer. llama.cpp in the VM sends its compute graphs over the VM's vsock device to a per-VM renderer on your Mac, which runs them on the GPU with Metal. The model, prompts and sampling stay in the VM. It works in headless VMs and is fastest for small models.
  • Shim. For large models that fit in the VM's memory, the launcher loads a small public library that makes the VM's own paravirtual Metal report the features that stock Metal code needs. Loading models runs at native speed, and snapshot and fork keep working.

Hoy's background service starts the renderer when the VM starts, restarts it if it stops, and stops it with the VM. VMs without GPU acceleration have no renderer and use no extra memory.

Measured speed#

Measured on an Apple M2 Mac with 24 GB (macOS 26.7), a 12 GiB VM with auto, llama.cpp llama-bench (512-token prompt, 128 generated tokens, Q4_K_M models), with nothing else running. Percentages compare the VM with the same build running on the Mac itself.

ModelPath auto choseReading the promptGenerating textOn the Mac itself (prompt / generate)
Qwen2.5 0.5Brenderer94%97%2,812 / 154 tokens/s
Llama 3.2 3Brenderer99%82%427 / 41 tokens/s
Llama 3.1 8Bshim100%98%177 / 19 tokens/s

On the 3B model the other path generates faster (shim: 99%, renderer: 82%). Force a path with HOY_GPU_PATH if you care about the last 15%. The numbers come from one machine and will differ on yours; a busy Mac or a second VM lowers them.

Limits#

  • Experimental. Behavior, names and defaults can still change. Use it with VMs you trust: the Mac-side renderer reads what the VM sends it, and its hardening against hostile VMs is not finished.
  • llama.cpp only. The package holds llama.cpp. Your own programs get the GPU only if they accept an injected library (the shim path) or are built with Hoy's backend (the renderer path); list yours with hoy-gpu programs and register more in ~/.config/hoy-gpu/programs/<name>.conf. The renderer accepts only the llama.cpp version in the package.
  • The official Ollama app does not get the GPU. It is signed with the hardened runtime and carries no entitlement that allows injected libraries, so macOS removes DYLD_INSERT_LIBRARIES and the shim cannot load into it. hoy-gpu run says so ("no GPU acceleration ... it runs on the VM's plain virtual GPU") and starts it anyway. What that means in practice: Llama 3.2 3B generated 41.5 tokens/s on the Mac and 0.26 tokens/s in the VM, through hoy-gpu run or without it. We do not re-sign Ollama and we do not change macOS security settings in the VM. Use llama-server, which has an OpenAI-compatible API, or another program built for this.
  • Snapshots. A VM cannot be snapshotted while a program in it has a renderer session open (exit 5). Stop the program, then snapshot. Restore and fork work. After a snapshot exists, auto prefers the shim for new programs.
  • Headless VMs have no Metal device, so only the renderer path exists there. A model that does not fit the renderer's GPU memory per program then runs without GPU help, and hoy-gpu says why. --gpu-acceleration shim needs a VM with graphics.
  • GPU memory. The renderer has a GPU memory budget that Hoy sets from the VM's memory and what the Mac has beside it, and each program gets an equal share (hoy-gpu doctor shows it, for example 4 GiB on a 24 GB Mac). A model larger than the share goes to the shim path when the VM has graphics and enough memory. In a VM without graphics, a model larger than the share runs without GPU help.
  • Loading models over the renderer path is slower than on the Mac, because the weights cross the VM's vsock device. The shim path loads at native speed.
  • What was measured. One Apple M2 Mac with a macOS 26.6 VM. Other chips and macOS 27 guests are not covered by the numbers above.