arkey guidesRSS
All guides
guide2026-11-12inference

Beginner's Guide: How to Run an AI Model on Your Own Computer

How local AI fits together, for a beginner. There are three inference engines. Every app, window and terminal tool is a front for one of them, either a GUI or a TUI. The guide explains model files (GGUF, safetensors, AWQ), how much GPU memory each size needs, and how system RAM lets a mixture-of-experts model run on a small GPU.

inferencequantizationgpulocal-modelsBy Julian AbeledaProject arkeyModels qwen3-14b, qwen3.8-27b, qwen3-30b-a3b, qwen3.8-flash-next, deepseek-v4.1-flash, kimi-k3Created 2026-11-12Edited 2026-11-12
==============================================================================
  ARKEY GUIDES
  BEGINNER'S GUIDE: HOW TO RUN AN AI MODEL ON YOUR OWN COMPUTER
==============================================================================
  Version 1.0                                      Written 2026-11-12
  Platforms: Mac, Windows, Linux
  Read first: How to Use AI From the Terminal
==============================================================================

  TABLE OF CONTENTS                                           SEARCH CODE
  ------------------------------------------------------------------------
  1. What this guide gets you ................................... [INTR]
  2. The map .................................................... [MAPP]
  3. The engine ................................................. [ENGN]
  4. The front: GUI or TUI ...................................... [FRNT]
  5. The model, and where it comes from ......................... [MODL]
  6. Model files: safetensors, GGUF, AWQ ........................ [FILE]
  7. Quantization: making a model smaller ....................... [QUNT]
  8. How much GPU a dense model needs ........................... [DENS]
  9. Mixture of experts, and where RAM comes in ................. [MOEX]
  10. Run your first model ...................................... [PLAY]
  11. Use it for real work: Arkey ............................... [ARKY]
  12. My setup .................................................. [MYSU]
  13. When it goes wrong ........................................ [HELP]
  14. Word list ................................................. [WORD]
  15. Version history ........................................... [VERS]

  To jump to a section, press Ctrl+F (Cmd+F on a Mac) and type its
  search code with the brackets.

1. What this guide gets you [INTR]

At the end you can answer four questions about any local AI setup:

  1. Which engine is doing the work?
  2. Which file is the model, and what format is it?
  3. Does it fit in my GPU's memory?
  4. If it does not fit, can system RAM make up the difference?

You also run one model on your own computer, in section 10. In section 11 you connect it to the terminal AI tools from the first guide.

A local model is an AI model that runs on your computer. Nothing is sent to a company. It costs nothing per message. It works with no internet once the file is downloaded.

This guide uses the terminal. If you have not used one, read How to Use AI From the Terminal first.

2. The map [MAPP]

Every local setup has the same four layers. The names change. The layers do not.

   YOU
    |   type a message
    v
 +------------------------------------------------------------------------+
 | 4. FRONT        what you type into. It does none of the math.          |
 |                                                                        |
 |    GUI: a window or a web page     TUI: a program in the terminal      |
 |    LM Studio, Jan, Open WebUI,     Codex, Claude Code, Kimi Code,      |
 |    DeepSeek Harness ...            Crush, Ollama ...                   |
 +------------------------------------------------------------------------+
    |   passes your message down
    v
 +------------------------------------------------------------------------+
 | 3. ENGINE       the program that does the math                         |
 |                                                                        |
 |    llama.cpp              vLLM                 TensorRT-LLM            |
 +------------------------------------------------------------------------+
    |   reads                   |   reads              |   reads
    v                           v                      v
 +------------------------------------------------------------------------+
 | 2. MODEL FILE   the numbers the model learned                          |
 |                                                                        |
 |    GGUF                   safetensors          TensorRT engine file    |
 |    (quant inside)         (full, AWQ, FP8)     (built for one GPU)     |
 +------------------------------------------------------------------------+
    |   is loaded into          |                      |
    v                           v                      v
 +------------------------------------------------------------------------+
 | 1. MEMORY       where the file sits while it runs                      |
 |                                                                        |
 |    VRAM, RAM or both      VRAM                 VRAM (NVIDIA only)      |
 +------------------------------------------------------------------------+

   Model files come from Hugging Face (section 5).

Read it from the bottom up when you plan a setup:

  1. Memory. How much VRAM and RAM do you have? This decides everything above it.
  2. Model file. Which model, in which format, at which size, fits that memory?
  3. Engine. Which engine reads that format on your hardware?
  4. Front. The GUI or TUI you type into. This is the least important choice, and you can change it at any time.

Most beginners choose in the other order. They pick a front first, because it is the part they can see. Then they do not know why a model is slow or will not load.

3. The engine [ENGN]

An inference engine is the program that runs the model. It loads the model file into memory. It does the math on your GPU or CPU. It gives back the next word. Inference means running a model to get an answer.

There are three main engines.

EngineMade byRuns onReadsBuilt for
llama.cppAn open-source projectMac, Windows, Linux. Any GPU, or no GPU.GGUFOne person on one computer
vLLMAn open-source projectLinux, with NVIDIA or AMD GPUssafetensorsA server answering many people at once
TensorRT-LLMNVIDIANVIDIA GPUs onlyIts own engine file, built from safetensorsThe most speed from one NVIDIA GPU

What makes an engine an engine

Two things define an engine:

  1. Its model layout. The file format it loads, and how it arranges the model in memory.
  2. Its kernels. The small programs that do the model's math on the chip.

If a project has its own layout and its own kernels, it is an engine. If it uses another engine's layout and kernels, it is not a new engine. It is a fork or a front.

  • A fork is a copy of an engine's code with changes. A fork of llama.cpp still loads GGUF and still runs llama.cpp's kernels, with some edits. It is llama.cpp.
  • A front is the program you type into. It passes your message to an engine. Section 4 covers fronts.

This is why the list above is three long and not thirty. There are many forks and many fronts. Each one traces back to one of these three.

The test

To find the engine behind any tool, ask two questions:

  1. What file does it load? GGUF means llama.cpp.
  2. Whose code does the math?

A note for Mac users

Apple has its own engine, MLX. It runs only on Apple chips and reads its own file format. It passes the test above, so it is a real fourth engine. It is left out of the main three because it runs on one company's hardware. llama.cpp also runs on a Mac.

4. The front: GUI or TUI [FRNT]

The front is the part you see and type into. It takes your message, hands it to an engine, and shows the answer. It does none of the model's math.

The front is what people talk about most, because it is the visible part. The part that matters more is behind it: the engine. The engine decides which models you can run, how much memory they need and how fast they answer. The front decides what the window looks like.

A front is one of two kinds.

KindWhat it isYou use it by
GUIGraphical user interface. A desktop window, or a web page in your browser.Clicking and typing in a box
TUIText user interface. A program inside the terminal.Typing commands and requests

GUI fronts

FrontWhat it isEngine behind it
llama.cpp's chat pageA web page the engine serves itself. Section 10 uses it.llama.cpp
LM StudioA desktop app with a model browser and sliders for settings.llama.cpp. On a Mac it also carries MLX.
JanA desktop chat app. Open source.llama.cpp
KoboldCppOne file to run, with a web page. Made for stories and role-play.A fork of llama.cpp
Open WebUIA chat website on your own computer.None of its own. It connects to an engine you run.
DeepSeek HarnessA web page for chat and agent work, from DeepSeek.None of its own. It connects to a model you point it at.

TUI fronts

FrontWhat it isEngine behind it
OllamaOne command downloads and runs a model. It is a command more than a full-screen TUI.Built on llama.cpp's code. Loads GGUF. Newer versions run some models on Ollama's own code, built on the same base library.
Codex, Claude Code, Kimi Code, CrushThe terminal AI tools from the first guide.None of their own. They connect to a company's model, or to your engine through a translator (section 11).
ArkeyA menu that starts the engine and opens a tool (section 11).llama.cpp

Do not go by the name

A name does not tell you which layer a program is on. Check what it does.

DeepSeek Harness is the clearest case. The word harness sounds like a part under the hood, and many people take it for an engine. The word itself is fair: a harness is the program that runs the steps and tools around a model. It is still not an engine. Start it and it opens a web page at an address on your own computer. You type in a box. It sends your message to a model that runs somewhere else. On the map it is a front, and a GUI one.

The test from section 3 settles any name. What file does it load, and whose code does the math? If the answer is "none" and "someone else's", it is a front.

What this means for you

  • You do not need a separate front. llama.cpp serves its own chat page. My own setup in section 12 has no GUI in it.
  • A model that fits behind one front fits behind the others. Memory is decided by the model file and the engine, not the front.
  • A setting in the front is a setting in the engine. "GPU layers" in LM Studio is llama.cpp's --gpu-layers. Learn the engine's word once and you can find it in every front.
  • A front can be behind its engine. A new model can work in llama.cpp before the fronts pick it up.
  • A front hides the settings. The engine's flags, such as --cpu-moe in section 9, are easiest to reach when you run the engine yourself.
  • Choose the front last, and for how it feels. Choose the engine for your hardware.

5. The model, and where it comes from [MODL]

A model is a very large table of numbers. The numbers are called weights, or parameters. A model's size is how many it has. 14B means 14 billion.

Almost every open model is published on Hugging Face. It is a website that hosts model files, like an app store with no prices. Each model has a page, called a repo.

How to read a repo name:

   Qwen / Qwen3-14B-GGUF
   ----   -----------------
    |        |    |    |
    |        |    |    +---- the file format (section 6)
    |        |    +--------- the size: 14 billion weights
    |        +-------------- the model family and version
    +----------------------- who uploaded it

Who uploaded it matters. There are two kinds of uploader:

  • The maker. Qwen, google, deepseek-ai, moonshotai. This is the original model.
  • A converter. unsloth, bartowski, lmstudio-community. They take the maker's model and publish it in other formats and sizes. The model is the same. The file is different.

When a maker does not publish the format you need, a converter usually has.

6. Model files: safetensors, GGUF, AWQ [FILE]

These three words are often listed together. They are not the same kind of thing. Two are file formats. One is a way of shrinking.

WordWhat it isEngineWhat you get
safetensorsA file formatvLLM, TensorRT-LLMThe model as the maker published it. A folder of files.
GGUFA file formatllama.cppThe whole model in one file, usually already shrunk.
AWQA way of shrinkingvLLMA shrunk model, stored in safetensors files.

safetensors

The format makers publish in. It holds the weights and nothing else. The name says it is safe: the file cannot carry a program, only numbers. Older formats could.

A safetensors model is a folder. The weights are in one or more .safetensors files. The settings and the word list are in small files beside them. It is usually full size: 16 bits for every weight.

GGUF

The format llama.cpp reads. One file holds everything: the weights, the word list and the settings. You download one file and run it.

A GGUF repo holds many versions of the same model, each shrunk by a different amount. The name of each file says how much:

   Qwen3-14B-Q4_K_M.gguf
             -- - -
             |  | |
             |  | +---- M = medium. S is smaller, L is larger.
             |  +------ K = the method used to shrink it
             +--------- Q4 = about 4 bits for each weight

Q4_K_M is the common default. Q8_0 is nearly full quality and about twice the size.

AWQ

AWQ stands for activation-aware weight quantization. It is a method that shrinks a model to 4 bits for each weight. The result is saved as safetensors files, so the folder looks like a normal model. vLLM reads it.

AWQ is to vLLM what Q4_K_M is to llama.cpp: the usual way to make a model fit in less memory on that engine.

Others you will see

NameWhat it isEngine
FP88 bits for each weightvLLM, TensorRT-LLM
GPTQAnother 4-bit method, like AWQvLLM
NVFP4NVIDIA's 4-bit formatvLLM, TensorRT-LLM
MLXApple's format, in 4-bit and 8-bitMLX

You do not need to learn them. Read the name for two things: how many bits, and which engine.

7. Quantization: making a model smaller [QUNT]

Quantization is storing each weight with fewer bits. The short word is quant.

A maker publishes each weight as a 16-bit number. A quant stores it in 8, 5 or 4 bits. The file gets smaller by the same share. Some accuracy is lost, more as the bits go down.

The size of any model file is one sum:

   weights  x  bits for each weight  /  8  =  bytes

   14 billion  x  16 bits  /  8  =  28 GB    (full size)
   14 billion  x   8 bits  /  8  =  14 GB    (Q8)
   14 billion  x   4 bits  /  8  =   7 GB    (4 bits)

Real files are a little larger than the sum. A Q4_K_M file uses closer to 5 bits for each weight on average, because the method keeps the most important parts at higher precision.

The real sizes for one model, Qwen3-14B:

FileBits for each weightSize
safetensors, full1629.5 GB
GGUF Q8_0815.7 GB
GGUF Q6_Kabout 612.1 GB
GGUF Q5_K_Mabout 510.5 GB
AWQ410.0 GB
GGUF Q4_K_Mabout 49.0 GB

The same model goes from 29.5 GB to 9.0 GB. This is why a model that needs a data-center GPU at full size runs on a gaming GPU as a quant.

8. How much GPU a dense model needs [DENS]

Start with dense models. They are the simple case.

A dense model uses every weight for every word it writes. So the whole file has to be in fast memory the whole time.

VRAM and RAM

Your computer has two kinds of memory.

MemoryWhere it isSpeed for AI
VRAMOn the GPUFast
RAMOn the main board, used by the CPUMuch slower

On a Mac with an Apple chip there is one pool of memory shared by both. The model can use most of it, not all: macOS and your other programs need their share. Count about two thirds of it as VRAM. That share is a rule of thumb, not a measurement.

To see how much you have:

MAC

Click the Apple menu, then About This Mac. Read the Memory line.

WINDOWS

Press Ctrl + Shift + Esc. Click Performance, then GPU. Read Dedicated GPU memory.

LINUX

With an NVIDIA GPU:

nvidia-smi

The rule

The model file has to fit in VRAM, with room left over.

The room left over holds the conversation. Leave about 2 GB for a short chat, and more for long documents. This allowance is a rule of thumb. It was not measured for this guide.

14B

Qwen3-14B. File sizes are from Hugging Face. The GPU column is the file size plus the allowance, rounded up to a size GPUs are sold in.

FileSizeSmallest GPU that fits it
GGUF Q4_K_M9.0 GB12 GB
AWQ10.0 GB12 GB
GGUF Q5_K_M10.5 GB16 GB
GGUF Q6_K12.1 GB16 GB
GGUF Q8_015.7 GB24 GB
safetensors, full29.5 GB48 GB

27B

Qwen3.8-27B. GGUF sizes are from the unsloth conversion.

FileSizeSmallest GPU that fits it
GGUF Q4_K_M16.6 GB24 GB
GGUF Q5_K_M19.9 GB24 GB
GGUF Q6_K22.0 GB24 GB, with a short chat only
GGUF Q8_029.0 GB32 GB, with a short chat only
FP830.9 GB48 GB
safetensors, full55.6 GB80 GB, or two GPUs

Common GPU sizes

VRAMExamplesDense model it holds at Q4_K_M
8 GBRTX 4060Up to about 8B
12 GBRTX 3060 12 GB, RTX 407014B
16 GBRTX 4060 Ti 16 GB, RTX 508014B at a higher quant
24 GBRTX 3090, RTX 409027B
32 GBRTX 509027B at a higher quant
48 GB and upWorkstation and data-center GPUsLarger models, or full size

What happens when a dense model does not fit

llama.cpp can split a model: some layers in VRAM, the rest in RAM. The model runs. It runs much slower, because every word has to pass through the slow part. For a dense model, RAM is a way to make it load. It is not a way to make it usable.

So for a dense model the choice is simple. Pick a smaller quant, pick a smaller model, or get more VRAM.

9. Mixture of experts, and where RAM comes in [MOEX]

The largest open models are not dense. They are mixture-of-experts models, written MoE. For these, RAM matters.

What an expert is

An MoE model splits most of its weights into groups called experts. For each word, a small part of the model, the router, picks a few experts. Only those experts do work for that word. The rest sit idle.

   DENSE                               MIXTURE OF EXPERTS

   every word uses every weight        every word uses a few experts

   +--------------------------+        +----+----+----+----+----+----+
   |##########################|        |    | ## |    |    | ## |    |
   |##########################|        +----+----+----+----+----+----+
   |##########################|        |    |    |    | ## |    |    |
   |##########################|        +----+----+----+----+----+----+
   +--------------------------+        |    |    |    |    |    |    |
                                       +----+----+----+----+----+----+
   ## = weights used for this word       ## = experts used for this word

An MoE model has two sizes, and the name often gives both:

   Qwen3-30B-A3B
         ---  ---
          |    |
          |    +---- A3B = 3 billion weights are active for each word
          +--------- 30B = 30 billion weights in total

The total decides how much memory the model takes. The active part decides how much work each word takes.

Why this brings RAM in

A dense model in RAM is slow because every word reads every weight from slow memory.

An MoE model in RAM reads only the few experts each word picked. Most of the file is not touched for that word. So the experts can sit in RAM, and the model stays usable.

llama.cpp has a setting for exactly this:

FlagWhat it does
--cpu-moeKeeps all the expert weights in RAM. The rest of the model goes to the GPU.
--n-cpu-moe NKeeps the expert weights of the first N layers in RAM. The rest go to the GPU.

With these, the memory map changes:

   DENSE MODEL                         MoE MODEL WITH --cpu-moe

   +------------------+                +------------------+
   | VRAM             |                | VRAM             |
   |                  |                |  attention       |
   |  the whole model |                |  router          |
   |                  |                |  the conversation|
   +------------------+                +------------------+
   +------------------+                +------------------+
   | RAM              |                | RAM              |
   |  (not used)      |                |  all the experts |
   |                  |                |  (most of file)  |
   +------------------+                +------------------+

   Needs: VRAM >= file size            Needs: RAM >= file size
                                              VRAM for the rest

For a dense model, VRAM is the limit. For an MoE model, RAM is the limit and VRAM sets the speed. The more experts you can also fit in VRAM, the faster it runs. --n-cpu-moe is the setting you turn to move experts from RAM to VRAM a layer at a time.

No speed was measured for this guide. How fast an MoE model runs from RAM depends on the RAM's speed, the CPU and the model. Expect it to be slower than a model that sits fully in VRAM.

A small MoE model: 30B total, 3B active

Qwen3-30B-A3B. The Q4_K_M file is 18.6 GB.

SetupDoes it run?
24 GB GPUYes. The whole file fits in VRAM.
12 GB GPU and 32 GB RAMYes, with --cpu-moe. The experts sit in RAM.
12 GB GPU and 16 GB RAMTight. The file is 18.6 GB and the two memories add up to 28 GB. Use --n-cpu-moe to put part of the experts in VRAM, and close other programs.

Compare the dense 27B from section 8. Its Q4_K_M file is 16.6 GB, a little smaller. On a 12 GB GPU it loads only by putting layers in RAM, and then every word waits on the slow part. The MoE model is larger, and each word reads only a few experts from RAM. The difference is the architecture, not the size.

The large ones: Flash Next, DeepSeek, Kimi

The models that compete with the paid ones are all MoE, and all large. These are the Q4 GGUF sizes from the unsloth conversions.

ModelTotal weightsActive for each wordQ4 fileSmallest file published
Qwen3.8-Flash-Next125 billion, plus a 51 billion lookup table6 billion111.3 GB70.5 GB
DeepSeek-V4.1-FlashNot read for this guideNot read for this guide500.7 GB334.8 GB
Kimi-K32.8 trillion104 billion1,508.7 GB466.4 GB

What each one asks of a computer:

Qwen3.8-Flash-Next. No single home GPU holds 111 GB. With the experts in RAM, the limit moves to RAM: a computer with 128 GB of RAM holds the Q4 file. Only 6 billion weights are active for each word, fewer than a dense 8B model uses. Its model card says the lookup table, called an n-gram embedding, is easier to keep outside the GPU than experts are. This is the large model a home computer can reach.

DeepSeek-V4.1-Flash. The smallest file is 334.8 GB. That is more RAM than a desktop computer takes. It needs a workstation or a server with several hundred GB of RAM.

Kimi-K3. The smallest file is 466.4 GB, and 104 billion weights are active for each word. That active part alone is larger than any dense model in section 8. This is a data-center model. It is listed here to show where the top of the range is.

The rule for MoE

  1. Read the total size. Your RAM and VRAM together have to hold the file.
  2. Read the active size. It tells you how much work each word is.
  3. Put everything that is not an expert in VRAM. Put experts in VRAM until it is full. The rest go to RAM.

10. Run your first model [PLAY]

Pick the model from your memory, using sections 8 and 9. Then pick a path. Path A is the one to start with. It is the engine alone, with its own chat page.

Your memoryModel to tryFile
8 GBQwen3-8BQ4_K_M
12 or 16 GBQwen3-14BQ4_K_M, 9.0 GB
24 GB or moreQwen3.8-27BQ4_K_M, 16.6 GB
12 GB VRAM and 32 GB RAMQwen3-30B-A3B, with --cpu-moeQ4_K_M, 18.6 GB

Path A: llama.cpp

This is the engine with no front. You see every setting.

MAC and LINUX

brew install llama.cpp

WINDOWS

winget install llama.cpp

ALL THREE

One command downloads the file from Hugging Face and starts the engine:

llama-server -hf Qwen/Qwen3-14B-GGUF:Q4_K_M

When it says the server is listening, open this address in your browser:

http://localhost:8080

A chat page opens. It is running on your computer. To stop the engine, press Ctrl + C in the terminal.

For the MoE model with its experts in RAM:

llama-server -hf unsloth/Qwen3-30B-A3B-GGUF:Q4_K_M --cpu-moe

Path B: vLLM

For a Linux computer with an NVIDIA GPU. vLLM is a server engine. Use it when several people or programs will use the model at once.

pip install vllm
vllm serve Qwen/Qwen3-14B-AWQ

Path C: TensorRT-LLM

For NVIDIA GPUs. It first builds an engine file for your exact GPU, which takes extra steps. It is not a beginner's first engine. Follow NVIDIA's own guide.

Optional: a different front

Not needed. If you want a desktop window, download LM Studio or Ollama from its website. Both run on Mac, Windows and Linux, and both run llama.cpp underneath.

In LM Studio, search for the model, click download, then click chat.

In Ollama, one command downloads the model and starts a chat:

ollama run qwen3:14b

I ran none of the commands in this section for this guide. They are from each project's own pages.

11. Use it for real work: Arkey [ARKY]

Section 10 ends with a chat page. That proves the model runs. It does not yet do work on your files. For that, the terminal tools from the first guide have to use your local model in place of a company's model.

The problem

The tools and the engines do not speak the same language. Each one sends requests in its maker's own format, called a protocol.

ProgramProtocol it speaks
CodexOpenAI Responses
Claude CodeAnthropic Messages
CrushOpenAI Chat Completions
llama.cpp serverOpenAI Chat Completions first. It also accepts the other two, and its own notes say it makes no strong claim that they match.

So a tool can be pointed straight at a llama.cpp server, and simple requests work. The gaps show up in real work: how the model's thinking is sent back, how tools are named, and what happens when you change from a local model to a company's model. A translator in the middle closes those gaps and gives every tool one address to talk to.

The two parts

MoonBridge is the translator. It is a small program that sits between a tool and a model. It takes a request in one protocol and passes it on in another. I keep a version of it for Arkey: moon-bridge-arkey.

Arkey is the program I built to run all the layers of the map from one menu. It is open source: arkey_v3. You pick a tool and you pick where the model runs. Arkey starts the engine, starts MoonBridge, connects them, and opens the tool.

   YOU
    |
    v
 +------------------------------------------------------------------------+
 | ARKEY           one menu: pick a tool, pick a route                    |
 +------------------------------------------------------------------------+
    |   opens
    v
 +------------------------------------------------------------------------+
 | TOOL            Codex        Kimi Code        Crush                    |
 +------------------------------------------------------------------------+
    |   sends a request in its own protocol
    v
 +------------------------------------------------------------------------+
 | MOONBRIDGE      translates the request for the route you picked        |
 +------------------------------------------------------------------------+
    |                       |                        |
    v                       v                        v
 +--------------------+  +---------------------+  +----------------------+
 | LOCAL              |  | SERVER              |  | FRONTIER             |
 | llama.cpp on this  |  | a llama.cpp server  |  | a company's model:   |
 | computer. Arkey    |  | on another computer |  | DeepSeek, Codex,     |
 | starts it.         |  | you can reach.      |  | Claude.              |
 +--------------------+  +---------------------+  +----------------------+

This is the model-agnostic setup from the first guide, built. The tool stays the same. The route changes from one menu.

The three routes

RouteWhere the model runsWhen to use it
Localllama.cpp on this computerThe model fits your memory (sections 8 and 9).
ServerA llama.cpp server on another computerYou have a second computer with a bigger GPU, or a rented one.
FrontierA company's serversThe job needs a model too large to run yourself.

Install it

Arkey runs on Mac and Linux. It needs Git, Go 1.25 or newer, llama.cpp, and at least one terminal tool already installed from the first guide. Arkey does not include the tools. It makes its own copy of the ones you have and leaves your normal install untouched.

MAC

brew install git go llama.cpp

LINUX

Install Git and Go with your package manager, and llama.cpp as in section 10.

BOTH

git clone https://github.com/JulianAbeleda/arkey_v3.git
cd arkey_v3
scripts/install.sh

Open a new terminal window and start it:

arkey

The menu

  TUI
    Arkey Codex
    Arkey Claude     (not available yet)
    Arkey Kimi
    Arkey Crush
  Config
    Local
      llama.cpp
        your installed GGUF models
        Enter loads, r refreshes, d unloads
    Server
      one row for each server you listed
      Enter tests the server and makes it the route
    Frontier
      DeepSeek
      Codex
      Claude
    GPU Auto-scan
  Exit

A first run:

  1. Open Config, then Local, then llama.cpp. Your GGUF files from section 10 are listed.
  2. Press Enter on one. Arkey checks your GPU, starts llama.cpp and loads the model.
  3. Go back and open TUI. Press Enter on Arkey Codex.
  4. Codex opens. It is now running on your local model. Give it the request from the first guide: Make a folder on my Desktop named hello world.

Everything is in the open

Arkey does not hide what it does. Each layer of the map is a file you can open and read.

WhatWhere
Which model, engine, port and context size are selected~/.config/arkey/config.toml
MoonBridge's routes and translations~/.config/arkey/moonbridge.yml
Logs of what was started and what was sent~/.local/state/arkey
Arkey's copies of your tools~/.local/libexec/arkey/clients

To add a server route, list it in config.toml. The address is the other computer's llama.cpp server:

[[servers]]
label = "My other computer"
origin = "http://192.168.1.50:8080"

What it does not do yet

  • One local engine. The Local route runs llama.cpp. vLLM and TensorRT-LLM are not started by Arkey.
  • Claude Code. It is in the menu and marked unavailable. The translator does not yet accept Claude Code's protocol.
  • Windows. Not covered by this guide.

12. My setup [MYSU]

This is what I run every day. It has two computers. Every front in it is a TUI.

   LAPTOP                                    DESKTOP
   MacBook Air, Apple M5, 16 GB              Ubuntu Linux, with the GPUs
 +----------------------------------+      +---------------------------+
 | Terminal                         |      |                           |
 |   |                              |      |  ENGINE                   |
 |   v                              |      |  serves one large model   |
 | ARKEY          the menu          |      |  on port 8080             |
 |   |                              |      |                           |
 |   v                              |      |  MODEL FILE               |
 | TOOL           Codex, Kimi Code, |      |  held in the GPUs' VRAM   |
 |   |            Crush             |      |                           |
 |   v                              |      +---------------------------+
 | MOONBRIDGE     the translator    |                    ^
 |   |                              |                    |
 |   +--- Server route -------------|--- private network-+
 |   |                              |
 |   +--- Local route ----> llama.cpp on the laptop
 |   |                      Qwen3-8B, GGUF
 |   |
 |   +--- Frontier route -> DeepSeek, Claude, OpenAI
 +----------------------------------+

The parts

Layer of the mapWhat I use
Front, GUINone.
Front, TUIArkey's menu, started with arkey. Then Codex, Kimi Code and Crush. Claude Code is installed and waits on the translator.
TranslatorMoonBridge, on the laptop.
Engine, laptopllama.cpp from Homebrew.
Model, laptopQwen3-8B as one GGUF file. It fits in the laptop's 16 GB.
Engine and model, desktopA server on the desktop holds a larger model in its GPUs. The laptop reaches it over a private network.
FrontierDeepSeek, Claude and OpenAI, each through its own API key.

How I use it

  1. Most days: the Server route. I work on the laptop. The model runs on the desktop. The laptop stays cool and its battery lasts, and the model is larger than the laptop could hold. This is sections 8 and 9 in practice: put the model where the memory is.
  2. Away from the desktop: the Local route. Qwen3-8B runs on the laptop with no internet. It is a smaller model, and it is always there.
  3. When the job is too big: the Frontier route. One menu choice sends the same tool to a company's model.

The tool does not change between the three. I change the route in Arkey's menu and keep working.

Why two computers

A laptop has little memory for a model. A desktop with GPUs has a lot, and it is loud and stays at home. A private network joins them, so the laptop uses the desktop's GPUs from anywhere. I use Tailscale for this. It gives each of your computers an address only you can reach.

On the desktop, the engine has to accept connections from other computers. For llama.cpp that is one flag:

llama-server -hf Qwen/Qwen3-14B-GGUF:Q4_K_M --host 0.0.0.0

Only do this on a private network. With that flag, any computer that can reach the desktop can use the model.

On the laptop, the desktop is one entry in Arkey's config.toml, as in section 11.

What next

The tools in this setup are harnesses. The next guide explains what a harness is and builds one: What a Harness Is, and How AI Gets to Use Your Computer.

13. When it goes wrong [HELP]

What you seeWhyFix
out of memory, or the model will not loadThe file is larger than your VRAM.Use a smaller quant or a smaller model. For an MoE model, add --cpu-moe.
It loads, and each word takes secondsPart of a dense model is in RAM.Use a smaller quant so the whole file fits in VRAM.
It runs on the CPU, and the GPU is idleThe engine was installed without GPU support, or no layers were sent to the GPU.In llama.cpp add --gpu-layers all. In a GUI front, raise the GPU layers setting.
The download stops partwayThese files are large.Run the same command again. It resumes.
An MoE model is very slow with --cpu-moeAll the experts are in RAM.Use --n-cpu-moe N with a lower N, so more experts go to VRAM.
The computer freezes while loadingRAM is full and the computer is using the disk as memory.Close other programs, or use a smaller file.
vLLM will not install on a Mac or on WindowsvLLM is made for Linux.Use llama.cpp.

14. Word list [WORD]

  ACTIVE WEIGHTS   The weights an MoE model uses for one word.
  FRONT            What you type into: a GUI or a TUI. Does none of the math.
  AWQ              A method that shrinks a model to 4 bits. For vLLM.
  DENSE            A model that uses every weight for every word.
  ENGINE           The program that runs the model.
  EXPERT           One group of weights in an MoE model.
  FORK             A copy of an engine's code, with changes.
  GUI              A front that is a window or a web page.
  GGUF             The one-file model format llama.cpp reads.
  HUGGING FACE     The website where model files are published.
  INFERENCE        Running a model to get an answer.
  KERNEL           A small program that does the model's math on the chip.
  MOONBRIDGE       A translator between a tool's protocol and a model's.
  MoE              Mixture of experts. Only a few experts work on each word.
  PROTOCOL         The format a program sends its requests in.
  QUANT            A model stored with fewer bits for each weight.
  RAM              The computer's main memory. Slow for AI.
  ROUTE            Where the model runs: local, a server, or a company.
  SAFETENSORS      The file format makers publish models in.
  TUI              A front that runs inside the terminal.
  VRAM             The GPU's memory. Fast for AI.
  WEIGHT           One number the model learned. Also called a parameter.

15. Version history [VERS]

  1.0   2026-11-12   First version.