Beginner's Guide: How to Run an AI Model on Your Own Computer
How local AI fits together, for a beginner. There are three inference engines. Every app, window and terminal tool is a front for one of them, either a GUI or a TUI. The guide explains model files (GGUF, safetensors, AWQ), how much GPU memory each size needs, and how system RAM lets a mixture-of-experts model run on a small GPU.
==============================================================================
ARKEY GUIDES
BEGINNER'S GUIDE: HOW TO RUN AN AI MODEL ON YOUR OWN COMPUTER
==============================================================================
Version 1.0 Written 2026-11-12
Platforms: Mac, Windows, Linux
Read first: How to Use AI From the Terminal
==============================================================================
TABLE OF CONTENTS SEARCH CODE
------------------------------------------------------------------------
1. What this guide gets you ................................... [INTR]
2. The map .................................................... [MAPP]
3. The engine ................................................. [ENGN]
4. The front: GUI or TUI ...................................... [FRNT]
5. The model, and where it comes from ......................... [MODL]
6. Model files: safetensors, GGUF, AWQ ........................ [FILE]
7. Quantization: making a model smaller ....................... [QUNT]
8. How much GPU a dense model needs ........................... [DENS]
9. Mixture of experts, and where RAM comes in ................. [MOEX]
10. Run your first model ...................................... [PLAY]
11. Use it for real work: Arkey ............................... [ARKY]
12. My setup .................................................. [MYSU]
13. When it goes wrong ........................................ [HELP]
14. Word list ................................................. [WORD]
15. Version history ........................................... [VERS]
To jump to a section, press Ctrl+F (Cmd+F on a Mac) and type its
search code with the brackets.
1. What this guide gets you [INTR]
At the end you can answer four questions about any local AI setup:
- Which engine is doing the work?
- Which file is the model, and what format is it?
- Does it fit in my GPU's memory?
- If it does not fit, can system RAM make up the difference?
You also run one model on your own computer, in section 10. In section 11 you connect it to the terminal AI tools from the first guide.
A local model is an AI model that runs on your computer. Nothing is sent to a company. It costs nothing per message. It works with no internet once the file is downloaded.
This guide uses the terminal. If you have not used one, read How to Use AI From the Terminal first.
2. The map [MAPP]
Every local setup has the same four layers. The names change. The layers do not.
YOU
| type a message
v
+------------------------------------------------------------------------+
| 4. FRONT what you type into. It does none of the math. |
| |
| GUI: a window or a web page TUI: a program in the terminal |
| LM Studio, Jan, Open WebUI, Codex, Claude Code, Kimi Code, |
| DeepSeek Harness ... Crush, Ollama ... |
+------------------------------------------------------------------------+
| passes your message down
v
+------------------------------------------------------------------------+
| 3. ENGINE the program that does the math |
| |
| llama.cpp vLLM TensorRT-LLM |
+------------------------------------------------------------------------+
| reads | reads | reads
v v v
+------------------------------------------------------------------------+
| 2. MODEL FILE the numbers the model learned |
| |
| GGUF safetensors TensorRT engine file |
| (quant inside) (full, AWQ, FP8) (built for one GPU) |
+------------------------------------------------------------------------+
| is loaded into | |
v v v
+------------------------------------------------------------------------+
| 1. MEMORY where the file sits while it runs |
| |
| VRAM, RAM or both VRAM VRAM (NVIDIA only) |
+------------------------------------------------------------------------+
Model files come from Hugging Face (section 5).
Read it from the bottom up when you plan a setup:
- Memory. How much VRAM and RAM do you have? This decides everything above it.
- Model file. Which model, in which format, at which size, fits that memory?
- Engine. Which engine reads that format on your hardware?
- Front. The GUI or TUI you type into. This is the least important choice, and you can change it at any time.
Most beginners choose in the other order. They pick a front first, because it is the part they can see. Then they do not know why a model is slow or will not load.
3. The engine [ENGN]
An inference engine is the program that runs the model. It loads the model file into memory. It does the math on your GPU or CPU. It gives back the next word. Inference means running a model to get an answer.
There are three main engines.
| Engine | Made by | Runs on | Reads | Built for |
|---|---|---|---|---|
| llama.cpp | An open-source project | Mac, Windows, Linux. Any GPU, or no GPU. | GGUF | One person on one computer |
| vLLM | An open-source project | Linux, with NVIDIA or AMD GPUs | safetensors | A server answering many people at once |
| TensorRT-LLM | NVIDIA | NVIDIA GPUs only | Its own engine file, built from safetensors | The most speed from one NVIDIA GPU |
What makes an engine an engine
Two things define an engine:
- Its model layout. The file format it loads, and how it arranges the model in memory.
- Its kernels. The small programs that do the model's math on the chip.
If a project has its own layout and its own kernels, it is an engine. If it uses another engine's layout and kernels, it is not a new engine. It is a fork or a front.
- A fork is a copy of an engine's code with changes. A fork of llama.cpp still loads GGUF and still runs llama.cpp's kernels, with some edits. It is llama.cpp.
- A front is the program you type into. It passes your message to an engine. Section 4 covers fronts.
This is why the list above is three long and not thirty. There are many forks and many fronts. Each one traces back to one of these three.
The test
To find the engine behind any tool, ask two questions:
- What file does it load? GGUF means llama.cpp.
- Whose code does the math?
A note for Mac users
Apple has its own engine, MLX. It runs only on Apple chips and reads its own file format. It passes the test above, so it is a real fourth engine. It is left out of the main three because it runs on one company's hardware. llama.cpp also runs on a Mac.
4. The front: GUI or TUI [FRNT]
The front is the part you see and type into. It takes your message, hands it to an engine, and shows the answer. It does none of the model's math.
The front is what people talk about most, because it is the visible part. The part that matters more is behind it: the engine. The engine decides which models you can run, how much memory they need and how fast they answer. The front decides what the window looks like.
A front is one of two kinds.
| Kind | What it is | You use it by |
|---|---|---|
| GUI | Graphical user interface. A desktop window, or a web page in your browser. | Clicking and typing in a box |
| TUI | Text user interface. A program inside the terminal. | Typing commands and requests |
GUI fronts
| Front | What it is | Engine behind it |
|---|---|---|
| llama.cpp's chat page | A web page the engine serves itself. Section 10 uses it. | llama.cpp |
| LM Studio | A desktop app with a model browser and sliders for settings. | llama.cpp. On a Mac it also carries MLX. |
| Jan | A desktop chat app. Open source. | llama.cpp |
| KoboldCpp | One file to run, with a web page. Made for stories and role-play. | A fork of llama.cpp |
| Open WebUI | A chat website on your own computer. | None of its own. It connects to an engine you run. |
| DeepSeek Harness | A web page for chat and agent work, from DeepSeek. | None of its own. It connects to a model you point it at. |
TUI fronts
| Front | What it is | Engine behind it |
|---|---|---|
| Ollama | One command downloads and runs a model. It is a command more than a full-screen TUI. | Built on llama.cpp's code. Loads GGUF. Newer versions run some models on Ollama's own code, built on the same base library. |
| Codex, Claude Code, Kimi Code, Crush | The terminal AI tools from the first guide. | None of their own. They connect to a company's model, or to your engine through a translator (section 11). |
| Arkey | A menu that starts the engine and opens a tool (section 11). | llama.cpp |
Do not go by the name
A name does not tell you which layer a program is on. Check what it does.
DeepSeek Harness is the clearest case. The word harness sounds like a part under the hood, and many people take it for an engine. The word itself is fair: a harness is the program that runs the steps and tools around a model. It is still not an engine. Start it and it opens a web page at an address on your own computer. You type in a box. It sends your message to a model that runs somewhere else. On the map it is a front, and a GUI one.
The test from section 3 settles any name. What file does it load, and whose code does the math? If the answer is "none" and "someone else's", it is a front.
What this means for you
- You do not need a separate front. llama.cpp serves its own chat page. My own setup in section 12 has no GUI in it.
- A model that fits behind one front fits behind the others. Memory is decided by the model file and the engine, not the front.
- A setting in the front is a setting in the engine. "GPU layers" in LM Studio is llama.cpp's
--gpu-layers. Learn the engine's word once and you can find it in every front. - A front can be behind its engine. A new model can work in llama.cpp before the fronts pick it up.
- A front hides the settings. The engine's flags, such as
--cpu-moein section 9, are easiest to reach when you run the engine yourself. - Choose the front last, and for how it feels. Choose the engine for your hardware.
5. The model, and where it comes from [MODL]
A model is a very large table of numbers. The numbers are called weights, or parameters. A model's size is how many it has. 14B means 14 billion.
Almost every open model is published on Hugging Face. It is a website that hosts model files, like an app store with no prices. Each model has a page, called a repo.
How to read a repo name:
Qwen / Qwen3-14B-GGUF
---- -----------------
| | | |
| | | +---- the file format (section 6)
| | +--------- the size: 14 billion weights
| +-------------- the model family and version
+----------------------- who uploaded it
Who uploaded it matters. There are two kinds of uploader:
- The maker.
Qwen,google,deepseek-ai,moonshotai. This is the original model. - A converter.
unsloth,bartowski,lmstudio-community. They take the maker's model and publish it in other formats and sizes. The model is the same. The file is different.
When a maker does not publish the format you need, a converter usually has.
6. Model files: safetensors, GGUF, AWQ [FILE]
These three words are often listed together. They are not the same kind of thing. Two are file formats. One is a way of shrinking.
| Word | What it is | Engine | What you get |
|---|---|---|---|
| safetensors | A file format | vLLM, TensorRT-LLM | The model as the maker published it. A folder of files. |
| GGUF | A file format | llama.cpp | The whole model in one file, usually already shrunk. |
| AWQ | A way of shrinking | vLLM | A shrunk model, stored in safetensors files. |
safetensors
The format makers publish in. It holds the weights and nothing else. The name says it is safe: the file cannot carry a program, only numbers. Older formats could.
A safetensors model is a folder. The weights are in one or more .safetensors files. The settings and the word list are in small files beside them. It is usually full size: 16 bits for every weight.
GGUF
The format llama.cpp reads. One file holds everything: the weights, the word list and the settings. You download one file and run it.
A GGUF repo holds many versions of the same model, each shrunk by a different amount. The name of each file says how much:
Qwen3-14B-Q4_K_M.gguf
-- - -
| | |
| | +---- M = medium. S is smaller, L is larger.
| +------ K = the method used to shrink it
+--------- Q4 = about 4 bits for each weight
Q4_K_M is the common default. Q8_0 is nearly full quality and about twice the size.
AWQ
AWQ stands for activation-aware weight quantization. It is a method that shrinks a model to 4 bits for each weight. The result is saved as safetensors files, so the folder looks like a normal model. vLLM reads it.
AWQ is to vLLM what Q4_K_M is to llama.cpp: the usual way to make a model fit in less memory on that engine.
Others you will see
| Name | What it is | Engine |
|---|---|---|
| FP8 | 8 bits for each weight | vLLM, TensorRT-LLM |
| GPTQ | Another 4-bit method, like AWQ | vLLM |
| NVFP4 | NVIDIA's 4-bit format | vLLM, TensorRT-LLM |
| MLX | Apple's format, in 4-bit and 8-bit | MLX |
You do not need to learn them. Read the name for two things: how many bits, and which engine.
7. Quantization: making a model smaller [QUNT]
Quantization is storing each weight with fewer bits. The short word is quant.
A maker publishes each weight as a 16-bit number. A quant stores it in 8, 5 or 4 bits. The file gets smaller by the same share. Some accuracy is lost, more as the bits go down.
The size of any model file is one sum:
weights x bits for each weight / 8 = bytes
14 billion x 16 bits / 8 = 28 GB (full size)
14 billion x 8 bits / 8 = 14 GB (Q8)
14 billion x 4 bits / 8 = 7 GB (4 bits)
Real files are a little larger than the sum. A Q4_K_M file uses closer to 5 bits for each weight on average, because the method keeps the most important parts at higher precision.
The real sizes for one model, Qwen3-14B:
| File | Bits for each weight | Size |
|---|---|---|
| safetensors, full | 16 | 29.5 GB |
GGUF Q8_0 | 8 | 15.7 GB |
GGUF Q6_K | about 6 | 12.1 GB |
GGUF Q5_K_M | about 5 | 10.5 GB |
| AWQ | 4 | 10.0 GB |
GGUF Q4_K_M | about 4 | 9.0 GB |
The same model goes from 29.5 GB to 9.0 GB. This is why a model that needs a data-center GPU at full size runs on a gaming GPU as a quant.
8. How much GPU a dense model needs [DENS]
Start with dense models. They are the simple case.
A dense model uses every weight for every word it writes. So the whole file has to be in fast memory the whole time.
VRAM and RAM
Your computer has two kinds of memory.
| Memory | Where it is | Speed for AI |
|---|---|---|
| VRAM | On the GPU | Fast |
| RAM | On the main board, used by the CPU | Much slower |
On a Mac with an Apple chip there is one pool of memory shared by both. The model can use most of it, not all: macOS and your other programs need their share. Count about two thirds of it as VRAM. That share is a rule of thumb, not a measurement.
To see how much you have:
MAC
Click the Apple menu, then About This Mac. Read the Memory line.
WINDOWS
Press Ctrl + Shift + Esc. Click Performance, then GPU. Read Dedicated GPU memory.
LINUX
With an NVIDIA GPU:
nvidia-smi
The rule
The model file has to fit in VRAM, with room left over.
The room left over holds the conversation. Leave about 2 GB for a short chat, and more for long documents. This allowance is a rule of thumb. It was not measured for this guide.
14B
Qwen3-14B. File sizes are from Hugging Face. The GPU column is the file size plus the allowance, rounded up to a size GPUs are sold in.
| File | Size | Smallest GPU that fits it |
|---|---|---|
GGUF Q4_K_M | 9.0 GB | 12 GB |
| AWQ | 10.0 GB | 12 GB |
GGUF Q5_K_M | 10.5 GB | 16 GB |
GGUF Q6_K | 12.1 GB | 16 GB |
GGUF Q8_0 | 15.7 GB | 24 GB |
| safetensors, full | 29.5 GB | 48 GB |
27B
Qwen3.8-27B. GGUF sizes are from the unsloth conversion.
| File | Size | Smallest GPU that fits it |
|---|---|---|
GGUF Q4_K_M | 16.6 GB | 24 GB |
GGUF Q5_K_M | 19.9 GB | 24 GB |
GGUF Q6_K | 22.0 GB | 24 GB, with a short chat only |
GGUF Q8_0 | 29.0 GB | 32 GB, with a short chat only |
| FP8 | 30.9 GB | 48 GB |
| safetensors, full | 55.6 GB | 80 GB, or two GPUs |
Common GPU sizes
| VRAM | Examples | Dense model it holds at Q4_K_M |
|---|---|---|
| 8 GB | RTX 4060 | Up to about 8B |
| 12 GB | RTX 3060 12 GB, RTX 4070 | 14B |
| 16 GB | RTX 4060 Ti 16 GB, RTX 5080 | 14B at a higher quant |
| 24 GB | RTX 3090, RTX 4090 | 27B |
| 32 GB | RTX 5090 | 27B at a higher quant |
| 48 GB and up | Workstation and data-center GPUs | Larger models, or full size |
What happens when a dense model does not fit
llama.cpp can split a model: some layers in VRAM, the rest in RAM. The model runs. It runs much slower, because every word has to pass through the slow part. For a dense model, RAM is a way to make it load. It is not a way to make it usable.
So for a dense model the choice is simple. Pick a smaller quant, pick a smaller model, or get more VRAM.
9. Mixture of experts, and where RAM comes in [MOEX]
The largest open models are not dense. They are mixture-of-experts models, written MoE. For these, RAM matters.
What an expert is
An MoE model splits most of its weights into groups called experts. For each word, a small part of the model, the router, picks a few experts. Only those experts do work for that word. The rest sit idle.
DENSE MIXTURE OF EXPERTS
every word uses every weight every word uses a few experts
+--------------------------+ +----+----+----+----+----+----+
|##########################| | | ## | | | ## | |
|##########################| +----+----+----+----+----+----+
|##########################| | | | | ## | | |
|##########################| +----+----+----+----+----+----+
+--------------------------+ | | | | | | |
+----+----+----+----+----+----+
## = weights used for this word ## = experts used for this word
An MoE model has two sizes, and the name often gives both:
Qwen3-30B-A3B
--- ---
| |
| +---- A3B = 3 billion weights are active for each word
+--------- 30B = 30 billion weights in total
The total decides how much memory the model takes. The active part decides how much work each word takes.
Why this brings RAM in
A dense model in RAM is slow because every word reads every weight from slow memory.
An MoE model in RAM reads only the few experts each word picked. Most of the file is not touched for that word. So the experts can sit in RAM, and the model stays usable.
llama.cpp has a setting for exactly this:
| Flag | What it does |
|---|---|
--cpu-moe | Keeps all the expert weights in RAM. The rest of the model goes to the GPU. |
--n-cpu-moe N | Keeps the expert weights of the first N layers in RAM. The rest go to the GPU. |
With these, the memory map changes:
DENSE MODEL MoE MODEL WITH --cpu-moe
+------------------+ +------------------+
| VRAM | | VRAM |
| | | attention |
| the whole model | | router |
| | | the conversation|
+------------------+ +------------------+
+------------------+ +------------------+
| RAM | | RAM |
| (not used) | | all the experts |
| | | (most of file) |
+------------------+ +------------------+
Needs: VRAM >= file size Needs: RAM >= file size
VRAM for the rest
For a dense model, VRAM is the limit. For an MoE model, RAM is the limit and VRAM sets the speed. The more experts you can also fit in VRAM, the faster it runs. --n-cpu-moe is the setting you turn to move experts from RAM to VRAM a layer at a time.
No speed was measured for this guide. How fast an MoE model runs from RAM depends on the RAM's speed, the CPU and the model. Expect it to be slower than a model that sits fully in VRAM.
A small MoE model: 30B total, 3B active
Qwen3-30B-A3B. The Q4_K_M file is 18.6 GB.
| Setup | Does it run? |
|---|---|
| 24 GB GPU | Yes. The whole file fits in VRAM. |
| 12 GB GPU and 32 GB RAM | Yes, with --cpu-moe. The experts sit in RAM. |
| 12 GB GPU and 16 GB RAM | Tight. The file is 18.6 GB and the two memories add up to 28 GB. Use --n-cpu-moe to put part of the experts in VRAM, and close other programs. |
Compare the dense 27B from section 8. Its Q4_K_M file is 16.6 GB, a little smaller. On a 12 GB GPU it loads only by putting layers in RAM, and then every word waits on the slow part. The MoE model is larger, and each word reads only a few experts from RAM. The difference is the architecture, not the size.
The large ones: Flash Next, DeepSeek, Kimi
The models that compete with the paid ones are all MoE, and all large. These are the Q4 GGUF sizes from the unsloth conversions.
| Model | Total weights | Active for each word | Q4 file | Smallest file published |
|---|---|---|---|---|
| Qwen3.8-Flash-Next | 125 billion, plus a 51 billion lookup table | 6 billion | 111.3 GB | 70.5 GB |
| DeepSeek-V4.1-Flash | Not read for this guide | Not read for this guide | 500.7 GB | 334.8 GB |
| Kimi-K3 | 2.8 trillion | 104 billion | 1,508.7 GB | 466.4 GB |
What each one asks of a computer:
Qwen3.8-Flash-Next. No single home GPU holds 111 GB. With the experts in RAM, the limit moves to RAM: a computer with 128 GB of RAM holds the Q4 file. Only 6 billion weights are active for each word, fewer than a dense 8B model uses. Its model card says the lookup table, called an n-gram embedding, is easier to keep outside the GPU than experts are. This is the large model a home computer can reach.
DeepSeek-V4.1-Flash. The smallest file is 334.8 GB. That is more RAM than a desktop computer takes. It needs a workstation or a server with several hundred GB of RAM.
Kimi-K3. The smallest file is 466.4 GB, and 104 billion weights are active for each word. That active part alone is larger than any dense model in section 8. This is a data-center model. It is listed here to show where the top of the range is.
The rule for MoE
- Read the total size. Your RAM and VRAM together have to hold the file.
- Read the active size. It tells you how much work each word is.
- Put everything that is not an expert in VRAM. Put experts in VRAM until it is full. The rest go to RAM.
10. Run your first model [PLAY]
Pick the model from your memory, using sections 8 and 9. Then pick a path. Path A is the one to start with. It is the engine alone, with its own chat page.
| Your memory | Model to try | File |
|---|---|---|
| 8 GB | Qwen3-8B | Q4_K_M |
| 12 or 16 GB | Qwen3-14B | Q4_K_M, 9.0 GB |
| 24 GB or more | Qwen3.8-27B | Q4_K_M, 16.6 GB |
| 12 GB VRAM and 32 GB RAM | Qwen3-30B-A3B, with --cpu-moe | Q4_K_M, 18.6 GB |
Path A: llama.cpp
This is the engine with no front. You see every setting.
MAC and LINUX
brew install llama.cpp
WINDOWS
winget install llama.cpp
ALL THREE
One command downloads the file from Hugging Face and starts the engine:
llama-server -hf Qwen/Qwen3-14B-GGUF:Q4_K_M
When it says the server is listening, open this address in your browser:
http://localhost:8080
A chat page opens. It is running on your computer. To stop the engine, press Ctrl + C in the terminal.
For the MoE model with its experts in RAM:
llama-server -hf unsloth/Qwen3-30B-A3B-GGUF:Q4_K_M --cpu-moe
Path B: vLLM
For a Linux computer with an NVIDIA GPU. vLLM is a server engine. Use it when several people or programs will use the model at once.
pip install vllm
vllm serve Qwen/Qwen3-14B-AWQ
Path C: TensorRT-LLM
For NVIDIA GPUs. It first builds an engine file for your exact GPU, which takes extra steps. It is not a beginner's first engine. Follow NVIDIA's own guide.
Optional: a different front
Not needed. If you want a desktop window, download LM Studio or Ollama from its website. Both run on Mac, Windows and Linux, and both run llama.cpp underneath.
In LM Studio, search for the model, click download, then click chat.
In Ollama, one command downloads the model and starts a chat:
ollama run qwen3:14b
I ran none of the commands in this section for this guide. They are from each project's own pages.
11. Use it for real work: Arkey [ARKY]
Section 10 ends with a chat page. That proves the model runs. It does not yet do work on your files. For that, the terminal tools from the first guide have to use your local model in place of a company's model.
The problem
The tools and the engines do not speak the same language. Each one sends requests in its maker's own format, called a protocol.
| Program | Protocol it speaks |
|---|---|
| Codex | OpenAI Responses |
| Claude Code | Anthropic Messages |
| Crush | OpenAI Chat Completions |
| llama.cpp server | OpenAI Chat Completions first. It also accepts the other two, and its own notes say it makes no strong claim that they match. |
So a tool can be pointed straight at a llama.cpp server, and simple requests work. The gaps show up in real work: how the model's thinking is sent back, how tools are named, and what happens when you change from a local model to a company's model. A translator in the middle closes those gaps and gives every tool one address to talk to.
The two parts
MoonBridge is the translator. It is a small program that sits between a tool and a model. It takes a request in one protocol and passes it on in another. I keep a version of it for Arkey: moon-bridge-arkey.
Arkey is the program I built to run all the layers of the map from one menu. It is open source: arkey_v3. You pick a tool and you pick where the model runs. Arkey starts the engine, starts MoonBridge, connects them, and opens the tool.
YOU
|
v
+------------------------------------------------------------------------+
| ARKEY one menu: pick a tool, pick a route |
+------------------------------------------------------------------------+
| opens
v
+------------------------------------------------------------------------+
| TOOL Codex Kimi Code Crush |
+------------------------------------------------------------------------+
| sends a request in its own protocol
v
+------------------------------------------------------------------------+
| MOONBRIDGE translates the request for the route you picked |
+------------------------------------------------------------------------+
| | |
v v v
+--------------------+ +---------------------+ +----------------------+
| LOCAL | | SERVER | | FRONTIER |
| llama.cpp on this | | a llama.cpp server | | a company's model: |
| computer. Arkey | | on another computer | | DeepSeek, Codex, |
| starts it. | | you can reach. | | Claude. |
+--------------------+ +---------------------+ +----------------------+
This is the model-agnostic setup from the first guide, built. The tool stays the same. The route changes from one menu.
The three routes
| Route | Where the model runs | When to use it |
|---|---|---|
| Local | llama.cpp on this computer | The model fits your memory (sections 8 and 9). |
| Server | A llama.cpp server on another computer | You have a second computer with a bigger GPU, or a rented one. |
| Frontier | A company's servers | The job needs a model too large to run yourself. |
Install it
Arkey runs on Mac and Linux. It needs Git, Go 1.25 or newer, llama.cpp, and at least one terminal tool already installed from the first guide. Arkey does not include the tools. It makes its own copy of the ones you have and leaves your normal install untouched.
MAC
brew install git go llama.cpp
LINUX
Install Git and Go with your package manager, and llama.cpp as in section 10.
BOTH
git clone https://github.com/JulianAbeleda/arkey_v3.git
cd arkey_v3
scripts/install.sh
Open a new terminal window and start it:
arkey
The menu
TUI
Arkey Codex
Arkey Claude (not available yet)
Arkey Kimi
Arkey Crush
Config
Local
llama.cpp
your installed GGUF models
Enter loads, r refreshes, d unloads
Server
one row for each server you listed
Enter tests the server and makes it the route
Frontier
DeepSeek
Codex
Claude
GPU Auto-scan
Exit
A first run:
- Open Config, then Local, then llama.cpp. Your GGUF files from section 10 are listed.
- Press
Enteron one. Arkey checks your GPU, starts llama.cpp and loads the model. - Go back and open TUI. Press
Enteron Arkey Codex. - Codex opens. It is now running on your local model. Give it the request from the first guide:
Make a folder on my Desktop named hello world.
Everything is in the open
Arkey does not hide what it does. Each layer of the map is a file you can open and read.
| What | Where |
|---|---|
| Which model, engine, port and context size are selected | ~/.config/arkey/config.toml |
| MoonBridge's routes and translations | ~/.config/arkey/moonbridge.yml |
| Logs of what was started and what was sent | ~/.local/state/arkey |
| Arkey's copies of your tools | ~/.local/libexec/arkey/clients |
To add a server route, list it in config.toml. The address is the other computer's llama.cpp server:
[[servers]]
label = "My other computer"
origin = "http://192.168.1.50:8080"
What it does not do yet
- One local engine. The Local route runs llama.cpp. vLLM and TensorRT-LLM are not started by Arkey.
- Claude Code. It is in the menu and marked unavailable. The translator does not yet accept Claude Code's protocol.
- Windows. Not covered by this guide.
12. My setup [MYSU]
This is what I run every day. It has two computers. Every front in it is a TUI.
LAPTOP DESKTOP
MacBook Air, Apple M5, 16 GB Ubuntu Linux, with the GPUs
+----------------------------------+ +---------------------------+
| Terminal | | |
| | | | ENGINE |
| v | | serves one large model |
| ARKEY the menu | | on port 8080 |
| | | | |
| v | | MODEL FILE |
| TOOL Codex, Kimi Code, | | held in the GPUs' VRAM |
| | Crush | | |
| v | +---------------------------+
| MOONBRIDGE the translator | ^
| | | |
| +--- Server route -------------|--- private network-+
| | |
| +--- Local route ----> llama.cpp on the laptop
| | Qwen3-8B, GGUF
| |
| +--- Frontier route -> DeepSeek, Claude, OpenAI
+----------------------------------+
The parts
| Layer of the map | What I use |
|---|---|
| Front, GUI | None. |
| Front, TUI | Arkey's menu, started with arkey. Then Codex, Kimi Code and Crush. Claude Code is installed and waits on the translator. |
| Translator | MoonBridge, on the laptop. |
| Engine, laptop | llama.cpp from Homebrew. |
| Model, laptop | Qwen3-8B as one GGUF file. It fits in the laptop's 16 GB. |
| Engine and model, desktop | A server on the desktop holds a larger model in its GPUs. The laptop reaches it over a private network. |
| Frontier | DeepSeek, Claude and OpenAI, each through its own API key. |
How I use it
- Most days: the Server route. I work on the laptop. The model runs on the desktop. The laptop stays cool and its battery lasts, and the model is larger than the laptop could hold. This is sections 8 and 9 in practice: put the model where the memory is.
- Away from the desktop: the Local route. Qwen3-8B runs on the laptop with no internet. It is a smaller model, and it is always there.
- When the job is too big: the Frontier route. One menu choice sends the same tool to a company's model.
The tool does not change between the three. I change the route in Arkey's menu and keep working.
Why two computers
A laptop has little memory for a model. A desktop with GPUs has a lot, and it is loud and stays at home. A private network joins them, so the laptop uses the desktop's GPUs from anywhere. I use Tailscale for this. It gives each of your computers an address only you can reach.
On the desktop, the engine has to accept connections from other computers. For llama.cpp that is one flag:
llama-server -hf Qwen/Qwen3-14B-GGUF:Q4_K_M --host 0.0.0.0
Only do this on a private network. With that flag, any computer that can reach the desktop can use the model.
On the laptop, the desktop is one entry in Arkey's config.toml, as in section 11.
What next
The tools in this setup are harnesses. The next guide explains what a harness is and builds one: What a Harness Is, and How AI Gets to Use Your Computer.
13. When it goes wrong [HELP]
| What you see | Why | Fix |
|---|---|---|
out of memory, or the model will not load | The file is larger than your VRAM. | Use a smaller quant or a smaller model. For an MoE model, add --cpu-moe. |
| It loads, and each word takes seconds | Part of a dense model is in RAM. | Use a smaller quant so the whole file fits in VRAM. |
| It runs on the CPU, and the GPU is idle | The engine was installed without GPU support, or no layers were sent to the GPU. | In llama.cpp add --gpu-layers all. In a GUI front, raise the GPU layers setting. |
| The download stops partway | These files are large. | Run the same command again. It resumes. |
An MoE model is very slow with --cpu-moe | All the experts are in RAM. | Use --n-cpu-moe N with a lower N, so more experts go to VRAM. |
| The computer freezes while loading | RAM is full and the computer is using the disk as memory. | Close other programs, or use a smaller file. |
| vLLM will not install on a Mac or on Windows | vLLM is made for Linux. | Use llama.cpp. |
14. Word list [WORD]
ACTIVE WEIGHTS The weights an MoE model uses for one word.
FRONT What you type into: a GUI or a TUI. Does none of the math.
AWQ A method that shrinks a model to 4 bits. For vLLM.
DENSE A model that uses every weight for every word.
ENGINE The program that runs the model.
EXPERT One group of weights in an MoE model.
FORK A copy of an engine's code, with changes.
GUI A front that is a window or a web page.
GGUF The one-file model format llama.cpp reads.
HUGGING FACE The website where model files are published.
INFERENCE Running a model to get an answer.
KERNEL A small program that does the model's math on the chip.
MOONBRIDGE A translator between a tool's protocol and a model's.
MoE Mixture of experts. Only a few experts work on each word.
PROTOCOL The format a program sends its requests in.
QUANT A model stored with fewer bits for each weight.
RAM The computer's main memory. Slow for AI.
ROUTE Where the model runs: local, a server, or a company.
SAFETENSORS The file format makers publish models in.
TUI A front that runs inside the terminal.
VRAM The GPU's memory. Fast for AI.
WEIGHT One number the model learned. Also called a parameter.
15. Version history [VERS]
1.0 2026-11-12 First version.