Local models
Download open models from Hugging Face and run them on your GPU, on your PC or a server.
Settings → Local models downloads open models and runs them with llama.cpp, on your PC or on a server's GPU. They then appear under the Local models provider like any other model.
Install the runtime
Install llama.cpp gets the right build for the machine from GitHub: CUDA (matching your NVIDIA driver), Vulkan or CPU, on Windows or Linux. InkBot offers updates when a newer build is out.
Find a model
Search Hugging Face's GGUF models. Sort by best match, trending, downloads, likes or date, and filter by size (including fits my GPU) or by whether the model can see images.
Pick a quant
A model's quants are ranked best-first for this machine, each with an estimate of whether it fits on the GPU, read from the file's own header before downloading. Split files are grouped, and a vision model's projector downloads with it. Downloads resume if interrupted.
Chat with it
Pick it for a bot. The model loads on first use (a "Loading…" step in the chat) and unloads when idle or to make room for another.
Fitting it in memory
The memory planner counts the weights, the context cache and buffers. If the model doesn't fit on the GPU, mixture-of-experts layers move to system memory first, then whole layers. While you change a model's settings, a live preview shows where its memory goes.
Per model you can set the context length, GPU layers, cache type and experts on the CPU. Several GPUs of the same make are used together.
Draft models
Give a model a smaller draft model from the same family for speculative decoding: the draft guesses ahead and the big model checks, which can make replies much faster with the same output.
Other options
- A Hugging Face token for gated or private models.
- The models folder where downloads go.
- Models already downloaded by LM Studio are offered too.
- View log shows what llama-server printed for a loaded (or failed) model.