What is a local LLM?

The three parts of a local setup

A large language model (LLM) learns patterns in text during training. When you ask a chat model a question, it uses those patterns to choose tokens that form a response. Running that calculation on your own device is called local inference.

Model

The learned weights, plus the configuration and tokenizer needed to use them. Think of the model file as something you download once and use repeatedly.

Runtime

Software that loads those weights into memory and runs the calculations on your CPU, GPU, or both. Ollama and llama.cpp are examples of software you can use for local inference.

Interface

The place you send the prompt and read the reply. It could be a terminal, a desktop app such as LM Studio, or an app you write yourself.

Follow a prompt

Say you type “Suggest two titles for my houseplant journal.” The interface passes the request to the runtime. The runtime converts it into tokens, runs the model, and converts the generated tokens back into text.

1. Your prompt
2. Local runtime
3. Model weights
4. Generated reply

With a downloaded model and a local runtime, the main calculation can happen without an internet connection. Installing software or fetching weights usually uses the network. Cloud model selections, web search, remote plug-ins, and sync can also send data out of the device.

Try it

Take an everyday task, such as rewriting a meeting reminder. Write down what data goes in and what comes out. Once you finish an installation lesson, send a sample reminder, disconnect from the internet, and send a second sample. A working reply is a useful check that this configuration can perform inference offline.

Check your understanding

Which part changes when you switch from a desktop chat window to a Python script that calls the same local model?

The interface changes. The runtime and weights can stay the same.

Sources

• Hugging Face LLM Course · How do Transformers work?
• LM Studio · Offline operation