What is an LLM, and which language model powers your AI agent?

An LLM, short for large language model, is an AI model trained on huge amounts of text, which lets it write, summarize, translate, and assess text on its own. ChatGPT, Claude, and Gemini all run on one. An AI agent needs one too: the language model does the thinking, and the agent does the work.

So what is an LLM exactly, how does it work, and where are its limits? That is what you will find below, without the jargon. After that, we get to the question that matters most for a business: which model do you put under your AI agent, and do you choose an API, a subscription, or a local model?

Small cobalt-blue blocks flowing in a line toward a large dark sculpture, with one loose block beside it, on a matte black background
A language model reads text as a sequence of small pieces and keeps predicting the next one.

How does a large language model work?

A language model does not read text as words but as tokens: small pieces of text. A token can be a short word, part of a longer word, or a punctuation mark. During training, the model saw enormous amounts of text and learned, again and again, to predict which token is likely to come next. That sounds simple. But a model that does it well enough can also answer an email, summarize a contract, or write code.

A second step follows that first training. The model learns to follow instructions and to answer in a conversation. That is why you can simply ask it something instead of having it complete a text. After that, the model itself no longer changes because of what you type. It does not learn from your conversations, unless the provider later uses them to train a new version. Whether that happens is in the provider’s terms.

What is a context window?

The context window is the amount of text the model can look at in one go: your question, the earlier messages, the documents you pass in, and its own answer. Anything outside it, the model does not see. That is why a long conversation sometimes seems to “forget” what was said at the very start.

For an AI agent, that window matters. An agent reads more than your question: it also reads files, search results, and tool output. According to the Hermes documentation on local models with Ollama, Hermes needs at least 64,000 tokens to work with tools. A model with a smaller window can still chat, but it quickly gets stuck on real tasks.

Why does an LLM sometimes make things up?

A language model picks the most likely continuation. Usually that is also the right answer, but not always. When the model does not know something, it can still give a convincing answer. That is called a hallucination. In a publication from September 2025, OpenAI describes hallucinations as “plausible but false statements generated by language models.” According to the same publication, this happens partly because training and evaluation reward models for guessing instead of admitting uncertainty.

For a business, that means: do not let a model recite facts from memory when it can look them up. An agent that first checks your own documents, your CRM, or a source, and names that source, makes fewer mistakes. And a person keeps checking what goes out the door, such as a quote, a price, or a legal statement.

What is the difference between an LLM and an AI agent?

An LLM produces text. It cannot send an email, open a file, or look up a customer by itself. An AI agent adds tools, memory, and tasks. The language model decides which step makes sense, the agent carries out that step and hands the result back to the model. That is how Hermes, the open-source agent Voltti works with, operates: Hermes handles the tools, the memory, and the rules, and you choose the language model.

That distinction is practical. You can swap the model without rebuilding the agent. Your skills, connections, and memory stay in place; only the thinking comes from a different model. There are three ways to connect such a model to your agent.

RouteHow it worksWhat you payWhere your text goesGood fit for

API from a model provider

Your agent sends each request to the provider with an API key

Per use, per token

To the provider, under its terms

Variable usage and the strongest models

Existing subscription

You sign in with a subscription you already have, if the provider allows it

Your subscription, sometimes with extra usage credits

To the provider, under its terms

Teams that already have a subscription

Local model

The model runs on a computer in your own network

The hardware and the electricity

Stays on your own hardware

Sensitive data and a fallback option

How do you choose a model for your AI agent?

There is no single best model for every business. Which route fits depends on your tasks, your data, and your budget. With an API, you pay per use and get direct access to the strongest models. With a subscription, you use what you already pay for, but what that subscription actually covers differs per provider. The Hermes documentation on model providers lists, per provider, which subscriptions work and what they cover. A local model runs on your own hardware, so your text does not leave your network. Hermes can manage such a model for you; the page on local models explains how.

With every route, check three things. The model must be able to call tools, because a model without tool calling can only chat. The context window must be large enough for your tasks. And you need to know what the provider does with your data. You can find open-weight models to run locally on Hugging Face, for example.

You also do not have to pick just one route. In our own Hermes setup at Voltti, we use cloud models and local models side by side, and switch per task with the /model command. When a cloud model stopped in the middle of a task, a local model took over the work. You can read how that works in a local model as a fallback for your AI agent. If you are torn between the cloud and your own hardware, local AI vs cloud AI: which fits your business helps with that choice. For the hardware itself, see our guide on which computer you need to run AI locally.

What does this mean for your business?

A language model is only one part of the picture. What it delivers for your business depends on the agent around it: which tools it may use, which data it may see, and who checks the result. Voltti sets up Hermes for businesses in Belgium and the Netherlands and leaves the choice of model to you: an API, a subscription, or a local model. If you want your data to stay in your own network, take a look at local AI for businesses.

Portrait of Seppe Gadeyne

Seppe Gadeyne is a developer and the owner of Voltti. He sets up AI agents for companies in Belgium and the Netherlands.

Updated on

Frequently asked questions about LLMs

01Is ChatGPT an LLM?

Not quite. ChatGPT is a chatbot: an app in which you talk to a language model. The language model itself is the LLM. ChatGPT uses OpenAI’s models, and Claude and Gemini are the chatbots for the models from Anthropic and Google.

02Can an LLM run locally?

Yes. You can download open-weight models and run them on your own computer, so your text does not leave your network. For agent work, you need a model that can call tools and has a large enough context window, which also means enough memory in your computer.

03Why does an LLM sometimes make things up?

A language model picks the most likely continuation, even when it does not really know the answer. That produces a convincing but wrong answer: a hallucination. So let an agent look up and name its sources, and have a person check what goes out the door.

04Does an LLM learn from my business data?

The model does not change because of what you type in a conversation. A provider may use conversations to train a future version, depending on its terms and your settings. So read those terms, or choose a local model for sensitive data.

05Which LLM is best for an AI agent?

That depends on your tasks. Check whether the model calls tools reliably, how large its context window is, what it costs, and where your data goes. You can also combine a strong cloud model with a local model as a fallback.

Which model fits your AI agent?

Discuss your pilot