Can Hermes switch to a local model automatically?

Yes, on outages, rate limits, authentication errors, and repeatedly empty responses. You set the local model up as a fallback provider. Hermes switches per turn and tries the main model again first with the next message. Hermes treats a refusal by a safety filter as a final result. In that case, you switch yourself with /model.

Is a local model as good as a cloud model?

Usually not. The strongest cloud models stay better for most tasks, and a local model is slower. A local model is mainly insurance: the work keeps going when the cloud model stops. Pick one that handles long sessions with many tool calls.

Do I lose my session when I switch models?

No. Hermes keeps the session history and the tools, and the new model continues where the previous one stopped. The new model does read the whole history again, because the prompt cache does not carry over. For a session of more than 100,000 tokens, Hermes asks for confirmation first.

What hardware do you need for a local backup model?

That depends on the model. We use a workstation with an RTX 5090 and 128 GB of system RAM for a large model with a long context window. A smaller model runs on less, but holds up less well in a long agent session. Our article on what computer you need to run AI locally lists the options per budget.

Can you use a local model to get around a safety filter?

No, not to get around the rules. A filter can also block legitimate work, such as maintenance on your own systems. A local model then lets that work continue. You stay responsible for what the agent does. The same agreements and human oversight still apply.

What does a local fallback model cost?

That depends on the hardware and the model, so you receive a custom quote. With a local model, you pay no provider for usage, but hardware and electricity weigh more. If you already have a suitable computer, a smaller backup model can be enough.

What if your cloud model stops? A local model as a fallback for your AI agent

An AI agent that works with a cloud model depends on a service you do not control. A safety filter, an outage, a rate limit, or a new rule can make the model stop in the middle of a task. A local model as a fallback keeps the work going: the same agent, the same tools, and the same session, only a different model. In AI terms, this is an LLM fallback: a second model your agent switches to when the first one does not answer.

This is not theory. On October 22, 2026, the cloud model in one of our own sessions stopped answering, and a local model took over the work. Below, you can read why this happens, why retrying often does not fix it, and how to set up a fallback that works when you need it.

An AI agent that keeps working through a local workstation while its line to the cloud is down

Why can a cloud model stop in the middle of a task?

A cloud model runs at a provider, and that provider has a say in whether your request gets an answer. There are four common reasons why it sometimes does not.

The first is a safety filter. Large models run behind classifiers that stop requests that could lead to misuse. Those filters are not perfect. According to Anthropic’s documentation on refusals, regular security work can trigger such a filter too. In our experience, such a filter does not only look at your last question. For an agent, what it just read counts as well: files, logs, and tool output.

The second and third reasons are more familiar: an outage at the provider, or a limit you hit. Think of a rate limit or the usage included in your subscription. The fourth is a policy change. A provider can retire a model or change its rules. The model then refuses requests today that worked yesterday.

Why does retrying often not help?

With a refusal, you send the same context to the same model again, and that context caused the refusal. So retry on a different model. With an outage or a limit, waiting usually does help.

Anthropic puts it plainly in the same documentation: “Re-sending a refused request to the same model usually earns another refusal.” The advice is to point the retry at a different model.

A refusal also does not show up in your monitoring. Technically, it is a successful response, an HTTP 200 without content, not an error. If you only watch for errors, you see nothing. With Anthropic, you recognize a refusal by the stop_reason field with the value refusal; you can log that field separately. Sometimes the agent itself does not know what happened either. That is exactly what we ran into.

Why the model stops	What you see	Does retrying help?	What works instead
Safety filter	An empty answer or a short standard text, no error	Usually not: the same context gets the same refusal	A different model, such as a local model
Provider outage	Server errors or timeouts	Sometimes, after a while	Switching to another model automatically
Limit reached	A message that you send too many requests or ran out of credit	Only after the reset time	Switching automatically, or waiting
Policy change	Refusals or a model that no longer exists	No	Choosing another model or provider

What happened in our own session?

On October 22, 2026, the cloud model refused every message in the middle of a long maintenance task, because a safety filter fired. We switched to a local model in the same session, and it finished the work.

We were working with our own Hermes installation at Voltti on technical maintenance on our own servers. Halfway through, every message got the same short reply: the request had not been processed. Retrying and rephrasing did not help. The agent itself suspected a context length problem or an outage. The Hermes log showed the real reason: the filter reacted to the technical content the agent had just read. Our own question was simply “continue.”

We then started the local model on our workstation: Qwen3.8-Flash-Next on Strata. In the same session, we switched models with the /model command. Hermes kept the full session history, the tools, and the agreements made earlier in the session. The local model finished the work in 66 model calls, with a context that grew from about 121,000 to 212,000 tokens.

It was slower than the cloud. With a warm cache, a turn took 3 to 20 seconds; without the cache, 70 to 115 seconds. And it was not flawless: a script the model wrote had two bugs, which it found itself in a local test. One count in the final summary did not match an earlier report. You check things like that with any model.

How does a local model take over a running session?

In Hermes, a local model is simply another provider with an OpenAI-compatible endpoint. You can switch in two ways: yourself with /model, or automatically with fallback_providers.

With /model, you change models yourself inside a running session; the documentation on models explains how that works. With fallback_providers, you set a backup model (a fallback) that Hermes switches to automatically. The setting is a list, so you can add several backup models that Hermes tries in order. A generic example:

fallback_providers:
  - provider: custom
    model: my-local-model
    base_url: http://localhost:8000/v1
    key_env: LOCAL_API_KEY

According to the documentation on fallback providers, Hermes switches automatically on rate limits and server errors, after its own retries. It also switches on authentication errors and on responses that are repeatedly empty or unusable. The fallback applies per turn. The documentation puts it this way: “Fallback is turn-scoped: each new user message starts with the primary model restored.” So with the next message, Hermes tries the main model again first. A refusal by a safety filter, however, is a final result: Hermes shows it and does not retry. After a refusal, you switch manually with /model.

Expect two things when you switch. The previous model’s prompt cache does not carry over, so the new model reads the whole history again. For a session of more than 100,000 tokens, Hermes therefore asks for confirmation first. If the new model has a smaller context window, the session may already be close to its compression threshold. Hermes then compresses the session with the next message.

What can you expect from a local model as a backup?

A local model is insurance, not a replacement for a strong cloud model. It is worth something if it is fast enough, has a long context window, and calls tools reliably. A session with 200,000 tokens and dozens of commands is very different from a chat question.

We still use the cloud model for most of our work. Our workstation has an RTX 5090 with 32 GB of VRAM and 128 GB of system RAM. Qwen3.8-Flash-Next ran at about 79 tokens per second on it, with a context window of 512,000 tokens. In our own task set, on October 21 and 22, 2026, it was the fastest and most reliable model on that hardware. These are measurements on one machine, not a general benchmark; the full comparison is online.

With a smaller computer, you choose a smaller model and expect less. Our article on what computer you need to run AI locally explains which hardware fits which model.

What should you watch for when you set up a fallback?

A local model as a fallback only works if you prepare it before you need it. These are the points that made the difference for us:

Use one agent framework for both routes. The handover worked because the session and the tools stayed the same; only the model changed.
Keep the local model ready. Know the start script, know what else runs on the graphics card, and test the startup now and then.
Pick a model that handles long sessions with many tool calls, not just one that scores well on chat questions.
Log refusals separately from errors. A refusal is a successful response without content and stays invisible otherwise.
Set up automatic fallback for outages and limits, and agree on who switches manually after a refusal.
Check the work the same way, whichever model does it: backups before, checks after, and a way back if something goes wrong.

A fallback is not a way around the rules. You use it for work that falls within your own agreements, with the same human oversight as usual. You check what the local model does just as carefully as the work of the cloud model.

How does Voltti set this up?

Voltti installs Hermes Agent on your side, sets up the fallback route, and provides support for businesses in Belgium and the Netherlands. We add a local model when the use case and the hardware justify it.

A hybrid setup is the starting point: Hermes runs on your side, and you pick the model that fits each task. We cover how to make that choice in local AI vs. cloud AI. The local model then serves as a backup or handles tasks where data has to stay on your own hardware. For what stays on your side and what does not, see what stays local in a hybrid AI setup.

You receive a custom quote, because hardware and model choice differ for every business. Our page on local AI for businesses covers the options. If you want to know what happens when the hardware itself fails, read what happens to your AI environment if hardware fails.