Running Local AI Agents with Ollama, OpenCode & Hermes
Running AI agents locally means executing both model inference and the agent runtime on your own machine instead of relying on external APIs. If you want a breakdown of model setup, quantization, and GPU constraints, see Running AI Models Locally. The system is typically split into two parts: a local inference server and an orchestration layer that handles tools, reasoning loops, and task execution.
In most setups, inference is handled by Ollama, while orchestration is handled by OpenCode. For GUI-based workflows, LM Studio exposes an OpenAI-compatible API. For stricter and more structured agent behavior, Hermes is used as an agent framework pattern built around controlled tool execution and multi-step reasoning.
Why Run AI Agents Locally
Local AI agent systems are used because they remove external dependencies while increasing control over execution.
- Data stays fully local (privacy by design)
- No API costs or rate limits
- Predictable latency without network dependency
- Works offline after model download
- Full control over tools, prompts, and execution flow
This makes local agents suitable for coding assistants, automation pipelines, and experimental AI systems.
What Quantization Means
Quantization is the process of reducing a model’s numerical precision (for example FP16 to INT4 or INT8) to reduce memory usage and improve inference speed, usually with a small trade-off in output quality.
Modern local inference engines like Ollama rely heavily on quantized models because full precision models are too large for consumer hardware.
Lower quantization reduces VRAM requirements but slightly reduces reasoning quality. Higher quantization improves quality but increases memory usage.
Local Model VRAM Guide
| Model Size | Quantization | VRAM | Use Case |
|---|---|---|---|
| 3B–4B | Q4 | 3–5 GB | lightweight automation, CLI tools |
| 7B | Q4 | 6–8 GB | default agent setup |
| 7B | Q5 | 8–10 GB | improved reasoning and tool use |
| 13B | Q4 | 10–12 GB | multi-step workflows |
| 13B | Q5 | 12–16 GB | stronger coding agents |
| 34B | Q4 | 20–24 GB | advanced reasoning setups |
| 70B | Q4 | 40 GB+ | research-grade systems |
These values assume GPU-first inference using runtimes such as Ollama or LM Studio.
How to Run AI Agents Locally (Ollama + OpenCode)
The most practical setup today uses Ollama as the runtime and OpenCode as the agent layer.
One-command setup (recommended)
1ollama launch opencode --model qwen3.5
This command:
- starts or connects to Ollama automatically
- configures OpenCode integration
- connects the agent layer to the selected model
- removes manual environment configuration
Interactive setup:
1ollama launch
OpenCode integration is officially supported by Ollama: https://docs.ollama.com/integrations/opencode
Hermes agent setup
1ollama launch hermes --model qwen3.5
Hermes integration is designed for structured agent workflows: https://docs.ollama.com/integrations/hermes
This setup is typically used when you need:
- strict tool calling formats
- deterministic multi-step reasoning
- consistent structured outputs
It is more suitable for autonomous systems than casual chat use.
Model selection for OpenCode workflows
Model choice is critical for agent performance.
- qwen3.5: balanced general-purpose agent behavior
- qwen2.5-coder: stronger tool execution and code consistency
- deepseek-coder: higher quality code generation and debugging
These models integrate directly through Ollama without additional configuration.
Ollama runtime: https://ollama.com
OpenCode agent system: https://opencode.ai
LM Studio in the same ecosystem
LM Studio provides a GUI-first alternative to Ollama:
It exposes an OpenAI-compatible API at:
This makes it interchangeable with Ollama in most agent frameworks while offering a visual model management workflow.
Hermes-style agent systems
Hermes represents a structured agent design approach focused on reliability over flexibility.
Instead of free-form chat behavior, it enforces:
- structured tool calls (often JSON-based)
- deterministic multi-step reasoning
- controlled execution loops
This makes it useful for autonomous coding agents and workflow automation systems where repeatability matters.
Recommended Local Agent Stack
A practical setup typically consists of:
- Runtime: Ollama or LM Studio
- Model layer: Llama, Qwen, DeepSeek, Hermes-style models
- Agent layer: OpenCode or Hermes-based frameworks
- Tools: filesystem, shell, APIs
- Interface: CLI or IDE integration
In this architecture, Ollama provides inference, OpenCode handles orchestration, and the model selection defines capability ceilings.
When Local Agents Make Sense
Local agent systems are appropriate for:
- coding automation and developer tooling
- offline-first AI workflows
- privacy-sensitive environments
- experimental agent architectures
They are less suitable when:
- very large frontier models are required
- multimodal workloads dominate
- hardware is insufficient for even quantized models
Limitations
- VRAM is the primary constraint
- RAM offloading works but significantly slows inference
- smaller models reduce reasoning depth
- agent reliability depends heavily on model choice
- setup complexity is higher than cloud-based APIs
Summary
Local AI agents are practical today using Ollama, OpenCode, LM Studio, and Hermes. With quantized models and correct hardware-aware selection, full agent systems can run locally without external dependencies, while still scaling based on VRAM and compute constraints.