Many people have asked me about my local agent stack and how I set it up.[reference:106] So I thought it might be useful to put together a little tutorial on how to set up a local coding agent using open-source tools and open-weight LLMs.[reference:107]
Why Go Local?
For many coding workflows, a local setup is an interesting alternative to proprietary services such as GPT in Codex or Opus in Claude Code.[reference:108] The local setup is transparent, inspectable, and free to run apart from hardware and electricity costs. It also stays fully under your control, and you can modify the coding harness in any way you like.[reference:109]
Your motivations for using local LLMs and coding harnesses may include:
- Predictable, fixed costs if you reach your subscription plan limits, and immunity to API price changes.[reference:110]
- Reproducibility; sometimes it's nice if a model is upgraded and it solves all your queries more reliably, though this can also break existing workflows.[reference:111]
- Offline use in the classic airplane flight scenario with slow or no internet.[reference:112]
- Privacy: for organizing and processing receipts, I'd be more comfortable with a local model ingesting them rather than sending data over to OpenAI or Anthropic.[reference:113]
The Architecture
We will use a locally served LLM together with a local coding harness that can read files, make edits, run commands, and verify changes.[reference:114] The LLM serves as the engine that provides the reasoning and code generation, while the surrounding harness provides the operating environment that allows the LLM to do meaningful coding work in local projects.[reference:115]
Assessment Notes
When selecting models for a local coding agent, consider these factors:
- Checking RAM usage at long contexts to see if the model is suitable for real work[reference:116]
- Measuring prefill and decoding tok/sec to see whether it's fast enough to not be annoying[reference:117]
- Making sure the model has sufficient tool-calling capabilities in theory[reference:118]
- Doing a security audit of the agent framework[reference:119]
- Assessing whether the model can solve some more challenging tasks when used in a coding harness[reference:120]
Getting Started
Raschka recommends Ollama for its cross-platform support (macOS, Linux, Windows).[reference:121] The full tutorial covers setting up the environment, connecting the LLM to the harness, and running practical coding tasks.
The Bottom Line
Local solutions become more and more attractive each day.[reference:122] While I still primarily alternate between Codex and Claude Code as my daily drivers, local solutions are a viable and increasingly capable alternative.[reference:123] If you have the hardware, they are practically free to run.[reference:124]
Given that proprietary services may become more restrictive over time, it's a good idea to be comfortable with open-weight alternatives as a backup.[reference:125] This tutorial provides a solid foundation for building a fully local coding agent that can handle real work.