Running local models is good now
vickiboykis.com
June 16, 2026
6 min read
🔥🔥🔥🔥🔥
67/100
Summary
Local AI models have significantly improved in performance and usability. Various models such as Mistral 7B, Gemma 3, OpenAI OSS-20B, and Qwen 3 MOE have been successfully run on a 2022 M2 Mac with 64 GB RAM and 1TB storage using different setups including llama.cpp, llama-cpp-python, and LM Studio.
Key Takeaways
- Local models have significantly improved in performance and usability, allowing for agentic coding with around 75% accuracy and speed compared to frontier models.
- The author primarily uses the Gemma-4-26b-a4b model in LM Studio for tasks such as refactoring Python scripts, proofreading, and writing unit tests.
- Recent advancements in local models, such as GPT-OSS and Gemma 4, have made previously impossible tasks feasible within the last six months.
- The setup for running local agentic models involves using an inference engine and an agent harness, with the author currently utilizing Pi and LM Studio.
Community Sentiment
Positives
- Qwen3.6-27B is proving to be a highly capable local model for coding tasks, demonstrating its utility in everyday applications without requiring cloud inference.
- The ability to run local models like Qwen3.6-27B offers users more control and potentially lowers long-term costs compared to subscription-based cloud services.
- Gemma 4 excels in pipeline automation tasks, outperforming larger models like Qwen, which highlights the importance of task-specific optimization in AI applications.
- Local models are expected to improve significantly, which could lead to a more competitive landscape against hosted models, benefiting users seeking cost-effective solutions.
Concerns
- Many users find that larger models, like Claude Sonnet 4.6, can feel inferior in conversational quality, indicating that size does not always equate to better performance.
- Running local models often requires substantial hardware investments, making them inaccessible for many users and limiting their practical use.
- The performance of local models can be inconsistent, with issues like slow inference and incorrect outputs, which can hinder productivity in complex tasks.
- Concerns about the increasing costs of running local models and the potential monopolization of resources by larger companies raise ethical questions about accessibility.
