The rankings tell you which model to run. This page tracks the projects that make running it
possible: the people shrinking models small enough to fit your machine, and the engines that
serve them fast. We watch their GitHub so you can check one page instead of eight.
Updated July 27, 2026 · Live stars and releases from GitHub · 8 projects tracked
Latest releases
as of July 27, 2026
llama.cppb10152
fit : count nextn (MTP) blocks in ngpulayers so front layers stay on GPU (26177)
We've been hard at work doing low level improvements in the kernels. Over 90 commits since v0.17.0 (June 3, 2026), themed around fine-tuning very large spars...
The projects that let you bend an open model to your own data, on hardware you actually own.
Unsloth
Single-GPU fine-tuning and dynamic quantization
69.0k stars
Quantization Single-GPU QLoRA
Hand-written GPU kernels that make fine-tuning a model on one consumer GPU roughly twice as fast while using far less VRAM, with no measurable quality hit. It is also the source of the dynamic GGUF quants that shrink giant mixture-of-experts models enough to fit a single big-RAM machine.
Follow if you fine-tune or run large models on one GPU or a Mac. This is the project that makes "it fits" possible.
A wrapper that lets you fine-tune across several GPUs from a single config file, on top of Hugging Face, DeepSpeed and FSDP. It handles the distributed-training plumbing so you do not hand-wire it yourself.
Follow if you have more than one GPU and want repeatable training runs without writing framework glue.
A community hub that supports more than 100 model families and gives you a no-code web UI for fine-tuning. It can hook into Unsloth's fast kernels under the hood.
Follow if you want to prototype fine-tunes by clicking rather than scripting, across a lot of model families.
Hugging Face's library for the post-training step that teaches a model to reason and follow preferences, with open implementations of DPO, GRPO and PPO. This is how open models learn to think before they answer.
Follow if you care how reasoning models are actually built, or want to align a model to your own preferences.
The engines that load the weights and answer requests, from a single Mac to a production cluster.
llama.cpp
Run models on CPU, Mac and edge devices
121.8k stars
Apple Silicon GGUF CPU
The C/C++ engine behind most consumer local AI. It runs large models on Apple Silicon, plain CPUs and modest GPUs through the GGUF quantized format. Ollama and LM Studio are both built on top of it.
Follow if you run models on a Mac or a machine without a big GPU. This is the foundation everything else sits on.
The standard engine for serving open models at volume. Its PagedAttention method squeezes far more throughput out of GPU memory, which is why most serious local deployments run on it.
Follow if you are hosting a model for a team or an app and need an OpenAI-compatible API that holds up under load.
A newer serving engine built for heavy concurrency and multi-step agent loops. Its RadixAttention automatically caches shared prompt prefixes, which cuts cost and latency when the same context gets reused.
Follow if you run agents or high-volume structured-output jobs locally and care about latency.
The size of a quant file is, in practice, the RAM you need to run it. These are the top models
from our rankings,
plus popular smaller ones, with the real GGUF file size of every quant. Pick your hardware and
see exactly which version of each model fits. Leave a few GB of headroom for context.
Cyan = the everyday sweet spot (Q4). "UD" = an Unsloth dynamic quant, which holds quality better at
a smaller size. Sizes refreshed July 27, 2026 from real GGUF files.
This is a highlight reel: the full directory has quant sizes for every ranked open model.
Which one is actually best?
You can see what fits your RAM above. For how these models rank on quality, by real
human-preference voting, plus separate coding and vision scores, see the live rankings.
What it tracks: the open-source projects that make local AI practical, split into tools that fine-tune models and engines that serve them.
Live signal: stars, last activity, and the latest release for each project come straight from the GitHub API, refreshed once a day. A release shows "n/a" only when a project has not cut a tagged release.
Quant sizes are real: the "shop by RAM" table reads the actual GGUF file size of each quant from Hugging Face, not an estimate, so the RAM numbers match what you will actually download.
What we do not do: we do not scrape rumor or hot takes. The signal here is shipped code, real releases, and real file sizes, which is the part that actually matters.
Which model to pick is ranked on the rankings page, by quality and what fits your rig.
Local AI toolkit FAQ
What are the best tools to run open-source LLMs locally? +
For running a model, llama.cpp is the foundation for Macs and CPUs, while vLLM and SGLang serve models at scale on GPUs. For shrinking and fine-tuning models, Unsloth leads on a single GPU. This page tracks all of them with their latest releases.
What is the difference between fine-tuning and serving an LLM? +
Fine-tuning changes a model by training it further on your own data (tools like Unsloth, Axolotl, TorchTune). Serving loads a finished model and answers requests fast (tools like llama.cpp, vLLM, SGLang). Most local setups use one of each.
Which tool should I use to run a local model on a Mac? +
llama.cpp, usually through Ollama or LM Studio, which are built on top of it. It runs quantized GGUF models on Apple Silicon with no GPU required.
How much RAM do I need to run a local model, and which quant should I pick? +
A simple rule: the quant file size is roughly the RAM you need, plus a few GB of headroom for context. Q4 (Q4_K_M) is the everyday sweet spot, good quality at about a quarter the full size. If a model is just over your RAM, an Unsloth dynamic (UD) quant often squeezes it in. The table on this page lists the real size of each quant for popular models so you can match one to your hardware.
How often is this toolkit list updated? +
Daily. The stars, last activity, latest release, and quant file sizes are pulled straight from GitHub and Hugging Face once a day. Last refreshed July 27, 2026.
Want this running inside your business?
BlueFort AI is the content side of BlueFort IT. When you want local, private AI set up and managed
for real, openly or on your own hardware, that's their day job.
