We are a group for people interested in running, evaluating, and building with AI locally. We are not interested in how GPU Rich you are -- we are a community focused on digging into relevant research, tools, and practical techniques for hosting models on consumer and prosumer hardware. We care about comparing real-world performance across different systems and models and sharing local agentic workflows we actually use day to day -- in as few FLOPs as you can muster.

Topics will range from inference stacks like vLLM, SGLang, and llama.cpp to hardware tradeoffs, model quality, optimization, benchmarking, and emerging local-first AI tooling. The scope is intentionally broad: if it helps us better understand, run, or build useful AI systems without relying entirely on the cloud, it’s fair game.

Join us and free the FLOPs. 

Cohere Labs Discord Channel: #local-ai

Roadmap & pre-requisites (or lack thereof)

1. There really are no pre-requisites. From a hardware perspective, getting involved with "local AI" can be as cheap or as expensive as you want it to be.

   • that being said, topics that will provide helpful context for this group are transformers, kv-caches, speculative decoding (DFlash, MTP, Eagle, etc)

2. Topics that will be covered and recurring in some meaningful sense include:

   • Inference engines (e.g. vLLM, llama.cpp, sglang)

   • Evaluations (general benchmarking, agentic evaluations (generating your own agent traces), and performance testing -- especially as it pertains to quantization tradeoffs for local serving)

The ethos of local-ai as a group, ironically is not even local. Your reasons to try things with local AI models may differ -- utility, curiosity, industry requirements, etc, but philosophically, the goal is climbing as far down and owning as much of the [agentic] AI Stack as possible.

For example, most people sit at the top -- they maybe own their skills for claude code, custom tools, plugins, hooks etc. The spirit of this group asks "how can we own the next level down of the stack to increase my ownership". Hence, the next step down the ladder might be to use an open source agent harness pointing to an open source model you deploy to your private cloud with zero data retention agreements. But maybe that is not enough for you -- you, dear reader, need to push the limits of what can be accomplished on local hardware. So you buy a GPU and host an LLM with llama.cpp. Another step down.

Each step down the ladder brings in different skills that build upon each other in unique ways. If you're the kind of person who asks how they can climb down that ladder, you're in the right place. 

Or if you just think local AI models are cool and powerful and want to build ridiculous home automations, that's also a perfectly good reason to be here :D