lithos-metal

lithos-metal is an open-source LLM inference engine for Apple silicon. It focuses on generating high-performance Metal megakernels and serving local models through APIs that common agent clients already understand. The project is aimed at developers who want low-latency local generation on Macs instead of routing every coding-agent request to a remote model provider. The repository describes three technical pillars: layer-wise mixer megakernels, GPU-resident generation control, and DSpark speculative decoding. In practice, lithos-metal serves a target model, pairs it with a matching draft head when available, and verifies multiple proposed tokens in a target pass. The default quick start uses Homebrew, then starts a local server with a Qwen target model. During startup the server downloads model assets, prepares cached weight packs, selects device-specific kernel recipes, compiles kernels, and warms them up. lithos-metal exposes the model through familiar interfaces. The README documents OpenAI-compatible chat completions, Responses API support, Anthropic Messages compatibility, SSE streaming, tool calls handled by the client, and Ollama-style /api/chat support. It also includes launchers for opencode, Claude Code, Codex, and Hermes so a developer can start the server and then run a client with temporary environment configuration rather than rewriting global settings. The project is tuned for modern Apple silicon. The README notes macOS 26+, a registered chip backend, and enough unified memory for the target model, draft model, and context. It lists a 32K default context capacity and describes model lifecycle options, busy behavior, one-generation-at-a-time scheduling, and request fields that are accepted or rejected by the Ollama adapter. That level of detail is useful for builders because it sets expectations before they wire the server into a coding workflow. lithos-metal is best for Mac-based AI developers, local inference tinkerers, and teams exploring private coding-agent setups on workstation hardware. It is not a general hosted API or a simple consumer chat app. The project is still early, with a focused model matrix and hardware requirements, but it is a strong fit for OpenTools because it gives builders a concrete path to run agent-friendly model serving on Apple silicon using open-source infrastructure. It is especially relevant for developers testing whether a local workstation can support day-to-day coding-agent loops without depending on a cloud endpoint for every generation. The local server shape also makes it easier to test clients that expect OpenAI, Anthropic, or Ollama-style APIs. Teams should still benchmark their target model and memory budget before adopting it, but the project gives a practical, documented path for experimentation.