Time to First Token: LLM Inference Serving Roadmap
A practical guide to Time to First Token, a 10-week curriculum for building, measuring, tuning, and benchmarking an LLM inference service.
Time to First Token: LLM Inference Serving Roadmap
Key takeaways#
- Time to First Token is a 10-week learning roadmap for engineers who want to operate LLM inference services, not just read serving blog posts.
- The curriculum builds one OpenAI-compatible inference service across 50 short sessions.
- Topics include vLLM, SGLang, roofline analysis, Prometheus and Grafana, load testing, quantization, speculative decoding, KV-cache behavior, and cost-aware routing.
- The repository is best treated as a resource for inference engineers and MLOps teams rather than a standalone tool.
What it covers#
Time to First Token teaches LLM serving by growing one service over ten weeks. The README argues against disconnected experiments: each session feeds a single artifact that is deployed, instrumented, tested under load, tuned, routed, and written up as a reproducible benchmark. The end goal is an OpenAI-compatible inference service running on rented GPU infrastructure with dashboards and measurements that explain real production behavior.
Who should use it#
The roadmap is aimed at engineers who know Python, transformers at a high level, and the command line. It does not require prior Kubernetes, CUDA, or serving experience. That makes it useful for application engineers moving into AI infrastructure, MLOps engineers who need more model-serving depth, and platform teams that want shared vocabulary around latency, throughput, queue depth, and cost per request.
Structure#
The plan is 10 weeks, 5 sessions per week, about 30 minutes per session, plus buffer days. It starts with the roofline model because prefill and decode bottlenecks explain many later serving decisions. It then moves into vLLM deployment and internals, metrics with Prometheus and Grafana, SGLang and RadixAttention, load testing to 1000 concurrent requests, quantization, speculative decoding, cost routing, and final benchmark publication.
Practical outcomes#
By the end, learners should have a serving stack they configured and tuned, Grafana dashboards for TTFT and inter-token latency, a load-test harness, benchmark variants across FP16, FP8, INT4, speculative decoding, and KV eviction, and a cost-aware router with token budgeting. Those outputs make the resource more practical than a reading list. The repository also links to primary materials from vLLM, Modal, SGLang, Horace He, and related papers.
Why builders should care#
Latency and cost decide whether AI products feel usable. Time to first token affects perceived responsiveness, while inter-token latency, batching, cache pressure, and routing shape throughput and unit economics. This roadmap gives builders a path for measuring those factors instead of guessing. It pairs conceptual reading with build days, so teams can translate terms like paged KV cache, continuous batching, and speculative decoding into measurements they can see on their own service.
Limitations#
This is not a packaged inference platform. Learners still need GPU access through providers such as RunPod, Modal, Lambda, vast.ai, or Colab, and some sessions may require paid GPU time. The value is the curriculum, sequence, and benchmark discipline. If you only need to serve a model today, use vLLM, SGLang, a managed inference API, or a hosted platform. If you want to understand what happens after the API call enters the serving layer, this resource is a strong place to start.