Signal: Claude 3.7 Sonnet Introduces Dynamic Reasoning Tokens
Anthropic unifies standard fast autoregressive inference and extended test-time thinking into a single dynamically allocated token budget.
Curated Topic Index
Inference & LLMs
High-throughput LLM serving, quantization, and memory-bandwidth optimizations.
Brief: Multi-Head Latent Attention and Fine-Grained MoE Routing
How decoupled key-value compression and fine-grained expert routing achieved unprecedented inference throughput without performance degradation.
Deep Dive: Speculative Decoding in High-Concurrency Serving
A comprehensive systems analysis of how draft-verification loops double token generation throughput while maintaining identical output distributions.