Scaling

Model Inference and Scaling: Cutting GPU Costs with Latency, Throughput, Quantization and the KV-Cache

A practical guide to latency, throughput, quantization, KV-cache and batching, with everyday analogies, focused on lowering the cost of GPU inference.