Systems ObservatoryProject evidence passportView ML Infrastructure track

PythonMLI

vllm-lite

Paged KV cache and continuous batching around a small model.

90 tests; paged scheduling reaches 2.9x sequential throughput and 5.5x better median TTFT.

Inspect source
vllm-lite project overview
Verified result94%KV slot utilization vs 21%

Evidence chain

The claim, with its attack surface exposed.

Recruiters can scan the result. Engineers can inspect how it was produced and where it stops being valid.

  1. Problem

    Static batching strands KV memory and creates head-of-line blocking under variable request lengths.

  2. Mechanism

    Paged cache, continuous batching, chunked prefill, prefix caching, and speculative decoding.

  3. Attack

    Same weights, sampling, traffic distribution, and memory budget across sequential, static, and paged schedulers.

  4. Boundary

    CPU-scale simulator with a small model; it does not claim production GPU kernel performance.

Reproduce it

One command to the test boundary.

The repository contains the implementation, tests, benchmark harness, and documented limitations behind this page.

git clone https://github.com/asp53826/vllm-lite && cd vllm-lite && pip install -r requirements.txt && pytest
Verified withscheduler benchmarkmemory accountingtoken invariance