Instrument model boundaries
Hooks synchronize the accelerator at layer entry and exit, recording start time, duration, shapes, phase, and device without changing model code.
Open-source LLM observability
Every layer. Every token. Every millisecond.
LayerLens reveals exactly when each transformer layer executes—from prompt prefill through every KV-cache decode step—so inference bottlenecks become visible, measurable, and comparable.
VALIDATED · vLLM + OLLAMA + HF
Real runs across three inference stacks. Compare model-level throughput, engine telemetry, and exact layer timing wherever the runtime exposes it.
Loading instrumented layer events…
Lab · exact products · no new quantization
Share the multiplication, not the KV cache.
Gate and up projections consume the same activation. Product Atlas computes each bit-identical x[j] × weight value once, then routes it to every matching output in both matrices. The operation count fell 76.97% on a real Qwen3.5 BF16 MLP pair.
7.34M → 1.69M multipliesExact stored weight bits0 added quantizationReference only
Qwen3.5 0.8B · layer 0 MLPMEASURED
Honest gate: the inspectable CPU prototype is 9.54× slower than optimized dense matvec. A fused kernel must win on wall time and memory before this becomes an inference feature.
Bottom-up profiler · scalar → head → layer
Repeated values are not always reusable work.
The new profiler walks from exact BF16 bit patterns through sigmoid and SiLU, activation coordinates and tiles, sibling matrix products, attention heads, blocks, and the layer stack. It keeps candidates only when both the dependency boundary and measured wall time make sense.
Sigmoid / SiLU domainMatrix productsActivation tilesAttention headsLayer states
Exact reuse observed37 VECTORS · QWEN3.5
Unary lookups are bit-exact but 2.35–2.45× slower on CPU. Contextual 4-value tiles and attention-head vectors had 0% exact reuse. Product fusion remains the strongest kernel candidate.
Raw experiment · memory + SQLite · bit exact
Cache only where the math actually repeats.
LayerLens captured native BF16 inputs and outputs from Qwen3.5, then replayed each projection with no cache, an exact memory cache, and a persistent SQLite cache. The early projection saved work; the contextual projection did not. Admission is decided per module from measured wall time.
7 / 37 exact hits44.04M multiplies skipped12.88% query-path saving0-bit output change
Latency vs native dense · lower is betterMEDIAN × 7
Decision: admit layer-0 QKV only after exact reuse is observed; reject layer-3 MLP at 0% hits. Setup makes both first short passes slower. Reopened SQLite is page-cache warm, not cold-disk latency.
No cache · no hash · fixed BF16 arithmetic
The lookup was fast. Rebuilding the number was not.
LayerLens precomputed every possible 8-bit significand product, then used real Qwen weights and activations. Parallel direct lookup beat native elementwise multiplication, but sign/exponent reconstruction and unfused accumulation erased the gain in the complete projection.
1M real products128 KiB fixed table62.4% faster lookup aloneBF16 output identical162.76× slower projection
Measured latency ratio · lower is better4 THREADS · MEDIAN × 7
Finding: the 0.38× direct read validates parallel lookup as a candidate. The 6.69× completed products and 162.76× projection reject this unfused implementation. Next work must remove reconstruction or fuse it with accumulation.
Exact optimization · no output change
Schedule shared prefixes while their KV blocks are still warm.
LayerLens now simulates a bounded full-block LRU cache, then groups queued requests by the longest reusable prefix. It reports the layer-token evaluations an engine can skip without storing prompt text.
0% → 50% prefix hits1,152 layer-token ops avoided≈35.03 fixed-matmul GFLOPsOutput preserving
Prefix-aware plan6 REQUESTS · 4 BLOCKS
$ layerlens-cache-plan workload.json \
--block-size 4 \
--capacity-blocks 4 \
--layers 24
FCFS 0 / 2304 layer-token ops avoided
PREFIX 1152 / 2304 layer-token ops avoided
ORDER 0 → 2 → 4 → 1 → 3 → 5
One trace contract · any runtime
Plug LayerLens into your inference stack.
Use the built-in collectors, embed the Python API, or publish a third-party adapter. Installed adapters appear in the same CLI automatically—without changing LayerLens source.
vLLMOllamaHugging FaceCustom adaptersPrivacy-safe defaults
Runtime-neutral captureLayerLens 0.10
$ layerlens-capture --list-runtimes ollama vllm my-runtime $ layerlens-capture \ --runtime my-runtime \ --model provider/model \ --prompt "Benchmark prompt" \ --output traces/run.json
Separate prefill from decode
Step zero represents prompt prefill and first-token latency. Every following step isolates one KV-cache decode pass.
Benchmark the same way
Portable JSON traces make model, runtime, hardware, and configuration comparisons reproducible and easy to share.