EXTRA (not in the plan): the vendor's own benchmarks/bench_latency.py --scenario json --iterations 50 (tag v0.41.0), run on this box 2026-10-20 ~11:41 UTC+8 against the installed headroom-ai 0.41.0. Headroom Latency Benchmark ======================================== Generating test scenarios... 14 scenarios ready Running benchmarks (50 iterations, 3 warmup)... [1/14] JSON: Search Results (100 items)... 5.8ms (p50), 48% compression, 4,868 tokens saved [2/14] JSON: Search Results (500 items)... 6.0ms (p50), 49% compression, 24,364 tokens saved [3/14] JSON: Search Results (1K items)... 6.2ms (p50), 49% compression, 48,799 tokens saved [4/14] JSON: Search Results (5K items)... 7.8ms (p50), 49% compression, 244,236 tokens saved [5/14] JSON: API Responses (500 items)... 6.0ms (p50), 44% compression, 17,033 tokens saved [6/14] JSON: Database Rows (1K rows)... 5.8ms (p50), 63% compression, 27,455 tokens saved [7/14] JSON: String Array (100 strings)... 5.8ms (p50), 77% compression, 807 tokens saved [8/14] JSON: String Array (500 strings)... 5.8ms (p50), 95% compression, 4,619 tokens saved [9/14] JSON: String Array (1K strings)... 5.8ms (p50), 97% compression, 9,374 tokens saved [10/14] JSON: Number Array (200 numbers)... 6.0ms (p50), 87% compression, 1,079 tokens saved [11/14] JSON: Number Array (1K numbers)... 6.0ms (p50), 96% compression, 5,846 tokens saved [12/14] JSON: Mixed Array (250 items)... 6.1ms (p50), 50% compression, 1,132 tokens saved [13/14] JSON: Flat Object (100 keys)... 5.8ms (p50), 1% compression, 63 tokens saved [14/14] JSON: Nested Object (3 arrays)... 5.8ms (p50), 42% compression, 9,542 tokens saved ==================================================================================================== HEADROOM LATENCY BENCHMARK ==================================================================================================== COMPRESSION OVERHEAD BY SCENARIO ---------------------------------------------------------------------------------------------------- Scenario Tokens In Saved Ratio p50 p95 p99 Mean ---------------------------------------------------------------------------------------------------- JSON: Search Results (100 items) 10.2K 4.9K 48% 5.8ms 7.1ms 7.5ms 6.0ms JSON: Search Results (500 items) 50.2K 24.4K 49% 6.0ms 7.2ms 7.7ms 6.1ms JSON: Search Results (1K items) 100.5K 48.8K 49% 6.2ms 8.4ms 9.6ms 6.6ms JSON: Search Results (5K items) 502.6K 244.2K 49% 7.8ms 9.6ms 12.6ms 8.0ms JSON: API Responses (500 items) 38.9K 17.0K 44% 6.0ms 7.5ms 10.1ms 6.2ms JSON: Database Rows (1K rows) 43.7K 27.5K 63% 5.8ms 6.9ms 7.5ms 6.0ms JSON: String Array (100 strings) 1.1K 807 77% 5.8ms 7.4ms 9.8ms 6.1ms JSON: String Array (500 strings) 4.9K 4.6K 95% 5.8ms 6.8ms 7.1ms 5.8ms JSON: String Array (1K strings) 9.6K 9.4K 97% 5.8ms 7.5ms 9.7ms 6.1ms JSON: Number Array (200 numbers) 1.2K 1.1K 87% 6.0ms 7.4ms 7.5ms 6.1ms JSON: Number Array (1K numbers) 6.1K 5.8K 96% 6.0ms 7.2ms 60.1ms 7.1ms JSON: Mixed Array (250 items) 2.3K 1.1K 50% 6.1ms 7.3ms 7.8ms 6.3ms JSON: Flat Object (100 keys) 6.1K 63 1% 5.8ms 7.3ms 10.1ms 6.1ms JSON: Nested Object (3 arrays) 22.8K 9.5K 42% 5.8ms 7.1ms 7.3ms 5.9ms PER-TRANSFORM BREAKDOWN (selected scenarios) -------------------------------------------------------------------------------- Scenario Transform p50 (ms) % Total -------------------------------------------------------------------------------- JSON: Search Results (100 items) content_router 5.7ms 99% JSON: Search Results (500 items) content_router 5.9ms 99% JSON: Search Results (1K items) content_router 6.1ms 99% JSON: Search Results (5K items) content_router 7.7ms 99% JSON: API Responses (500 items) content_router 5.9ms 99% JSON: Database Rows (1K rows) content_router 5.8ms 99% JSON: String Array (100 strings) content_router 5.8ms 99% JSON: String Array (500 strings) content_router 5.7ms 99% JSON: String Array (1K strings) content_router 5.7ms 99% JSON: Number Array (200 numbers) content_router 5.9ms 99% JSON: Number Array (1K numbers) content_router 5.9ms 99% JSON: Mixed Array (250 items) content_router 6.1ms 99% JSON: Flat Object (100 keys) content_router 5.8ms 99% JSON: Nested Object (3 arrays) content_router 5.8ms 99% COST-BENEFIT ANALYSIS ---------------------------------------------------------------------------------------------------- Reference model: Claude Sonnet 4.5 (0.03ms/token prefill) Scenario Compress LLM Saved Net Benefit $/1K Reqs ---------------------------------------------------------------------------------------------------- JSON: Search Results (100 items) 5.8ms 146ms +140.2ms $ 14.60 JSON: Search Results (500 items) 6.0ms 731ms +725.0ms $ 73.09 JSON: Search Results (1K items) 6.2ms 1464ms +1457.8ms $ 146.40 JSON: Search Results (5K items) 7.8ms 7327ms +7319.3ms $ 732.71 JSON: API Responses (500 items) 6.0ms 511ms +505.0ms $ 51.10 JSON: Database Rows (1K rows) 5.8ms 824ms +817.8ms $ 82.36 JSON: String Array (100 strings) 5.8ms 24.2ms +18.4ms $ 2.42 JSON: String Array (500 strings) 5.8ms 139ms +132.8ms $ 13.86 JSON: String Array (1K strings) 5.8ms 281ms +275.4ms $ 28.12 JSON: Number Array (200 numbers) 6.0ms 32.4ms +26.4ms $ 3.24 JSON: Number Array (1K numbers) 6.0ms 175ms +169.4ms $ 17.54 JSON: Mixed Array (250 items) 6.1ms 34.0ms +27.8ms $ 3.40 JSON: Flat Object (100 keys) 5.8ms 1.9ms -3.9ms $ 0.19 JSON: Nested Object (3 arrays) 5.8ms 286ms +280.4ms $ 28.63 BREAK-EVEN ANALYSIS -------------------------------------------------------------------------------- Minimum tokens saved for compression to pay for itself in latency: GPT-4o Mini (0.01ms/token): JSON: Search Results (100 items) need 579, save 4.9K -> ALWAYS WINS JSON: Search Results (500 items) need 595, save 24.4K -> ALWAYS WINS JSON: Search Results (1K items) need 621, save 48.8K -> ALWAYS WINS JSON: Search Results (5K items) need 776, save 244.2K -> ALWAYS WINS JSON: API Responses (500 items) need 597, save 17.0K -> ALWAYS WINS JSON: Database Rows (1K rows) need 584, save 27.5K -> ALWAYS WINS JSON: String Array (100 strings) need 583, save 807 -> ALWAYS WINS JSON: String Array (500 strings) need 577, save 4.6K -> ALWAYS WINS JSON: String Array (1K strings) need 581, save 9.4K -> ALWAYS WINS JSON: Number Array (200 numbers) need 600, save 1.1K -> ALWAYS WINS JSON: Number Array (1K numbers) need 595, save 5.8K -> ALWAYS WINS JSON: Mixed Array (250 items) need 614, save 1.1K -> ALWAYS WINS JSON: Flat Object (100 keys) need 583, save 63 -> OVERHEAD > SAVINGS JSON: Nested Object (3 arrays) need 583, save 9.5K -> ALWAYS WINS GPT-4o (0.03ms/token): JSON: Search Results (100 items) need 193, save 4.9K -> ALWAYS WINS JSON: Search Results (500 items) need 198, save 24.4K -> ALWAYS WINS JSON: Search Results (1K items) need 207, save 48.8K -> ALWAYS WINS JSON: Search Results (5K items) need 258, save 244.2K -> ALWAYS WINS JSON: API Responses (500 items) need 199, save 17.0K -> ALWAYS WINS JSON: Database Rows (1K rows) need 194, save 27.5K -> ALWAYS WINS JSON: String Array (100 strings) need 194, save 807 -> ALWAYS WINS JSON: String Array (500 strings) need 192, save 4.6K -> ALWAYS WINS JSON: String Array (1K strings) need 193, save 9.4K -> ALWAYS WINS JSON: Number Array (200 numbers) need 200, save 1.1K -> ALWAYS WINS JSON: Number Array (1K numbers) need 198, save 5.8K -> ALWAYS WINS JSON: Mixed Array (250 items) need 204, save 1.1K -> ALWAYS WINS JSON: Flat Object (100 keys) need 194, save 63 -> OVERHEAD > SAVINGS JSON: Nested Object (3 arrays) need 194, save 9.5K -> ALWAYS WINS Claude Sonnet 4.5 (0.03ms/token): JSON: Search Results (100 items) need 193, save 4.9K -> ALWAYS WINS JSON: Search Results (500 items) need 198, save 24.4K -> ALWAYS WINS JSON: Search Results (1K items) need 207, save 48.8K -> ALWAYS WINS JSON: Search Results (5K items) need 258, save 244.2K -> ALWAYS WINS JSON: API Responses (500 items) need 199, save 17.0K -> ALWAYS WINS JSON: Database Rows (1K rows) need 194, save 27.5K -> ALWAYS WINS JSON: String Array (100 strings) need 194, save 807 -> ALWAYS WINS JSON: String Array (500 strings) need 192, save 4.6K -> ALWAYS WINS JSON: String Array (1K strings) need 193, save 9.4K -> ALWAYS WINS JSON: Number Array (200 numbers) need 200, save 1.1K -> ALWAYS WINS JSON: Number Array (1K numbers) need 198, save 5.8K -> ALWAYS WINS JSON: Mixed Array (250 items) need 204, save 1.1K -> ALWAYS WINS JSON: Flat Object (100 keys) need 194, save 63 -> OVERHEAD > SAVINGS JSON: Nested Object (3 arrays) need 194, save 9.5K -> ALWAYS WINS Claude Opus 4 (0.08ms/token): JSON: Search Results (100 items) need 72, save 4.9K -> ALWAYS WINS JSON: Search Results (500 items) need 74, save 24.4K -> ALWAYS WINS JSON: Search Results (1K items) need 77, save 48.8K -> ALWAYS WINS JSON: Search Results (5K items) need 97, save 244.2K -> ALWAYS WINS JSON: API Responses (500 items) need 74, save 17.0K -> ALWAYS WINS JSON: Database Rows (1K rows) need 73, save 27.5K -> ALWAYS WINS JSON: String Array (100 strings) need 72, save 807 -> ALWAYS WINS JSON: String Array (500 strings) need 72, save 4.6K -> ALWAYS WINS JSON: String Array (1K strings) need 72, save 9.4K -> ALWAYS WINS JSON: Number Array (200 numbers) need 75, save 1.1K -> ALWAYS WINS JSON: Number Array (1K numbers) need 74, save 5.8K -> ALWAYS WINS JSON: Mixed Array (250 items) need 76, save 1.1K -> ALWAYS WINS JSON: Flat Object (100 keys) need 72, save 63 -> OVERHEAD > SAVINGS JSON: Nested Object (3 arrays) need 72, save 9.5K -> ALWAYS WINS JSON results saved to: /workspace/triedagents-tools/runs/2026-10-20/tools/headroom/c3_extra_vendor_bench_latency.json