# High-Volume Render Benchmark Story: "Time-box the first high-volume benchmark" (`backlog/epics/05_cli_rendering_engine_and_debenu_integration.md`, 5 points, spike). Target under test (`project_config.md`): **render 100,000 records at 300 DPI in under 10 minutes.** **Bottom line: the current render path does NOT meet the target and is not close.** Throughput degrades sharply as the accumulated in-memory PDF document grows, so the small-scale numbers from Sprint 1 (392 records, ~1.8s) are not representative of high-volume behavior at all. This is a real, material risk and is being escalated as a new backlog story rather than fixed silently inside this spike (out of scope per the story's own conversation notes: "a spike-style story to gather performance information, not a final optimization guarantee"). ## Environment - Machine: Intel Core i7-8750H @ 2.20GHz, ~16 GB RAM, Windows 10.0.26200 (Windows 11 Pro). - Build: `dotnet build src/EnvelopeRenderer.Cli/EnvelopeRenderer.Cli.csproj -c Release` (net10.0, Release configuration — not `dotnet run`/Debug, to avoid understating a production-representative number). - Debenu Quick PDF Library 10.13, 64-bit DLL, licensed via the project's `key.txt` (`DebenuLicenseKeyResolver`, walked up from the executable directory — same resolution path a desktop-launched render would use). - CLI invoked directly (no desktop app in the loop), template `sample-data/sample-envelope-template.xml` (3 `` elements per page: 1 static, 2 dynamic columns — same template Sprint 1 verified against). ## Benchmark scenario (repeatable) 1. Generate a ~100k-record CSV with the same shape/quoting as the real sample data by repeating the 392 data rows from `sample-data/87700 - 999999 - Wilson Township.csv` verbatim 256 times (header written once): `256 * 392 = 100,352` data rows. Rows are exact repeats rather than synthetic-but-unique data — acceptable for a throughput/bottleneck spike since neither CsvHelper parsing nor the render merge loop branches on row content, only on column names and value length, and the CLI's per-record work is O(1) regardless (see "Root cause" below). Not committed to the repo (30+ MB; regenerate on demand with the one-line `awk` command in this file's git history / the Day 1 scrum log, or any equivalent script). 2. Run the CLI directly: `EnvelopeRenderer.Cli.exe --template sample-envelope-template.xml --csv --output .pdf`, with `DEBENU_LICENSE_KEY` resolved via `key.txt` exactly as production/desktop launches do. 3. Record the `PROGRESS render ... elapsedMs= completed=` stream (already emitted by the CLI per `CLI_CONTRACT.md` — no extra instrumentation needed) and the final PDF size on success. 4. 300 DPI: this template/render path does not embed raster images or otherwise vary output by DPI (text-only, vector text drawn via `DrawText` at exact point coordinates) — there is no DPI knob in the current text-only pipeline, so "at 300 DPI" is satisfied vacuously for this slice. This will matter once image handling (epic 7) lands and should be re-benchmarked then. ## Results ### Full ~100k-record run — timed-boxed, not run to completion The run was allowed to proceed for **138.8 seconds of wall time** and was then deliberately stopped (time-boxed) once the throughput trend below made the outcome unambiguous — projected full-run time is many multiples of the 10-minute target (see "Extrapolation"). No output PDF exists for this run since `Save()` is only called once every record has been added (see "Root cause"); the process was killed before reaching record 100,352, so nothing was written to disk for this run specifically. Representative samples from the real `PROGRESS render` stream (full log evidence recorded during this spike, not fabricated): | Elapsed | Records completed | Records/sec in the preceding ~10s window | |---|---|---| | 108 ms | 1 | — | | 1,109 ms | 442 | ~399/s (first second, before the effect below kicks in) | | 10,147 ms | 1,389 | ~105/s | | 20,276 ms | 1,930 | ~53/s | | 30,416 ms | 2,346 | ~41/s | | 40,521 ms | 2,696 | ~35/s | | 50,626 ms | 3,004 | ~30/s | | 60,809 ms | 3,282 | ~27/s | | 71,009 ms | 3,533 | ~25/s | | 81,245 ms | 3,766 | ~23/s | | 91,447 ms | 3,985 | ~22/s | | 101,725 ms | 4,184 | ~19/s | | 112,015 ms | 4,381 | ~19/s | | 122,238 ms | 4,555 | ~17/s | | 132,617 ms | 4,724 | ~16/s | | 138,843 ms (stopped here) | 4,827 | ~15/s and still falling | Throughput is **monotonically decreasing** — not a one-time warm-up cost — and had not leveled off to a stable floor by the time the run was stopped at 4,827 of 100,352 records (4.8%). Process working-set memory grew from ~63 MB to ~76 MB over the same window (modest, not itself a risk at this scale) while CPU stayed pegged at effectively one full core the whole time — this is a compute-bound slowdown, not an I/O or memory-pressure one. ### Smaller run — full completion, for a real (non-extrapolated) data point To have at least one *complete, verified* higher-volume data point rather than relying only on extrapolation, the same scenario was also run to completion at 2,000 records (first 2,000 rows of the same generated dataset): ``` PROGRESS startup elapsedMs=0 completed=0 ... PROGRESS complete elapsedMs=20775 completed=2000 exit=0 ``` - Elapsed: **20.775 seconds** for 2,000 records (~96 records/sec average over the whole run — already well down from the ~399/s seen in the first second of the 100k attempt, confirming the same degradation curve). - Output: real, valid PDF, `%PDF-1.4` header confirmed, **2,302,387 bytes** (~1.15 KB/page — in line with Sprint 1's 392-page/~1.3 MB result, so per-page *output size* is not the problem). - Exit code `0`, matching `CLI_CONTRACT.md`. This confirms the CLI still produces correct output at this scale — the risk is purely **throughput**, not correctness. ### Comparison to Sprint 1's only prior data point Sprint 1 Batch 2 verified 392 records in ~1.8s (~217 rec/s using the throttled progress numbers in `CLI_CONTRACT.md`'s smoke example). That number was accurate for *its own scale* but is not representative of high-volume behavior — the degradation only becomes visible past roughly the first 1,000–2,000 accumulated pages, well beyond what a 392-row sample template exercises. This is exactly the kind of thing this spike exists to catch before more work lands on the render hot path (per the Sprint 1 retrospective action item). ## Root cause (spike-level investigation, not a fix) The CLI's own C# merge loop (`RenderEngine.Render`, `CsvRecordSource.ReadRecords`) does **O(1)** work per record: CSV rows are streamed one at a time (no full-file buffering), and each record produces a small fixed-size list of `TextDraw`s with no data structure that grows with the number of records processed so far, other than a `Dictionary` font-handle cache keyed by distinct font name (this template uses exactly one font, so that cache never grows past size 1). `DebenuPdfRenderer.AddPage` calls a fixed, constant number of Debenu API functions per page (`NewPage`/`SetPageDimensions`/`SetFillColor`/`SelectFont`/`SetTextSize`/`DrawText` x3) with no loop over prior pages. That leaves the Debenu Quick PDF Library 10.13 native document object model itself as the strongly suspected source of the slowdown: the whole PDF is built up in memory (confirmed — no PDF bytes are written to disk until the single `SaveToFile` call after every record has been added, so a killed 100k run leaves no output file at all, as seen above) and each additional `NewPage`/`DrawText` call appears to cost more as the number of already-added pages grows. This is consistent with, though not proven to be, an internal data structure in the vendor DLL that is scanned or re-walked per operation (e.g., an internal page/object list) rather than one with O(1) amortized append. This repo has no access to Debenu's internal source to confirm further; treating it as an external-library characteristic to design around, not something to patch, is the appropriate scope for a spike. ## Verdict against the target **Off track.** Using the observed trend (throughput still falling at ~15 rec/s and not yet at a floor when stopped at record 4,827), a full 100,352-record run would take at least on the order of **45–90+ minutes** — 5x to 10x+ over the 10-minute target — and possibly worse, since the curve had not plateaued. This is a material risk, not a rounding-error miss. ## Follow-up backlog work captured Logged as both a new backlog story and a technical debt entry so it survives past this spike: - New story: "Investigate and address high-volume render throughput degradation" — added to `backlog/epics/05_cli_rendering_engine_and_debenu_integration.md` (Status: Ready). Scope: characterize whether the degradation is genuinely Debenu-internal (e.g. by probing whether periodic `Save`+reopen batching, or Debenu's `AddSubsettedFont`/page-count-related settings, change the curve) and pick a concrete mitigation (candidates to evaluate, not yet decided: batching output into multiple PDF files and concatenating, periodic incremental saves, or an alternate Debenu API usage pattern) before any further feature work depends on rendering 1M-record jobs (`project_config.md`'s stated ceiling). - Technical debt: `logs/technical_debt_log.md`, 2026-09-14 entry — "release-quality high-volume benchmark target is currently missed by 5-10x+ at realistic scale." Not fixed as part of this spike, per the story's own conversation notes ("not a final optimization guarantee") and to avoid silently absorbing a scope-changing investigation into a 5-point spike. ## Effect of Batch 3 (progress reporting) and the license-key resolver on this number Both were already in place for this run (the CLI always emits `PROGRESS` events and always resolves the license key via `DebenuLicenseKeyResolver`) and neither is a plausible contributor: - The progress reporter is throttled to at most once per second of wall time (`ConsoleProgressReporter`) and does O(1) work per throttled write; it cannot explain a monotonically *decreasing* per-record rate, since its own overhead is constant per record (a delegate call) and near-zero when throttled. - License key resolution (`DebenuLicenseKeyResolver.Resolve`) runs exactly once at startup, before any records are processed, so it cannot affect per-record throughput at all. Both are ruled out; the bottleneck is isolated to the per-page Debenu document-building cost described above.