[rmorozov]

Problem

Task 0359: prompt templates need to fit retrieved content into a token budget. This is the last step of the approved 0358/0359 plan (/mnt/project-files/0358/plan.md, sections 4 and 7). The contract is in docs/cpp-only-filters.md and docs/api-2.0.md section 3.3, merged in #766.

Resulting behaviour

• Tokenizer is a new class in include/jinja2cpp/prompt.h, set through Settings::tokenizer.
  • It has a shared immutable implementation, like PromptEscaper, and compares by identity.
  • It takes a count function and an optional prefix function.
  • Without a count function it estimates ceil(utf8 bytes / 4), for narrow and wide templates alike.
• count_tokens counts a string with the tokenizer. An empty string costs no call. A count above the int64 range saturates.
• truncate_tokens(n, end='...') keeps the longest prefix that, together with end, fits in n tokens.
  • It uses the prefix callback when one is given.
  • Offsets the host returns are clamped and rounded down to a code-point start; wide text keeps surrogate pairs whole.
  • An end longer than n raises TemplateRuntimeError.
• pack(budget, attribute, by, reverse, reserve, stop) keeps items in their original order.
  • Items are measured lazily, at most once each, and none are measured after the budget is spent or stop ends the packing.
  • by sorts with the code behind sort, which is extracted as SortedOrderByPaths/IsLessKey; Sort's instruction counts are unchanged.
  • Items are copied out as list does, which applies MaxSequenceSize and never holds a lent item past the statement.
• Errors in callbacks: a callback exception, std or not, fails the render with UserCallableException. For a std exception, GetException() is set.
• Bench:
  • New case bench/cases/prompt_budget: 200 chunks, then pack + neutralize|fence with a word-counting tokenizer.
  • The driver's --dump-dir fails if one render exceeds max_token_calls.
  • The case is C++ only.
  • The gate in benchmark.yml and preflight.sh counts the base's own cases when the base driver predates the new settings keys.
• Task files: 0358 is set to done (Add PromptEscaper and the neutralize and fence filters (0358, PR 1) #777, Prompt autoescape mode (0358, PR 2) #784). Follow-ups are filed as 0408 (a capability marker for the bench gate instead of a grep chain) and 0409 (bound the truncate_tokens shrink loop against a non-monotonic tokenizer).

Validation

• Debug build: gcc with -Werror. The full ctest passes 8/8 on this head, rebased onto 3660cfd.
• Sanitizers (verifier): a clang-18 ASan+UBSan build passes the full suite, 10819 tests, with no reports.
• New tests: 89 rows, narrow and wide. They follow the plan's table and add lifetime, sequence-cap and call-count cases.
• Bench assertion (verifier): it fails as expected when max_token_calls is lowered to 10.
• preflight: scripts/preflight.sh --perf against 3660cfd passes every check. No existing case moves by more than about +0.4%. prompt_budget is new: Render 1.31M instructions and 76 allocations.

Limits

• The plan's example a b... comes out as a b ... with the test's word tokenizer, whose prefix keeps the trailing space. That is a property of the test fixture, not of the filter.
• Not built here with MSVC or Apple Clang, and a 16-bit wchar_t is exercised only by CI.

Process

• Roles: implementer (Sonnet) and verifier (Sonnet, high effort).
• Verifier: 1 round, with no blocking findings. Of its nits, the count saturation is fixed, the shrink bound is filed as 0409, and the bench gate as 0408.
• Red CI pushes: none caused by this PR. On 803555c, C++26 clang-20 Debug failed while apt installed clang-20, before any build; the one re-run passed.
• Codex: 1 finding, 1 fixed (P1: truncate_tokens did not check maxSequenceSize on long host text; now it does, with a narrow and wide test). The security review, asked for again here to cover Add PromptEscaper and the neutralize and fence filters (0358, PR 1) #777 and Prompt autoescape mode (0358, PR 2) #784, did not run.

---

Generated by Claude Code

[rmorozov]

@codex review

@codex security review — please include the prompt escaping paths merged in #777 and #784 (src/prompt_escape.cpp, src/prompt_filters.cpp, the prompt autoescape sites), whose security reviews did not run.

---

Generated by Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-06T11:13:32.321850Z	f9b1f42	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f9b1f42759

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[rmorozov]

C++26 clang-20 Debug shared=ON failed at 803555c before anything was built. The apt-get install of clang-20 timed out three times on the Azure mirror (the libclang-rt-20-dev download). No code from this PR ran, so the failure is not this PR's. I've re-run the job once.

---

Generated by Claude Code