Jinja2's error classes and messages (0432), Jinja2's generate() items (0436)

docs/api-2.0.md

@@ -318,6 +318,17 @@ overload set next to the three of `RenderTo`). `RenderOptions` grows by a pointe
 a public struct that must land before G1, after which it cannot grow. The `SecurityError` half of
 the task was done by 0357 S2c; only the docs changed.
 
+**Jinja2's error classes and generate items (tasks 0432, 0436, before G1).** `ErrorCode` gains
+`TemplateSyntaxError = 1023` and `TemplateAssertionError = 1024` (additive; `ExtraParams[0]` is
+Jinja2's message). `UnexpectedStatement` keeps the tag in `ExtraParams[0]` and adds Jinja2's sentence
+as `ExtraParams[1]`; `TemplateRuntimeError` adds the Python class (`"TypeError"`, `"ValueError"`) as
+`ExtraParams[1]` where the engine knows it. `RenderOptions` grows by `bool writeEachPiece`: a sink
+gets one `Write` per piece Jinja2's `generate()` yields (text run, `{{ }}` value, macro or call
+output up to the 512-character buffer), so the Python module's `generate()` yields Jinja2's items.
+Considered: a separate sink callback per piece (a second sink interface for one flag) and
+splitting in the module (the module cannot see node boundaries). Like `generationSpans`, the growth
+of `RenderOptions` must land before G1.
+
 **Prompt escaping and token budgets (tasks 0358, 0359, decided by Ruslan 2026-11-05).** A new,
 always installed header `jinja2cpp/prompt.h` (core, included by `template_env.h`, not behind
 `JINJA2CPP_WITH_CHAT_TEMPLATES`, whose switch must not change the core ABI) adds:

docs/python.md

@@ -112,10 +112,10 @@ engines on a Release build of the module.
 - `env.select_template`, `env.get_or_select_template`, `env.list_templates(extensions=None,
   filter_func=None)` and `env.join_path`.
 - `template.generate()` and `template.stream()` with `TemplateStream` (`dump`,
-  `enable_buffering`, `disable_buffering`). They yield the chunks the engine writes to its
-  `OutputSink` (up to 512 characters each), and an error after the chunks written before it, as
-  Jinja2 does; but the whole render runs at the first item, and the chunks are not Jinja2's
-  one item per output node (lazy rendering is docs/tasks/0436).
+  `enable_buffering`, `disable_buffering`). They yield Jinja2's items (a run of text, the value of
+  a `{{ }}`, a macro or call block's output; the engine's `RenderOptions::writeEachPiece`, 0436),
+  and an error after the items written before it, as Jinja2 does; but the whole render runs at the
+  first item, on the caller's thread (lazy rendering is docs/tasks/0457).
 - `template.make_module(vars=None, shared=False, locals=None)` and `template.module`, from the
   engine's `Template::MakeModule`: the exports are attributes of a `TemplateModule`, `str()` is
   the rendered body, `__html__` is the same as Markup. An exported macro (also inside an exported
@@ -449,8 +449,16 @@ that call: using it after the call returns raises `TemplateRuntimeError`.
   raises `TemplateAssertionError`), and returns an opaque object that only `jinja2cpp.meta`
   takes, not Jinja2's node tree. `find_referenced_templates` gives the names of the tags in
   source order, `None` for a name that is not a literal.
-- `generate()` gives the output as one item, so `stream().enable_buffering(n)` has nothing to
-  combine and an error in the template surfaces at the first `next()` (0355).
+- `generate()` runs the whole template at the first item, where Jinja2 runs it item by item (0457).
+  Jinja2 folds constant output into the text around it (`a{{ 1 }}b` is one item, here three), and
+  yields `super()`, `self.block()` and a recursive loop's call as one item, a macro's output of
+  more than 512 characters as one item, and an empty `{{ }}` as `''` (0456).
+- A Python loader's `get_source` runs again when a template it gave fails to compile, to show
+  the line of the error (twice more with the default fallback, which loads it through Jinja2);
+  Jinja2 calls it once.
+- Tracebacks name the classes `jinja2cpp.exceptions.*`, where Jinja2's say `jinja2.exceptions.*`
+  (0432). They show the template's file and line as a frame, as Jinja2's do: the line of the call
+  for an error a callable raised, and the line of a syntax error.
 - The `Context` a `pass_context` callable gets sees every variable of the call site, the loop
   variables, `{% with %}` names and macro locals included, where Jinja2's holds only the
   arguments, the globals and the template's top-level names; its `vars` and `parent` both hold
@@ -490,4 +498,3 @@ that call: using it after the call returns raises `TemplateRuntimeError`.
   `UndefinedError`. `sort` and `unique` of an unsafe or missing attribute under
   `StrictUndefined` do not raise as Jinja2's do (0413). `sum(attribute=)` of a missing attribute
   is now an `UndefinedError` as in Jinja2 (it was a `TemplateRuntimeError`).
-- Not yet: real streaming (`generate` gives the whole output as one item, docs/tasks/0355).

docs/release-2.0.md

@@ -81,7 +81,7 @@ tag; **A** = accepted for 2.0, tracked, does not block.
 | Gate | What must hold | Tasks | Owner | B/A | Evidence |
 |---|---|---|---|---|---|
 | G0 Scope | Every task in `docs/tasks/` is `done`, `dropped`, or on the postponed list of section 5; nothing is postponed without Ruslan's word | all open tasks | Ruslan | B | `grep -l 'status: open' docs/tasks/0*.md` on the candidate, checked against section 5 |
-| G1 API/ABI freeze | Inline namespace `v2` in every public header, `fwd.h`, no leaked macros or vendored names; `nm` shows `jinja2::v2::`; a consumer built against 1.3.2 headers fails to link against 2.0 (static and shared); the **API wave** has landed first: 0250 (one error contract for the public API) and 0252 (allocator and memory contract, PR #575) settle before the rest, then 0253 (error contract P1), 0262 (`Settings::Limits`), 0266 (`ConfinedFileSystem`), 0172, 0178, 0180, 0181, 0036, 0097, 0098, 0080, 0233, 0236, 0183, 0039, 0358, 0359, 0377 (every task that adds to or changes public headers) follow it, and 0357 (the Python module moves to the public API only, Ruslan 2026-10-30) lands last; decisions of section 8 recorded | [0076](tasks/0076-abi-namespace-and-headers.md) and the API wave | every listed task is done (0039 last, #685), and so is 0186's O2 (oracle version macros in `version.h`). Of those added since 2026-10-30, 0353 (#782), 0358, 0359 and 0357 (the Python module on the public API only; CI builds it against the installed package) and 0377 (chat-template assistant mask via `RenderOptions::generationSpans`, #820) are done; the freeze waits on Ruslan's word; 0441 (a list or tuple subclass iterates as in Python, #827) is done | B | 0076's tests in CI; negative link check |
+| G1 API/ABI freeze | Inline namespace `v2` in every public header, `fwd.h`, no leaked macros or vendored names; `nm` shows `jinja2::v2::`; a consumer built against 1.3.2 headers fails to link against 2.0 (static and shared); the **API wave** has landed first: 0250 (one error contract for the public API) and 0252 (allocator and memory contract, PR #575) settle before the rest, then 0253 (error contract P1), 0262 (`Settings::Limits`), 0266 (`ConfinedFileSystem`), 0172, 0178, 0180, 0181, 0036, 0097, 0098, 0080, 0233, 0236, 0183, 0039, 0358, 0359, 0377 (every task that adds to or changes public headers) follow it, and 0357 (the Python module moves to the public API only, Ruslan 2026-10-30) lands last; decisions of section 8 recorded | [0076](tasks/0076-abi-namespace-and-headers.md) and the API wave | every listed task is done (0039 last, #685), and so is 0186's O2 (oracle version macros in `version.h`). Of those added since 2026-10-30, 0353 (#782), 0358, 0359 and 0357 (the Python module on the public API only; CI builds it against the installed package) and 0377 (chat-template assistant mask via `RenderOptions::generationSpans`, #820) are done; the freeze waits on Ruslan's word; 0441 (a list or tuple subclass iterates as in Python, #827) is done; 0432 (`ErrorCode::TemplateSyntaxError` and `TemplateAssertionError`) and 0436 (`RenderOptions::writeEachPiece`) add to public headers and land before it | B | 0076's tests in CI; negative link check |
 | G2 Known correctness defects | `*context` callable binds positional args; self-referencing namespace/dict/list does not leak under LSan; unsigned above `INT64_MAX` renders the value or fails loudly; with `Settings::finalize` set, `{{ }}` output is still escaped under autoescape (high priority: untrusted values rendered raw) | [0183](tasks/0183-context-callable-positional-arg.md), [0110](tasks/0110-namespace-self-reference-leak.md), [0172](tasks/0172-value-unsigned-overflow.md), [0288](tasks/0288-finalize-autoescape.md) | met: 0183, 0110, 0172 and 0288 are done | B | their unit rows; sanitizer jobs with `detect_leaks=1`; 0288's `autoescape.*finalize*` parity cases |
 | G3 Embedding and trust contract | Lifetime/cache/callback scenarios pass under ASan/UBSan/LSan and TSan; `docs/security.md` states the trust model, what `RealFileSystem` does with `../` and absolute names, the fixed caps, and a reporting contact | [0223](tasks/0223-embedding-lifecycle-integration-tests.md), [0224](tasks/0224-template-trust-and-loader-boundary.md) | 0223, 0224, 0264 and 0327 done; open: 0256, 0263, 0265, 0267, 0324, 0347, 0361, 0365 | B | sanitizer jobs; 0224's disposable-file probes |
 | G4 Migration | `MIGRATION.md` covers renames and behaviour changes with before/after; the migration script rewrites our 1.x-style tests warning-free in CI; a 1.3.2 consumer fixture upgrades against the installed RC | [0077](tasks/0077-migration-script.md), [0222](tasks/0222-downstream-1x-migration-trial.md) | 0077 done (#767); open: 0222 (after G1) and 0388 | B | CI job of 0077 and 0222 |

docs/tasks/0432-upstream-suite-error-types-and-messages.md

@@ -1,9 +1,9 @@
 ---
-status: open
+status: in-progress
 priority: medium
 area: parity
 depends: [0355]
-touches: [python/jinja2cpp/exceptions.py, python/src/errors.cpp, src/template_parser.cpp, src/statements.cpp]
+touches: [python/jinja2cpp/_debug.py, python/jinja2cpp/loaders.py, python/src/errors.cpp, python/src/loaders.cpp, src/template_parser.cpp, src/statements.cpp, src/error_info.cpp, include/jinja2cpp/error_info.h]
 shares: [python/tests/upstream/expected-failures.txt]
 ---
 # Error classes, messages and tracebacks the upstream Jinja2 suite checks
@@ -41,3 +41,38 @@ raises (CPython allows a synthetic frame through `types.TracebackType`).
 
 **Done when.** The listed upstream tests pass through the module, and the C++ messages are
 the same where the C++ API reports them.
+
+## Decisions
+
+Engine half (additive public API):
+
+- `ErrorCode::TemplateSyntaxError = 1023` and `ErrorCode::TemplateAssertionError = 1024`: a construct
+  Jinja2 rejects with a message of its own (its `TemplateSyntaxError`, and the compile-time
+  `TemplateAssertionError`). `ExtraParams[0]` is Jinja2's message. A template that ends inside a
+  statement, trans-block misuse and a required block with content are `TemplateSyntaxError`; an unknown
+  pluralization variable, a trans variable defined twice and a `caller` parameter with no default that the
+  body uses are `TemplateAssertionError`.
+- `UnexpectedStatement` keeps `ExtraParams[0]` = the tag and gains `ExtraParams[1]` = Jinja2's sentence
+  (`Encountered unknown tag 'endif'. Jinja was looking for ...`, parser.py `_fail_ut_eof`); `ToString` prints
+  it. A tag name that is no tag at all now reports it too (it was `UnexpectedToken`).
+- `ErrorCode::TemplateRuntimeError` gets `ExtraParams[1]` = `"TypeError"` or `"ValueError"` where the engine
+  knows the class Jinja2 raises (not iterable, undefined slice bound, doubled `caller`, invalid `xmlattr`
+  key); absent otherwise. The `xmlattr` message is Jinja2's, with the key's repr.
+- A name a `from ... import` cannot find is an undefined value with Jinja2's message (`the template 'm'
+  (imported on line 1) does not export the requested name 'x'`).
+- `{% include x %}` tries the names of a host sequence in turn, as for a template list.
+- A block name with a hyphen is `TemplateSyntaxError` with Jinja2's message.
+
+Module half:
+
+- The module raises `TemplateAssertionError` for 1024, Python's `TypeError`/`ValueError` for 17 with that
+  `ExtraParams[1]`, `TemplatesNotFound` (`.templates`, `.name`) for an include of several names, and a
+  FileSystemLoader's TemplateNotFound names its search path as it is at the raise (`jinja2cpp.loaders._not_found`).
+- A syntax error in a template the render includes carries that template's name, file and source; the source
+  of a template from a Python loader is read again from its `get_source` for the diagnostic.
+- Tracebacks get a template frame (`jinja2cpp._debug.add_template_frame`, Jinja2's fake_traceback): the call's
+  line between the caller and a callable's frames, and the line of a syntax error.
+- Deliberate: tracebacks name `jinja2cpp.exceptions.*` (`test_debug.py::TestDebug::test_regular_syntax_error`,
+  `test_syntax_error`); setting `__module__` to `jinja2.exceptions` would make pickling resolve to Jinja2's class.
+
+**Result.** Upstream suite 607 -> 626 of 856 with this task (all listed tests pass but the two deliberate ones).

docs/tasks/0436-python-lazy-generate.md

@@ -1,9 +1,9 @@
 ---
-status: open
+status: in-progress
 priority: medium
 area: parity
 depends: [0355]
-touches: [python/src/environment.cpp, python/jinja2cpp/environment.py]
+touches: [python/src/environment.cpp, python/jinja2cpp/environment.py, include/jinja2cpp/template.h, src/renderer.h, src/out_stream.h, src/statements.cpp]
 shares: [python/tests/upstream/expected-failures.txt]
 ---
 # `Template.generate` renders on demand, in Jinja2's chunks
@@ -27,3 +27,22 @@ engine's (`test_api.py::TestStreaming::test_basic_streaming`, `test_buffered_str
 
 **Done when.** `next(t.generate())` returns before a later part of the template runs (a
 callable with a side effect shows it), and the two upstream tests pass or are `deliberate`.
+
+## Decisions
+
+- Items (part 2, done): `RenderOptions::writeEachPiece` (public, additive): a body flushes the
+  render's sink stream after each node; a macro or call block holds the flush
+  (`OutStream::PieceHold`), so its output is one item, as Jinja2's. It costs one register test per
+  body node when off (preflight `--perf`: largest change +0.64%). `generate()` renders with it at the
+  first item on the caller's thread and yields the items.
+- On demand (part 1): a worker-thread render was built and verified on this branch (commit
+  dc2a6fd9: daemon thread taking turns with the consumer, back-pressure of one 512-character
+  batch, contextvars copy, Ctrl-C through the poll). The verifier found it cannot keep Jinja2's
+  semantics: the template's callables run on another OS thread, so sqlite objects, `threading.local`
+  state and locks the caller holds break, gevent's patched Thread deadlocks, and a daemon consumer at
+  interpreter exit aborts (pthread_exit unwinding through noexcept destructors). Moved to 0457 for a
+  render that resumes on the caller's thread (decision card 2026-11-10, recommended option, taken
+  while Ruslan's answer was open).
+
+**Result.** `test_basic_streaming` and `test_buffered_streaming` pass (upstream 626 -> 628 of 856).
+Left: 0456 (items Jinja2 folds into one), 0457 (lazy rendering).

docs/tasks/0456-generate-items-folding.md

@@ -0,0 +1,37 @@
+---
+status: open
+priority: low
+area: parity
+depends: [0436]
+touches: [src/renderer.h, src/statements.cpp, src/template_parser.cpp]
+shares: [python/tests/test_generate.py]
+---
+# generate() items where Jinja2 folds output into one
+
+**Problem.** With `RenderOptions::writeEachPiece` (0436) the module's `generate()` yields Jinja2's
+items for text, `{{ }}`, loops, conditions, macro and call blocks, but these cases differ from
+Jinja2 3.1.6 (checked with the pinned oracle):
+
+- Constant output is folded by Jinja2's compiler into the text around it: `a{{ 1 }}b` is one item
+  `"a1b"` in Jinja2, three here (`"a"`, `"1"`, `"b"`).
+- `{{ super() }}` is one item in Jinja2 (the parent block's output joined), several here.
+- A recursive loop (`{% for i in seq recursive %}...{{ loop(x) }}...{% endfor %}`) is buffered by
+  Jinja2 and yielded as one item; here each piece is an item.
+- `{{ self.b() }}` is one item in Jinja2 (`A5B`), three here.
+- Text split by a comment (`a{# c #}b`) or by `{% raw %}` is one item in Jinja2, two here.
+- Constants of other types fold too: `{{ none }}{{ true }}{{ 1.5 }}` is one item in Jinja2, three
+  here; under autoescape `{{ '<' }}a{{ '<' }}` is one item, three here.
+- `{{ x }}` of an empty value (an undefined name) is an item `''` in Jinja2; the sink never gets an
+  empty write, so no item here. The same for an empty macro call (`{{ m() }}`), an empty filter
+  block, `{{ '' }}` and `{{ x if not x }}`.
+- A macro or call block whose output is longer than the stream's 512-character buffer is one item
+  in Jinja2 and several here (the buffer flushes when full; found by the 0436 verifier round 2):
+  a 900-character macro gives `[384, 384, 132]`.
+
+**Proposal.** Hold the piece flush (`OutStream::PieceHold`) in `super()` and in a recursive loop's
+body, and mark constant-folded runs when the parser merges them (or merge adjacent constant
+nodes at parse time, which also helps the render). Let a held piece grow past the buffer (or hand the sink the held run in one
+call). Add the cases to
+`OutputTest.SinkGetsEachPieceWhenAsked` and python/tests/test_generate.py.
+
+**Done when.** These cases yield Jinja2's items through `generate()`.

docs/tasks/0457-python-generate-on-demand.md

@@ -0,0 +1,31 @@
+---
+status: open
+priority: medium
+area: parity
+depends: [0436]
+touches: [python/src/environment.cpp, python/jinja2cpp/environment.py]
+shares: [python/tests/test_generate.py, python/tests/upstream/expected-failures.txt]
+---
+# `Template.generate` renders on demand, on the caller's thread
+
+**Problem.** Jinja2's `generate()` runs the template as the consumer asks for items, so a server
+sends the first bytes before the render ends and holds one item in memory. Since 0436 the module
+yields Jinja2's items, but the whole render runs at the first `next()`.
+
+0436 built and verified a worker-thread render (branch commit dc2a6fd9, decisions in the 0436 task
+file) and dropped it: a Python callable the template calls then runs on another OS thread, which
+breaks sqlite objects made in the caller's thread, `threading.local` state (Django connections),
+locks the caller holds, and gevent (its patched `Thread` is a greenlet on the caller's thread); and
+a daemon consumer at interpreter exit aborts.
+
+**Proposal.** Resume the engine's render on the caller's thread: run it on a stack of its own (a
+fiber: Windows fibers, `ucontext` or a small assembly switch on POSIX, with ASan's
+`__sanitizer_start_switch_fiber`) and switch back at each sink write. Between two items no Python
+frame of the render is live (a callable has returned before the engine writes), so CPython's thread
+state needs no swapping; check that claim, the engine's stack checks on a stack of its own
+(recursion_guard.cpp skips them there now), macOS arm64 support, and the cost per switch. Needs a
+plan (architect) before code.
+
+**Done when.** `next(t.generate())` returns before a later part of the template runs (a callable
+with a side effect shows it), callables run on the caller's thread, and closing the generator ends
+the render.