Public RenderOptions, FindReferencedTemplates and sandbox policy (0357 S2b, S2c)

CMakeLists.txt

@@ -263,6 +263,7 @@ set(JINJA2CPP_SOURCES
     src/prompt_escape.cpp
     src/prompt_filters.cpp
     src/public_host_object.cpp
+    src/public_sandbox_policy.cpp
     src/token_budget.cpp
     src/python_format.cpp
     src/recursion_guard.cpp
@@ -272,6 +273,7 @@ set(JINJA2CPP_SOURCES
     src/statements.cpp
     src/string_converter_filter.cpp
     src/template.cpp
+    src/template_render_options.cpp
     src/template_dict.cpp
     src/template_env.cpp
     src/template_module.cpp
@@ -530,6 +532,7 @@ if (JINJA2CPP_BUILD_TESTS)
         test/helpers_tests.cpp
         test/host_object_test.cpp
         test/public_host_object_test.cpp
+        test/public_sandbox_policy_test.cpp
         test/sandbox_policy_test.cpp
         test/i18n_test.cpp
         test/limits_test.cpp
@@ -548,6 +551,7 @@ if (JINJA2CPP_BUILD_TESTS)
         test/arena_switch_test.cpp
         test/ordered_map_test.cpp
         test/recursion_limits_test.cpp
+        test/referenced_templates_test.cpp
         test/slot_frame_test.cpp
         test/statements_tets.cpp
         test/support_types_test.cpp
@@ -565,11 +569,6 @@ if (JINJA2CPP_BUILD_TESTS)
         test/value_ptr_test.cpp
         test/parity/parity_test.cpp
     )
-    # The referenced-templates walk is a header-only internal of the Python module, which links
-    # the library statically: it calls the parser, which a shared library does not export
-    if (NOT JINJA2CPP_BUILD_SHARED)
-        list(APPEND JINJA2CPP_TEST_SOURCES test/referenced_templates_test.cpp)
-    endif ()
     if (JINJA2CPP_WITH_CHAT_TEMPLATES)
         list(APPEND JINJA2CPP_TEST_SOURCES test/chat_templates/chat_template_test.cpp test/chat_template_preset_test.cpp)
     endif ()

docs/api-2.0.md

@@ -176,7 +176,9 @@ locations), 0255 (engine runtime errors).
   `Load` replaces the template for its own object only, and a module keeps its tree, reversing
   the second half of 0284 D1a. 0301 raises it for a stored macro of a module that ended with an
   earlier call; 0300 for a template callable lent to a host's call and called after it), 20 `DataReadError` (`ToValue` could not
-  read a JSON document: its `JsonTraits` reported an error; 0311). `UnexpectedException` (1) now means "an exception the
+  read a JSON document: its `JsonTraits` reported an error; 0311), 21 `SecurityError` (a sandbox refused an attribute or a
+  list or dict method: `ISandboxPolicy`, `Settings::immutableContainers`; 0357 B7, until then an `UndefinedError` with a
+  marker in `extraParams[1]`). `UnexpectedException` (1) now means "an exception the
   engine did not raise and could not attribute", with location and `GetException()`.
 
 **Against Python.** Same: syntax, undefined and not-found errors keep their codes, with
@@ -837,7 +839,9 @@ Recommend A now, C only if someone asks.
      `UserCallable::keepUndefined`. `None` stays the default.
    - **Rendering.** `RenderOptions`, with per-render globals and a `std::function` poll;
      call-site information for `*context` callables; hex digits for `BigInt`;
-     `Template::FindReferencedTemplates`.
+     `Template::FindReferencedTemplates`. The globals are a `shared_ptr<const ValuesMap>`, not a
+     call-scoped borrow: a value of them that a callable keeps stays valid after the render, as
+     an environment global's does (S2b).
    - **Sandbox.** An optional public `ISandboxPolicy`, at no cost when it is unset. Both the
      native attribute path and the direct method-call path honour it.
    - **Growth after the freeze.** Through `IHostObject2 : IHostObject`, found once per object.

docs/design/0357-python-module-public-api.md

@@ -181,3 +181,9 @@ Codex agrees with option 2 and with composition (D1). It also agrees with D3: an
 - **D6: Codex disagrees with the switch-only sandbox.** Under B7 alone, a subclassed `SandboxedEnvironment` that rejects a native method never sees that operation. Native methods are resolved and called directly (`src/expression_evaluator.cpp:1685-1696`), and native attribute safety is decided in `src/value_methods.cpp:2747-2764`. So the loss is in what the policy enforces, not only in message text. Codex prefers an optional public policy callback, with no cost when it is unset, honoured on both the native attribute path and the direct method-call path. A fixed built-in sandbox is acceptable only as an explicit, owner-approved reduction of the drop-in contract.
 
 **Effect on the plan.** B1 gains three export capabilities: lossless entries, the tuple flag and the identity token. All three belong to the public `GenericList`/`GenericMap` export views and also help C++ users. The D6 recommendation changes to **D6b, an optional `ISandboxPolicy`**, because parity in what the policy enforces is a goal and ordinary renders pay nothing.
+
+## 10. S2b and S2c as built (2026-11-09)
+
+- **B3.** `RenderOptions{ shared_ptr<const ValuesMap> globals; std::function<void()> poll; }` and `Template::RenderTo(out|sink, params, options)`. The globals are shared, not borrowed: a value of them that a callable keeps stays valid after the render (`LendGlobal`, as for the environment's globals). A poll alone makes a render count its steps (`PollOnlyLimits`), and a render the polled one starts with no poll of its own is polled too. Same `RenderCore<true>` instantiation as the module's `RenderInternal`; the ordinary render is unchanged (instruction gate).
+- **B6.** `Template::FindReferencedTemplates()`; wide templates now give their names (they gave `nullopt` before).
+- **B7 / D6b.** `jinja2::ISandboxPolicy` in `jinja2cpp/sandbox_policy.h`, held by `Settings::sandboxPolicy` (identity equality; a template keeps the policy it was loaded with, 0427). `NativeAttribute(obj, name) -> optional<Value>` is asked where the engine's policy was; `Format(...) -> optional<Value>` returns nullopt by default, and the engine's own `str.format` then reads a field's attribute through `NativeAttribute`. `Settings::immutableContainers` replaces `Dialect::ImmutableSandbox` for users; `ErrorCode::SecurityError` (21) replaces the `extraParams[1]` marker; `UndefinedDescription::security` marks a refused attribute. The engine's interface is `IEngineSandboxPolicy` with one bridge (`src/public_sandbox_policy.cpp`), as for host objects.

docs/python.md

@@ -345,15 +345,15 @@ inside the adapters that let the engine read Python objects, not in generated co
   Jinja2's `OverflowError("Range too big. The sandbox blocks ranges larger than MAX_RANGE
   (100000).")`.
 - **The engine's own values** (the template's strings, numbers, lists, dicts, `loop`, `cycler`,
-  macros) have no Python attributes. The environment holds an `ISandboxPolicy` (src/sandbox_policy.h,
-  internal until 0357 makes it public) that the engine asks where a native value's attribute is
+  macros) have no Python attributes. The environment holds an `IEngineSandboxPolicy` (src/sandbox_policy.h,
+  the engine's side of the public `ISandboxPolicy` the module moves to in 0357 S3) that the engine asks where a native value's attribute is
   missing, after it finds one of its methods, and at `str.format`. A private name (`__class__`,
   `__init__`, `_x`) on one is asked of `is_safe_attribute` with a stand-in object of the Python
   type and is refused with the same undefined; so is a public name when the subclass's
   `is_safe_attribute` is not Jinja2's. `format_map` of a string, which the engine lacks, and
   `str.format` are Python's, run through `SandboxedFormatter`. `ImmutableSandboxedEnvironment`
   also refuses the methods of the engine's lists and dicts that change them (`append`, `update`,
-  ...) with Jinja2's message (`Dialect::ImmutableSandbox`, the read-only containers of the
+  ...) with Jinja2's message and `ErrorCode::SecurityError` (`Dialect::ImmutableSandbox`, the read-only containers of the
   transformers preset).
 - **Cost.** A plain `Environment` pays a null check where the policy would be asked: the checks
   are behind a pointer the render already holds, which only a sandboxed environment sets. A sandboxed render calls `is_safe_attribute` once

docs/tasks/0357-python-module-public-api.md

@@ -62,3 +62,23 @@ freeze.
   710,482; mitsuhiko_table 8,211,299 / 9,131,276 / 8,890,719. D7 is relaxed for host-data cases
   (api-2.0.md section 7 item 10); `detail::MakeHostValueInPlace` and
   `HostResult::SetHostObjectInPlace` are frozen ABI.
+- S2b (B3, B6), stacked on #800: `RenderOptions` (`globals` as `shared_ptr<const ValuesMap>`, so
+  a value a callable keeps outlives the render as an environment global does through
+  `LendGlobal`; `poll` as `std::function<void()>`) with `Template::RenderTo(out|sink, params,
+  options)` over the `RenderCore<true>` instantiation; a poll alone makes the render count its
+  steps, and a render a polled one starts (a callable rendering another template) is polled too.
+  `Template::FindReferencedTemplates()` is public, and `test/referenced_templates_test.cpp` now
+  builds in shared mode. Found on the way: in a wide template every referenced name came back as
+  `nullopt` (the walk read narrow strings only); fixed, with a row. The module's poll moved to
+  the `std::function` form of `detail::t_hostPoll`; its render entry is unchanged until S3.
+- S2c (B7, D6b), same PR as S2b: public `jinja2::ISandboxPolicy` (`jinja2cpp/sandbox_policy.h`):
+  `NativeAttribute(const Value&, string_view) -> optional<Value>` and `Format(self, formatMap, params) ->
+  optional<Value>`, nullopt by default for the engine's own `str.format`, which now reads a field's
+  attribute through `NativeAttribute` (Jinja2's `SandboxedFormatter`). Held by `Settings::sandboxPolicy`
+  (identity equality), bridged to the engine's policy, renamed `IEngineSandboxPolicy`
+  (`src/public_sandbox_policy.cpp`). `Settings::immutableContainers` is the public switch for
+  `Dialect::ImmutableSandbox`. `ErrorCode::SecurityError` (21) replaces the `extraParams[1]` marker on an
+  `UndefinedError`; `UndefinedDescription::security` makes and reads a refused attribute's undefined value.
+  0427 decided: a template keeps the policy it was loaded with. Template callables (a macro, `caller`)
+  reach `NativeAttribute` as an empty value, so S3 must keep the module's stand-in for them.
+

docs/tasks/0427-sandbox-policy-and-template-cache.md

@@ -1,5 +1,5 @@
 ---
-status: open
+status: done
 priority: low
 area: api
 depends: [0354, 0357]
@@ -20,3 +20,10 @@ it was loaded with (as it keeps the delimiters). Whichever is chosen, `TemplateE
 Python module's per-load refresh of the public-names flag follow it.
 
 **Done when.** The decision is in `docs/design/0357` and a C++ test pins it.
+
+**Decision (0357 S2c).** A template keeps the policy it was loaded with, as it keeps its delimiters and other
+settings: the public policy is `Settings::sandboxPolicy`, compared by identity in `Settings ==`, so a Load after a
+change gets a fresh `LoadSettings` and the cache's own comparison of settings applies as for any other setting.
+Pinned by `PublicSandboxPolicyTest.ATemplateKeepsThePolicyItWasLoadedWith`. The internal `SetSandboxPolicy` and
+`SetDialect` go with the Python module's switch (0357 S3/S4).
+

include/jinja2cpp/error_info.h

@@ -49,6 +49,7 @@ enum class ErrorCode
     ResourceLimitExceeded = 18,    //!< A limit was passed: one of Settings::Limits, a built-in cap or a host's budget. ExtraParams[0] contains the message, ExtraParams[1] (for Settings::Limits) the name of the limit (`"maxRenderSteps"`)
     TemplateExpired = 19,          //!< A macro of a \ref BasicTemplateModule called a macro an earlier call stored, whose module ended with that call (docs/tasks/0301). ExtraParams[0] contains the macro's name, ExtraParams[1] is "ended". A Load no longer ends a module (docs/tasks/0321). A macro or `caller` a host's call got as an argument and called after that call ended has ExtraParams[0] "lent" (docs/tasks/0300)
     DataReadError = 20,            //!< \ref ToValue could not read a JSON document: its JsonTraits reported an error. ExtraParams[0] contains the message, ExtraParams[1] the JSON path of the value
+    SecurityError = 21,            //!< A sandbox refused an operation (Jinja2's SecurityError): the use of an attribute \ref ISandboxPolicy refused (an undefined value with `security` set, \ref UndefinedDescription), a method that changes a list or dict under Settings::immutableContainers. ExtraParams[0] contains the message
     ExpectedStringLiteral = 1001,  //!< String literal expected
     ExpectedIdentifier = 1002,     //!< Identifier expected
     ExpectedSquareBracket = 1003,  //!< ']' expected

include/jinja2cpp/host_object.h

@@ -327,6 +327,9 @@ struct UndefinedDescription
     bool isAttr = true;
     //! Replaces the message built from the fields above when not empty
     std::string hint;
+    //! A sandbox refused the attribute (Jinja2's `unsafe_undefined`): a use that fails reports
+    //! ErrorCode::SecurityError instead of ErrorCode::UndefinedError (\ref ISandboxPolicy)
+    bool security = false;
 };
 
 /*!

include/jinja2cpp/sandbox_policy.h

@@ -0,0 +1,80 @@
+#ifndef JINJA2CPP_SANDBOX_POLICY_H
+#define JINJA2CPP_SANDBOX_POLICY_H
+
+#include "config.h"
+#include "user_callable.h"
+#include "value.h"
+
+#include <optional>
+#include <string_view>
+
+JINJA2CPP_SUPPRESS_DLL_INTERFACE_WARNINGS_BEGIN
+
+namespace jinja2
+{
+inline namespace v2
+{
+/*!
+ * \brief A host's sandbox for the values of the engine (Jinja2's `SandboxedEnvironment`)
+ *
+ * Set as \ref Settings::sandboxPolicy, it decides what a template may do with the engine's own values: strings,
+ * numbers, lists, dicts, and template objects such as `loop`, a macro or a namespace. A host object (\ref IHostObject)
+ * is never asked about: it applies its own policy in \ref IHostObject::GetAttr and \ref IHostObject::Call. Unset,
+ * the default, a render pays one null check where a policy would be asked.
+ *
+ * The policy is asked:
+ * - for every attribute or method a template reads from a value of the engine by name (`s.upper`, `l.append`,
+ *   `x|attr('n')`, `loop.index`), found or not, before the engine's own answer is used;
+ * - for `str.format`, which it may perform itself (\ref Format).
+ *
+ * Together with \ref Settings::immutableContainers, \ref ErrorCode::SecurityError and the `security` flag of
+ * \ref UndefinedDescription it is what the Python module's `SandboxedEnvironment` and
+ * `ImmutableSandboxedEnvironment` are built on (docs/python.md). A template keeps the policy it was loaded with,
+ * as it keeps its other settings. Called from any thread that renders, so it must be thread-safe.
+ */
+class JINJA2CPP_EXPORT ISandboxPolicy
+{
+public:
+    ISandboxPolicy() = default;
+    ISandboxPolicy(const ISandboxPolicy&) = delete;
+    ISandboxPolicy& operator=(const ISandboxPolicy&) = delete;
+    ISandboxPolicy(ISandboxPolicy&&) = delete;
+    ISandboxPolicy& operator=(ISandboxPolicy&&) = delete;
+    virtual ~ISandboxPolicy();
+
+    /*!
+     * \brief The attribute `name` of `obj`, as the sandbox lets the template read it
+     *
+     * Jinja2's `is_safe_attribute`: return `std::nullopt` to leave the engine's answer (the attribute, the bound
+     * method, or an undefined value when there is none), or the value the template reads instead. A refused name
+     * is usually `MakeUndefined` with `security` set, whose use fails with \ref ErrorCode::SecurityError; a value
+     * the engine lacks (a method of Python's `str`) may be any value, a \ref UserCallable among them.
+     *
+     * @param obj   The value, as a host callable gets it: `loop` and a namespace as a snapshot of their
+     *              attributes, a macro or `caller` as an empty value
+     * @param name  The attribute's name
+     */
+    [[nodiscard]] virtual std::optional<Value> NativeAttribute(const Value& obj, std::string_view name) const = 0;
+
+    /*!
+     * \brief `self.format(*args, **kwargs)`, or `self.format_map(mapping)` when `formatMap`, in the sandbox
+     *
+     * Return the result, or `std::nullopt`, the default, for the engine's own `str.format`, which then reads the
+     * attribute a format field names (`{0.name}`) through \ref NativeAttribute, as Jinja2's `SandboxedFormatter`
+     * does: a refused one formats as its undefined value. The engine asks with `formatMap` false only: its
+     * strings have no `format_map`, so a policy that wants one returns it from \ref NativeAttribute (a callable
+     * that calls `Format(self, true, ...)` itself), as the Python module does.
+     *
+     * @param self       The string
+     * @param formatMap  Whether the call is `format_map`, whose one argument is the mapping
+     * @param params     The call's arguments, as a callable declared with ArgInfo::VarArgs and ArgInfo::VarKwArgs
+     *                   (and UserCallable::keepUndefined) gets them
+     */
+    [[nodiscard]] virtual std::optional<Value> Format(const Value& self, bool formatMap, const UserCallableParams& params) const;
+};
+} // namespace v2
+} // namespace jinja2
+
+JINJA2CPP_SUPPRESS_DLL_INTERFACE_WARNINGS_END
+
+#endif // JINJA2CPP_SANDBOX_POLICY_H