Repository navigation
Pin loop definitions and carry region coordinates through task execution - #74345
Merged
Merged
Conversation
ashb
added this pull request to stack #74339
October 6, 2026 15:06
kaxil
reviewed
Oct 6, 2026
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 7, 2026 13:57
3b98054 to
adce91a
Compare
ashb
removed this pull request from stack #74339
October 7, 2026 15:20
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 7, 2026 15:22
adce91a to
45978c7
Compare
ashb
added this pull request to stack #74410
October 7, 2026 15:22
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 7, 2026 20:57
45978c7 to
6829d3c
Compare
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 8, 2026 13:48
8b8d409 to
94a81e0
Compare
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 8, 2026 13:53
94a81e0 to
7774d1b
Compare
ashb
marked this pull request as ready for review
October 8, 2026 13:58
ashb
requested review from
XD-DENG,
bolkedebruin,
o-nikolas and
pierrejeambrun
as code owners
October 8, 2026 13:58
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 8, 2026 16:06
7774d1b to
07542b1
Compare
kaxil
approved these changes
Oct 8, 2026
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 8, 2026 20:45
07542b1 to
fb403e9
Compare
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 9, 2026 10:32
fb403e9 to
4741eef
Compare
ashb
force-pushed
the
task-loops-stack-4
branch
2 times, most recently
from
October 9, 2026 16:28
2f7bed1 to
ca94467
Compare
ashb
force-pushed
the
task-loops-stack-4
branch
2 times, most recently
from
October 9, 2026 20:46
15795c7 to
b3f9aa1
Compare
ashb
force-pushed
the
task-loops-stack-4
branch
2 times, most recently
from
October 10, 2026 07:05
b93e57d to
fc3aa41
Compare
A task in a loop or mapped region is identified by (region_id, region_index), but running code, the Execution API and log readers still address it by the public map_index. If one value served both, a loop pass would look like a mapped expansion. We derive the public index from the Dag definition pinned to the try instead of storing it: a task exposes its region index as map_index only when its pinned definition expands, and -1 otherwise. Ordinary mapped tasks therefore keep reporting exactly what they did before. The scheduler and API server must understand loops without importing user code, so the serialized Dag carries a loop group definition they can read. The Execution API and the supervisor schema take the region fields as optional, versioned fields so that older task SDKs keep working against a newer API server. Lookups of earlier runs refuse loop producers, because an iteration of one run has no counterpart to resolve in another run. A log path and a remote log id have to come out the same for the process that writes the log and for every reader that fetches it later, retired attempts included, and one loop pass must not overwrite the logs of the pass before it. Building both from one prepared, batched context gives workers, the triggerer and log readers the same answer without a query per task. Only loop tasks gain a pass position; ordinary mapped tasks keep their existing path, so logs already written stay readable.
ashb
force-pushed
the
task-loops-stack-4
branch
from
October 10, 2026 16:18
fc3aa41 to
2fd9c2a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A task in a loop or mapped region is identified by (region_id, region_index),
but running code, the Execution API and log readers still address it by the
public map_index. If one value served both, a loop pass would look like a mapped
expansion. We derive the public index from the Dag definition pinned to the try
instead of storing it: a task exposes its region index as map_index only when
its pinned definition expands, and -1 otherwise. Ordinary mapped tasks therefore
keep reporting exactly what they did before.
The scheduler and API server must understand loops without importing user code,
so the serialized Dag carries a loop group definition they can read.
The Execution API and the supervisor schema take the region fields as optional,
versioned fields so that older task SDKs keep working against a newer API
server. Lookups of earlier runs refuse loop producers, because an iteration of
one run has no counterpart to resolve in another run.
A log path and a remote log id have to come out the same for the process that
writes the log and for every reader that fetches it later, retired attempts
included, and one loop pass must not overwrite the logs of the pass before it.
Building both from one prepared, batched context gives workers, the triggerer
and log readers the same answer without a query per task. Only loop tasks gain
a pass position; ordinary mapped tasks keep their existing path, so logs already
written stay readable.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.