Repository navigation
gantry acr artifact streaming - #836
Draft
Vaibhav Patel (vpatelsj) wants to merge 51 commits into
Draft
Vaibhav Patel (vpatelsj) wants to merge 51 commits into
Vaibhav Patel (vpatelsj) wants to merge 51 commits into
Conversation
Vaibhav Patel (vpatelsj)
changed the title
Vaibhav Patel (vpatelsj)
force-pushed
the
copilot/gantry-acr-artifact-streaming
branch
from
September 25, 2026 12:06
3d5f410 to
92842e1
Compare
Six agent knobs were duplicated across the config struct, defaults, env vars, CLI flags, validation, chart values, JSON schema, and the rendered ConfigMap. Four of them were tuning values an operator cannot evaluate from the outside, so keep only the feature gate and the signed-origin host allowlist and move the rest into internal/gantry/streaming beside the behavior they govern. NewOriginClient no longer takes tuning parameters and cannot fail, and streaming.Options now falls back to defaults instead of rejecting a zero value, which removes an error path from agent startup. Also stop writing logConfig.logLevel on the host. The configurator owns p2pConfig only; the test now pins that unrelated OverlayBD settings are left exactly as the host had them.
The live streaming e2e sent sig=redacted, so the plan's secret-free logs contract passed without testing anything. Use a sentinel SAS value and assert it is absent from every Gantry pod's logs and metrics after the request has demonstrably transited the agent.
Add a stale-provider e2e: remove the blob from the provider's containerd while its DHT record survives, then assert the next range fails over to the signed origin inside one request and the peer serves nothing more. libp2p has no protocol-level withdraw, so this is the real production shape rather than a simulated error. Make the test origin reject any request whose URI did not arrive byte-for-byte and give it the ACR data-path form, doubled slash and percent-encoded query included. Raw signed-URL preservation through the real listener was previously only covered by a unit test with a hand-set RequestURI. Factor the shared cluster setup into a fixture so the two scenarios do not duplicate it. Document the operations surface the plan required and was missing: metrics reference, four alert rules separating Gantry, ACR, peer-reuse and allowlist failures, an incident table, and a compatibility matrix naming who owns each component.
ACR exposes the Artifact Streaming manifest under a different digest that control-plane tag metadata never reports. Resolve it through the node-local AKS helper instead, which returns Docker-Content-Digest.
Selectable() ignored lease expiry, so dead holders stayed rankable and kept SelectableCount() at the full holder count. That satisfied the early-return in attemptClaim, so reclaimableChairs never ran and the stale set was self-sustaining. Rank and the counts now treat a chair as usable only when it is within the abandon threshold, measured against an injectable store clock.
The blob serve cap exists to preserve bandwidth per large-layer stream. Artifact Streaming index reads inherited it accidentally: a few KiB on the container-start path queued behind 1 GiB transfers. Small bounded ranges now take a separate semaphore, and only ranges with both bounds present qualify so an unbounded transfer cannot claim the lane. Range fetches also get their own h2 transport, since sharing one connection with bulk streams put them behind the same flow-control window and idle-ping fate.
PrefetchManifestChildren only dispatched PleasePull to ranked chairs, so a node that served a manifest never fetched the layers itself. Under Artifact Streaming containerd never requests the blobs either, so the per-node Resolve path never fired and coverage stalled at a few replicas per layer instead of one per node. OnManifestServed now seeds every child into the local content store, preferring peers and falling back to origin. Readiness no longer gates on chair availability: a node with no selectable chair can still serve and seed, and gating kept it out of rotation.
Every node seeds every layer, so an origin fallback turns one cold image into nodes x layers concurrent origin pulls. At 1000 nodes that was up to 4000 concurrent 1 GiB pulls through the node-local ACR helper, which starved the OverlayBD range reads that gate container start: 174 GB left origin and not one layer ever committed. Seeding is now peer-only by default and polls the swarm for longer instead of giving up after ten minutes. Bootstrapping a cold image stays the chair prefetch path's job, which pulls from a small designated set of nodes. Also expose gantry_layer_seed_total through the benchmark scrape allowlist, which silently dropped it before.
The containerd content store is all-or-nothing, so a peer could not serve any part of a 1 GiB layer until it held the whole thing. Under Artifact Streaming nothing is ever complete during a cold start, which left every node reading OverlayBD indexes straight from origin: 1000 nodes x up to 32 concurrent origin reads each, against a registry that answers 429. Ranges fetched from a peer or origin are now kept in a disk-backed cache and served back out of the transfer endpoint even when the blob itself is absent, and the node announces the digest so the swarm can find it. Only reads up to MaxCachedRangeBytes are admitted, so bulk layer traffic cannot evict the small index reads that actually gate container start.
The advertiser proves a digest is serveable by opening it in the content store and reconciles announcements against that same inventory, so a node holding only cached ranges was invisible: Notify refused to announce it and reconcile would have withdrawn it anyway. Peers could never discover a range source, which left the range cache purely local and the origin stampede intact.
Measured on a node after a successful run: the range cache held 40 layers, 41 GB apparent but only 52 MB allocated. Probing a 1.08 GB layer shows the only bytes a container touches are the head and the last ~1 MB - 0.13% of the image. Everything in between is never read at mount time. Fetching those on demand makes the first CreateContainer fail, and the pod then waits out kubelet backoff, which is what put a ~130s floor under startup with a min of 130s and a max of 178s - a fixed cost, not contention. Serving a manifest now warms both regions for every layer, peer-first and jittered so a cold image does not put the whole fleet on origin at once.
p90 startup was 15s but p95 was 44s, which is the shape of a minority of nodes starting before their index warm finished. Halve the spread and double the per-node concurrency; origin only carried 1.72 GB across the fleet, so there is ample headroom.
Measured at 1000 nodes. Concurrency 16 with a 500ms spread against concurrency 8 with 2s: p90 42s vs 15s, max 80s vs 50s, job window 103s vs 70s, p50 unchanged at 11s vs 10s. Warming harder just makes nodes contend, so restore the settings that produced the better distribution. One sample per configuration, so treat the p50 difference as noise; the p90 and window gaps are large enough to act on.
Vaibhav Patel (vpatelsj)
force-pushed
the
copilot/gantry-acr-artifact-streaming
branch
from
October 16, 2026 10:07
acbf8fb to
7e68855
Compare
…tifact-streaming # Conflicts: # .github/workflows/nightly.yaml # .github/workflows/release.yaml # docs/content/guides/gantry.md
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.