Skip to content

Sign the final release provenance - #4550

Merged
kubernetes-prow[bot] merged 1 commit into
kubernetes:masterfrom
saschagrunert:sign-final-provenance
Oct 30, 2026
Merged

kubernetes-prow[bot] merged 1 commit into
kubernetes:masterfrom
saschagrunert:sign-final-provenance

Conversation

@saschagrunert

@saschagrunert saschagrunert commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

What type of PR is this?

/kind feature

What this PR does / why we need it:

Signs the final release provenance in place, like the stage run signs its provenance since #4507. krel release generates provenance-<version>.json per version and copies it to the release bucket, but the release build runs krel sign blobs before that step, so the published provenance.json has never been signed.

PushArtifacts now signs the final attestations first, as the account named by GOOGLE_SERVICE_ACCOUNT_NAME (krel-staging, which the release step already sets for the image and blob signatures), so a signing failure stops the release before anything is published. Without that variable the release fails instead of signing with other credentials, which the documented verify command would reject. Signing inside krel rather than in a later Cloud Build step keeps the plain statement from ever being public, and krel knows the versions of a release. A version without a final attestation fails official releases (--nomock), so that they never publish without provenance; mock releases skip it with a warning. krel sign attestation --in-place and this share the new Signer.SignFilesInPlace and Signer.WriteBundleFile, and the krel release help says that local runs need GOOGLE_SERVICE_ACCOUNT_NAME too.

https://dl.k8s.io/release/<version>/provenance.json becomes a sigstore bundle instead of the plain statement of v1.37.0 and older. It verifies with:

cosign verify-blob-attestation --bundle provenance.json --type slsaprovenance1 \
  --certificate-identity krel-staging@k8s-releng-prod.iam.gserviceaccount.com \
  --certificate-oidc-issuer https://accounts.google.com kubectl

Which issue(s) this PR fixes:

Part of #2616

Special notes for your reviewer:

fake_release_impl.go is regenerated with counterfeiter v6.12.2, the version of the other fakes, so it only changes the signing method. A signer factory on defaultReleaseImpl lets the unit test cover the success path, the paths passed and the signing account.

The verify command was checked against a staged bundle signed as krel-staging. The "Verify Signed Kubernetes Artifacts" page on kubernetes.io should get it once this ships.

Tested end to end with a mock stage (Cloud Build fc5b9edb in k8s-release) and a mock release (f2b2aac0) of master, with krel built from a branch that combines #4555, #4550 and #4551: the stage attached the signed image provenance to 54 mock images (cosign verify-attestation passes) and signed provenance.json in place; the release verified that stage provenance and all 451 staged artifacts, signed the final provenance in place before publishing, and the published provenance.json verifies against the released kubectl with cosign verify-blob-attestation.

Does this PR introduce a user-facing change?

The provenance attestation published with each Kubernetes release (`provenance.json` on dl.k8s.io) is now a sigstore bundle signed as `krel-staging@k8s-releng-prod.iam.gserviceaccount.com` instead of an unsigned in-toto statement.

@kubernetes-prow kubernetes-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority labels Oct 7, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from jimangel and jrsapi October 7, 2026 07:06
@kubernetes-prow kubernetes-prow Bot added area/release-eng Issues or PRs related to the Release Engineering subproject sig/release Categorizes an issue or PR as relevant to SIG Release. approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 7, 2026
@saschagrunert
saschagrunert force-pushed the sign-final-provenance branch 2 times, most recently from 6fa8e31 to 15844dc Compare October 7, 2026 07:14
@kubernetes-prow kubernetes-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 7, 2026
@saschagrunert
saschagrunert force-pushed the sign-final-provenance branch from 15844dc to fb78017 Compare October 7, 2026 07:14
@saschagrunert
saschagrunert force-pushed the sign-final-provenance branch 2 times, most recently from e0dc4ba to 6268957 Compare October 8, 2026 10:13
@kubernetes-prow kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 12, 2026
@kubernetes-prow kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 13, 2026
@kubernetes-prow kubernetes-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 13, 2026
@saschagrunert saschagrunert changed the title WIP: Sign the final release provenance Sign the final release provenance Oct 13, 2026
@kubernetes-prow kubernetes-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 13, 2026
@saschagrunert

Copy link
Copy Markdown
Member Author

@kubernetes/release-engineering PTAL

@saschagrunert

Copy link
Copy Markdown
Member Author

@kubernetes/release-engineering PTAL

krel release generated the final provenance attestations and published
them unsigned, since the release job signs the artifacts before krel
release writes them. They are now signed in place like the stage run
signs its provenance, as the account named by
GOOGLE_SERVICE_ACCOUNT_NAME, which the release job impersonates to sign
the artifacts, before anything of the release is published, so that a
signing failure stops it. Without that account the release fails
instead of signing with other credentials. A version without a final
attestation fails official releases, mock releases skip it with a
warning, like they only warn about a stage provenance that doesn't
check. The published provenance.json becomes a sigstore bundle. The
in-place signing of the krel sign attestation command moves into the
attestation package, so that both use it.

Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
Comment thread pkg/anago/release.go
// warning, like mock releases only warn when the stage provenance doesn't
// check.
func (d *defaultReleaseImpl) SignProvenance(versions []string, required bool) error {
account := os.Getenv("GOOGLE_SERVICE_ACCOUNT_NAME")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: SignProvenance reads GOOGLE_SERVICE_ACCOUNT_NAME directly, while everything else in the impl comes from d.options. No other Go code in the repo reads this variable; it only appears in the krel release help text. Could we add it to ReleaseOptions (for example as a --signing-account flag that defaults to the env var)? Then the account would be visible in the options and easier to set in tests. Fine as a follow-up.

@kubernetes-prow kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 30, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: saschagrunert, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow
kubernetes-prow Bot merged commit fa71cc8 into kubernetes:master Oct 30, 2026
11 checks passed
@kubernetes-prow kubernetes-prow Bot added this to the v1.38 milestone Oct 30, 2026
@saschagrunert
saschagrunert deleted the sign-final-provenance branch October 30, 2026 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/release-eng Issues or PRs related to the Release Engineering subproject cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/release Categorizes an issue or PR as relevant to SIG Release. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants