Build a trusted container delivery pipeline
- Advanced
- 30 min
- Published
- Verified
What you'll learn
- Build and push an image with SBOM and
provenance: mode=maxattestations fromdocker/build-push-action - Gate a release on Docker Scout policy with
exit-code, and confirm that a failing image is never signed or tagged - Sign the image by digest with keyless Cosign, using the workflow's GitHub OIDC identity instead of a stored key
- Verify the signature against the exact certificate identity and OIDC issuer, and explain what changes that identity
Your team releases myorg/my-app by running docker push from whichever
laptop is closest. The image works, but when someone asks which commit
1.0 was built from, whether it passed your vulnerability policy, or who
pushed it, nothing in the registry can answer. Anyone with push access could
overwrite the tag tomorrow, and a deployment would pull the new image without
noticing.
A release pipeline fixes this by making one place, a GitHub Actions workflow, the only thing that produces a releasable image. It builds the image, records how it was built, checks it against policy, and signs it. The signature is tied to the workflow’s identity, so a cluster can later require “signed by this workflow, on this branch” and refuse everything else.
In this guide, you write that workflow for myorg/my-app, watch its policy
gate stop a vulnerable build before anything is signed, fix the build, and
then verify the signed release from your own machine.
What you’ll build
One workflow file, .github/workflows/release.yml, that runs on every push to
main:
- Build the image and push it under a commit tag, with an SBOM and max-level provenance attached.
- Analyze it with Docker Scout and list fixable vulnerabilities in the job summary.
- Evaluate policy with Docker Scout, and fail the job if any policy isn’t met.
- Sign the image digest with keyless Cosign, using the workflow’s GitHub OIDC token, and attach a signed SBOM attestation with the same identity.
- Publish the release tag,
1.1, on that same digest.
The order is the point. A release tag only ever lands on an image that passed the gate and carries a signature.
The terminals in this guide are simulations. Click through them to see what each command prints, then run the real commands against your own repository.
Prerequisites
To follow along on your own machine and repository, you need:
- A GitHub repository for the app, with GitHub Actions enabled. This guide
uses
myorg/my-app, default branchmain. - A Docker Hub repository to push to, here
docker.io/myorg/my-app, and a Docker Hub access token with read and write access for the workflow to use. - Access to Docker Hardened Images, which the app’s base image comes from.
The same Docker account signs in to
dhi.io. See Get started with Docker Hardened Images. - Cosign and Docker with Buildx installed locally, to check the results.
- Optionally, the GitHub CLI, for setting secrets and checking runs from the terminal. The GitHub web UI works too.
You don’t need to have signed an image before. If you want to understand signatures and digests first, see Prove where your container image came from.
The starting point
The app is a small Express API in ~/my-app, with package.json,
package-lock.json, and index.js. It’s built from a Docker Hardened Image
in two stages:
# syntax=docker/dockerfile:1
FROM dhi.io/node:22-alpine3.23-dev AS builder
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
FROM dhi.io/node:22-alpine3.23
ENV PATH=/app/node_modules/.bin:$PATH
COPY --from=builder --chown=node:node /usr/src/app /app
WORKDIR /app
CMD ["node", "index.js"]To see why this base image is a good starting point, see Build your application on a trusted foundation.
The current release, 1.0, was pushed by hand. Ask Cosign whether the
release workflow signed it. A keyless signature is checked against two
values: the identity in the signing certificate and the OIDC issuer that
vouched for it. Set both once, so the commands that follow stay readable:
~/my-app
export IDENTITY=https://github.com/myorg/my-app/.github/workflows/release.yml@refs/heads/main
export ISSUER=https://token.actions.githubusercontent.com
cosign verify docker.io/myorg/my-app@sha256:9d2e6bf06421cd8681531046a17c390561e0cc2c6ee42bb6d6f56d66e9b2eb6c --certificate-identity $IDENTITY --certificate-oidc-issuer $ISSUER
Error: no signatures found error during command execution: no signatures found
The command verifies by digest, not by the 1.0 tag, because a tag can be
moved to a different image at any time. Nothing has signed this digest, so
there is nothing to verify. That’s the gap the workflow closes.
Give the workflow registry access
The workflow pushes to Docker Hub and pulls the base image from dhi.io, so
it needs a Docker username and access token. Store the username as a
repository variable and the token as a secret, so the token never appears in
the workflow file or its logs:
~/my-app
gh variable set DOCKERHUB_USERNAME --body myorg-ci
✓ Created variable DOCKERHUB_USERNAME for myorg/my-app
gh secret set DOCKERHUB_TOKEN
✓ Set Actions secret DOCKERHUB_TOKEN for myorg/my-app
Notice what the workflow doesn’t need: a signing key. Keyless signing gets a short-lived certificate from Sigstore at signing time, so there’s no private key to store, rotate, or leak.
Write the release workflow
Create .github/workflows/release.yml. The file name and the branch matter.
They become part of the signing identity, as you’ll see when you verify the
signature.
name: release
on:
push:
branches: [main]
permissions:
contents: read
id-token: write
env:
IMAGE: docker.io/myorg/my-app
VERSION: "1.1"
jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Check out the repository
uses: actions/checkout@v6
- name: Log in to Docker Hub
uses: docker/login-action@v4
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to Docker Hardened Images
uses: docker/login-action@v4
with:
registry: dhi.io
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Tag the candidate with the commit
id: meta
uses: docker/metadata-action@v6
with:
images: ${{ env.IMAGE }}
tags: type=sha
# 1. Build, attest, and push the candidate
- name: Build and push
id: build
uses: docker/build-push-action@v7
with:
push: true
sbom: true
provenance: mode=max
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# 2. Analyze: report fixable vulnerabilities in the job summary
- name: Analyze vulnerabilities
uses: docker/scout-action@v1.26.0
with:
command: cves
image: registry://${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
only-severities: critical,high
only-fixed: true
# 3. Gate: fail the job if any policy isn't met
- name: Evaluate policy
uses: docker/scout-action@v1.26.0
with:
command: policy
image: registry://${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
policy-config: policies.json
exit-code: true
# 4. Sign the digest with the workflow's OIDC identity
- name: Install Cosign
uses: sigstore/cosign-installer@v4.1.2
with:
cosign-release: v3.1.3
- name: Sign the image
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: cosign sign --yes "${IMAGE}@${DIGEST}"
- name: Attest the SBOM
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: |
docker buildx imagetools inspect "${IMAGE}@${DIGEST}" \
--format '{{ json .SBOM.SPDX }}' > sbom.spdx.json
cosign attest --yes --type spdxjson \
--predicate sbom.spdx.json "${IMAGE}@${DIGEST}"
- name: Verify the signature
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: >
cosign verify "${IMAGE}@${DIGEST}"
--certificate-identity "https://github.com/myorg/my-app/.github/workflows/release.yml@refs/heads/main"
--certificate-oidc-issuer "https://token.actions.githubusercontent.com"
# 5. Publish: point the release tag at the signed digest
- name: Publish the release tag
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: docker buildx imagetools create --tag "${IMAGE}:${VERSION}" "${IMAGE}@${DIGEST}"Next to the workflow, create policies.json. The policy step reads it from
the checked-out repository to adjust Scout’s built-in policies:
{
"policies": [
{
"name": "copyleft-license",
"enabled": false
}
]
}The rest of this section walks through what each part establishes.
Permissions: id-token: write
id-token: write lets the job request an OIDC token from GitHub. That token
is the workflow’s proof of who it is, and Cosign exchanges it for a signing
certificate. Without this permission, the sign step fails, because the job
has no identity to sign with. contents: read is all the job needs to check
out the code. Granting nothing else keeps the token GitHub gives the job as
narrow as possible.
Build: attestations come from the build
The build step pushes the image under a commit tag such as sha-c765d4f,
not under 1.1. That’s a candidate, not a release.
sbom: true and provenance: mode=max make BuildKit generate the
attestations while it builds, and store them with the image in the registry.
They can’t be added at the end, because only the build knows what went into
the image. Max-level provenance records the build’s materials, such as the
base image digests, the source repository and commit, and the build
parameters.
docker/build-push-action adds provenance by default, but only at
mode=min for a private repository, so setting mode=max explicitly gives
you the same evidence either way. Attestations need the image pushed
directly to a registry, which is why this step uses push: true rather than
loading the image into the runner.
The step’s digest output, steps.build.outputs.digest, is the content
address of exactly what was pushed. Every step after this one refers to the
image by that digest, never by a tag, so they all operate on the same
artifact even if someone moves a tag in the meantime.
Analyze and gate: a report and a decision
The two Scout steps look similar, but they do different jobs. cves with
only-fixed: true lists the critical and high vulnerabilities that have a
fix, so the person reading the job summary knows what to do. It reports and
doesn’t fail.
policy evaluates the image against Docker Scout’s policies, such as no
fixable critical or high vulnerabilities, supply chain attestations present,
and a non-root default user. With exit-code: true, an unmet policy fails
the step, and GitHub Actions skips every step after it. That’s the gate:
signing and publishing never run.
The registry:// prefix tells Scout to read the image and its attestations
from the registry rather than looking for a local copy.
This workflow uses Scout’s built-in policies, adjusted by policies.json.
The built-in No copyleft licenses policy flags every package under a GPL,
LGPL, or MPL license, and Alpine packages in the hardened base image, such as
libgcc, libstdc++, and alpine-baselayout-data, carry those licenses.
Every image on this base fails it, so policies.json turns that policy off
and the gate decides on vulnerabilities, attestations, and the base image.
Tuning and writing policies is covered in
Enforce your organization’s container standards.
The step pins docker/scout-action@v1.26.0 rather than @v1, because the
v1 tag still points to release 1.20.4, which refuses command: policy
without an organization input. The action passes policy-config through
to Scout even though its action.yml doesn’t declare the input, so GitHub
Actions may warn about an unexpected input on this step.
On a CI system other than GitHub Actions, the same gate is
docker scout policy <image> --policy-config policies.json --exit-code,
which exits with code 2 when a policy isn’t met.
Sign: keyless, by digest
sigstore/cosign-installer installs Cosign and verifies the release it
downloads. cosign sign without --key signs keylessly:
- Cosign generates a short-lived key pair in memory.
- It sends the job’s GitHub OIDC token to Sigstore’s certificate authority, Fulcio, which issues a certificate for that key. The certificate records who the token says the signer is: this workflow file, in this repository, on this ref.
- It signs the image digest, records the signature in Sigstore’s public transparency log, Rekor, and pushes the signature to the registry next to the image.
The key is discarded when the job ends. --yes accepts Sigstore’s prompt
about recording the signature in a public log, which a CI job can’t answer.
Signing "${IMAGE}@${DIGEST}" rather than a tag matters for the same reason
as everywhere else: the signature covers that digest, and only that digest.
Attest: a signed SBOM
The SBOM and provenance BuildKit generated are stored in the image index, so
the signature on the index digest covers them. They aren’t signed
attestations in their own right, though, and tools that check attestations
at deploy time, such as Kyverno, look for in-toto attestations signed with
Cosign. The Attest the SBOM step closes that gap. imagetools inspect
extracts the SPDX SBOM BuildKit generated, and cosign attest --type spdxjson
wraps it in an in-toto statement with the predicate type
https://spdx.dev/Document, signs it keylessly with the same workflow
identity, and attaches it to the same digest.
The verify step after it is a self-check. If the identity the workflow signs with ever drifts from the one your cluster expects, for example because someone renamed the file, the release fails here instead of in production.
Publish: tag the signed digest
docker buildx imagetools create adds the 1.1 tag to the image already in
the registry, without rebuilding. Because the source is a single image index,
the command makes an exact copy, so the digest doesn’t change, and the
signature and attestations attached to that digest apply to 1.1 as well.
Bump VERSION when you cut the next release.
Watch the gate stop a release
Commit the workflow and push it to main. Merging a pull request into
main triggers it the same way.
~/my-app
git add .github/workflows/release.yml policies.json
git commit -m "Add the release workflow"
[main ca70089] Add the release workflow 2 files changed, 114 insertions(+) create mode 100644 .github/workflows/release.yml create mode 100644 policies.json
git push origin main
To https://github.com/myorg/my-app.git 85e3794..ca70089 main -> main
gh run list --workflow release.yml --limit 1
STATUS TITLE WORKFLOW BRANCH EVENT ID ELAPSED AGE X Add the release workflow release main push 18240561273 1m48s about 1 minute ago
The run failed. Open it, and the Evaluate policy step shows why. This is an excerpt of that step’s log:
Policy status FAILED (5/6 policies met)
Health score B (76%!)(string=)
Status │ Policy │ Results
────────┼────────────────────────────────────────────────┼─────────────────────────────
✓ │ Default non-root user │
! │ Fixable critical or high vulnerabilities found │ 0C 4H 0M 0L
✓ │ No high-profile vulnerabilities │ 0C 0H 0M 0L
✓ │ No outdated base images │
✓ │ No unapproved base images │ 0 deviations
✓ │ Supply chain attestations │ 0 deviationsThis failure is expected. The hardened base image is fine. The four high
vulnerabilities are in npm packages the app’s lockfile pins: express 4.19.2
pulls in versions of body-parser and path-to-regexp that have known fixes.
The Analyze vulnerabilities step’s job summary lists each CVE and its
fixed version. The %!)(string=) after the health score is a formatting bug
in the action’s output, not part of the result. The signing and publishing
steps show as skipped.
The workflow did what it’s for. Check that nothing releasable came out of
it. The candidate was pushed, so it’s in the registry, but it has no
signature, and 1.1 doesn’t exist:
~/my-app
cosign verify docker.io/myorg/my-app@sha256:3b1fd2bf606f10f0759a25dd638452fce8d07b45edaaab8d5a50ef321006ca91 --certificate-identity $IDENTITY --certificate-oidc-issuer $ISSUER
Error: no signatures found error during command execution: no signatures found
docker buildx imagetools inspect docker.io/myorg/my-app:1.1
ERROR: docker.io/myorg/my-app:1.1: not found
An unsigned candidate in the registry is harmless as long as the thing that deploys images requires a signature. That’s the job of admission control, covered in Only run trusted images in production.
Fix the build and release
Update express to a version whose dependencies include the fixes, then
commit the new lockfile. For how to find the right version and confirm a fix
locally before you push, see
Find and fix vulnerabilities before you ship.
~/my-app
npm install express@4.22.3
changed 13 packages, and audited 69 packages in 2s 16 packages are looking for funding run `npm fund` for details found 0 vulnerabilities
git commit -am "Update express to 4.22.3"
[main c765d4f] Update express to 4.22.3 2 files changed, 97 insertions(+), 85 deletions(-)
git push origin main
To https://github.com/myorg/my-app.git ca70089..c765d4f main -> main
gh run list --workflow release.yml --limit 1
STATUS TITLE WORKFLOW BRANCH EVENT ID ELAPSED AGE ✓ Update express to 4.22.3 release main push 18240790415 2m09s about 1 minute ago
This time every policy passes, so the job goes on to sign the digest, verify
the signature, and publish 1.1.
Verify the release
Check the result the way a consumer would: from outside the pipeline, with nothing but the image reference and the expected identity.
First, resolve 1.1 to its digest, and confirm the attestations are there:
~/my-app
docker buildx imagetools inspect docker.io/myorg/my-app:1.1
Name: docker.io/myorg/my-app:1.1
MediaType: application/vnd.oci.image.index.v1+json
Digest: sha256:4f7c9e691546946bebacce3f32f6b78b49ee74d60f918976ee6c4682e8d99dc2
Manifests:
Name: docker.io/myorg/my-app:1.1@sha256:7a0c5505b7c245714462f06877d937b0c2a7ea0ca22f51e13ad24e964d04ac59
MediaType: application/vnd.oci.image.manifest.v1+json
Platform: linux/amd64
Name: docker.io/myorg/my-app:1.1@sha256:e83d14ee91ded25195b4ec6140096806802e3c440140d24223752ccbaf960809
MediaType: application/vnd.oci.image.manifest.v1+json
Platform: unknown/unknown
Annotations:
vnd.docker.reference.digest: sha256:7a0c5505b7c245714462f06877d937b0c2a7ea0ca22f51e13ad24e964d04ac59
vnd.docker.reference.type: attestation-manifestThe second manifest, with the attestation-manifest reference type, holds
the SBOM and provenance for the linux/amd64 image. The index digest,
sha256:4f7c9e…, is the same one the build step pushed as sha-c765d4f,
because publishing copied it rather than rebuilding.
Now verify that digest’s signature against the release workflow’s identity:
~/my-app
cosign verify docker.io/myorg/my-app@sha256:4f7c9e691546946bebacce3f32f6b78b49ee74d60f918976ee6c4682e8d99dc2 --certificate-identity $IDENTITY --certificate-oidc-issuer $ISSUER
Verification for index.docker.io/myorg/my-app@sha256:4f7c9e691546946bebacce3f32f6b78b49ee74d60f918976ee6c4682e8d99dc2 -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates
Cosign checked three things: the signature covers this digest, the transparency log has a record of it, and the certificate chains to Sigstore’s certificate authority and names the identity and issuer you asked for. You didn’t need a public key, because the certificate is the key’s credential.
To see what the identity check buys you, ask for a signature from a
different branch. A workflow run on dev, or on someone’s fork, signs with a
different identity:
~/my-app
cosign verify docker.io/myorg/my-app@sha256:4f7c9e691546946bebacce3f32f6b78b49ee74d60f918976ee6c4682e8d99dc2 --certificate-identity https://github.com/myorg/my-app/.github/workflows/release.yml@refs/heads/dev --certificate-oidc-issuer $ISSUER
Error: no matching signatures: none of the expected identities matched what was in the certificate, got subjects [https://github.com/myorg/my-app/.github/workflows/release.yml@refs/heads/main] with issuer https://token.actions.githubusercontent.com
The signature is valid, but it isn’t from the identity you required, so verification fails. The error also shows you the identity the certificate actually carries, which is a handy way to find the exact string when you write a verification policy.
Understand the signing identity
The certificate behind each signature names two things, and anything that verifies your images, including an admission policy, has to match both:
| Field | Value for this workflow |
|---|---|
| Certificate identity | https://github.com/myorg/my-app/.github/workflows/release.yml@refs/heads/main |
| OIDC issuer | https://token.actions.githubusercontent.com |
The issuer says GitHub Actions vouched for the signer. The identity says which workflow: the repository, the path of the workflow file, and the Git ref the run was triggered on. Fulcio copies them from the OIDC token GitHub issues to the job, so a job can’t choose its own.
That makes the identity sensitive to changes you might not think of as security changes:
- Renaming or moving the workflow file changes the path, and so the identity.
- Triggering on a tag instead of a branch gives
...release.yml@refs/tags/v1.1, a different identity per release. If you release from tags, verify with--certificate-identity-regexpand a tightly anchored pattern rather than a fixed string. - Moving the steps into a reusable workflow makes the certificate name the reusable workflow’s file, not the caller’s.
- Running from a fork or another branch produces a different identity, as the failed verification above showed.
Each of these is a deliberate change for whoever maintains the verification policy, which is the point: only one workflow on one branch produces signatures that verify.
What the signature does and doesn’t prove
A valid signature proves that this release workflow, running on main,
signed this exact digest, and that the digest passed your policy gate on the
way, because the workflow can’t reach the sign step otherwise. It doesn’t
prove the code is good. Anyone who can push to main can get a commit
signed, so protect the branch with required reviews and status checks. The
gate is also only as strict as the policies it evaluates, and only as current
as the vulnerability data at build time. A CVE disclosed next week won’t
unsign this image.
Where to go next
The pipeline now produces images that carry their own evidence: an SBOM, a provenance record, and a signature and signed SBOM attestation tied to one workflow identity. Nothing enforces it yet. A cluster still runs an unsigned image if you hand it one.
Only run trusted images in production
adds a Kyverno admission policy that requires a signature and a signed SBOM
attestation from exactly the identity and issuer in this guide, so the production namespace refuses
everything else.
For reference depth on the pieces used here, see Docker Docs on attestations in GitHub Actions and evaluating Scout policies, and Sigstore’s OIDC identities in Fulcio.
Part of a learning path
This works on its own and is also included in the paths below. Open a path to see the complete route.
Learning path
Secure your software supply chain
Step 6 of 7
- Step 1: Build your application on a trusted foundation
- Step 2: Find and fix vulnerabilities before you ship
- Step 3: Document vulnerabilities that don't affect your application
- Step 4: Enforce your organization's container standards
- Step 5: Prove where your container image came from
- Step 6: Build a trusted container delivery pipeline (this page)
- Step 7: Only run trusted images in production
Table of contents