Signing Docker Images with Notary V2 in Azure Pipelines

Let's learn how to sign Docker images using Notary V2 in Azure Pipelines.

Tools

Env

First, we'll set some helpful environment variables to make the rest of the tutorial easier to follow.

export RESOURCE_GROUP=notaryexport 
export ACR_NAME=xeolexampleregistryexport 
export AKV_NAME=xeolexamplekeyvault

And then setup some variables for our certificate and private key.

export CERT_SUBJECT="CN=wabbit-networks.io,O=Notary,L=Seattle,ST=WA,C=US"
export KEY_NAME=mysigningkey
export CERT_PATH=$KEY_NAME.pem

1.Setup the Infrastructure

We'll setup an Azure Container Registry, an Azure Key Vault, and then allow our Azure pipelines to have access to both of them.

1.1 Create the Container Registry

# (OPTIONAL): create a resource group
az group create --name $RESOURCE_GROUP --location eastus

# create the container registry
acr create --resource-group $RESOURCE_GROUP --name $ACR_NAME --sku Premium

1.2 Create the Key Vault

This will create an Azure Key Vault where we'll store our signing certificate and private key.

az keyvault create --name $AKV_NAME --resource-group $RESOURCE_GROUP --location eastus

2.Create the Signing Certificate

We need to create an x509 certificate and private key to sign and verify our images. Here we create a self-signed certificate, but if you have an existing certificate you can also upload it to Azure Key Vault.

This is a certificate policy that is used to create the certificate and private key with Azure Key Vault.

cat < ./cert_policy.json
    {
       "issuerParameters": {
       "certificateTransparency": null,
       "name": "Self"
       },
       "x509CertificateProperties": {
       "ekus": [
           "1.3.6.1.5.5.7.3.3"
       ],
       "keyUsage": [
           "digitalSignature"
       ],
       "subject": "$CERT_SUBJECT",
       "validityInMonths": 12
       }
    }
    EOF

The EKU listed here (1.3.6.1.5.5.7.3.3) is an OID that specifies this certificate is for code signing. (ref). The subject is used later as trust identity when verifying the signature.

Now, we can use the policy to create the certificate and private key

az keyvault certificate create -n $KEY_NAME --vault-name $AKV_NAME -p @cert_policy.json

Get the links to the certificate and private key in order to use them later.

az keyvault certificate show -n $KEY_NAME --vault-name $AKV_NAME --query 'id' -o tsv
az keyvault certificate show -n $KEY_NAME --vault-name $AKV_NAME --query 'kid' -o tsv

Remember to note these values for later.

3.Setup pipelines access to the Key Vault and Container Registry

We'll now setup our Azure DevOps pipelines to have access to the Key Vault and Container Registry.

3.1. Setup Azure Key Vault Service Principal in ADO

- Visit Project Settings for your Azure DevOps project

- Click Service Connections and then New Service Connection and select Azure Resource Manager

- Select Service Principal (automatic) and then click Next.

- Select your Azure subscription and resource group, and then click Save.

- Set the name of the service connection to AzureServiceConnection and click Save.

- Click on the newly created service connection and then click Manage Service Principal and note

the Application (client) ID

export APPLICATION_ID=\

3.2 Setup Azure Key Vault RBAC

We need to give our Azure DevOps service connection access to the Key Vault.

export KEY_VAULT_ID=$(az keyvault show --name $AKV_NAME --query id -o tsv)
az role assignment create --role "Key Vault Crypto User" --assignee $APPLICATION_ID --scope $KEY_VAULT_ID
az role assignment create --role "Key Vault Reader" --assignee $APPLICATION_ID --scope $KEY_VAULT_ID

These roles will allow the service principal to read and sign with the certificate and private key.

3.3 Setup Azure Container Registry Service Principal in ADO

- Visit Project Settings for your Azure DevOps project

- Click Service Connections and then New Service Connection and select Docker Registry.

- Select Azure Container Registry and then click Next.

- Select Authentication Type Service Principal. Choose your Azure subscription. And set Azure

Container Registry to the registry you created earlier. Click Save.

- Set the name of the service connection to ACRServiceConnection and click Save.

3.4 Setup Azure Container Registry RBAC

We need to give our Azure DevOps service connection access to the Container Registry.

export REGISTRY_ID=$(az acr show --name $ACR_NAME --query id -o tsv)
az role assignment create --role "AcrPush" --assignee $APPLICATION_ID --scope $REGISTRY_ID
az role assignment create --role "AcrPull" --assignee $APPLICATION_ID --scope $REGISTRY_ID
az role assignment create --role "AcrImageSigner" --assignee $APPLICATION_ID --scope $REGISTRY_ID

This will allow the service principal to push and pull images from the registry, as well as sign images with Notary V2.

4. Setup the Azure Pipeline

Now, we are finally ready to setup our Azure Pipeline. We'll use the Notation Azure Key Vault Plugin to sign our images.

branches:
 include:
   - main

pool:
 vmImage: 'ubuntu-latest'

variables:
 ACR_NAME: "xeolexampleregistry"
 AKV_NAME: "xeolexamplekeyvault"

 KEY_NAME: "xeol"
 CERT_PATH: "./$(KEY_NAME).pem"
 CERT_SUBJECT: "CN=xeol.io,O=Notary,L=San Fransisco,ST=CA,C=US"
 CERT_ID: ""
 KEY_ID: ""

 IMAGE_REGISTRY: "$(ACR_NAME).azurecr.io"
 IMAGE_REPOSITORY: "signed"
 IMAGE_TAG: "$(Build.BuildId)"

 NOTATION_VERSION: "1.0.0-rc.7"
 NOTATION_AZURE_KV_VERSION: "1.0.0-rc.2"

jobs:
- job: Build
 displayName: 'Build Docker Image'
 pool:
   vmImage: 'ubuntu-latest'
 steps:
 - task: Docker@2
   displayName: Build and push an image to container registry
   inputs:
     command: buildAndPush
     repository: $(IMAGE_REPOSITORY)
     Dockerfile: '**/Dockerfile'
     containerRegistry: ACRServiceConnection
     tags: '$(IMAGE_TAG)'

- job: Sign
 displayName: 'Sign Docker Image'
 pool:
   vmImage: 'ubuntu-latest'
 steps:
 - script: |
     set -ex
     checksum_file="notation_${NOTATION_VERSION}_checksums.txt"
     tar_file="notation_${NOTATION_VERSION}_linux_amd64.tar.gz"
     curl -Lo ${checksum_file} "https://github.com/notaryproject/notation/releases/download/v$NOTATION_VERSION/${checksum_file}"
     curl -Lo ${tar_file} "https://github.com/notaryproject/notation/releases/download/v$NOTATION_VERSION/${tar_file}"
     grep ${tar_file} ${checksum_file} | shasum --check
     sudo tar xvzf ${tar_file} -C /usr/bin/ notation
   displayName: 'Install Notation'

 - script: |
     set -ex
     install_path=~/.config/notation/plugins/azure-kv
     checksum_file="notation-azure-kv_${NOTATION_AZURE_KV_VERSION}_checksums.txt"
     tar_file="notation-azure-kv_${NOTATION_AZURE_KV_VERSION}_linux_amd64.tar.gz"
     mkdir -p ${install_path}
     curl -Lo ${checksum_file} "https://github.com/Azure/notation-azure-kv/releases/download/v${NOTATION_AZURE_KV_VERSION}/${checksum_file}"
     curl -Lo ${tar_file} "https://github.com/Azure/notation-azure-kv/releases/download/v${NOTATION_AZURE_KV_VERSION}/${tar_file}"
     grep ${tar_file} ${checksum_file} | shasum --check
     tar xvzf ${tar_file} -C ${install_path} ./notation-azure-kv
     notation plugin ls
   displayName: 'Install Notation Azure Key Vault Plugin'

 - task: Docker@2
   displayName: 'Docker Login'
   inputs:
     containerRegistry: 'AzureServiceConnection'
     command: 'login'

 - task: AzureCLI@2
   inputs:
     azureSubscription: 'AzureServiceConnection'
     scriptType: 'bash'
     scriptLocation: 'inlineScript'
     inlineScript: |
       set -ex
       az keyvault certificate download --file $(CERT_PATH) --id $(CERT_ID) --encoding PEM
       notation key add $(KEY_NAME) --plugin azure-kv --id $(KEY_ID)
       notation sign --signature-format cose --key $KEY_NAME $(IMAGE_REGISTRY)/$(IMAGE_REPOSITORY):$(IMAGE_TAG)
       notation ls $(IMAGE_REGISTRY)/$(IMAGE_REPOSITORY):$(IMAGE_TAG)
   displayName: 'Sign Image with Notation'

If everything succeeds you should see a successful output from the notary sign command. 🥳

Successfully signed ***/signed@sha256:9c206fb5100b0d4972c9a5d3e6a6238c62307e1c28efbf56d8a06f31147c5f84