Verify Container Image Signatures in Kubernetes using Notary or Cosign or both

Connaisseur v2.0 adds support for multiple keys and signature solutions.

Connaisseur is an admission controller to integrate container image signature verification and trust pinning into a Kubernetes cluster. Now, with the version 2.0 release several important features for a modern Kubernetes infrastructure are added.

But let’s first briefly review some basics.

In short....

Integrity and provenance of container images deployed to a Kubernetes cluster can be ensured via digital signatures. On a very basic level, this requires two steps:

1. Signing container images after building
2. Verifying the image signatures before deployment

Connaisseur aims to solve step two.

Implemented as mutating admission controller, it intercepts resource creation or update requests sent to the Kubernetes cluster, identifies all container images and verifies their signatures against pre-configured public keys. Based on the result, it either accepts or denies those requests.

For more details, please check out our docs or previous blog posts on Connaisseur basics using Notary and initial support of Cosign.

What’s new?

Many of the features in v2.0.0 were inspired by feedback from the community that greatly helped to steer development over the last year ever since the initial release. The central configuration via helm/values.yaml has been re-designed accompanied by implementing validation modules for different signing solutions (e.g. Notary / Docker Content Trust or Sigstore / Cosign) which now for example allows to use multiple keys, signing solutions and registries for validation.... at the same time. Main improvements are:

• multi-key support
• multi-validator support
• overall better usability
• improved Cosign support (incl. auth. for private registries)
• all new documentation

This leads to a few very useful consequences: In modern Kubernetes infrastructures, container images may be pulled from different even external (public) sources for deployment. Besides for example using separate keys for different registries for security reasons, specifically in case of external images one may not even have control over which image signing solution is used. Say Cosign is used for internal images, but on top e.g Redis from Docker Hub is deployed as a database which — being part of Docker Official Images — is publicly available signed via Docker Content Trust for Notary. Using Connaisseur v2.0, the internal images could be validated using the internal key with a Cosign validator while Docker Hub’s public root key with a Notary validator could be set up for all Docker Official Images (for a list of a available images see here). Which validator to use for which image can be then be granularly configured via Connaisseur’s image policy. It is even possible to directly block certain images and sources implementing something like allow- or denylists.

In fact as Docker Hub Official Images are very common, the respective public root key is directly shipped pre-configured with Connaisseur. We expect that other big projects like Kubernetes will begin publishing public keys for their images, as for example Distroless recently did, and might then extend the pre-configured validators.

To learn more about v2.0 and how to use it, check out our new docs 📝

However, a demo says more than a thousand words....

Verifying Container Image Signatures for Kubernetes — fast!

Getting started just got even faster 🚀

⚠️ Only try this on a test cluster as Connaisseur will actually block all unsigned images ⚠️

Connaisseur can be fully configured via its helm/values.yaml. However, as it ships pre-configured with public keys for its own and Docker Official Images, you only need to clone the repository for a quick test:

$ git clone https://github.com/sse-secure-systems/connaisseur.git
$ cd connaisseur

And install Connaisseur via Helm:

$ helm install connaisseur helm --atomic --create-namespace --namespace connaisseur

Once installation has finished, you are good to go. Successful verification can be tested via official Docker images like hello-world:

$ kubectl run hello-world --image=docker.io/hello-worldpod/hello-world created

Or our signed testimage:

$ kubectl run demo --image=docker.io/securesystemsengineering/testimage:signed

pod/demo created

In both cases, the pod is created. However, when trying to deploy an unsigned image:

$ kubectl run demo --image=docker.io/securesystemsengineering/testimage:unsigned

Error from server: admission webhook \"connaisseur-svc.connaisseur.svc\" denied the request: Unable to find signed digest for image docker.io/securesystemsengineering/testimage:unsigned.

Connaisseur denies the request and returns an error (...) Unable to find signed digest(...). Since the images above are signed using Docker Content Trust, you can inspect the trust data using docker trust inspect --pretty <image-name>.

To uninstall Connaisseur use:

$ helm uninstall connaisseur --namespace connaisseur

Congrats 🎉 you just validated the first images in your cluster!

But can we use both, Notary and Cosign?

Absolutely! Let’s add signature verification for Distroless docker images from Google 😃

The required Cosign public key is published in the Distroless repository. Now, we need to edit the helm/values.yaml. Find the default validator under .validators and:

• Set the type to cosign
• Remove the host entry
• Uncomment the default trust root
• Add the public key from the Distroless repository

The result should look similar to this:

- name: default
  type: cosign  # or other supported validator (e.g. \"cosign\")
  trust_roots:
  # # the `default` key is used if no key is specified in image policy
  - name: default
    key: |  # enter your public key below
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZzVzkb8A+DbgDpaJId/bOmV8n7Q
      OqxYbK0Iro6GzSmOzxkn+N2AKawLyXi84WSwJQBK//psATakCgAQKkNTAA==
      -----END PUBLIC KEY-----

We need to install again via:

$ helm install connaisseur helm --atomic --create-namespace --namespace connaisseur

Now, we can validate both, Docker Official’s Notary-based signatures and Distroless’ Cosign signatures:

# distroless
$ kubectl run distroless --image=gcr.io/distroless/static-debian10

pod/distroless created
# docker official
$ kubectl run hello-world --image=docker.io/hello-world

pod/hello-world created

While our unsigned test image or any other unsigned image for that matter is still denied:

$ kubectl run demo --image=docker.io/securesystemsengineering/testimage:unsigned

Error from server: admission webhook \"connaisseur-svc.connaisseur.svc\" denied the request: Unable to find signed digest for image docker.io/securesystemsengineering/testimage:unsigned.

That’s it. You now enforce signed Docker Official and Distroless images in your cluster and block any unsigned images.

To get started configuring and verifying your own images and signatures, please follow our setup guide 🚀

What’s next?

After this major feature release, we plan to focus on performance, patching and stability improvements like increasing parallelization, implementing a more performant server and load tests. Furthermore, as Cosign is bound for its 1.0 release, we hope to expose some of the more advanced features. Also, the modularization of Connaisseur will simplify adding more signing solutions, so we hope to look into that as well.

We really appreciate input from the community, so please keep sharing your feedback as GitHub Discussions, issues or contribute directly via PRs 🙏

Thanks to everyone who has contributed so far ❤️

• GitHub Repository: https://github.com/sse-secure-systems/connaisseur
• Documentation: https://sse-secure-systems.github.io/connaisseur/