Container Image Signatures in Kubernetes
Container image signatures are a rarely implemented security feature, even though images' contents are ever changing and hard to get a grasp of, making it easy for attackers to hide malicious content in them. A main reason for that is that the most popular container orchestrator Kubernetes has no native support for image signatures or their verification. Connaisseur is a Kubernetes admission controller that tries to change that, by allowing only signed images into a cluster and ensure only trusted and unmodified content is deployed, thus amp up your security.
tl;dr
- Docker Content Trust (DCT) is a way to sign your Docker images
- It uses Notary to store signing data
- Kubernetes doesn't support DCT natively
- Connaisseur (GitHub) is a Kubernetes admission controller that intercepts requests sent to the cluster
- It verifies the signatures of all image references found in the requests
- It denies any requests trying to deploy unsigned images
Digital signatures are a well-known approach for maintaining the integrity of any data transferred all around the web. Whether it's for signing emails, using TLS certificates or app signatures for popular stores such as Google Play or Apple's App Store. It's an overall appropriate solution that provides a lot of trust and security, in a world where your credentials are at a constant risk of being stolen and your machine is at a constant risk for abuse, such as bitcoin mining.
With Docker and Kubernetes, a new landscape has been opened up in this world, full of containerized applications in the form of Docker images, all ready to be pulled from your favorite image registry. But when pulling these images, how sure can you be of their contents? Are there no malicious services hidden in there somewhere? Docker images change all the time and that freshly pulled nginx image could already look a bit different, updated with the latest security patch, or maybe updated with something else, more sketchy...? It's hard to say, not only for images coming from a public registry, but also for the ones you built yourself. You would have to go into the image and scan for this "malicious content", how ever that may look like.
So why not use digital signatures for solving this problem, like we always do? That way we make sure we know from whom the image is coming and that there are no bitcoin miners in there (except you pulled a bitcoin mining image of course). For docker-compose, my colleagues at SSE have developed trusted-compose. But how does the situation look like in the context of Kubernetes? Well, let's see.
A Game of Tag
Before talking about image signatures themselves, a quick reminder on how images can be addressed. There are two options, either address them by tag or by digest. Using the tag, the image content can vary over time as the tag might be overwritten with newer versions of the image, especially if you use the tag latest. The digest on the other hand, is unique for each image, as the digest is a SHA256 hash over the image content that cannot be changed. So for the same image you can either use image:tag or image@sha256:d19357..., while the second option always gives you the same content and the first could change with newer versions of tag.
Now you could say, why not use digests all the time, if tampering with the images content is your concern. The same argument could be made around the Domain Name System (DNS) system. Why not exclusively use IP addresses to get rid of DNS spoofing? Just use 172.217.20.110 instead of google.com and no one can redirect you to a fake version of Google. Obviously we are not doing that. Domain names are much more tangible and give you a better understanding on where you are or going. 172.271.20.110 seems like a random number that is hard to remember. Maybe you haven't even noticed that the second IP address isn't valid at all, which shows how easy it is to mess things up here.
The same is true for image tags and digests, which is why it's a lot more common to use tags instead, even though this opens up this issue. And even if you were to use digests exclusively, one cannot tell whether it is a legitimate one just by looking at it, as the source of trust is not clear at all.
Docker Content Trust and Notary
Container image signatures are nothing new per se. Docker actually developed its own system to sign images, called Docker Content Trust, which is tightly connected to yet another system called Notary. Notary functions as a server that stores manifest files of "trusted resources", which are signed by a trusted entity (that's you). In the context of Docker Content Trust that means that all signed or to be signed images are the "trusted resources" for which some manifest files exist. These files contain a mapping between tags and digests in a 1:1 relation, with the manifest files being signed with a private key. That is the image signature. It's not created from the image itself, but only from a mapping that pins down a specific distinct version of an image, when referencing its tag.
You can activate Docker Content Trust, by setting the DOCKER_CONTENT_TRUST environment variable to 1 and DOCKER_CONTENT_TRUST_SERVER to a Notary instance URL. When you then try to pull let's say image:tag, a lookup for image's manifest files will be made in the Notary instance. The signatures of these manifests will be verified, so you can trust the tag-digest mappings. Should the signatures all be valid, the manifests will be searched for tag to determine the digest. This digest can be referred to as the "signed" version of the image, which is then used for pulling the image from the registry instead of the original tag (The registry never gets consulted for looking up a tag's image digest; Docker Content Trust sidesteps this lookup ...). If the signatures are not valid, or the tag can't be found (or no manifest files exist to begin with), the image never gets pulled, as there is no signature.
How do you create signatures with Docker Content Trust? You can just use Docker as usual. Under the hood, it will update the manifest files for the image you are trying to sign or create new ones should they not exist. The current tag is added or updated with the digest and afterwards the manifest file is newly signed.
This covers the basic functionalities of Docker Content Trust and Notary, leaving out some more complex additions such as freshness guarantee and delegation roles. If you want to learn more about Notary and the underlying system called The Update Framework, go to their Github page.
All that is left is to activate Docker Content Trust in Kubernetes and we are all good, right? Well there lies the problem. Activating this feature in the Docker daemon of Kubernetes doesn't prevent you from pulling/deploying unsigned images whatsoever. It essentially ignores the feature and thus defeats the whole purpose of doing image signatures in the first place. So, what can we do?
Connaisseur
Connaisseur solves this problem. It's an open source solution developed by SSE, inspired by the fact that image integrity for Kubernetes emerged as a recurring topic during our work. Portieris (by IBM) is a solution simliar to ours, but its functionality is currently restricted to the IBM Cloud. That's one of the reasons why we decided to take things into our own hands and work on the matter.
Design
Connaisseur is implemented as an admission controller, which means after a resource that is to be deployed to a cluster has gone through the authentication and authorization steps, it will be verified by Connaisseur before actually hitting the cluster. In order to do this verification, Connaisseur searches for all image references in the resource and collects them in a list. For each image in the list, Connaisseur will look up the manifest files in a Notary instance and validate their signatures. If valid, the corresponding digest for the tag is extracted and the original image reference is changed to use the digest instead of the tag. Otherwise, should the signature not be valid, or if the given tag is not present in the manifest files or the manifest files themselves are not there, Connaisseur will deny the resource, stopping its deployment completely.
That's Connaisseur in a nutshell: You want to deploy an image? Connaisseur will only deploy the signed version, or none at all.
Signing your Images
In theory that sounds great. But what about reality? Well see for yourself. First you are going to create an unsigned and signed image, using Docker Hub as your image registry and their Notary instance, then install Connaisseur in your cluster and lastly check whether the signature verification works.
So take two arbitrary images, let's say image:unsigned and image:signed.
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
phbelitz/image signed e6d91653fd50 19 minutes ago 83.7MB
phbelitz/image unsigned 1601ff33dbe9 21 minutes ago 83.7MB
python 3.7-alpine 6ca3e0b1ab69 3 weeks ago 73.1MB
You can go ahead and push the image:unsigned image into you Docker Hub repository, as you don't want a signature for it. For the the one you want signed, you have to activate Docker Content Trust before pushing. As the Notary instance you can use the public one from Docker.
export DOCKER_CONTENT_TRUST=1
export DOCKER_CONTENT_TRUST_SERVER=https://notary.docker.io
Now push the the image, just as you would do with the unsigned one. When doing so, you'll be asked to enter a passphrase for a new root and repository key. These keys are automatically generated and are used for creating the image signatures. They will reside on your machine at ~/.docker/trust/private.
The repository key will be used, should you create new signatures for image, whether that means updating the existing tags or adding new ones. The root key will be needed when creating signatures for completely different images (e.g. image2). In this case a new repository key will be generated as well. Meaning your organization will always have only one root key, but many repository keys, one for each image.
$ docker push phbelitz/image:signed
The push refers to repository [docker.io/phbelitz/image]
6dc8ae05970a: Pushed
13d4f082f19a: Pushed
122b7df5671a: Pushed
5afde73d5bad: Pushed
b10be96d1b4e: Pushed
003d0b48eda4: Pushed
408e53c5e3b2: Pushed
50644c29ef5a: Pushed
signed: digest: sha256:6e7a18a418d4996681ea3bd1757fcf61f70d7cb32902fc4ce85d025c4a633465 size: 1994
Signing and pushing trust metadata
You are about to create a new root signing key passphrase. This passphrase
will be used to protect the most sensitive key in your signing system. Please
choose a long, complex passphrase and be careful to keep the password and the
key file itself secure and backed up. It is highly recommended that you use a
password manager to generate the passphrase and keep it safe. There will be no
way to recover this key. You can find the key in your config directory.
Enter passphrase for new root key with ID 4d8aa7e:
Repeat passphrase for new root key with ID 4d8aa7e:
Enter passphrase for new repository key with ID 7082f7c:
Repeat passphrase for new repository key with ID 7082f7c:
Finished initializing \"docker.io/phbelitz/image\"
Successfully signed docker.io/phbelitz/image:signed
After that you'll need to get the public part of your root key, as Connaisseur will need it to verify all the signatures. Go to your ~/.docker/trust/private directory and use OpenSSL to generate the public key:
$ cd ~/.docker/trust/private
$ sed '/^role:\\sroot$/d' $(grep -iRl \"role: root\" .) > root-priv.key
$ openssl ec -in root-priv.key -pubout -out root-pub.pem
read EC key
Enter PEM pass phrase:
writing EC key
$ rm root-priv.key
$ cd -
Connaisseur in Action
Ok! The final step is the verification. Clone the Connaisseur repository from Github (git clone git@github.com:sse-secure-systems/connaisseur.git) and adjust the configuration. Take a look at the helm/values.yaml for that. There are three values you have to change, the notary.auth.user, notary.auth.password and notary.rootPubKey. For notary.auth.user and notary.auth.password use your Docker Hub credentials, as the Notary instance will use the same here. For notary.rootPubKey put in the public part of the root key you just created. Save the changes and go into the root directory of the Connaisseur repository.
A simple make install should do the rest:
$ make install
bash helm/certs/gen_certs.sh
Generating RSA private key, 4096 bit long modulus (2 primes)
.....................................................................................................................++++
..........................................................................................................++++
e is 65537 (0x010001)
Generating RSA private key, 4096 bit long modulus (2 primes)
.......................++++
...............................................++++
e is 65537 (0x010001)
Signature ok
subject=CN = connaisseur-svc.connaisseur.svc
Getting CA Private Key
kubectl create ns connaisseur || true
namespace/connaisseur created
kubectl config set-context --current --namespace connaisseur
Context \"minikube\" modified.
helm install connaisseur helm --wait
NAME: connaisseur
LAST DEPLOYED: Wed Jul 29 13:42:05 2020
NAMESPACE: connaisseur
STATUS: deployed
REVISION: 1
TEST SUITE: None
The moment of truth has come. Create a sample namespace and switch to it:
$ kubectl create ns sample
namespace/sample created
$ kubectl config set-context --current --namespace sample
Context \"minikube\" modified
Now try creating a pod with the unsigned image. It should give an error, since there is no signature for it:
$ kubectl run unsigned --image=phbelitz/image:unsigned --port=5000
Error from server: admission webhook \"connaisseur-svc.connaisseur.svc\" denied the request: could not find signed digest for image \"docker.io/phbelitz/image:unsigned\" in trust data.
Great success! Works as intended. Do the same with the signed image. Here the pod should be created, as a signature is present and valid:
$ kubectl run signed --image=phbelitz/image:signed --port=5000 pod/signed created
Tada, that works as well 🎉. Now if you check all the pods that are currently running, all you'll see is the signed pod, but not the unsigned one, since this one got denied:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
signed 1/1 Running 0 2m6s
And that's it!
Fade-out
A quite simple example on how container image signatures in Kubernetes can work. In more sophisticated approaches, you can also let your images be signed during a CI/CD pipeline and then be verified at deployment time. This can especially be useful in the growing GitOps approaches that are getting more and more popular. Solutions like Flux, where an operator inside Kubernetes listens on image registries and applies any changes, whether they include malicious content or not, could be secured with Connaisseur. With Flux and Connaisseur together, you need no kubectl access to your cluster anymore, thus reducing the attack surface and at the same time ensuring that only signed images are admitted for deployment. Security all over the place!
If you want to learn more about Connaisseur, for example how the image policy works, go visit the Connaisseur GitHub page. Also feel free to give us feedback or add some contributions, it's much appreciated. Now go ahead and secure your clusters. Cheers!
