In the last two weeks, I was working on releasing the first OSS release of Kamus, a secret management solution for Kubernetes. To release Kamus, I had to publish a few docker images to Docker Hub and the CLI – which is deployed as an NPM package. This was the first time I had to deal with deploying packages to a repository. Doing this process thought made me thinks a lot about the (in)security of this process, and I want to share these thoughts with you. It’s all sums up to one small question – what’s inside the box? Do you know what you install when you download a package from a repository?
First, let’s clear up the terms used in this post: There is a package. This package is upload by a maintainer to a repository (NPM, Docker hub, NuGet, PIP etc). The user uses a package manager tool to download the package from the repository and use it.
Why should we care?
Package repositories become a critical part of the software we build. One vulnerable package could make an entire application vulnerable, which can result in a data breach, denial of service or any other kind of attack (Equifax breach is a great example). Today, when our code is running in docker container, it just becomes worst: Think about downloading a random VM from the internet and using it to run your production application (yes, I know containers are not VM). This is what we usually do when building containers. And if that’s not enough – we use package managers to install software on the personal computer we use (and we just had an example for such an attack with the event stream vulnerability).
I’m telling all that not to scare you, but to point out how much trust we put on package managers. Tools like Docker hub, NPM, NuGet, PIP, and others, which we all used on a daily basis. How much can we trust these tools? Do they worth our trust? To answer these questions, we need first to define the threats, and what controls we would like to see in place.
A question of trust
It’s all boiled down to one simple question: Can I trust the package I’m downloading from the repository? Can I tell for sure it was published by it’s authors? Can I know for sure who are the authors? How can I, as a package maintainer, guarantee the authenticity of my packages? (ok, more than one question, but you get the general idea).
And what happens when a CI/CD is involved? Pushing packages manually is an error-prone and time-consuming process. I want to have CI/CD process, without weakening the process of publishing packages. How can I achieve that?
Possible Controls
I want to offer a few controls I that should exist in a package repository. These are controls that can help mitigate the threats I mentioned before and create a more secure (and trustable) package repositories. Nothing here is new, but for some reason, not all of them are available today – and we should work on fixing that:
- Least privilege. Create a user for a CI/CD that has the minimum required permissions, usually only publish. The CI/CD process *should* not run as admin, for example. Apply the same rule for all users that have access to the repository.
- Short lived tokens. The tokens used for authentication by the CI/CD should have expiration dates. Create tokens that are valid for six months or less if possible.
- MFA. Enforce MFA for all users log in. Requires the user to re-authenticate before publishing a package.
- Audit. Have a clear audit log describing all the actions performed, including simple way to export the data. The audit logs should have properties like source ip, authenticated user etc.
- Verified maintainers. Have a clear way for package maintainers to prove their identity. I, for example, should not have the possibility to publish packages under Microsoft name, under any circumstance.
- Signed packages. Allow package maintainers to sign their packages.
Existing controls in NPM and Docker hub
Now that I have a list of controls, let’s take a look at what controls are available to use. This could help us estimate how secure these repositories are:
| NPM | Dockerhub | |
|---|---|---|
| Least privilege | Tokens are created for a user, and has support for access level - read only or read and push | The permissions are per group, not per user. You can create a group only for your CI/CD user, but this can be hard to maintain when have many images |
| Short-lived tokens | No | No token, only user authentication. There is no way to create short-lived passwords |
| MFA | You can enforce MFA for all package maintainers, either for admin or for publish. Once this setting is enabled, you can't push with a token - no CI/CD integration. | No |
| Audit | No | No |
| Verified authors | No | Some support |
| Signed packages | Yes | Yes |
From the table, it should be clear that NPM is a bit more secure than Docker hub. You can enforce MFA and there is support for tokens. The main downside of NPM is how the MFA is implemented. When I worked on Kamus CLI, I wanted to enforce MFA for all the users. Kamus is a very sensitive product – users need to trust it with their secrets. Unfortunately, as you can see on the table, you must choose between having a CI/CD and enforcing MFA. Currently, you cannot require MFA and have a token that can be used by a CI/CD system (Liran Tal asked for it more than a year ago). Due to the sensitivity of Kamus, it was a very hard choice here – and the final decision was to publish it manually for now. Personally, I hate to choose between security and usability.
The lack of short-lived tokens and MFA (yes, I don’t think we can say that MFA is supported currently in NPM) is also something that does not makes me happy. The combinations of this with the lack of audit logs is what lead me to loss the trust in this systems. I hope we can see these areas improved in the future – this is critical for all of us.
Looking forward
In this post, I tried to share an initial list of controls that any package repository should support. I used this list to test NPM and Dockerhub, and the results are disappointing. I need your help for the next steps – to make a real impact. What we can do? One idea is to start a project similar to OWASP ASVS -what do you think on “package repository verification standard”?. The project should contains a list of controls that each package repository should have. My list is just my list – I’m sure there are controls I missed.
Another direction is including more package repositories in the table – what is the status of Nuget? And what about Helm charts? Maybe this list should be part of the same project. Finally, the most important part is communication. Every developer is a costumer of a package repository. We all affected. We all should require from repository owners to build more secure system. There is so much we can do to make this process a bit more secure. Will you join me?