Me and my co-authors have submitted this paper for publication at SCORED '24 and it got accepted.
Like all submissions it is still part of a shepherding process until September 1st.
Until then we will still do some polishing and make a few changes that were requested by the anonymous reviewers and shepherd, who I am told will avoid this thread until after publication.
I wanted to use the remaining time as an opportunity to also ask for honest feedback from the Nix community, so if you’re interested feel free to take a look at the current draft of the paper and share/discuss your thoughts here.
While this is a scientific paper, we wrote it with the Nix community in mind and with the hope that it might have positive impact on the understanding and further development of Nix and other cloud build systems, which is why we care about your feedback. Please be aware that at the same time we cannot promise to actually incorporate your feedback, since the audience for the paper is much wider (though probably smaller) than the Nix community by itself and there are limits to what we can actually change in roughly two weeks.
If there is something you would prefer to say in private, you can find my email address in the PDF, and I think you can also DM me on here.
@mschwaig I just watched your NixCon talk which did a great job explaining the trust and threat models for cloud builds.
As I understand it, you’d like to add “more stuff” to the Nix signatures. Kind of like a generic metadata payload. Consumers could then employ more sophisticated policies when they decide if they should trust a specific signature. One of the things that could be in this metadata is build host provenance so you can be sure the builder ran a specific builder host image that you trust, for example.
However (and this might simply be to my lacking understanding of techniques like TPM etc), I don’t understand how you could verify that the output content hasn’t been modified after the build ran? Wouldn’t you then need to sign the output hash with some sort of TPM key? How would such a solution scale? Would a consumer need to have a list of trusted “public TPM keys”? It doesn’t sound scalable. Most likely my understanding of the provenance part of your talk is flawed, so I would be grateful if you could explain it a bit further? I think my question boils down to: What key is used to sign the signatures? A key that is unique to each builder, or a key that is unique to a cloud provider (that runs many builder hosts)?
Thanks for watching my talk and asking about this.
I agree with your summary. I am also still working on this topic and and a more specific proposal for the signature format.
Yes, you have to bind the signing key to the hardware of the builder, for example by generating it in a TPM 2.0. You can then have the host generate a key attestation about that signing key and include it with the signature. This attestation can at the same time make signed statements about the measurement values (PCR hashes) observed at boot. There is also a certificate chain that links all of this this to the manufacturer of the chip.
Section 6.3.3 of the paper has a longer explanation of this, but also feel free to message me with more questions about it or follow up here.
If everybody had to configure this manually in order to use https://cache.nixos.org, I agree that this would not be scalable. And I do not outline a solution to that problem in the paper.
If one organization maintains a number of builders, re-signing each path with one organization-specific key and trusting that one would be the wrong solution, because that gets rid of the end-to-end guarantees and means we could not do revocation or would need some extra mechanism for it.
I think an organization would instead want to publish a signed list of the currently trusted builders in some way, and consumers can use that, (or fork/adapt it for more advanced use cases where they adapt the trust model of the organization in some way).
I think a transparency logs, nixpkgs or another trusted repo, or some signed list on the web with a short validity period, would all be worth looking into as specific mechanisms to do this. I have not actually looked deeper into this problem yet.
There are also use cases that do not need to scale without user intervention, like trusting a number of builders your friends or trusted community members maintain. Everyone maintaining their specific list of those trusted builders and the verification criteria for them manually sounds fine to me.
TLDR: The signing keys should be specific to the builder, and we still have to come up with some way for organizations to aggregate/curate those keys and the applicable criteria, which preserves the end to end guarantees, but allows for adaptations over over time. I think this is a solvable problem.
Let me know if you have any further questions or perspective on this. I’m also happy to have meetings about these issues (with people personally or with their businesses).
Thank you for your detailed answer, this makes it much more clear to me (and also aligns with my intuition of the problem).
I think an organization would instead want to publish a signed list of the currently trusted builders in some way, and consumers can use that,
Yes this sounds like a reasonable approach. In fact, if you outsource builds to an organization you will have some legal agreement with that organization, so when you ask the organization if a given TPM signature is actually produced by the organization then you shift the burden of proof over to the organization, and they would take on some level of legal responsibility for replying truthfully. On the other hand, it is maybe not conceptually much different than simply having the organization sign the TPM signature with an organization-specific key. Having the extra callback to verify signatures would help with key rotation, though.
I’m curious, have you had any discussions with the Nix team about extending the signature format (and perhaps the narinfo format too)? It seems like a non-controversial first step that could be valuable for experimentation, and useful regardless what solution is actually selected in the end.
Finally, would it make sense to think about sigstore (https://www.sigstore.dev/) usage/compatibility? It seems sigstore has gained traction in the industry.
I have had conversations with people from the Nix team about my work in general and some ideas for changes like that, after joining one of their public meetings. They have been very generous with their time and attention. One of my personal goals with working on a signature format is actually getting it upstreamed. Since I do not have a specific proposal ready to share, I think its is still too early to actually have that conversation.
Hayden Blauzvern from the sigstore project was actually one of the other presenters at the workshop where I presented my paper. I had a really nice conversation with him as well, but we did not discuss this specifically. I still have to learn more about sigstore, in order to get an idea about what an integration could look like. It is definitely one of the things we should look into.
@mschwaig I’ve been coming up with some ways to extend the signature format too (mostly surrounding support of multiple signature algorithms) and it would be compatible and complement your ideas, we should chat. Loved your talk.
@numint That sounds interesting! I have been looking into formats like JWS, which also makes the signing algorithm more configurable. I did not spend any time on looking into configuring algorithms or using different algorithms yet. I’m happy to have a meeting about this, I will PM you.
