Kubernetes container runtime interface + Nix?

Development
1

hi there,
i’d had this idea and wanted to bounce it off of others to check if it holds water.

we’ve had some NixOps tools out there, but i was wondering if it could be useful to combine Nix with one of the more mature existing tools out there, Kubernetes. now, we have some integration there already, but more specifically, i’d been wondering if it could be useful there to skip intermediate container runtimes there. so far, Kubernetes’ container runtime interface (CRI) in its implementations seems to have been integrated mostly with the Open Container Initiative (OCI)'s container specification - think Docker/Podman.

so i guess i’m wondering:

  • could Nix interface with CRI more directly than thru containers, given a CRI implementation geared toward Nix?
  • would that make sense, i.e. add value over what is currently offered by the existing NixOps / Kubernetes ecosystems?
2 Likes
2

Actually from a kubernetes perspective you would mostly user containerd as container runtime. There is also something like CRI-O, but I never used that. Docker has basically been entirely deprecated by now.

And since Kubernetes is a container orchestration platform, I’d say that is what I expect from it: to orchestrate OCI containers. So I am not entirely sure if I understand what it would mean to “skip intermediate container runtimes” when it comes to kubernetes.

Because when you define kubernetes resources, in particular deployments, you will have to specify containers based on images for your deployment. And these need to come from somewhere.

3

thanks, that makes sense.
i suppose the most straight-forward integration then would be to have it use nix-generated containers?

i suppose one thing that seemed awkward for me when using containers (using docker) before was that caching at that level seemed far less sensible compared to how caching is done in the nix ecosystem.
but i suppose in your setup that may be solved already?

4

what exactly are you trying to do?

if i know that, i can probably say if you need to deploy a behemoth of a container ochestration?

You’ll find nix can probably be used in very clever ways to avoid these mamoth and complex container managment systems… but sometimes not…

i’d love to know more…

5

@nixinator thanks for challenging me on this. initially when i started thinking of using kubernetes (before nix came into the picture) i’d been thinking of:

  • separating servers vs state (nix: separated by default, just specify external storage)
  • declarative infrastructure setups (nix: nix for software + terraform for external services?)
  • ways to override/customize existing packages (nix: overlays built-in)
  • restarting failing instances (systemd setting restart on failure)
  • redundancy / load-balancing (nix: deploy / configure existing load balancers?)
  • health/performance dashboards (nix: use just any generic server dashboard?)
  • sanity checks (e.g. works with open policy agent) (nix: nix-policy)
  • git-centered continuous delivery, think argo CD, web interface optional (nix: any NixOps tools?)

in retrospect i imagine most of these are covered by NixOps tools already - i’ll admit i haven’t been particularly familiar with them so far.

1 Like
6

You are right!

once you start thinking in a reproduable and declartive way… you will find lots of these tools are no longer needed.

most of these tools, and the state they manage just go away.

You’ll then start to look at the whole ‘devops’ stacks as overly complex…

a bit like a ‘stack of plates’…hard to keep spinning and the higher it gets the harder it is to keep them from SMASHING onto the floor. You really need to avoid ‘devops’ word soup if you can.

it may look good on your CV, but it’s not going to any favours for your mental health, or actually ‘understanding’ the things you are deploying.

for me perfect design in not when you add code or features, it’s when you can’t take anything more away.

I know which one is sane, and one that is insane

k8s might seems like a great solution…that does everything, the same with docker…

but i don’t like YAML, i think it’s Rammel… but if you want to spend your days writing templating code…thats up to you… .it certainly doesn’t bring me any kind of joy.

You can achieve a lot of what these tools by using standard nix tools.

not all , but most.

I can send you horror story after horror story of deployments of these full stack (of plates) deployments…

However i understand a lot of people don’t have the option of doing rm -rf k8s and rm -rf docker…

but if your building a green field app, then try it with pure nix, it will be fun.

The first thing they do with nix is try to get it run all the tools they think they need, but once they run it, the discover they don’t need the legacy tools like ansible and docker at all.

A lot of ‘devops cloud services’, seem to be 3 line shell scripts, than have a fancy name, and a fancy logo, and they charge you 9.99 month, i’ll never still don’t know how they do it… it’s a pretty good scam.

good luck…i hope you have as much fun with nix that i do… Say no to word soup CNCF tools!

any questions, just let me know.

May the nix be with you, always.

1 Like