I’m using NixOS for more than 2 years at this point, both on my vps and laptops. So configuration-wise I am OK with (and I really like) the approach.
But on development, it’s another story. I’m working at a Python-shop, and people here used to handle stuff with docker/poetry/pyenv/whatever… I wanted to nix-ify some projects but it’s always a mess with, say, medium-sized django project. Maybe I am too dumb to understand why we’re doing what (since I mostly search github to get code examples and do it), it always feels half-done. Also mostly I realize it’s breaking after some time (with non-pure “use-my-channels” approach of course).
With the introduction of flakes, which made me excited, since there will be no more impure stuff, no more breakage by time. But the problem is, I understand even less now. Tried to convert my configuration to a flake, but my def myfunction(args) brain couldn’tunderstandwhy my import sometimes needs an argument and sometimes not. (links are just examples to things I skipped by saying “…alright”).
Anyway too much whining. Thanks for reading so far. 2 main questions:
Is flake system OK to invest right now? Or do we expect changes which can break it?
Do you recommend any repositories which I can gaze, both nixifying-django and flaky-configuration wise? (preferably as light as possible)
Flakes are in the latest stable release Nix 2.4 as an experimental feature. Even though Flakes are still subject to change it is highly unlikely that they are going to go away again. Also the Flakes changes over the last year were relatively minor (for my use-case at least).
I would point out that there are some oddities about flakes still (recently someone asked why local flakes are copied into the nix store constantly, for example, which causes tons of write and disk usage overhead for the smallest change).
They’ll probably get ironed out over time, but this may be a long process. The UX isn’t perfect yet, so going into it expecting perfection may leave you disappointed - and trying to get someone who objects to the usage of nix in the first place (as is more likely when you’re not in these parts) to accept flakes may be utterly impossible.
Figuring out how flakes work and how to use them also took a deep dive into nixpkgs for me, which made me finally learn how it all works, but may also be mildly frustrating, especially since documentation is sparse.
I keep meaning to write a basic set of goal-oriented “howtos” to try and document the what and how of flakes.
All of that said, personally, I think flakes are a much more explicit way of managing the things nix can do, and they are very nice for a wide range of use cases non-flake nix only marginally supports. I think it would be a major step backwards to not continue stabilizing the feature.
What are your issues with Django and Nix? I’m using them together since two years without issues, first with all the packages from Nixpkgs with some minor dependencies added manually and lately with the help of mach-nix, which help packaging what isn’t in Nixpkgs.
The only real issue I had from a Python-Nix PoV is that usually Django “sites” aren’t Python packages but trees of Python modules and other files, that the Django “manage” utility is able to work on…
Yes, Python packaging is in a sad state, but what the writer describes is dependency vendoring vs using the distro packages. This affects most languages that have modern package management, does it help so much here? It’s again a “Nixpkgs, mach-nix or poetry2nix or foo2nix?” thing
My main problem is that I have no idea what I am doing. I hope I am doing everything wrong here and that’s not what should I do to have a half-baked development environment (this is just the package, I need more hacks on shell.nix to make a database ready if I understood correct etc.).
But you want really to work (i.e. developing) on this “serveradmin” package or something else? Because you are packaging it in the recipe while in my mind a development environment should have the dependencies packages, but what I’m working on should be editable. Usually my Python dev environment involves installing the dependencies together with some package to do a minimal dev env automation like pydoit or gnumake, then my local flake devShell will enable a local vrtualenv and do a pip install -e pkg of the packages I’m developing.
Make or Doit will then allow the developer to deal with automating the development setup like:
start the needed services: how? It depends on the project;
create the db and load latest definition;
load test/dev data;
use Django manage.py to run all that bunch of modules and configuration
The settings.py for the development is managed like a normal python module. I don’t see the point of building it.
The production.py is even more simple because it mostly load variables from the (production) environment. Deciding stuff like ports, /var paths, passwords, logging, etc. is all another question that for me doesn’t matter so much in development, where for example when I’m using PostgreSQL it is configured to always trust system identity. In production those things matter very much instead, and they influence the decision made for a bunch of other services, like the reverse proxy, log collector, backup system, secret storage, etc. but they are managed in the final NixOps configuration for the system and the container used for each installation
At NixCon 2019 there was a presentation by a guy from the brilliant group that develops DeckDeckGo, and I’ve initially stolen some ideas for the management of background services like PostgreSQL from their dev environment while other things like automation I do it differently as I said before.
I’d like to have it packaged so I can deploy it too, but of course development is more important requirement, especially when my target is injecting the nix into our ecosystem more. As I said I also made a shell file already, which has nasty shell hooks etc in it. But looks like that’s the default way, considering the example you sent.
Of course the development environment should be editable, but not the dependencies, no? I guess I need to be patient to allow the ecosystem to get mature (for example I just realized there is a tool called poetry2nix which sounded magical, but that fade away once I realized private repositories are not supported yet).
But anyway, thanks for the detailed response and suggestions. I’ll look for more examples in public repos and follow/support some projects.
While I was working at microsoft (azure-ml), I would use nix to bring in the more PITA models (pytorch, tensorflow), and just use venv to give me an editable install (e.g. pip install -e .). This had the benefit that a lot of domain specific python packages could just be fetched impurely, and as long as it was just pure python, it worked fine in conjunction with nix.
Also, it made it easier to switch between interpreter versions. I could do nix-shell -A py37 or any other relevant interpreter at the time, and get a development shell with a new interpreter and most of my dependencies cached globally. And nix is one of the few platforms I can think of where we usually also have the upcoming dev cpython interpreter.
If you have many repos, having a global cache is nice as well. Much faster to tear down and build new environments.
I haven’t tried any of the machnix or xxx2nix tools to see how development environments work with nix.
Do you recommend any repositories which I can gaze, both nixifying-django and flaky-configuration wise? (preferably as light as possible)
I understand “experimental” quite differently from how I hear it used in Nix documentation.
I am going all in on flakes because of the wonderful improvements they bring to packaging and reproducability.
I’ve thus far:
rewritten a number of project CI pipelines to use flakes over Dockerfiles
prototyping a reproducible macOS development environment
For me, the big risks right now are lack of documentation and the time to get something done.
Me too!
For some example sources to read from I’ve been using the “experimental” flakes search and chasing down each repo’s flake.nix.
Me too!