Keys features
Congratulations! Some audio discussion from when this was WIP, at timestamp 16:04:
Very nice niche.
But why is the config file not made with Nix syntax? Or TOML.
congrats on the release!
i like the comparison table - it’s kinda cool to see an overview like that.
Why KDL for the manifest language?
KDL has a syntax that isn’t a burden to use a configuration language — which helps explain why is very popular for its age. JSON has no comments, one must watch commas, & is verbose with quotations. TOML doesn’t nest well. YAML is overly complex. Nickel is great, but needs to be transformed into one of these others. EDN was considered, but KDL felt better to write – especially the fresh-cmd syntax using
$&|as node names.
Someone had written KDL parser for opam & I like its ergonomics. There is no Nix parser. I also really dislike TOML’s ergonomics & it wouldn’t fit here.
There is no Nix parser.
Even if you don’t have available the bindings to the Nix API, the usual approach is to call nix as a subprocess, and then parse the resulting configuration that has been transformed to some other language. Either directly nix eval --json -f ./config.nix or with a build step in-between nix build -f ./config.nix -o ./config.kdl. This is how terranix works, for example.
In the comparison table, there is a row called “User-written freshness logic”, could you please elaborate what you mean by that?
It looks like freshness (fresh-cmd and fresh_value) is nixtamal’s equivalent for nix flake update or npins update/upgrade
Technically, yes, but evaluation with Nix, transforming to JSON, decoding, then processing sounds like a strictly worse/expensive route. Since the manifest requires a tool & is strictly used to create the lockfile + its loader shim, any configuration language could be be a valid option (I initially thought of Nickel since I like it’s type system, but again it felt like the evaluation, & transformation to JSON was not worth it). KDL happens to offer, in my opinion, a better UX + better readability for this task—which is why newer tooling is choosing it like niri & zellij. Nix is great, but the language wouldn’t be my first choice.
The dual to “fresh” is “stale”. Is your input “stale”, let’s check the fresh-cmd to find out!
I didn’t think the word “latest” was quite appropriate since it’s up to you to decide what “fresh” means in your context which often isn’t “latest”. The idea is that: so long as it can be executed in your shell & returns a string that can be compared against, you can write honestly whatever you want. This also pairs well with the command named nixtamal refresh (same same as pkcon refresh) which runs the fresh-cmd which can short-circuit trying to prefetch your input.
In the showcase, I showed 2 additional examples like getting the latest version from Security/Server Side TLS - MozillaWiki as to consume the JSON file in a setup as well as another querying the JSON API of Hugging Face to see if a specific model is the most recent. But you can stat a file to see if its been modified, parse an Atom feed & select if a tag is less than some value, or really whatever you decide is going to be your own logic for that input.
What is the motivation? I had been using flakes but I was frustrated by the lack of support for my preferred VCSs, Darcs & Pijul. For a time I had been using Smederee or darcs hub’s latest tarball of my projects since flakes were never going to support these VCSs, but there was a problem: since these weren’t using a ‘supported’ VCS, flakes just blindly redownloaded the whole tarball every nix flake update—just to report back that the hashes are the same + here’s our new latest checked time stamp. This is wasteful to my bandwidth (I tether a lot) as well as to the servers of these smaller forges. I wanted a way to short-circuit this with by running my own command that compares against a known value that was quicker & cheaper… like fetching & parsing a static file.
Now, I’ve still been using Darcs, but without darcs ls-remote I don’t have a way to see if my local is synced up with latest weak hash (XOR of the patch set since this isn’t a snapshot-based VCS). Instead I just added * posthook … command that echoes new weak hash to a file I can cURL. Again, the genericness of fresh-cmd saved me from needlessly downloading the whole repository just to check if we are at a new patch set.
It was interesting that while I was writing Nixtamal, a post, The Silicon Valley Stack Doesn’t Work Here: Why Africa Will Lead the Post-Bloat Web, is published talking about some of the wasteful ways Western developers treat the network—& avoiding that waste in network requests for large downloads was explicitly an issues I am looking to tackle with Nixtamal.
(seems I should add this to the FAQs)
This is really cool by itself, but I do feel sad about the fragmentations of Nix inputs locking. It seems that every week there is a new solution for the same problem, just because flakes didn’t do it right… I really hope that flakes were properly designed so that everyone can use the same tooling 
I would rather see a little bit of competition in the space than put this sort of thing in the compiler 
I do think this might be the only tool trying to pin things outside the builtins which is pretty novel.
Really neat project! Coincidentally was discussing declarative pinning around when it got released.
One of the concerns that popped up with this approach is that you cannot sync the manifest with the lockfile when you use the inputs, you have to have a manual step beforehand. Flakes can update the lockfile for you without any intervention, whereas here users would have to make wrappers over their already existing tooling to get the same behavior. I’m not sure how much of a issue this is UX wise, but I’m curious on your thoughts on it.
Also, some things I wanted to point out from the faq:
I do think this might be the only tool trying to pin things outside the builtins
npins can also use nixpkgs fetchers! Though its definitely more intuitive for users when its the default. Maybe in the future there can also be added support for nixpkgs fetcher’s which don’t have a builtin. I do experience limitations with git+ssh when using them but otherwise they really should be preferred most of the time.
Automatic sync has actually been 50:50 annoying:useful to me in the past where I got an update / download prematurely. VCSs make it easy to revert, but if you haven’t set one up yet, you get changes that you might not yet be ready for. You could use something like entr, watchexec, or wrap like inotify to listen for changes & then run nixtamal lock, but I don’t know about it.
- niv is made in haskell, so I think it got conflated with npins ;p
Yes, that was definitely a jab at npins. I wanted the features you see here in npins, but I wasn’t interested in learning Rust (or spend more time on proprietary MS GitHub) to try to add any fixes for the behaviors I was interested in.
- in npins you do not need to drop a input to change it, you can just add it again and it’ll be replaced
True, it more like ‘upsert’ but you still have to know the key. The time I spent with npin a while back, I found it easier to just modify the lockfile… to which I made a small JSON error & was turned off by the whole UX.
npins can also use nixpkgs fetchers
Interesting. I hadn’t checked in on the project since this happened.
I’m currently working on being able to specify if you would prefer eval vs. build fetch times project-wide & by input. A contributor pointed out some use cases for preferring one or the other. I am also looking at preferring the system’s Nixpkgs & doing a feature check to see if I can use it over trying to import Nixpkgs twice as the fetchers are fairly stable & not worth slowing nix-build down to just to get fetchdarcs which my system already has. This has a tradeoff of purity for being quicker.
Someone had written KDL parser for opam & I like its ergonomics. There is no Nix parser.
Are you saying there is no Nix parser written in OCaml?
Why would there be? It’s not a small task & Nix doesn’t have a formal spec as far as I recall which is why Tvix/Snix have so many corner cases to cover with the moving target that is C++ Nix.
0.1.1-beta made it to NixOS unstable (see: NixOS Search). 0.2.0-beta has an open pull request.
This version has bugfixes as well as a new feature: fetch-time: build | eval on supported inputs (file, archive, git) which allows you to choose eval-time fetchers (so builtins.fetch*) over the build time ones (pkgs.fetch*). This puts the user in control, but can technically also free one from needing Nixpkgs now (tho Nixpkgs is required if using patches, using Darcs/Pijul, or you want offline caching, hashing, & all the other things we have come to expect from pkgs.fetch*).
Thank you everyone for the feedback. I made a number of changes to the static site to cover some questions I got from various platforms. Special shout-out to @WeetHet for fixing the Darwin build & suggesting fetch-time.
TODO: fetch-time needs to added to nixtamal show.
One emergent, albeit “impure”, feature I found was using
inputs = import ./nix/tamal { bootstrap-nixpkgs = <nixpkgs>; }
In certain cases I think it’s a fine enough tradeoff to prefer the system’s pkgs.fetch* as they are stable enough for bootstrapping (& we do do some feature checking anyhow). This matters since the builtins caching is stored differently causing the user to download Nixpkgs twice (once with builtins for bootstrapping, then again when imported as inputs.nixpkgs). I have been trialing this pattern today
# release.nix
import ./release.nix { }
# release-lib.nix
{ system ? builtins.currentSystem, bootstrap-nixpkgs ? null }:
let
inputs = import ./nix/tamal { inherit system bootstrap-nixpkgs; };
pkgs = import inputs.nixpkgs { inherit system; overlays = [/*…*/]; };
in
{ inherit (pkgs.my-scope) default dev-shell my-package; }
# release-dev.nix
import ./release-lib.nix { bootstrap-nixpkgs = <nixpkgs>; }
# shell.nix
(import ./release-dev.nix).dev-shell
# default.nix
(import ./release.nix).default
Which lets me use the system’s Nixpkgs for the dev-shell where I mostly want Nix to bootstrap as quickly as possible, but can also do nix-build ./release-dev.nix -A my-package with the same tradeoff. This is not pure evaluation from top to bottom, but everything beyond fetching the sources into the Nix store is pure which often fits “good enough” criteria. release-lib.nix could be extended with architectures if needed like how upstream Nixpkgs does it, but for my use cases, this hasn’t been needed. Alternatively I could just set fetch-time=eval for Nixpkgs if it’s supported (meaning patches weren’t added).
This one is small & adds fetch-time to nixtamal show so you can better understand how nixtamal sees/understands your manifest.
Bug fixes & add a new TUI-like UX for refresh & lock commands—which cuts down on noise & make the phase much clearer.
1.0.0 is around the corner with the upcoming features being a basic nixtamal upgrade command to upgrade lock+manifest to latest as well as Fossil support (awaiting merge).