So today I noticed that home manager was pulling obsidian from the github source rather than via the nix cache (Just to be clear I have not specified it to this). I would much prefer it to only use nixpkgs and not directly from source, am I able to achieve this in some way?
Would setting max-jobs to 0 be the way to go?
I am quite new to nixos so still getting to grips with things, my aim is to only use nixpkgs for packages as I feel it improves security and reproducibility so I do not want nix pulling directly from upstream.
To be clear, by using the nixpkgs repository (and derivations — build recipes defined in it) you trust the recipes (which pull the sources from upstream repositories; or repackage prebuilt binaries in the case that there’s no source available or source packaging is infeasible). hydra.nixos.org pulls the same sources as you would have locally, had you run the same build locally without substituting given enough compute resources. The cache serves as a sort of way to memoise builds (including downloading sources).
So what you are getting from the substituter is doing less compute locally (and building an entire nixos closure from source takes a beefy machine quite a bit of time). By using the cache.nixos.org you are trusting more – the NixOS build farm – to faithfully build the derivations, in addition to trusting upstream sources/binaries that are pulled in the derivations.
Sorry if I’m being too obvious with my lengthy explanation.
I think I got some of my terminology wrong in my post.
So am I correct in the assumption that most packages come from cache.nixos.org and are either built via hydra or locally via the nixpkg recipes. Some exceptions exist where the recipe pulls directly from upstream due to license reasons (like obsidian) which is where the allow unfree option maybe plays a part?
The recipe doesn’t know whether it’s going to pull from hydra or from upstream. Everything in nixpkgs is defined purely in terms of building from upstream / from source. The ability for those build results to be substituted from cache.nixos.org instead of being built locally is entirely logic that exists outside of nixpkgs; Nix itself knows when the recipe (which, again, is just trying to build from source) matches one that the cache.nixos.org already built, and just downloads that result instead of building it. There is no distinction between these two flows within nixpkgs. In Obsidian’s case, Nix (not nixpkgs) simply always finds that cache.nixos.org did not build it.
ok, so that makes sense. But perhaps I could make my clarification question clearer.
Generally, when you install packages on nixos,via the stable channel for example, most of them packages come from nixos cache whereas some are pulled from source and built locally due to redistribution clauses.
My thought process is we think of cache.nixos.org as a “repository” of sorts to distribute pre built binaries based on nixpkgs derivations, though propriety software can be the exception.
That’s an approximation, but if you override any aspects of a package, then it will get built locally since it probably won’t match anything in that cache.
If you use an overlay that changes said package, then everything that depends on said package will also get built locally.
Either way, nixpkgs trusts upstreams, so if you trust nixpkgs, you better trust the upstreams of the packages you use too. What you asked for isn’t possible.
That’s fair, I think my initial question was based on an incorrect assumption/understanding of nix anyway. But that has improved thanks to this thread.
I am not really concerned on if things are built locally, initially I was confused why in this case, obsidian was not pulling from cache and how many other packages would do the same. I was more or less wondering on if the majority of packages come from the nixos domain with a few exceptions such as unfree software.
Normally yes; the exceptions would be things like overrides, as well as dependents of overlays, unfree/non-redistributable software, software marked broken or insecure, and packages that would be faster to build locally than remotely (or are impossible to build remotely as they differ across every system).
Since every computer has a different hostname (or at least there are many hostnames) caching the file in cache.nixos.org wouldn’t be useful and it’s easy enough to build locally.
They shouldn’t be broken (as in not building), as hydra tries to build them and if the build fails you can’t update the channel to the broken version of the package.
My understanding is that hydra first builds the package and in case it is successful adds the build to the cache only after that will the channel progress. Otherwise i think you are correct.
I’m referring to meta.broken, which is set to true for probably thousands of packages.
Similarly, if meta.hydraPlatforms doesn’t include your arch, it won’t be in the cache.
Etc. etc.
The contents of /etc, for example, will include hundreds of derivations that must be built locally since hydra can’t know about/store every possible config.
Or to put it simply, with a fully stock configuration and the correct channels/flake inputs, yes, this sentence is a correct statement.
No guarantees if you explicitly disable the cache in your config, use a no longer suported NixOS version, or use the release-* branches.
There is an additional edge case. Occasionally a build might also fail consistently, which will result in the package being uncached. This can happen either due to mistakes in packaging that slipped past review, or simply spurious build failures that cannot be controlled against (very rare, but has happened).
The contents of /etc, for example, will include hundreds of derivations that must be built locally since hydra can’t know about/store every possible config.
That’s a good example, that makes a lot of sense. Thanks for clarifying.
Or to put it simply, with a fully stock configuration and the correct channels/flake inputs, yes, this sentence is a correct statement.
No guarantees if you explicitly disable the cache in your config, use a no longer suported NixOS version, or use the release-* branches.
There is an additional edge case. Occasionally a build might also fail consistently, which will result in the package being uncached. This can happen either due to mistakes in packaging that slipped past review, or simply spurious build failures that cannot be controlled against (very rare, but has happened).
Thanks that really clears it up. I am trying to keep it simple at the moment so I am just using 26.05 channel plus 26.05 home manager channel btw. I am yet to try overrides/overlays anyway.