Can't wrap my head around derivation dependencies

Help
1

Hi there,

I’m a Nix noob. I’ve read all the nixos manual, a good chunk of the nix manual and started to read nixpkgs. But still a noob :smile:

I also have a personal server running nix, started to build my own derivation and even committed a (very small) fix for one package. Still learning a lot, but I like what I’m seeing, so definitely interested !

I knew that Nix was a one-of-its-kind, with some steep learning curve, but I’m ready to pay the price. Plus, most of the information is in the documentations so we can’t say that there’s no manual.

However, I’m stuck on a particular point : derivation dependencies. I can say a lot of things on the documentation, some part are easier to get than others, but this part… this part is too much. I read it countless time, but still can’t understand it.

So here I am. Can someone ELI5 the derivation dependencies ?

Thanks a lot for your help !

EDIT : forgot to say this. As a retribution, I commit myself to propose a patch for the documentation, with proper diagrams and explanation, in order to enhance this section.

5 Likes
2

Not sure this qualifies as explain-like-I’m-5, nor does it reflect a complete understanding of that section of the manual. However, I don’t think it’s necessary to understand it for practical purposes of packaging most software. So with that covered, here’s my attempt:

  • build is the system we’re building our derivation on
  • host is the system we’re building our derivation for
  • target is the system the output of our derivation will be able to build things for.

It’s rare for host not to be the same as target, and you’ll find that a large portion of nixpkgs doesn’t really support cross-compiling well, let alone even consider the case of “canadian cross” (where we have build, host and target all distinct).

depsAB is simply put software that will run on the A system and produce things that run on the B system.

Most of the time, all you need to care about is buildInputs (depsHostTarget) which is stuff you need to link against, and nativeBuildInputs (depsBuildHost) which is stuff you need to run as part of the build.

Hope this helps, even if it doesn’t clarify the manual section itself!

6 Likes
3

Hey,

First of all, thanks a lot for your help and your explanation.

So, if you are ok with that, I would like to separate your answer in multiple sections, and then we can discuss them. Also, I would like to rewrite it using my words (yeah, I’m that kind of guy, and also not a native english speaker.).

Cross compilation glossary

So:

  • build : where we build the derivation / package
  • host: where we’ll install the derivation / package
  • target : a server where we will run the derivation / package from the build to build the derivation / package for the host

As mentioned, target is a very specific case and in most cases, target == build. However, I think it’s important to define it because the word target is used 35 times in section 3.3 of the nixpkg documentation.

So the target is a potential third server used for Canadian Cross compilation. build generates a package that is not the final package, but a package that is able to build the final package, the one that can be run by host.

In more details:

  • build build the cross-compilation package for target
  • target get the cross-compilation package and run it. This will generate the final package.
  • host get the final package and run it.

As mentioned by @lheckemann, it’s kind of rare to see this case, but I wanted to try to explain it.

So the different use cases, by order of complexity:

build == target == host

The most common case : a server that is building for itself. So you create the derivation, build it, then host it. The target step here doesn’t exist.

build == target, separate host

A server build the final package, and host gets it and uses it. For example, a dedicated server used only for building package for other hosts. If I’m not mistaken, this is the case of the Nixos package cache: if available, our servers downloads packages from the cache, and those packages have been produced by another server.
Also, the target step doesn’t exist.

separate build, target and host

Canadian Cross, as described above. Rare case.

I’ll write a second post regarding the actual dependencies, and how they are used, but could you guys review what I have written above and correct it if needed ?

Thanks !

4

I think you (or I)'ve misunderstood the target part: a gcc with build i686-linux, host armv7l-linux, and target riscv64-linux would be built on i686-linux, run on armv7l-linux, and build things for riscv64-linux.

As far as I understand, the very existence of the target platform is only due to a weakness in gcc’s (this might be wrong) and binutils’s (most tools nowadays support multi-target, but arguably one of the most important ones, the assembler, doesn’t) design by which they can only emit code for one target platform, decided at compile time, which complicates the whole cross-compilation process quite a bit. Again as far as I understand, LLVM doesn’t have this problem and allows selecting multiple target platforms, making it possible to build a “cross-compilation god compiler” :smiley:

Might also be good to hear from the wizard who designed all this, @Ericson2314, if he has a moment :slight_smile:

5

Yeah, this is what I’ve understood it as well. Is it incorrect ? Or is there a part in my previous post where I may let you think that I did not understand it that way ?

6

I’m fairly new to this too, but I think you’ve confused host and target. Target is only meaningful when the derivation we’re considering produces software.

Back to the GCC example, building GCC on machineA (build) that will be shipped to machineB (host) which will run it in order to produce a soft that will execute on machineC (target).

BUT if you’re talking about anything else than an old compiler, let’s say firefox, you’ll simply ignore the target platform as it becomes irrelevant (firefox will not produce software).

7

So here’s the source of truth.

There are three system names that the build knows about: the machine you are building on ( build ), the machine that you are building for ( host ), and the machine that GCC will produce code for ( target ).

So it seems that I swapped the host and the target:

  • the host is where the compiler build by the build is sent. host will then execute the cross-compiler to produce the binary
  • the target is the where the binary will run

If we take the schema from wikipedia:
https://upload.wikimedia.org/wikipedia/commons/c/cb/Example_of_Canadian_Cross%2C_scheme.svg

Server A is build, server B is host and server C is target ?

8

To be clear, if you want to use nix(os/pkgs/…) and don’t need cross-compilation, you don’t need to care about such details :wink:

9

@vcunat: let’s talk about that then. What’s the bare minimum you need to know to be able to build packages ?

10

This should have you covered for most of your usage and contribution to nixpkgs.

1 Like
11

So buildInputs is the list of package dependencies for the final package to run, and nativeBuildInputs the list of package dependencies for the final package to be built, right ?

Let’s take an example then: emby package.

I have several questions here:

  • there’s a bunch of packages put as argument of the function. They are then used in different phases as explicit dependencies. So do we need to put them as buildInputs ?
  • If not, is it a best practice to put all dependencies, even explicit, in buildInputs ?
  • Why the developer put the runtime dependecies in propagatedBuildInputs and not directly in buildInputs ?

Once again, thanks a lot for your help !

Cheers,

12

You don’t have to. buildInputs provides standard facilities to “build against” that package, e.g. for C that would be extending include and library paths. Packages can customize what happens when they’re put into build inputs, so naturally different languages/frameworks may have different approaches. So, you may just use e.g. direct ${package} references in the build scripts, but you will get just paths to the packages and not any “other goodies”.

Inputs propagated from emby will be “automatically added” to buildInputs of any package that adds emby into its build inputs (it behaves transitively). That can be useful sometimes, but I don’t know why it was used in this particular case.

1 Like
13

To be honest, I’m not sure to fully understand what you called “other goodies”.

If I would risk an example:

  • for emby, you need to call the executable mono, so here there’s an explicit dependency. No need to put mono in buildInputs
  • if the software package requires a specific library for example, then you’ll need to put it in buildInputs in order to have a package that can work.

Correct ?

14

The goodies depend on the package, e.g. visibility of build inputs by pkg-config is added, if you add pkgconfig to your (native) build inputs. Mono might also have some hooks that help you building with it (now or in future), and you will probably get these only if you add it into build inputs…

15

Goodies also include having a bin dir (if present) added to $PATH, and also quite importantly the setting of NIX_CFLAGS and NIX_LDFLAGS such that the cc wrapper makes the compiler search the appropriate paths for headers and libraries as part of stdenv.

16

Yes, though I avoided mentioning $PATH because that goodie gets a bit complicated for cross-compilation…

17

I also had trouble to understand this part of the manual and it is much more confusing than informative for most people IMHO.
I really think that the manual should be simplified, and move all these stuff in an “advanced” section that is really interesting for very specific case.

3 Likes
18

Agreed.

I’m starting to build new packages and nixos modules, so I played around with buildInputs. It seems to do the trick, and as mentioned by a lot of people in this thread, you don’t really need anything else.

This part is extremely confusing, given the fact that:

  • it’s focused on cross-compilation, where most of people are doing a package where host == target
  • the algorithm part with the -1 / 0 / +1 offset is ridiculously complex
  • the sheer complexity of this part of the documentation could potentially drive away people to build their own packages

Bottom line:

  • mentioned the buildInputs and the nativeBuildInputs and explain them
  • put every other attributes in reference
  • put the algorithm explanation in an Appendix, as “Advanced”

EDIT : typo

3 Likes
19

How can I control whether my package does this, when added to buildInputs?

20

I do not think you can control that. Every package transitively in your buildInputs that contains a /bin directory will get added to the (HOST_)PATH:

1 Like