Rust In Nix Discussion Thread

Development
#1

Preface

We had @edolstra on NixOS Office Hours today to talk about Rust in Nix.

A lot of good points were raised there, and we’d like to give a space for these to continue to be discussed as it’s something @edolstra is interested in.

If you weren’t present in the hours, our stream has been uploaded on our Youtube channel.

If you have additional questions, things to clarify, or want to offer additional insight this can be the place.
Interaction is also appropriate on GitHub and the #nixos-dev Freenode channel.

Thanks :fire: worldofpeace.

12 Likes
#2

Really enjoyed it, been rust curious for about 4 years, always good to drink the memory-safe koolaid

2 Likes
#3

Funny you should specifically note memory safe as there was some discussion about the method used to interface rust and c++ was less than safe with respect to memory…

#4

I’m not an experienced Rust developer by any means, but to me, it seems one of the key points in writing good software is “do not do the things the compiler manual tells you not to do”. The fact that it works for you now does not provide me with much confidence that it will work for everyone, or in all cases, especially after I’ve asked anyone knowledgeable about this.

In the best case, by doing this, you lose nearly all the memory safety benefits of Rust, and instead introduce really hard to think about cross-language move semantics. I cannot begin to think how this will work with concurrency.

You seem to think that the only sort of undefined behavior this will lead to is a crash in case of struct incompatibility, but violating the assumptions the compiler and optimizer are making is, IMHO, a recipe for disaster. I don’t think you can predict in what ways optimized code will break in cases where you are doing something that it assumes you will never do.

Nearly un-debuggable rare segfaults and security issues are the least of it. Silent data corruption is another fun one. Especially in codepaths that you are not testing (errors, malicious input), this will be a problem. The recent bug report of error: numeric field did not have utf-8 text: �ps�� when getting cksum for � may be a good candidate already.

Since this layer constitutes the basic building blocks of the Rust code, I’m really afraid that we’re introducing a source of bugs that will haunt us for years to come.

5 Likes
#5

I’m not a Rust expert. I really like Rust though and thus was happy to hear that some Rust code had founds way into Nix. This is something that I was aware of prior to the call already. I hadn’t looked at it though. I was less enthusiastic about what I learned during the call in terms of how the C++/Rust interop actually works.

During the call some time was spent arguing about the ways in which this could break, segfault or lead to undefined behavior. I don’t have the insight to make statements about the likelihood of things breaking in horrible ways. But frankly I’m not sure I need to:

While there were different opinions on the likelihood of things going wrong and just how bad things could go wrong it seemed to me that nobody in the call argued that this was a nice and clean interface - this, as far as I understand, simply isn’t currently possible.

From my point of view we are thus dealing with tech debt. Adding tech debt can be completely fine and a good way forward. I think there are just some constraints to doing so successfully and responsibly:

  • Has to be added consciously and explicitly
  • Has to be the result of some cost/risk/benefits considerations
  • There has to be a plan of how to repay/reduce it

I would be really interested to hear opinions on the above bullet points in the context of the Rust code that has been added to Nix and the way forward. I am thinking and wondering about:

  • What could a nice, clean and stable interface for Rust look like in the future?
  • Should more Rust code be added before such an interface exists?
  • What is the benefit of having the Rust code we have right now considering the risks versus f.ex https://github.com/NixOS/nix/pull/3259 ?
  • What could be a plan for an incremental transition from C++ to Rust? Maybe the current approach was more of a spike and now it’s a matter of figuring out how to do it in a cleaner and safer way?
6 Likes
#6

A few years ago the position was that rewriting Nix in Rust is not the time best spent.

A few years ago we got rid of Perl to make contributions easier to Nix (one weird language instead of two).

I’m not against Rust, but if the end goal is not rewriting everything in Rust (which is probably a lot of work), then this feels quite backwards. Is there any reason the rewrite wouldn’t be a separate repository? Same as we did with hnix.

My questions are:

  • what changed that now spending time rewriting Nix to Rust makes sense?
  • is the goal to rewrite everything to Rust? If not, how to decide what’s C++ and what is Rust?
  • why not a separate repository?

I wasn’t able to attend office hours, so I’m sorry if these questions were repeated. I just feel like this is a big change in Nix and should probably be discussed (as RFC?).

11 Likes
#7

@domenkozar I showed up to office hours late but my impression was that eelco wanted to try it out as an experiment to see how well it would work. I don’t think this means we will go down the rust path without an RFC forever. I understand that. Sometimes you need to dive in and experiment to really understand how something will play out.

Also I agree with your opinion.

#8

I’m all for experiments, but I’m not sure what’s the goal of this experiment. Was that discussed?

#9

As mentioned I didn’t catch the whole call but I believe one of the contributing factors was that all the cool kids are writing in rust these days. As someone with 20+ years of c++ experience I understand why people don’t like the language and would want something like rust instead… but like you said:

2 Likes
#10

Is there a recording somewhere of the office hours? If not, could that become policy?

#11

It’s in the OP since the thread was opened.

also there’s this https://github.com/worldofpeace/events/tree/master/office-hours

1 Like
#12

office hours are posted here: NixOS - YouTube

1 Like
#13

I think @edolstra 's main point was that writing rust was a a lot more concise (references a one liner in rust to do a GET request vs a non-trivially large file which handles all the libcurl HTTP logic) and developer friendly. A lot of the pain with using a c++ toolchain is erased with how helpful cargo (specifically packages) is in conjunction with friendlier compilation errors; and these gains would help a lot in feature velocity and maintenance.

Then again I watched it a week ago

1 Like
#14

Similar to what has been mentioned on Reddit, another way to experiment with Rust in Nix could be rewriting smaller utility programs in Rust.

One such project that I have been interested in is nix-top, a nix-build monitor by @samueldr.
Playing with smaller utilities outside of the Nix core might be a good place to start, particularly for those (like myself) that consider themselves Rustaceans, but by no means experts.

Some additional resources that might help here:
How to not RiiR
How to RiiR

And of course, just about anything from the Servo team or Mozilla in general
e.g. Servo Engines

2 Likes
#15

I haven’t really looked at it but apparently there’s a library for safe Rust↔C++ interop

https://github.com/dtolnay/cxx

3 Likes
#16

I don’t have anything useful to add except to make a feature request- please add an option to drop to repl on error if you decide to “RIIR”

1 Like
#17

Going to post a comment that was on youtube here (not sure how to reach that person outside here)

Thiago Machado - YouTube
Thiago Machado

Seeing the Vec definition at 31:00, this is indeed dangerous.
You are using Vec as an opaque type on C++ side, but since Rust’s Vec definition is generic and has no repr C into it, the fields may have their orderings changed (for whatever reason the compiler finds it appealing to do so).

I have seen webrender using “FfiVec”, which basically redefines Vec but repr C attached to it, and it is used as a read-only data from the C++ side:
/ - mozsearchGENERATED/gfx/webrender_bindings/webrender_ffi_generated.h#688

So the Rust, before exposing an object as a Vec, first would have to convert that Vec into this FfiVec and only then expose it.

I myself have problems with that… if anyone finds a good solution, or a macro or whatever, please link me :smiley:

#18

yepp, cos nix is not the bottleneck

a better return-on-investment could be something like bazelNix
to get distributed and incremental builds by default
to make hydra more efficient = more attractive for self-hosting

and while were at it, lets replace gnu with fuchsia
to get app isolation by default

new kids want new toys, progress for the sake of progress ; )

#19

If someone wonders what a Nix in Rust would actually look like, this would be a basic example:

currently able to evaluate things like:

let
  add = a: b: a + b;
in
  add 1 2

which involves complex things like scopes, built-ins, currying, identifiers and laziness

3 Likes
#20

I apologize for digging up this thread, but just wanted to chime in that – as a hobbyist that continues to have fun learning rust but will probably never learn much C++ – I would also love to have nix in rust.

I’ve already run into issues like Segmentation fault when building ISO image with nixFlakes · Issue #4246 · NixOS/nix · GitHub that I imagine would be impossible in (safe) rust.

I’ve been humbled by how gracious the nix community has been in accepting contributions to nixpkgs and nix-darwin even from a noob like me. Thanks to the helpful compiler, the excellent tooling, and the language design, if nix were written in rust I think I could safely make (almost certainly minor) contributions in ways that I don’t think I’ll ever have the confidence or competence to accomplish in C++.

Every few months I try to investigate related projects like toros (linked above), if it’s not a problem I’ll post new finds in this thread for future searchers.

I don’t mean to step on any toes, I am incredibly grateful for all of the effort and expertise that has been poured into nix. Thank you to all the maintainers and geniuses that make it available for hobbyists like myself.

Hosted by Flying Circus.