Done: reMarkable1 cross-compile toolchain for Nix

Hi all,

I’ve been working on on off the last few weeks on adding cross-compilation support for the reMarkable 1 tablet. I would greatly appreciate people trying out the toolchain and:

  • Building various community projects, especially the Qt ones which might require a closed-source Qt library to be packaged as well
  • (if you have a tablet), test the compiled binaries

The company provides a toolchain from which a cross toolchain can be made, see it packaged into an overlay. When I adjusted the PR to not depend on the company provided toolchain and instead build everything from source, it seems that the linking behavior is changed and binaries wouldn’t run on the tablet, so I would have to either statically link everything or install Nix on the target.

Would it be possible to get dynamic linking to work again, perhaps linking against the libraries that are in the official toolchain?

Regards,

siraben

9 Likes

I’m very interested in making open toolchain work, however I only have rM2 so that is what I am targeting.

Looking at your open PR it seemed easy enough to modify dynamicLinker so it points to proper path on device. However that is not the case as resulting binaries have set linker to: /usr/lib/ld.so.1 for some reason.

Right now I cannot figure out why that is the case.

My branch is here: GitHub - jkopanski/nixpkgs at remarkable2

Did you had any luck?

2 Likes

It was also easy to modify dynamicLinker on my PR to /lib/ld-linux-armhf.so.3, however the resulting binaries cannot be run on the reMarkable tablet. Here’s the result of running file on hello. It seems to look for a nonexistent linker in the Nix store.

root@reMarkable:~# file ./hello-2020-11-16 
./hello-2020-11-16: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/3s2x09bbl1xss1lcx7a6dbw69zmip3gx-glibc-2.32-armv7l-unknown-linux-gnueabihf/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped

Have you tried compiling it statically yet? That is what I was planning for my reM1 but haven’t tried that yet. You mentioned it being an option in the OP.

Yes, compiling it statically works. IMO we should still try to dynamically link as much as possible with on-device libraries.

It seems that is using standard nix linker. If you look at the dynamicLinker if-then-else tree be sure to put remarkable condition before generic arm one. I did that with special isRemarkable flag.

However even with that my programs compile with linker pointed to nonsensical:

file /nix/store/xrm21bz3p2na8yxckg4g89b5kcf754jy-hello-2.10-armv7l-unknown-linux-gnueabihf/bin/hello
/nix/store/xrm21bz3p2na8yxckg4g89b5kcf754jy-hello-2.10-armv7l-unknown-linux-gnueabihf/bin/hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, for GNU/Linux 2.6.32, not stripped

I have no clue where it comes from.
Inspecting binutils-wrapper it seems that there is correct nix-support/dynamic-linker file with proper contents.

Yeah, I made the right change to the dynamic linker, see this commit. Do we need to override something else? For instance, using the proprietary toolchain, this can be done.

It seems that you forgot to set the flag in remarkable1 example system. Take a look at my example

I saw, that proprietary toolchain fix, but it does set the proper contents for nix-support/dynamic-linker file. Theoretically the same happens when we override dynamicLinker. For some reason tho its getting ignored?

Ah, right. I just set the flag now and got

root@reMarkable:~# file hello-2020-11-17
hello-2020-11-17: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, for GNU/Linux 2.6.32, not stripped

So we have the same result, not really sure what to do with that linker and where it comes from.

Here are the only places in nixpkgs that contains lib.so.1. I also checked if the if-else tree was properly taking that branch by raising an error, and it was correct.

$ ag "ld.so.1"    
pkgs/development/compilers/fpc/mark-paths.patch
35:-  const defdynlinker='/lib/ld.so.1';
36:+  const defdynlinker='@dynlinker-prefix@/lib/ld.so.1';
55:-  const defdynlinker='/lib/ld.so.1';
56:+  const defdynlinker='@dynlinker-prefix@/lib/ld.so.1';
88:-  const defdynlinker='/lib/ld.so.1';
89:+  const defdynlinker='@dynlinker-prefix@/lib/ld.so.1';

pkgs/build-support/bintools-wrapper/default.nix
63:    else if targetPlatform.system == "powerpc-linux"  then "${libc_lib}/lib/ld.so.1"
64:    else if targetPlatform.isMips                     then "${libc_lib}/lib/ld.so.1"
66:    else if stdenv.lib.hasSuffix "pc-gnu" targetPlatform.config then "ld.so.1"

I think this might work;

But looking at the value for postFixup this doesn’t make sense, since the exact same logic is there already.

diff --git a/pkgs/build-support/bintools-wrapper/default.nix b/pkgs/build-support/bintools-wrapper/default.nix
index 06ae699d4ad..2f3e106bcd9 100644
--- a/pkgs/build-support/bintools-wrapper/default.nix
+++ b/pkgs/build-support/bintools-wrapper/default.nix
@@ -256,6 +256,11 @@ stdenv.mkDerivation {
       echo "-macosx_version_min 10.12 -sdk_version 10.12 -no_uuid" >> $out/nix-support/libc-ldflags-before
     ''
 
+    + optionalString targetPlatform.isRemarkable ''
+      echo "/lib/ld-linux-armhf.so.3" >$out/nix-support/dynamic-linker
+      echo "-dynamic-linker /lib/ld-linux-armhf.so.3" > $out/nix-support/libc-ldflags-before
+    ''
+
     ##
     ## User env support
     ##

if you inspect $out/nix-support/dynamic-linker correct value will be filled there already, since that is determined by this big if ... else dynamicLinker part

I think this value /usr/lib/ld.so.1 comes from binutils. I think that is the default set if the one passed as parameter is not found.

binutils source

Indeed, the thing we passed was invalid, so it must have fallen back on that.

In general, this patch is doing too much / trying to hard:

  • We already have generic logic getting the interpreter for 32-bit arm linux.
  • No need to patch anything if we don’t hit the default anyways
  • No need for prebuilt tools

All that remains is the zero-gravitas kernel config and example platform that uses it.

Initial implementation of remarkable1 cross-compile by siraben · Pull Request #102503 · NixOS/nixpkgs · GitHub has been merged! Huge thanks to @Ericson2314 for helping provide feedback. For any reMarkable users out there, here’s how to use it.

  1. Ensure that your nixpkgs channel is up to date (or equivalent with niv and flakes). You can check if it can cross-compile to reMarkable 1 by running
$ nix eval -f '<nixpkgs>' 'lib.systems.examples.remarkable1'
  1. Create a non-root user on the tablet, e.g. useradd siraben && passwd siraben. Ensure that you have passwordless SSH set up by using ssh-copy-id.
  2. The root partition on the tablet has very limited space (22 MB), so, as root, mkdir -p /nix /opt/nix && mount --bind /opt/nix /nix. The bind can be made persistent by adding the following line to /etc/fstab
/opt/nix /nix none bind,nofail 0,0
  1. Install Nix on the device. I used GitHub - DavHau/nix-on-armv7l: Nix without NixOS on arm32v7 / armv7l / Raspberry Pi OS / Raspbian and decompressed the tarball in the releases and ran the install script.
  2. Using nix-build and nix-copy-closure, one can cross-build from their machine and transfer it to the tablet, like so. The NIX_SSHOPTS is needed because nix isn’t available unless .profile is sourced.
$ NIX_SSHOPTS="source .profile;" nix-copy-closure --to siraben@10.11.99.1 "$(nix-build -f '<nixpkgs>' -A pkgs.pkgsCross.remarkable1.hello)"

Happy hacking!

4 Likes

So glad to find someone doing this. I’ve had my eye on the Remarkable for a while, but they never answer my questions on whether the system is open to outside development and what tools will be available. I think I’ll buy one now!

1 Like

Also see GitHub - reHackable/awesome-reMarkable: A curated list of projects related to the reMarkable tablet. Worth noting reMarkable company provides toolchain and is responsive on their linux repo, but cannot provide support for third party mods.

1 Like

Based on this work, reMarkable 2 cross-compilation has also been added. Cross-compile configuration for reMarkable 2 tablet by tadfisher · Pull Request #105432 · NixOS/nixpkgs · GitHub

2 Likes

Just wanted to say thanks, got my reMarkable 2 today and immediately got Nix to work on it. Feels like living in the future :slight_smile:

4 Likes