New project: mobile-nixos [progress thread]


#1

Hi!

I have been chipping away at booting a nix-built system mobile devices. I don’t really know how to introduce the project, and it’s a bit early to have a complete demo. This builds heavily off the efforts of the postmarketOS project, and as a goal will try to keep building off that project, benefiting from their fixes, and conversely, upstreaming our work should be a goal.

The future

First, I’ll need to disassemble the pile of hardcoding and hacks currently making it boot. It isn’t that high, but building on top of it is bound to hurt! This is mainly cleaning up the initramfs, making device quirks configurable and verifying by porting an armv7 device (which I already own).

The goal, then, is to boot a proper NixOS system, not to make a derivative distribution.

This project, repository and overlay will hold anything that doesn’t have as much value (initially) upstream. It will also hold any patches and fixes specific to mobile device compatibility. While this is true now, I wouldn’t be against working on upstreaming all this into nixpkgs and NixOS proper!

Current state

This currently has been developed with only one android device on hand, which means that I may be making architectural decisions that would hinder the integration of non-android-like system. This should be considered as a bug and fixed!

Furthermore, this is a big project, and I know there are some things I’m unfamiliar with, when doing nix derivation and builds. Please comment and critique my work, this should help me getting better with nix!

Don’t hesitate to contribute! If you have other devices which are postmarketOS compatible, and are interested, hit me up here, or on IRC.

Here’s the repo:


2018-06-10

It is possible to build a fastboot compatible boot.img with an initramfs that will dump graphics into the framebuffer, start the android usb network thing, and start dropbear (with passwordless login).

This makes it possible to play around with the system, without even touching the internal storage.

2018-06-11

Got the framebuffer configured properly. It was previously configured so that Red and Blue channels were swapped. In glorious potatovision, a pair of photos showing how the temporary splash screen looks like on my device. (Sure, this could easily be faked.) Now that I have the framebuffer looking right, I’m planning to refactor the hacks for the initrd and then looking into booting into stage 2.


#2

That’s cool! Would it eventually be like not-os with custom set of NixOS modules, or would it support original NixOS modules?

Anyway, don’t hesitate to provide quarterly project updates here, we are all very excited about NixOS expansion!


#3

Ideally, NixOS proper will be used, with additional modules defined in the mobile-nixos repository for everything mobile-specific (e.g. there are services needed for some hardware on some mobile platforms).

And yes, I intend to post updates regularly. Actually, this initial post was made in the excitation of getting something working so quickly.


#4

Nice work!

Did you have a look at @telent’s NixWRT project? There are probably has some overlap in terms of maintaining small closure sizes and cross-compiling to embedded hardware.


#5

While smaller closure sizes would be better, I don’t really know what kind of overlap there would be.

A quick look at that project, and it looks quite nice! Though I think I’m going in a different direction architecturally (see next post). Unless I’m mistaken NixWRT intends to not use NixOS, but be built using nixpkgs. I intend mobile-nixos to be a full NixOS installation in the end; what I’m working on is the tooling to build bootable systems and provide necessary additional packages.


#6

2018-06-17 update

Now with more declarative device configuration.

I have been working since last Wednesday (it’s now Sunday) on (ab)using the declarative configuration system from <nixkpkgs/nixos>. I really like how I believe it will allow device configurations and descriptions to be smaller by extending and merging configurations. Right now it’s a bit overkill since there is one device, an android-system based one, but once other devices are added this will show its true potential.

This is not to be confused with the NixOS system configuration! What has been implemented is using the configuration system to build (for now) a basic stage-1 including kernel. There are no current plans for integration with NixOS configuration (as I don’t really know how to handle it). Every configurations have been prefixed mobile.*, so adding those configurations to an NixOS system later on should be possible, assuming implementation can work.

Next steps

These are not ordered chronologically.

More targets

Both x86_64 and AArch64 qemu targets should be added, this may allow some work to have a better turnaround, but will especially force the declarative configuration to be tested using a non-android system.

Additionally, I have an armv7l device (Nexus 7 2013, flo) to which I can port mobile-nixos. I’ll first need to re-install my armv7l SBC as SD card corrupted on a forced shutdown. I’m hoping the qemu-user-arm shenanigans will allow a full build of everything needed. (on rare occasions it fails due to qemu emulation discrepancies or bugs)

Framebuffer agent / menu / lib

Complete framebuffer + inputs use will be needed for the over-arching goal of using a full NixOS system including rollbacks. This means that either a NIHnot-in-house new implementation of something, or look into existing work (e.g. android recoveries).

Implement switching roots

To continue booting, switching roots (stage-2) is necessary! This shouldn’t be much of an issue, but as of right now nothing has been implemented.

Explore kexec options

This will need more documentation elsewhere. In #1 I described how mobile-nixos could be built like a tertiary bootloader, where it would be used to kexec into another kernel+initrd, allowing the NixOS system to have a full control on the kernel without touching the boot.img partition. The non-kexec method (simpler stage-1 to stage-2 switch root) wouldn’t allow either safe stage-1 changes nor safe kernel changes (without a host system to recover).

NixOS configuration and “bringup”

I expect a bit of work will be needed to get a NixOS system working. First, I’ll need a way to generate a virgin or pre-configured image, like the sd-image, but in a way that’s installable on the target device. As some devices won’t have a VT terminal (e.g. my asus-z00t), a solution will have to be found, either pre-configure an ssh or adb service, or find out if fbterm or a similar tool can be used.

Then, as part of the configuration, some common configurations should be explored. Will it be possible to merge system and userdata together using LVM so a bit more space can be used by NixOS?

Finally, what kind of issues will happen once I try getting stage-2 running?


Evening update

Well, I added qemu-x86_64 support. This isn’t as useful as it could be as networking fails to work properly, but once figured out, this will help with some stuff.

Do note that qemu has an horizontal framebuffer, this will actually help in ensuring nothing is badly hardcoded to work only on portrait devices; not all devices are portrait!


#7

This does look very cool, and I look forward to getting some time to play with it

One question, which is probably not a question you relish, but: any plans to make it run Android apps (in a sandbox or in emulation or …)? I’m mostly pretty happy with FOSS on my phone (thanks in large part to fdroid) but there are a couple of apps (e.g. my banking app) which realistically I don’t think will ever have web or linux-native alternatives. It looks like Qemu will run arm guests on an arm host, maybe that would be the way to go

NixWRT intends to not use NixOS, but be built using nixpkgs.

This is correct. My hope is to be able to get NixWRT building images for devices with as little as 4MB flash (currently at 4244k and struggling to get rid of that last little bit), I think your phones are probably a bit higher-spec :slight_smile:


#8

This is correct. My hope is to be able to get NixWRT building images for devices with as little as 4MB flash (currently at 4244k and struggling to get rid of that last little bit), I think your phones are probably a bit higher-spec :slight_smile:

And don’t get me wrong, I’m watching NixWRT from afar, so one day it’ll replace OpenWRT with it when possible!


One question, which is probably not a question you relish, but: any plans to make it run Android apps (in a sandbox or in emulation or …)? I’m mostly pretty happy with FOSS on my phone (thanks in large part to fdroid) but there are a couple of apps (e.g. my banking app) which realistically I don’t think will ever have web or linux-native alternatives. It looks like Qemu will run arm guests on an arm host, maybe that would be the way to go

I really don’t know how possible or impossible it would be. Especially apps that will want a secure device (like those that use SafetyNet). The best bet for random Android apps may be anbox, but anything that needs to pass SafetyNet or use hardware (like NFC) I’m pretty sure won’t work. That’s definitely out of scope for my little project! The ARM QEMU guests idea also has merits, though it would still fail on at least the special hardware cases.


#9

Today I found https://github.com/cleverca22/not-os which is also focused on building a initrd + squashfs


#10

I am already indebted to not-os, I took some inspiration from it when figuring our the initial build for the initrd. While I don’t know how much more it will contribute, I’m pretty sure the tests will help me figure out how to handle tests.


#11

A few questions for you:

  • What init system is it running? It looks like you are trying to reuse as much of NixOS as possible but I would kind of be surprised if SystemD works well on mobile. Reusing some of Not-OS might be a good idea because I think Runit is much more portable.

  • Have you looked into cross compiling it? I feel like this would be pretty important for this kind of thing but it looks like it’s all x86_64 right now? I’m definitely interested in this because I don’t think anyone else has successfully cross compiled a NixOS system (although in theory it should work fine). If you need any help in this I can definitely help with cross stuff.


#12

Yes

But I haven’t figured out some things yet. I think cross-compiling is still a bit in flux with nixpkgs, so I’m waiting a bit before investing actual time to figure out thing. Then, when I’ll try and figure out, I’ll be asking questions.

There is someone who did achieve a full cross-compilation of AArch64 NixOS, and I replicated the results locally too.

And finally, this project isn’t all x86_64. There are two devices currently implemented: qemu-x86_64, which is x86_64, and asus-z00t, which is AArch64. Both boot and are almost par for features (x86_64 qemu’s networking doesn’t work yet, haven’t looked at it much).

After my work day, I can look into fleshing out my needs for cross-compilation. Meanwhile, if you already have documentation handy about cross-compilation, I would like to read them :).

[EDIT] For AArch64, I have been relying on native non-cross compilation.


#13

Currently, it boots up until init. I only have a stage-1 implemented.

I had a preliminary look at how I’ll first handle switching roots, since I can’t really handle generations yet. During this 3-days week-end (holiday here) I’ll be implementing switching root.

Though, I have to say, I don’t know why systemd wouldn’t be right for mobile. Are there particular known reasons for saying that? Why would runit work better?

For now, I am focusing into using as-stock-as-possible NixOS, with added fixes and quirks for the environment. As there is no support for an alternative init yet, I’ll probably not support it yet. That is, unless it really doesn’t work well. If it doesn’t, then I’ll flesh out plans to manage an alternative init.

I, personally, wouldn’t be surprised if systemd manages fine on mobile. After all, it manages fine on SBCs with less powerful hardware, and less available memory.


#14

Cross should be fairly stable right now. You just need a line like this in your config:

nixpkgs.crossSystem = lib.systems.examples.aarch64-multiplatform;


#15

Hi, there are a couple issues I am facing with cross-compiling, but only one which I really don’t know how to even begin tackling.

The kernel I build is old, and possibly bad code from a downstream OEM source dump. That kernel, on native AArch64, builds with gcc6 and gcc7, but only the gcc6 build boots.

With native (AArch64) compilation, no issues, gcc6 is already built and I can build the kernel.

With cross-compilation, it seems gcc6 needs to be built. It seems that gcc6 won’t build as it is. This is probably since gcc6 is being built with gcc7 due to warnings/errors [-Werror] (clever on IRC seems to think so, I thought so too). gcc6 probably needs to be built with gcc6 or gcc5. Building with gcc5 seems a non-starter as it itself needs to be built (I think). Building with gcc6 seems to always turn out to cause infinite recursion. My feeling is that simply overriding the compiler from gcc6 to be gcc6 is not the right solution; it would need to be built with a host-system(?) gcc6 and not whatever it is trying to be building it with.

Once I have gcc6 figured out, I’m probably fine to at least make it work, I hope.

Any pointer, @matthewbauer? I have this branch (wip/cross) where I have currently set it to always build with cross-compilation. I gisted the log, but github (un)helpfully truncates it *sighs*. The relevant error as a comment.


#16

2018-06-25

Thanks to the help from @matthewbauer I was able to get cross-compiling working right. At the top of wip/cross, I can built for my device and boot it (fbv for splash screens currently segfaults).

I still have to entirely grok the whole nativeBuildInputs, buildInputs, buildPackages, etc… I do know about 3.3. Specifying dependencies and Chapter 5. Cross-compilation. It’s more of a matter of really understanding rather than kinda understanding.

I believe my issues with fbv are mostly caused by its non-standard build system. I’ll have to figure out how to properly pass it libraries so it builds right. This’ll be fun to debug :).

Following steps should probably be adding a QEMU AArch64 target, this should help figure out some of the issues with cross-compiling. I also should look into making it automatic; when building something that’s non-native, automatically use the right platform. Then: finish up stage-2 boots.


#17

2018-07-04

Some updates!

I have been working on this off and on, making some progress, though there’s still the elusive root switching to stage-2 to implement.

New device

I have added support for asus-flo, otherwise known as Nexus 7 2013 (WiFi), thus strengthening the boot.img build system.

It fully builds using the downstream android kernel, the mainline kernel is yet to be implemented. Why the downstream kernel? Multiple reasons:

  • Hardware support in mainline: I don’t know how it compares to downstream.
  • Allows testing a 3.4 downstream kernel with armv7.

I will eventually also add the mainline kernel, as it’s always better to have mainline support when possible.

Improvements / enhancements

BGRA / RGBA

One major nitpick I had was with how the framebuffer configuration was seemingly wrong. I may have done something wrong elsewhere, but I believe everything I did was right. What’s happening is that both my devices seem to default to a BGR(A) framebuffer format, while seemingly presenting themselves as RGB(A) framebuffers. This meant that the red channel and blue channels were inverted.

For asus-z00t it wasn’t as much an issue, as I could use fbset to change the parameters to the framebuffer. It seems that asus-flo's parameters cannot handle being changed to BGR(A)[1]. I dove into the kernel sources and “fixed”(opinions may differ) the drivers so they (1) could switch to BGR(A) (2) defaulted to BGR(A). When a config option existed to set the defaulf framebuffer format, I added another one, otherwise I simply overruled the defaults.

I would love anyone knowing about framebuffer and/or kernel development to take a look at the patches, I don’t think they could cause issues, but I don’t actually know.

I’d like to know as soon as possible if I did something bad (wrt/ framebuffers) as I otherwise will recommend this approach to “fixing” RGBA/BGRA, and making all supported devices work as initialized in-kernel, instead of fixing them during boot.

Splash screen

Other than that, and partially related, I have decided to forget about fbv and instead use ply-image; I initially didn’t use ply-image because of an error printed on the CLI, but after reading the source, and testing, it looks like it’s a warning and doesn’t affect its used for splash screen purposes.

What’s good with ply-image is that it’s an actively (not much is needed) maintained program expressly used for boot splashes. It is maintained by the chromium team. It even has support for basic animations, which will definitely help spruce up the boot.

Soon I’ll switch the temporary splashes and make some more permanent ones.

Next

Stage-2! I’m itching since I basically have a hacky stage-2 working with qemu. I need to figure out a cleaner approach that will work better for android devices.

Then, once stage-2 is working, work on making this useful, so porting stuff for nixos, trying to make it usable.


1: It may be that my patch fixes setting the framebuffer config through fbset. I haven’t tested if I change the CONFIG option back to its previous default and apply fbset instead.


#18

2018-11-19

Some updates, months later.

I have been busy with other things lately, but was just starting to look back at this.

Sadly, I have lost the use of my main development device. The Zenfone 2 Laser I was using, and keeping aside for that work, is still fine. It still works as it should, and probably would be a fine device for a couple years, but sadly, the battery isn’t. It has puffed-up a bit too much to my tastes. I’m waiting on word back from Asus as to whether I can get a genuine part. This slows down my progress as I now have zero dedicated AArch64 devices for this project.

New device

Well, I still have an AArch64 device, it’s simply not dedicated.

I have added preliminary support for the Moto Z Play, codenamed motorola-addison. Model number XT1635-02.

I say preliminary since the support isn’t quite there. First of all, the device ships stock with a 32 bit system image, and thus, Lineage OS supports it as a 32 bit system.

There is a branch in the Lineage OS kernel tree which adds support for AArch64, but it has two issues. First, it barfs in the dmesg logs an awful amount of trace logs; this may not be a big issue, mainly annoying. What’s a real issue, though, is that it will not detect the SD card. This means that working on stage-2 can’t really be done, unless I want to wipe the device.

I have some ideas to figure out things, but not much. If anyone with deep knowledge with these kind of devices can help, I’d gladly pick their brain.

Next

Not really useful to you all, but the next step here is to figure out what to do with the whole hardware situation. Maybe try some ideas. Hopefully get something figured out hardware-wise. I think it’s possible to run most android device without a battery by supplying the power directly to the pins the battery connects to, depending on the response from Asus, I might as well try that.