systemd stage3


systemd (Photo credit: James O’Gorman)

In my quick review of systemd, I left a few points hanging for further elaboration.

I mentioned that there are no official stage3 tarballs with systemd. Without them, the only way to get a systemd system is to upgrade via the guide.

As I get used to it, I’m going to need a way to install systemd repeatedly and consistently. I have therefore created my own stage3 tarball.



I have a gist with the various scripts that I wrote. Consider this blog post the README.

In essence, a stage3 tarball is a basic rootfs directory structure with just enough binaries and libraries to install more stuff. Add a kernel, bootloader, some must-have packages and configuration files and you have a bootable snowflake.

They’re easy to make, but care needs to be taken to make them sufficiently generic and small enough for distribution. Mine comes in a just over 98MB.

Some things to consider:

  • You don’t have to mess with your real rootfs. –root and –config-root (control the $ROOT and $PORTAGE_CONFIGROOT variables respectively) are good ways to create new rootfs directory trees. This behaves much like debian’s debootstrap or yum –installroot.
  • Use binpkgs, we don’t need full build logs or compilation artifacts. Emerging with packages also requires a smaller dependency set (no build dependencies on the target), so less packages need to be installed.
  • This means that creating the final target system occurs in two steps, one –buildpkg but not –usepkg, then again with –usepkg. These are the chroot-prepare and chroot directories.

Pre-emerge tricks

The very minimum that emerge needs to know about the target, is the make.profile symlink. This is at ./etc/make.profile relative to $PORTAGE_CONFIGROOT and points to a profile in $PORTDIR/profiles.

root@localhost ~ # ls -l chroot/etc/make.profile
lrwxrwxrwx 1 root root 46 Aug 31 22:10 chroot/etc/make.profile -> /usr/portage/profiles/default/linux/amd64/13.0

Here, emerge (the program itself) and the portage tree (/usr/portage) are located on my real filesystem. I’m actually doing this all in a virtual machine dedicated to building gentoo root filesystems, so “real” is a subjective term.

If I wanted to use the defaults, I could create a naive stage3 tarball in two commands.

# emerge --{config-,}root=chroot world
# tar xzf stage3-naive.tar.gz -C chroot .

Add systemd

To force systemd, I have changed the global USE flags to “-consolekit systemd”, so that packages will be compiled with systemd awareness, and added sys-apps/systemd to the world set.

I also added net-misc/dhcpcd, sys-apps/dbus and sys-apps/iproute2 to the world file because they are useful to have and not part of the system set. I have a larger list of world dependencies that include app-editors/vim, app-portage/eix, sys-kernel/dracut (and keywords to unmask it), sys-boot/grub plus some portage, filesystem and networking tools.

Compiling packages

Create the binpkgs, saving them to a PKGDIR somewhere. Defaults to /usr/portage/packages.

# EMERGE_FLAGS="--buildpkg --update --jobs"
# mkdir "chroot-prepare" "chroot"
# tar xavpf stage-template.tar.gz -C chroot
# emerge $EMERGE_FLAGS --config-root=chroot --root=chroot-prepare \

This is where most of the time will be spent. It is good to have a strong multicore machine with enough RAM for this stage. Add –jobs (unbounded) and set MAKEOPTS (in make.conf) if you can without crashing the build host. VMs are really useful for this eventuality.

We could tarball up chroot-prepare, but it includes a few extras that we won’t necessarily need to get a working stage3. It also misses out something critical that exposes a bug in the portage tree.

Emerge proper

# emerge $EMERGE_FLAGS --usepkgonly --config-root=chroot --root=chroot \

Ideally, this command would work. However there are a few bugs in the area where sys-apps/dbus (a dependency of systemd) will not be installed correctly. It has a pkg_setup phase that calls enewgroup and enewuser from the user.eclass eclass. Which, in their current incarnations are not ROOT aware, preventing dbus from starting at boot.

The gist includes a patch to the eclass that I should attempt to get merged. Given the previous attempts by others, and that this only works for recent linux distros I won’t hold my breath.

The other half of fixing dbus is that the required programs to call enew{user,group} also require files provided by sys-libs/glibc (for /usr/bin/getent), sys-libs/pam, sys-auth/pambase, sys-apps/shadow and sys-apps/baselayout.

Thanks go to dev-util/strace (and following which open() calls failed because pam was not yet installed) and qfile (of app-portage/portage-utils) for hunting down the needed packages. I’m not sure what the proper way to fix this is since my patched eclass requires permission checking in the chroot, not the dbus ebuild itself.

This knowledge lets us create a working stage3.

# DBUS_DEPS="sys-libs/glibc \
    sys-libs/pam \
    sys-auth/pambase \
    sys-apps/shadow \
# emerge $EMERGE_FLAGS --usepkgonly --config-root=chroot --root=chroot \
    --oneshot --nodeps $DBUS_DEPS
# emerge $EMERGE_FLAGS --usepkgonly --config-root=chroot --root=chroot \

And finally,

# tar cJf stage3-systemd.tar.xz -C chroot .



  1. Pingback: systemd | Fragments
  2. giorgio

    hi! …tryin’ your systemd-stage3 in virtualbox-gentoo host. I can’t change root passwd so I can’t login after reboot. any advice? thanks!
    ps now emerging sudo, let’s give it a try.

  3. bencord0

    Typically I blank out the root password line in /etc/shadow, then the only way to login is using the console tty0, i.e. if I have physical access. Gentoo doesn’t ask for a password if the corresponding line in /etc/shadow is empty. I then get a script (something like cloud-init) to pull a ssh key from somewhere on the network.

    You might have to change the password out-of-band. Try mounting the rootfs in another live environment and reset the password by “chroot /mnt/gentoo passwd”. Another trick is to append your own /etc/passwd to the tarball before you use it.

    Also, you might want to check if the /root/ directory exists, I dont think I emerged any packages that create it.

  4. bencord0

    I’m starting to get curious about problems that people are seeing.

    > same problem. after chroot and passwd , gives module error
    What kind of module is erroring? Kernel modules are not included, but missing Python or PAM modules might be symptomatic of a missing dependency, or some other hackery/bug with the way packages are installed. It would be good to figure this out.

    > also cant build stage3-systemd x86 stage-template.tar.gz missing
    I have updated the gist [1] with some tweeks after the initial blog post. There is an outline about what to include in the stage-tempate tarball. I didn’t upload one since it is an easy thing to recreate (or possible integrate into the script).

    Finally, I should also point out that it has been over a month since I made that tarball. The portage tree has moved on and the majority of packages have been version bumped. You really should be making your own stages as and when you need them.
    Once I get a bit more practice, I plan to start doing automated builds. How that will work I still haven’t decided, but it probably warrants a new blog post to itself.


    • bencord0

      That makes life complicated.

      I was under the impression that systemd still relys on gettys to spawn login which (via pam) verify against /etc/shadow.
      I personally have been running “sed -i -e ‘/root/ s/*//’ /etc/shadow” and remove root’s password. This has the effect of denying remote root login and allowing me not to let others do an offline attack on my shadow file. It should still permit root login on any terminals listed in /etc/securetty. That is, based on my experiance in openrc.

      I don’t think systemd fiddles with that, but I could be mistaken. If you figure out which file systemd is actually using, it would be a great help.

    • bencord0

      Apologies if I come across mean when I type this, but I have included instructions about how to reproduce my efforts.

      The gist has all the things that I am using to create a stage3. Extract it onto a filesystem, add a kernel and bootloader and it is just enough to boot and get basic connectivity.

      This procedure can be extended to creating entire stage4s. The easiest way would be to add packages to the world file in your own stage-template tarball, then run the commands to build the fat stage tarball again. I only included systemd, dbus, iproute2 and dhcpcd because that was the minimum that I needed to get ping working.

      Other good packages that could be bundled include sys-fs/btrfs-progs, sys-kernel/dracut and sys-boot/grub:2.
      That, along with a kernel (and modules) should be enough to bootstrap a system. An idea for another blog post perhaps.

    • bencord0

      Ahah! Missing PAM module.

      I copied over the strace binary from one of my hosts and dug around to see what passwd was searching for.

      # strace passwd 2>&1|grep open|tail -n 20
      open(“/etc/pam.d/system-auth”, O_RDONLY) = 4
      open(“/lib64/security/”, O_RDONLY|O_CLOEXEC) = 5
      open(“/etc/”, O_RDONLY|O_CLOEXEC) = 5
      open(“/lib64/tls/x86_64/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No suc
      open(“/lib64/tls/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file
      open(“/lib64/x86_64/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such fi
      open(“/lib64/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or d
      open(“/usr/lib64/tls/x86_64/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No
      open(“/usr/lib64/tls/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such f
      open(“/usr/lib64/x86_64/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No suc
      open(“/usr/lib64/”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file
      open(“/etc/localtime”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directo
      open(“/etc/pam.d/other”, O_RDONLY) = 3
      open(“/lib64/security/”, O_RDONLY|O_CLOEXEC) = 4
      open(“/etc/passwd”, O_RDONLY) = 3
      open(“/etc/passwd”, O_RDONLY) = 3
      open(“/etc/passwd”, O_RDONLY|O_CLOEXEC) = 3
      open(“/etc/shadow”, O_RDONLY|O_CLOEXEC) = 3
      open(“/etc/passwd”, O_RDONLY|O_CLOEXEC) = 3
      open(“/etc/shadow”, O_RDONLY|O_CLOEXEC) = 3

      That solves the mystery. sys-libs/cracklib is missing. That provides such files as:
      /usr/lib64/cracklib_dict.* and /lib64/ etc.

      Emerge sys-apps/cracklib, or just copy those files from another host.
      This should have been brought in as one of the dependencies of sys-apps/shadow (USE=cracklib).

  5. martin hyska

    can y posts sample of stage3-systemd amd64, i really need it. have big trouble to change it from openrc

  6. martin hyska

    Thx, but how can I make stage3 for x86? I try to go step by step and dont know how make symlink for make.profile. Whz use ls -l ? How is the valid path for x86? I dont find any such path on my comp.

    • bencord0

      I’ve created a script to create the stage-template tarball.

      There’s a line that sets the portage profile,
      ln -sf /usr/portage/profiles/default/linux/amd64/13.0 stage-template/etc/make.profile

      Change the “amd64” to “x86” and that should be enough for your purposes. I don’t have an x86 32-bit kernel to build or test this however.

      “ls -l” is used in the main post to show the symlink’s existence. It is created with “ln -s”.

  7. martin hyska

    Just I am tring to build it. Maybe It will be good step to merge scripts to 1 file aand add architecture test.

    • bencord0

      Feel free to do that.

      I personally think that the original script does too much. It handles chroot creation, both the preparatory emerge and the real emerge, final touchups and packaging.

      Those should be separate steps that can be run in isolation. Something also needs to be done about uploading to a storage server afterwards and further testing the final tarball.

      I also want to research into bundling a kernel and other mechanics to boot. Eventually leading to a consistent environment to make rebuilds.

  8. martin hyska

    Seem that on x86 cant build it. I get several errors. openssl give me +bindist use,dracut give me testing repo. There was also ude blosked bz systemd. Now I am treing testing repo.

  9. martin hyska

    Also give me emerge failed systemd-208-r2. In log some econf failed. Which version do y use for build?

  10. martin hyska

    On official web there wrote, that dbus is compiled with -systemd, can I set use manualy in world file?

  11. bencord0

    I have no idea what the state of the x86 tree is. I suspect that you will probably need to be using ACCEPT_KEYWORDS=”~x86″, or just add them to /etc/portage/package.keywords. I can’t really be helpful in this area anymore since I abandoned 32-bit systems years ago.

    openssl’s bindist useflag is there to enable/disable elliptic curves, it’s harmless either way, but impacts distribution of compiled binaries, and has to match the flag set in openssh. dracut is still in testing, but grub2 is now stable.

    Systemd is stable, there really shouldn’t be blockages, but you will have to resolve these yourself. DBus will need the systemd USE flag. You can’t boot without it. You can set USE flags in /etc/portage/package.use.

    I haven’t hardcoded versions in my scripts. It is my hope that it should work on the latest portage tree.

  12. martin hyska

    And minimal packages to boot are dbus and systemd only? I also get some warning that cant find kernel sources from systemd, but dont know, if its important. Gentoo use suck me, If i cant run, i return back to archlinux

    • bencord0

      You need more than systemd and dbus. You need things like PAM to let you login, gettys to setup terminals, bootloaders, kernerls and eventually init (provided by systemd). Gentoo defines a system set, the minimum packages that are needed to create a self sustaining system that can install more stuff later on.

      In the gentoo world, systemd isn’t ready yet. As it stands, if you pay careful attention, and do the steps yourself, it is possible to boot using systemd and get the basic services working.
      The traditional openrc based init mechanisms are what I would recommend for current usage in Gentoo. They are well tested, offer much more flexibility in the transitional period as the Gentoo ecosystem accepts systemd as an alternative. The choice provided by alternatives are the central reason for USE flags.

      If Arch is useful to you, and provides a systemd system that “just works”, then you should stick with what you know. Fedora also provides a first-class environment for systemd.
      In Gentoo, systemd might never be the primary init system, but efforts will be made to make it a feasible alternative. In an idea world, it should be possible to enable USE=”systemd”, recompile the system and reboot. Gentoo is not there yet, but there has been a lot of progress recently. I have proven that systemd is a viable target within the Gentoo meta-distribution, even if it isn’t fully baked yet.

      I am not a Gentoo developer and cannot speak for Gentoo itself. But as a user of Gentoo for many years, I can give advice and pointers on what I have found useful and report back to the Internet using my blog here.

      Technical notes: In theory, the minimum needed to use Gentoo is the system set. Adding USE=”-consolekit systemd” brings in the systemd alternative for use.
      Grub2, the linux kernel and dracut are also nice things that play well with systemd. The systemd install is looking for kernel sources under (chroot)/usr/src/linux provided by one of the sys-kernel/* packages. I am not installing one, but instead providing one externally. A choice I have found difficult to acheive in other distributions.

      Is there a reason you’re using Gentoo for your tasks in the first place?

  13. martin hyska

    I was using arch and there will be problem with downgrade and no versioning. Often I update all packages and system freeze, just testing repo seems ustable than on others distros.I want to try gentoo, love compile from source, but dont like use mismash. I would like gnome3 on my htc shift and red, that is developed with systemd and gnome-shell. maybe it good to install with. Just i am try it. But can install it from original stage3, give me some udev problems. I want to try it without udev and LFS is way. Now this scrypt is strange for x86 it emerge me 2 version of systemd 204 and 208. I dont know why. maybe thats problem.

  14. bencord0

    Using cutting edge software (systemd, gnome3) that necessitates extra repos ontop of what the main distributions have already tested is always going to be a dangerous task.
    I must admit, one of the reasons that I like gentoo is that when things do break, at least there is a way to fix things without requiring a full system reinstall. See my Gentoo FTW post on other reasons.

    Compiling from source will enivitably bring the mismash. Apparently, most people don’t care for the customisations. There are plenty of binary packaged distros for them.

    If you need a Gnome3 desktop, then a quick google brings up[1]. You probably want to be playing with a distro that the developers are using, and has a rich testing environment. Gentoo, LSF and potentially even Arch are way too configurable to provide a consistent experance that has had enough testing.

    This makes me suggest that Fedora is the way to go. It is a well tested, systemd/gnome3 based distro. The documentation for the most common user tasks are excellent, if a bit lacking if you want to do anything too deep (such as recompiling the system).
    I should note that Fedora is the target platform for systemd, so all features should work well in that environment. I don’t know what platform the gnome3 devs are using.

    Sidenote: For KDE fans, OpenSUSE is the way to go.


  15. martin hyska

    Y are using some X on gentoo? I love tune it and dont like system like ubuntu, which do everything alone. Love philosofy of gentoo and i would like to see it on my htc shift

    • bencord0

      Can you emerge that version of systemd onto a normal x86 install of Gentoo? Those look like ebuild problems.

      rm “${D}”/usr/share/man/man8/{halt,poweroff,reboot,runlevel,shutdown,telinit}.8 || die;

      The ebuild is trying to remove files that don’t exist. I haven’t seen this problem before and I suspect that this is probably a bug in the ebuild itself. It might be worth filing it on

  16. martin hyska

    No, i cant emege. I try different versions and its same. In normal install is the same error. Once I have it installed, but have many problems with previous udev install and I format it. Now I cant get it again.

  17. bencord0

    I’m using latest in ~amd64, sys-apps/systemd-208-r2.
    Udev and systemd can’t be installed at the same time. The udev codebase is part of the systemd, so will block systemd from being installed at the same time.

    I have no idea what to do about the ebuild errors that you are seeing.

  18. bencord0

    In short, no. You will need to setup a toolchain and cross compile. Alternatively, boot into a x86 installation of Gentoo and compile it natively.

  19. martin hyska

    Seems, that gentoo distro goes wrong. mininimal x86 live cd has no emerge. Live dvd cant run scrypts due to root permitions. Last stage3 dont emerge –sync dont work. Many and many things does gentoo unusable, but its philosophy is great

    • bencord0

      I don’t think you’re using them properly. The minimal environments are just enough so that you can boot, setup networking, download stage3s, setup bind mounts and chroot. The stage3 installation process has you emerge new software from inside the chroot.
      The Full hybrid dvd drops you into a user kde session. You need to open up a konsole and gain root access.
      If “emerge –sync” is failing due to network issues, but you know that it works outside of the chroot, then you forgot to tell the chrooted envionment where to do DNS lookups. A mistake that I often forget. One solution is to “cp -L /etc/resolv.conf /mnt/gentoo/etc/” from outside of the chroot.

  20. martin hyska

    Just try to fresh install and now szstemd install pass, but fail lvm2. Which use is otional set to compile? Can y post your make.conf

    • bencord0

      root@localhost ~ # cat /etc/portage/make.conf
      DRACUT_MODULES=”btrfs lvm systemd”
      FEATURES=”buildpkg parallel-fetch parallel-install”
      USE=”-consolekit device-mapper systemd”

    • bencord0

      This occurs when you’re upgrading an openrc system to systemd. A lot of the deps need to be told to use systemd. Best way is to set them under /etc/portage/package.use on a case by case basis.

  21. martin hyska

    just seems than my stage3 x86 nearly to work. one big problem is some profiles error, which i get on emegre –sync. eselect profile list give me no profile. make.profile shows to path, which not exists. how can i add some profile?

  22. martin hyska

    hmm, there is bug in portage 2.2.7 cant create /usr/portage/profiles/repo_name file. I also cant compile with my build. Dont know, where is problem. GCC is part of which package? Threre is no emerege gcc in your script.

    • bencord0

      repo_name is created when you sync the tree of the host machine.
      GCC is pulled in as part of the @system set.
      I would prefer not pulling in anything using –nodeps, but a chrootless install is horriby horriby broken.
      As is a cross compiled install-from-blank, Gentoo does it’s best, but some packages like python really don’t like being crosscompiled. (The resultant python binary is used as part of the build process, which is really damn useful, except that it won’t run on the host).

      I actually don’t think that GCC is the problem. I’ve been poking around a bit, and it appears that sys-devel/binutils doesn’t run eselect properly in post_install when you give an alternative ROOT.
      $ ln -s /usr/x86_64-pc-linux-gnu/binutils-bin/*/* /usr/bin/
      should resolve that issue, or just emerge binutils again (binary pkgs ftw!).

      I haven’t figured out an elegant solution for that yet.

  23. martin hyska

    can y make some user friendly build? just can call it stage4 with mc istead of vim, with lsusb and wifi tools workinkg. it will be good step to use it well

    • bencord0

      I could, but I won’t.

      It’s a slippery slope to me essentially creating my own Gentoo based distro. And I really don’t want to do that either.
      My experiments with systemd ended when I managed to create a booting system.

      For a full Gentoo experiance that uses systemd…
      1/. Wait for sys-apps/systemd to mature a bit in Gentoo and there will eventually be proper guides on Hopefully, without hacks.
      2/. Wait for someone who knows what they are doing and is working withing the namespace to make stage3s.
      3/. Use It’s a very interesting project and they have far suprior build scripts and infrastructure than I do.
      4/. Experiment yourself.

    • bencord0

      :) I would, but I haven’t really setup a donations scheme yet.
      I’m actually planning on moving this blog to be hosted under my own domain. Maybe I’ll setup a system then, but I haven’t decided on a system yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s