Build and install Sysbox from sources on Gentoo

[j-landru]

Build and install Sysbox from source on Gentoo

Installing Sysbox on unsupported distro is not yet detail documented, as it took me a while to make binaries on Gentoo, I share here some personal notes about building Sysbox from source on Gentoo, hoping it can be useful for others or maybe receive some feedback or guidelines.

Context

I'm working on polymorphic installless network and system lab all in one images ( see abcd ) where some lab model will be containerized on rootful docker runtime. The whole plateform image itself could be virtualized on HVM hypervisors (KVM, virtulabox, or others) as well on common container runtime (lxd, podman or docker and possibly k8s) in rootless mode. So two virtualization levels are in place, the outer (HVM or rootless container runtime) and the inner level (nested) as rootful docker for Gns3 or Kathara network mockups. When the outer virtualization context is docker we face running rootful docker inside rootless docker environement. In that case sysbox-runc seems more robust way compared to native rootless docker. As my daily working environment is Gentoo, installing Sysbox on that distro is a must.

Docker and Make files

Like most application or system binaries, Sysbox build is Makefile based, and as stated in #718 Sysbox dev legitimately choose to "build exclusively inside a Docker container (to keep the host clean); however the main reason to have per-distro Dockerfiles is not so much for building, but rather to also test Sysbox inside the container, and the container to mimic the specific distros where Sysbox runs".

Dockerfiles

Inspired by the Sysbox Debian-like dockerfiles, the three dockerfiles below are a translation attempt replacing Debian packages by their Gentoo ebuild counterparts. These images are built upon gentoo/stage3:amd64-systemd, the official Gentoo stage reference image based on systemd.

_

Disclaimer : The three dockerfiles needed to produce sysbox-runc binary, displayed in that post, are not yet tested at this stage there's no warranty about successful container run of these images.
_

Nota : : Building from alternate openrc init base image gentoo/stage3:amd64-openrc maybe possible but has not been invesitgated yet.

Nota : Manual docker install inside the image using get-docker.sh replaced by docker ebuild as this install script is not Gentoo friendly.

Nota : Sybox build needs to be run on rootful docker env as some phase are run with --privileged mode. So switch your docker env to rooful, pointing to the right daemon.json conf file and restart Docker runtime before building.

• The sysbox/sysbox-dockerfiles/gentoo-stage3-amd64-systemd/Dockerfile dockerfile

#
# Gentoo stage3 + Systemd
#
# Description:
#
# This image serves as a basic reference example for user's looking to run
# ArchLinux + Systemd inside a system container, in order to deploy various
# services within the system container, or use it as a virtual host environment.
#
# Usage:
#
# $ docker run --runtime=sysbox-runc -it --rm --name=syscont nestybox/archlinux-systemd
# $ docker run --runtime=sysbox-runc -it --rm --name=gentoo-stage3-amd64-systemd gentoo/stage3:amd64-systemd
#
# This will run systemd and prompt for a user login; the default user/password
# in this image is "admin/admin".

FROM gentoo/stage3:amd64-systemd

# remove man pages and locale data
#RUN rm -rf /archlinux/usr/share/locale && rm -rf /archlinux/usr/share/man

# The following systemd services don't work well (yet) inside a sysbox container
# (due to lack of permissions inside unprivileged containers)
RUN systemctl mask systemd-journald-audit.socket systemd-udev-trigger.service systemd-networkd-wait-online.service

# The following systemd services are not needed
RUN systemctl mask systemd-firstboot.service

# switch default target from graphical to multi-user
RUN systemctl set-default multi-user.target

# Create default 'admin/admin' user
RUN useradd --create-home --shell /bin/bash admin && echo "admin:admin" | chpasswd

# Set systemd as entrypoint.
ENTRYPOINT [ "/sbin/init", "--log-level=err" ]

• The sysbox/sysbox/sysbox-in-docker/Dockerfile.gentoo-stage3-amd64-systemd dockerfile

#
# Sysbox-In-Docker Container Dockerfile (Gentoo)
#
# This Dockerfile creates the sysbox-in-docker container image, which holds
# all Sysbox binaries and its dependencies. The goal is to allow users to run
# an entire Sysbox sandbox within a container.
#
# NOTE: Sysbox is a container runtime and thus needs host root privileges. As a
# result, this image must be run as a privileged container, and a few resources
# must be bind-mounted to meet Sysbox requirements as well as those of system-level
# apps running in inner containers. Notice that within the privileged container,
# inner containers launched with Docker + Sysbox will be strongly isolated from the
# host by Sysbox (e.g., via the Linux user-namespace).
#
# Instructions:
#
# * Image creation:
#
#   $ make sysbox-in-docker gentoo-stage3-amd64-systemd
#
# * Container creation:
#
# docker run -d --privileged --rm --hostname sysbox-in-docker --name sysbox-in-docker \
#                -v /var/tmp/sysbox-var-lib-docker:/var/lib/docker \
#                -v /var/tmp/sysbox-var-lib-sysbox:/var/lib/sysbox \
#                -v /lib/modules/$(uname -r):/lib/modules/$(uname -r):ro \
#                -v /usr/src/kernels/$(uname -r):/usr/src/kernels/$(uname -r):ro \
#                nestybox/sysbox-in-docker:gentoo-stage3-amd64-systemd
#

FROM gentoo/stage3:amd64-systemd


RUN emerge-webrsync && emerge -v \
    app-portage/gentoolkit \
    app-misc/jq \
    sys-fs/fuse \
    sys-apps/lsb-release \
    app-shells/bash-completion \
    dev-util/dialog && \
    # as get-docker.sh not gentoofriendly !! so emerge docker
    emerge -v \
    app-containers/docker \
    app-containers/docker-buildx \
    app-containers/docker-cli && \
    # Housekeeping
    # eclean -d distfiles && eclean -d packages
    rm -rf /var/cache/distfiles/ && rm -rf /var/db/repos/gentoo/

##  Docker get-docker.sh not Gentoo friendly, Docker install emerge above
# Install Docker.
#RUN curl -fsSL https://get.docker.com -o get-docker.sh
#RUN sh get-docker.sh

# S6 process-supervisor installation & configuration.
ADD /~https://github.com/just-containers/s6-overlay/releases/download/v2.1.0.2/s6-overlay-amd64-installer /tmp/
RUN chmod +x /tmp/s6-overlay-amd64-installer && /tmp/s6-overlay-amd64-installer /

ENV \
    # Pass envvar variables to agents
    S6_KEEP_ENV=1 \
    # Direct all agent logs to stdout.
    S6_LOGGING=0 \
    # Exit container if entrypoint fails.
    S6_BEHAVIOUR_IF_STAGE2_FAILS=2

COPY s6-services /etc/services.d/
COPY sysbox /etc/cont-init.d/

# Copy Sysbox artifacts.
COPY sysbox-mgr /usr/bin/sysbox-mgr
COPY sysbox-fs /usr/bin/sysbox-fs
COPY sysbox-runc /usr/bin/sysbox-runc

ENTRYPOINT ["/init"]

• The sysbox/tests/Dockerfile.gentoo-2 dockerfile

#
# Sysbox Test Container Dockerfile (Gentoo image inspired by Dockerfile.debian-ullseye)
#  disclaimer : deb package name translated to emerge counterpart
#               probably not fully completed to run all test suite ?!
#               
#
# This Dockerfile creates the sysbox test container image. The image
# contains all dependencies needed to build, run, and test sysbox.
#
# The image does not contain sysbox itself; the sysbox repo
# must be bind mounted into the image. It can then be built,
# installed, and executed within the container.
#
# The image must be run as a privileged container (i.e., docker run --privileged ...)
# Refer to the sysbox Makefile test targets.
#
# This Dockerfile is based on a similar Dockerfile in the OCI runc
# github repo, but adapted to sysbox testing.
#
# Instructions:
#
# docker build -t sysbox-test .
#

FROM gentoo/stage3:amd64-systemd

# Desired platform architecture to build upon.
ARG sys_arch
ENV SYS_ARCH=${sys_arch}
ARG target_arch
ENV TARGET_ARCH=${target_arch}

#ENV DEBIAN_FRONTEND=noninteractive

# too old for Gentoo, at time of writing
#ARG k8s_version=v1.20.2

# CRI-O & crictl version for testing sysbox pods; CRI-O 1.20 is required as it
# introduces rootless pod support (via the Linux user-ns)
#ARG crio_version=1.20
#ARG crio_os=Debian_11
#ARG crictl_version=v1.20.0

RUN echo -e 'FEATURES="-ipc-sandbox -network-sandbox -pid-sandbox"\nUSE="-doc"\n' >> /etc/portage/make.conf && \
    emerge-webrsync && emerge \
       app-portage/gentoolkit \
       app-admin/sudo \
       dev-vcs/git \
       app-misc/jq \
       dev-libs/libaio \
       dev-libs/protobuf \
       dev-libs/protobuf-c \
       dev-libs/libnl \
       net-libs/libnet \
       sys-process/lsof \
       sys-fs/fuse \
       sys-apps/lsb-release \
       app-shells/bash-completion \
       dev-util/dialog \
       app-text/tree \
       dev-util/strace \
       dev-util/shellcheck-bin && \
    # as get-docker.sh not gentoofriendly !! so emerge docker
    echo "app-containers/cri-o" > /etc//portage/package.accept_keywords/cri-o && \
    emerge -v \
       app-containers/docker \
       app-containers/docker-buildx \
       app-containers/docker-cli \
       sys-cluster/kubectl \
       app-containers/cri-o \
       app-containers/cri-tools && \
    echo "# Housekeeping \n" && \
    echo "# raw quick and dirty 'eclean -d distfiles && eclean -d packages'\n" && \
    rm -rf /var/cache/distfiles/*.* && rm -rf /var/cache/binpkgs/*.* && rm -rf /var/db/repos/gentoo/*.*

# from Bulleye test dockerfile
RUN    echo ". /etc/bash_completion" >> /etc/bash.bashrc \
    && ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa \
    && echo "    StrictHostKeyChecking accept-new" >> /etc/ssh/ssh_config



# Install Golang
RUN wget https://go.dev/dl/go1.19.6.linux-${sys_arch}.tar.gz && \
    tar -C /usr/local -xzf go1.19.6.linux-${sys_arch}.tar.gz && \
    /usr/local/go/bin/go env -w GONOSUMDB=/root/nestybox

ENV GOPATH /go
ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
RUN go env -w GONOSUMDB=/root/nestybox && \
    mkdir -p "$GOPATH/src" "$GOPATH/bin" && \
    chmod -R 777 "$GOPATH"

# Add a dummy user for the rootless integration tests; needed by the
# `git clone` operations below.
RUN useradd -u1000 -m -d/home/rootless -s/bin/bash rootless

# install bats
RUN cd /tmp \
    && git clone /~https://github.com/sstephenson/bats.git \
    && cd bats \
    && git reset --hard 03608115df2071fff4eaaff1605768c275e5f81f \
    && ./install.sh /usr/local \
    && rm -rf /tmp/bats

# install protoc compiler for gRPC
RUN if [ "$sys_arch" = "amd64" ] ; then arch_str="x86_64"; \
    elif [ "$sys_arch" = "arm64" ]; then arch_str="aarch_64"; \
    else echo "Unsupported platform: ${sys_arch}"; exit; fi \
    && curl -LO /~https://github.com/protocolbuffers/protobuf/releases/download/v3.15.8/protoc-3.15.8-linux-${arch_str}.zip \
    && unzip protoc-3.15.8-linux-${arch_str}.zip -d $HOME/.local \
    && export PATH="$PATH:$HOME/.local/bin" \
    && go install github.com/golang/protobuf/protoc-gen-go@latest \
    && export PATH="$PATH:$(go env GOPATH)/bin"

##  Docker get-docker.sh not Gentoo friendly, Docker install emerge above
## Install Docker
##RUN curl -fsSL https://get.docker.com -o get-docker.sh \
##    && sh get-docker.sh
ADD https://raw.githubusercontent.com/docker/docker-ce/master/components/cli/contrib/completion/bash/docker /etc/bash_completion.d/docker.sh

## kubectl emerged above
### Install Kubectl for K8s integration-testing. Notice that we are explicitly
### stating the kubectl version to download, which should match the K8s release
### deployed in K8s (L2) nodes.
##RUN curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -  \
##    && echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list \
##    && apt-get update \
##    && apt-get install kubectl="${k8s_version#v}"-00 \
##    && apt-get clean -y \
##    && rm -rf /var/lib/apt/lists/*

## shellceck-bin emerged above
### shellcheck for lint of shell scripts
##RUN apt-get update && apt-get install -y shellcheck

# Go Dlv for debugging
RUN go install github.com/go-delve/delve/cmd/dlv@latest

## kubectl emerged above
### Install Kubectl for k8s-in-docker integration-testing. Notice that we are explicitly
### stating the kubectl version to download, which should match the K8s release
### deployed in the K8s-in-docker nodes (L2).
##RUN curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -  \
##    && echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list \
##    && apt-get update \
##    && apt-get install kubectl:${sys_arch}="${k8s_version#v}"-00 \
##    && apt-get clean -y \
##    && rm -rf /var/lib/apt/lists/*

## app-containers/cri-o and app-containers/cri-tools emerged above
### CRI-O and crictl for testing deployment of pods with sysbox (aka "sysbox pods")
##RUN echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/${crio_os}/ /"| tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list \
##    && echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/${crio_version}/${crio_os}/ /"| tee /etc/##apt/sources.list.d/##devel:kubic:libcontainers:stable:cri-o:${crio_version}.list \
##    && curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${crio_version}/${crio_os}/Release.key | sudo apt-key add - \
##    && curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/${crio_os}/Release.key | sudo apt-key add - \
##    && apt-get update && apt-get install -y --no-install-recommends conntrack:${sys_arch} cri-o:${sys_arch} cri-o-runc:${sys_arch} \
##    && wget /~https://github.com/kubernetes-sigs/cri-tools/releases/download/${crictl_version}/crictl-${crictl_version}-linux-${sys_arch}.tar.gz \
##    && sudo tar zxvf crictl-${crictl_version}-linux-${sys_arch}.tar.gz -C /usr/local/bin \
##    && rm -f crictl-${crictl_version}-linux-${sys_arch}.tar.gz

# Container CNIs (needed by CRI-O)
RUN cd /root \
    && git clone /~https://github.com/containernetworking/plugins \
    && cd plugins \
    && git checkout -b v0.9.1 v0.9.1 \
    && ./build_linux.sh \
    && mkdir -p /opt/cni/bin \
    && cp bin/* /opt/cni/bin/

# Dasel (for yaml, toml, json parsing) (/~https://github.com/TomWright/dasel)
# Note: manually download Dasel v1 as our testContainerInit script does not yet support Dasel v2.
RUN wget /~https://github.com/TomWright/dasel/releases/download/v1.27.2/dasel_linux_${sys_arch} &&  mv dasel_linux_${sys_arch} dasel && chmod +x dasel \
    && mv ./dasel /usr/local/bin/dasel

# Use the old definition for SECCOMP_NOTIF_ID_VALID in /usr/include/linux/seccomp.h
#
# This is needed because the definition changed in the mainline kernel
# on 06/2020 (from SECCOMP_IOR -> SECCOMP_IOW), and some distros we
# support have picked it up in their latest releases / kernels
# updates. The kernel change was backward compatible, so by using the
# old definition, we are guaranteed it will work on kernels before and
# after the change. On the other hand, if we were to use the new
# definition, seccomp notify would fail when sysbox runs in old
# kernels.
RUN sed -i 's/^#define SECCOMP_IOCTL_NOTIF_ID_VALID[ \t]*SECCOMP_IOW(2, __u64)/#define SECCOMP_IOCTL_NOTIF_ID_VALID   SECCOMP_IOR(2, __u64)/g' /usr/include/linux/seccomp.h

# sysbox env
RUN useradd sysbox

# test scripts
COPY scr/testContainerInit /usr/bin
COPY scr/testContainerCleanup /usr/bin
COPY scr/buildContainerInit /usr/bin
COPY scr/sindTestContainerInit /usr/bin
COPY bin/userns_child_exec_${sys_arch} /usr/bin

RUN mkdir -p /root/nestybox
WORKDIR /root/nestybox/sysbox
CMD /bin/bash

Makefile

Before launching the first make sysbox-static try, some Makefile adjustments are needed :

1. add "gentoo" familly to the image base distro filter ;
2. Due to some ebuild install blocks (net-misc/socat-1.7.4.4-r1 for example), a quick and dirty workaround was to remove some docker build env vars (--build-arg target_arch=$(TARGET_ARCH)) on test-image target.

--- Makefile    2023-10-20 17:02:50.438440972 +0200
+++ Makefile-sysbox-gentoo      2023-10-20 17:03:49.087418853 +0200
@@ -93,7 +93,7 @@
 export KERNEL_REL
 
 # Sysbox image-generation globals utilized during the sysbox's building and testing process.
-ifeq ($(IMAGE_BASE_DISTRO),$(filter $(IMAGE_BASE_DISTRO),centos fedora redhat almalinux rocky amzn))
+ifeq ($(IMAGE_BASE_DISTRO),$(filter $(IMAGE_BASE_DISTRO),centos fedora redhat almalinux rocky amzn gentoo))
        IMAGE_BASE_RELEASE := $(shell cat /etc/os-release | grep "^VERSION_ID" | cut -d "=" -f2 | tr -d '"' | cut -d "." -f1)
        KERNEL_HEADERS := kernels/$(KERNEL_REL)
 else
@@ -473,11 +473,16 @@
        @exit 1
 endif
 
+
 test-img: ## Build test container image
 test-img:
+#      @printf "\n** Building the test container **\n\n"
+#      @cd $(TEST_DIR) && docker build -t $(TEST_IMAGE) \
+#              --build-arg sys_arch=$(SYS_ARCH) --build-arg target_arch=$(TARGET_ARCH) \
+#              -f Dockerfile.$(IMAGE_BASE_DISTRO)-$(IMAGE_BASE_RELEASE) .
        @printf "\n** Building the test container **\n\n"
        @cd $(TEST_DIR) && docker build -t $(TEST_IMAGE) \
-               --build-arg sys_arch=$(SYS_ARCH) --build-arg target_arch=$(TARGET_ARCH) \
+               --build-arg sys_arch=$(SYS_ARCH)  \
                -f Dockerfile.$(IMAGE_BASE_DISTRO)-$(IMAGE_BASE_RELEASE) .
 
 test-img-systemd: ## Build test container image that includes systemd

Manual install

Copy binaries

use make install or manually copy binaries in /usr/bin

Install systemd units

From Sysbox debian package extract systemd stuff and move it ot accordingly dirs :

1. download latest (0.6.2-0 at time of writing) sysbox deb package

# cd /tmp && curl -LJO https://downloads.nestybox.com/sysbox/releases/v0.6.2/sysbox-ce_0.6.2-0.linux_amd64.deb

2. Extract lib dir from data.tar.xz inside deb file

# ar -xv sysbox-ce_0.6.2-0.linux_amd64.deb && tar -tvf data.tar.xz ./lib && tar -xvf data.tar.xz ./lib

3. verify file contents :

• cat ./lib/sysctl.d/99-sysbox-sysctl.conf
• cat ./lib/systemd/system/sysbox.service
• cat ./lib/systemd/system/sysbox-mgr.service
• cat ./lib/systemd/system/sysbox-fs.service

4. Optionnally adjust to your system, then if that really what you want, copy or move those files to their accordingly dir under /usr/lib subtree
5. start sysbox

# systemctl start sysbox
# systemctl status sysbox

Set Docker env aware of Sysbox

Set Docker env aware of Sysbox using ./scr/docker-cfg --sysbox-runtime=enable script or manually edit your /etc/docker/daemon.json conf file.

• Rootless Sysbox aware docker daemon.json example

  "userns-remap": "xxx-id",
  "ipv6": true,
  "experimental": true,
  "fixed-cidr-v6": "2001:db8:d0c4:515b::/64",
  "ip6tables": true,
  "exec-opts": ["native.cgroupdriver=systemd"],
  "experimental": true,
  "cgroup-parent": "docker.slice",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3" 
  },
  "runtimes": {
     "sysbox-runc": {
         "path": "/usr/bin/sysbox-runc"
     }
  }
 
}

Nota : Adjust userns-remap stanza value to your corresponding userid remapped
Nota : fixed-cidr-v6 stanza : 2001:db8::/32 documentation IPv6 prefix example, adjust to your own GUA or ULA IPv6 prefix.

• restart Docker

# systemctl restart docker

References

• Sysbox Developer's Guide: Building & Installing

[ctalledo]

Thank you very much @j-landru for listing out the steps to build and use Sysbox in Gentoo.

Regarding

Nota : Adjust userns-remap stanza value to your corresponding userid remapped

Is that really needed? Normally we use Docker without userns-remap with Sysbox, and Sysbox then generates the containers in the user-namespace (Docker is unaware).

[j-landru]

As it was my previous docker rootless config, I thought it could be a "good practice" to leave it as is in the principle of "less privilege".

But I'm not a sysbox expert, let me know if it's the case or not. Maybe a notice could be added in the doc as en ehancement ?