Upgrading K3os
As I have described in a previous article, I like k3s. I like it so much that it hosts all of my public services. This is why my servers run K3os, which is a minimal operating system with the bare-minimum to run K3s. In essence, K3os is nothing more than the diagram on the K3os website:
But this also implies that it barely has any maintenance, I have not done any large maintenance since installing it, upgrades have all been smooth, and I never needed to edit any configuration files. Sadly K3os is no longer supported by its original creator Rancher, after being bought by Suse. Because K3os is so small, I wondered how hard it would be to maintain it all by myself. At the risk of sounding like Calibre’s maintainer, I cloned all relevant repositories.
k3os-kernel
The linux kernel in k3os is based on the Ubuntu kernel. Compiling this kernel is extremely easy, Rancher wrapped the entire buildprocess into a custom tool called ‘Dapper’. This means that running sudo make will run a docker container which will then compile the kernel in a """controlled""" environment.
At the end of the buildprocess, you will be left with a dist/ directory which contains a compiled K3os kernel. At the point of deprecation, k3os’s kernel was based on Ubuntu Focal 5.4.0.
user@irondesktop:~/git/k3os/k3os-kernel (focal/lts)$ tree dist/ | less
dist/
├── artifacts
│ ├── kernel-extra-generic_amd64.tar.xz
│ ├── kernel-generic_amd64.tar.xz
│ └── kernel-headers-generic_amd64.tar.xz
└── generic
├── headers
│ ├── lib
│ │ └── modules
│ │ └── 5.4.0-88-generic
│ │ └── build -> /usr/src/linux-headers-5.4.0-88-generic
│ └── usr
│ ├── share
│ │ └── doc
│ │ ├── linux-headers-5.4.0-88
│ │ │ ├── changelog.Debian.gz
│ │ │ └── copyright
│ │ └── linux-headers-5.4.0-88-generic
│ │ ├── changelog.Debian.gz
│ │ └── copyright
│ └── src
│ ├── linux-headers-5.4.0-88
│ │ ├── arch
│ │ │ ├── alpha
│ │ │ │ ├── boot
│ │ │ │ │ ├── bootloader.lds
│ │ │ │ │ └── Makefile
:For my test, which would show me how much effort it would be to maintain K3os, I wanted to upgrade the kernel to the newest Ubuntu LTS, which is currently Ubuntu Noble with Linux 6.8.0. Unfurtunately, it wasn’t as simple as simply changing the version numbers.
Dockerfile.dapper
Dapper’s controlled build environment is based on a docker container. This docker container is defined using a Dockerfile. Dapper will then run the defined Dockerfile, and execute a bunch of custom build scripts. These build scripts can do everything you need them to do. At the end of the build process, Dapper will copy the output from the docker container to the local disk and clean everything up. K3os’s kernel was based on an Ubuntu Focal image, which downloaded the kernel sources from the ubuntu repositories. The sources are not included in the actual k3os-kernel repository. It looks like this:
ARG BUILD=library/buildpack-deps:focal
ARG UBUNTU=library/ubuntu:focal
ARG DOWNLOADS=/usr/src/downloads
FROM ${UBUNTU} AS ubuntu
ARG DOWNLOADS
ARG LINUX_FIRMWARE=linux-firmware=1.187.17
ARG LINUX_SOURCE=linux-source-5.4.0=5.4.0-88.99
ENV DEBIAN_FRONTEND=noninteractive
RUN set -x \
&& apt-get --assume-yes update \
&& apt-get --assume-yes download \
${LINUX_FIRMWARE} \
${LINUX_SOURCE} \
&& mkdir -vp ${DOWNLOADS} \
&& mv -vf linux-firmware* ${DOWNLOADS}/ubuntu-firmware.deb \
&& mv -vf linux-source* ${DOWNLOADS}/ubuntu-kernel.deb
FROM ${BUILD}
ARG DOWNLOADS
COPY --from=ubuntu ${DOWNLOADS}/ ${DOWNLOADS}/
RUN apt-get --assume-yes update \
&& apt-get --assume-yes install --no-install-recommends --upgrade \
bc \
bison \
... left out for brevity
########## Dapper Configuration #####################
ENV DAPPER_ENV VERSION DEBUG
ENV DAPPER_DOCKER_SOCKET true
ENV DAPPER_SOURCE /source
ENV DAPPER_OUTPUT ./dist
ENV DAPPER_RUN_ARGS --privileged
ENV EDITOR=vim \
PAGER=less \
SHELL=/bin/bash
WORKDIR ${DAPPER_SOURCE}
########## General Configuration #####################
ARG DAPPER_HOST_ARCH
ENV ARCH $DAPPER_HOST_ARCH
ENV DOWNLOADS ${DOWNLOADS}
ENTRYPOINT ["./scripts/entry"]
CMD ["ci"]My first test consisted of building the latest Focal kernel, which is 5.4.0-182.202. Simply changing the LINUX_SOURCE argument to the latest version (for focal) worked flawlessly. Success!
However, changing the docker image to Noble, and the kernel to linux-source-6.8.0 does not work. This is because Ubuntu’s linux-source packages for some reason no longer include the build-script that was included for Focal. This is what the original build script looked like, after exctracting the .tar.bz2.
KERNEL_DIR=build/kernel
# some hacking
mkdir -p ${KERNEL_DIR}/debian/stamps
chmod a+x ${KERNEL_DIR}/debian*/scripts/*
chmod a+x ${KERNEL_DIR}/debian*/scripts/misc/*
# kernel
pushd ${KERNEL_DIR}
unset -v ARCH KERNEL_DIR
debian/rules clean
# see https://wiki.ubuntu.com/KernelTeam/KernelMaintenance#Overriding_module_check_failures
debian/rules binary-headers binary-generic \
do_zfs=false \
do_dkms_nvidia=false \
do_dkms_nvidia_server=false \
skipabi=true \
skipmodule=true \
skipretpoline=true
popdIt simply enters the source directory, and calls the debian/rules build-script with the binary-headers and binary-generic target. This taget should create a bunch of .deb packages which can be installed into an Ubuntu system. However, the newer linux-source-6.8.0 packages no longer include the debian/rules build scripts? This let me on a side-quest where I attempted to try to use the bindeb-pkg target which the kernel supports. These side-quests eventually led to nowhere.
When I started to investigate the reason why Ubuntu no longer included the debian/rules script, I discovered that they do still use it! I have no clue why the linux-source-. packages no longer include it, but Ubuntu’s git does. Since the build-guide still claims that the apt packages should contain the build script, I think that this may be a mistake in Ubuntu’s linux-source packages. After some hacking, I changed the k3os-kernel build scripts to pull the Ubuntu 6.8.0 kernel from source and generate a build. Success! I had upgraded the k3os-kernel to 6.8.0, but without building a new k3s image, it is useless on its own.
k3os
K3os, just like k3os-kernel, uses the Dapper build tool. Which meant that creating an usable iso was as simple as calling sudo make. By default, k3os downloads many components, including the k3os-kernel.
user@irondesktop:~/git/k3os/k3os (master)$ tree dist/
dist/
├── artifacts
│ ├── k3os-amd64.iso
│ ├── k3os-initrd-amd64
│ ├── k3os-kernel-amd64.squashfs
│ ├── k3os-kernel-version-amd64
│ ├── k3os-rootfs-amd64.tar.gz
│ └── k3os-vmlinuz-amd64
├── images.tar
└── images.txt
1 directory, 8 filesRunning the k3os-amd64.iso is simple too, Rancher included a script which executes the image in qemu. Simply running sudo ./scripts/run results in the following:
GNU GRUB version 2.06
┌────────────────────────────────────────────────────────────────────────────┐
│*k3OS LiveCD & Installer │
│ k3OS Installer │
│ k3OS Rescue Shell │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────────┘
Use the ↑ and ↓ keys to select which entry is highlighted.
Press enter to boot the selected OS, `e' to edit the commands
before booting or `c' for a command-line.
and, after booting:
[ 1.365100] Floppy drive(s): fd0 is 2.88M AMI BIOS
[ 1.378179] FDC 0 is a S82078B
[ 1.416728] random: crng init done
[ 2.446569] loop1: detected capacity change from 0 to 1251440
, ,
,------------|'------'| _ ____
/ . '-' |-' | | |___ \
\/| | | | | __ __) | ___ ___
| .________.'----' | |/ / |__ < / _ \ / __|
| | | | | < ___) || (_) |\__ \
\___/ \___/ |_|\_\|____/ \___/ |___/
k3OS v0.21.5-k3s2r1
Kernel 5.4.0-88.99-generic on an x86_64 (/dev/ttyS0)
================================================================================
NIC State Address
eth0 UP fe80::5054:ff:fe12:3456/64
================================================================================
Welcome to k3OS (login with user: rancher)
k3os-21404 login: This is all really cool, and well. But its still the old version. Changing the downloaded kernel out for a few local files is as easy as changing the RUN to a COPY in the respective dockerfile.
# Download kernel
RUN mkdir -p /usr/src
# RUN curl -fL $KERNEL_XZ -o /usr/src/kernel.tar.xz
# RUN curl -fL $KERNEL_EXTRA_XZ -o /usr/src/kernel-extra.tar.xz
# RUN curl -fL $KERNEL_HEADERS_XZ -o /usr/src/kernel-headers.tar.xz
COPY kernel-generic_amd64.tar.xz /usr/src/kernel.tar.xz
COPY kernel-extra-generic_amd64.tar.xz /usr/src/kernel-extra.tar.xz
COPY kernel-headers-generic_amd64.tar.xz /usr/src/kernel-headers.tar.xzTime for a rebuild later, and running the test script! and… its broken? This had me confused for a while, and required multiple side-quests to resolve.
After copying the kernel files into the k3os build process, it extracts them, creates an initramfs, and prepares the intird configuration:
# Extract to /usr/src/root
RUN mkdir -p /usr/src/root && \
cd /usr/src/root && \
tar xvf /usr/src/kernel.tar.xz && \
tar xvf /usr/src/kernel-extra.tar.xz && \
tar xvf /usr/src/kernel-headers.tar.xz
# Create initrd
RUN mkdir /usr/src/initrd && \
rsync -a /usr/src/root/lib/ /lib/ && \
depmod $KVERSION && \
mkinitramfs -k $KVERSION -c lz4 -o /usr/src/initrd.tmp
# Generate initrd firmware and module lists
RUN mkdir -p /output/lib && \
mkdir -p /output/headers && \
cd /usr/src/initrd && \
lz4cat /usr/src/initrd.tmp | cpio -idmv && \
find lib/modules -name \*.ko > /output/initrd-modules && \
echo lib/modules/${KVERSION}/modules.order >> /output/initrd-modules && \
echo lib/modules/${KVERSION}/modules.builtin >> /output/initrd-modules && \
find lib/firmware -type f > /output/initrd-firmware && \
find usr/lib/firmware -type f | sed 's!usr/!!' >> /output/initrd-firmwareBut using our new 6.8.0 kernel, the last step fails as lib/firmware, and lib/modules does not exist. Many google-queries, guides and attempts later; I figured out that mkinitramfs does show all files in its verbose logging, but it does not actually include them in the final binary. I don’t know why it doesn’t do this, as it works fine for the original linux 5.4.0, and it also worked fine for linux 6.5.0 which I tried in one of the undocumented side-quests.
I never figured out why mkinitramfs doesn’t work properly with the 6.8.0 kernel, but another tool, called dracut is also capable of generating an initramfs image. Replacing mkinitramfs with dracut worked flawlessly.
# Create initrd
RUN mkdir /usr/src/initrd && \
rsync -a /usr/src/root/lib/ /lib/ && \
depmod $KVERSION && \
dracut /usr/src/initrd.tmp $KVERSION --lz4And after that, k3os was up and running with linux kernel 6.8.0. Wow! But we’re still not done upgrading other components of k3os. Its still running an old version of k3s, and all other installed packages are based on an outdated Alpine Linux version.
Updating k3s in k3os
K3s is installed using an install script that is downloaded from github
ARG REPO
ARG TAG
FROM ${REPO}/k3os-base:${TAG}
ARG ARCH
ENV ARCH ${ARCH}
ENV VERSION v1.23.3+k3s1
ADD https://raw.githubusercontent.com/rancher/k3s/${VERSION}/install.sh /output/install.sh
ENV INSTALL_K3S_VERSION=${VERSION} \
INSTALL_K3S_SKIP_START=true \
INSTALL_K3S_BIN_DIR=/output
RUN chmod +x /output/install.sh
RUN /output/install.sh
RUN echo "${VERSION}" > /output/versionSimply replacing the version with ENV VERSION v1.30.0+k3s1 is all that is required to upgrade the k3s version. Nice.
Updating all other base programs
As I wrote before, k3os’s base libraries are based of Alpine linux. Simply replacing the version of 3.14 to 3.19 does sadly not work.
FROM alpine:3.19 as base
ARG ARCH
RUN apk --no-cache add \
bash \
bash-completion \
blkid \
busybox \
ca-certificates \
connman \
conntrack-tools \
coreutils \
... left out for brevityThis is because Alpine linux changed the default cgroups setup in alpine version 3.16. This is simply solved by adding an /etc/rc.conf file to the k3os overlay which contains the rc_cgroup_mode="legacy" property. And that is all there is to it!
The conclusion
I was curious how much effort it would be to upgrade K3os, and not only was it suprisingly easy to upgrade k3os to the latest version of all software on it, I was able to do it in a reasonable timespan. In a previous article I wrote “I doubt that I have the time and knowledge to actually pull it off. It would be a colossal task.” about updating k3os, but I was completely wrong. In the near future, I will probably attempt to install the upgraded k3os on my servers.
, ,
,------------|'------'| _ ____
/ . '-' |-' | | |___ \
\/| | | | | __ __) | ___ ___
| .________.'----' | |/ / |__ < / _ \ / __|
| | | | | < ___) || (_) |\__ \
\___/ \___/ |_|\_\|____/ \___/ |___/
k3OS v1.30-k3s-1-g2804e85
Kernel 6.8.0-31-generic on an x86_64 (/dev/ttyS0)
================================================================================
NIC State Address
eth0 UP fe80::5054:ff:fe12:3456/64
================================================================================
Welcome to k3OS (login with user: rancher)
k3os-5787 login: rancher
Password:
Welcome to k3OS!
Refer to https://github.com/rancher/k3os for README and issues.
The default mode of k3OS is to run a single node cluster. Use "kubectl"
to access it. The node token in /var/lib/rancher/k3s/server/node-token
can be used to join agents to this server.
k3os-5787 [~]$ kubectl version
Client Version: v1.30.0+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0+k3s1If you spot any mistakes or if you would like to contact me, visit the contact page for more details.
