Raspberry 4 development environment

Recently I obtained a CM4 Raspberry 4 compute module and an IO base board. Developing software for the board can be done in several ways. Very straightforward is to use the device as a desktop PC. You can install an editor and compiler and start coding. However this approach has some serious drawbacks:

• Compiling on the device is slow
• When you update the boot parameters there is always the chance that the device won’t boot. You will have to move the SD card to another desktop to make fixes. In the case of EMMC a special procedure is required to access the files.
• By default Linux will continuously write a lot of data to the SD card/EMMC. This shortens the lifespan of the memory.
• You have to think about keeping the OS clean. Especially when experimenting with installation various packages.

In this post I show how to setup the perfect environment for Raspberry software development. I will cover the following topics:

CM4 on IO base board

• Configure net boot using TFTP.
  – This allows easy changes to boot parameters such as cmdline.txt, config.txt. If the device fails to boot you can easily roll back and reboot without swapping SD cards.
• Configure Raspberry rootfs over NFS.
  – A central location is used for all the OS files. This makes it easy to create backups and to keep your OS clean.
• Making the rootfs read only, using overlay fs for temporary changes. and syncing changes back to the rootfs.
  – Avoids making unwanted changes to the OS.
  – This avoids writing continuously to the SD card or EMMC which improves reliability.
• Configure a cross compiler and chroot
  – You can compile applications for the Raspberry on a fast desktop machine. The built code is available immediately on the device through the NFS share.
  – The cross compiler will use the NFS root as a ‘sysroot’ which will allow you to link against ARM libraries installed on the device

Preparation

First we setup some scripts and required packages.

1. On the Raspberry install the following:

# for the NFS client
sudo apt-get install nfs-common
# needed to change absolute symlinks to relative symlinks
sudo apt-get install symlinks

2. Fix symbolic links that contain absolute paths and change them into relative symlinks. This will solve issues later on when using the rootfs as a sysroot for cross compilation.

cd /usr/lib/aarch64-linux-gnu/
sudo symlinks -c -r .

3. Later on we use an overlay fs to support a read only root file system. To setup this during boot an initial ramdisk (initrd) is required. Create the initrd:

# make boot read/write
mount -o remount,rw /boot
sudo mkinitramfs -o /boot/initrd

If you get errors that certain realtek firmwares are missing you can copy these from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/rtl_nic to /lib/firmware/rtl_nic/

4. We will setup overlay fs and use a script by Paul Ridgway:

sudo curl https://gist.githubusercontent.com/paul-ridgway/d39cbb30530442dca416734c3ee70162/raw/c490df8be1976dd062a8b5f429ef42ed1b393ecb/ro-root.sh 
-o /bin/ro-root.sh 

5. I had some issues while getting “boot from NFS” working. The boot process was hanging. To get it working I configured a static IP (this is explained in a later paragraph). Furthermore I disabled the swap file:

sudo systemctl disable dphys-swapfile

6. Next setup a NFS share “rpi_rootfs” and “rpi_boot” on your NAS or development server. For the time being the shares must be writable, don’t enable root squash. Mount the shares on the Raspberry:

sudo mkdir /mnt/rpi_rootfs
sudo mkdir /mnt/rpi_boot
sudo mount -t nfs :/rpi_rootfs /mnt/rpi_rootfs
sudo mount -t nfs :/rpi_boot /mnt/rpi_boot

7. Now copy the boot partition and root partition to the NFS shares. I use rsync for this:

# remove --dry-run to actually perform the copying, instead of just testing what would be done.
sudo rsync -aAXv --dry-run --delete --exclude={“/etc/fstab”,”/boot/*”,”/dev/*”,”/proc/*”,”/sys/*”,”/tmp/*”,”/run/*”,
”/mnt/*”,”/media/*”,”/ro/*”,”/rw/*”,”/lost+found”} / /mnt/rpi_rootfs/# remove --dry-run to actually perform the copying, 
instead of just testing what would be done.
sudo rsync -aAXv --dry-run --delete --exclude={“/etc/fstab”,”/boot/*”,”/dev/*”,”/proc/*”,”/sys/*”,”/tmp/*”,”/run/*”,
”/mnt/*”,”/media/*”,”/ro/*”,”/rw/*”,”/lost+found”} /boot/ /mnt/rpi_boot/

Network boot using TFTP

Booting using TFTP allow easy access of to the Raspberry boot parameters, even in the case you made a mistake and the RPi won’t boot.

Setup a TFTP server on your NAS or developlment server that will server the files from a rpi_boot share.

We first need to make sure that the CM4 module will no longer boot from eMMC or sd card but will fetch the files from the TFTP server. This can be done by writing the eeprom. Perform the following steps on the development machine:

1. clone the https://github.com/raspberrypi/usbboot repository. Read the Readme.txt

cd usbboot
make

2. clone the eeprom repo: https://github.com/raspberrypi/rpi-eeprom.git

3. create a folder network_eeprom

4 .copy the latest eeprom to this dir:

cp rpi-eeprom/firmware/stable/pieeprom-2022–02–08.bin network_eeprom

5. copy bootcode4.bin and copy a signature

cp usbboot/recovery/bootcode4.bin network_eeprom
cp usbboot/recovery/pieeprom.sig network_eeprom

6. Create network_eeprom/boot.conf file

I had issues getting NFS boot working with DHCP so I configured a static fixed IP.

[all]
BOOT_UART=0
WAKE_ON_GPIO=0
POWER_OFF_ON_HALT=1# see https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#BOOT_ORDER
# 0xf2 means restart (0xf) and then try network boot (0x02) BOOT_ORDER=0xf2# the address of your TFTP server
TFTP_IP=192.168.68.130
# static client IP, no dhcp
CLIENT_IP=192.168.68.111
SUBNET=255.255.255.0
GATEWAY=192.168.68.1ENABLE_SELF_UPDATE=1

7. embed the boot.conf settings in an eeprom binary

cd network_eeprom
../rpi-eeprom/rpi-eeprom-config -config boot.conf -out pieeprom.bin pieeprom-2022–02–08.bin

8. update the signature

sha256sum pieeprom.bin

Use the output of the command above to replace the first line in pieeprom.sig

9. Boot the CM4 in mass storage mode (see link). Flash the eeprom by running the following command on the development machine:

cd rpiboot
sudo ./rpiboot -d ../network_eeprom/

Reboot the Raspberry in normal startup mode. If all is well it should boot using the TFTP server!

Read only rootfs with overlay file system

Keeping the root filesystem read only helps avoiding wear on the eMMC/SD card. Modifications are stored in memory, this helps keeping the filesystem clean while experimenting with various packages.

First update rpi_boot/config.txt so that the raspberry will the initrd. Edit the file on the rpi_boot share not on the boot partition (we are net booting now!). Add the following:

# add this to /boot/config.txt
initramfs initrd followkernel
ramfsfile=initrd
ramfsaddr=-1

Reboot the Raspberry and check that it still boots. Next update rpi_boot/cmdline.txt so that the ro-root.sh script we downloaded earlier is used. Add to the end of cmdline.txt:

init=/bin/ro-root.sh

Again reboot the CM4. If you login and type “mount”, there should be a line

overlayfs-root on / type overlay (rw,noatime,lowerdir=/mnt/lower,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work)

As an example. Type “touch /home/pi/test_file.txt”. The file is created in memory on the overlay filesystem. It will be gone after an reboot. The file can be found in /rw/upper/home/pi/. The original read only root files system can be found in/ro.

Now when you install new applications (eg. with apt-get) you probably want them to be available after a reboot. After making changes you can synchronize the overlay to the root filesystem after which the changes are permanent. This ‘syncing’ operation is a bit of a hack, as overlay fs doesn’t really support syncing while the filesystem is mounted. So after a sync you always need to reboot the Raspberry!

# make the ro folder read/write
sudo mount -o /remount,rw /ro# sync the files including the changes in 'upper' to the lower filesystem
sudo rsync -aAXv --dry-run --delete --exclude={“/etc/fstab”,”/boot/*”,”/dev/*”,”/proc/*”,”/sys/*”,”/tmp/*”,”/run/*”,”/mnt/*”,
”/media/*”,”/ro/*”,”/rw/*”,”/lost+found”} / /ro/# always reboot after this operation

sudo reboot

Rootfs over NFS

In this step we replace the “lower” directory which the overlay is using by an NFS mount. Update the rpi_boot/cmdline.txt file as follows:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootwait rw nfsroot=192.168.68.130:/rpi_rootfs 
ip=192.168.68.111::192.168.68.1:255.255.255.0:rpi::off root=/dev/nfs elevator=deadline init=/bin/ro-root.sh

1. I use a static ip. The format of the ip string is detailed here.
2. Some have added “smsc95xx.turbo_mode=N” as it would fix some network issues. I have seen no network issues.
3. Some people are using extra options to nfsroot,
   nfsroot=192.168.68.130:/rpi_rootfs,ip,v3
   I didn’t need these options to get it working.

After rebooting check your mounts. There should be a line that indicates the NFS share is mounted.

> mount
...
192.168.68.130:/rpi_rootfs on /ro type nfs (ro,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,
hard,nolock,proto=tcp,port=2049,timeo=600,retrans=10,sec=sys,mountaddr=192.168.68.130,mountvers=3,
mountport=30000,mountproto=tcp,local_lock=all,addr=192.168.68.130)
...

Sidenote: It took me quite some time to get NFS working. Somehow the boot process just hanged. I have a switch which support port mirroring. Using wireshark I was able to check that NFS was actually working, but that some other networking issue caused problems.

Finally the Raspberry is now operating completely using network storage!

Cross compiling

To develop applications for the Raspberry we need a ‘cross’ compiler. This is a compiler that runs on the development machine, but builds applications that run on the target (Raspberry) machine. Furthermore we want this compiler to use the execution environment (libraries and so forth) that the Raspberry root filesystem provides.

1. First we characterize the Raspberry execution environment:

# check the version of binutils
>ld -v
GNU ld (GNU Binutils for Debian) 2.35.2
# check the version of glibc
>ldd --version
ldd (Debian GLIBC 2.31-13+rpt2+rpi1+deb11u2) 2.31
>/lib/aarch64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.31-13+rpt2+rpi1+deb11u2) stable release version 2.31.
# check the kernel version
>uname -a
Linux raspberrypicm4 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

2. Setup crosstool-ng. Crosstool-ng will build the cross compiler toolchain for us. I performed these steps inside a Docker container to avoid having to install many packages into my development environment.

git clone https://github.com/crosstool-ng/crosstool-ng

3. Build crosstool-ng

# note: check the home dir paths below
cd crosstool-ng/
./bootstrap 
mkdir build
cd build/
./configure — prefix=/home/develop/.local
make -j $(nproc)
make install
export PATH=/home/develop/.local/bin:$PATH

4. Configure and build the Raspberry cross compilation toolchain

mkdir ~/RPi_toolchain
cd ~/RPi_toolchain
# use rpi4 template
ct-ng aarch64-rpi4-linux-gnu

You can use “ct-ng menuconfig” to customize the toolchain. The resulting configuration can be save in the file “.config”. At least the following options have to be updated:

4.1 Raspberry uses multiarch organization to support 32 and 64 bit libraries. This needs a binutils patch. In order to have crosstool-ng process patches we me must configure:
— Paths and misc options → Patches origin → Bundled, then local
— Paths and misc options → Patches origin → Local patch directory ${CT_TOP_DIR}/patches
— C Compiler →Core GCC extra config =–enable-multiarch

If you check .config afterwards the following options should have been set:

CT_PATCH_BUNDLED_LOCAL=y
CT_PATCH_ORDER=”bundled,local”
CT_PATCH_USE_LOCAL=y
CT_LOCAL_PATCH_DIR=”${CT_TOP_DIR}/patches”
CT_CC_GCC_EXTRA_CONFIG_ARRAY="--enable-multiarch"

4.2. Set the binutils version (see step 1)

— Binary utilities → Version of binutils = 2.35.2

I couldn’t select 2.35.2, only 2.35.1. I selected 2.35.1 and updated .config afterwards. Unfortunately this is reset every time you make changes via menuconfig!

CT_BINUTILS_VERSION=”2.35.2"

4.3. Set the glibc version (see step 1)
— C-library → Version of glibc= 2.31

4.4. Set the kernel version (see step 1)
— Operating system → Version of linux = 5.9.16

Choose the first version lower than your kernel (<5.10.92)

4.5. Choose GCC version
— C compiler → Version of gcc = 10.3.0

I tried the latest one as well (11.2) but it was incompatible with GLibC 2.31.

4.6. Disable read only output
— Paths and misc options → Render the toolchain read-only = no

4.6. Setup binutils patches

mkdir -p ~/RPi_toolchain/patches/binutils/2.35.2cd ~
wget https://ftp.debian.org/debian/pool/main/b/binutils/binutils-source_2.35.2-2_all.deb
mkdir binutils
cd binutils
ar x ../binutils-source_2.35.2–2_all.deb 
tar xf data.tar.xz
cp usr/src/binutils/patches/129_multiarch_libpath.patch ~/RPi_toolchain/patches/binutils/2.35.2/.

5. Build the toolchain

Verify versions:

>ct-ng show-config
 Languages : C,C++
 OS : linux-5.9.16
 Binutils : binutils-2.35.1
 Compiler : gcc-10.3.0
 C library : glibc-2.31
 Debug tools : gdb-11.2
 Companion libs : expat-2.4.1 gettext-0.21 gmp-6.2.1 isl-0.24 libiconv-1.16 mpc-1.2.1 mpfr-4.1.0 ncurses-6.2 zlib-1.2.11
 Companion tools :
> export DEB_TARGET_MULTIARCH=arm-linux-gnueabihf
> ct-ng build

After building check that the binutils patch was applied.

> ./aarch64-rpi4-linux-gnu-ld --verbose | grep -i “search”...SEARCH_DIR("=/usr/local/lib/arm-linux-gnueabihf")....

6. Using/Testing
To test that the generated toolchain is working and searches the correct paths for libraries I tried a “hello world” app that makes use of ncurses.

6.1. On the raspberry. Install ncurses using apt-get, sync the rootfs and reboot

6.2 Hello world ncurses-test.c

#include <ncurses.h> /* ncurses.h includes stdio.h */
#include <string.h>int main()
{
 char mesg[]=”Enter a string: “; /* message to be appeared on the screen */
 char str[80];
 int row,col; /* to store the number of rows and *
 * the number of colums of the screen */
 initscr(); /* start the curses mode */
 getmaxyx(stdscr,row,col); /* get the number of rows and columns */
 mvprintw(row/2,(col-strlen(mesg))/2,”%s”,mesg);
 /* print the message at the center of the screen */
 getstr(str);
 mvprintw(LINES — 2, 0, “You Entered: %s”, str);
 getch();
 endwin();return 0;
}

6.3 Build ncurses-test.c using the cross compiler. Building the app is straight forward.

#!/usr/bin/env bash
set -eux
# set the location of the toolchain
install_dir=”/home/dinne/x-tools/aarch64-rpi4-linux-gnu”
export PATH=”${PATH}:${install_dir}/bin”aarch64-rpi4-linux-gnu-gcc \
 --sysroot /mnt/rpi_rootfs \
 -v -c -o ncurses-test.o \
 ncurses.caarch64-rpi4-linux-gnu-gcc \
 --sysroot=/mnt/rpi_rootfs \
 -v ncurses-test.o \
 -o ncurses-test \
 -lcurses -ldlcp ncurses-test /mnt/rpi_roots/home/pi/.

If you login on to the CM4 you can run it. Another approach is to run it in an emulated environment via a chroot, as shown in the next section.

Note: I had some problems that the compiled app could not be ran on the target and was giving errors regarding a missing library. You can see which libraries the app wants to load using ldd. Some paths are hardcoded in gcc itself. The runtime linker (searcH) paths can be modified if you encouter issues:

# Not required
# -Wl, — dynamic-linker=”/lib/ld-linux-aarch64.so.1" \
# -Wl, — rpath=”/lib” 
# -L /mnt/rpi_rootfs/usr/lib/gcc/aarch64-linux-gnu/10

Chroot

You can emulate ARM binaries on the development PC.

1. Install the ARM emulator:

sudo apt install qemu binfmt-support qemu-user-static

2. Check that ARM binaries are linked to the emulator:

update-binfmts — display

3. Mount the raspberry root fs on your development machine. Eg add to /etc/fstab

> mkdir /mnt/rpi_rootfs# add to /etc/fstab
...
192.168.68.130:/rpi_rootfs /mnt/rpi_rootfs nfs defaults 0 0
...> sudo mount /mnt/rpi_rootfs

4. Enable the chroot, and perform a test

sudo chroot /mnt/rpi_rootfs> /home/pi/ncurses-test

In the past I have had issues with other setups. This stackoverlow post is a good starting point solving possible problems: https://unix.stackexchange.com/questions/41889/how-can-i-chroot-into-a-filesystem-with-a-different-architechture

Conclusion

Finally we have a good starting point to work on Raspberry Pi applications.