I tried to apt-get dist-upgrade my Rasberry Pi 2 Model B to Raspbian Buster this weekend. Some things broke!
udev, journald and timesyncd crash
On startup this looks like systemd failures like “Failed to start udev Kernel Device Manager”, with errors like:
error while loading shared libraries: /usr/lib/arm-linux-gnueabihf/libarmmem.so: cannot restore segment prot after reloc: Operation not permitted
This was particularly nasty because the system fails to boot enough to start sshd, so physical access is needed to repair it.
This problem is described on stackexchange and on the Raspbian bug tracker.
libarmmem is an LD_PRELOADed library from the raspi-copies-and-fills package, containing ARM-optimized versions of memcpy, strlen, etc. The 0.11 package is incompatible with the other buster binaries, presumably due to the newer libc/toolchain.
The immediate solution is to apt-get purge raspi-copies-and-fills, but this presumably leaves you with some slow, ubiquitous libc routines. The solution to get optimized, working libraries was to build a new version of raspi-copies-and-fills with the buster toolchain. One extra layer of confusion was added because the Debian source (ala apt-get source) package corresponding to raspi-copies-and-fills_0.11 seems to be out of sync with Raspbian’s own repository, so I ended up forking my own raspi-copies-and-fills branch.
iptables is broken, nf_tables is disabled
Debian buster has deprecated iptables in favour of nftables, so /usr/sbin/iptables is by default an nftables wrapper. But the stable kernel from the raspberrypi-kernel_1.20190401-1 package doesn’t have the nf_tables kernel module, so all your iptables and nft commands will fail.
You can use /usr/sbin/iptables-legacy instead, or install a recent (4.19.50-v7+) kernel using rpi-update.
Building out-of-tree kernel modules after rpi-update
Unfortunately, rpi-update doesn’t install things in an idiomatic Debian way (there’s no .deb package, it just shoves binaries into /boot). In addition, rpi-update doesn’t install anything equivalent to raspberrypi-kernel-headers, which is required to build modules out-of-tree.
The separate rpi-source utility does automate grabbing the sources, navigating a couple of layers of indirection to find the right commits in various github repos. I ended up following this process manually to avoid inflicting 1GB of kernel sources on my Pi’s SD card just to get 25MB of headers. It involved grabbing Module7.symvers from the rpi-firmware repo, the tree from raspberrypi/linux, /proc/config.gz from the running kernel, and then finally running
make olddefconfig && make modules_prepare