Raspberry Pi: update breaks Raspbian Stretch

Today, the three Raspberry Pis that we have on our network went down. They were all running Raspbian (Debian for Raspberry Pi) Stretch.

While this issue can be solved, it is serious enough to require to remove the micro-SD card and manually fix the the root filesystem. Therefore, it seems you cannot fix this issue unless you have physical access to your system.

Here are details to attract attention to this issue…

As I started telling you, our systems were down, well almost. Some services were still running, as they were still responding to ping through our VPN. However, SSH access was no longer available:

$ ssh scan
ssh_exchange_identification: Connection closed by remote host

After connecting a serial cable to one of the Pis, and adding init=/bin/sh to the /boot/cmdline.txt file. I found that I couldn’t seem to execute at least some executables. Everything I tried to execute was causing a segmentation fault.

It was time to remove the micro-SD card and look at system logs. Inspecting /var/log/apt/history.log revealed that the raspi-copies-and-fills package was updated yesterday (March. 11, 2019). This allowed me to make a search for issue reports with this package name. Indeed, before having such a lead, I couldn’t find what I was looking for, as there are too many discussions about the use of the Raspberry Pi! So, here’s what I quickly found following this lead:

raspi-copies-and-fills package gone AWOL? (Raspberry Pi forums)
Ras3 crashes after update (Raspberry Pi forums)

These posts have all details. All you need to do is take away the micro-SD card, repair the second partition with e2fsck -f /dev/mmcblk0p2 and remove the etc/ld.so.preload file inside this partition.

Note, that at the time of this writing, this issue has already been fixed, so it is safe to upgrade your Pi if it is still up and running, or right after repairing your Raspbian root filesystem.

This incident is very unfortunate, as you need to physically access your board to recover from it. I hope you don’t run updates as frequently as we do (or right after the time when the update was issued), and that your Pis are not impacted, otherwise possibly forcing you to travel or to crawl into difficult to access places to reach your boards.

However, I don’t want blame the community volunteers running Raspbian. They have made a terrific job maintaining this distro which had been flawlessly running for more many years on our systems. This seems as good as what we get from commercial distributions.

I hope that not too many services ran by Raspberry Pis will be disrupted because of this issue. However, that may be yet another way to prove how popular such devices are.

Author: Michael Opdenacker

Michael Opdenacker is the founder of Bootlin, and was its CEO until 2021. He is best known for all the free embedded Linux and kernel training materials that he created together with Thomas Petazzoni. He is always looking for ways to increase performance, reduce size and boot time, and to maximize Linux' world domination. More details... View all posts by Michael Opdenacker