Today, the three Raspberry Pis that we have on our network went down. They were all running Raspbian (Debian for Raspberry Pi) Stretch.
While this issue can be solved, it is serious enough to require to remove the micro-SD card and manually fix the the root filesystem. Therefore, it seems you cannot fix this issue unless you have physical access to your system.
Here are details to attract attention to this issue…
As I started telling you, our systems were down, well almost. Some services were still running, as they were still responding to ping through our VPN. However, SSH access was no longer available:
$ ssh scan ssh_exchange_identification: Connection closed by remote host
After connecting a serial cable to one of the Pis, and adding
init=/bin/sh to the
/boot/cmdline.txt file. I found that I couldn’t seem to execute at least some executables. Everything I tried to execute was causing a segmentation fault.
It was time to remove the micro-SD card and look at system logs. Inspecting
/var/log/apt/history.log revealed that the
raspi-copies-and-fills package was updated yesterday (March. 11, 2019). This allowed me to make a search for issue reports with this package name. Indeed, before having such a lead, I couldn’t find what I was looking for, as there are too many discussions about the use of the Raspberry Pi! So, here’s what I quickly found following this lead:
raspi-copies-and-fills package gone AWOL? (Raspberry Pi forums)
Ras3 crashes after update (Raspberry Pi forums)
These posts have all details. All you need to do is take away the micro-SD card, repair the second partition with
e2fsck -f /dev/mmcblk0p2 and remove the
etc/ld.so.preload file inside this partition.
Note, that at the time of this writing, this issue has already been fixed, so it is safe to upgrade your Pi if it is still up and running, or right after repairing your Raspbian root filesystem.
This incident is very unfortunate, as you need to physically access your board to recover from it. I hope you don’t run updates as frequently as we do (or right after the time when the update was issued), and that your Pis are not impacted, otherwise possibly forcing you to travel or to crawl into difficult to access places to reach your boards.
However, I don’t want blame the community volunteers running Raspbian. They have made a terrific job maintaining this distro which had been flawlessly running for more many years on our systems. This seems as good as what we get from commercial distributions.
I hope that not too many services ran by Raspberry Pis will be disrupted because of this issue. However, that may be yet another way to prove how popular such devices are.
4 thoughts on “Raspberry Pi: update breaks Raspbian Stretch”
On those Raspberry Pi with more modern CPU cores than the ARM11 of the single CPU thingies it might be an idea to purge/mask the offending raspi-copies-and-fills package anyway: https://bugs.launchpad.net/raspbian/+bug/1573473
My Hass.io got corrupted after update last night, not sure if it’s raspbian based, bit think it might has to be something with this. It can’t boot and port 22 is closed. Will try this fix tonight, hope it works…
Hey, there is discussion about this issue (mentioning this article) on TechRepublic:
Finally I know why my Pi stop working 3days ago. But unfortunately my Pi is 200 km away form me. SAD.
Hope they’ll carefully check before release any update.