While this issue can be solved, it is serious enough to require to remove the micro-SD card and manually fix the the root filesystem. Therefore, it seems you cannot fix this issue unless you have physical access to your system.
Here are details to attract attention to this issue…
As I started telling you, our systems were down, well almost. Some services were still running, as they were still responding to ping through our VPN. However, SSH access was no longer available:
$ ssh scan ssh_exchange_identification: Connection closed by remote host
After connecting a serial cable to one of the Pis, and adding
init=/bin/sh to the
/boot/cmdline.txt file. I found that I couldn’t seem to execute at least some executables. Everything I tried to execute was causing a segmentation fault.
It was time to remove the micro-SD card and look at system logs. Inspecting
/var/log/apt/history.log revealed that the
raspi-copies-and-fills package was updated yesterday (March. 11, 2019). This allowed me to make a search for issue reports with this package name. Indeed, before having such a lead, I couldn’t find what I was looking for, as there are too many discussions about the use of the Raspberry Pi! So, here’s what I quickly found following this lead:
These posts have all details. All you need to do is take away the micro-SD card, repair the second partition with
e2fsck -f /dev/mmcblk0p2 and remove the
etc/ld.so.preload file inside this partition.
Note, that at the time of this writing, this issue has already been fixed, so it is safe to upgrade your Pi if it is still up and running, or right after repairing your Raspbian root filesystem.
This incident is very unfortunate, as you need to physically access your board to recover from it. I hope you don’t run updates as frequently as we do (or right after the time when the update was issued), and that your Pis are not impacted, otherwise possibly forcing you to travel or to crawl into difficult to access places to reach your boards.
However, I don’t want blame the community volunteers running Raspbian. They have made a terrific job maintaining this distro which had been flawlessly running for more many years on our systems. This seems as good as what we get from commercial distributions.
I hope that not too many services ran by Raspberry Pis will be disrupted because of this issue. However, that may be yet another way to prove how popular such devices are.