Thanks to community contributions, our engineer Luca Ceresoli has recently published a fix to the zynqmp-pmufw-builder repository that allows building a fully working PMU Firmware binary. Rebooting had previously been broken for a long time.

Building the PMU Firmware

For the uninitiated, the PMU (Platform Management Unit) is an auxiliary Microblaze processor present on the Xilinx/AMD ZynqMP System-on-Chips, whose role includes handling power states and power domains, as well as rebooting the chip. The PMU Firmware (often called PMUFW) is the small program running on it, and it is loaded during the early boot process. The PMUFW source code is available in the Xilinx embeddedsw git repository.

A few years ago Luca has published zynqmp-pmufw-builder, a simple script to allow building the PMU Firmware. One goal of that simple project is to make the build process easy and quick, without needing a complex Yocto setup. Another goal it to document the build process itself by being as simple and readable as possible.

The reboot bug

Unfortunately, about one year ago it has been reported that the PMUFW binaries built by this script are hanging forever during the reboot process, making it unusable in most cases.

Since the embeddedsw code being built was the same, the toolchain used was the first suspect: zynqmp-pmufw-builder leverages crosstool-NG to build a basic Microblaze toolchain, while Xilinx/AMD supports either using their Vitis IDE or the meta-xilinx Yocto layers which build their own Microblaze toolchain. There must be a difference in the toolchain that creates this bug, maybe due to one of the 50+ patches that the meta-microblaze layer by Xilinx applies on top of gcc.

A workaround…

Recently AMD engineer Ibai Erkiaga Elorza gave a big contribution by investigating the problem in deep detail. He found the issue in the initialization of the TCM memories during the RPU cluster reset process: the PMUFW code is initializing the TCM memories using memset(), which does not guarantee that memory will be written using 32-bit accesses as required. The toolchain that zynqmp-pmufw-builder used to build via crosstool-NG does not work because it implements memset() using 8-bit accesses. Other toolchains work for one reason or another.

Ibai also found that disabling CT_LIBC_NEWLIB_ENABLE_TARGET_OPTSPACE in the crosstool-NG configuration would avoid this problem. This is only a workaround because the code should work with every reasonable toolchain, and it also requires to rebuild the toolchain, but it did the trick!

Luca added this workaround, which finally allowed zynqmp-pmufw-builder to build PMUFW binaries which are able to reboot correctly! He also published new binary images for the ZynqMP-based Kria SoMs that have reboot working properly. Building binaries for non-Kria ZynqMP boards should work as well now.

…and a proper fix!

Shortly later, AMD engineer Neal Frager went even further and prepared a patch to fix the bug at its root: the PMUFW code. His patch is simply replacing the memset() calls with a for loop using 32-bit pointers. With this simple change the toolchain workaround is not required anymore.

Luca added Neal’s patch to the zynqmp-pmufw-builder script along with code to apply it  before building the PMUFW. He then removed the workaround from the crosstool-NG configuration and verified that everything is working, including reboot. The result is on the zynqmp-pmufw-builder master branch.

We would like to thank Ibai and Neal for their effort in getting this very annoying issue solved!