\documentclass[aspectratio=169,obeyspaces,spaces,hyphens,dvipsnames]{beamer}
\usepackage[utf8]{inputenc}
\usepackage{lmodern}% http://ctan.org/pkg/lm
\usepackage{minted}
\usepackage{hyperref}
\usepackage{xcolor}
\usepackage{pgfplots}
\usepackage{tikz}
\usepackage[normalem]{ulem}

\mode<presentation>
\usetheme{Bootlin}

\def\signed #1{{\leavevmode\unskip\nobreak\hfil\penalty50\hskip2em
  \hbox{}\nobreak\hfil(#1)
  \parfillskip=0pt \finalhyphendemerits=0 \endgraf}}

\newsavebox\mybox
\newenvironment{aquote}[1]
  {\savebox\mybox{#1}\begin{quotation}}
  {\signed{\usebox\mybox}\end{quotation}}

\title{Finding the Best Block Filesystem}
\authors{Michael Opdenacker}
\email{michael.opdenacker@bootlin.com}
\slidesurl{https://bootlin.com/pub/conferences/2023/eoss/}
\institute{Bootlin}
\conference{Embedded Linux Conference 2023}

\begin{document}

\addtocontents{toc}{\protect\setcounter{tocdepth}{-1}}
\section{Finding the Best Block Filesystem}
\addtocontents{toc}{\protect\setcounter{tocdepth}{2}}

\begin{frame}
  \begin{center}
     \Huge
     Strč prst skrz krk \\
     \vfill
     \pause
     \huge
     Happy to be in the Czech Republic! \\
     \pause
     But let's do the presentation in English this time ;-) \\
     \pause
     \vfill
     \footnotesize \url{https://en.wikipedia.org/wiki/Strč\_prst\_skrz\_krk}
  \end{center}
\end{frame}

\begin{frame}{Michael Opdenacker}
\begin{columns}
  \column{0.75\textwidth}
  \begin{itemize}
  \item Founder and Embedded Linux engineer at Bootlin:
    \begin{itemize}
    \item Embedded Linux {\bf expertise}
    \item {\bf Development}, consulting and training
    \item Strong open-source focus
    \end{itemize}
  \item About myself:
    \begin{itemize}
      \item Always happy to learn from every new project,\\
            and share what I learn.
      \item Initial author of Bootlin's freely available embedded Linux,
	    kernel and boot time reduction training materials (\url{https://bootlin.com/docs/})
      \item Current documentation maintainer for the Yocto Project
      \item Current maintainer of the Elixir Cross Referencer, making it
            easier to study the sources of big C projects like the Linux kernel.
            See \url{https://elixir.bootlin.com}.
    \end{itemize}
  \end{itemize}
  \column{0.25\textwidth}
    \begin{center}
    \vspace{0.5cm}
    \includegraphics[width=0.5\textwidth]{graphics/logo-cc.pdf}\\
    \vspace{0.5cm}
    \includegraphics[width=\textwidth]{graphics/elixir.pdf}
    \end{center}
\end{columns}
\end{frame}

\begin{frame}{Abstract}

\small \em It can be difficult to find the most appropriate filesystem
for your embedded system's eMMC or SD card storage. You can benchmark your
system with each of them, but it can be time consuming. In this talk, we will
compare all the actively maintained block filesystems supported in the Linux
kernel: Ext2, Ext4, XFS, Btrfs, F2FS, SquashFS and EROFS. Each of them will
be properly introduced, with its basic design principles and main features.

\vspace{1em}
We will then compare each filesystem in terms of kernel module size and load
time, filesystem mount time (important for boot time), filesystem size, as
well as read and write performance on a few simple scenarios. We will also
look for the best compression algorithms for filesystems with compression
options. Performance comparisons will be run both on a 32 bit ARM board and
on a 64 bit ARM one, both using a fast SD card as storage device.

\vspace{1em}
Filesystem performance can really depend on the benchmark, on your storage
and on your CPU, so no universal results should be expected. However, you
will learn what the best solution is in specific hardware configurations
and testcases.
\end{frame}

\begin{frame}{Disclaimer}
	\begin{itemize}
		\item I'm not a filesystems expert
		\item I'm just a regular embedded Linux engineer who was
			  given (a significant amount of) time by his company
              to do some research on this topic (thanks!)
    \end{itemize}
\end{frame}

\begin{frame}{Main goals}
    \scriptsize Not my first presentation on this topic:
    \url{https://bootlin.com/pub/conferences/2010/elce/elce2010-flash-filesystems.pdf}
    \includegraphics[height=0.75\textheight]{graphics/2010-presentation.png}
\end{frame}

\begin{frame}{Main goals}
  \begin{itemize}
    \item Rotating block devices are very rare now, especially in
	  embedded systems.
    \item Solid State Disks didn't exist when some filesystems were
      created. Filesystems have also evolved since 2010, in particular
      new ones have appeared (F2FS, EROFS).
    \item So, the main goal is to help you find out which filesystems
      are likely to work best for your embedded projects with Solid State
      storage.
    \item We chose to make our tests on MMC/SD, the most common
	  type of storage on embedded systems.
  \end{itemize}
\end{frame}

\begin{frame}{How to know whether a device is Solid State or not?}
  \begin{itemize}
    \item Check whether \code{/sys/block/<device>/queue/rotational} contains \code{0}.
    \item Unfortunately, this doesn't work for USB mass storage, always reported
	  as rotational.
    \item Correct information for MMC/SD.
  \end{itemize}
\end{frame}

\subsection{Available filesystems}

\begin{frame}{Ext2}
  One of the earliest Linux filesystem, introduced in 1993
  \begin{itemize}
    \item \kdochtml{filesystems/ext2}
    \item Still actively supported. Low metadata overhead, module size and RAM usage
    \item But risk of metadata corruption after an unclean shutdown.
	  You then need to run \code{e2fsck}, which takes time and may need operator intervention.
	  Can't reboot autonomously.
    \item First successor: \code{ext3} (2001), addressing this limitation with {\em Journaling}
	  \footnote[frame]{https://en.wikipedia.org/wiki/Journaling\_file\_system},
	  but wasn't scaling well. Now deprecated.
    \item {\em Journalism} reduces corruption and loss of information ;-)
    \item Date range: December 14, 1901 -- January 18, 2038!
  \end{itemize}
  Not recommended for embedded systems!
\end{frame}

\begin{frame}{Ext4}
	The modern successor of Ext2
	\begin{itemize}
		\item First introduced in 2006, filesystem with Journaling, without \code{ext3} limitations.
		\item Still actively developed (new features added). However, considered in 2008
		      by Ted Ts'o as a "stop-gap" based on old technologies.
		\item The default filesystem choice for many GNU/Linux distributions (Debian, Ubuntu)
		\item The \code{ext4} driver also supports \code{ext2} and \code{ext3} (one driver is sufficient).
		\item Noteworthy feature: transparent encryption (but compression not available).
		\item Date range: 1901 -- 2446. Fine for embedded systems without a 400 Y+ warranty!
		\item Minimum partition size to have a journal: 8MiB.
		\item Minimum partition size without a journal: 256KiB (only 32 inodes!).
		\item Create the filesystem with \code{mkfs.ext4}.
	\end{itemize}
	\url{https://en.wikipedia.org/wiki/Ext4}
\end{frame}


\begin{frame}{XFS}
	A Journaling filesystem
	\begin{itemize}
		\item Since 1994 (started by Silicon Graphics for the IRIX OS)
		\item Actively maintained and developed by Red Hat now
		\item Features: variable block size, direct I/O, online growth...
		\item Minimum partition size: 16MiB (9.7MiB of free space)
		\item Create the filesystem with \code{mkfs.xfs}.
	\end{itemize}
	\url{https://en.wikipedia.org/wiki/XFS}
\end{frame}

\begin{frame}{Btrfs}
	A copy-on-write filesystem (another Czech word!)
	\begin{itemize}
		\item Pronounced as "better F S", "butter F S" or "b-tree F S", since 2009.
		\item A modern filesystem with many advanced features: volumes, snapshots, transparent compression...
		\item Minimum partition size: 109MiB (only 32MiB of free space).
		\item Create the filesystem with \code{mkfs.btrfs}.
	\end{itemize}
	\url{https://en.wikipedia.org/wiki/Btrfs}
\end{frame}

\begin{frame}{F2FS --- Flash-Friendly File System}
	A log-structured filesystem
	\begin{itemize}
		\item Since 2012 (started by Samsung, actively maintained)
		\item Designed from the start to take into account the
		      characteristics of solid-state based storage (eMMC, SD, SSD)
		\item In particular, trying to make most writes sequential (best on SSD)
		\item Support for transparent encryption and compression (LZO,
              LZ4, Zstd), possible on a file by file (or file type)
              basis, through extended file attributes.
		\item Maximum partition size: 16TB, maximum file size: 3.94TB
		\item Minimum partition size: 52MiB (8MiB free space)
		\item Create the filesystem with \code{mkfs.f2fs}.
	\end{itemize}
	\url{https://en.wikipedia.org/wiki/F2FS}
\end{frame}

\begin{frame}{NILFS --- New Implementation of a Log-structured File System}
	A log-structured filesystem too, also known as NILFS2
	\begin{itemize}
		\item Since 2005 (started by NTT, maintained but not very actively)
		\item Treating the storage medium as a circular buffer, new blocks are always written to the end.
		\item Provides continuous snapshotting, easy to restore files modified or deleted at any recent time.
		      This create a weird behavior though: all past files (even erased ones) are kept and
		      this fills up all available space. Needs to run and configure \code{nilfs_cleanerd} to avoid this
		      (you also need to stop it before you can unmount your partition!).
		\item Supposed to be great at latency (minimizes seek time) and be the best at handling
		      many small files.
		\item Minimum partition size: 129MiB (48MiB free space)
		\item Create the filesystem with \code{mkfs.nilfs2}.
	\end{itemize}
	\url{https://en.wikipedia.org/wiki/NILFS}
\end{frame}

\begin{frame}{SquashFS --- A Read-Only and Compressed File System}
	The most popular choice for this usage
	\begin{itemize}
		\item Started by Phillip Lougher, since 2009 in the mainline kernel, actively maintained.
		\item Fine for parts of a filesystem which can be read-only (kernel, binaries...)
		\item Used in most live CDs and live USB distributions
		\item Supports several compression algorithms (Gzip, LZO, XZ, LZ4, Zstd)
		\item Supposed to give priority to compression ratio vs read performance
		\item Suitable for very small partitions
		\item Create a filesystem image with \code{mksquashfs}.
	\end{itemize}
	\url{https://en.wikipedia.org/wiki/SquashFS}
\end{frame}

\begin{frame}{EROFS --- Enhanced Read-Only File System}
	A more recent read-only, compressed solution
	\begin{itemize}
		\item Started by Gao Xiang (Huawei), since 2019 in the mainline kernel.
		\item Used in particular in Android phones (Huawei, Xiaomi, Oppo...)
		\item Supposed to give priority to read performance vs compression ratio
		\item EROFS implements compression into fixed 4KB blocks (better for read
		      performance), while SquashFS uses fixed-sized blocks of uncompressed
		      data.
		\item Unlike Squashfs, EROFS also allows for random access to files in
		      directories.
		\item Development seems more active than on SquashFS.
		\item Suitable for very small partitions
		\item Create a filesystem image with \code{mkfs.erofs}.
	\end{itemize}
	\url{https://en.wikipedia.org/wiki/EROFS}
\end{frame}

\begin{frame}{Filesystems we didn't test}
	\begin{itemize}
    \item Ext2: obsolete in 2038
	\item JFS: supported but legacy
	\item ReiserFS: lacks support, going away in a few years
	\item CramFS: supported but legacy
	\item BcacheFS: not merged yet!
	\end{itemize}
\end{frame}

\subsection{Raw benchmarks}

\begin{frame}{How SD cards are organized}
    \begin{columns}
    \column{0.5\textwidth}
    \includegraphics[width=\textwidth]{graphics/sdcard-architecture.pdf}
    \column{0.5\textwidth}
    \small
    \begin{itemize}
	\item NAND flash is organized in {\em pages} (typically 2-4K) and in
	          {\em erase blocks} (typically 128K).
	\item However, SD cards group several erase blocks together in {\em segments}
	      (typically 4M) which allows to address and manage more storage space.
	      The storage will never erase anything smaller than a segment.
    \end{itemize}
    \end{columns}
\end{frame}

\begin{frame}{Comparing SD cards}
    \includegraphics[width=\textwidth]{graphics/sd-cards.pdf}
\end{frame}

\begin{frame}[fragile]{How to infer the page size?}
    \begin{columns}
    \column{0.6\textwidth}
    \footnotesize
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{sh}
#!/bin/sh
b=512
while [ $b -le 67108864 ]
do
        echo -n $b
        time -f ",%e" chrt -f 99 dd status=none if=64M.img \
                      of=/dev/mmcblk0 bs=$b conv=fdatasync
        b=$(($b *2))
done
    \end{minted}
    \end{block}
    \includegraphics[width=\textwidth]{graphics/page-size.pdf}
    \column{0.4\textwidth}
    \footnotesize
    Test writing with increasing block sizes and find the best one:
        \begin{itemize}
	\item Any write smaller than 4K is inefficient.
	\item Writing bigger (sequential) blocks is unnecessary.
	\item Good to make sure that filesystems use at least 4K blocks.
	\end{itemize}
    \end{columns}
\end{frame}

\begin{frame}{Raw write tests on 8 SD cards}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/1.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/2.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/3.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/4.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/5.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/6.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/7.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-write/8.pdf}
\end{frame}

\begin{frame}{Raw write tests winner(s)}
    \begin{columns}
    \column{0.5\textwidth}
    \includegraphics[height=0.8\textheight]{graphics/raw-write/4k-write-performance.pdf}
    \column{0.5\textwidth}
    \includegraphics[width=\textwidth]{graphics/sandisk-extreme-64gb.jpg}\\
    \vspace{1cm}
    \scriptsize Note: first time I scan an SD card with a scanner!
    \end{columns}
\end{frame}

\begin{frame}[fragile]{How to infer the segment size?}
    \scriptsize
    \begin{columns}
    \column{0.51\textwidth}
	A good solution is to use {\em flashbench} from
	Arnd Bergmann:
	\begin{itemize}
    \item Available as a Debian/Ubuntu package.
    \item Can help to find the segment size by reading across several
          block size boundaries.
	\item Also offers useful \code{-f} option for looking for
		  special "FAT" sectors with better performance.
    \item See {\em Optimizing Linux with cheap flash drives}\\
		  \url{https://lwn.net/Articles/428584/}
	\end{itemize}
	\scriptsize \url{https://git.linaro.org/people/arnd/flashbench.git/}

    \column{0.49\textwidth}
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{text}
$ sudo flashbench -a /dev/mmcblk0
align 4294967296  pre 1.39ms  on 1.47ms  post 1.28ms  diff 139µs
align 2147483648  pre 1.3ms   on 1.47ms  post 1.33ms  diff 158µs
align 1073741824  pre 1.24ms  on 1.36ms  post 1.21ms  diff 139µs
align 536870912   pre 1.26ms  on 1.39ms  post 1.29ms  diff 114µs
align 268435456   pre 1.24ms  on 1.38ms  post 1.25ms  diff 131µs
align 134217728   pre 1.26ms  on 1.38ms  post 1.24ms  diff 130µs
align 67108864    pre 1.4ms   on 1.54ms  post 1.52ms  diff 81.6µs
align 33554432    pre 1.3ms   on 1.48ms  post 1.33ms  diff 161µs
align 16777216    pre 1.35ms  on 1.47ms  post 1.35ms  diff 116µs
align 8388608     pre 1.22ms  on 1.41ms  post 1.32ms  diff 140µs
align 4194304     pre 1.17ms  on 1.32ms  post 1.18ms  diff 147µs
align 2097152     pre 1.21ms  on 1.28ms  post 1.2ms   diff 78.4µs
align 1048576     pre 1.18ms  on 1.26ms  post 1.19ms  diff 81.4µs
align 524288      pre 1.19ms  on 1.27ms  post 1.19ms  diff 77.5µs
align 262144      pre 1.19ms  on 1.28ms  post 1.17ms  diff 96.5µs
align 131072      pre 1.16ms  on 1.25ms  post 1.18ms  diff 81.2µs
align 65536       pre 1.16ms  on 1.25ms  post 1.18ms  diff 74.4µs
align 32768       pre 1.18ms  on 1.28ms  post 1.17ms  diff 103µs
    \end{minted}
    \end{block}
    Tests on SD card 2 (KINGSTON 16 GB Class 10 U1)\\
    Here, there is a clear level change at 4M.\\
    That's less obvious with several of the other SD cards
    we surveyed, but in all cases, 4M turns out to be a safe
    choice (no risk to take a value that's the double of
    the actual one).
    \end{columns}
\end{frame}

\begin{frame}{Raw read tests on 8 SD cards}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/1.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/2.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/3.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/4.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/5.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/6.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/7.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/raw-read/8.pdf}
    \scriptsize See the level change at 1M!
\end{frame}

\subsection{Filesystem benchmarks}

\begin{frame}[fragile]{Ext4 vs... \pause Ext4}
    \small
    \begin{columns}
    \column{0.7\textwidth}
	\begin{itemize}
		\item Ext4 was created originally for rotating block storage.
			  Couldn't find any \code{mkfs.ext4} option which could help
			  on solid state storage.
		\item \code{mkfs.ext4}'s default block size, 4K, is a good value according
			  to our raw write tests.
		\item Did you know? By default, this command will automatically select suitable
		      \code{inode_ratio}, \code{block_size} and \code{inode_size} values
		      according to the size of the filesystem. You can override this
	          with the \code{-T} option and profiles in the \code{/etc/mke2fs.conf} file.
	\end{itemize}
    \column{0.3\textwidth}
      \begin{block}{}
      \begin{minted}[fontsize=\tiny]{text}
...
[fs_types]
        ...
        small = {
                inode_ratio = 4096
        }
        floppy = {
                inode_ratio = 8192
        }
        big = {
                inode_ratio = 32768
        }
        huge = {
                inode_ratio = 65536
        }
        ...
        largefile4 = {
                inode_ratio = 4194304
                blocksize = -1
        }
        hurd = {
             blocksize = 4096
             inode_size = 128
             warn_y2038_dates = 0
        }
      \end{minted}
      \end{block}
      \tiny Profiles in \code{/etc/mke2fs.conf}
    \end{columns}
\end{frame}

\begin{frame}{XFS vs XFS}
	\begin{itemize}
		\item Didn't find options which could increase performance on
              Solid State storage.
        \item By default, \code{mkfs.xfs} picks a 4K block size, which
		      is good with SD cards.
    \end{itemize}
\end{frame}

\begin{frame}[fragile]{Btrfs vs Btrfs}
    \begin{columns}
    \column{0.5\textwidth}
	\begin{itemize}
        \item By default, \code{mkfs.btrfs} picks a 4K block size, which
		      is good with SD cards.
        \item \code{mkfs.btrfs} does detect an SSD when given an SD
			  card partition.
        \item In past versions, \code{mkfs.btrfs} was then creating the
              filesystem with the \code{-m single}. This corresponds
              to not duplicating metadata to two physical
			  locations (\code{-m dup}).
		\item However, this information is not trustworthy and we
			  now always have \code{-m dup} by default.
        \item See \href{https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html#man-mkfs-dup-profiles-on-a-single-device}
			  {DUP profiles on a single device}.
    \end{itemize}
    \column{0.5\textwidth}
		\includegraphics[width=\textwidth]{graphics/btrfs-single.pdf}\\
		\scriptsize
		\vspace{1cm}
		\code{mkfs.btrfs -m single} reduces space (-3.5\%) and increases read and
             write performance. Keeping this option!
    \end{columns}
\end{frame}

\begin{frame}[fragile]{Btrfs ssd options}
    \begin{columns}
    \column{0.5\textwidth}
    \small
	\begin{itemize}
        \item Btrfs offers SSD related mount options: \code{ssd},
              \code{ssd_spread} and \code{nossd}.
        \item \code{ssd} is enabled by default if your storage
		      is detected as SSD:\\
			  \code{[  915.149865] BTRFS info (device mmcblk0p3): enabling ssd optimizations}\\
			  Otherwise (USB mass storage), you may need to set it manually.
		\item Didn't manage to use \code{ssd_spread}, was causing
			  \code{No space left on device} errors even though
			  there was plenty of space.
		\item See \url{https://btrfs.readthedocs.io/en/latest/ch-mount-options.html}.
    \end{itemize}
    \column{0.5\textwidth}
		\includegraphics[width=\textwidth]{graphics/btrfs-nossd.pdf}\\
		\scriptsize
		\vspace{1cm}
		The \code{ssd} mount option brings minor read and write speedups
        (< 1\%).
    \end{columns}
\end{frame}

\begin{frame}[fragile]{Btrfs compression options}
    \small
    \begin{columns}
    \column{0.6\textwidth}
	\begin{itemize}
        \item Btrfs also offers mount options for compression:
              \code{compress=none|zlib|lzo|zstd} (default: \code{none}).
		      See \href{https://btrfs.readthedocs.io/en/latest/ch-mount-options.html}{mount options}.
	    \item Compression helps a bit with read time (- 10\% at best),
			  but significantly hurts write time (though acceptably with
              \code{lzo}), at least on the 1GHz CPU we tested.
		\item Compression is smart: giving up if file contents turn out
              not to be compressible (e.g. video).
		\item Sticking to \code{none}, otherwise would use
			  \code{lzo} if compression is needed, or \code{zlib} if
              compression rate matters more than performance.
		\item Here, what you choose will depend on your system, how
			  often you write, what kind of files...
    \end{itemize}
    \column{0.4\textwidth}
		\includegraphics[width=\textwidth]{graphics/btrfs-compression.pdf}\\
		\includegraphics[width=\textwidth]{graphics/btrfs-compression-used-space.pdf}\\
		\scriptsize
		\vspace{1cm}
    \end{columns}
\end{frame}

\begin{frame}{F2FS vs F2FS}
    \footnotesize
    \begin{columns}
    \column{0.6\textwidth}
	\begin{itemize}
        \item F2FS compression options are not very straightforward to
              use. They allow for selective compression of files
              (specific files, files with a given extension). However,
              it's possible to try to compress all files.
		\item Need to create the filesystem as follows:\\
              \code{mkfs.f2fs -O compression,extra_attr}
        \item Need to mount the filesystem as follows:\\
			  \code{-o compress_extension=*,compress_algorithm=lzo|lz4|zstd|lzo-rle}
        \item Also tried the \code{lazytime} and \code{atgc,gc_merge}
			  mount options, but they didn't help with performance, except marginally
              for video writing.
		\item See \url{https://www.kernel.org/doc/html/latest/filesystems/f2fs.html}
			  and \url{https://wiki.archlinux.org/title/F2FS}.
    \end{itemize}
    \column{0.4\textwidth}
		\scriptsize
		\includegraphics[width=\textwidth]{graphics/f2fs-compression.pdf}\\
		\vspace{1cm}
		Compression can be useful, but here there are no sizable
		performance benefits. Keeping the uncompressed version.
    \end{columns}
\end{frame}

\begin{frame}{nilfs2 vs nilfs2}
	\begin{itemize}
        \item Didn't see any \code{mount} option
			  which could help with performance.
        \item See \url{https://www.kernel.org/doc/html/latest/filesystems/nilfs2.html}
		\item \code{mkfs.nilfs2} options:
			\begin{itemize}
			\item \code{-b} (block-size) already has a good default value (4K)
			\item \code{-B} (blocks-per-segment) too (8M)
			\end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{SquashFS vs SquashFS}
\begin{columns}
  \footnotesize
  \column{0.6\textwidth}
	Comparing the read performance of the various compression options
    (using a Raspberry Pi OS Lite root filesystem, 1.2 GB, ARM binaries).\\
	\includegraphics[height=0.6\textheight]{graphics/squashfs-compression.pdf}\\
	\code{lzo} looks like the best compromise in terms of speed and space.
	We will use SquashFS with this compression scheme.
  \column{0.4\textwidth}
  \footnotesize
  Notes: tried several promising non-default kernel configuration options:
  \begin{itemize}
	\item \kconfig{SQUASHFS_4K_DEVBLK_SIZE}: enforcing 4K blocks. This slightly degraded performance.
	\item \kconfig{SQUASHFS_FILE_DIRECT}: directly decompressing to the file cache. Didn't see
	      any noticeable impact on read performance.
	\item Haven't tried the multi-threaded decompression options yet.
  \end{itemize}
\end{columns}
\end{frame}

\begin{frame}{EROFS vs EROFS (1)}
\begin{columns}
  \small
  \column{0.6\textwidth}
	Comparing the read performance of the various compression options (using an Raspberry Pi OS Lite
	root filesystem (1.2 GB, ARM binaries).
	\includegraphics[height=0.6\textheight]{graphics/erofs-compression.pdf}\\
  \column{0.4\textwidth}
  \footnotesize
  We tried 4 options to create an EROFS image with \code{mkfs.erofs}:
  \begin{itemize}
	\item No option: no compression
	\item \code{-zlz4hc}: compresses with the {\em MicroLZMA} compressor (Linux 5.16+)
	\item \code{-zlz4hc,12}: compresses with the {\em MicroLZMA} compressor, best compression
	\item \code{-C65536}: adding {\em big pcluster} feature for bigger clusters (Linux 5.13+) 
  \end{itemize}
\end{columns}
\end{frame}

\begin{frame}{EROFS vs EROFS (2)}
\begin{columns}
  \small
  \column{0.6\textwidth}
	\includegraphics[width=\textwidth]{graphics/erofs-read-time.pdf}\\
  \column{0.4\textwidth}
  \footnotesize
  \begin{itemize}
	\item We will continue tests with \code{-zlz4hc,12}, giving the best time results.
	\item There just seems to be a penalty on the host machine creating the image!
  \end{itemize}
\end{columns}
\end{frame}

\subsection{Filesystem benchmarks - Real comparisons}

\begin{frame}{Kernel module size}
    \includegraphics[height=0.8\textheight]{graphics/module-size.pdf}
\end{frame}

\begin{frame}{Module loading time}
    \includegraphics[height=0.8\textheight]{graphics/init-time.pdf}
\end{frame}

\begin{frame}{Mounting time}
    \includegraphics[height=0.8\textheight]{graphics/mount-time.pdf}
\end{frame}

\begin{frame}{Boot time: module loading and mounting time}
    \includegraphics[height=0.8\textheight]{graphics/init-plus-mount-time.pdf}
\end{frame}

\begin{frame}[fragile]{Used space}
	\begin{columns}
    \column{0.6\textwidth}
    \includegraphics[width=\textwidth]{graphics/used-space.pdf}
    \column{0.4\textwidth}
    \small
    Contents: Raspberry Pi OS Lite root filesystem, 1.2 GB, ARM binaries.
	\end{columns}
\end{frame}

\begin{frame}[fragile]{Reading time}
	\begin{columns}
    \column{0.6\textwidth}
    \includegraphics[width=\textwidth]{graphics/read-time.pdf}
    \column{0.4\textwidth}
    \small Reading all the files in the filesystem
    (contents of a Raspberry Pi OS Lite root filesystem, 1.2 GB, ARM binaries).
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{sh}
#!/bin/sh
tar cf /dev/null /mnt/data
    \end{minted}
    \end{block}
	\end{columns}
\end{frame}

\begin{frame}[fragile]{Writing time}
	\begin{columns}
    \column{0.6\textwidth}
    \includegraphics[width=\textwidth]{graphics/write-time.pdf}
    \column{0.4\textwidth}
    \small Copying a Debian ARM root filesystem (319M) 5 times
    from a pre-loaded ramfs to the target filesystem.
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{sh}
#!/bin/sh
num=5
for i in `seq $num`
do
    cp -r /mnt/ramfs/debian-arm /mnt/data/$i
    echo "Copying $i / $num"
done
sync
    \end{minted}
    \end{block}
	\end{columns}
\end{frame}

\begin{frame}[fragile]{Reading, removing and writing time}
	\begin{columns}
    \column{0.6\textwidth}
    \includegraphics[width=\textwidth]{graphics/read-write-time.pdf}
    \column{0.4\textwidth}
    \small Continued from the previous tests. Remove 1 directory
    out of 5 (Debian ARM root filesystem, 319M), copy the oldest
    remaining one to a new one, all this 5 times.
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{sh}
#!/bin/sh
num=5
i=1
while [ $i -le $num ]
do
    echo "Loop $i..."
    rm -rf /mnt/data/$i
    i=`expr $i + 1`
    cp -r /mnt/data/$i /mnt/data/`expr $i + 4`
done
sync
    \end{minted}
    \end{block}
	\end{columns}
\end{frame}

\begin{frame}[fragile]{Removing time}
	\begin{columns}
    \column{0.6\textwidth}
    \includegraphics[width=\textwidth]{graphics/remove-time.pdf}
    \column{0.4\textwidth}
    \small Continued from the previous tests. Remove all 5 directories
    (Debian ARM root filesystem, 319M).
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{sh}
#!/bin/sh
/bin/rm -rf /mnt/data/*
sync
    \end{minted}
    \end{block}
	\end{columns}
\end{frame}

\begin{frame}[fragile]{Sequential write time}
	\begin{columns}
    \column{0.6\textwidth}
    \includegraphics[width=\textwidth]{graphics/video-write-time.pdf}
    \column{0.4\textwidth}
    \small New test. Copy a preloaded video
    (\code{big_buck_bunny_720p_surround.avi}, 317M)
    to the filesystem, 5 times.
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{sh}
#!/bin/sh
for i in `seq 5`
do
    cp /mnt/ramfs/video.avi /mnt/data/video.avi.$i
done
sync
    \end{minted}
    \end{block}
	\end{columns}
\end{frame}

\begin{frame}[fragile]{Sequential read time}
	\begin{columns}
    \column{0.6\textwidth}
    \includegraphics[width=\textwidth]{graphics/video-read-time.pdf}
    \column{0.4\textwidth}
    \small Continued from the previous test. After unmounting and
    remounting the filesystem, read the stored videos
    (\code{big_buck_bunny_720p_surround.avi}, 317M, 5 times)
    \begin{block}{}
    \begin{minted}[fontsize=\tiny]{sh}
#!/bin/sh
cat /mnt/data/video.avi.* > /dev/null
    \end{minted}
    \end{block}
	\end{columns}
\end{frame}

\begin{frame}{The BIG picture}
    \includegraphics[width=0.32\textwidth]{graphics/init-plus-mount-time.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/read-time.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/write-time.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/read-write-time.pdf}
    % \includegraphics[width=0.32\textwidth]{graphics/remove-time.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/video-write-time.pdf}
    \includegraphics[width=0.32\textwidth]{graphics/video-read-time.pdf}
\end{frame}

\begin{frame}{Our observations}
    \includegraphics[height=0.8\textheight]{graphics/rating.pdf}\\
    \scriptsize {\bf Caution:} specific to our own tests!
\end{frame}

\begin{frame}{What to remember}
    \begin{itemize}
    \item Seems like Ext2 is going away (2038 limit)
    \item EROFS seems to be the fastest read-only filesystem as expected.
    \item SquashFS is great to minimize space while keeping very good
          read performance.
	\item Ext4 remains a very good default choice for read-write filesystem
          in all aspects.
	\item F2FS seems to be the second best choice.
    \item Btrfs turns out to be bulky, complicated and powerful,
          but created with \code{-m single}, is a solid choice too (except for boot time).
    \item XFS is a pretty good choice too, and easy to use.
    \item Compression doesn't seem to help with performance (at least
		  with our rather slow, single-core CPU).
    \item Nilfs2 can give you great results, but also the worst ones,
          depending on your usage scenario.
    \end{itemize}
    Always try with your own hardware and applications!
\end{frame}

\begin{frame}{Lessons learned}
    \begin{itemize}
	\item Flashing an SD card with \code{dd}: the {\bf page size} (usually 4K) is the best block size. Don't exceed 1M.
	\item Reading an SD card with \code{dd}: the {\bf segment size} is one of the best block sizes. Bigger or
	      smaller blocks degrade performance.
	\item Tried to use \code{flashbench -f} to look for special "FAT" segment with better
	      performance. Didn't find anything noticeable.
	\item Anyway, for Journaling filesystems, the journal cannot fit in the first sectors,s
	      as it is too big anyway. You may store it elsewhere though, if you have faster storage.
    \end{itemize}
\end{frame}

\begin{frame}{Limitations}
    \begin{itemize}
	\item Haven't tested on ARM64 yet (faster CPU), and with multi-core CPUs either
    \item Haven't tested real random writing (modifying random files in
          place) and reading.
    \item Tests made only on one SD card, not on eMMC, and neither on USB
          nor on NVME, SATA...
    \end{itemize}
\end{frame}

\begin{frame}{Further resources}
	\begin{itemize}
		\item Peter Chubb: SD cards and filesystems for embedded systems (2015)\\
		      \url{http://mirror.linux.org.au/pub/linux.conf.au/2015/Case_Room_2/Friday/SD_Cards_and_filesystems_for_Embedded_Systems.webm}
	    \item Richard Weinberger: EROFS vs. SquashFS: A Gentle Benchmark (2022)\\
		      \url{https://blog.sigma-star.at/post/2022/07/squashfs-erofs/}
	\end{itemize}
\end{frame}

\questionslide

\end{document}
