When we, the PMDK team, want to check performance of our library, either to see if there was any regression or if our tweaks did a good job, we run benchmarks. One of them is FIO. It helps us simulate synthetic traffic of reads and writes to a pmem device. In this blog post I will introduce this tool and explain how we commonly use it.
For starters, links to needed software:
And, of course, to execute any FIO tests using pmem engines you need PMDK installed in your system.
Tests can be run on emulated pmem or real hardware (devices
presented to system by NVDIMM Firmware Interface Table (NFIT), according to ACPI
Specification v6.0+). Either way you have to prepare device(s) before test
execution. First step is to configure a namespace (emulated pmem is delivered
already as a namespace). For this step you can use command ndctl
create-namespace. There are two main options to configure namespace in our
case, either you choose mode fsdax or devdax. Namespace configured with
fsdax is a block device with support of dax operations. It can host a
dax-capable file system (e.g. ext4 or xfs) and should be configured like:
ndctl create-namespace --type=pmem --mode=fsdax --region=X [--align=4k]
mkfs.ext4 /dev/pmemX
mkdir /mnt/pmem
mount /dev/pmemX /mnt/pmem -o daxIn all cases regions should be created and maintained using administrative tool specific for your platform. Optional parameter align can be used to change alignment, while default for both mentioned modes is 2M.
Namespace with devdax mode is a character device and is a raw access analogue of filesystem-dax. Configuration is simpler, because there’s no need for file system preparation:
ndctl create-namespace --type=pmem --mode=devdax --region=X [--align=4k]To sum up, when you execute the following commands:
ls /mnt/pmem
ls /dev/dax*You should see mounted file system or dev-dax device available to use.
Installation of FIO is pretty straightforward and is described on its GitHub page. Execution is also not complicated, but it requires preparation of the actual benchmark workload (workload is a set of options describing how FIO will generate the traffic for measurements). We’ll see how workloads look for specific jobs below, but first a few general rules:
time_based parameter is not set FIO will run until
requests transfer data of given size; duration of a job would be unpredictable),FIO is generating I/O traffic using engines specific for the job. To specify
which engine is used in a job there’s workload’s option ioengine=my_engine.
All of them are described in their c file and all have corresponding
examples. Few of them are related to persistent memory:
libpmem
This engine reads/writes data using libpmem library. Works on a namespace
created in fsdax mode. Full example workload for generating traffic of
sequential reads using this engine can be found here. There are
additional comments within the jobfile to explain specific parameters.
dev-dax
It also uses libpmem library, but as the name suggest it is specified to work
with device-dax devices. Our full example workload DaxSeqR.fio
shows how to properly use FIO with /dev/dax. Since we don’t work on a “regular”
file we use a little trick to achieve more realistic measurements - we set a
separate space for each thread doing reads/writes using option
offset_increment=(int)GB. This way threads’ requests do not overlap each
other and results are not cached in the processor. In case FIO reads the same
part over and over (using different threads on the same space) it ends up not
reading from the device.
pmemblk
This engine is using libpmemblk library. Results delivered by this engine will
not show you the best performance of your hardware, only what this specific
library is capable of. While using this engine, blocksize and
size of a file are given as part of filename option, like here:
filename=/mnt/pmem6/testjob,512,1024000
#size=1024000M
#bs=512This is a bit different approach, comparing with other engines which use parameter “bs” and “size” (see commented part above). Full example workload doing traffic of sequential reads for pmemblk can be found here.
mmap
It’s the most “basic” of mentioned engines, because its purpose is just to read
from/write to a memory mapped region. It can be used with pmem, but is not
tailor-made. It generates traffic doing memcpy to/from memory region. Difference
between this engine and libpmem is that it doesn’t use PMDK library and hence
doesn’t take advantages of functions specific for writing to persistent memory.
Again, full example working workload is available: MmapSeqR.fio
Command to run FIO is: fio [options] [jobfile] ....
Since we use workloads defined in a file (as opposed to specifying parameters in command line), we execute FIO with command like:
numactl -N 0 fio --output=my_workload.json --output-format=json my_workload.fio
numactl command guarantees that processes are pinned to selected
numa node. The same can also be achieved by assigning CPU mask for FIO using
option cpus_allowed. We chose JSON format to save our results in
it, for more convenient automatic parsing. Last parameter is our input file.
In the resulting file (here called my_workload.json) we look for a list of [“jobs”] and then a section with the name of our job (“jobname”). There is all the benchmark results that our simulated traffic has delivered, including i.a. bw (bandwidth averaged per second, in kiB), iops (IOPS count) and lat_ns (total latency in nanoseconds) divided into read/write sub-sections and additional data like CPU usage (in %). For details, please see “Interpreting the output” section in FIO’s HOWTO page (you can look at “normal” output, since it has similar attributes as in “json” formatted output).