MapperX 性能测试

1 搭建 dm-cache 环境

1.1 创建物理卷

1.1.1 修改虚拟机设置，添加两块虚拟磁盘

1.1.2 对新建的磁盘进行分区及格式化的工作（以其中一块为例）

执行 fdisk /dev/sdb，

$ fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x6f3b5f16.

Command (m for help):

在上一步骤的基础上输入：m，

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   g   create a new empty GPT partition table
   G   create an IRIX (SGI) partition table
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

然后根据提示输入：n，以添加新的分区，

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p):

依次输入 p 和 1，

Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1

接着便会提示卷的起始扇区和结束扇区，都保持默认按回车的即可（意思是只分一个区），

First sector (2048-4194303, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4194303, default 4194303):
Using default value 4194303
Partition 1 of type Linux and of size 2 GiB is set

Command (m for help):

输入 “w” 保存并退出，

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

再次使用 “fdisk -l ”这个命令来查看会发现出现了/dev/sdb1（说明已经完成了分区工作）,

$ fdisk -l
............
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     4194303     2096128   83  Linux

Disk /dev/mapper/centos-root: 18.2 GB, 18249416704 bytes, 35643392 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
............

1.1.3 初始化分区为物理卷

使用 pvcreate 命令新建物理卷，

1 2	$ pvcreate /dev/sdb1 Physical volume "/dev/sdb1" successfully created.

查看当前的物理卷，

$ pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               centos
  PV Size               <19.00 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              4863
  Free PE               0
  Allocated PE          4863
  PV UUID               Y0aMSP-YKlD-Y2yH-c01s-4FME-4loY-GqqDVw

  "/dev/sdb1" is a new physical volume of "<2.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdb1
  VG Name
  PV Size               <2.00 GiB
  Allocatable           NO
  PE Size               0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               i3qKtd-WqIU-aIxY-0niW-OwLd-GFSC-HjKXu2

物理卷 /dev/sdb1 出现在列表中

1.2 将物理卷添加到卷组

使用 vgextend 命令将物理卷添加到卷组，

1 2	$ vgextend centos /dev/sdb1 Volume group "centos" successfully extended

1.3 在卷组上创建逻辑卷

使用 lvcreate 命令创建逻辑卷，

1 2	$ lvcreate -n dm_cache -L 1G centos /dev/sdb1 Logical volume "dm_cache" created.

1.4 将 fast LV 设置成 main LV 的高速缓存

使用 lvconvert 命令建立高速缓存磁盘和大容量磁盘的关联，

$ lvconvert --type cache --cachepool dm_cache centos/dm_main
  WARNING: Converting centos/dm_cache to cache pool's data volume with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert centos/dm_cache? [y/n]: y
  Converted centos/dm_cache to cache pool.
  Logical volume centos/dm_main is now cached.

查看当前的逻辑卷，

$ lvdisplay
  --- Logical volume ---
  LV Path                /dev/centos/dm_main
  LV Name                dm_main
  VG Name                centos
  LV UUID                FsTspi-GPTM-bCXd-oLZ6-4hdo-OUJo-LErddE
  LV Write Access        read/write
  LV Creation host, time localhost.localdomain, 2021-12-12 02:21:08 -0500
  LV Cache pool name     dm_cache
  LV Cache origin name   dm_main_corig
  LV Status              available
  # open                 0
  LV Size                19.90 GiB
  Cache used blocks      0.05%
  Cache metadata blocks  2.15%
  Cache dirty blocks     0.00%
  Cache read hits/misses 5 / 44
  Cache wrt hits/misses  0 / 0
  Cache demotions        0
  Cache promotions       9
  Current LE             5095
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2

2 安装性能测试工具

2.1 安装 fio

1	$ yum install -y fio

fio 是一个多线程 I/O 生成工具,可以生成多种 I/O 模式,用来测试磁盘设备的性能(也包含文件系统:如针对网络文件系统 NFS 的 I/O 测试)。

2. 下载 MSR Cambridge Traces

点击链接 http://iotta.snia.org/traces/block-io/388，进入下载页面，

3 下载、编译、安装 MapperX

根据 MapperX README.md 的提示步骤，下载、编译、安装 MapperX
编译内核，

1	$ make CONFIG_DM_CACHE=m -j4

编译指定模块，

1	$ make CONFIG_DM_CACHE=m modules SUBDIRS=drivers/md

安装模块，

1	$ sudo make modules_install

安装内核，

1	$ sudo make install

重启虚拟机，

$ reboot

4 Benchmark

4.1 Random Write Latency

4.1.1 sync

dm-cache 默认的写策略是 write through，确保每次写到高速缓存的数据都能同步写到大容量低速设备上，不必担心数据丢失的风险，

1 2	$ dmsetup status centos/dm_main 0 41738240 cache 8 41/2048 128 16384/16384 1690 123 1718562 23599 94 16478 0 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -

查看高速缓存的块大小，

1
2
3

$ lvs -o+chunksize centos/dm_main
  LV      VG     Attr       LSize  Pool       Origin          Data%  Meta%  Move Log Cpy%Sync Convert Chunk
  dm_main centos Cwi-a-C--- 19.90g [dm_cache] [dm_main_corig] 0.05   2.15            0.00             64.00k

默认是 64 KB

运行 fio 开始测试，

1	$ fio -filename=/dev/centos/dm_main -direct=1 -iodepth=1 -thread -rw=randwrite -ioengine=psync -bs=64k -size=100M -numjobs=50 -group_reporting -name=rand_100write_64k_clat > rand_100write_64k_clat

实验数据，

$ cat rand_100write_64k_clat
rand_100write_64k_clat: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1
...
fio-3.7
Starting 50 threads

rand_100write_64k_clat: (groupid=0, jobs=50): err= 0: pid=3135: Sun Dec 12 07:50:37 2021
  write: IOPS=5154, BW=322MiB/s (338MB/s)(5000MiB/15519msec)
    clat (usec): min=150, max=209870, avg=9572.78, stdev=5985.26
     lat (usec): min=151, max=209874, avg=9575.53, stdev=5985.52
    clat percentiles (usec):
     |  1.00th=[   529],  5.00th=[  1647], 10.00th=[  6849], 20.00th=[  7570],
     | 30.00th=[  8029], 40.00th=[  8455], 50.00th=[  8979], 60.00th=[ 10159],
     | 70.00th=[ 10814], 80.00th=[ 11469], 90.00th=[ 12387], 95.00th=[ 13435],
     | 99.00th=[ 27132], 99.50th=[ 44827], 99.90th=[ 84411], 99.95th=[115868],
     | 99.99th=[154141]
   bw (  KiB/s): min= 1792, max=14336, per=2.00%, avg=6589.61, stdev=1289.25, samples=1497
   iops        : min=   28, max=  224, avg=102.72, stdev=20.18, samples=1497
  lat (usec)   : 250=0.04%, 500=0.80%, 750=1.44%, 1000=1.10%
  lat (msec)   : 2=2.26%, 4=1.90%, 10=50.81%, 20=40.11%, 50=1.15%
  lat (msec)   : 100=0.32%, 250=0.07%
  cpu          : usr=0.16%, sys=1.47%, ctx=79970, majf=0, minf=31
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,80000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=322MiB/s (338MB/s), 322MiB/s-322MiB/s (338MB/s-338MB/s), io=5000MiB (5243MB), run=15519-15519msec

clat（Completion Latency）p50为8979，avg为9572.78，p99.9为84411

重新运行 fio，其他参数不变，设置 iodepth=16，测试 IOPS，

1 2	write: IOPS=5111, BW=319MiB/s (335MB/s)(5000MiB/15652msec) iops : min= 42, max= 271, avg=102.47, stdev=23.19, samples=1492

将 dm-cache 的块大小更改成 128KB，

$ lvconvert --splitcache centos/dm_main
  Logical volume centos/dm_main is not cached and cache pool centos/dm_cache is unused.
$ lvconvert --type cache --cachepool dm_cache centos/dm_main --chunksize 128
Do you want wipe existing metadata of cache pool centos/dm_cache? [y/n]: y
  Logical volume centos/dm_main is now cached.

重复上面的实验，先测 clat，

$ fio -filename=/dev/centos/dm_main -direct=1 -iodepth=1 -thread -rw=randwrite -ioengine=psync -bs=128k -size=100M -numjobs=50 -group_reporting -name=rand_100write_128k_clat > rand_100write_128k_clat
 clat (usec): min=294, max=286567, avg=16790.77, stdev=11746.64
 
 clat percentiles (usec):
     |  1.00th=[  1532],  5.00th=[  4146], 10.00th=[  7373], 20.00th=[ 13566],
     | 30.00th=[ 15401], 40.00th=[ 16909], 50.00th=[ 17695], 60.00th=[ 18220],
     | 70.00th=[ 18744], 80.00th=[ 19268], 90.00th=[ 20317], 95.00th=[ 21365],
     | 99.00th=[ 32375], 99.50th=[ 65274], 99.90th=[263193], 99.95th=[270533],
     | 99.99th=[283116]

然后再测 iops，

1
2
3

$ fio -filename=/dev/centos/dm_main -direct=1 -iodepth=16 -thread -rw=randwrite -ioengine=psync -bs=128k -size=100M -numjobs=50 -group_reporting -name=rand_100write_128k_iops > rand_100write_128k_iops
write: IOPS=3133, BW=392MiB/s (411MB/s)(5000MiB/12764msec)
iops        : min=   39, max=   88, avg=62.72, stdev= 9.68, samples=1215

将 dm-cache 的块大小更改成 256KB，

$ lvconvert --splitcache centos/dm_main
  Logical volume centos/dm_main is not cached and cache pool centos/dm_cache is unused.
$ lvconvert --type cache --cachepool dm_cache centos/dm_main --chunksize 256
Do you want wipe existing metadata of cache pool centos/dm_cache? [y/n]: y
  Logical volume centos/dm_main is now cached.

测试 clat，

$ fio -filename=/dev/centos/dm_main -direct=1 -iodepth=1 -thread -rw=randwrite -ioengine=psync -bs=256k -size=100M -numjobs=50 -group_reporting -name=rand_100write_256k_clat > rand_100write_256k_clat
 clat (usec): min=349, max=266827, avg=26564.90, stdev=14122.46
 
 clat percentiles (usec):
     |  1.00th=[  1696],  5.00th=[  6980], 10.00th=[ 21627], 20.00th=[ 24511],
     | 30.00th=[ 25297], 40.00th=[ 25822], 50.00th=[ 26346], 60.00th=[ 26870],
     | 70.00th=[ 27395], 80.00th=[ 28181], 90.00th=[ 30016], 95.00th=[ 33162],
     | 99.00th=[ 80217], 99.50th=[125305], 99.90th=[221250], 99.95th=[231736],
     | 99.99th=[258999]

测试 iops，

1
2
3

$ fio -filename=/dev/centos/dm_main -direct=1 -iodepth=16 -thread -rw=randwrite -ioengine=psync -bs=256k -size=100M -numjobs=50 -group_reporting -name=rand_100write_256k_iops > rand_100write_256k_iops
write: IOPS=1678, BW=420MiB/s (440MB/s)(5000MiB/11916msec)
iops        : min=   13, max=   75, avg=33.92, stdev= 6.59, samples=1107

4.1.2 normal

将 dm-cache 的 write mode 更改为 writeback，

$ lvconvert --splitcache centos/dm_main
  Logical volume centos/dm_main is not cached and cache pool centos/dm_cache is unused.
$ lvconvert --type cache --cachepool dm_cache centos/dm_main --chunksize 64 --cachemode writeback
Do you want wipe existing metadata of cache pool centos/dm_cache? [y/n]: y
  Logical volume centos/dm_main is now cached.

重复上面的实验，

当 dm-cache 块大小为 64KB 时，clat 为，

clat (usec): min=122, max=254254, avg=3028.95, stdev=8495.91
  
clat percentiles (usec):
 |  1.00th=[   293],  5.00th=[   498], 10.00th=[   717], 20.00th=[  1106],
 | 30.00th=[  1450], 40.00th=[  1778], 50.00th=[  2114], 60.00th=[  2442],
 | 70.00th=[  2868], 80.00th=[  3556], 90.00th=[  4686], 95.00th=[  5604],
 | 99.00th=[ 11076], 99.50th=[ 58459], 99.90th=[147850], 99.95th=[183501],
 | 99.99th=[235930]

iops 为，

1 2	write: IOPS=14.8k, BW=925MiB/s (970MB/s)(5000MiB/5405msec) iops : min= 100, max= 782, avg=322.91, stdev=88.47, samples=435

当 dm-cache 块大小为 128KB 时，clat 为，

clat (usec): min=117, max=502708, avg=4851.00, stdev=8269.09

clat percentiles (usec):
 |  1.00th=[   502],  5.00th=[   898], 10.00th=[  1401], 20.00th=[  2278],
 | 30.00th=[  2933], 40.00th=[  3458], 50.00th=[  3949], 60.00th=[  4490],
 | 70.00th=[  5080], 80.00th=[  5932], 90.00th=[  8029], 95.00th=[  9634],
 | 99.00th=[ 18482], 99.50th=[ 43254], 99.90th=[128451], 99.95th=[152044],
 | 99.99th=[308282]

iops 为，

1 2	write: IOPS=9541, BW=1193MiB/s (1251MB/s)(5000MiB/4192msec) iops : min= 56, max= 503, avg=202.64, stdev=75.92, samples=337

当 dm-cache 块大小为 256KB 时，clat 为，

clat (usec): min=205, max=310102, avg=8785.39, stdev=9813.20
 
clat percentiles (usec):
 |  1.00th=[   668],  5.00th=[  1172], 10.00th=[  2278], 20.00th=[  4752],
 | 30.00th=[  5932], 40.00th=[  6849], 50.00th=[  7635], 60.00th=[  8455],
 | 70.00th=[  9503], 80.00th=[ 11207], 90.00th=[ 14222], 95.00th=[ 17695],
 | 99.00th=[ 31065], 99.50th=[ 42730], 99.90th=[149947], 99.95th=[185598],
 | 99.99th=[278922]

iops 为，

1 2	write: IOPS=5323, BW=1331MiB/s (1395MB/s)(5000MiB/3757msec) iops : min= 18, max= 233, avg=119.38, stdev=31.13, samples=278

TODO

Benchmark
实验结果分析