I have done a fair bit of testing and experimenting to maximize KVM VM performance on ZFS. Hopefully someone finds the below useful. Guests/Benchmark used: WinXP Pro + Crystal Disk Mark 1 pass at 1000MG SL 6.4 + Gnome Disk Utility (palimpsest) R/W disk benchmark function Each guest uses a file backed sparse raw disk located on an NFS mounted directory ("/foo"). The NFS mounted directory is interconnected via a dual 10Gbe Twin-ax copper from the targeted storage server disk subsystem and filesystem. The connection is bonded using Linu x bonding (mode=4 miimon=250 xmit_hash_policy=layer2+3). NFS server e xport options were (rw,async,no_root_squash). The NFS client was left with defa ult mount options. The NFS version used was NFSv3 only. NFSv4 was disabled on the storage server via "RPCNFSDARGS = -N 4". *NOTES* ZFS, EXT4, MD+LVM, or BTRFS disk subsystem configuration details are specified in each below test configuration. Low latency Intel SSD models were utilized in all cases wherein SSD are being used. When ext4 is used on top of a zvol with default mount options the journal/metadata is synchronous and therefore offloaded to the ZIL. Jumbo frames were enabled on all 10Gbe interfaces. The following network tuning was applied for to optimize for 10Gbe net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.core.wmem_max = 16777216 net.core.rmem_max = 16777216 net.core.wmem_default = 65536 net.core.rmem_default = 87380 net.core.netdev_max_backlog = 30000 WinXP qemu-kvm commandline: /usr/libexec/qemu-kvm -name winxpfoo -M rhel6.4.0 -cpu qemu64,-svm -enable-kvm -m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig -nodefaults -drive file=/foo/winxp.img,if=none,id=drive-virtio-disk0,format=raw,cach e=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-dis k0,id=virtio-disk0 -spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qx l -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 & SL 6.4 qemu-kvm commandline: /usr/libexec/qemu-kvm -name sl64 -M rhel6.4.0 -cpu qemu64,-svm -enable-kv m -m 3072 -smp 2,sockets=1,cores=2,threads=1 -nodefconfig -nodefaults -drive file=/vmstore/foo/SL64hd0.img,if=none,id=drive-virtio-disk0,format= raw,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-dis k0,id=virtio-disk0,bootindex=2 -drive file=/vmstore/foo/SL64hd1.img,if=none,id=drive-virtio-disk1,format= raw,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-dis k1,id=virtio-disk1 -spice port=5900,addr=3.57.109.210,disable-ticketing -k en-us -vga qx l -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 & -- Test Config #1 -- Storage Server: Sunfire x4140 RAM: 32GB CPU: 2x AMD Opteron 2439SE HBA: LSI 3801 Disks: 8 146GB 10K 2.5" SAS Disk Config: LSI HW IME Raid + Linux LVM Filesystem: LVM+ext4 Network: Intel x520DA2 dual port 10Gbe OS: SL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Filesystem: LSI HW Raid+ ext4 Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Benchmark Results (MB/s): **** WinXP **** seq: 295.4 189.7 512k: 298.8 188.9 4K: 14.75 12.99 4K QD32: 16.16 14.42 **** SL 6.4 (MB/s) **** Min Read: 654.3 Max Read: 704.8 Avg Read: 682.5 Min Write: 233.6 Max Write: 551.1 Avg Write: 465.8 AVG Access: 0.5ms -- End Test Config #1 -- -- Test Config #2 -- Storage Server: Sunfire x4270m2 RAM: 72GB CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 24 300GB 10K 2.5" SAS Disks: 4 128GB Intel SATA SSD Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC Filesystem: ZFS ZVOL+ext4 Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror mirror ... log cache zfs create -s -V 100G -o volblocksize=64K das0/foo mkfs.ext4 -L foo /dev/das0/foo mount /dev/das0/foo /zvol/foo -o noatime **** XP (MB/s) **** seq: 315.4 273.4 512k: 302.4 263.0 4K: 12.94 8.688 4K QD32: 14.45 13.13 **** SL 6.4 (MB/s)**** Min Read: 630.0 Max Read: 698.2 Avg Read: 662.8 Min Write: 357.6 Max Write: 1.1 GB/s Avg Write: 895.3 AVG Access: 0.5ms -- End Test Config #2 -- -- Test Config #3 -- Storage Server: Sunfire x4270m2 RAM: 72GB CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 24 300GB 10K 2.5" SAS Disks: 4 128GB Intel SATA SSD Disk Config: ZFS Raid10 + Raid 0 ZIL + Raid 0 L2ARC Filesystem: ZFS Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror mirror ... log cache zfs create das0/foo **** XP (MB/s) **** seq: 111.4 57.36 512k: 95.64 54.33 4K: 11.80 2.878 4K QD32: 11.31 3.095 -- End Test Config #3 -- -- Test Config #4 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: ZFS Raid10 + ZIL + L2ARC Filesystem: ZFS ZVOL+ext4 Network: Intel x520DA2 dual port 10Gbe OS: Fedora 18 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror mirror log cache < ssd> zfs create -s -V 100G -o volblocksize=64K das0/foo mkfs.ext4 -L foo /dev/das0/foo mount /dev/das0/foo /zvol/foo -o noatime **** XP (MB/s) **** seq: 196.5 200.2 512k: 195.2 188.8 4K: 9.665 7.043 4K QD32: 12.02 8.140 **** SL 6.4 (MB/s) **** Min Read: 423.7 Max Read: 575.3 Avg Read: 526.7 Min Write: 44.4 Max Write: 668.8 Avg Write: 540.5 AVG Access: 1.4ms -- End Test Config #4 -- -- Test Config #5 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: ZFS Raid10 + ZIL + L2ARC Filesystem: ZFS Network: Intel x520DA2 dual port 10Gbe OS: Fedora 18 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ZFS Configuration Details/Commands zpool create das0 mirror mirror log cache < ssd> zfs create das0/foo **** XP (MB/s) **** seq: 152.5 126.2 512k: 147.4 121.8 4K: 9.189 7.481 4K QD32: 11.90 8.003 **** SL 6.4 (MB/s) **** Min Read: 409.3 Max Read: 564.4 Avg Read: 511.6 Min Write: 237.1 Max Write: 658.0 Avg Write: 555.3 AVG Access: 2.9ms -- End Test Config #5 -- -- Test Config #6 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: Linux MD Raid10 + LVM + ext4 Filesystem: ext4 Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/ext4 Configuration Details/Commands 4 Disk SSD Linux MD Raid 10 Stripe Size 1MB 100GB LV(foo) from VG0 via PV MD device (ext4 data ) 2 Disk SSD Linux MD Raid 0 Stripe Size 1MB 1GB LV(journaldev) from VG1 via PV MD device (ext4 journal/metadata) ext4 filesystem lv-foo, external journal on lv-journaldev (external journal_data_ordered) mounted at /vg0/foo (mount options rw,noatime,journal_checksum,journal_async_commit) **** XP (MB/s) **** seq: 251.3 203.9 512k: 249.0 217.7 4K: 10.12 9.012 4K QD32: 13.18 11.54 **** SL 6.4 (MB/s) **** Min Read: 523.3 Max Read: 663.0 Avg Read: 606.4 Min Write: 239.9 Max Write: 656.0 Avg Write: 563.8 AVG Access: 0.8ms -- End Test Config #6 -- -- Test Config #7 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: BTRFS Raid10 data + raid0 metadata Filesystem: BTRFS Network: Intel x520DA2 dual port 10Gbe OS: Fedora 18 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/BTRFS Configuration Details/Commands mkfs.btrfs -L foo -m raid0 -d raid10 btrfs subvolume create /das0/foo **** XP (MB/s) **** seq: 195.4 180.0 512k: 180.8 174.5 4K: 9.898 7.183 4K QD32: 11.87 8.248 **** SL 6.4 (MB/s) **** Min Read: 465.4 Max Read: 554.6 Avg Read: 514.1 Min Write: 450.5 Max Write: 670.3 Avg Write: 525.3 AVG Access: 0.8ms -- End Test Config #7-- -- Test Config #8 -- Storage Server: Sunfire x4170 RAM: 72GB CPU:2x Intel E5540 HBA: LSI 92118i Disks: 4 200GB Intel SATA SSD Disks: 2 100 GB Intel SATA SSD Disk Config: BTRFS Raid10 data + raid0 metadata Filesystem: BTRFS Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 KVM/NFS Client: Sunfire X4170m2 RAM: 72GB RAM CPU: 2x Intel E5540 HBA: LSI 92118i Disks: 2 146GB 10K 2.5" SAS Disk Config: LSI HW IS Raid Network: Intel x520DA2 dual port 10Gbe OS: OEL 6.4 Filesystem/BTRFS Configuration Details/Commands mkfs.btrfs -L foo -m raid0 -d raid10 btrfs subvolume create /das0/foo **** XP (MB/s) **** seq: 232.1 123.4 512k: 243.0 213.6 4K: 9.393 9.170 4K QD32: 12.04 10.74 **** SL 6.4 (MB/s) **** Min Read: 508.0 Max Read: 652.4 Avg Read: 594.4 Min Write: 409.6 Max Write: 670.4 Avg Write: 525.8 AVG Access: 0.5 -- End Test Config #8 --