High write latency on Intel Pro 1500 SSD

[SDoni]

Hello!

I recently upgraded my laptop with a brand new Intel Pro 1500 SSD (180 GB). While trying out various measurements using Linux tools - I'm a performance addict - I discovered something interesting.

I've recorded some IO operations (mostly write sync) with latency of about 170 ms ! I couldn't believe it: an SSD 1 OOM slower than an HDD?

Workload: normal working sessions, no stress at all. OS is Ubuntu 14.04 (trim enabled by default via a crontab weekly job), application running were Chromium browser and a Win 7 KVM VM with MS office open. Filling status:

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda5 101G 71G 25G 74% /

none 4,0K 0 4,0K 0% /sys/fs/cgroup

udev 3,8G 4,0K 3,8G 1% /dev

tmpfs 768M 1,3M 767M 1% /run

none 5,0M 0 5,0M 0% /run/lock

none 3,8G 38M 3,8G 1% /run/shm

none 100M 68K 100M 1% /run/user

/dev/sda2 57G 33G 24G 58% /media/ste/Windows7_OS

Here is an example of the latency data, which I recorded and confirmed with multiple tools: Brendan Gregg ftrace scripts (https://github.com/brendangregg/perf-tools brendangregg/perf-tools · GitHub), blktrace/blkparse/btt, iostat -x 1.

Example:

sudo ./iolatency:

>=(ms) .. <(ms) : I/O |Distribution |

0 -> 1 : 0 | |

1 -> 2 : 0 | |

2 -> 4 : 0 | |

4 -> 8 : 1 |# |

8 -> 16 : 1 |# |

16 -> 32 : 0 | |

32 -> 64 : 0 | |

64 -> 128 : 0 | |

128 -> 256 : 32 |# |

sudo ./iosnoop:

Tracing block I/O. Ctrl-C to end.

COMM PID TYPE DEV BLOCK BYTES LATms

jbd2/sda5-166 166 WS 8,0 227094344 53248 177.84

0 WS 8,0 227094448 4096 2.02

kworker/u16:0 20402 W 8,0 178377528 4096 175.48

kworker/u16:0 20402 W 8,0 178377472 4096 175.46

kworker/u16:0 20402 W 8,0 230578432 4096 175.46

kworker/u16:0 20402 W 8,0 178377696 4096 175.50

kworker/u16:0 20402 W 8,0 230431976 4096 176.35

kworker/u16:0 20402 W 8,0 230409184 4096 176.36

kworker/u16:0 20402 W 8,0 121944064 4096 178.21

kworker/u16:0 20402 WM 8,0 121944088 4096 178.24

kworker/u16:0 20402 WM 8,0 121944112 4096 178.24

kworker/u16:0 20402 WM 8,0 159693064 4096 178.79

kworker/u16:0 20402 WM 8,0 205830272 4096 178.81

kworker/u16:0 20402 WM 8,0 226801776 4096 179.03

kworker/u16:0 20402 WM 8,0 285522744 4096 179.04

kworker/u16:0 20402 WM 8,0 289716352 4096 179.30

kworker/u16:0 20402 WM 8,0 289716480 4096 179.32

kworker/u16:0 20402 WM 8,0 289717120 4096 179.56

kworker/u16:0 20402 WM 8,0 289782040 4096 179.58

kworker/u16:0 20402 WM 8,0 302301320 4096 179.59

kworker/u16:0 20402 WM 8,0 306493488 4096 179.59

I have tried the same measurements on a older and cheaper SSD (with the same OS and applications) and did not see such latency peaks.

Am I missing something? What could be the cause of this?

Thank you!

[Jose_H_Intel1]

Hello stef3a, the following information may be helpful:

• https://wiki.freeswitch.org/wiki/SSD_Tuning_for_Linux SSD Tuning for Linux - FreeSWITCH Wiki
• http://unix.stackexchange.com/questions/93791/benchmark-ssd-on-linux-how-to-measure-the-same-things-... performance - Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows - Unix & Linux S…
• http://askubuntu.com/questions/87035/how-to-check-hard-disk-performance How to check hard disk performance - Ask Ubuntu
• http://www.howtogeek.com/62761/how-to-tweak-your-ssd-in-ubuntu-for-better-performance/ How to Tweak Your SSD in Ubuntu for Better Performance

[SDoni]

Hello Joe,

Thanks for the reply.

I have already checked the OS-level knobs, no further tuning seems to be needed on a stock Ubuntu 14.04. I have checked for:

- partition alighment

- TRIM use and actual functionality

- ext4

- noatime on fstab

As regards the benchmarks, I think you missed my point. What I have provided is a trace of IO latency, which is basically a measure of IO latency during real-life workloads. My point was not related to getting low benchmark scores, which doens't seem the case by the way. I have tested the SSD with fio on random read and write workloads on 8 GB datasets, as per Intel procedures, and got results in line with the specs.

Despite this, during actual use (and light load!), the device is showing horrible latency peaks.

Any hint about that?

[SDoni]

Here are some iostat -x 1 statistics:

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

sda 0,00 0,00 1,00 0,00 4,00 0,00 8,00 0,18 176,00 176,00 0,00 176,00 17,60

sda 0,00 0,00 0,00 1,00 0,00 12,00 24,00 0,18 176,00 0,00 176,00 176,00 17,60

sda 0,00 10,00 0,00 33,00 0,00 172,00 10,42 5,79 175,39 0,00 175,39 5,45 18,00

sda 0,00 0,00 0,00 3,00 0,00 16,00 10,67 0,52 173,33 0,00 173,33 58,67 17,60

sda 0,00 0,00 0,00 3,00 0,00 32,00 21,33 0,52 172,00 0,00 172,00 57,33 17,20

sda 0,00 0,00 0,00 2,00 0,00 16,00 16,00 0,34 172,00 0,00 172,00 86,00 17,20

sda 0,00 1,00 0,00 23,00 0,00 108,00 9,39 3,94 171,13 0,00 171,13 7,83 18,00

sda 0,00 0,00 0,00 14,00 0,00 60,00 8,57 2,34 166,86 0,00 166,86 16,00 22,40

sda 0,00 0,00 1,00 22,00 4,00 148,00 13,22 3,68 159,83 0,00 167,09 7,65 17,60

sda 0,00 12,00 0,00 8,00 0,00 236,00 59,00 1,10 138,00 0,00 138,00 27,50 22,00

As you can see, with a very low write IOPS (even 1 per second!), the response time is constantly above 170 ms.

[Jose_H_Intel1]

Please allow us more time to investigate about this.