topic Re: High write latency on Intel Pro 1500 SSD in Solid State Drives (NAND)

High write latency on Intel Pro 1500 SSD

SDoni — Mon, 15 Sep 2014 10:52:20 GMT

Hello!

I recently upgraded my laptop with a brand new Intel Pro 1500 SSD (180 GB). While trying out various measurements using Linux tools - I'm a performance addict - I discovered something interesting.

I've recorded some IO operations (mostly write sync) with latency of about 170 ms ! I couldn't believe it: an SSD 1 OOM slower than an HDD?

Workload: normal working sessions, no stress at all. OS is Ubuntu 14.04 (trim enabled by default via a crontab weekly job), application running were Chromium browser and a Win 7 KVM VM with MS office open. Filling status:

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda5 101G 71G 25G 74% /

none 4,0K 0 4,0K 0% /sys/fs/cgroup

udev 3,8G 4,0K 3,8G 1% /dev

tmpfs 768M 1,3M 767M 1% /run

none 5,0M 0 5,0M 0% /run/lock

none 3,8G 38M 3,8G 1% /run/shm

none 100M 68K 100M 1% /run/user

/dev/sda2 57G 33G 24G 58% /media/ste/Windows7_OS

Here is an example of the latency data, which I recorded and confirmed with multiple tools: Brendan Gregg ftrace scripts (https://github.com/brendangregg/perf-tools brendangregg/perf-tools · GitHub), blktrace/blkparse/btt, iostat -x 1.

Example:

sudo ./iolatency:

>=(ms) .. <(ms) : I/O |Distribution |

0 -> 1 : 0 | |

1 -> 2 : 0 | |

2 -> 4 : 0 | |

4 -> 8 : 1 |# |

8 -> 16 : 1 |# |

16 -> 32 : 0 | |

32 -> 64 : 0 | |

64 -> 128 : 0 | |

128 -> 256 : 32 |# |

sudo ./iosnoop:

Tracing block I/O. Ctrl-C to end.

COMM PID TYPE DEV BLOCK BYTES LATms

jbd2/sda5-166 166 WS 8,0 227094344 53248 177.84

0 WS 8,0 227094448 4096 2.02

kworker/u16:0 20402 W 8,0 178377528 4096 175.48

kworker/u16:0 20402 W 8,0 178377472 4096 175.46

kworker/u16:0 20402 W 8,0 230578432 4096 175.46

kworker/u16:0 20402 W 8,0 178377696 4096 175.50

kworker/u16:0 20402 W 8,0 230431976 4096 176.35

kworker/u16:0 20402 W 8,0 230409184 4096 176.36

kworker/u16:0 20402 W 8,0 121944064 4096 178.21

kworker/u16:0 20402 WM 8,0 121944088 4096 178.24

kworker/u16:0 20402 WM 8,0 121944112 4096 178.24

kworker/u16:0 20402 WM 8,0 159693064 4096 178.79

kworker/u16:0 20402 WM 8,0 205830272 4096 178.81

kworker/u16:0 20402 WM 8,0 226801776 4096 179.03

kworker/u16:0 20402 WM 8,0 285522744 4096 179.04

kworker/u16:0 20402 WM 8,0 289716352 4096 179.30

kworker/u16:0 20402 WM 8,0 289716480 4096 179.32

kworker/u16:0 20402 WM 8,0 289717120 4096 179.56

kworker/u16:0 20402 WM 8,0 289782040 4096 179.58

kworker/u16:0 20402 WM 8,0 302301320 4096 179.59

kworker/u16:0 20402 WM 8,0 306493488 4096 179.59

I have tried the same measurements on a older and cheaper SSD (with the same OS and applications) and did not see such latency peaks.

Am I missing something? What could be the cause of this?

Thank you!

Re: High write latency on Intel Pro 1500 SSD

Jose_H_Intel1 — Tue, 16 Sep 2014 16:47:31 GMT

Hello stef3a, the following information may be helpful:

https://wiki.freeswitch.org/wiki/SSD_Tuning_for_Linux SSD Tuning for Linux - FreeSWITCH Wiki
http://unix.stackexchange.com/questions/93791/benchmark-ssd-on-linux-how-to-measure-the-same-things-as-crystaldiskmark-does-i performance - Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows - Unix & Linux S…
http://askubuntu.com/questions/87035/how-to-check-hard-disk-performance How to check hard disk performance - Ask Ubuntu
http://www.howtogeek.com/62761/how-to-tweak-your-ssd-in-ubuntu-for-better-performance/ How to Tweak Your SSD in Ubuntu for Better Performance

Re: High write latency on Intel Pro 1500 SSD

SDoni — Thu, 18 Sep 2014 13:53:54 GMT

Hello Joe,

Thanks for the reply.

I have already checked the OS-level knobs, no further tuning seems to be needed on a stock Ubuntu 14.04. I have checked for:

- partition alighment

- TRIM use and actual functionality

- ext4

- noatime on fstab

As regards the benchmarks, I think you missed my point. What I have provided is a trace of IO latency, which is basically a measure of IO latency during real-life workloads. My point was not related to getting low benchmark scores, which doens't seem the case by the way. I have tested the SSD with fio on random read and write workloads on 8 GB datasets, as per Intel procedures, and got results in line with the specs.

Despite this, during actual use (and light load!), the device is showing horrible latency peaks.

Any hint about that?

Re: High write latency on Intel Pro 1500 SSD

SDoni — Thu, 18 Sep 2014 14:28:03 GMT

Here are some iostat -x 1 statistics:

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

sda 0,00 0,00 1,00 0,00 4,00 0,00 8,00 0,18 176,00 176,00 0,00 176,00 17,60

sda 0,00 0,00 0,00 1,00 0,00 12,00 24,00 0,18 176,00 0,00 176,00 176,00 17,60

sda 0,00 10,00 0,00 33,00 0,00 172,00 10,42 5,79 175,39 0,00 175,39 5,45 18,00

sda 0,00 0,00 0,00 3,00 0,00 16,00 10,67 0,52 173,33 0,00 173,33 58,67 17,60

sda 0,00 0,00 0,00 3,00 0,00 32,00 21,33 0,52 172,00 0,00 172,00 57,33 17,20

sda 0,00 0,00 0,00 2,00 0,00 16,00 16,00 0,34 172,00 0,00 172,00 86,00 17,20

sda 0,00 1,00 0,00 23,00 0,00 108,00 9,39 3,94 171,13 0,00 171,13 7,83 18,00

sda 0,00 0,00 0,00 14,00 0,00 60,00 8,57 2,34 166,86 0,00 166,86 16,00 22,40

sda 0,00 0,00 1,00 22,00 4,00 148,00 13,22 3,68 159,83 0,00 167,09 7,65 17,60

sda 0,00 12,00 0,00 8,00 0,00 236,00 59,00 1,10 138,00 0,00 138,00 27,50 22,00

As you can see, with a very low write IOPS (even 1 per second!), the response time is constantly above 170 ms.

Re: High write latency on Intel Pro 1500 SSD

Jose_H_Intel1 — Fri, 19 Sep 2014 22:21:33 GMT

Please allow us more time to investigate about this.

Re: High write latency on Intel Pro 1500 SSD

SDoni — Thu, 09 Oct 2014 07:31:48 GMT

Hello Joe,

Any update on this one?

Re: High write latency on Intel Pro 1500 SSD

Jose_H_Intel1 — Thu, 09 Oct 2014 19:33:25 GMT

What results do you get with no programs running, including the Windows* 7 VM?

Re: High write latency on Intel Pro 1500 SSD

SDoni — Tue, 28 Oct 2014 08:26:53 GMT

Hello Joe,

Same results even with no VM,

Re: High write latency on Intel Pro 1500 SSD

Jose_H_Intel1 — Thu, 30 Oct 2014 17:33:06 GMT

Please allow us more time to investigate.

Re: High write latency on Intel Pro 1500 SSD

SDoni — Fri, 06 Mar 2015 12:38:17 GMT

Hi Joe,

Any news on this one?

Re: High write latency on Intel Pro 1500 SSD

ASouz7 — Fri, 06 Mar 2015 19:20:44 GMT

Hello Stef3a,

We are checking with our engineers and as soon as we have an answer we will let you know.

Re: High write latency on Intel Pro 1500 SSD

ASouz7 — Mon, 23 Mar 2015 20:29:56 GMT

Hello stef3a,

We have tested the same drive and did not found the sames latencies. In this particular case we recommend that you contact your nearest Intel Contact Center at your earliest convenience for extra support and other options. Here is the link: http://www.intel.com/p/en_US/support/contactsupport Contact Support - In you are located in The United States, you may call this number: 916 377 7000 Mon to Fri from 7:00 A.M to 5:00 P.M. (PST)

Re: High write latency on Intel Pro 1500 SSD

KPodl — Thu, 09 Apr 2015 14:07:27 GMT

This is still not solved , I did manage to get in W540 new PRO 1500 180GB, and until I secure erased it I had write level as low as 2MB/s ... after secure erasing, I did formatted it to only 135GB (25% OP) and now all is back as it should be, qestion is , FW is LSTi (CVDA407600YX1802GN) - so I know that this is SF-2281, and it has his "ups-and-downs" (not so PRO choice as a controller IMHO) but .. does it have new FW? Will it handle as a drive in USB case (how good GC process is?) ?

Re: High write latency on Intel Pro 1500 SSD

ASouz7 — Thu, 09 Apr 2015 20:03:02 GMT

Hello kpodlaski,

The drive will work just fine connected via USB, only the performance will be impacted by the USB connection. As for the firmware, the LSTi is the latest version for the time being.

Re: High write latency on Intel Pro 1500 SSD

KPodl — Fri, 10 Apr 2015 06:55:16 GMT

That's great to know, I already see limitations of cheap USB3.0 enclosures - so that is what I'm aware. More what I'm curious is how well Garbage Collection will help (having +25% space available). Also ... is it worth having Thunderbolt case (as I've learned lately, Thunderbolt should show drive as SATA device , and therefore allow to pass Trim command to the drive from Intel SSD Tool box) was it tested?

Re: High write latency on Intel Pro 1500 SSD

ASouz7 — Fri, 10 Apr 2015 20:37:55 GMT

Hello kpodlaski,

We can't guarantee it as we have not tested these devices. In theory it should work. If you are willing to try, please know that there is no guarantee that work.