Intel® SSD DC P3700 Series I/O 0 QID 0 timeout, reset controller

[SWarm]

We have a few servers with nvme drives that are currently experiencing controller resets during I/O.

Affected drives:

Node Model Version Namepace Usage Format FW Rev

---------------- -------------------- -------- -------- -------------------------- ---------------- --------

/dev/nvme0n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme1n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme2n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme3n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme4n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme5n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme6n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme7n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme8n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

/dev/nvme9n1 INTEL SSDPE2MD800G4 1.0 1 800.17 GB / 800.17 GB 512 B + 0 B 8DV10131

They run on Debian Wheezy 7.11 with a 3.16 kernel.

some logging:

[Tue Sep 5 11:28:48 2017] nvme 0000:05:00.0: I/O 0 QID 0 timeout, reset controller

[Tue Sep 5 11:28:50 2017] nvme 0000:05:00.0: Device shutdown incomplete; abort shutdown

[Tue Sep 5 11:28:50 2017] nvme 0000:05:00.0: Cancelling I/O 0 QID 0

[Tue Sep 5 11:28:50 2017] nvme 0000:05:00.0: Cancelling I/O 1 QID 0

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 239 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 240 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 241 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 242 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 243 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 244 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 245 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 246 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 247 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 248 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 249 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 250 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 251 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 252 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 253 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 254 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 255 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 256 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 257 for MSI/MSI-X

[Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 258 for MSI/MSI-X

[Sat Sep 9 18:23:21 2017] nvme 0000:03:00.0: I/O 0 QID 0 timeout, reset controller

[Sat Sep 9 18:23:23 2017] nvme 0000:03:00.0: Device shutdown incomplete; abort shutdown

[Sat Sep 9 18:23:23 2017] nvme 0000:03:00.0: Cancelling I/O 0 QID 0

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 219 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 220 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 221 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 222 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 223 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 224 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 225 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 226 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 227 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 228 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 229 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 230 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 231 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 232 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 233 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 234 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 235 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 236 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 237 for MSI/MSI-X

[Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 238 for MSI/MSI-X

[Sun Sep 10 15:07:25 2017] nvme 0000:83:00.0: I/O 0 QID 0 timeout, reset controller

[Sun Sep 10 15:07:28 2017] nvme 0000:83:00.0: Device shutdown incomplete; abort shutdown

[Sun Sep 10 15:07:28 2017] nvme 0000:83:00.0: Cancelling I/O 0 QID 0

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 319 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 320 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 321 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 322 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 323 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 324 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 325 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 326 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 327 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 328 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 329 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 330 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 331 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 332 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 333 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 334 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 335 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 336 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 337 for MSI/MSI-X

[Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 338 for MSI/MSI-X

[Tue Sep 12 15:06:34 2017] nvme 0000:03:00.0: I/O 0 QID 0 timeout, reset controller

[Tue Sep 12 15:06:36 2017] nvme 0000:03:00.0: Device shutdown incomplete; abort shutdown

[Tue Sep 12 15:06:36 2017] nvme 0000:03:00.0: Cancelling I/O 0 QID 0

[Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 219 for MSI/MSI-X

[Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 220 for MSI/MSI-X

[Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 221 for MSI/MSI-X

[Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 222 for MSI/MSI-X

[Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 223 for MSI/MSI-X

[Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 224 for MSI/MSI-X

[Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 225 for MSI/MSI-X

[Tue Sep 1...

[idata]

Hi p0nts,

We understand your situation regarding your Intel® SSDs DC P3700 Series.It seems that you are running an old firmware version.We recommend you to update the firmware of the SSDs by using the https://downloadcenter.intel.com/download/27144/Intel-SSD-Data-Center-Tool?product=87278 Intel® SSD Data Center Tool.You need to run the following commands as with root privileges: 1. isdct show -intelssd : to find out the index of your drives. It may be different as the tool only detects Intel(R) SSDs. 2. isdct load intelssd 1(replace one with the index shown). Please let us know if that works. If you run into any issues again provide us with new system logs. Regards, Junior M.

[SWarm]

Hi Junior,

Thank you for your answer, I am currently upgrading the drives so I will let you know about the results

[idata]

Hi p0nts,

Thanks for the update, we would be waiting for the results. Regards, Junior M.

[idata]

Hi p0nts,

We haven't heard back from you in a while. We would like to know if there is something else we can assist you with.Regards,Junior M.