topic Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2 in Solid State Drives (NAND)

Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

PPete7 — Sat, 04 Jun 2016 09:40:03 GMT

I have just purchased two Supermicro SuperServer model http://www.supermicro.com/products/system/2u/2028/SYS-2028U-TN24R4T_.cfm SYS-2028U-TN24R4T+ systems with motherboard http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRU-i_.cfm X10DRU-i+ and latest BIOS v2.0. Each system has 24 Intel 750 NVMe SSD model http://ark.intel.com/products/86741/Intel-SSD-750-Series-1_2TB-2_5in-PCIe-3_0-20nm-MLC SSDPE2MW012T4X1 disks. The systems have Windows Server 2012 R2 with all updates and https://downloadcenter.intel.com/download/23929/Intel-SSD-Data-Center-Family-for-NVMe-Drivers Intel 750 SSD driver v1.5.0.1002. None of these disks is used for booting the OS – just for data storage.

The problem is that the Intel driver IaNVMe.sys crashes each system on reboot (see below console screenshot). The only way I can boot into Windows now is to go into SAFE MODE and rename the IaNVMe.sys driver. Then perform a normal boot, which then succeeds by using Microsoft NVMe driver.

On a related note, I installed the https://downloadcenter.intel.com/download/25771/-Intel-Rapid-Storage-Technology-Enterprise-NVMe-Intel-RSTe-NVMe-RAID-Driver Intel NVMe SSD RAID driver on just one of these systems. My hope is to use this to create RAID sets from the NVMe disks. It too (IaRNVMe.sys) is crashing the system on reboot. See the second screenshot below.

I have engaged Microsoft and they have collected MEMORY dumps that are being analyzed. We were able to capture the drivers causing the crashes by enabling the https://support.microsoft.com/en-us/kb/244617 Windows Driver Verifier. Microsoft hopes to give me results of the analysis so that I can pass on to Intel if necessary.

Until then, does anyone have any idea why this is happening? Are these known issues? Are there better drivers?

Thanks!

KPA

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

idata — Mon, 06 Jun 2016 18:23:31 GMT

Hello KPA,

As we understand, your SuperMicro SuperServer 2028U-TN24R4T+ is getting BSOD's when it boots after you installed the Intel® SSD Data Center Family for NVMe Drivers 1.5.0.1002. Please check the following items about this situation:

- This server has 24x NVMe Hot-swap 2.5" drive bays. We advise you to check on this with https://www.supermicro.com/support/index.cfm SuperMicro support, since the manual for the http://www.supermicro.com/products/system/2u/2028/SYS-2028U-TN24R4T_.cfm SuperServer 2028U-TN24R4T+ indicates that there may be additional requirements for the server to operate in a fully populated 24x NVME drive configuration.

- It is advised to obtain the latest BIOS, software and drivers (including chipset) for your server from SuperMicro.

- The Intel® SSD 750 Series is a high end (enthusiast) consumer driver, however, it is not recommended/validated for use in Enterprise systems. Furthermore, most server systems have a list of tested drives, or may even require OEM drives to operate correctly. In your case SuperMicro has a list of supported 2.5" NVME drives, and notes about their usage in the following link:

http://www.supermicro.com/support/resources/HDD/display.cfm%3FCategory%3DSS%26sz%3DN.%26ctrl%3D73%26id%3DD25E4F34CBF58DE33A94A5AF7F81F5E4%26NVMe%3D1%26svrid%3D85118 http://www.supermicro.com/support/resources/HDD/display.cfm?Category=SS&sz=N.&ctrl=73&id=D25E4F34CBF58DE33A94A5AF7F81F5E4&NVMe=1&svrid=85118

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

PPete7 — Mon, 06 Jun 2016 23:38:46 GMT

Hi JB,

Thanks for your feedback. Below is my response.

1. Regarding your reference that "the manual for the http://www.supermicro.com/products/system/2u/2028/SYS-2028U-TN24R4T_.cfm SuperServer 2028U-TN24R4T+ indicates that there may be additional requirements for the server to operate in a fully populated 24x NVME drive configuration", the only thing I found in the manual that you referenced is below (see screenshot), explaining that expansion cards are required to add more drives up to 24. Of course this is already the case and the Windows Server 2012 R2 OS can see all 24 NVMe drives. Please point out any other concerns that you see so that I can follow up.

2. Regarding the need for the latest BIOS and drivers (including chipset) from Supermicro, that is already the case:

3. Regarding your last comment stating "The Intel® SSD 750 Series is a high end (enthusiast) consumer driver, however, it is not recommended/validated for use in Enterprise systems", what does this really mean? Intel has provided https://downloadcenter.intel.com/download/23929/Intel-SSD-Data-Center-Family-for-NVMe-Drivers Windows Server 2012 R2 drivers for these NVMe SSDs and that is exactly what I am using these drives for. Whether my use is for enterprise or not seems of little import here as long as I am using the drives and drivers on supported OS. Please let me know if I am wrong about this. Furthermore, I contacted Supermicro regarding supported NVMe SSD drives for these servers, their statement is that they have not tested the Intel 750 NVMe SSD drives but they have other end users who have successfully used them in Supermicro computers.

Hence, the fundamental question is, should the https://downloadcenter.intel.com/download/23929/Intel-SSD-Data-Center-Family-for-NVMe-Drivers Windows Server 2012 R2 Intel drivers be crashing the OS? If the answer is NO, I would expect you would want to look into this and solve the problem for me and others since these are your drivers.

Thanks and I look forward to your feedback.

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

idata — Tue, 07 Jun 2016 00:58:12 GMT

Thanks for the information, the concern regarding compatibility is that since this server uses multiple backplanes, riser board and hot-plug support, we would advise to use SSD's tested by the vendor. Still, we will check further on our end.

In the meantime, please let us know any additional logs or errors you have received from the system. If you prefer to send logs privately, we can use a different contact method.

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

PPete7 — Tue, 07 Jun 2016 01:45:11 GMT

I have MEMORY dumps that I generated for Microsoft Support regarding the failing IaNVMe.sys. Let me know if you want to start analyzing them concurrently with Microsoft and I will send them over. Let me know where to send them if this is what you prefer.

Thanks.

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

idata — Tue, 07 Jun 2016 20:54:39 GMT

Hello KPA,

We are checking with other resources if there are additional requirements, or any compatibility implications that may be causing this condition.Systems supporting this amount or NVMe drives were recently released, so we would like to confirm if the problem is the driver itself, or something not working well in this configuration. For testing, please setup a simple configuration (such as single backplane with 1 drive) and check if the system boots this way or not. If it works, please start adding more SSD's and test again, then try with the additional backplanes and drives, until we can determine at which point this happens.Also, let us know if when you use the Microsoft driver, the system works well or not.Have you tried using the https://downloadcenter.intel.com/download/23931/Intel-SSD-Data-Center-Tool Intel® SSD Data Center Tool to manage the drives?You can actually add logs to the thread using the Advanced editor, ir the logs are too big, or if you prefer to do this privately, let us know and we will use other contact method.JB

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

PPete7 — Wed, 08 Jun 2016 09:03:58 GMT

Hi JB,

I will plan on reducing the number of drives as you suggested to determine if this has any effect on the crash. I will report back to you. Meanwhile, all 24 drives work fine with the Microsoft driver.

Regarding your question about whether or not I have tried the https://downloadcenter.intel.com/download/23931/Intel-SSD-Data-Center-Tool Intel SSD Data Center Tool ... what do you want me to test with this tool?

The memory dump and related files that Microsoft generated for this crash is a 52MB zip file. Let me know how you prefer that I send this file.

Thanks.

KPA

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

idata — Wed, 08 Jun 2016 18:59:33 GMT

Thank you for the update, we will be waiting for results of the test about the amount of drives.

We would like to confirm how the SSD's are detected by ISDCT in this condition, so we can get the output of the command: "isdct show –intelssd" in a text file.

We will let you know soon about reviewing the system dumps.

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

PPete7 — Wed, 08 Jun 2016 22:05:47 GMT

I have the results of the "isdct show –intelssd" command. It has serial numbers of my SSDs so I don't believe it should be posted here. Please let me know how to send the text file to you. Thanks.

KPA

Re: Intel 750 NVMe SSD Driver (IaNVMe.sys) Crashing Windows Server 2012 R2

idata — Thu, 09 Jun 2016 00:59:45 GMT

Hello KPA,

Please check your private messages for additional information about this ticket.