Firmware Downgrade

[AWill17]

Does anyone know if it is possible to downgrade the firmware on the P3700? We are struggling with a VMware VSAN implementation, and support has told us that the drives need to be running firmware 8DV10131, and the current firmware 8DV10171 is causing problems. Our vendor has attempted to get an answer directly from Intel with no luck.

Thanks,

Andy

[DChen55]

I have a mix of VSAN 6.1/6.2 hybrid and all flash clusters, all using P3700 or P3600 for cache. I can tell you that in VSAN 6.2, using the HCL firmware/driver combo you would see very poor performance, congestion and latency problems. I recently built a new 6.2 VSAN all flash cluster that happened to ship with 8DV10131 (HCL) firmware. Using 1.0e.1.1-1OEM.550.0.0.1391871 HCL driver, there are severe write performance related issues. Result is the same after upgrading to firmware 8DV10171. Between VMWare and Intel or whoever is responsible for updating the HCL, I don't think any real testing went into it before it got 6.2 qualified. Testing the HCL combo even for 5 minutes, one would immediately notice a major latency/congestion issue, on even light stress testing. I believe it also has something to do with the new checksum functionality added in 6.2, if disabled in the storage profile, all performance returns back to normal levels.

Personally I don't think it's an issue of downgrading firmware but for Intel to release a new inte-nvme driver (and or firmware update) that resolves issues discovered for version for 6.2. Also none of the issues exist on 6.1 (probably because checksum feature isn't on 6.1).

Here are my findings from 6.2 AF VSAN using Intel P3700 400GB for write cache and 4x Intel S3510 800GB for capacity:

P3700 400GB, firmware 8DV10131, intel-nvme 1.0e.1.1-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

P3700 400GB, firmware 8DV10131, intel-nvme 1.0e.2.0-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

P3700 400GB, firmware 8DV10171, intel-nvme 1.0e.1.1-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

P3700 400GB, firmware 8DV10171, intel-nvme 1.0e.2.0-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

P3700 400GB, firmware 8DV10171, nvme 1.2.0.27-4vmw.550.0.0.1331820 driver - no latency/congestion problems, sequential writes limited to 250MB/s, no issues if disabling checksum

[idata]

elerium,

Thanks for this information. We will make sure to verify with our team and send this findings to the proper department.As soon as we have any news or updates we will be posting it here.

[idata]

Elerium & Emulsifide,

We have found some other users experiencing a similar issue reported in this https://communities.vmware.com/thread/533769?start=0&tstart=0 thread by elerium, we are checking from our side, but it has been the first time we have made aware of it. We will share any findings as soon as we have any updates.NC

[idata]

elerium,

We would like to inform you that one of the big advantages of our Datacenter drives is the high lasting data integrity.

Enabling the software Checksum feature available in VSAN 6.2 will add additional latency which results in an overall performance reduction, therefore due to our drive's high reliability towards preventing data corruption, it may not be necessary to enable this feature in VSAN unless the application requires so.

NC

[idata]

elerium,

We haven't heard from you in a while. We would like to know if you still need our assistance.