topic Re: Intel 750 not showing delete support under ESXi? in Solid State Drives (NAND)

Intel 750 not showing delete support under ESXi?

shopk — Wed, 28 Oct 2015 11:33:30 GMT

Hi everyone, I've hit an unexpected issue with some new Intel 750's (1.2TB) and ESXi 6 (latest update). I have installed the latest driver as recommended (to improve the performance of the drives) and while they are working I have noticed that ESXi isn't attempting to reclaim any free space (and forcing it doesn't work). A check of the VAAI values of each drive showed the following:

VAAI Plugin Name:

ATS Status: unsupported

Clone Status: unsupported

Zero Status: unsupported

Delete Status: unsupported

While I expected ATS/Clone/Zero to be unsupported, I didn't expect to see Delete as unsupported. Using a standard SSD through an AHCI controller will show Delete as supported (as TRIM seems to get converted to DISCARD/UNMAP).

Has anyone else seen this? I'm concerned that as a result my performance is going to degrade over time as there will be a lack of proper unmapping of free space.

Re: Intel 750 not showing delete support under ESXi?

jbenavides — Thu, 29 Oct 2015 01:00:29 GMT

Hello shopkins,

We will need some additional information to check about the "Delete Status: Unsupported". Please let us know the following:

- Computer brand and model

- BIOS version- SSD model number- SSD firmware version- Driver version used

- Are you using an Add in card, or a 2.5 inch form factor SSD?

- What type of PCIe slot are you using? (PCIe generation and number of lanes)

Re: Intel 750 not showing delete support under ESXi?

shopk — Thu, 29 Oct 2015 10:39:46 GMT

Hi Jonathon,

Additional information below (I've put quite a bit). I should add that all four of the drives are working and are currently being used as datastores. The concern is that the UNMAP support doesn't appear to be recognised and that as a result there will be a performance degredation. Looking at the nvme driver on SourceForge it does appear to support UNMAP, hence my confusion around why shows Delete as unsupported.

- Supermicro X9DRW-7TPF+

- BIOS 3.2 (latest available)

- Intel 750 1.2TB AIC (PCI Gen-3 x4, not the 2.5" form-factor version)

- Unknown firmware version (would be what was shipped with the card, didnt see a firmware upgrade available a few days ago)

- VMW-ESX-5.5.0-intel-nvme-1.0e.1.1-2791094.zip (exact driver from ESXi below)

- PCIe slots are x8 but have been clocked down to x4 (they work in either mode)

> Module info

[root@esxi:~] vmkload_mod -s intel-nvme

vmkload_mod module information

input file: /usr/lib/vmware/vmkmod/intel-nvme

Version: 1.0e.1.1-1OEM.550.0.0.1391871

License: BSD

Required name-spaces:

com.vmware.vmkapi# v2_2_0_0

Parameters:

> Storage Path List (for one device, I have four)

[root@esxi:~] esxcli storage core path list

pscsi.vmhba4-pscsi.0:0-naa.65fdfe40353136333030305731xxxxx

UID: pscsi.vmhba4-pscsi.0:0-naa.65fdfe40353136333030305731xxxxx

Runtime Name: vmhba4:C0:T0:L0

Device: naa.65fdfe40353136333030305731xxxxx

Device Display Name: Local NVMe Disk (naa.65fdfe40353136333030305731xxxxx)

Adapter: vmhba4

Channel: 0

Target: 0

LUN: 0

Plugin: NMP

State: active

Transport: parallel

Adapter Identifier: pscsi.vmhba4

Target Identifier: pscsi.0:0

Adapter Transport Details: Unavailable or path is unclaimed

Target Transport Details: Unavailable or path is unclaimed

Maximum IO Size: 131072

> Device VAAI Status

[root@esxi:~] esxcli storage core device vaai status get -d naa.65fdfe40353136333030305731xxxxxx

naa.65fdfe40353136333030305731xxxxxx

VAAI Plugin Name:

ATS Status: unsupported

Clone Status: unsupported

Zero Status: unsupported

Delete Status: unsupported

> Storage Adapter List

[root@esxi:~] esxcli storage core adapter list

HBA Name Driver Link State UID Capabilities Description

-------- ------------ ---------- -------------- ------------ ------------------------------------------------------------------------------

vmhba0 megaraid_sas link-n/a unknown.vmhba0 (0000:0b:00.0) LSI / Symbios Logic MegaRAID SAS Fusion Controller

vmhba2 rste link-n/a pscsi.vmhba2 (0000:02:00.0) Intel Corporation Patsburg 4-Port SATA/SAS Storage Control Unit

vmhba3 intel-nvme link-n/a pscsi.vmhba3 (0000:81:00.0) Intel Corporation NVM-Express PCIe SSD

vmhba4 intel-nvme link-n/a pscsi.vmhba4 (0000:03:00.0) Intel Corporation NVM-Express PCIe SSD

vmhba5 intel-nvme link-n/a pscsi.vmhba5 (0000:04:00.0) Intel Corporation NVM-Express PCIe SSD

vmhba6 intel-nvme link-n/a pscsi.vmhba6 (0000:82:00.0) Intel Corporation NVM-Express PCIe SSD

vmhba32 usb-storage link-n/a usb.vmhba32 () USB

Re: Intel 750 not showing delete support under ESXi?

shopk — Thu, 29 Oct 2015 10:47:47 GMT

On the subject of firmware, can someone comment on what the approach is for upgrading the firmware on systems that aren't running Windows... According to http://www.intel.com/support/ssdc/hpssd/sb/CS-035687.htm Intel® High Performance Solid-State Drive — Firmware Versions for Intel® Solid State Drives the firmware update tool doesn't support the 750's, which makes things a little tricky.

Re: Intel 750 not showing delete support under ESXi?

shopk — Thu, 29 Oct 2015 15:02:03 GMT

Quick update, firmware has been updated on all 4 drives however ESXi still shows the Delete capability as unsupported.

Before

[root@localhost ~]# isdct show -intelssd

- IntelSSD CVCQ5163000Wxxxxxx -

DeviceStatus: Healthy

Firmware: 8EV10135

FirmwareUpdateAvailable: Firmware=8EV10171 Bootloader=8B1B0131

ModelNumber: INTEL SSDPEDMW012T4

ProductFamily: Intel SSD 750 Series

SerialNumber: CVCQ5163000Wxxxxxx

Index: 0

DevicePath: /dev/nvme0n1

Bootloader: 8B1B012E

After

[root@localhost ~]# isdct show -intelssd

- IntelSSD CVCQ5163000Wxxxxxx -

DeviceStatus: Healthy

Firmware: 8EV10171

FirmwareUpdateAvailable: The selected Intel SSD contains current firmware as of this tool release.

ModelNumber: INTEL SSDPEDMW012T4

ProductFamily: Intel SSD 750 Series

SerialNumber: CVCQ5163000Wxxxxxx

Index: 0

DevicePath: /dev/nvme0n1

Bootloader: 8B1B0131

Re: Intel 750 not showing delete support under ESXi?

jbenavides — Thu, 29 Oct 2015 15:43:15 GMT

Hello shopkins,

Thank you for all the information you have provided. After reviewing it, the main recommendation I have at this time is to use the Intel-NVMe* driver 1.0e.2.0, this can be downloaded from VMware website at: https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI55-INTEL-INTEL-NVME-10E20&productId=396# product_downloads Download VMware Virtual SAN

Please let us know if this helps, or if the issue persists so we can consider further options.

Re: Intel 750 not showing delete support under ESXi?

shopk — Thu, 29 Oct 2015 17:02:17 GMT

Jonathon,

Driver has been upgraded and the system has been rebooted. Sadly the VAAI support still shows that Delete isnt supported:

[root@esxi:~] vmkload_mod -s intel-nvme

vmkload_mod module information

input file: /usr/lib/vmware/vmkmod/intel-nvme

Version: 1.0e.2.0-1OEM.550.0.0.1391871

License: BSD

Required name-spaces:

com.vmware.vmkapi# v2_2_0_0

Parameters:

[root@esxi:~] esxcli storage core device vaai status get

t10.NVMe____INTEL_SSDPEDMW012T4_CVCQ5163000Wxxxxxx__00000001

VAAI Plugin Name:

ATS Status: unsupported

Clone Status: unsupported

Zero Status: unsupported

Delete Status: unsupported

As a comparison, a Samsung EVO 840 250GB shows the following:

[root@testesxi:~] esxcli storage core device vaai status get

t10.ATA_____Samsung_SSD_840_EVO_250GB________________S1DDNEADxxxxxxF______

VAAI Plugin Name:

ATS Status: unsupported

Clone Status: unsupported

Zero Status: supported

Delete Status: supported

With the above SSD, both Zero and Delete are supported which helps the long-term performance of the drive.

Re: Intel 750 not showing delete support under ESXi?

shopk — Fri, 30 Oct 2015 23:02:23 GMT

I've found the following page (http://omnia.oeey.com/VMware/vSphere/NVMe VMware/vSphere/NVMe - Omnia) which shows something interesting in comparison to my results... Their output of esxcfg-scsidevs -l shows that the VAAI status is unknown, which in ESXi terms means its not got full VAAI compatibility but does have at least one feature. The difference in their results (aside from using a P3600) appears to be that their driver is 'nvme' rather than 'intel-nvme', so I am assuming they are using the OOTB ESXi version.

Can someone from Intel confirm if the external intel-nvme driver should have the VAAI capability, or if the only way to get it is to revert back to the OOTB driver and suffer the performance drop.

Re: Intel 750 not showing delete support under ESXi?

jbenavides — Fri, 30 Oct 2015 23:26:54 GMT

Hello shopkins,

We will engage additional resources to review this issue, please allow us some time to perform additional research and we will provide any new information as soon as it is available.

Re: Intel 750 not showing delete support under ESXi?

ASouz7 — Fri, 20 Nov 2015 19:58:37 GMT

Hello shopkins,

Thank you for your time. Our engineering department tested with both Intel® SSD 750 Series and Intel® SSD DC P3600 Series and the outcome was the following:

For the Intel® SSD 750 Series, it is not on the VMWare* hadware compatibility list and therefore, it will have less endurance that the Intel® SSD DC P3600.

For the Intel® SSD DC P3600, delete status is supported under ESXi 6, but not on 5.5 so our recommendation is to update to ESXi 6.

Overprovioning may reduce the need for TRIM.

Re: Intel 750 not showing delete support under ESXi?

shopk — Fri, 20 Nov 2015 20:03:46 GMT

Hi aleki, I appreciate that the 750 series are not on the VMware HCL, that said, they both use the same NVMe driver and both device models do advertise TRIM capability. From an ESXi perspective I am running ESXi 6 (not 5.5). The 5.5 is from the driver version from your website (as the 5.5 driver is used for 6.0 as well).

Is TRIM capability on the 750-series being deliberately disabled by your driver? (when running under ESXi 6)

Re: Intel 750 not showing delete support under ESXi?

jbenavides — Mon, 23 Nov 2015 19:40:23 GMT

Hello shopkins,

Please take into consideration that the Intel® SSD 750 series is a high performance consumer drive, recommended for use in client and workstations storage. The 750 is not certified for use with ESXi and it is not listed in the VMware Hardware Compatibility List.

The Intel® SSD 750 Series have many things in common with the Intel® SSD Data Center Family drives, so the same driver may recognize it and make it operational in your system; however, it does not have the same hardware and capabilities that the Data Center drives have. Since this is not a recommended configuration, we cannot guarantee that all features will be available for it in ESXi, hence, TRIM is not supported for the Intel® SSD 750 Series in VMware ESXi.

Re: Intel 750 not showing delete support under ESXi?

shopk — Mon, 23 Nov 2015 19:48:17 GMT

Jonathon,

I appreciate that the 750 isn't on the VMware HCL, however that doesn't really answer the question as to if it is VMware (specifically ESXi) that is disabling TRIM support for the 750-series or the updated driver provided by Intel. Many consumer SSD's provide TRIM capability under ESXi (I've tested a few myself) and running the 750 under Linux also shows TRIM capability, so I'd like to understand where support for TRIM is being disabled in this instance.

Re: Intel 750 not showing delete support under ESXi?

jbenavides — Wed, 25 Nov 2015 00:22:52 GMT

Hello shopkins,

I understand that this can be confusing. The Intel NVMe driver for VMware ESXi* 5.5 does not provide TRIM Support with the Intel® SSD 750 Series, and We understand that the native NVMe in VMware ESXi* 6.0 provides TRIM support for other NVMe drives.

Please keep in mind that the Intel® SSD 750 series were not designed to be used in Data Center environments, so we cannot guarantee full support and compatibility with ESXi*.

Re: Intel 750 not showing delete support under ESXi?

SChan70 — Fri, 18 Dec 2015 07:17:44 GMT

Related question, did you have to do something special to get 750 to work with ESXi? Mine does not even show up in 5.5 U3? Thanks