The time has finally come! VMware has announced their newest release of vSphere 7.0 and I couldn’t be more excited! There is a lot to be excited about with this release, as noted by legendary Jason Massae himself in his blog post here. But if I am being completely honest, the thing I am most excited about is NVMe over Fabrics (NVMe-oF)! I am, and always have been, a protocol nerd. I enjoy learning about how data is transferred, translated, and everything that goes in to data processing. So the fact that VMware will now support NVMe-oF it has been kind of like a dream come true for me :).
I will be blogging about quite a few topics in the next while when it comes vSphere 7.0 and storage integration. So this is just the first of many to come and would recommend you keep an eye out for new blogs if you are interested. If there is anything in particular you would like to know about, please leave a comment and let me know!
Which transport protocol will Pure Storage support with vSphere 7.0 GA?
As noted in Jason Massae’s blog post referenced previously, VMware is initially supporting two transport protocols for NVMe-oF: Fibre Channel (FC) and RDMA over Converged Ethernet – version 2 (RoCE v2). Whether or not this gets expanded to other protocols in the future, we’ll see!
As for Pure Storage the initial integration will be supporting RDMA over Converged Ethernet (RoCE v2), also notated as NVMe-RoCE or NVME-RDMA. If you would like to understand more about this technology and the benefits I would highly recommend reading NVMe over RoCE Storage Fabrics for Noobs. It is a great read and breaks things down in a digestible manner for everyone.
Configuring NVMe-RDMA in vSphere 7.0
Now that we know which protocol is supported let’s dive into how you can configure ESXi for NVMe-RDMA connectivity.
Configuration of RDMA NICs on ESXi:
If the ESXi hosts are utilizing Mellanox adapters for NVMe-oF connectivity and the driver is nmlx4_core, you must enable RoCEv2. This is because the default operating mode is RoCEv1 for this particular driver and RoCEv2 is required for NVMe-oF connectivity.
In order to determine which driver your Mellanox adapters are using, look under the “Driver” section when running the following command on the ESXi host(s):
[root@barney:~] esxcfg-nics -l |grep -E 'Name|Mellanox' Name PCI Driver Link Speed Duplex MAC Address MTU Description vmnic4 0000:42:00.0 nmlx5_core Up 25000Mbps Full ec:0d:9a:82:5a:32 9000 Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA) vmnic5 0000:42:00.1 nmlx5_core Up 25000Mbps Full ec:0d:9a:82:5a:33 9000 Mellanox Technologies ConnectX-4 Lx EN NIC; 25GbE; dual-port SFP28; (MCX4121A-ACA)
In the above example my ESXi hosts are using the Mellanox adapter with the nmlx5_core driver.
The steps below are only applicable to the nmlx4_core driver. No action is required if the Mellanox adapter is using the nmlx5_core driver.
1. Run the following SSH command on all the ESXi hosts that will be connected via NVMe-RDMA:
esxcli system module parameters set -p enable_rocev2=1 -m nmlx4_core
2. Reboot the host.
Once the host has been rebooted you are able to confirm the change has taken affect by running the following command:
[root@barney:~] esxcli system module parameters list --module=nmlx4_core |grep -E 'Name|enable_rocev2' Name Type Value Description enable_rocev2 int 1 Enable RoCEv2 mode for all devices
If you see the “Value” as “1”, then you are all set. If the setting did not take, then there will be no “Value” listed, it will simply be blank.
The following steps must be taken for all Broadcom RDMA adapters.
1. Ensure that the (NIC + RDMA mode) is enabled for the RDMA NICs on the “Device Configuration” page in the BIOS of the host.
Refer to the vendor documentation on how to enter BIOS and modify this value.
2. Once you have confirmed NIC + RDMA mode is abled, run the following commands on the ESXi hosts to enable RoCEv2:
esxcli system module parameters set -m bnxtnet -p disable_roce=0 esxcli system module parameters set -m bnxtroce -p disable_rocev2=0
3. Reboot the host.
Again, to verify the settings have taken place, run the following commands:
[root@barney:~] esxcli system module parameters list --module=bnxtnet |grep -E 'Name|disable_roce' Name Type Value Description disable_roce bool 0 Disable the RoCE support. 0: enable RoCE support, 1: disable RoCE support. [Default: 1]
[root@barney:~] esxcli system module parameters list --module=bnxtroce |grep -E 'Name|disable_rocev2' Name Type Value Description disable_rocev2 bool 0 set to 1 to disable ROCEv2 support, set to 0 to enable RoCEv2 support (default)
The “Values” from both commands should show “0”, just as it does above.
Once you have enabled the appropriate configuration on your physical RDMA NICs (if applicable), the next step is to configure the vSwitches, port groups, and vmkernel ports on the ESXi hosts. If you have set up iSCSI previously you will notice that it is a very similar setup and configuration process.
NOTE: The below configuration is only one option (using standard vSwitches). If you would like to use a vSphere Distributed Switches (vDS) then that is also an acceptable configuration option as well.
The important points to consider when setting up your environment:
– At least 2 different port groups are required.
– Each port group should have only 1 physical RDMA NIC port in the “Active adapters” section. The other adapter port(s) should be in the “Unused adapters” section or not added at all.
– Ensure MTU is configured consistently between the vmkernel adapters and the virtual switch.
Example setup configuration for standard vSwitches:
1. Select the desired ESXi host, select the “Configure” tab, locate the “Networking” section, and select “Virtual switches”.
2. Select “VMkernel Network Adapter”.
3. Select “New standard switch” and input the desired MTU size.
4. Add one of the desired Physical NIC ports to the “Active Adapters” leaving the “Standby” and “Unused” adapters empty.
5. Input the desired name of your Port Group “Network label” and any other applicable settings. Ensure you leave all of the “Enabled services” unchecked.
6. Input the desired IP address and subnet mask for the vmknic.
7. Review all of the listed fields and ensure the configuration is correct, then “Finish” the setup.
Step 8: Repeat steps 1 – 7 for the other vSwitch, port group, and vmkernel port.
After each ESXi host is properly configured and setup with network connectivity the next step is to create the Software NVMe-RDMA Storage Adapters.
It is important to point out at that you will add at least two “Software NVMe over RDMA adapters” to each ESXi host. This is better outlined in Step #2 below.
1. Select the desired ESXi host, select the “Configure” tab, locate the “Storage” section, and select “Storage Adapters”.
Once you are on the “Storage Adapters” page, select “Add Software Adapter”.
2. Select “Add software NVMe over RDMA adapter” and choose which “RDMA device” you want to add.
You will repeat this process for all RDMA ports you plan on using.
If you have more than 2 RDMA adapters available, and do not plan on using all of them, you can look at the following ESXi host CLI output to compare which physical ports are associated with each virtual RDMA adapter:
[root@barney:~] esxcli rdma device list Name Driver State MTU Speed Paired Uplink Description ------- ---------- ------ ---- ------- ------------- ----------- vmrdma0 nmlx5_rdma Active 4096 25 Gbps vmnic4 MT27630 Family [ConnectX-4 LX] vmrdma1 nmlx5_rdma Active 4096 25 Gbps vmnic5 MT27630 Family [ConnectX-4 LX]
3. After you have added all applicable Software NVMe-RDMA Storage Adapters the next step is to configure the “Controllers” for every adapter.
You will select the applicable adapter you wish to configure, click on “Controllers” and then select “Add Controller”.
4. You can decide to “Automatically discover controllers” or “Enter controller details manually”. For simplicity, automatic discovery is recommended.
You will repeat this process for ALL FlashArray IP addresses dedicated for NVMe-RDMA connectivity.
After you have configured your ESXi hosts the next step is recording the NVMe Qualified Name (NQN) of each ESXi host you plan on connecting to the FlashArray.
[root@barney:~] esxcli nvme info get Host NQN: nqn.2014-08.com.vmware:nvme:barney
NOTE: Until NVMe-oF is implemented into the vSphere API, this is the only known way to get this information. So if you find a way to do it otherwise, please let me know!
From the FlashArray perspective, there isn’t going to be much of a difference from setting up iSCSI or FC. If there are questions for setting up hosts and host groups, then you can refer to the FlashArray Configuration KB as that should sufficiently guide you through this process.
As a complimentary addition to that KB, below is an example of adding an NQN to a host object on the FlashArray:
That’s it! Once you have all of that configured you are all set and can begin enjoying the beauty that is NVMe-RDMA!
Please leave a comment if you feel like something is missing or have clarifying questions. I would love to hear from you!