Differences

This shows you the differences between two versions of the page.

--- acelab:opennebula_sr-iov_vmm_driver [2013/09/25 14:56]
dmacleod
+++ acelab:opennebula_sr-iov_vmm_driver [2021/12/09 16:42] (current)
@@ Line 1: / Line 1: @@
 ====== OpenNebula KVM SR-IOV Driver ======
-With the recent release of an OFED which supports SR-IOV on Infiniband HCAs it is now possible to use verbs from inside a VM. This VMM driver supports these Infiniband HCAs, and any other SR-IOV network device, in OpenNebula.
+With the release of an OFED which supports SR-IOV on Infiniband HCAs it is now possible to use verbs from inside a VM. This VMM driver supports these Infiniband HCAs, and any other SR-IOV network device, in OpenNebula.
 Demonstration Video:
@@ Line 17: / Line 17: @@
   - The maximum number of VMs with SR-IOV interfaces that the host can support is limited by the number of VFs the root device exposes. Usually around 64.
   - To use this driver with Ethernet SR-IOV devices you will need to modify the VM context script and take into consideration MAC prefixes. ([[#ethernet_sr-iov_devices|More info]])
-  - Network isolation is only supported on Mellanox ConnectX-3 HCAs with OFED v2.0-3.0.0 and a patched version of the ovswitch VNM driver is required. ([[#network_isolation|More info]])
+  - Network isolation with SR-IOV interfaces is only supported on Mellanox ConnectX-3 HCAs with OFED v2.0-3.0.0 and a patched version of the ovswitch VNM driver is required. ([[#network_isolation|More info]])
   - IPv6 has not been tested.
 ===== Testing Environment =====
   - CentOS 6.4
-  - Mellanox OFED 2.0
+  - Mellanox OFED 2.0-3.0.0
   - libvirt 0.10.2
   - OpenNebula 4.2
 ===== Installation =====
@@ Line 56: / Line 55: @@
 x07 0x00 0x04
-. On VM hosts using this driver edit ///etc/libvirt/qemu.conf// and change the user and group to “root” then restart libvirtd.
+. If using the driver in "mlnx_ofed2" mode you need to specify the device and port number of the interface. To find the device ID execute:
+    ls /sys/class/infiniband
+You will see a list of Infiniband devices "mlx4_<x>". The device ID is the <x>. To find the port ID execute:
+    ls /sys/class/infiniband/mlx4_<x>/ports
+You will see a list of numbers representing the ports available on the device.
-. On the head node restart OpenNebula.
+Each interface which you have created a map for needs an additional file called "<map_name>_ofed", eg. "ib0_ofed". The file contains two lines specifying its associated device and port.
-. Create a virtual network in OpenNebula. The “Bridge” field must contain “sriov_” prepended to the name of the file that contains the VF mapings, e.g. “sriov_ib0”
+Example,
-. Update the contextualisation scripts for the VMs with the code provided [[#context_script_modification|here]].
+///var/lib/one/remotes/vmm/kvm-sriov/vf_maps/ib0_ofed//:
+   device 0
+   port   1
-. Create hosts to use the “kvm_sriov” driver.
+. On VM hosts using this driver edit ///etc/libvirt/qemu.conf// and change the user and group to “root” then restart libvirtd.
+. On the head node restart OpenNebula.
+. Create a virtual network in OpenNebula. The “Bridge” field must contain “sriov_” prepended to the name of the file that contains the VF mapings, e.g. “sriov_ib0”
+. Update the contextualisation scripts for the VMs with the code provided [[#context_script_modification|here]].
+. Create hosts to use the “kvm_sriov” driver.
 ===== Appendix =====
 ==== Configuration Options ====
+|VF_MAPS_PATH|The path where the driver must look for the VF map files|
+|DUMMY_BRIDGE|The bridge to which the dummy interfaces will be connected to|
+|DUMMY_MAC_PREFIX|The prefix used to differentiate SR-IOV devices from virtual devices|
+|DRIVER_MODE|Either "generic" or "mlnx_ofed2". Generic mode passes IP address to VMs through dummy interfaces. Mlnx_ofed2 mode encodes IP address in VF GUIDs|
+|HPC_MODE|Either "on" or "off". HPC mode improves the performance of VMs. The host's CPU is passed through to the VM, may break migration. The memory backing for the VM is set to huge pages, [[#huge_pages|additional host configuration required]]. If a VM has as many vCPUs and the host has CPUs it is detected as a special case and the hosts NUMA architecture is passed to the VM. If a VM has as many vCPUs as a single NUMA node on the host the VM's CPUs will be pinned to a NUMA node.|
 ==== Libvirt “save” command ====
@@ Line 101: / Line 119: @@
 The context script will need to be updated to support the Ethernet devices, however the changes are minimal and simple.
+==== Huge Pages ====
+Enabling huge pages on KVM virtual machines will greatly improve memory performance. To enable huge pages on your VM hosts follow these instructions:
+http://www.linux-kvm.com/content/get-performance-boost-backing-your-kvm-guest-hugetlbfs
+Huge page backed VMs will be enabled if "HPC_MODE" is set "on".
+==== Network Isolation ====
+This driver supports network isolation on all virtual interfaces through the standard OpenNebula VNM drivers. Isolation on SR-IOV devices is only supported on Mellanox ConnectX-3 HCAs. The Infiniband isolation is implemented through a modified version of the "ovswitch" driver which ships with OpenNebula.
+To use Infiniband isolation the host and network must be set to use the Open vSwitch driver. The IB Pkey is set in the VLAN field when creating a network. Note, the Pkey must be specified in decimal. It will be converted into hexadecimal and a full key will be generated from it.
+The driver assumes the IB fabric has been correctly configured with the requested partitions. If during deployment the requested pkey is not found the virtual function will be blocked. If you do not specify a pkey the default partition will be used.
 ===== Driver =====
@@ Line 181: / Line 213: @@
             MASK=$(get_mask)
             GATEWAY=$(get_gateway)
-            echo "Writing"
             gen_iface_conf > /etc/sysconfig/network-scripts/ifcfg-${DEV}
           fi

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools