Differences

This shows you the differences between two versions of the page.

--- opennebula_sr-iov_vmm_driver [2013/05/15 14:30]
dmacleod [Prerequisites and Limitations]
+++ opennebula_sr-iov_vmm_driver [2021/12/09 16:42] (current)
@@ Line 1: / Line 1: @@
-====== OpenNebula KVM-SRIOV Driver ======
+====== OpenNebula KVM SR-IOV Driver ======
 Mellanox has recently released a driver which supports SR-IOV on their ConnectX-3 HCA family. This means it is now possible to use Infiniband verbs from inside a VM. To support these Infiniband HCAs, and any other SR-IOV network device, in OpenNebula I developed an SR-IOV VMM driver.
@@ Line 13: / Line 13: @@
   - A virtual bridge is required. The VM will attach interfaces to this bridge but the OS will not use them. They are only required to pass IP address information into the VM. The bridge does not require any external connectivity, it just needs to exist.
   - VF usage tracking is implemented in the ///tmp// file system. During a fatal host error the VF usage tracking might become out of sync with actual VF usage. You will have to manually recover. ([[opennebula_sr-iov_vmm_driver#vf_usage_tracking|More info]])
-  - A modified context script is required to decode the SR-IOV interface information inside the VM. Examples are given for Infiniband and Ethernet.
+  - A modified context script is required to decode the SR-IOV interface information inside the VM. Examples are given for Infiniband.
   - IPv6 has not been tested.
 ===== Testing Environment =====
@@ Line 26: / Line 26: @@
-. Extract kvm-srivo.tar.gz to ///var/lib/one/remotes/vmm/kvm-sriov//
+. Extract {{:kvm-sriov.tar.gz|kvm-sriov.tar.gz}} to ///var/lib/one/remotes/vmm/kvm-sriov//
 . Edit ///etc/one/oned.conf// and add:
@@ Line 40: / Line 40: @@
 https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/sect-Virtualization_Host_Configuration_and_Guest_Installation_Guide-SR_IOV-How_SR_IOV_Libvirt_Works.html
-. In the ///var/lib/one/remotes/vmm/kvm-sriov/vf_maps// directory create a file with the name of the VFs root device, e.g. “ib0”. In the file write the bus, slot and function for each VF you want the driver to use. The address must be in hexadecimal. Each line in the file represents a VF, the bus, slot and function addresses are separated by a space “ “. **There must be no additional text, spaces or lines in the table**.
+. In the ///var/lib/one/remotes/vmm/kvm-sriov/vf_maps// directory create a file with the name of the VFs root device, e.g. “ib0”. In the file write the bus, slot and function for each VF you want the driver to use. The address must be in hexadecimal. Each line in the file represents a VF, the bus, slot and function addresses are separated by a single space character “ “. **There must be no additional text, spaces or lines in the table**.
 Example, four virtual functions:
@@ Line 69: / Line 69: @@
-The “save” command saves the state of the VM's memory to the host's disk. This operation fails when a VM contains an SR-IOV device because it is passed through the hypervisor and not defined by it. As a resut the devices memory state cannot be read, cause the save operation to fail.
+The “save” command saves the state of the VM's memory to the host's disk. This operation fails when a VM contains an SR-IOV device because it is passed through the hypervisor and not defined by it. As a resut the devices memory state cannot be read, causing the save operation to fail.
 ==== Hot attaching SR-IOV Network Interfaces ====
@@ Line 79: / Line 79: @@
-Libvirt does provide a built-in method for tracking VF usage but it does not support the Mellanox OFED 2 drivers. As a result I had to create my own usage tracking mechanism. If you are using this driver with an SR-IOV card witch support VF assignment from a pool you can use the instructions found below to update the scripts:
+Libvirt does provide a built-in method for tracking VF usage but it does not support the Mellanox OFED 2 drivers. As a result I had to create my own usage tracking mechanism. If you are using this driver with an SR-IOV card which supports VF assignment from a pool then you can use the instructions found below to update the scripts:
 http://wiki.libvirt.org/page/Networking#Assignment_with_.3Cinterface_type.3D.27hostdev.27.3E_.28SRIOV_devices_only.29
-In the hosts ///tmp// directory and vf_interfaces directory will be created by the driver. Inside the vf_interfaces directory a directory will be created for each SR-IOV root device. Inside the root device tracking folder files will be created to indicate that a VF is in use. For example, if the first, third and fourth VF are in use you see:
+In the host's ///tmp// directory a vf_interfaces directory will be created by the driver. Inside the vf_interfaces directory a directory will be created for each SR-IOV root device. Inside the root device tracking folder files will be created to indicate that a VF is in use. For example, if the first, third and fourth VF are in use you see:
 ///tmp/vf_interfaces/ib0//:
@@ Line 88: / Line 88: @@
+===== Driver =====
+  * {{:kvm-sriov.tar.gz|SR-IOV KVM Driver}}
+==== Context Script Modification ====
+Edit the //00-network// script (in our case: /srv/one-context.d/00-network). Update the gen_network_configuration() and gen_iface_conf() functions.
+  gen_iface_conf() {
+    cat <<EOT
+    DEVICE=$DEV
+    BOOTPROTO=none
+    ONBOOT=yes
+    ####################
+    # Update this      #
+    ####################
+    TYPE=$TYPE
+    #-------------------
+    NETMASK=$MASK
+    IPADDR=$IP
+    EOT
+      if [ -n "$GATEWAY" ]; then
+        echo "GATEWAY=$GATEWAY"
+      fi
+      echo ""
+  }
+  gen_network_configuration()
+  {
+    IFACES=`get_interfaces`
+    for i in $IFACES; do
+        MAC=`get_mac $i`
+        ###################################
+        # Update this                     #
+        ###################################
+        ib_vf=$(echo "$MAC" | cut -c 1-2)
+        ib_pos=$(echo "$MAC" | cut -c 5)
+        if  [ "$ib_vf" == "AA" ]; then
+          DEV="ib"$ib_pos
+          UPCASE_DEV=`upcase $DEV`
+          TYPE="Infiniband"
+        else
+          DEV=`get_dev $i`
+          UPCASE_DEV=`upcase $DEV`
+          TYPE="Ethernet"
+        fi
+        #-----------------------------------
+        IP=$(get_ip)
+        NETWORK=$(get_network)
+        MASK=$(get_mask)
+        GATEWAY=$(get_gateway)
+        ####################################
+        # Update this                      #
+        ####################################
+        if  [ "$ib_vf" == "AA" ]; then
+          gen_iface_conf > /etc/sysconfig/network-scripts/ifcfg-ib$ib_pos
+        else
+          gen_iface_conf > /etc/sysconfig/network-scripts/ifcfg-${DEV}
+        fi
+        #-----------------------------------
+    done
+  }
+Take note of:
+  - "$ib_vf" == "AA". This means that the script is expecting AA and the MAC prefic for SR-IOV devices.
+  - gen_iface_conf > /etc/sysconfig/network-scripts/ifcfg-ib$ib_pos. The ifcfg file is being generated for Infiniand.

wiki.chpc.ac.za

User Tools

Site Tools

Differences

Page Tools