wiki.chpc.ac.za

User Tools

Site Tools

You are here: start » opennebula_sr-iov_vmm_driver
Trace: • atat • quantum_espresso_checkpointing_restarting • port_issues • ultrafluidx
opennebula_sr-iov_vmm_driver

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
opennebula_sr-iov_vmm_driver [2013/05/15 14:34]
dmacleod [Files] 		opennebula_sr-iov_vmm_driver [2021/12/09 16:42] (current)
Line 13: Line 13:
   - A virtual bridge is required. The VM will attach interfaces to this bridge but the OS will not use them. They are only required to pass IP address information into the VM. The bridge does not require any external connectivity, it just needs to exist.   - A virtual bridge is required. The VM will attach interfaces to this bridge but the OS will not use them. They are only required to pass IP address information into the VM. The bridge does not require any external connectivity, it just needs to exist.
   - VF usage tracking is implemented in the ///tmp// file system. During a fatal host error the VF usage tracking might become out of sync with actual VF usage. You will have to manually recover. ([[opennebula_sr-iov_vmm_driver#vf_usage_tracking|More info]])   - VF usage tracking is implemented in the ///tmp// file system. During a fatal host error the VF usage tracking might become out of sync with actual VF usage. You will have to manually recover. ([[opennebula_sr-iov_vmm_driver#vf_usage_tracking|More info]])
-  - A modified context script is required to decode the SR-IOV interface information inside the VM. Examples are given for Infiniband and Ethernet.+  - A modified context script is required to decode the SR-IOV interface information inside the VM. Examples are given for Infiniband.
   - IPv6 has not been tested.   - IPv6 has not been tested.
 ===== Testing Environment ===== ===== Testing Environment =====
Line 26: Line 26:
  
  
-1. Extract kvm-srivo.tar.gz to ///var/lib/one/remotes/vmm/kvm-sriov//+1. Extract {{:kvm-sriov.tar.gz|kvm-sriov.tar.gz}} to ///var/lib/one/remotes/vmm/kvm-sriov//
  
 2. Edit ///etc/one/oned.conf// and add: 2. Edit ///etc/one/oned.conf// and add:
Line 40: Line 40:
 https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/sect-Virtualization_Host_Configuration_and_Guest_Installation_Guide-SR_IOV-How_SR_IOV_Libvirt_Works.html  https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/sect-Virtualization_Host_Configuration_and_Guest_Installation_Guide-SR_IOV-How_SR_IOV_Libvirt_Works.html 
  
-4. In the ///var/lib/one/remotes/vmm/kvm-sriov/vf_maps// directory create a file with the name of the VFs root device, e.g. “ib0”. In the file write the bus, slot and function for each VF you want the driver to use. The address must be in hexadecimal. Each line in the file represents a VF, the bus, slot and function addresses are separated by a space “ “. **There must be no additional text, spaces or lines in the table**.+4. In the ///var/lib/one/remotes/vmm/kvm-sriov/vf_maps// directory create a file with the name of the VFs root device, e.g. “ib0”. In the file write the bus, slot and function for each VF you want the driver to use. The address must be in hexadecimal. Each line in the file represents a VF, the bus, slot and function addresses are separated by a single space character “ “. **There must be no additional text, spaces or lines in the table**.
  
 Example, four virtual functions: Example, four virtual functions:
Line 79: Line 79:
  
  
-Libvirt does provide a built-in method for tracking VF usage but it does not support the Mellanox OFED 2 drivers. As a result I had to create my own usage tracking mechanism. If you are using this driver with an SR-IOV card witch support VF assignment from a pool you can use the instructions found below to update the scripts:+Libvirt does provide a built-in method for tracking VF usage but it does not support the Mellanox OFED 2 drivers. As a result I had to create my own usage tracking mechanism. If you are using this driver with an SR-IOV card which supports VF assignment from a pool then you can use the instructions found below to update the scripts:
 http://wiki.libvirt.org/page/Networking#Assignment_with_.3Cinterface_type.3D.27hostdev.27.3E_.28SRIOV_devices_only.29  http://wiki.libvirt.org/page/Networking#Assignment_with_.3Cinterface_type.3D.27hostdev.27.3E_.28SRIOV_devices_only.29 
  
-In the hosts ///tmp// directory and vf_interfaces directory will be created by the driver. Inside the vf_interfaces directory a directory will be created for each SR-IOV root device. Inside the root device tracking folder files will be created to indicate that a VF is in use. For example, if the first, third and fourth VF are in use you see:+In the host's ///tmp// directory a vf_interfaces directory will be created by the driver. Inside the vf_interfaces directory a directory will be created for each SR-IOV root device. Inside the root device tracking folder files will be created to indicate that a VF is in use. For example, if the first, third and fourth VF are in use you see:
  
 ///tmp/vf_interfaces/ib0//: ///tmp/vf_interfaces/ib0//:
Line 89: Line 89:
     3     3
  
-===== Files =====+===== Driver =====
  
-  * SR-IOV KVM Driver +  * {{:kvm-sriov.tar.gz|SR-IOV KVM Driver}}
-  * Infiniband Context Script +
-  * Ethernet Context Script+
  
 +==== Context Script Modification ====
 +
 +
 +Edit the //00-network// script (in our case: /srv/one-context.d/00-network). Update the gen_network_configuration() and gen_iface_conf() functions.
 +
 +  gen_iface_conf() {
 +    cat <<EOT
 +    DEVICE=$DEV
 +    BOOTPROTO=none
 +    ONBOOT=yes
 +    ####################
 +    # Update this      #
 +    ####################
 +    TYPE=$TYPE
 +    #-------------------
 +    NETMASK=$MASK
 +    IPADDR=$IP
 +    EOT
 +      if [ -n "$GATEWAY" ]; then
 +        echo "GATEWAY=$GATEWAY"
 +      fi
 +      echo ""
 +  }
 +
 +  gen_network_configuration()
 +  {
 +    IFACES=`get_interfaces`
 +    for i in $IFACES; do
 +        MAC=`get_mac $i`
 +        ###################################
 +        # Update this                     #
 +        ###################################
 +        ib_vf=$(echo "$MAC" | cut -c 1-2)
 +        ib_pos=$(echo "$MAC" | cut -c 5)
 +        if  [ "$ib_vf" == "AA" ]; then
 +          DEV="ib"$ib_pos
 +          UPCASE_DEV=`upcase $DEV`
 +          TYPE="Infiniband"
 +        else
 +          DEV=`get_dev $i`
 +          UPCASE_DEV=`upcase $DEV`
 +          TYPE="Ethernet"
 +        fi
 +        #-----------------------------------
 +        IP=$(get_ip)
 +        NETWORK=$(get_network)
 +        MASK=$(get_mask)
 +        GATEWAY=$(get_gateway)
 +        ####################################
 +        # Update this                      #
 +        ####################################
 +        if  [ "$ib_vf" == "AA" ]; then
 +          gen_iface_conf > /etc/sysconfig/network-scripts/ifcfg-ib$ib_pos
 +        else
 +          gen_iface_conf > /etc/sysconfig/network-scripts/ifcfg-${DEV}
 +        fi
 +        #-----------------------------------
 +    done
 +  }
 +  
 +Take note of:
 +  - "$ib_vf" == "AA". This means that the script is expecting AA and the MAC prefic for SR-IOV devices.
 +  - gen_iface_conf > /etc/sysconfig/network-scripts/ifcfg-ib$ib_pos. The ifcfg file is being generated for Infiniand.
/app/dokuwiki/data/attic/opennebula_sr-iov_vmm_driver.1368621277.txt.gz · Last modified: 2021/12/09 16:42 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
CC Attribution-Noncommercial-Share Alike 3.0 Unported Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki