From f2d1cd2444616b91be7b99a0fefa9b4d0a5174b8 Mon Sep 17 00:00:00 2001 From: "Xinle.Guo" Date: Fri, 18 Feb 2022 16:17:57 +0800 Subject: [PATCH 10/10] vfio/doc: create a new document for using vfio Signed-off-by: Xinle.Guo --- docs/config_guidebook.md | 26 +++++-------- docs/vfio.md | 79 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 17 deletions(-) create mode 100644 docs/vfio.md diff --git a/docs/config_guidebook.md b/docs/config_guidebook.md index e0b8bdc..f541e16 100644 --- a/docs/config_guidebook.md +++ b/docs/config_guidebook.md @@ -447,32 +447,24 @@ Four properties can be set for PFlash device. ``` ### 2.11 VFIO -The VFIO driver is an IOMMU/device agnostic framework for exposing direct access to userspace, in a secure, IOMMU protected environment. Virtual machine often makes use of direct device access when configured for the highest possible I/O performance. +The VFIO driver is an IOMMU/device agnostic framework for exposing direct access to userspace, in a secure, +IOMMU protected environment. Virtual machine often makes use of direct device access when configured for the highest +possible I/O performance. -In order to successfully use VFIO device, it is mandatory that hardware supports virtualization and IOMMU groups. - -Assume user wants to access PCI device 0000:1a:00.3. -The device is attached to PCI bus, therefore user will make use of vfio-pci to manage the group: -```shell -# cmdline -modprobe vfio-pci -``` -Binding this device to the vfio-pci driver, it will create the VFIO group character devices for this group. -```shell -# cmdline -echo 0000:1a:00.3 > /sys/bus/pci/devices/0000:1a:00.3/driver/unbind -echo `lspci -ns 0000:1a:00.3 | awk -F':| ' '{print $5" "$6}'` > /sys/bus/pci/drivers/vfio-pci/new_id -``` Four properties are supported for VFIO device -* host: PCI device info in the system that contains domain, bus number, slot number and function number. +* host: PCI device info in the system that contains domain, bus number, slot number and function number. * id: VFIO device name. * bus: bus number of VFIO device. * addr: including slot number and function number. + ```shell -# cmdline -device vfio-pci,host=0000:1a:00.3,id=net,bus=pcie.0,addr=0x03.0x0[,multifunction=on] ``` +Note: the kernel must contain physical device drivers, otherwise it cannot be loaded normally. + +See [VFIO](./vfio.md) for more details. + ### 2.12 Chardev The type of chardev backend could be: stdio, pty, socket and file(output only). diff --git a/docs/vfio.md b/docs/vfio.md new file mode 100644 index 0000000..bd0aeea --- /dev/null +++ b/docs/vfio.md @@ -0,0 +1,79 @@ +# VFIO User Manual + +## Introduction + +The VFIO driver is an IOMMU/device agnostic framework for exposing direct access to userspace, in a secure, +IOMMU protected environment. Virtual machine often makes use of direct device access when configured for the highest +possible I/O performance. + +## Preparation + +In order to successfully use VFIO device, it is mandatory that hardware supports virtualization and IOMMU groups. +Execute the following command on your host OS to check whether the IOMMU has been turned on. +```shell +# dmesg | grep iommu +``` +If the IOMMU is turned on, the terminal display as follows: +```shell +iommu: Default domain type: Translated +hibmc-drm 0000:0a:00.0: Adding to iommu group 0 +ehci-pci 0000:7a:01.0: Adding to iommu group 1 +ehci-pci 0000:ba:01.0: Adding to iommu group 2 +ohci-pci 0000:7a:00.0: Adding to iommu group 3 +ohci-pci 0000:ba:00.0: Adding to iommu group 4 +xhci_hcd 0000:7a:02.0: Adding to iommu group 5 +... +``` +Assume user wants to access PCI device 0000:1a:00.3. +The device is attached to PCI bus, therefore user will make use of vfio-pci to manage the group: +```shell +# modprobe vfio-pci +``` + +## Bind VFIO device + +Binding this device to the vfio-pci driver, it will create the VFIO group character devices for this group. +```shell +# echo 0000:1a:00.3 > /sys/bus/pci/devices/0000:1a:00.3/driver/unbind +# echo `lspci -ns 0000:1a:00.3 | awk -F':| ' '{print $5" "$6}'` > /sys/bus/pci/drivers/vfio-pci/new_id +``` + +## Command line + +Four properties are supported for VFIO device +* host: PCI device info in the system that contains domain, bus number, slot number and function number. +* id: VFIO device name. +* bus: bus number of VFIO device. +* addr: including slot number and function number. +```shell +-device vfio-pci,host=0000:1a:00.3,id=net,bus=pcie.0,addr=0x03.0x0[,multifunction=on] +``` +Note: the kernel must contain physical device drivers, otherwise it cannot be loaded normally. + +## Hot plug management + +StratoVirt standard VM supports hot-plug VFIO devices with QMP. +Refer to qmp.md for specific command line parameters. + +### Example + +hot plug VFIO device: +```json +<- {"execute":"device_add", "arguments":{"id":"vfio-0", "driver":"vfio-pci", "bus": "pcie.1", "addr":"0x0", "host": "0000:1a:00.3"}} +-> {"return": {}} +``` +hot unplug VFIO device: +```json +<- {"execute": "device_del", "arguments": {"id": "vfio-0"}} +-> {"event":"DEVICE_DELETED","data":{"device":"vfio-0","path":"vfio-0"},"timestamp":{"seconds":1614310541,"microseconds":554250}} +-> {"return": {}} +``` + +## Unbind VFIO device + +If it is necessary to unbind VFIO device directly, you can execute the following command. +Note: assume uses hinic driver +```shell +# echo 0000:03:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind +# echo 0000:03:00.0 > /sys/bus/pci/drivers/hinic/bind +``` -- 2.25.1