六、Libvirt

對於Libvirt，在啓動虛擬機之前，首先需要define虛擬機，是一個XML格式的文件

列出所有的Instance

# virsh list
Id Name State
----------------------------------------------------
10 instance-00000006 running

# virsh dumpxml instance-00000006
<domain type='kvm' id='10'>
<name>instance-00000006</name>
<uuid>73b896bb-7c7d-447e-ab6a-c4089532f003</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<resource>
    <partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>2014.1.1</entry>
      <entry name='serial'>80590690-87d2-e311-b1b0-a0481cabdfb4</entry>
      <entry name='uuid'>73b896bb-7c7d-447e-ab6a-c4089532f003</entry>
    </system>
</sysinfo>
<os>
    <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
</os>
<features>
    <acpi/>
    <apic/>
</features>
<cpu mode='host-model'>
    <model fallback='allow'/>
</cpu>
<clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/disk'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:ae:f4:17'/>
      <source bridge='qbrc51a349e-87'/>
      <target dev='tapc51a349e-87'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/20'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-73b896bb-7c7d-447e-ab6a-c4089532f003</label>
    <imagelabel>libvirt-73b896bb-7c7d-447e-ab6a-c4089532f003</imagelabel>
</seclabel>
</domain>

我們發現裏面定義了虛擬化類型kvm, vcpu, memory, disk, pty等，需要注意的是network，是一個tap device，attach到了qbr上。

虛擬化有很多種類型，參考下面的文章

虛擬化技術

[轉]Virtualization Basics

當然虛擬機啓動了之後，通過進程的查看，便能看到複雜無比的參數

# ps aux | grep instance-00000006
libvirt+ 22200 6.3 0.4 5464532 282888 ? Sl 09:51 0:09 qemu-system-x86_64 -enable-kvm -name instance-00000006 -S -machine pc-i440fx-trusty,accel=kvm,usb=off -cpu SandyBridge,+erms,+smep,+fsgsbase,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 73b896bb-7c7d-447e-ab6a-c4089532f003 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2014.1.1,serial=80590690-87d2-e311-b1b0-a0481cabdfb4,uuid=73b896bb-7c7d-447e-ab6a-c4089532f003 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000006.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:ae:f4:17,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

然而誰能解釋這些參數是幹什麼的？

請仔細閱讀下面兩篇文章

QEMU KVM libvirt 手冊(3) - Storage Media

QEMU KVM Libvirt手冊(7): 硬件虛擬化

QEMU KVM Libvirt手冊(8): 半虛擬化設備virtio

machine參數是總線Architecture，通過qemu-system-x86_64 --machine ?查看，default就是參數中的值。

accel=kvm說明虛擬化使用的是kvm

cpu表示處理器的參數以及處理器的一些flags，可以使用命令qemu-system-x86_64 --cpu ?查看

smp是對稱多處理器，

-smp 1,sockets=1,cores=1,threads=1

qemu仿真了一個具有1個vcpu，一個socket，一個core，一個threads的處理器。

socket, core, threads是什麼概念呢

（1）socket就是主板上插cpu的槽的數目，也即管理員說的”路“
（2）core就是我們平時說的”核“，即雙核，4核等
（3）thread就是每個core的硬件線程數，即超線程

具體例子，某個服務器是：2路4核超線程（一般默認爲2個線程），那麼，通過cat /proc/cpuinfo看到的是2*4*2=16個processor，很多人也習慣成爲16核了！

SMBIOS全稱System Management BIOS，用於表示x86 architectures的硬件信息，包含BIOS，主板的信息，這裏都是openstack，是假的了

-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000006.monitor,server,nowait

-mon chardev=charmonitor,id=monitor,mode=control

這是一對，用unix socket方式暴露monitor，從而可以通過virsh操作monitor

rtc是指system clock, -no-hpet是指不用更精準的時間。

-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 是USB，連接到PCI總線0上，是device 0, function 1

下面兩個是一對

-drive file=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none

-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1

表示硬盤，drive指向文件，device使用virtio，連到pci的總線0上，是device 4, funtion 0

下面兩個是一對

-netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25

-device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:ae:f4:17,bus=pci.0,addr=0x3

表示網卡，用tap device，device使用virtio，連接到pci的總線0上，是device 3，function 0

下面兩個是一對

-chardev file,id=charserial0,path=/var/lib/nova/instances/73b896bb-7c7d-447e-ab6a-c4089532f003/console.log

-device isa-serial,chardev=charserial0,id=serial0

是chardev，將log重定向到console.log

下面兩個是一對，是pty

-chardev pty,id=charserial1

-device isa-serial,chardev=charserial1,id=serial1

這是顯卡

-device cirrus-vga,id=video0,bus=pci.0,addr=0x2

這是內存

-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

都連接到pci總線上，通過命令virsh # qemu-monitor-command instance-00000024 --hmp "info pci"可以查看pci總線上的所有設備。

這裏面有很多半虛擬化的設備，從而可以提高性能

[轉] KVM VirtIO paravirtualized drivers: why they matter

Virtio: An I/O virtualization framework for Linux

QEMU KVM Libvirt手冊(8): 半虛擬化設備virtio

[轉]Virtio balloon

除了硬件的虛擬化和半虛擬化，對於網絡，qemu和kvm本身也有自己的機制

QEMU KVM Libvirt手冊(9): network

QEMU Networking

Virtual Networking

同時對於存儲，也有自己的機制

QEMU KVM Libvirt手冊(11): Managing Storage

這一節最後要說的，就是libvirt對虛擬機的管理

有一個強大的工具叫monitor，可以進行多種操作，相當於機器的管理界面，也可以通過virsh進行操作，參考文章QEMU KVM libvirt手冊(2)

最重要的命令行工具就是virsh了，參考QEMU KVM Libvirt手冊(10):Managing Virtual Machines with libvirt

七、Neutron

這一步，就是講instance連接到已經創建好的網絡設備上

步驟33：創建qbr網橋

步驟34：創建veth pair，qvo和qvb

步驟35：將qvb添加到qbr上

步驟36：將qvo添加到br-int上

看起來複雜而清晰的連接過程，爲什麼要這樣，需要理解neutron中的網絡設備架構

其實很早就有人畫出來了，如下面的圖

在network Node上：

在Compute Node上：

當看到這裏，很多人腦袋就大了，openstack爲什麼要創建這麼多的虛擬網卡，他們之間什麼關係，這些dl_vlan, mod_vlan_vid都是什麼東東啊？

請參考文章neutron的基本原理

neutron的不同的private network之間是隔離的，租戶隔離技術三種常用的VLAN, GRE，VXLAN，各有優缺點

VLAN原理

A virtual LAN (VLAN) is a group of networking devices in the same broadcast domain.

有兩種VLAN

Static VLAN/Port-based VLAN

manually assign a port on a switch to a VLAN using an Interface Subconfiguration mode command.

Dynamic VLANs

the switch automatically assigns the port to a VLAN using information from the user device, such as its MAC address, IP address, or even directory information (a user or group name, for instance).
The switch then consults a policy server, called a VLAN membership policy server (VMPS), which contains a mapping of device information to VLANs.

有兩種connection

Access-Link Connections

a device that has a standardized Ethernet NIC that understands only standardized Ethernet frames
Access-link connections can only be associated with a single VLAN.

Trunk Connections

trunk connections are capable of carrying traffic for multiple VLANs.

IEEE’s 802.1Q

優點

Increased performance

reducing collisions
limiting broadcast traffic
Less need to be routed

Improved manageability

Manage logical groups

Increased security options

packets only to other members of the VLAN.

缺點

limited number of VLANs 4000 -> 1000

number of MAC addresses supported in switches

GRE的原理

Generic Routing Encapsulation (GRE) is a tunneling protocol that can encapsulate a wide variety of network layer protocols inside virtual point-to-point links over an Internet Protocol internetwork.

Header

優點

Resolve the VLAN and MAC limitations by encapsulating communications within p2p 'tunnels' which hid the guest MAC information exposing only the MAC addresses of host systems.

L2 to L3, after leaving the encapsulated L2 virtual network, the traffic is forwarded to a gateway which can de-encapsulate traffic and route it out onto the leveraged unencapsulated network.

缺點

Point to point tunnel

Pool extensibility

Few switches can understand GRE Header, so load distribution and ACL (both depends on IPs and ports) can not be applied

VXLAN原理

Allow for virtual machines to live in two disparate networks yet still operate as if they were attached to the same L2.

Components:

Multicast support, IGMP and PIM
VXLAN Network Identifier (VNI): 24-bit segment ID
VXLAN Gateway
VXLAN Tunnel End Point (VTEP)
VXLAN Segment/VXLAN Overlay Network

When VM1 wants to send a packet to VM2, it needs the MAC address of VM2 this is the process that is followed:
VM1 sends a ARP packet requesting the MAC address associated with 192.168.0.101
This ARP is encapsulated by VTEP1 into a multicast packet to the multicast group associated with VNI 864
All VTEPs see the multicast packet and add the association of VTEP1 and VM1 to its VXLAN tables
VTEP2 receives the multicast packet decapsulates it, and sends the original broadcast on portgroups associated with VNI 864
VM2 sees the ARP packet and responds with its MAC address
VTEP2 encapsulates the response as a unicast IP packet and sends it back to VTEP1 using IP routing
VTEP1 decapsulates the packet and passes it on to VM1
At this point VM1 knows the MAC address of VM2 and can send directed packets to it as shown in in Figure 2: VM to VM communication:
VM1 sends the IP packet to VM2 from IP address 192.168.0.100 to 192.168.0.101
VTEP1 takes the packet and encapsulates it by adding the following headers:
VXLAN header with VNI=864
Standard UDP header and sets the UDP checksum to 0×0000, and the destination port being the VXLAN IANA designated port. Cisco N1KV is currently using port ID 8472.
Standard IP header with the Destination being VTEP2’s IP address and Protocol 0×011 for the UDP packet used for delivery
Standard MAC header with the MAC address of the next hop. In this case it is the router Interface with MAC address 00:10:11:FE:D8:D2 which will use IP routing to send it to the destination
VTEP2 receives the packet as it has it’s MAC address as the destination. The packet is decapsulated and found to be a VXLAN packet due to the UDP destination port. At this point the VTEP will look up the associated portgroups for VNI 864 found in the VXLAN header. It will then verify that the target, VM2 in this case, is allowed to receive frames for VNI 864 due to it’s portgroup membership and pass the packet on if the verification passes.
VM2 receives the packet and deals with it like any other IP packet.