<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hello Eddie,<br>
<br>
I've tried the way you suggested me, and I got exactly the same
qemu line to be executed.<br>
I suggest you to use my way of using the vendor-id and don't mess
with qemu arguments in the xml file.<br>
<br>
Regards,<br>
George<br>
<br>
On 02/11/2015 02:54 μμ, Eddie Yen wrote:<br>
</div>
<blockquote
cite="mid:CAHZFSBq-uCa00UZDU+Wd7VA059d7Sj1JpmjjVVNZ6rF8KBs+2Q@mail.gmail.com"
type="cite">
<div dir="ltr">For now the latest driver is 358.50, and my guest
using latest driver wo any problem.
<div>But I'm using the method that AW talking about, so maybe
give it a try?</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-11-02 20:47 GMT+08:00 Georgios
Kourachanis <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:geo.kourachanis@gmail.com" target="_blank">geo.kourachanis@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>It's the same thing, either by adding them with
qemuarguments, or with the wrapper.<br>
<br>
The thing is to use the hyper-v functions. That's what
the hyper-v vendor-id patch has given to us. The ability
of hidding the hyper-v functions from nvidia GPUs so
that we can use them!<br>
<br>
Also, I've tried with a null name for the vendor-id, I
got the same performance.<br>
<br>
The nvidia drivers I'm currently using are 358.50<br>
<br>
Moreover, could you suggest me a nice software to test
the VM's performance in general? I don't really like
passmark.
<div>
<div class="h5"><br>
<br>
<br>
<br>
On 02/11/2015 02:11 μμ, Eddie Yen wrote:<br>
</div>
</div>
</div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">OK, but I still suggest that remove
Hyper-V function tags in your XML.
<div>Because we don't know about what new tricks
inside the driver that NVIDIA wants to
"surprise" us.</div>
<div><br>
</div>
<div>For me, my GTX980 works well by using upon
edits. But I'm using 4820K which didn't need ACS
patch and wo intel graphic.</div>
<div>So I'm not sure it may cause by patch or sth.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-11-02 20:04
GMT+08:00 Georgios Kourachanis <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:geo.kourachanis@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:geo.kourachanis@gmail.com">geo.kourachanis@gmail.com</a></a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hello Eddie,<br>
<br>
Thanks for answering, though:<br>
<br>
What you suggest me to do, I've already
done it with this way:<br>
<br>
<font color="#000000"><span><tt>/usr/local/bin/qemu-system-x86_64.hv</tt><tt>:</tt><tt><br>
</tt><tt>#!/bin/sh</tt><tt><br>
</tt><tt>exec
/usr/bin/qemu-system-x86_64 `echo
"\$@" | \</tt><tt><br>
</tt><tt>sed
's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'</tt></span><tt><br>
<br>
<br>
and by changing the emulator qemu to
this line:<br>
<br>
<</tt></font><font color="#000000"><tt>emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator></tt><tt><br>
</tt></font><br>
I'm just giving the ID "GoobyPLS" to the
vendor. I'll try without a vendor name to
see if it changes anything.<br>
<br>
Also, I'm using the qemu git version
"r41983.g3a958f5" so it already contains
the patch that helps us use the lines
above.
<div>
<div><br>
<br>
<br>
<br>
<br>
On 02/11/2015 03:53 πμ, Eddie Yen
wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">According from AW's
blog:
<div>"<span
style="font-size:13.1999998092651px;line-height:18.4799995422363px;font-family:Arial,Helvetica,sans-serif"> For
this step we again need to run </span><span>virsh
edit</span><span
style="font-size:13.1999998092651px;line-height:18.4799995422363px;font-family:Arial,Helvetica,sans-serif"> on
the VM. Within the</span><span><features></span><span
style="font-size:13.1999998092651px;line-height:18.4799995422363px;font-family:Arial,Helvetica,sans-serif"> section,
remove everything between the </span><span><hyperv></span><span
style="font-size:13.1999998092651px;line-height:18.4799995422363px;font-family:Arial,Helvetica,sans-serif"> tags,
including the tags themselves.0</span>"</div>
<div>and</div>
<div>"<span
style="font-size:13.1999998092651px;line-height:18.4799995422363px;font-family:Arial,Helvetica,sans-serif">Additionally,
within the </span><span><clock></span><span
style="font-size:13.1999998092651px;line-height:18.4799995422363px;font-family:Arial,Helvetica,sans-serif"> tag,
find the timer named </span><span>hypervclock</span><span
style="font-size:13.1999998092651px;line-height:18.4799995422363px;font-family:Arial,Helvetica,sans-serif">,
remove the line containing this
tag completely. Save and exit
the edit session.</span>"</div>
<div><br>
</div>
<div>I found that these still exist
in your XML file, so try to do
this:</div>
<div><br>
</div>
<div>1. Remove these tags.</div>
<div>2. Re-compile QEMU and
re-install it with this patch</div>
<div><a moz-do-not-send="true"
href="http://www.spinics.net/lists/kvm/msg121742.html"
target="_blank">http://www.spinics.net/lists/kvm/msg121742.html</a><br>
</div>
<div>3. Add these tags between
</devices> and
</domain></div>
<div><br>
</div>
<div><qemu:commandline></div>
<div> <qemu:arg
value='-cpu'/></div>
<div> <qemu:arg
value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv)vendor_id='/><br>
</div>
<div></qemu:commandline></div>
<div><br>
</div>
<div>I'm using GTX980, too. Before
that, I got poor 3D performance in
Windows 10, after this patch and
edition, I got performance back.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-11-02
1:43 GMT+08:00 Georgios
Kourachanis <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:geo.kourachanis@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:geo.kourachanis@gmail.com">geo.kourachanis@gmail.com</a></a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div>
<div bgcolor="#FFFFFF"
text="#000000"> <tt>Hello
</tt><tt>all,</tt><tt><br>
</tt><tt><br>
</tt><tt>I had been using
Xen </tt><tt>with some
AMD GPUs </tt><tt>for
almost 2 years till
about June 2015, when I
then found out that KVM
and libvirt could do the
same stuff I was
interested in with
nvidia GPUs, too. I
needed the CUDA cores so
I did change to an ASUS
GTX 980 Strix. But
unfortunatelly, I don't
get any good performance
output from it. On
native windows 7/10
installation it's a
beast though.</tt><tt><br>
</tt><tt>I also have an
AMD R7 250 which works
great with KVM. But
let's not mess with it.</tt><tt><br>
</tt><tt><br>
</tt><tt>Let me get to the
point:</tt><tt><br>
</tt><tt><br>
</tt><tt>I have no
problems as for the
installation of Windows
or OVMF or
passing-through or
anything else. The only
problem is the GTX980's
performance.</tt><tt><br>
</tt><tt>The performance
had a significant boost
when I used the latest
qemu branch with the
hyper-v trick, but
still, not getting what
many people seem to
claim in this mailing
list "almost-native"
(even with nvidia GPUs).</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>Here is my
system's specs:</tt><tt><br>
</tt><tt><br>
</tt><tt>Archlinux with
4.1.6-1-vfio (with the
ACS patch ALONE)</tt><tt><br>
</tt><tt>Intel Core i7</tt><tt>
3770 ( I use the igpu
for the archlinux)</tt><tt><br>
</tt><tt>24GiB RAM</tt><tt><br>
</tt><tt>ASUS GTX 980
Strix</tt><tt><br>
</tt><tt>Sapphire R7 250</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>lspci
(only pass-through'd
stuff):</tt><tt><br>
</tt><tt><br>
</tt><tt>01:00.0 VGA
compatible controller:
NVIDIA Corporation GM204
[GeForce GTX 980] (rev
a1)</tt><tt><br>
</tt><tt>
Subsystem: ASUSTeK
Computer Inc. Device
8518</tt><tt><br>
</tt><tt> Kernel
driver in use: vfio-pci</tt><tt><br>
</tt><tt> Kernel
modules: nouveau</tt><tt><br>
</tt><tt>01:00.1 Audio
device: NVIDIA
Corporation GM204 High
Definition Audio
Controller (rev a1)</tt><tt><br>
</tt><tt>
Subsystem: ASUSTeK
Computer Inc. Device
8518</tt><tt><br>
</tt><tt> Kernel
driver in use: vfio-pci</tt><tt><br>
</tt><tt> Kernel
modules: snd_hda_intel</tt><tt><br>
</tt><tt>02:00.0 VGA
compatible controller:
Advanced Micro Devices,
Inc. [AMD/ATI] Oland PRO
[Radeon R7 240/340]</tt><tt><br>
</tt><tt>
Subsystem: PC Partner
Limited / Sapphire
Technology Device e266</tt><tt><br>
</tt><tt> Kernel
modules: radeon</tt><tt><br>
</tt><tt>02:00.1 Audio
device: Advanced Micro
Devices, Inc. [AMD/ATI]
Cape Verde/Pitcairn HDMI
Audio [Radeon HD
7700/7800 Series]</tt><tt><br>
</tt><tt>
Subsystem: PC Partner
Limited / Sapphire
Technology Device aab0</tt><tt><br>
</tt><tt> Kernel
driver in use:
snd_hda_intel</tt><tt><br>
</tt><tt> Kernel
modules: snd_hda_intel</tt><tt><br>
</tt><tt>08:00.0 USB
controller: ASMedia
Technology Inc. ASM1042
SuperSpeed USB Host
Controller</tt><tt><br>
</tt><tt>
Subsystem: ASRock
Incorporation
Motherboard</tt><tt><br>
</tt><tt> Kernel
driver in use: vfio-pci</tt><tt><br>
</tt><tt> Kernel
modules: xhci_pci</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>booting
lines:</tt><tt><br>
</tt><tt><br>
</tt><tt>linux
/boot/vmlinuz-linux-vfio
root=UUID=XXXX rw
intel_iommu=on
pcie_acs_override=downstream
isolcpus=2-3,6-7
nohz_full=2-3,6-7</tt><tt><br>
</tt><tt>initrd
/boot/intel-ucode.img
/boot/initramfs-linux-vfio.img</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>/etc/fstab:</tt><code><br>
<br>
hugetlbfs /hugepages
hugetlbfs defaults 0 0</code><tt><br>
</tt>
<hr size="2" width="100%"><tt>/etc/sysctl.d/40-hugepage.conf:</tt><tt><br>
</tt><tt><br>
</tt><tt>vm.nr_hugepages =
8000</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>/etc/modules-load.d/vfio.conf:</tt><tt><br>
</tt><tt><br>
</tt><tt>kvm</tt><tt><br>
</tt><tt>kvm-intel</tt><tt><br>
</tt><tt>vfio</tt><tt><br>
</tt><tt>vfio-pci</tt><tt><br>
</tt><tt>vfio_iommu_type1</tt><tt><br>
</tt><tt>vfio_virqfd</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>/etc/modprobe.d/kvm.conf:</tt><tt><br>
</tt><tt><br>
</tt><tt>options kvm
ignore_msrs=1</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>/etc/modprobe.d/kvm-intel.conf:</tt><tt><br>
</tt><tt><br>
</tt><tt>options kvm-intel
nested=1</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>/etc/modprobe.d/vfio_iommu_type1.conf:</tt><tt><br>
</tt><tt><br>
</tt><tt>options
vfio_iommu_type1
allow_unsafe_interrupts=0</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt>/etc/modprobe.d/vfio-pci.conf:</tt><tt><br>
</tt><tt><br>
</tt><tt>options vfio-pci
ids=10de:13c0,10de:0fbb,1002:6613,1002:aab0,1b21:1042</tt><tt><br>
</tt>
<hr size="2" width="100%"><tt><br>
</tt><tt>And the virsh
xml:</tt><tt><br>
</tt><tt><br>
</tt><tt><domain
type='kvm'></tt><tt><br>
</tt><tt>
<name>windows_10</name></tt><tt><br>
</tt><tt>
<uuid>63045df8-c782-4cfd-abc7-a3598826ae83</uuid></tt><tt><br>
</tt><tt> <memory
unit='KiB'>6553600</memory></tt><tt><br>
</tt><tt>
<currentMemory
unit='KiB'>6553600</currentMemory></tt><tt><br>
</tt><tt>
<memoryBacking></tt><tt><br>
</tt><tt>
<hugepages/></tt><tt><br>
</tt><tt>
</memoryBacking></tt><tt><br>
</tt><tt> <vcpu
placement='static'>4</vcpu></tt><tt><br>
</tt><tt> <cputune></tt><tt><br>
</tt><tt> <vcpupin
vcpu='0' cpuset='2'/></tt><tt><br>
</tt><tt> <vcpupin
vcpu='1' cpuset='3'/></tt><tt><br>
</tt><tt> <vcpupin
vcpu='2' cpuset='6'/></tt><tt><br>
</tt><tt> <vcpupin
vcpu='3' cpuset='7'/></tt><tt><br>
</tt><tt>
</cputune></tt><tt><br>
</tt><tt> <os></tt><tt><br>
</tt><tt> <type
arch='x86_64'
machine='pc-i440fx-2.4'>hvm</type></tt><tt><br>
</tt><tt> <loader
readonly='yes'
type='pflash'>/usr/local/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader></tt><tt><br>
</tt><tt>
<nvram>/var/lib/libvirt/qemu/nvram/windows_nvidia_VARS.fd</nvram></tt><tt><br>
</tt><tt> </os></tt><tt><br>
</tt><tt>
<features></tt><tt><br>
</tt><tt> <acpi/></tt><tt><br>
</tt><tt> <apic/></tt><tt><br>
</tt><tt> <pae/></tt><tt><br>
</tt><tt>
<hyperv></tt><tt><br>
</tt><tt> <relaxed
state='on'/></tt><tt><br>
</tt><tt> <vapic
state='on'/></tt><tt><br>
</tt><tt>
<spinlocks state='on'
retries='8191'/></tt><tt><br>
</tt><tt>
</hyperv></tt><tt><br>
</tt><tt> <kvm></tt><tt><br>
</tt><tt> <hidden
state='on'/></tt><tt><br>
</tt><tt> </kvm></tt><tt><br>
</tt><tt> <vmport
state='off'/></tt><tt><br>
</tt><tt>
</features></tt><tt><br>
</tt><tt> <cpu
mode='host-passthrough'></tt><tt><br>
</tt><tt> <topology
sockets='1' cores='4'
threads='1'/></tt><tt><br>
</tt><tt> </cpu></tt><tt><br>
</tt><tt> <clock
offset='localtime'></tt><tt><br>
</tt><tt> <timer
name='rtc'
tickpolicy='catchup'/></tt><tt><br>
</tt><tt> <timer
name='pit'
tickpolicy='delay'/></tt><tt><br>
</tt><tt> <timer
name='hpet'
present='no'/></tt><tt><br>
</tt><tt> <timer
name='hypervclock'
present='yes'/></tt><tt><br>
</tt><tt> </clock></tt><tt><br>
</tt><tt>
<on_poweroff>destroy</on_poweroff></tt><tt><br>
</tt><tt>
<on_reboot>restart</on_reboot></tt><tt><br>
</tt><tt>
<on_crash>restart</on_crash></tt><tt><br>
</tt><tt> <pm></tt><tt><br>
</tt><tt>
<suspend-to-mem
enabled='no'/></tt><tt><br>
</tt><tt>
<suspend-to-disk
enabled='no'/></tt><tt><br>
</tt><tt> </pm></tt><tt><br>
</tt><tt> <devices></tt><tt><br>
</tt><tt>
<emulator>/usr/local/bin/qemu-system-x86_64.hv</emulator></tt><tt><br>
</tt><tt> <disk
type='block'
device='disk'></tt><tt><br>
</tt><tt> <driver
name='qemu' type='raw'
cache='none'/></tt><tt><br>
</tt><tt> <source
dev='/dev/mapper/vg_ssd-lv_kvm_NVIDIA'/></tt><tt><br>
</tt><tt> <target
dev='sda'
bus='scsi'/></tt><tt><br>
</tt><tt> <boot
order='1'/></tt><tt><br>
</tt><tt> <address
type='drive'
controller='0' bus='0'
target='0' unit='0'/></tt><tt><br>
</tt><tt> </disk></tt><tt><br>
</tt><tt> <disk
type='block'
device='disk'></tt><tt><br>
</tt><tt> <driver
name='qemu' type='raw'
cache='none'/></tt><tt><br>
</tt><tt> <source
dev='/dev/mapper/vg_raid5-lv_xen_ntfs_files'/></tt><tt><br>
</tt><tt> <target
dev='sdb'
bus='scsi'/></tt><tt><br>
</tt><tt> <address
type='drive'
controller='0' bus='0'
target='0' unit='1'/></tt><tt><br>
</tt><tt> </disk></tt><tt><br>
</tt><tt>
<controller
type='usb' index='0'></tt><tt><br>
</tt><tt> <address
type='pci'
domain='0x0000'
bus='0x00' slot='0x01'
function='0x2'/></tt><tt><br>
</tt><tt>
</controller></tt><tt><br>
</tt><tt>
<controller
type='pci' index='0'
model='pci-root'/></tt><tt><br>
</tt><tt>
<controller
type='scsi' index='0'
model='virtio-scsi'></tt><tt><br>
</tt><tt> <address
type='pci'
domain='0x0000'
bus='0x00' slot='0x06'
function='0x0'/></tt><tt><br>
</tt><tt>
</controller></tt><tt><br>
</tt><tt> <interface
type='bridge'></tt><tt><br>
</tt><tt> <mac
address='52:54:00:e9:85:8f'/></tt><tt><br>
</tt><tt> <source
bridge='xenbr0'/></tt><tt><br>
</tt><tt> <model
type='e1000'/></tt><tt><br>
</tt><tt> <address
type='pci'
domain='0x0000'
bus='0x00' slot='0x03'
function='0x0'/></tt><tt><br>
</tt><tt>
</interface></tt><tt><br>
</tt><tt> <hostdev
mode='subsystem'
type='pci'
managed='yes'></tt><tt><br>
</tt><tt>
<source></tt><tt><br>
</tt><tt>
<address
domain='0x0000'
bus='0x01' slot='0x00'
function='0x0'/></tt><tt><br>
</tt><tt>
</source></tt><tt><br>
</tt><tt> <address
type='pci'
domain='0x0000'
bus='0x00' slot='0x0a'
function='0x0'
multifunction='on'/></tt><tt><br>
</tt><tt>
</hostdev></tt><tt><br>
</tt><tt> <hostdev
mode='subsystem'
type='pci'
managed='yes'></tt><tt><br>
</tt><tt>
<source></tt><tt><br>
</tt><tt>
<address
domain='0x0000'
bus='0x01' slot='0x00'
function='0x1'/></tt><tt><br>
</tt><tt>
</source></tt><tt><br>
</tt><tt> <address
type='pci'
domain='0x0000'
bus='0x00' slot='0x0a'
function='0x1'/></tt><tt><br>
</tt><tt>
</hostdev></tt><tt><br>
</tt><tt> <hostdev
mode='subsystem'
type='pci'
managed='yes'></tt><tt><br>
</tt><tt>
<source></tt><tt><br>
</tt><tt>
<address
domain='0x0000'
bus='0x08' slot='0x00'
function='0x0'/></tt><tt><br>
</tt><tt>
</source></tt><tt><br>
</tt><tt> <address
type='pci'
domain='0x0000'
bus='0x00' slot='0x08'
function='0x0'/></tt><tt><br>
</tt><tt>
</hostdev></tt><tt><br>
</tt><tt>
<memballoon
model='virtio'></tt><tt><br>
</tt><tt> <address
type='pci'
domain='0x0000'
bus='0x00' slot='0x05'
function='0x0'/></tt><tt><br>
</tt><tt>
</memballoon></tt><tt><br>
</tt><tt>
</devices></tt><tt><br>
</tt><tt></domain></tt><tt><br>
</tt>
<hr size="2" width="100%"><tt><br>
</tt><tt>/usr/local/bin/qemu-system-x86_64.hv</tt><tt>:</tt><tt><br>
</tt><tt>#!/bin/sh</tt><tt><br>
</tt><tt>exec
/usr/bin/qemu-system-x86_64
`echo "\$@" | \</tt><tt><br>
</tt><tt>sed
's|hv_time|hv_time,hv_vendor_id=GoobyPLS|g'</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>And some notes:</tt><tt><br>
</tt><tt><br>
</tt><tt>1) Using "</tt><tt><topology
sockets='1' cores='4'
threads='1'/></tt><tt>"
instead o</tt><tt>f </tt><tt>"</tt><tt><topology
sockets='1' cores='</tt><tt>2</tt><tt>'
threads='</tt><tt>2</tt><tt>'/></tt><tt>"
provided about 2% boost
in GPU performance. No
change in RAM or CPU
tests. I've tested with
the passmark</tt><tt>.</tt><tt><br>
</tt><tt><br>
</tt><tt>2) I tried using
the emulatorpin method
Alex says on a mail here
on </tt><tt>vfio-users</tt><tt>,
but I didn't notice any
changed in GPU
performance. I didn't
test it on the CPU side
though.</tt><tt><br>
</tt><tt><br>
</tt><tt>3) The main
problem of the
performance lack is that
a specific game that
I've been playing isn't
quite playable. That
game has been mentioned
before here on the list,
it's Tera (european
version (gameforge),
although american
version</tt><tt>(enmasse)
has exactly the same
performance).</tt><tt><br>
</tt><tt><br>
</tt><tt>4) Every othe</tt><tt>r
game I managed to play
is quite playable,
though I haven't tested
them to see if they run
on native speeds.</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>I'd really want
some help on this
matter, I really want to
make my server run this
VM with the nvidia GPU.
I hate dual booting
Windows </tt><tt>>_></tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>Thank</tt><tt>s!</tt><tt><br>
</tt> </div>
</div>
</div>
<br>
_______________________________________________<br>
vfio-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:vfio-users@redhat.com"
target="_blank">vfio-users@redhat.com</a><br>
<a moz-do-not-send="true"
href="https://www.redhat.com/mailman/listinfo/vfio-users"
rel="noreferrer"
target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>