[libvirt-users] 'virsh capabilities' on Debian Wheezy-amd64 reports different cpu to Wheezy-i386 (on same hardware)
Struan Bartlett
struan.bartlett at NewsNow.co.uk
Mon Mar 3 20:37:05 UTC 2014
On 03/03/2014 14:53, Martin Kletzander wrote:
> On Mon, Mar 03, 2014 at 02:15:43PM +0000, Struan Bartlett wrote:
>>
>> On 03/03/2014 13:42, Martin Kletzander wrote:
>>> On Mon, Mar 03, 2014 at 11:15:51AM +0000, Struan Bartlett wrote:
>>>> On 03/03/2014 10:55, Martin Kletzander wrote:
>>>>> On Mon, Mar 03, 2014 at 10:47:03AM +0000, Struan Bartlett wrote:
>>>>>> On 03/03/2014 10:44, Martin Kletzander wrote:
>>>>>>> On Mon, Mar 03, 2014 at 10:30:11AM +0000, Struan Bartlett wrote:
>>>>>>>> Hi Martin
>>>>>>>>
>>>>>>>> Thanks for your response. Here's the output of that grep:
>>>>>>>>
>>>>>>>> # grep ^flags /proc/cpuinfo | sort -u
>>>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
>>>>>>>> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall
>>>>>>>> nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
>>>>>>>> xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor
>>>>>>>> ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2
>>>>>>>> x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat xsaveopt
>>>>>>>> pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
>>>>>>>>
>>>>>>>> I've just managed to install the libvirt-bin:amd64 package on the same
>>>>>>>> machine, on the wheezy-i386 distribution. The output of 'virsh
>>>>>>>> capabilities' is now reported correctly, and my VMs that required
>>>>>>>> SandyBridge are now booting.
>>>>>>>>
>>>>>>>> Have you any further suggestions?
>>>>>>>>
>>>>>>> Oh, I missed the fact that it was 32 bit distro. In that case, I
>>>>>>> guess some cpu features won't be available. No other ideas.
>>>>>> Ok thanks Martin. If the amd64 and i386 versions of the package are
>>>>>> behaving differently in this respect, on otherwise the same hardware,
>>>>>> kernel and distro, then I can only presume this is a bug, so my next
>>>>>> step would be to raise this on the development list.
>>>>> Just a guess, but our CPUID instruction might be translated to
>>>>> assembly code differently. You can try running 'cpuid -ir' (or 'cpuid
>>>>> -ir1' if it's the same on all cpus) on both machines, that will give
>>>>> you the results libvirt is processing. If that output is different
>>>>> than there is nothing libvirt can do. If it's the same, it might be a
>>>>> libvirt bug.
>>>> You might be onto something here. The following are run on the same
>>>> physical machine:
>>>>
>>>> # apt-get install cpuid:i386
>>>> # cpuid -ir1 >/tmp/cpuid-ir1-i386
>>>> # apt-get install cpuid:amd64
>>>> # cpuid -ir1 >/tmp/cpuid-ir1-amd64
>>>> # diff -ubw /tmp/cpuid-ir1-i386 /tmp/cpuid-ir1-amd64
>>>> --- /tmp/cpuid-ir1-i386 2014-03-03 11:13:00.839208222 +0000
>>>> +++ /tmp/cpuid-ir1-amd64 2014-03-03 11:13:13.283307377 +0000
>>>> @@ -10,11 +10,11 @@
>>>> 00000008 00000000 00000000 00000000 00000000
>>>> 00000009 00000000 00000000 00000000 00000000
>>>> 0000000a 07300403 00000000 00000000 00000603
>>>> -0000000b 00000000 00000000 000000e8 0000002c
>>>> +0000000b 00000000 00000000 0000006f 0000002c
>>> This one has *probably* no significance in libvirt code since we only
>>> call cpuid functions 0x00000001, 0x00000007 and 0x80000001 (which is
>>> enough for us to determine all we need).
>>>
>>>> 0000000c 00000000 00000000 00000000 00000000
>>>> 0000000d 00000000 00000000 00000000 00000000
>>>> 80000000 80000008 00000000 00000000 00000000
>>>> -80000001 00000000 00000000 00000001 2c100000
>>>> +80000001 00000000 00000000 00000001 2c100800
>>> But this is something.
>>>
>>> This means in second run the CPU reports it has "syscall" and in the
>>> first one it doesn't. Since "syscall" is x86_64 instruction used
>>> instead of i386's "int 0x80" (IIRC) it makes sense that it is missing
>>> on 32-bit CPU (when the code is running in 32-bit mode).
>>>
>>>> 80000002 20202020 6e492020 286c6574 58202952
>>>> 80000003 286e6f65 43202952 45205550 36322d35
>>>> 80000004 204c3035 20402030 30382e31 007a4847
>>>>
>>>> Could this be enough difference to throw cpu-detection off?
>>>>
>>> Even one bit is enough if that bit represents a flag which is not in
>>> the CPU you are looking for (i.e. SandyBridge).
>>>
>>> I must admin I was too lazy to compare the differences between the
>>> processor definitions you've sent (basically because I'm fairly
>>> certain we have no such mistakes in our code O:-) ).
>>>
>>> But I did that now and indeed, you can see that the difference is only
>>> in the "syscall" feature. You can do that yourselves using the info
>>> from "cpu_map.xml" if you don't believe me ;)
>>>
>>> Martin
>> Martin
>>
>> Many thanks for helping me get to the bottom of this, though could you
>> clarify what you mean when you say that the difference is only in the
>> "syscall" feature? I've looked at cpu_map.xml, and compared the features
>> for n270 (pentiumpro+coreduo+ssse3) with SandyBridge
>> (pentiumpro+Conroe+Penryn+Nehalem+Westmere+pclmuldq+SandyBridge) and am
>> seeing quite a few differences (see below). The main issue though for
> You have to also add the features that libvirt added in the output of
> 'virsh capabilities' and then try to match the differences. I did it
> by hand, so there might be a mistake or two, but not this much. Add
> the features from 'virsh capabilities' reported with x86_64 libvirt to
> the SandyBridge-features and the features from i386 libvirt into
> n270-features and you'll see.
Thanks very much for clarifying.
>
>> me, is that the libvirt architecture need not be representative of the
>> virtualisation architecture. Apart from the issue of CPU detection, a
>> user should otherwise be fine running libvirt-bin:i386 with
>> qemu-system-x86_64:amd64 (which is our aim, leveraging Debian Multiarch
> It... should, yes. But given the current implementation there is
> nothing we can do about this. There's huge work being done by Jiri
> (from libvirt) and Eduardo (from qemu) making libvirt more aware of
> qemu CPUs. That should help in the future, but since they are
> experiencing many problems (and I'm considering only those which I
> know of, I'm sure they have more of them) it won't take place in very
> near future (just my guess).
Ok, thanks.
>
>> to run only the 64bit binaries that really need 64bit capability or
>> large RAM, while keeping the memory footprint of other binaries small).
>> Is there a workaround? Just customise the cpu_map.xml file?
>>
> You *should not* customize that. libvirt might suggest different
> CPUs, it will get overridden with updates etc. However, I cannot
> force you to keep the file that way...
>
> How about trying out the x32 ABI [1] [2]? Does Debian make something
> like even possible?
Thanks for these links. I'll check them out. But it sounds like the
easiest route forward is just to accept we have to run libvirt-bin:amd64
as well as qemu:amd64. That's not a huge extra burden, and something I
expect we can live with. And it's really good to understand why
libvirt-bin:i386 detects different CPU types.
Struan
>
> Martin
>
> [1] http://en.wikipedia.org/wiki/X32_ABI
> [2] https://sites.google.com/site/x32abi/
>
>> # diff -ubw /tmp/n270-features /tmp/SandyBridge-features
>> --- /tmp/n270-features 2014-03-03 14:05:02.003880511 +0000
>> +++ /tmp/SandyBridge-features 2014-03-03 14:04:56.063834563 +0000
>> @@ -1,25 +1,38 @@
>> + <feature name='aes'/>
>> <feature name='apic'/>
>> + <feature name='avx'/>
>> <feature name='clflush'/>
>> <feature name='cmov'/>
>> + <feature name='cx16'/>
>> <feature name='cx8'/>
>> <feature name='de'/>
>> <feature name='fpu'/>
>> <feature name='fxsr'/>
>> + <feature name='lahf_lm'/>
>> + <feature name='lm'/>
>> <feature name='mca'/>
>> <feature name='mce'/>
>> <feature name='mmx'/>
>> - <feature name='monitor'/>
>> <feature name='msr'/>
>> <feature name='mtrr'/>
>> <feature name='nx'/>
>> <feature name='pae'/>
>> <feature name='pat'/>
>> + <feature name='pclmuldq'/>
>> <feature name='pge'/>
>> <feature name='pni'/>
>> + <feature name='popcnt'/>
>> <feature name='pse'/>
>> + <feature name='pse36'/>
>> + <feature name='rdtscp'/>
>> <feature name='sep'/>
>> <feature name='sse'/>
>> <feature name='sse2'/>
>> + <feature name='sse4.1'/>
>> + <feature name='sse4.2'/>
>> <feature name='ssse3'/>
>> + <feature name='syscall'/>
>> <feature name='tsc'/>
>> - <feature name='vme'/>
>> + <feature name='tsc-deadline'/>
>> + <feature name='x2apic'/>
>> + <feature name='xsave'/>
>>>> Struan
>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>>> Struan
>>>>>>>>
>>>>>>>> On 03/03/2014 10:00, Martin Kletzander wrote:
>>>>>>>>> On Fri, Feb 28, 2014 at 03:45:01PM +0000, Struan Bartlett wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> On a range of Dell servers containing Intel 64bit processors, 'virsh
>>>>>>>>>> capabilities' reports the cpu differently on Debian Wheezy-amd64 and
>>>>>>>>>> Wheezy-i386. The results given by the Wheezy-i386 version seem very
>>>>>>>>>> wrong (since n270 is an Atom processor). Apart from architecture, the
>>>>>>>>>> package versions of libvirt-bin are identical: 1.2.1-1~bpo70+1.
>>>>>>>>>> /usr/share/libvirt/cpu_map.xml files are identical. Is this a known
>>>>>>>>>> issue? Details for one server are:
>>>>>>>>>>
>>>>>>>>>> # cat /proc/cpuinfo| head -n 26
>>>>>>>>>> processor : 0
>>>>>>>>>> vendor_id : GenuineIntel
>>>>>>>>>> cpu family : 6
>>>>>>>>>> model : 45
>>>>>>>>>> model name : Intel(R) Xeon(R) CPU E5-2650L 0 @ 1.80GHz
>>>>>>>>>> stepping : 7
>>>>>>>>>> microcode : 0x70d
>>>>>>>>>> cpu MHz : 1800.054
>>>>>>>>>> cache size : 20480 KB
>>>>>>>>>> physical id : 0
>>>>>>>>>> siblings : 16
>>>>>>>>>> core id : 0
>>>>>>>>>> cpu cores : 8
>>>>>>>>>> apicid : 0
>>>>>>>>>> initial apicid : 0
>>>>>>>>>> fpu : yes
>>>>>>>>>> fpu_exception : yes
>>>>>>>>>> cpuid level : 13
>>>>>>>>>> wp : yes
>>>>>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>>>>>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>>>>>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
>>>>>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64
>>>>>>>>>> monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1
>>>>>>>>>> sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat
>>>>>>>>>> xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
>>>>>>>>>> bogomips : 3600.10
>>>>>>>>>> clflush size : 64
>>>>>>>>>> cache_alignment : 64
>>>>>>>>>> address sizes : 46 bits physical, 48 bits virtual
>>>>>>>>>> power management:
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>> </proc/cpuinfo for processors 1..31 snipped here for brevity>
>>>>>>>>>>
>>>>>>>>> Check if all 32 CPUs have *exactly* the same flags, I remember an
>>>>>>>>> issue when we reported a wrong cpu because the probing code was
>>>>>>>>> scheduled on one of them which had one flag missing. If package and
>>>>>>>>> cpu_map.xml are the same, this is the only thing I can think of.
>>>>>>>>> Simple 'grep ^flags /proc/cpuinfo | sort -u' should do. If only one
>>>>>>>>> line is printed out than I don't know where the problem might be...
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>>> # Running Wheezy-amd64 libvirt-bin1.2.1-1~bpo70+1
>>>>>>>>>> # virsh capabilities
>>>>>>>>>>
>>>>>>>>>> <cpu>
>>>>>>>>>> <arch>x86_64</arch>
>>>>>>>>>> <model>SandyBridge</model>
>>>>>>>>>> <vendor>Intel</vendor>
>>>>>>>>>> <topology sockets='2' cores='8' threads='2'/>
>>>>>>>>>> <feature name='pdpe1gb'/>
>>>>>>>>>> <feature name='osxsave'/>
>>>>>>>>>> <feature name='dca'/>
>>>>>>>>>> <feature name='pcid'/>
>>>>>>>>>> <feature name='pdcm'/>
>>>>>>>>>> <feature name='xtpr'/>
>>>>>>>>>> <feature name='tm2'/>
>>>>>>>>>> <feature name='est'/>
>>>>>>>>>> <feature name='smx'/>
>>>>>>>>>> <feature name='vmx'/>
>>>>>>>>>> <feature name='ds_cpl'/>
>>>>>>>>>> <feature name='monitor'/>
>>>>>>>>>> <feature name='dtes64'/>
>>>>>>>>>> <feature name='pbe'/>
>>>>>>>>>> <feature name='tm'/>
>>>>>>>>>> <feature name='ht'/>
>>>>>>>>>> <feature name='ss'/>
>>>>>>>>>> <feature name='acpi'/>
>>>>>>>>>> <feature name='ds'/>
>>>>>>>>>> <feature name='vme'/>
>>>>>>>>>> </cpu>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # Running Wheezy-i386 libvirt-bin1.2.1-1~bpo70+1
>>>>>>>>>> # virsh capabilities
>>>>>>>>>>
>>>>>>>>>> <cpu>
>>>>>>>>>> <arch>x86_64</arch>
>>>>>>>>>> <model>n270</model>
>>>>>>>>>> <vendor>Intel</vendor>
>>>>>>>>>> <topology sockets='2' cores='8' threads='2'/>
>>>>>>>>>> <feature name='lahf_lm'/>
>>>>>>>>>> <feature name='lm'/>
>>>>>>>>>> <feature name='rdtscp'/>
>>>>>>>>>> <feature name='pdpe1gb'/>
>>>>>>>>>> <feature name='avx'/>
>>>>>>>>>> <feature name='osxsave'/>
>>>>>>>>>> <feature name='xsave'/>
>>>>>>>>>> <feature name='aes'/>
>>>>>>>>>> <feature name='tsc-deadline'/>
>>>>>>>>>> <feature name='popcnt'/>
>>>>>>>>>> <feature name='x2apic'/>
>>>>>>>>>> <feature name='sse4.2'/>
>>>>>>>>>> <feature name='sse4.1'/>
>>>>>>>>>> <feature name='dca'/>
>>>>>>>>>> <feature name='pcid'/>
>>>>>>>>>> <feature name='pdcm'/>
>>>>>>>>>> <feature name='xtpr'/>
>>>>>>>>>> <feature name='cx16'/>
>>>>>>>>>> <feature name='tm2'/>
>>>>>>>>>> <feature name='est'/>
>>>>>>>>>> <feature name='smx'/>
>>>>>>>>>> <feature name='vmx'/>
>>>>>>>>>> <feature name='ds_cpl'/>
>>>>>>>>>> <feature name='dtes64'/>
>>>>>>>>>> <feature name='pclmuldq'/>
>>>>>>>>>> <feature name='pbe'/>
>>>>>>>>>> <feature name='tm'/>
>>>>>>>>>> <feature name='ht'/>
>>>>>>>>>> <feature name='ss'/>
>>>>>>>>>> <feature name='acpi'/>
>>>>>>>>>> <feature name='ds'/>
>>>>>>>>>> <feature name='pse36'/>
>>>>>>>>>> </cpu>
>>>>>>>>>>
>>>>>>>>>> Kind regards
>>>>>>>>>>
>>>>>>>>>> Struan Bartlett
>>>>>>>>>>
>>>>>>>>>>
>>
--
Struan Bartlett
NewsNow Publishing Limited
Tel: +44 (0)845 838 8890
Fax: +44 (0)845 838 8898
The UK's #1 News Portal:
> www.NewsNow.co.uk <http://www.NewsNow.co.uk> (est. 1998)
Also tailored for Mobile:
> mobile.NewsNow.co.uk <http://mobile.NewsNow.co.uk/>
Now with FREE Personalisation:
> Register <http://www.NewsNow.co.uk/register/>
Bespoke B2B Internet News Monitoring:
> Internet News Monitoring
<http://www.newsnow.co.uk/services/newsmonitoring/>
Bespoke B2B Headlines for Websites:
> Editorial-In-A-Box <http://www.newsnow.co.uk/services/websites/>
NewsNow Publishing Limited, trading also as NewsNow.co.uk, is a company
registered in England and Wales under company no. 3435857 with
registered office The Euston Office, 1 Euston Square, 40 Melton Street,
London NW1 2FD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20140303/2b4de94c/attachment.htm>
More information about the libvirt-users
mailing list