[Avocado-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console
Cleber Rosa
crosa at redhat.com
Wed May 2 16:22:23 UTC 2018
On 04/20/2018 12:17 AM, Philippe Mathieu-Daudé wrote:
> Cross-posting qemu-devel + avocado-devel.
>
>> While previously working on a Super I/O refactor, I encountered some problems
>> at runtime, after building the codebase successfully and running qtests.
>> I had to manually start to boot different guests and check the bootlog.
>>
>> I wanted to give a try at Avocado which seems designed to simplify that kind
>> of functional tests.
>>
>> I applied Amador Pahim work following Cleber Rosa documentation from
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
>> however I had to modify few things to parse the boot console.
>> Since his work is not merged, I included it in this series.
>>
>> The tests simply expect to find a string reported by Linux printk when a
>> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A"
>> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
>> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
>> device are correctly detected:
>>
>>
>> self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>> self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>> self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>> self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
>>
>> Example of the tests output:
>>
>> $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml
>> JOB ID : 695094c9bbe8f6011226da7c2031c2c53e949910
>> JOB LOG : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>> (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s)
>> (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s)
>> (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s)
>> (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s)
>> (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s)
>> (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s)
>> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>> JOB TIME : 13.31 s
>>
>> Please apologize my ugly Python, this series is just a proof-of-concept :)
>> I couldn't figure out how to use the @skipUnless(correct arch) decorator.
>
> Eduardo asked me to share my first impressions after trying this
> framework with QEMU.
>
> So far it seems more designed to run tests _inside_ a qemu instance
> (think user-space), eventually testing packages/scripts.
>
The first tests written indeed involved interacting with the guest OS
more often than not. I tried to show one example that did not envolve a
guest here:
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03468.html
Just as short exercise, this is another example (sorry for the lack of
creativity):
class CpuSmp(test.QemuTest):
"""
:avocado: enable
"""
def test(self):
smp_max = int(self.params.get('smp_max', default=255))
self.vm._args.extend(['-smp', '%s' % smp_max])
self.vm.launch()
res = self.vm.qmp('query-cpus-fast')
self.assertEqual(len(res[u'return']), smp_max)
> My tests are targeting the machine model itself, if the devices are
> correctly instantiated and so.
> The fastest approach was to check the Linux kernel bootlog, but if you
> look at the TestAlphaClipperBoot console you'll see a boot "firmware" is
> executed before the kernel. Casually this firmware also send information
> on the console, but what if the console is not accessible?
> We can use a chardev for the ioport80 POST, but some firmwares post boot
> events via I2C, SPI, CAN...
>
> To improve the testing, my idea is to use the Tracing framework.
> The machine trace events would be logged in a db and avogado would
> verify some of the trace events, did they occur? in the correct sequence
> order? with the correct arguments? We can also check relative
> timestamps, is this sequence timing fast enough?
>
Right, I think I see where you're going here.
For the record, the Avocado test runner has the ability to check the
test output against a previously recorded file, and that can determine
the test status. Maybe a second implementation of the same concept can
be applied to your idea.
> I plan to use the gdb API to insert breakpoints and follow part
> execution flow, eventually injecting (faulty) events.
>
Just to be clear, do you mean the Avocado GDB API?
> The multi-arch/machines support is a bit weak yet.
>
Yes. One of the ways I believe the multi-arch/machines support can be
significantly improved is by using some level of parameter
standardization, together with tools to generate the intended test
"matrix". On a quick experiment with "pict" (an indepedent
combinatorial testing tool) on this matter, I came up with:
---
arch: x86_64, ppc64, ppcemb, ppc, aarch64, alpha, s390x
machine_type: pc, q35, pseries-2.12, virt, 40p, bamboo, g3beige, mac99,
mpc8544ds, ppce500, prep, sam460ex, clipper, s390-ccw-virtio-2.12
machine_accel: kvm, tcg
smp_max: 1, 4, 8, 15, 248, 255
IF [arch] = "x86_64" THEN
[machine_type] = "q35" OR
[machine_type] = "pc";
IF [arch] = "ppc" THEN
[machine_type] = "40p" OR
[machine_type] = "bamboo" OR
[machine_type] = "g3beige" OR
[machine_type] = "mac99" OR
[machine_type] = "mpc8544ds" OR
[machine_type] = "ppce500" OR
[machine_type] = "prep" OR
[machine_type] = "sam460ex";
IF [arch] = "alpha" THEN [machine_type] = "clipper";
IF [arch] = "ppc64" THEN [machine_type] = "pseries-2.12";
IF [arch] = "ppcemb" THEN [machine_type] = "bamboo";
IF [arch] = "aarch64" THEN [machine_type] = "virt";
IF [arch] = "s390x" THEN [machine_type] = "s390-ccw-virtio-2.12";
# On my machine, only x86_64 can do KVM
IF NOT [arch] = "x86_64" THEN [machine_accel] = "tcg";
IF [machine_type] = "q35" THEN [smp_max] = 255;
IF [machine_type] = "pc" THEN [smp_max] = 255;
IF [machine_type] = "s390-ccw-virtio-2.12" THEN [smp_max] = 248;
IF [machine_type] = "virt" THEN [smp_max] = 8;
IF [machine_type] = "mpc8544ds" THEN [smp_max] = 15;
IF [machine_type] = "clipper" THEN [smp_max] = 4;
IF [machine_type] = "40p" THEN [smp_max] = 1;
IF [machine_type] = "bamboo" THEN [smp_max] = 1;
IF [machine_type] = "g3beige" THEN [smp_max] = 1;
IF [machine_type] = "mac99" THEN [smp_max] = 1;
IF [machine_type] = "sam460ex" THEN [smp_max] = 1;
IF [machine_type] = "ppce500" THEN [smp_max] = 1;
IF [machine_type] = "prep" THEN [smp_max] = 1;
---
With this *very partial* definition of the arch/machine support in QEMU,
a single test can be run in 22 different scenarios. I've used it to run
the 4 VNC tests and the one CpuSmp test printed above, resulting in 110
different test executions:
https://cleber.fedorapeople.org/results.html
The first three parameters are already supported by avocado_qemu, and
more can follow.
And to be fully honest, "avocado_qemu" has more weaknesses than
strengthens at the moment. That's partially because it's so young (and
immature), but, can be made into our ideal grown "being".
> I'm personally more interested in automatically testing [real world]
> firmware or full machine boot process, the kind of integration testing I
> can not do with qtests.
>
> I liked the possibility to generate coredumps, or the replay function.
>
> I also liked to be able to write a test on how a machine boots, in ~20
> LOC (see TestAlphaClipperBoot2_6).
>
> The storage API is OK to fetch a full VM image, but not a single file
> like a kernel or a flash image.
>
As pointed elsewhere, fetching/caching is probably OK at this point.
But handling the (optional) gzip compression can use some TLC.
> Enough for my second try, good work :)
>
> I hope this was helpful.
>
> Regards,
>
> Phil.
>
Thanks for the review!
- Cleber.
More information about the Avocado-devel
mailing list