[Avocado-devel] [RFC PATCH 0/7] avocado: Add acceptance tests parsing the Linux boot console

[Philippe Mathieu-Daudé]

> Cross-posting qemu-devel + avocado-devel.
> 
>> While previously working on a Super I/O refactor, I encountered some problems
>> at runtime, after building the codebase successfully and running qtests.
>> I had to manually start to boot different guests and check the bootlog.
>>
>> I wanted to give a try at Avocado which seems designed to simplify that kind
>> of functional tests.
>>
>> I applied Amador Pahim work following Cleber Rosa documentation from
>> http://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg03891.html,
>> however I had to modify few things to parse the boot console.
>> Since his work is not merged, I included it in this series.
>>
>> The tests simply expect to find a string reported by Linux printk when a
>> device is detected/initialized, such "ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A"
>> and "i8042 KBD port at 0x60,0x64 irq 1" for the Super I/O chip, or such
>> "registered as PCnet/PCI II 79C970A" to confirms the PCI subsystem and network
>> device are correctly detected:
>>
>>
>>     self.assertIn(u'ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A', bootlog)
>>     self.assertIn(u'ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A', bootlog)
>>     self.assertIn(u'i8042 KBD port at 0x60,0x64 irq 1', bootlog)
>>     self.assertIn(u'i8042 AUX port at 0x60,0x64 irq 12', bootlog)
>>
>> Example of the tests output:
>>
>>     $ avocado run test_linux-boot-console.py -m test_linux-boot-console.py.data/parameters.yaml 
>>     JOB ID     : 695094c9bbe8f6011226da7c2031c2c53e949910
>>     JOB LOG    : /home/phil/avocado/job-results/job-2018-04-19T13.36-695094c/job.log
>>      (1/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;alpha-2582: PASS (4.76 s)
>>      (2/6) test_linux-boot-console.py:TestAlphaClipperBoot2_6.test_boot_console;mips-4a72: PASS (0.00 s)
>>      (3/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;alpha-2582: PASS (0.00 s)
>>      (4/6) test_linux-boot-console.py:TestMips4kcMaltaBoot2_6.test_boot_console;mips-4a72: PASS (3.92 s)
>>      (5/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;alpha-2582: PASS (0.00 s)
>>      (6/6) test_linux-boot-console.py:TestMips4kcMaltaBoot3_2.test_boot_console;mips-4a72: PASS (4.08 s)
>>     RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>>     JOB TIME   : 13.31 s
>>
>> Please apologize my ugly Python, this series is just a proof-of-concept :)
>> I couldn't figure out how to use the @skipUnless(correct arch) decorator.
> 
> Eduardo asked me to share my first impressions after trying this
> framework with QEMU.
> 
> So far it seems more designed to run tests _inside_ a qemu instance
> (think user-space), eventually testing packages/scripts.

Current examples use the same pattern, which I felt a bit restricting:

[QemuTest]
  .setUp: prepare VM arguments, launch the VM

  .test_*: run tests in the running VM

  .tearDown: shutdown the VM and cleanup dangling files

> My tests are targeting the machine model itself, if the devices are
> correctly instantiated and so.
> The fastest approach was to check the Linux kernel bootlog, but if you
> look at the TestAlphaClipperBoot console you'll see a boot "firmware" is
> executed before the kernel. Casually this firmware also send information
> on the console, but what if the console is not accessible?
> We can use a chardev for the ioport80 POST, but some firmwares post boot
> events via I2C, SPI, CAN...
> 
> To improve the testing, my idea is to use the Tracing framework.
> The machine trace events would be logged in a db and avogado would
> verify some of the trace events, did they occur? in the correct sequence
> order? with the correct arguments? We can also check relative
> timestamps, is this sequence timing fast enough?
> 
> I plan to use the gdb API to insert breakpoints and follow part
> execution flow, eventually injecting (faulty) events.
> 
> The multi-arch/machines support is a bit weak yet.
> 
> I'm personally more interested in automatically testing [real world]
> firmware or full machine boot process, the kind of integration testing I
> can not do with qtests.
> 
> I liked the possibility to generate coredumps, or the replay function.
> 
> I also liked to be able to write a test on how a machine boots, in ~20
> LOC (see TestAlphaClipperBoot2_6).
> 
> The storage API is OK to fetch a full VM image, but not a single file
> like a kernel or a flash image

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20180420/bc38bcb5/attachment.sig>