[Avocado-devel] Running tests in parallel
Lukáš Doktor
ldoktor at redhat.com
Thu Nov 24 13:48:29 UTC 2016
Dne 23.11.2016 v 16:26 Lucas Meneghel Rodrigues napsal(a):
>
>
> On Wed, Nov 23, 2016 at 12:46 PM Zubair Lutfullah Kakakhel
> <Zubair.Kakakhel at imgtec.com <mailto:Zubair.Kakakhel at imgtec.com>> wrote:
>
> Hi,
>
> On 11/23/2016 02:28 PM, Cleber Rosa wrote:
> >
> > On 11/23/2016 07:07 AM, Zubair Lutfullah Kakakhel wrote:
> >> Hi,
> >>
> >> Thank you for your comprehensive reply!
> >>
> >> Comments inline.
> >>
> >> On 11/22/2016 02:11 PM, Cleber Rosa wrote:
> >>> On 11/22/2016 07:53 AM, Zubair Lutfullah Kakakhel wrote:
> >>>> Hi,
> >>>>
> >>>
> >>> Hi Zubair,
> >>>
> >>>> There are quite a few threads about this and a trello card
> >>>> https://trello.com/c/xNeR2slj/255-support-running-tests-in-parallel
> >>>>
> >>>> And the discussion leads to a complex multi-host RFC.
> >>>> https://www.redhat.com/archives/avocado-devel/2016-March/msg00025.html
> >>>>
> >>>> Our requirement is simpler.
> >>>> All we wanted to do is run disjoint simple (c executables) tests in
> >>>> parallel.
> >>>>
> >>>
> >>> Sounds fair enough.
> >>>
> >>>> I was wondering if somebody has a WIP branch that has some level of
> >>>> implementation for this?
> >>>
> >>> I'm not familiar with a WiP or PoC on this (yet). If anyone has
> >>> experimented with it, I'd happy to hear about it.
> >>>
> >>>> Or If somebody is familiar with the code base, I'd appreciate some
> >>>> direction on how to implement this.
> >>>>
> >>>
> >>> Avocado already runs every single test in a fresh new process. This is,
> >>> at least theoretically, a good start. Also, the test process is
> >>> handled based on the standard Python multiprocessing module:
> >>>
> >>> https://github.com/avocado-framework/avocado/blob/master/avocado/core/runner.py#L363
> >>>
> >>>
> >>> The first experimentation I'd do would be to attempt using the also
> >>> Python standard multiprocessing.Pool:
> >>>
> >>> https://docs.python.org/2.7/library/multiprocessing.html#using-a-pool-of-workers
> >>>
> >>
> >> In this case, there would be a separate python thread for each test
> >> being run in parallel.
> >> Each python thread would actually call the test executable using a
> >> sub-process?
> >>
> >
> > Ideally, the Avocado test runner would remain a single process, that is,
> > without one additional thread (or process) to manage each *test* process.
> >
> >> That can be OK for Desktops but won't scale well for using avocado in
> >> memory
> >> constrained Embedded devices.
> >>
> >
> > I must admit I haven't attempted to run Avocado in resource constrained
> > environments. Can you explain what is your bigger concern?
>
> In our case, primarily memory. Even for dormant processes. Although
> cpu usage is also
> a concern.
>
> Imagine running Avocado on a slightly beefy WiFi router with 128
> Mbytes of RAM.
> One python process is slow/difficult. Run a few python processes in
> parallel.
> And the Kernel Out of Memory killer starts killing processes.
>
> >
> > Do you feel that Avocado (as a single process test *runner*) plus one
> > process for each *test* is not suitable to those environments?
>
> Avocado should only be running one process ideally.
> And each test should be running 'only' its process.
>
> I think we've confused the dialogue with terminology.
> Threads/processes/subprocesses/multiprocessing
> I'll attempt to clarify.
>
> My current understanding of Avocado
>
> Avocado-runner parent process
> runs - > Avocado test thread using multiprocessing.Process here [1]
> run - > Actual test executable using subprocess here [2]
>
>
> We use subprocess only if the test is an executable. If it's an avocado
> 'instrumented' test we execute the test class main entry point
>
Actually it's `multiprocessing.Process`, which means it's forked process
and this applies for any type of test.
>
> Is this correct?
> Is there a particular purpose the runner starts a separate thread which
> actually calls the test executable?
>
>
> Yes, test isolation. We don't want tests modifying and messing with the
> main runner process in any capacity.
Exactly
>
>
> Now coming back to running tests in parallel.
>
> You mentioned using multiprocessing.Pool. In that case, there could be
> a potential issue for constrained devices.
> e.g. Running 4 tests in parallel.
>
> Avocado-runner parent process
> runs - > Avocado test thread using multiprocessing.Process here [1]
> run - > Actual test process using subprocess here [2]
> runs - > Avocado test thread using multiprocessing.Process here [1]
> run - > Actual test process using subprocess here [2]
> runs - > Avocado test thread using multiprocessing.Process here [1]
> run - > Actual test process using subprocess here [2]
> runs - > Avocado test thread using multiprocessing.Process here [1]
> run - > Actual test process using subprocess here [2]
>
> So 4 tests would actually result in 9 processes being created.
> 1 runner in Python
> 4 of them mostly dormant Python multiprocessing. (their purpose is a
> bit unclear)
> 4 actual executables.
>
>
> I hope the use of multiprocessing is clearer now.
>
>
> Ideally, that number should be 5 for running 4 tests.
> Avocado-runner parent process
> run - > Actual test process using subprocess here [2]
> run - > Actual test process using subprocess here [2]
> run - > Actual test process using subprocess here [2]
> run - > Actual test process using subprocess here [2]
>
>
> I suppose you could hack the entire test loader system to avoid
> multiprocessing in the case we want to run executables, but I'm not a
> big fan of the idea.
Actually when thinking about this card I wanted to do just that. My
intention was to modify the `avocado.core.runner` main loop to not just
run a test and monitor it's process, but to run a list of tasks
(structure which encapsulates the process, queue, timeout, ...) and it'd
poll in turns for status of all those tasks. It actually does not
require much changes, just a nice structure and one additional `for`.
As for the UI I had a WiP with support for multiple tests, unfortunately
I already forgot where it was. The idea was the Result stored the
started tests and it cycled through the active ones every 2s. On exit it
wrote `\r` + the full test name and status followed by `\n`. Therefor if
I skip the `\r` it'd like something like this
(1-2/3) sleeptest.py:SleepTest.test: \
(1-2/3) sleeptest.py:SleepTest.test: |
(2-2/3) sleeptest.py:SleepTest.test: /
(2-3/3) sleeptest.py:SleepTest.test: -
(2/3) sleeptest.py:SleepTest.test: PASS (1.01 s)
(1-3/3) sleeptest.py:SleepTest.test: \
(1/3) sleeptest.py:SleepTest.test: PASS (1.03 s)
(3/3) sleeptest.py:SleepTest.test: |
(3/3) sleeptest.py:SleepTest.test: PASS (1.01 s)
with the `\r` it'd only show:
(2/3) sleeptest.py:SleepTest.test: PASS (1.01 s)
(1/3) sleeptest.py:SleepTest.test: PASS (1.03 s)
(3/3) sleeptest.py:SleepTest.test: PASS (1.01 s)
(but during the execution you'd see all the above, it'd only be
rewritten with the next line)
it's quite simple, you only need to store the active tests, index and
update index+tests on started/finished test.
Anyway, the UI is the last concern... Important is to extract the
test-info and allow parallel execution. As it's python, it'll always be
memory-intensive, but as far as you use linux it should not be that bad
as fork uses copy on write, so the two processes should share most of
the memory. Anyway my machine has 16G so I can't really tell....
Looking forward to your patches and +1 for improved runner rather than
multiprocessing.Pool.
Lukáš
>
>
> I hope this doesn't look even more confusing :)
>
> Regards,
> ZubairLK
>
> [1]
> https://github.com/avocado-framework/avocado/blob/master/avocado/core/runner.py#L363
> [2]
> https://github.com/avocado-framework/avocado/blob/master/avocado/utils/process.py#L273
>
> >
> > - Cleber.
> >
> >> Please correct me if I am reading this incorrectly.
> >>
> >> Regards,
> >> ZubairLK
> >>
> >>>
> >>> This would most certainly lead to changes in how Avocado currently
> >>> serially waits for the test status:
> >>>
> >>> https://github.com/avocado-framework/avocado/blob/master/avocado/core/runner.py#L403
> >>>
> >>>
> >>> Which ultimately is added to the (Job wide) results:
> >>>
> >>> https://github.com/avocado-framework/avocado/blob/master/avocado/core/runner.py#L455
> >>>
> >>>
> >>> Since the results for many tests will now be acquired in unpredictable
> >>> order, this will require changes to the ResultEvent based plugins (such
> >>> as the UI).
> >>>
> >>>> Thanks
> >>>>
> >>>> Regards,
> >>>> ZubairLK
> >>>>
> >>>
> >>> I hope this is a good initial set of pointers. If you feel adventurous
> >>> and wants to start hacking on this, you're more then welcome.
> >>>
> >>> BTW: we've had quite a number of features that started as
> >>> experiments/ideas/not-really-perfect-pull-requests from the community
> >>> that Avocado "core team" members embraced and pushed all the way to
> >>> completeness.
> >>>
> >
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 502 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20161124/f4d3f4fa/attachment.sig>
More information about the Avocado-devel
mailing list