[Avocado-devel] New questions

Wed Sep 14 17:06:55 UTC 2016

Dne 14.9.2016 v 18:46 Marcos E. Matsunaga napsal(a):
> Hi Lucas,
>
> Thanks for your answers. I have some comments below. And forgive me if I
> am repeating something.
>
>
> On 14/09/16 16:59, Lucas Meneghel Rodrigues wrote:
>>
>>
>> On Wed, Sep 14, 2016 at 8:32 AM Marcos E. Matsunaga
>> <Marcos.Matsunaga at oracle.com <mailto:Marcos.Matsunaga at oracle.com>> wrote:
>>
>>     Hi Folks,
>>
>>     I have some questions about how avocado works.
>>
>>     1. If I run avocado and give it a directory that has all tests. Is
>>     there
>>     a way to specify the order of execution? I mean, if I name the files
>>     001-xxx.py, 010-aa.py, will it execute 001-xxx.py before 010-aa.py
>>     or it
>>     doesn't follow an alphabetical order?
>>
>>
>>
>> There is - You can specify their order of execution in the command line:
>>
>> avocado run failtest.py raise.py doublefree.py
>> JOB ID     : 6047dedc2996815659a75841f00518fa0f83b1ee
>> JOB LOG    :
>> /home/lmr/avocado/job-results/job-2016-09-14T12.53-6047ded/job.log
>> TESTS      : 3
>>  (1/3) failtest.py:FailTest.test: FAIL (0.00 s)
>>  (2/3) raise.py:Raise.test: PASS (0.11 s)
>>  (3/3) doublefree.py:DoubleFreeTest.test: PASS (1.02 s)
>> RESULTS    : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
>> TESTS TIME : 1.13 s
>> JOB HTML   :
>> /home/lmr/avocado/job-results/job-2016-09-14T12.53-6047ded/html/results.html
> Yeah.. That would work if you want to run just a set of the tests, but
> think about a specific test that has hundreds of individual tests. If
> you can just give the directory and it sort it alphabetically before
> executing, that would be great. And with multiplex auto discovery, that
> would be even better.
As I mentioned in my response, currently it's not ordered, but we can 
discuss and make it alphabetical. It's just a matter of adding one 
`sorted` into a proper place (as it's iterative it'd be multiple-line 
fix, but still relatively simple). To not forget let's track it here:

https://trello.com/c/whLgcvtO/827-consider-sorting-the-fileloader-s-test-discovery

>>
>>
>>
>>     2. Lets take into consideration that same directory. Some of the
>>     scripts
>>     will have multiplex configuration files. Does avocado
>>     automatically look
>>     at some specific directory for those multiplex configuration
>>     files? I've
>>     tried to add them to the data, cfg and even the <script>.data
>>     directories, but it seems that it doesn't look for them automatically,
>>     only when I specify the option --multiplex, but then, the file will be
>>     used by all scripts and I was only able to specify a single
>>     multiplex file.
>>
>>
>> The original design assumption was that you'd execute only one test
>> that has a multiplex file, and provide the multiplex file with it, so
>> indeed what you wan't to do can't be done right now. I suppose
>> multiple tests with multiplex files and multiplex file auto detection
>> would be a nice feature to add moving forward, though.
> With a small standard definition, I think multiplex auto-discovery could
> be simple to implement. Let's say you have a <some dir>/test.py and when
> you execute it, check under <some dir>/test.py.data for a test.yaml. If
> it is there, just open it and use.  And, of course, you could add a
> default directory for multiplex files in avocado.conf and follow some
> hierarchy.
>
> Speaking of multiplex, when I was thinking about starting parallel jobs
> on different hosts, I came up with something like this in a multplex file:
>   1
> perf:
>
>   2
> SetUp:
>   3                 packages = 'perf
> netperf'
>   4
> server:
>   5                 hostname =
> 'perf1'
>   6                 ipaddress =
> '10.196.50.101'
>   7                 userid =
> 'root'
>   8
> client:
>   9                 hostname =
> 'perf2'
>  10                 ipaddress =
> '10.196.50.102'
>  11                 userid =
> 'root'
>
> I tried every combination I could to try to retrieve the two hostnames,
> but I was never able to do it. Wouldn't be a good idea to be able to
> retrieve the information from the multiplex file as
> "self.params.get('/perf/server/hostname')" ? That way, you could list
> all your remote clients (in this case) and be able to do whatever
> necessary to run jobs in parallel.
>
Running the jobs in parallel is tricky. It always depends on what do you 
want to do and where and creating one universal method is near to 
impossible (it'd be very complex with lots of options). Let's wait for 
the Job API and custom runner scripts to allow such behavior.

To demonstrate the complexity, you might want to run jobs in parallel:

1. the same test on multiple machines
2. use queues and distribute the tests to the machines in any order
3. create a server and run several clients
...

>>
>>
>>     3. I tried to find if it was possible to start multiple test processes
>>     in parallel, but it seems that avocado doesn't have anything like
>>     that.
>>     Lets say I have 4 guests and I want to execute performance tests while
>>     loading the 4 guests and I want to start tests on all 4 guests at the
>>     same time. It doesn't have a feature that will do that, right?
>>
>>
>> Executing tests in multiple remote machines was something that we
>> din't think about either, and one that would be very interesting to
>> add indeed.
> Yes.. That would be really interesting.
>>
>> What you can do for now is to start 4 parallel separate instances of
>> avocado, one for each machine. Each execution will create its own job
>> directory, though.
> I don't know if you are familiar with OpenMPI, but it does exactly that.
> It manage to start tests simultaneously on many hosts (it doesn't have
> to be guests). I think a functionality similar to that would be great.
> Avocado already copy the job to the remote host and retrieve the result.

The current limitation is that the remote runner runs full jobs and 
there is currently no way to merge results of the jobs. With the job 
APIs this problem should be addressed so let's wait for that. Also note 
that the Job API is very complex task and probably will be implemented 
incrementally. I'd not expect it to be ready in one month...

>>
>>
>>     Thanks for your time and help.
>>
>>
>> Cheers!
Regards,
Lukáš

>
> --
>
> Regards,
>
> Marcos Eduardo Matsunaga
>
> Oracle USA
> Linux Engineering
>
> “The statements and opinions expressed here are my own and do not
> necessarily represent those of Oracle Corporation.”
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 502 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20160914/2cd5c199/attachment.sig>