[Avocado-devel] Parameter System Overhaul

Wed Aug 16 22:03:58 UTC 2017

On 08/16/2017 04:34 PM, Ademar Reis wrote:
> On Wed, Aug 16, 2017 at 02:28:47PM -0400, Cleber Rosa wrote:
>>
>> It's not covered by the fallback mechanism.  What I mean is that Avocado
>> should probably have, as one of the test parameter system providers
>> which are enabled by default, one that looks for parameters in the test
>> data dir.  Something like:
>>
>>  1. test => examples/tests/sleeptest.py
>>  2. test's datadir (current concept) => examples/tests/sleeptest.py.data
>>  3. test's file-based param => examples/tests/sleeptest.py.data/params
>>
>> The code handling line 3, would be one of the test parameter system
>> providers.  Just one that is enabled by default.
> 
> OK, this makes sense to me and is aligned with what I propose, we're
> in sync.
> 

OK.

>>
>>>>
>>>>> Users who don't want any specificity and/or have a small test base
>>>>> with a low chance of clashes could simply ignore the prefix both
>>>>> when creating parameters and when making calls to params.get().
>>>>>
>>>>
>>>> Yes.
>>>>
>>>>>>
>>>>>> The second item will probably mean the definition of a new class to
>>>>>> the ``avocado.core.plugin_interfaces`` module, together with a new
>>>>>> dispatcher(-like) implementation in ``avocado.core.dispatcher``.
>>>>>>
>>>>>> Parameters availability: local .vs. external
>>>>>> ============================================
>>>>>>
>>>>>> Right now, all parameters are given to the test at instantiation time.
>>>>>> Let's say that in this scenario, all parameters are *local* to the
>>>>>> test.  Local parameters have the benefit that the test is self
>>>>>> contained and doesn't need any external communication.
>>>>>>
>>>>>> In theory, this is also a limitation, because all parameters must be
>>>>>> available before the test is started.  Even if other parameter system
>>>>>> implementations are possible, with a local approach, there would be a
>>>>>> number of limitations.  For long running tests, that may depend on
>>>>>> parameters generated during the test, this would be a blocker.  Also,
>>>>>> if a huge number of parameters would be available (think of a huge
>>>>>> cloud or database of parameters) they would all have to be copied to
>>>>>> the test at instantiation time.  Finally, it also means that the
>>>>>> source of parameters would need to be able to iterate over all the
>>>>>> available parameters, so that they can be copied, which can be a
>>>>>> further limitation for cascading implementations.
>>>>>>
>>>>>> An external approach to parameters, would be one that the test holds a
>>>>>> handle to a broker of parameter providers.  The parameter resolution
>>>>>> would be done at run time.  This avoids the copying of parameters, but
>>>>>> requires runtime communication with the parameter providers.  This can
>>>>>> make the test execution much more fragile and dependent on the external
>>>>>> communication.  Even by minimizing the number of communication
>>>>>> endpoints by communicating with the test runner only, it can still add
>>>>>> significant overhead, latency and point of failures to the test
>>>>>> execution.
>>>>>>
>>>>>> I believe that, at this time, the limitations imposed by local
>>>>>> parameter availability do *not* outweigh the problems that an external
>>>>>> approach can bring.  In the future, if advanced use cases require an
>>>>>> external approach to parameters availability, this can be reevaluated.
>>>>>
>>>>> If I understand your point correctly, this is an implementation
>>>>> detail. It depends on what the "contract" is between the test runner
>>>>> (parameter provider) and the test being run (the user of
>>>>> params.get()).
>>>>>
>>>>
>>>> It is an implementation detail, but one that will ultimately reflect on
>>>> what is available to test writers.
>>>>
>>>>> For example, should users assume parameters are dynamic and can
>>>>> change during the lifetime of a test, an therefore two identical
>>>>> calls to params.get() might return different values?  Should it be
>>>>> possible to change params (something like params.put()) at runtime?
>>>>>
>>>>> (IMO the answer should be no to both questions).
>>>>>
>>>>> If you have something different in mind, then it would be
>>>>> interesting to see some real use-cases.
>>>>>
>>>>
>>>> Defining what users could expect is what I was trying to get to, but you
>>>> were way more direct here.  I mentioned that iterating through the
>>>> available parameters wouldn't be possible with the "external" approach,
>>>> because test writers would be affected by those choices.
>>>>
>>>> When I said that the "limitations imposed by local parameter
>>>> availability do *not* outweigh the problems that an external approach
>>>> can bring", I basically stood by the "local parameter availability"
>>>> approach, which mean to users that it'd be possible to:
>>>>
>>>>  1) Iterate through parameters
>>>>  2) Change parameters
>>>>  3) Remove parameters
>>>>  4) Add parameters
>>>>
>>>> Now, it's not because it's possible that it should be supported or
>>>> encouraged.  For the record, #1 is already possible with the current
>>>> implementation:
>>>>
>>>> http://avocado-framework.readthedocs.io/en/53.0/api/core/avocado.core.html?highlight=avocadoparams#avocado.core.varianter.AvocadoParams.iteritems
>>>>
>>>> And so is, to some extent #4 (and depending on how you look at it, #2):
>>>>
>>>> http://avocado-framework.readthedocs.io/en/53.0/api/core/avocado.core.html?highlight=avocadoparams#avocado.core.varianter.Varianter.add_default_param
>>>>
>>>
>>> Right. But these are avocado.core APIs (implementation details) that
>>> should not be exposed or supported.
>>>
>>
>> The second one is not exposed to test writers, the first one is.
>>
>>> I've been assuming params.get() is the only params-related API
>>> exposed to test writers. Is that correct?
>>>
>>
>> No, all the methods in AvocadoParams are available:
>>
>> http://avocado-framework.readthedocs.io/en/53.0/api/core/avocado.core.html?highlight=avocadoparams#avocado.core.varianter.AvocadoParams
>>
>> So it's possible to do things like:
>>
>>  def test():
>>     for o, k, v in self.params.iteritems():
>>        self.log.debug("%s %s %s", o, k, v)
>>
>> Now, some clarifications: I got confused and pasted the wrong link to
>> add_default_param().  That is a Varianter method, so it's *not*
>> available to test writers.  I got confused because I was staring at the
>> AvocadoParams __init__ parameter "default_params".  And and instance of
>> AvocadoParams is what you get as the "self.params" on a test.
>>
>> Hopefully that last paragraph is not too confusing.
> 
> You once again linked to avocado.core documentation, but the object
> is available to test writers. I imagine you understand we have a
> leak here: an internal object is being exposed as part of the API to
> test writers.
> 

I did link to the "avocado.core" documentation because an instance of
that (internal only) class is what users currently have access to:

 >>> from avocado import Test
 >>> class MyTest(Test):
       def test(self):
           pass
 >>> test = MyTest()
 >>> test.params
 <AvocadoParams *: * ([]),0: /test/* ([])>

> So let me change my question:
> 
> I've been assuming params.get() is the only params-related API we
> *want* (or plan) to expose to test writers. Is that correct?
> 

That's correct.

And that's why I kept bringing back the discussion about params.iteritems().

> Thanks.
>    - Ademar
> 

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/avocado-devel/attachments/20170816/9dc12406/attachment.sig>