[Avocado-devel] Implement fetch-assets command line

[Cleber Rosa]

On Tue, Sep 24, 2019 at 04:42:01PM -0300, Willian Rampazzo wrote:
> Hello everyone,
> 
> I started working on Trello card
> https://trello.com/c/T3SC1sZs/1521-implement-fetch-assets-command-line
> as part of a broader card,
> https://trello.com/c/CKP7YS6G/1481-on-cache-check-for-asset-fetcher
> and I would like to bring my findings to a discussion.
> 
> One way to implement that would be to parse the test source looking
> for the fetch_asset call and execute it. In theory, it is a straight
> forward implementation.
> 
> I have started working in the parser. After some simple tests, some
> complex situations started to show up. Let me bring an existing
> example, examples/tests/assets.py:
> 
>     def setUp(self):
>         mirrors = ['https://mirrors.peers.community/mirrors/gnu/hello/',
>                    'https://mirrors.kernel.org/gnu/hello/',
>                    'http://gnu.c3sl.ufpr.br/ftp/',
>                    'ftp://ftp.funet.fi/pub/gnu/prep/hello/']
>         hello = 'hello-2.9.tar.gz'
>         hello_locations = ["%s/%s" % (loc, hello) for loc in mirrors]
>         hello_sig = 'hello-2.9.tar.gz.sig'
>         hello_sig_locations = ["%s/%s" % (loc, hello_sig) for loc in mirrors]
>         self.hello = self.fetch_asset(
>             name=hello,
>             locations=hello_locations)
>         self.hello_sig = self.fetch_asset(
>             name=hello_sig,
>             asset_hash='f3b9fae20c35740004ae7b8de1301836dab4ac30',
>             locations=hello_sig_locations)
> 
> When the parser finds the fetch_asset call, it needs to inspect its
> arguments, looking for variables. If any variable is found, it needs
> to walk back the code looking for its assignment, so it is able to
> execute it prior to the fetch_asset execution. If the variables
> consist of other variables, the parser, again, needs to walk back in
> the code looking for those assignments, and so on.
> 
> There are countless ways of creating the right side of a variable
> assignment, from a single string assignment to a function that builds
> the content, or conditional assignments.
> 
> My initial idea is to cover variables consisting of other variables
> and list comprehension, just like in the example. For now, other
> complex constructions would be out of this implementation.
>

I would actually start with something even simpler, that is, only
consider the fetch_asset() calls with literals as the `name`
parameter, and with `name` being a URL (that is, no locations are
given or considered).

If that works, we can have an example for a tool that checks/downloads
those, and increase the coverage later (to include, as you put it
"variables consisting of other variables and list comprehension").

> Any comment, concern, suggestion, is appreciated here as it would help
> to build a more robust code.
>

I consider this type of parsing a kind of "magic" (hopefully the good
kind).  I'd try the "bunny from a hat" trick before attempting the
"sawing a human" in half trick.

Cheers,
- Cleber.

> Best regards,
> 
> Willian Rampazzo
> Software Engineer
> Red Hat Brazil
> 2717 337F 7E4A 5FDF
>