[python-libvirt] RFC:PEP 484 type annotation - how to continue

[Daniel P. Berrangé]

On Mon, Oct 05, 2020 at 09:12:26AM +0200, Philipp Hahn wrote:
> Hello,
> 
> currently my Merge Quests
> <https://gitlab.com/libvirt/libvirt-python/-/merge_requests/9> is stuck
> because there are some open technical discussions:
> 
> 1. The Python binding consists of two layers:
>    - libvirtmod* is the low-level C-like API, which mostly is just a
> thin wrapper around the C-library. It is generated from the API.xml files
>   - The high-level libvirt* modules, providing an object oriented API,
> which is also generated from the API.xml files.
>   In both cases there are many exceptions defines via
> libvirt*-override-api.xml or coded in generator.py:
>   - skip_impl: Generate Python code, but skip C implementation
>   - custom_function: Generate neither C nor Python code - hand crafted code
>   - skip_function: Generate neither C nor Python code - not used at all
>   - function_skip_python_impl: Generate C code, but skip python impl
> 
> Previously the functions listed in "skip_impl" were also duplicated in
> the "libvirt-*override-api.xml" files, which is somehow hard to
> maintain. With d19699e48ce9c17481f94b4dc0715e18e05d5adb I removed all
> those names from "skip_impl", which are also listed in the overrides and
> added code to add them back when "generator.py" is invoked.
> For this to work I had to change several "files" attributes to start
> with "python", which looks okay to me, but still would like to get verified.

The generator.py is such a big hack I don't think it matters too much.
The real important thing is that the resulting code is sane.

> 
> 2. The low-level library works mostly with bytes, but the high-level
> Python API uses nested Lists, Tuples and Dicts. To get most out of the
> Python static type annotations the input and output parameters should be
> modled in as much detail as possible.
> This requires to store those types somewhere. My current
> 46592e28124858ed863bceba65c3dafa2bbb02cd adds some often used types to
> generator.py and then references them from the overrides.xml:
> 
> > +    'PyListSecret': ('', 'List["virSecret"]', '', ''),
> > +    'PyDictAny': ('', 'Dict[str, Any]', '', ''),
> 
> > +      <return type='PyListSecret' info='the list of secrets or None in case of error'/>
> 
> This still is very painful as there are many "used-once" types, which
> still must be declared and used in two places.
> So far the "return" type was used for the "C" type and currently I
> mis-use it for referencing it for my Python types. This doesn't look
> right, so I would prefer adding a new attribute which can be used to
> override the Python type.
> This also would probably nicely fit with functions like
> "virDomainOpenConsole", when "dev_name" is optional and can be "None",
> which the current type hint requires a "string".
> 
> Is extending the libvirt*-override-api.xml files okay or must this be
> coordinated with the other parts of the project?

These files follow the same schema as the main api.xml files shipped by
libvirt itself. That is just a matter of convenience though - the
overrides are only used in the libvirt-python generator.py, so there
is no reason why we can't record extra information in them.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|