<div dir="ltr">If it possible to fix a typo in the commit message while submitting it will be great:<div> </div><div>SLES4 -> SLES11 SP4 </div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 20, 2020 at 3:59 PM Daniel P. Berrangé <<a href="mailto:berrange@redhat.com">berrange@redhat.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Apr 20, 2020 at 03:37:16PM +0300, Sam Eiderman wrote: > The python3 bindings create unicode objects from application strings > on the guest (i.e. installed rpm, deb packages). > It is documented that rpm package fields such as description should be > utf8 encoded - however in some cases they are not a valid unicode > string, on SLES11 SP4 the following packages fail to be converted to > unicode using guestfs_int_py_fromstring() (which invokes > PyUnicode_FromString()): > > PackageKit > aaa_base > coreutils > dejavu > desktop-data-SLED > gnome-utils > hunspell > hunspell-32bit > hunspell-tools > libblocxx6 > libexif > libgphoto2 > libgtksourceview-2_0-0 > libmpfr1 > libopensc2 > libopensc2-32bit > liborc-0_4-0 > libpackagekit-glib10 > libpixman-1-0 > libpixman-1-0-32bit > libpoppler-glib4 > libpoppler5 > libsensors3 > libtelepathy-glib0 > m4 > opensc > opensc-32bit > permissions > pinentry > poppler-tools > python-gtksourceview > splashy > syslog-ng > tar > tightvnc > xorg-x11 > xorg-x11-xauth > yast2-mouse > > Fix this by globally changing guestfs_int_py_fromstring() > and guestfs_int_py_fromstringsize() to decode utf-8 with the "replace" > error handler: > > <a href="https://docs.python.org/3/library/codecs.html#error-handlers" rel="noreferrer" target="_blank">https://docs.python.org/3/library/codecs.html#error-handlers</a> > > For example, this will decode PackageKit's description on SLES4 the > following way: > > Backend: pisi > S.�ağlar Onur <<a href="mailto:caglar@pardus.org.tr" target="_blank">caglar@pardus.org.tr</a>> > > Signed-off-by: Sam Eiderman <<a href="mailto:sameid@google.com" target="_blank">sameid@google.com</a>> > --- > python/handle.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/python/handle.c b/python/handle.c > index 2fb8c18f0..427424707 100644 > --- a/python/handle.c > +++ b/python/handle.c > @@ -387,7 +387,7 @@ guestfs_int_py_fromstring (const char *str) > #if PY_MAJOR_VERSION < 3 > return PyString_FromString (str); > #else > - return PyUnicode_FromString (str); > + return PyUnicode_Decode(str, strlen(str), "utf-8", "replace"); > #endif > } > > @@ -397,7 +397,7 @@ guestfs_int_py_fromstringsize (const char *str, size_t size) > #if PY_MAJOR_VERSION < 3 > return PyString_FromStringAndSize (str, size); > #else > - return PyUnicode_FromStringAndSize (str, size); > + return PyUnicode_Decode(str, size, "utf-8", "replace"); > #endif > } Reviewed-by: Daniel P. Berrangé <<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>> Regards, Daniel -- |: <a href="https://berrange.com" rel="noreferrer" target="_blank">https://berrange.com</a> -o- <a href="https://www.flickr.com/photos/dberrange" rel="noreferrer" target="_blank">https://www.flickr.com/photos/dberrange</a> :| |: <a href="https://libvirt.org" rel="noreferrer" target="_blank">https://libvirt.org</a> -o- <a href="https://fstop138.berrange.com" rel="noreferrer" target="_blank">https://fstop138.berrange.com</a> :| |: <a href="https://entangle-photo.org" rel="noreferrer" target="_blank">https://entangle-photo.org</a> -o- <a href="https://www.instagram.com/dberrange" rel="noreferrer" target="_blank">https://www.instagram.com/dberrange</a> :| </blockquote></div>