[libvirt] [PATCH 0/5] Improve translation handling

Daniel P. Berrangé berrange at redhat.com
Wed Apr 18 16:15:23 UTC 2018


Ping, i'd like to get this merged before we get close to the
freeze so we have debugging time if something unexpected
happens...

On Thu, Apr 12, 2018 at 02:28:17PM +0100, Daniel P. Berrangé wrote:
> This series aims to improve the way we handle translations.
> 
> The key problems with our current approach
> 
>  - The pot & po files stored in GIT contain huge set of
>    annotations about source file names & line numbers.
>    These are out of date as soon as a change is commited
>    to git following a translation refresh. This makes
>    diffs impossible to meaningfully review, as they are
>    98% noise, 2% signal.
> 
>  - The po file messages are sorted by source location,
>    so when we move code between files, or rename files,
>    the po file message order changes for no good reason.
>    This makes diffs even more impossible to review.
> 
>  - The po files contain entries for all messages even
>    if most have no translation, bloating size of po/
>    data stored in git
> 
>  - Whenever 'make dist' is run, it alters all the pot
>    and po files, so developers need to then reset their
>    content to match git HEAD manually. This is caused
>    by having auto-generated content (source file locations)
>    mixed in with the static content (the actual translated
>    strings)
> 
> Overall the *.po  files we store consume 100MB, and when
> I refresh content from Zanata, it results in git commits
> that are many 10's of MB in size and impossible todo any
> meaningful review on.
> 
> After this series, we only minimized po files in git, with
> the redundated & outdated source locations info stripped.
> This stripped info is re-added automatically during build
> to create the real .po files, that we distribute, and/or
> upload to translators in Zanata.
> 
> As a result the po directory is only 19MB in size, and
> when refreshing from Zanata, we have git commits that
> clearly show *only* the altered translations, nothing
> else. The importance of this cannot be overstated - by
> having these clear diffs I discovered a serious bug in
> the Zanata client that has been screwing up translations
> in every project that uses Zanata by adding bogus "fuzzy"
> annotations.
> 
> Note that what I've posted here is a cut-down version of
> what will actually be pushed. I have cut out the 99 other
> patches that actally change the .po files. There's no
> meaningful way to review those commits because of the
> noise in the original .po files we're replacing. I have
> gone through manual steps, however, to validate that the
> .po files we generate from the new .mini.po files are
> identical to the original .po files we used to store,
> aside from the bogus fuzzy annotations due to broken
> Zanata.
> 
> The diffstat shown below covers the full set of patches
> on the branch at:
> 
>    https://github.com/berrange/libvirt/commits/i18n-4
> 
> The killer statistic is:
> 
>    418787 insertions(+), 4388281 deletions(-)
> 
> Daniel P. Berrangé (5):
>   po: provide custom make rules for po file management
>   po: remove language list from zanata configuration
>   po: add rules for integration with zanata
>   po: stop storing libvirt.pot in git
>   po: minimize & canonicalize translations stored in git
> 
>  .gitignore                   |     9 +-
>  ABOUT-NLS                    |     1 +
>  autogen.sh                   |     3 +-
>  bootstrap.conf               |    38 -
>  build-aux/minimize-po.pl     |    37 +
>  configure.ac                 |    33 +-
>  m4/virt-nls.m4               |    70 +
>  po/Makefile.am               |   107 +
>  po/{POTFILES.in => POTFILES} |     0
>  po/README.md                 |    75 +
>  po/af.mini.po                |    20 +
>  po/af.po                     | 46395 ------------------------------
>  po/am.mini.po                |    20 +
>  po/am.po                     | 46395 ------------------------------
>  po/anp.mini.po               |    20 +
>  po/anp.po                    | 46395 ------------------------------
>  po/ar.mini.po                |   762 +
>  po/ar.po                     | 46685 ------------------------------
>  po/{as.po => as.mini.po}     | 54246 ++++++++++-------------------------
>  po/ast.mini.po               |    20 +
>  po/ast.po                    | 46395 ------------------------------
>  po/bal.mini.po               |    20 +
>  po/bal.po                    | 46395 ------------------------------
>  po/be.mini.po                |    21 +
>  po/be.po                     | 46396 ------------------------------
>  po/bg.mini.po                |  1204 +
>  po/bg.po                     | 46733 ------------------------------
>  po/bn.mini.po                |   761 +
>  po/bn.po                     | 46684 ------------------------------
>  po/bn_IN.mini.po             |  9131 ++++++
>  po/bn_IN.po                  | 46729 ------------------------------
>  po/bo.mini.po                |    20 +
>  po/bo.po                     | 46395 ------------------------------
>  po/br.mini.po                |    20 +
>  po/br.po                     | 46395 ------------------------------
>  po/brx.mini.po               |    20 +
>  po/brx.po                    | 46395 ------------------------------
>  po/bs.mini.po                |   767 +
>  po/bs.po                     | 46693 ------------------------------
>  po/ca.mini.po                |  1618 ++
>  po/ca.po                     | 46555 ------------------------------
>  po/cs.mini.po                |  4146 +++
>  po/cs.po                     | 46751 ------------------------------
>  po/cy.mini.po                |   762 +
>  po/cy.po                     | 46701 ------------------------------
>  po/da.mini.po                |   982 +
>  po/da.po                     | 46704 ------------------------------
>  po/de.mini.po                | 22654 +++++++++++++++
>  po/de.po                     | 47656 -------------------------------
>  po/de_CH.mini.po             |    20 +
>  po/de_CH.po                  | 46395 ------------------------------
>  po/el.mini.po                |   724 +
>  po/el.po                     | 46857 ------------------------------
>  po/en_GB.mini.po             | 22271 +++++++++++++++
>  po/en_GB.po                  | 47573 -------------------------------
>  po/eo.mini.po                |    20 +
>  po/eo.po                     | 46395 ------------------------------
>  po/es.mini.po                | 22622 +++++++++++++++
>  po/es.po                     | 47748 -------------------------------
>  po/et.mini.po                |   761 +
>  po/et.po                     | 46684 ------------------------------
>  po/eu.mini.po                |   761 +
>  po/eu.po                     | 46684 ------------------------------
>  po/fa.mini.po                |    20 +
>  po/fa.po                     | 46395 ------------------------------
>  po/fi.mini.po                |  1281 +
>  po/fi.po                     | 46744 ------------------------------
>  po/fil.mini.po               |    20 +
>  po/fr.mini.po                |  4258 +++
>  po/fr.po                     | 46907 ------------------------------
>  po/fur.mini.po               |    20 +
>  po/ga.mini.po                |    21 +
>  po/gl.mini.po                |   761 +
>  po/gl.po                     | 46700 ------------------------------
>  po/{gu.po => gu.mini.po}     | 54191 ++++++++++-------------------------
>  po/he.mini.po                |   761 +
>  po/he.po                     | 46684 ------------------------------
>  po/hi.mini.po                | 13772 +++++++++
>  po/hi.po                     | 47013 -------------------------------
>  po/hr.mini.po                |    21 +
>  po/hr.po                     | 46396 ------------------------------
>  po/hu.mini.po                |   987 +
>  po/hu.po                     | 46709 ------------------------------
>  po/ia.mini.po                |    20 +
>  po/ia.po                     | 46395 ------------------------------
>  po/id.mini.po                |   728 +
>  po/id.po                     | 46691 ------------------------------
>  po/ilo.mini.po               |    20 +
>  po/ilo.po                    | 46395 ------------------------------
>  po/is.mini.po                |   761 +
>  po/is.po                     | 46684 ------------------------------
>  po/it.mini.po                |  6588 +++++
>  po/it.po                     | 46663 ------------------------------
>  po/{ja.po => ja.mini.po}     | 54737 ++++++++++-------------------------
>  po/ka.mini.po                |   761 +
>  po/ka.po                     | 46684 ------------------------------
>  po/kk.mini.po                |    20 +
>  po/kk.po                     | 46395 ------------------------------
>  po/km.mini.po                |    20 +
>  po/km.po                     | 46395 ------------------------------
>  po/{kn.po => kn.mini.po}     | 54156 ++++++++++-------------------------
>  po/ko.mini.po                | 12838 +++++++++
>  po/ko.po                     | 46723 ------------------------------
>  po/kw.mini.po                |    21 +
>  po/kw.po                     | 46396 ------------------------------
>  po/kw at kkcor.mini.po          |    21 +
>  po/kw at kkcor.po               | 46396 ------------------------------
>  po/kw at uccor.mini.po          |    21 +
>  po/kw at uccor.po               | 46396 ------------------------------
>  po/kw_GB.mini.po             |    21 +
>  po/kw_GB.po                  | 46396 ------------------------------
>  po/ky.mini.po                |    20 +
>  po/ky.po                     | 46395 ------------------------------
>  po/libvirt.pot               | 46394 ------------------------------
>  po/lt.mini.po                |   762 +
>  po/lt.po                     | 46685 ------------------------------
>  po/lv.mini.po                |   762 +
>  po/lv.po                     | 46685 ------------------------------
>  po/mai.mini.po               |    20 +
>  po/mai.po                    | 46395 ------------------------------
>  po/mk.mini.po                |  1138 +
>  po/mk.po                     | 46724 ------------------------------
>  po/{ml.po => ml.mini.po}     | 53963 ++++++++++-------------------------
>  po/mn.mini.po                |    20 +
>  po/mn.po                     | 46395 ------------------------------
>  po/{mr.po => mr.mini.po}     | 55484 +++++++++++-------------------------
>  po/ms.mini.po                |   189 +
>  po/ms.po                     | 46580 ------------------------------
>  po/my.mini.po                |    20 +
>  po/nb.mini.po                |   492 +
>  po/nb.po                     | 46667 ------------------------------
>  po/nds.mini.po               |    20 +
>  po/nds.po                    | 46395 ------------------------------
>  po/ne.mini.po                |    20 +
>  po/ne.po                     | 46395 ------------------------------
>  po/nl.mini.po                |  9868 +++++++
>  po/nl.po                     | 47062 -------------------------------
>  po/nn.mini.po                |   761 +
>  po/nn.po                     | 46684 ------------------------------
>  po/nso.mini.po               |   761 +
>  po/nso.po                    | 46684 ------------------------------
>  po/{or.po => or.mini.po}     | 53161 +++++++++-------------------------
>  po/{pa.po => pa.mini.po}     | 53711 ++++++++++-------------------------
>  po/pl.mini.po                | 10274 +++++++
>  po/pl.po                     | 47166 -------------------------------
>  po/pt.mini.po                |  1243 +
>  po/pt.po                     | 46587 ------------------------------
>  po/pt_BR.mini.po             | 22821 +++++++++++++++
>  po/pt_BR.po                  | 47565 -------------------------------
>  po/ro.mini.po                |   762 +
>  po/ro.po                     | 46685 ------------------------------
>  po/{ru.po => ru.mini.po}     | 54065 ++++++++++-------------------------
>  po/si.mini.po                |   761 +
>  po/si.po                     | 46684 ------------------------------
>  po/sk.mini.po                |   761 +
>  po/sk.po                     | 46684 ------------------------------
>  po/sl.mini.po                |   762 +
>  po/sl.po                     | 46685 ------------------------------
>  po/sq.mini.po                |    24 +
>  po/sq.po                     | 46399 ------------------------------
>  po/sr.mini.po                |  2203 ++
>  po/sr.po                     | 46910 ------------------------------
>  po/sr at latin.mini.po          |  2204 ++
>  po/sr at latin.po               | 46911 ------------------------------
>  po/sv.mini.po                |  2559 ++
>  po/sv.po                     | 46696 ------------------------------
>  po/{ta.po => ta.mini.po}     | 54968 +++++++++++-------------------------
>  po/{te.po => te.mini.po}     | 53427 ++++++++++-------------------------
>  po/tg.mini.po                |    20 +
>  po/tg.po                     | 46395 ------------------------------
>  po/th.mini.po                |   761 +
>  po/th.po                     | 46684 ------------------------------
>  po/tr.mini.po                |   761 +
>  po/tr.po                     | 46684 ------------------------------
>  po/tw.mini.po                |    20 +
>  po/tw.po                     | 46395 ------------------------------
>  po/{uk.po => uk.mini.po}     | 62575 +++++++++++++++++------------------------
>  po/ur.mini.po                |   761 +
>  po/ur.po                     | 46684 ------------------------------
>  po/vi.mini.po                |  8674 ++++++
>  po/vi.po                     | 47171 -------------------------------
>  po/wba.mini.po               |    20 +
>  po/wba.po                    | 46395 ------------------------------
>  po/yo.mini.po                |    20 +
>  po/yo.po                     | 46395 ------------------------------
>  po/zanata.xml                |    99 -
>  po/zh_CN.mini.po             | 23613 ++++++++++++++++
>  po/zh_CN.po                  | 46701 ------------------------------
>  po/zh_HK.mini.po             |    20 +
>  po/zh_HK.po                  | 46395 ------------------------------
>  po/zh_TW.mini.po             |   958 +
>  po/zh_TW.po                  | 46725 ------------------------------
>  po/zu.mini.po                |   761 +
>  po/zu.po                     | 46684 ------------------------------
>  src/internal.h               |    16 +-
>  src/libvirt-admin.c          |     2 +
>  src/libvirt.c                |     2 +
>  src/util/virgettext.c        |     3 +-
>  tools/virt-host-validate.c   |     4 +-
>  199 files changed, 418787 insertions(+), 4388281 deletions(-)
> 
> -- 
> 2.14.3
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|




More information about the libvir-list mailing list