2013/1/11 John Dennis <<a href="mailto:jdennis@redhat.com" target="_blank">jdennis@redhat.com</a>> <div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On 01/11/2013 10:04 AM, Petr Viktorin wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hello list, This discussion was started in private; I'll continue it here. On 01/10/2013 05:41 PM, John Dennis wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On 01/10/2013 04:27 AM, Petr Viktorin wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On 01/09/2013 03:55 PM, John Dennis wrote: </blockquote> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> And I could work on improving the i18n/translations infrastructure, starting by writing up a RFE+design. </blockquote></blockquote></blockquote> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Could you elaborate as to what you perceive as the current problems and what this work would address. </blockquote></blockquote> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Here are my notes: </blockquote> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Use fake translations for tests </blockquote> We already do (but perhaps not sufficiently). </blockquote> I mean use it in *all* tests, to ensure all the right things are translated and weird characters are handled well. See <a href="https://www.redhat.com/archives/freeipa-devel/2012-October/msg00278.html" target="_blank">https://www.redhat.com/archives/freeipa-devel/2012-October/msg00278.html</a> </blockquote> Ah yes, I like the idea of a test domain for strings, this is a good idea. Not only would it exercise our i18n code more but it could insulate the tests from string changes (the test would look for a canonical string in the test domain) </blockquote><div> FWIW, KDE also uses an empty .po (e.g. empty translated messages) in order to easier spot strings not marked for translations. </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Split up huge strings so the entire text doesn't have to be retranslated each time something changes/is added </blockquote> Good idea. But one question I have is should we be optimizing for our programmers time or the translators time? The Transifex tool should make available to translators similar existing translations (in fact it might, I seem to recall some functionality in this area). Wouldn't it be better to address this issue in Transifex where all projects would benefit? Also the exact same functionality is needed to support release versions. The strings between releases are often close but not identical. The Transifex tool should make available a close match from a previous version to the translator working on a new version (or visa versa). See your issue below concerning versions. IMHO this is a Transifex issue which needs to be solved there, not something we should be investing precious IPA programmers time on. Plus if it's solved in Transifex it's a *huge* win for *everyone*, not just IPA. </blockquote> Huh? Splitting the strings provides additional information (paragraph/context boundaries) that Transifex can't get otherwise. From what I hear it's a pretty standard technique when working with gettext. </blockquote> I'm not sure how splitting text into smaller units gives more context but I can see the argument for each msgid being a logical paragraph. We don't have too many multi-paragraph strings now so it shouldn't be too involved. </blockquote><div> One issue also discussed on this list is the problem of 100+ lines strings in man pages generated from ___doc___ tags in scripts. Those are a _real_ pain for translators to maintain when only one line is changed. Didn't have the time yet to explore splitting those strings, I need to take some to do so. </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> For typos, gettext has the "fuzzy" functionality that we explicitly turn off. I think we're on our own here. </blockquote> Be very afraid of turning on fuzzy matching. Before we moved to TX we used the entire gnu tool chain. I discovered a number of our PO files were horribly corrupted. With a lot of work I traced this down to fuzzy matches. If memory serves me right here is what happened. When a msgstr was absent a fuzzy match was performed and inserted as a candidate msgstr. Somehow the fuzzy candidates got accepted as actual msgstr's. I'm not sure if we ever figured out how this happened. The two most likely explanations were 1) a known bug in TX that stripped the fuzzy flag off the msgstr or 2) a translator who blindly accepted all "TX suggestions". (A suggestion in TX comes from a fuzzy match). But the real problem is the fuzzy matching is horribly bad. Most of the fuzzy suggestions (primarily on short strings) were wildly incorrect. I had to go back to a number of PO files and manually locate all fuzzy suggestions that had been promoted to legitimate msgstr's. A tedious process I hope to never repeat. BTW, if memory serves me correctly the fuzzy suggestions got into the PO files in the first place because we were running the full gnu tool chain (sorry off the top of my head I don't recall exactly which component inserts the fuzzy suggestion), but I think we've since turned that off, for a very good reason. <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Keep a history/repo of the translations, since Transifex only stores the latest version </blockquote> We already do keep a history, it's in git. </blockquote> It's not updated often enough. If I mess something up before a release and Transifex gets wiped, or if a rogue translator deletes some translations, the work is gone. </blockquote> Yes, updating more frequently is an excellent goal. </blockquote><div> Yes, please! Having nothing to translate for months on Transifex is not fun. Having a mass of new strings to translate every once in a while when a new set of strings is eventually pushed to Transifex when someone remembers to so is certainly not. </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Update the source strings on Transifex more often (ideally as soon as patches are pushed) </blockquote> Yes, great idea, this would be really useful and is necessary. <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Break Git dependencies: make it possible generate the POT in an unpacked tarball </blockquote> Are you talking about the fact our scripts invoke git to determine what files to process? If so then yes, this would be a good dependency to get rid of. However it does mean we somehow have to maintain a manifest list of some sort somewhere. </blockquote> A directory listing is fine IMO. We use it for more critical things, like loading plugins, without any trouble. Also, when run in a Git repo the Makefile can compare the file list with what Git says and warn accordingly. </blockquote> How do you which files in a directory should be scanned for translations? Perhaps it doesn't hurt to scan every file, we never tried scanning inappropriate files so I don't know what the consequence would be. A little history: Originally there was a manifest of every file to be scanned. Simo made some fixes to the i18n infrastructure and didn't like the manifest idea (it has an obvious downside, it has to be updated when source files are added/deleted/moved). Simo used git to get a list of source files. But that mechanism depended on identifying a source files via it's filename extension. Our scripts don't have an extension so those were hardcoded just like the original manifest. It didn't eliminate the maintenance problem, it just made it smaller. To my mind that was the worst of both worlds, it introduced a git dependency but didn't solve the manual manifest problem. One could do something else, if a file doesn't have an extension you can read the beginning of the file and look for a shebang (#!) interpreter line, that would identify the file as a script and hence a translation candidate. </blockquote><div> At least this is scriptable easily using file(1). </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> But perhaps your idea of scan everything and throw away (or ignore) anything which won't scan correctly because because it's not valid input is the best. As I said I don't think we ever explored that, but it might also be because in some instances we have to tell the scanner what type of file the input is (a chicken-n-egg problem). But it's an interesting idea and we should see how it works in practice. <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Figure out how to best share messages across versions (2.x vs. 3.x) so they only have to be translated once </blockquote> There is a crying need for this, but isn't this a Transifex issue? Why would we solving this in IPA? What about SSSD and every other project, they all have identical issues. As far as I can tell Transifex has never addressed this issue sufficiently (see above) and the onus is on them to do so. </blockquote> I don't think waiting for Transifex will solve the problem. </blockquote> Then what is your suggestion? Pull every msgid from every version, put it into one massive unified pot file and then split the resulting unified PO files back into version specific PO files? Well I suppose we wouldn't have to split the PO files, they could just contain translations that are never referenced, it would make them larger but wouldn't hurt anything. Of course merging the strings from every version into one unified POT would play havoc with the msgid references (where is the string located) unless the filename was modified to include it's git branch. </blockquote> <div> I'd see a few remarks here: - this massive .po file would grow wildly, especially when a typo is corrected in huge strings (__doc___), when additional sentences are added to those, etc. - breaking down bigger strings in smaller ones will certainly help here in avoiding duplicated content, - in Transifex, it is easy to upload a .po onto another branch, and only untranslated matching strings would be updated. I used it on ananconda where there are multiple branches between Fedora & RHEL5/6 & master, that worked easily without breaking anything. </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Just thinking off the top of my head. <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Clean up checked-in PO files even more, for nicer diffs </blockquote> A nice feature, but I'm wondering to extent we're currently suffering because of this. It's rare that we have to compare PO files. Plus diff is not well suited for comparing PO's because PO files with equivalent data can be formatted differently. That's why I wrote some tools to read PO files, normalize the contents and then do a comparison. Anyway my top level question is is this something we really need at this point? </blockquote> You're right that files have to be normalized to diff well.That's actually the point here :) </blockquote></blockquote><div> That would be nice before getting translated content from Transifex, or any other sources of translation, since every single po editor seems to use its own format, being one massive line including \n, or multi line by ending and reopening quotes to keep everything in 72/80 cols. NB: That one is usually not a problem for translators (except those using a mail based review process, where having 72 cols may help, or not). </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Anyway I'm just thinking of sorting the PO alphabetically - an extra option to msgattrib should do it. <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> - Automate & document the process so any dev can do it </blockquote> Excellent goal, we're not too far from it now, but of all the things on the list this is the most important. </blockquote></blockquote></blockquote><div> Indeed. Please setup automation on pushing strings to Transifex, ideally in a "master" branch. Then each time a new significant release is prepared: - advertise the strings freeze, - possibly create a specific branch in Transifex, using available translations from the master branch, - advertise the new branch for translators to work on it. My 2 cents, J. </div></div>-- Jérôme Fenal