From jjaggars at redhat.com Wed Sep 5 20:37:50 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Wed, 5 Sep 2012 16:37:50 -0400 (EDT) Subject: [sos-devel] isatty opinions In-Reply-To: <1965078835.705773.1346877101671.JavaMail.root@redhat.com> Message-ID: <1851494366.728121.1346877470523.JavaMail.root@redhat.com> So I've been reassigned the following JDR bug: https://issues.jboss.org/browse/JBPAPP-9565 The story behind this situation is long and gory, but to make it short the productization process removed some libraries that jython uses to call native code for what could be argued as fair reasons (could easily be argued the other way too I suppose.) Anyhow, it seems that without the jffi libraries JDR fails to execute while running on a solaris box due to the inability to find the proper native bits to run. The call in question is os.isatty(). We use this call in two spots today: [jjaggars sos (master $>)]$ ack isatty sosreport.py 270: if hasattr(sys, 'ps1') or not sys.stderr.isatty(): 310: if not sys.stdin.isatty(): The first is when we catch an exception in debug mode, if we don't have a tty we don't use pdb to debug. The second is when we setup logging; if we don't have a tty we disable some output stuff. How critical is it to attempt to detect an underlying tty for these cases? Can we safely trust our defaults and the user (or blame them for their mistakes if they do it wrong) for these seemingly trivial situations? Here are some more details to help frame things a bit more. I'm not sure how important it is to fix this issue, might mark as a will_not_fix. I'm also unsure if removing the isatty() calls will be enough to fix this issue altogether, it's possible that there are a whole mess of functions that follow the same failing code path that I don't know about yet and don't have a good way to test for. Thanks Jesse From bmr at redhat.com Thu Sep 6 13:19:24 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Thu, 06 Sep 2012 14:19:24 +0100 Subject: [sos-devel] isatty opinions In-Reply-To: <1851494366.728121.1346877470523.JavaMail.root@redhat.com> References: <1851494366.728121.1346877470523.JavaMail.root@redhat.com> Message-ID: <5048A2DC.6020505@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/05/2012 09:37 PM, Jesse Jaggars wrote: > So I've been reassigned the following JDR bug: > https://issues.jboss.org/browse/JBPAPP-9565 > > The story behind this situation is long and gory, but to make it > short the productization process removed some libraries that jython > uses to call native code for what could be argued as fair reasons > (could easily be argued the other way too I suppose.) > > Anyhow, it seems that without the jffi libraries JDR fails to > execute while running on a solaris box due to the inability to find > the proper native bits to run. The call in question is os.isatty(). > Ugh. I'm not sure I like this restriction although I do see that we're in a tough spot (I don't think that anyone wants to compile native binaries for Solaris). Is there really no way that this can be done with existing available binary code? > We use this call in two spots today: > > [jjaggars sos (master $>)]$ ack isatty sosreport.py 270: if > hasattr(sys, 'ps1') or not sys.stderr.isatty(): 310: if not > sys.stdin.isatty(): Is there some way we can shim this? The isatty checks are generally of interest in detecting interactive/non-interactive sessions and are important in Linux environments. I'd assume that in Jython execution we never have a controlling TTY, is that correct? In that case we can replace direct calls to isatty() with a function that always returns false under Jython but makes the os.isatty() call on other platforms. > The first is when we catch an exception in debug mode, if we don't > have a tty we don't use pdb to debug. The second is when we setup > logging; if we don't have a tty we disable some output stuff. It gates batch mode and disables coloured output (to avoid non-printing control characters on stdout). > How critical is it to attempt to detect an underlying tty for these > cases? Can we safely trust our defaults and the user (or blame them > for their mistakes if they do it wrong) for these seemingly trivial > situations? I would guess that there are quite a few users that run sos without a tty and expect that to imply batch mode. Without that check we'll start prompting for stuff and those users current modes of execution will hang. > Here are some more details to help frame things a bit more. I'm not > sure how important it is to fix this issue, might mark as a > will_not_fix. I'm also unsure if removing the isatty() calls will > be enough to fix this issue altogether, it's possible that there > are a whole mess of functions that follow the same failing code > path that I don't know about yet and don't have a good way to test > for. That's probably my biggest concern here: what else could we be missing? How does this affect future plans in terms of making certain core APIs "off-limits" for sos? Shimming a few corner cases isn't a big deal. Wrapping half the python standard library would be much more painful. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlBIotwACgkQ6YSQoMYUY96L1wCfVhGdF84lQkBeTLM4EsMDswOG InYAoKAipBYfwsO4l+7BWSIwFEOjh4uG =Gb+O -----END PGP SIGNATURE----- From jjaggars at redhat.com Thu Sep 6 15:27:24 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Thu, 6 Sep 2012 11:27:24 -0400 (EDT) Subject: [sos-devel] isatty opinions In-Reply-To: <5048A2DC.6020505@redhat.com> Message-ID: <487620129.3742052.1346945244084.JavaMail.root@redhat.com> More responses below... ----- Original Message ----- > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 09/05/2012 09:37 PM, Jesse Jaggars wrote: > > So I've been reassigned the following JDR bug: > > https://issues.jboss.org/browse/JBPAPP-9565 > > > > The story behind this situation is long and gory, but to make it > > short the productization process removed some libraries that jython > > uses to call native code for what could be argued as fair reasons > > (could easily be argued the other way too I suppose.) > > > > Anyhow, it seems that without the jffi libraries JDR fails to > > execute while running on a solaris box due to the inability to find > > the proper native bits to run. The call in question is os.isatty(). > > > Ugh. I'm not sure I like this restriction although I do see that > we're > in a tough spot (I don't think that anyone wants to compile native > binaries for Solaris). > Yep that's basically it. The community version just ships the .so files that come with jython, but that's not an acceptable option for EAP unfortunately. There was some discussion of making an exception somehow but that fizzled out. > Is there really no way that this can be done with existing available > binary code? > Could be, I haven't investigated trying with pure java and then falling back onto os.isatty. That's probably possible, but it promotes ugliness. > > We use this call in two spots today: > > > > [jjaggars sos (master $>)]$ ack isatty sosreport.py 270: if > > hasattr(sys, 'ps1') or not sys.stderr.isatty(): 310: if not > > sys.stdin.isatty(): > > Is there some way we can shim this? The isatty checks are generally > of > interest in detecting interactive/non-interactive sessions and are > important in Linux environments. > > I'd assume that in Jython execution we never have a controlling TTY, > is that correct? > This isn't entirely true, but sosreport is forced into batch mode operation in every case so maybe we could skip the whole thing in this case. > In that case we can replace direct calls to isatty() with a function > that always returns false under Jython but makes the os.isatty() call > on other platforms. > > > The first is when we catch an exception in debug mode, if we don't > > have a tty we don't use pdb to debug. The second is when we setup > > logging; if we don't have a tty we disable some output stuff. > > It gates batch mode and disables coloured output (to avoid > non-printing control characters on stdout). > So colored output was removed so the only real benefit is the setting of batch mode in this case. I wonder if defaulting to batch mode would be a miserable idea? Providing we add an --interactive option. > > How critical is it to attempt to detect an underlying tty for these > > cases? Can we safely trust our defaults and the user (or blame them > > for their mistakes if they do it wrong) for these seemingly trivial > > situations? > > I would guess that there are quite a few users that run sos without a > tty and expect that to imply batch mode. Without that check we'll > start prompting for stuff and those users current modes of execution > will hang. See above. > > > Here are some more details to help frame things a bit more. I'm not > > sure how important it is to fix this issue, might mark as a > > will_not_fix. I'm also unsure if removing the isatty() calls will > > be enough to fix this issue altogether, it's possible that there > > are a whole mess of functions that follow the same failing code > > path that I don't know about yet and don't have a good way to test > > for. > > That's probably my biggest concern here: what else could we be > missing? How does this affect future plans in terms of making certain > core APIs "off-limits" for sos? > > Shimming a few corner cases isn't a big deal. Wrapping half the > python > standard library would be much more painful. > Getting sosreport to work in jython has added some corner-case handling ugliness already, but it'd be nice to cut back on that I think. Might be a good idea to think about separating the UI layer from the actual machinery. > Regards, > Bryn. > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAlBIotwACgkQ6YSQoMYUY96L1wCfVhGdF84lQkBeTLM4EsMDswOG > InYAoKAipBYfwsO4l+7BWSIwFEOjh4uG > =Gb+O > -----END PGP SIGNATURE----- > From bmr at redhat.com Thu Sep 6 16:11:26 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Thu, 06 Sep 2012 17:11:26 +0100 Subject: [sos-devel] isatty opinions In-Reply-To: <487620129.3742052.1346945244084.JavaMail.root@redhat.com> References: <487620129.3742052.1346945244084.JavaMail.root@redhat.com> Message-ID: <5048CB2E.7060106@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/06/2012 04:27 PM, Jesse Jaggars wrote: > 09/06/2012 02:19 PM Bryn M. Reeves wrote: >> On 09/05/2012 09:37 PM, Jesse Jaggars wrote: >>> So I've been reassigned the following JDR bug: >>> https://issues.jboss.org/browse/JBPAPP-9565 > Yep that's basically it. The community version just ships the .so > files that come with jython, but that's not an acceptable option > for EAP unfortunately. There was some discussion of making an > exception somehow but that fizzled out. It might be worth revisiting that; I think there have been similar exceptions in the past (we have previously even supported products that we had to build natively on Solaris, although not recently iirc). > Could be, I haven't investigated trying with pure java and then > falling back onto os.isatty. That's probably possible, but it > promotes ugliness. Afaik Java still lacks a native isatty() equivalent (otherwise I guess Jython would just use that implementation). There's been a bug open for about 15 years so I'm not holding my breath for that: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4099017 > This isn't entirely true, but sosreport is forced into batch mode > operation in every case so maybe we could skip the whole thing in > this case. In Jython generally, sure, but in the scenario we're expecting sos to run would we expect a real tty? Either way it's really moot since you mentioned that we're always forcing batch mode there. > So colored output was removed so the only real benefit is the > setting of batch mode in this case. I wonder if defaulting to batch > mode would be a miserable idea? Providing we add an --interactive > option. I'd be very reluctant - this would be a change in behaviour that's been around since sysreport. We have a lot of product and training docs as well as considerable field knowledge that says that when you just run "sosreport" you get an interactive prompt-driven UI. We've talked about making some UI changes and I'd still like to carry on with that. It's possible this might give us a better avenue for doing this but I think forcing batch on a default invocation of "sosreport" is likely to be very unpopular with users and support engineers. >> I would guess that there are quite a few users that run sos >> without a tty and expect that to imply batch mode. Without that >> check we'll start prompting for stuff and those users current >> modes of execution will hang. I wouldn't be happy unleashing this as-is (i.e. as a change in behaviour of the existing sosreport command - if a user runs "sosreport" we want them to get the expected behaviour). > Getting sosreport to work in jython has added some corner-case > handling ugliness already, but it'd be nice to cut back on that I > think. Might be a good idea to think about separating the UI layer > from the actual machinery. This is absolutely something that I favour (although I'm concerned by some of the recent changes that I see adding interactivity in individual modules - we've never had any structured way to handle that and it's generally something I think we should avoid - for much the same reason as RPM avoids interaction during package installation and updates). Do we have any reasonable way to determine what needs to be avoided in the Jython environment? A list of unavailable standard library modules or functions would be a great help in understanding the implications of this. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlBIyy4ACgkQ6YSQoMYUY96RUQCgyI+tgYWqLBqyoxjNgzf7loM6 bWcAoMu2nlhzLRYZhzw91eMlRUkiKtEK =BmrO -----END PGP SIGNATURE----- From jjaggars at redhat.com Thu Sep 6 16:38:42 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Thu, 6 Sep 2012 12:38:42 -0400 (EDT) Subject: [sos-devel] isatty opinions In-Reply-To: <5048CB2E.7060106@redhat.com> Message-ID: <1842944748.4043120.1346949522017.JavaMail.root@redhat.com> ----- Original Message ----- > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 09/06/2012 04:27 PM, Jesse Jaggars wrote: > > 09/06/2012 02:19 PM Bryn M. Reeves wrote: > >> On 09/05/2012 09:37 PM, Jesse Jaggars wrote: > >>> So I've been reassigned the following JDR bug: > >>> https://issues.jboss.org/browse/JBPAPP-9565 > > Yep that's basically it. The community version just ships the .so > > files that come with jython, but that's not an acceptable option > > for EAP unfortunately. There was some discussion of making an > > exception somehow but that fizzled out. > > It might be worth revisiting that; I think there have been similar > exceptions in the past (we have previously even supported products > that we had to build natively on Solaris, although not recently > iirc). > I'll probably bring this back up. I'd like to avoid changing code for such a specific user base. > > Could be, I haven't investigated trying with pure java and then > > falling back onto os.isatty. That's probably possible, but it > > promotes ugliness. > > Afaik Java still lacks a native isatty() equivalent (otherwise I > guess > Jython would just use that implementation). There's been a bug open > for about 15 years so I'm not holding my breath for that: > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4099017 > Figures, scratch that. I'll likely go the route you mentioned previously and try to skip the isatty calls under jython if it comes down to code changes. > > This isn't entirely true, but sosreport is forced into batch mode > > operation in every case so maybe we could skip the whole thing in > > this case. > > In Jython generally, sure, but in the scenario we're expecting sos to > run would we expect a real tty? Either way it's really moot since you > mentioned that we're always forcing batch mode there. > > > So colored output was removed so the only real benefit is the > > setting of batch mode in this case. I wonder if defaulting to batch > > mode would be a miserable idea? Providing we add an --interactive > > option. > > I'd be very reluctant - this would be a change in behaviour that's > been around since sysreport. We have a lot of product and training > docs as well as considerable field knowledge that says that when you > just run "sosreport" you get an interactive prompt-driven UI. > > We've talked about making some UI changes and I'd still like to carry > on with that. It's possible this might give us a better avenue for > doing this but I think forcing batch on a default invocation of > "sosreport" is likely to be very unpopular with users and support > engineers. > > >> I would guess that there are quite a few users that run sos > >> without a tty and expect that to imply batch mode. Without that > >> check we'll start prompting for stuff and those users current > >> modes of execution will hang. > > I wouldn't be happy unleashing this as-is (i.e. as a change in > behaviour of the existing sosreport command - if a user runs > "sosreport" we want them to get the expected behaviour). > Agree here. And on the long list of things we would like to change, this behavior is very low on that list (if it even belongs.) > > Getting sosreport to work in jython has added some corner-case > > handling ugliness already, but it'd be nice to cut back on that I > > think. Might be a good idea to think about separating the UI layer > > from the actual machinery. > > This is absolutely something that I favour (although I'm concerned by > some of the recent changes that I see adding interactivity in > individual modules - we've never had any structured way to handle > that > and it's generally something I think we should avoid - for much the > same reason as RPM avoids interaction during package installation and > updates). > Makes sense to me. I wasn't aware of any interactive plugins. I'm certainly not a fan of that idea generally. > Do we have any reasonable way to determine what needs to be avoided > in > the Jython environment? A list of unavailable standard library > modules > or functions would be a great help in understanding the implications > of this. > I'm not sure. Generally though anything that is backed up by c should be avoided. C-based extensions simply do not work and code that relies on underlying c libraries requires the jffi binary stuff. The jffi bits wouldn't be a problem if it weren't for the productization requirements. At plugin load time, assuming you build your plugin with all imports made at the top of the file, sosreport will disable your plugin if it can't load all the libraries you depend upon. This provides a minimal safety net for these situations, but in the case of the os module, we don't run into linking errors simply by importing os. > Regards, > Bryn. > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAlBIyy4ACgkQ6YSQoMYUY96RUQCgyI+tgYWqLBqyoxjNgzf7loM6 > bWcAoMu2nlhzLRYZhzw91eMlRUkiKtEK > =BmrO > -----END PGP SIGNATURE----- > From bmr at redhat.com Thu Sep 6 17:20:55 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Thu, 06 Sep 2012 18:20:55 +0100 Subject: [sos-devel] isatty opinions In-Reply-To: <1842944748.4043120.1346949522017.JavaMail.root@redhat.com> References: <1842944748.4043120.1346949522017.JavaMail.root@redhat.com> Message-ID: <5048DB77.5080306@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/06/2012 05:38 PM, Jesse Jaggars wrote: >> Makes sense to me. I wasn't aware of any interactive plugins. I'm >> certainly not a fan of that idea generally. The main one I noticed while doing a complete audit last month was the EMC module. This prompts for SP IP addresses during setup() - that's something I would like to remove. There are a couple of others that seem to require command-line configuration in order to be useful. I can see that for some operations that's a requirement but ideally I'd like things to generate some useful data when run with nothing but defaults. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlBI23cACgkQ6YSQoMYUY97EfACguRI4YM8ODqqKcjsczqHay1CE NHIAoL3SMnirK1U9nc1VPH6N01nOh7ty =gh2v -----END PGP SIGNATURE----- From jjaggars at redhat.com Wed Sep 12 19:36:20 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Wed, 12 Sep 2012 15:36:20 -0400 (EDT) Subject: [sos-devel] isatty opinions In-Reply-To: <5048DB77.5080306@redhat.com> Message-ID: <1136541286.19495169.1347478580094.JavaMail.root@redhat.com> I've decided to attempt to fix this by avoiding os.isatty calls when executing on java. Trying to do this as cleanly as possible of course. In other news, Jython is very near to releasing a 2.7+ compatible version, meaning that we can drop all the backwards compatibility hacks for the 2.5.x stuff (like using simplejson for json and importing with_statement from __future__). Stay tuned ----- Original Message ----- > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 09/06/2012 05:38 PM, Jesse Jaggars wrote: > >> Makes sense to me. I wasn't aware of any interactive plugins. I'm > >> certainly not a fan of that idea generally. > > The main one I noticed while doing a complete audit last month was > the > EMC module. This prompts for SP IP addresses during setup() - that's > something I would like to remove. > > There are a couple of others that seem to require command-line > configuration in order to be useful. I can see that for some > operations that's a requirement but ideally I'd like things to > generate some useful data when run with nothing but defaults. > > Regards, > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAlBI23cACgkQ6YSQoMYUY97EfACguRI4YM8ODqqKcjsczqHay1CE > NHIAoL3SMnirK1U9nc1VPH6N01nOh7ty > =gh2v > -----END PGP SIGNATURE----- > From plambri at redhat.com Tue Sep 18 13:01:37 2012 From: plambri at redhat.com (Pierguido Lambri) Date: Tue, 18 Sep 2012 14:01:37 +0100 Subject: [sos-devel] Adding install files in the IPA plugin Message-ID: <20120918130137.GG16939@redhat.com> Hi, Would it be possible to enable few more IPA logs to capture with sosreport? The files are quite important when debugging installation issues: /var/log/ipaclient-install.log /var/log/ipaserver-install.log /var/log/ipareplica-install.log A patch should be as easy as: # diff -u ipa.py ipa-old.py --- ipa.py 2012-09-18 12:39:57.000000000 +0100 +++ ipa-old.py 2012-09-18 12:39:14.000000000 +0100 @@ -35,8 +35,5 @@ self.addCopySpec("/etc/ipa/ipa.conf") self.addCopySpec("/etc/krb5.conf") self.addCopySpec("/etc/krb5.keytab") - self.addCopySpec("/var/log/ipaclient-install.log") - self.addCopySpec("/var/log/ipaserver-install.log") - self.addCopySpec("/var/log/ipareplica-install.log") return Thanks, Pier