From jjaggars at redhat.com Tue Mar 6 22:24:01 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Tue, 6 Mar 2012 16:24:01 -0600 Subject: [sos-devel] RHEL7 date(s)? Message-ID: <20120306222401.GC592@localhost.localdomain> I assume we would like to have a major release of sosreport in time for RHEL7, but I'm not sure what date we are looking at for that, anybody know? Jesse From bmr at redhat.com Fri Mar 9 11:18:36 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Fri, 09 Mar 2012 11:18:36 +0000 Subject: [sos-devel] sosreport manifest? Message-ID: <4F59E70C.50704@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This is something that's come up a few times to assist in making the data in a sos tarball self-describing and discoverable (and to support the eventual creation of a library and language bindings to provide uniform access to the data). What would we want in the manifest and what would we want it to look like? Cheers, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9Z5wwACgkQ6YSQoMYUY94WngCeICv/wDs/PCSfSAadXXCeUCPR Th0AoLhTp2+Iqq2szsQFBTLQmhXAE0BO =eKdL -----END PGP SIGNATURE----- From adam.stokes at canonical.com Fri Mar 9 12:51:21 2012 From: adam.stokes at canonical.com (Adam Stokes) Date: Fri, 09 Mar 2012 07:51:21 -0500 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F59E70C.50704@redhat.com> References: <4F59E70C.50704@redhat.com> Message-ID: <4F59FCC9.60004@canonical.com> On 03/09/2012 06:18 AM, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > This is something that's come up a few times to assist in making the > data in a sos tarball self-describing and discoverable (and to support > the eventual creation of a library and language bindings to provide > uniform access to the data). > > What would we want in the manifest and what would we want it to look like? > > Cheers, > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk9Z5wwACgkQ6YSQoMYUY94WngCeICv/wDs/PCSfSAadXXCeUCPR > Th0AoLhTp2+Iqq2szsQFBTLQmhXAE0BO > =eKdL > -----END PGP SIGNATURE----- > > _______________________________________________ > Sos-devel mailing list > Sos-devel at redhat.com > https://www.redhat.com/mailman/listinfo/sos-devel > Would this be something outside of the diagnose routine? From bmr at redhat.com Fri Mar 9 13:29:19 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Fri, 09 Mar 2012 13:29:19 +0000 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F59FCC9.60004@canonical.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> Message-ID: <4F5A05AF.8060302@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/09/2012 12:51 PM, Adam Stokes wrote: > On 03/09/2012 06:18 AM, Bryn M. Reeves wrote: This is something > that's come up a few times to assist in making the data in a sos > tarball self-describing and discoverable (and to support the > eventual creation of a library and language bindings to provide > uniform access to the data). > > What would we want in the manifest and what would we want it to > look like? > > Cheers, Bryn. >> >> > Would this be something outside of the diagnose routine? > Yes - diagnose hasn't really seen much development lately and there are ongoing discussions about removing it (other tools outside of sos seem a better place for this functionality). The manifest would be descriptive metadata that tells consumers of a sos tarball what it contains and where. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9aBa8ACgkQ6YSQoMYUY95/YACgmvVUipsYss+5f9Q0keh8JZ5i uD4An27py0K/PmCRu0ptu+56R14n5KDv =9La9 -----END PGP SIGNATURE----- From jjaggars at redhat.com Fri Mar 9 20:10:58 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Fri, 9 Mar 2012 14:10:58 -0600 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5A05AF.8060302@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> Message-ID: <20120309201058.GB4168@localhost.localdomain> On Fri, Mar 09, 2012 at 01:29:19PM +0000, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 03/09/2012 12:51 PM, Adam Stokes wrote: > > On 03/09/2012 06:18 AM, Bryn M. Reeves wrote: This is something > > that's come up a few times to assist in making the data in a sos > > tarball self-describing and discoverable (and to support the > > eventual creation of a library and language bindings to provide > > uniform access to the data). > > > > What would we want in the manifest and what would we want it to > > look like? > > > > Cheers, Bryn. > >> > >> > > Would this be something outside of the diagnose routine? > > > > Yes - diagnose hasn't really seen much development lately and there > are ongoing discussions about removing it (other tools outside of sos > seem a better place for this functionality). > > The manifest would be descriptive metadata that tells consumers of a > sos tarball what it contains and where. > > Regards, > Bryn. > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk9aBa8ACgkQ6YSQoMYUY95/YACgmvVUipsYss+5f9Q0keh8JZ5i > uD4An27py0K/PmCRu0ptu+56R14n5KDv > =9La9 > -----END PGP SIGNATURE----- > > _______________________________________________ > Sos-devel mailing list > Sos-devel at redhat.com > https://www.redhat.com/mailman/listinfo/sos-devel Hey folks, To help me get a better idea what are some of the interface methods you would anticipate having in the long run? How would you go about using the library to work with a sosreport archive? Jesse From miclark at redhat.com Fri Mar 9 20:32:03 2012 From: miclark at redhat.com (Mike Clark) Date: Fri, 09 Mar 2012 14:32:03 -0600 Subject: [sos-devel] sosreport manifest? In-Reply-To: <20120309201058.GB4168@localhost.localdomain> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> Message-ID: <4F5A68C3.4000105@redhat.com> Hey folks, Bryn, I have basically the same question as Jesse here. But, let me go a step further. If the manifest is to "support the eventual creation of a library and language bindings to provide uniform access to the data," we really should start with that first. I don't think we even need to consider a manifest until the other conversation takes place. Cheers, Mike C. On 03/09/2012 02:10 PM, Jesse Jaggars wrote: > On Fri, Mar 09, 2012 at 01:29:19PM +0000, Bryn M. Reeves wrote: > On 03/09/2012 12:51 PM, Adam Stokes wrote: > >>> On 03/09/2012 06:18 AM, Bryn M. Reeves wrote: This is something > >>> that's come up a few times to assist in making the data in a sos > >>> tarball self-describing and discoverable (and to support the > >>> eventual creation of a library and language bindings to provide > >>> uniform access to the data). > >>> > >>> What would we want in the manifest and what would we want it to > >>> look like? > >>> > >>> Cheers, Bryn. > >>>> > >>>> > >>> Would this be something outside of the diagnose routine? > >>> > > Yes - diagnose hasn't really seen much development lately and there > are ongoing discussions about removing it (other tools outside of sos > seem a better place for this functionality). > > The manifest would be descriptive metadata that tells consumers of a > sos tarball what it contains and where. > > Regards, > Bryn. > >> >> _______________________________________________ >> Sos-devel mailing list >> Sos-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/sos-devel > > Hey folks, > > To help me get a better idea what are some of the interface methods you > would anticipate having in the long run? How would you go about using > the library to work with a sosreport archive? > > Jesse > > _______________________________________________ > Sos-devel mailing list > Sos-devel at redhat.com > https://www.redhat.com/mailman/listinfo/sos-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 554 bytes Desc: OpenPGP digital signature URL: From adam.stokes at canonical.com Fri Mar 9 21:31:03 2012 From: adam.stokes at canonical.com (Adam Stokes) Date: Fri, 09 Mar 2012 16:31:03 -0500 Subject: [sos-devel] sosreport manifest? In-Reply-To: <20120309201058.GB4168@localhost.localdomain> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> Message-ID: <4F5A7697.6010007@canonical.com> On 03/09/2012 03:10 PM, Jesse Jaggars wrote: > On Fri, Mar 09, 2012 at 01:29:19PM +0000, Bryn M. Reeves wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 03/09/2012 12:51 PM, Adam Stokes wrote: >>> On 03/09/2012 06:18 AM, Bryn M. Reeves wrote: This is something >>> that's come up a few times to assist in making the data in a sos >>> tarball self-describing and discoverable (and to support the >>> eventual creation of a library and language bindings to provide >>> uniform access to the data). >>> >>> What would we want in the manifest and what would we want it to >>> look like? >>> >>> Cheers, Bryn. >>>> >>> Would this be something outside of the diagnose routine? >>> >> Yes - diagnose hasn't really seen much development lately and there >> are ongoing discussions about removing it (other tools outside of sos >> seem a better place for this functionality). >> >> The manifest would be descriptive metadata that tells consumers of a >> sos tarball what it contains and where. >> >> Regards, >> Bryn. >> >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.4.12 (GNU/Linux) >> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ >> >> iEYEARECAAYFAk9aBa8ACgkQ6YSQoMYUY95/YACgmvVUipsYss+5f9Q0keh8JZ5i >> uD4An27py0K/PmCRu0ptu+56R14n5KDv >> =9La9 >> -----END PGP SIGNATURE----- >> >> _______________________________________________ >> Sos-devel mailing list >> Sos-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/sos-devel > Hey folks, > > To help me get a better idea what are some of the interface methods you > would anticipate having in the long run? How would you go about using > the library to work with a sosreport archive? > > Jesse > TBH the first thing that comes to mind is http://www.splunk.com/ and not necessarily trying to integrate with them but a more homegrown approach where it's all inclusive with sosreport. Thanks, Adam From bmr at redhat.com Mon Mar 12 11:14:19 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Mon, 12 Mar 2012 11:14:19 +0000 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5A68C3.4000105@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> Message-ID: <4F5DDA8B.6020300@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/09/2012 08:32 PM, Mike Clark wrote: > Hey folks, > > Bryn, I have basically the same question as Jesse here. But, let > me go a step further. If the manifest is to "support the eventual > creation of a library and language bindings to provide uniform > access to the data," we really should start with that first. I > don't think we even need to consider a manifest until the other > conversation takes place. So, my problem here is getting the users of the data to engage with the team working on sos (and any future access library). I've been trying to get requirements for downstream data users for the best part of a year but discussions have never come to anything particularly concrete. We need to get something done here as soon as possible - there is already a proliferation of code to access the data in an sosreport on a variety of languages. If we find ourselves wanting to make the kind of changes that we want in sos this is bound to lead to resistance because this code is very brittle (hard-coded paths are the norm). By having an answer as to how an external user of the data can discover the content of a tarball (without resorting to path crawling) we make a step towards that aim. I don't think we need anything fancy here, just a journal of what has been collected and where it is. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9d2osACgkQ6YSQoMYUY96hQgCg0REEkSnbOo/cxq5Qu4UwMkHO /B0AoN0jx3xRshn7LIkxbJM/jmzxsGI9 =3RPY -----END PGP SIGNATURE----- From ahecox at redhat.com Mon Mar 12 13:00:25 2012 From: ahecox at redhat.com (Andrew Hecox) Date: Mon, 12 Mar 2012 09:00:25 -0400 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5DDA8B.6020300@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> <4F5DDA8B.6020300@redhat.com> Message-ID: <4F5DF369.60405@redhat.com> On 03/12/2012 07:14 AM, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 03/09/2012 08:32 PM, Mike Clark wrote: >> Hey folks, >> >> Bryn, I have basically the same question as Jesse here. But, let >> me go a step further. If the manifest is to "support the eventual >> creation of a library and language bindings to provide uniform >> access to the data," we really should start with that first. I >> don't think we even need to consider a manifest until the other >> conversation takes place. > > So, my problem here is getting the users of the data to engage with > the team working on sos (and any future access library). I've been > trying to get requirements for downstream data users for the best part > of a year but discussions have never come to anything particularly > concrete. > > We need to get something done here as soon as possible - there is > already a proliferation of code to access the data in an sosreport on > a variety of languages. If we find ourselves wanting to make the kind > of changes that we want in sos this is bound to lead to resistance > because this code is very brittle (hard-coded paths are the norm). > > By having an answer as to how an external user of the data can > discover the content of a tarball (without resorting to path crawling) > we make a step towards that aim. > > I don't think we need anything fancy here, just a journal of what has > been collected and where it is. tar archives provide a journal/manifest of the names of files, and where they are. Because tar is a well known file format, there are already language bindings in place for everything anyone would want to do. -Ah > Regards, > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk9d2osACgkQ6YSQoMYUY96hQgCg0REEkSnbOo/cxq5Qu4UwMkHO > /B0AoN0jx3xRshn7LIkxbJM/jmzxsGI9 > =3RPY > -----END PGP SIGNATURE----- > > _______________________________________________ > Sos-devel mailing list > Sos-devel at redhat.com > https://www.redhat.com/mailman/listinfo/sos-devel -- Andrew Hecox Supervisor, Software Engineering Subscription Engineering ?????????? From bmr at redhat.com Mon Mar 12 14:17:09 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Mon, 12 Mar 2012 14:17:09 +0000 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5DF369.60405@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> <4F5DDA8B.6020300@redhat.com> <4F5DF369.60405@redhat.com> Message-ID: <4F5E0565.3090609@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/12/2012 01:00 PM, Andrew Hecox wrote: >> tar archives provide a journal/manifest of the names of files, >> and where they are. Because tar is a well known file format, >> there are already language bindings in place for everything >> anyone would want to do. Unfortunately this does not solve the problem; as I mentioned before the tar structure should be considered an ABI. It is subject to change and is not discoverable in a way that will support upcoming changes in a robust way. For instance, sos will soon need to have the ability to collect the same configuration file from multiple locations and to record this information in the archive. While it would be possible to search all archives for all possible paths this is inefficient and prone to failure. For instance, if we change the set of command line options used to run a program the path in the tarball will changes as might the content present at that path. This also cannot address more complex manipulations of the data than a simple "read this data from this path". In looking at the tools that consume sos data there is huge overlap and commonality in the information that they retrieve and the processing that they carry out. Reinventing this in each project that has overlapping requirements is not maintainable long-term and will hinder sos's ability to adapt to meet changing requirements (since there will be pressure to not make changes that would affect downstream consumers). Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9eBWUACgkQ6YSQoMYUY96fIQCgkMGxu0sSEFBSkU/NZ9HS39c9 vHMAn0i/+ee4waOa5HrU0jh16tLXi6QX =qeBC -----END PGP SIGNATURE----- From jjaggars at redhat.com Mon Mar 12 14:33:58 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Mon, 12 Mar 2012 09:33:58 -0500 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5E0565.3090609@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> <4F5DDA8B.6020300@redhat.com> <4F5DF369.60405@redhat.com> <4F5E0565.3090609@redhat.com> Message-ID: <20120312143358.GA31542@localhost.localdomain> On Mon, Mar 12, 2012 at 02:17:09PM +0000, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 03/12/2012 01:00 PM, Andrew Hecox wrote: > >> tar archives provide a journal/manifest of the names of files, > >> and where they are. Because tar is a well known file format, > >> there are already language bindings in place for everything > >> anyone would want to do. > > Unfortunately this does not solve the problem; as I mentioned before > the tar structure should be considered an ABI. It is subject to change > and is not discoverable in a way that will support upcoming changes in > a robust way. > > For instance, sos will soon need to have the ability to collect the > same configuration file from multiple locations and to record this > information in the archive. While it would be possible to search all > archives for all possible paths this is inefficient and prone to > failure. For instance, if we change the set of command line options > used to run a program the path in the tarball will changes as might > the content present at that path. > > This also cannot address more complex manipulations of the data than a > simple "read this data from this path". In looking at the tools that > consume sos data there is huge overlap and commonality in the > information that they retrieve and the processing that they carry out. > Reinventing this in each project that has overlapping requirements is > not maintainable long-term and will hinder sos's ability to adapt to > meet changing requirements (since there will be pressure to not make > changes that would affect downstream consumers). > > Regards, > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk9eBWUACgkQ6YSQoMYUY96fIQCgkMGxu0sSEFBSkU/NZ9HS39c9 > vHMAn0i/+ee4waOa5HrU0jh16tLXi6QX > =qeBC > -----END PGP SIGNATURE----- > > _______________________________________________ > Sos-devel mailing list > Sos-devel at redhat.com > https://www.redhat.com/mailman/listinfo/sos-devel I'm pretty ignorant here in that I'm not familiar with any of the consumer tools. I've only seen the sosviewer thing that was recently 'integrated' into SFDC. What are some of the other tools that people have written? I'd like to take a look at the sorts of things that the are doing more specifically to help me get a better understanding of what folks actually want. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From ahecox at redhat.com Mon Mar 12 14:35:50 2012 From: ahecox at redhat.com (Andrew Hecox) Date: Mon, 12 Mar 2012 10:35:50 -0400 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5E0565.3090609@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> <4F5DDA8B.6020300@redhat.com> <4F5DF369.60405@redhat.com> <4F5E0565.3090609@redhat.com> Message-ID: <4F5E09C6.6060809@redhat.com> On 03/12/2012 10:17 AM, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 03/12/2012 01:00 PM, Andrew Hecox wrote: >>> tar archives provide a journal/manifest of the names of files, >>> and where they are. Because tar is a well known file format, >>> there are already language bindings in place for everything >>> anyone would want to do. > > Unfortunately this does not solve the problem; as I mentioned before > the tar structure should be considered an ABI. It is subject to change > and is not discoverable in a way that will support upcoming changes in > a robust way. I'd guess to most consumers, the logical layout & content is an API found through tar bindings for whatever your specific languages are. > For instance, sos will soon need to have the ability to collect the > same configuration file from multiple locations and to record this > information in the archive. While it would be possible to search all > archives for all possible paths this is inefficient and prone to > failure. For instance, if we change the set of command line options > used to run a program the path in the tarball will changes as might > the content present at that path. what are these "same configuration file from multiple locations" use-cases? -Ah > This also cannot address more complex manipulations of the data than a > simple "read this data from this path". In looking at the tools that > consume sos data there is huge overlap and commonality in the > information that they retrieve and the processing that they carry out. > Reinventing this in each project that has overlapping requirements is > not maintainable long-term and will hinder sos's ability to adapt to > meet changing requirements (since there will be pressure to not make > changes that would affect downstream consumers). > > Regards, > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk9eBWUACgkQ6YSQoMYUY96fIQCgkMGxu0sSEFBSkU/NZ9HS39c9 > vHMAn0i/+ee4waOa5HrU0jh16tLXi6QX > =qeBC > -----END PGP SIGNATURE----- -- Andrew Hecox Supervisor, Software Engineering Subscription Engineering ?????????? From bmr at redhat.com Mon Mar 12 14:40:05 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Mon, 12 Mar 2012 14:40:05 +0000 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5E09C6.6060809@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> <4F5DDA8B.6020300@redhat.com> <4F5DF369.60405@redhat.com> <4F5E0565.3090609@redhat.com> <4F5E09C6.6060809@redhat.com> Message-ID: <4F5E0AC5.2010509@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/12/2012 02:35 PM, Andrew Hecox wrote: > On 03/12/2012 10:17 AM, Bryn M. Reeves wrote: On 03/12/2012 01:00 > PM, Andrew Hecox wrote: >>>>> tar archives provide a journal/manifest of the names of >>>>> files, and where they are. Because tar is a well known file >>>>> format, there are already language bindings in place for >>>>> everything anyone would want to do. > > Unfortunately this does not solve the problem; as I mentioned > before the tar structure should be considered an ABI. It is subject > to change and is not discoverable in a way that will support > upcoming changes in a robust way. > >> I'd guess to most consumers, the logical layout & content is an >> API found through tar bindings for whatever your specific >> languages are. That does not make an API. APIs have a published specification, are well-defined and stable for some defined purpose. None of those criteria are met by the structure of the tarball. > For instance, sos will soon need to have the ability to collect > the same configuration file from multiple locations and to record > this information in the archive. While it would be possible to > search all archives for all possible paths this is inefficient and > prone to failure. For instance, if we change the set of command > line options used to run a program the path in the tarball will > changes as might the content present at that path. > >> what are these "same configuration file from multiple locations" >> use-cases? It's a feature request we've received to permit sos to function in the case that multiple versions of the same package are installed on the system in different locations. Various distributions permit this in one manner or another so it's something we will have to support very soon. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9eCsUACgkQ6YSQoMYUY95ctACguWPWrHeN0/ZPvraqZ2uos13e x8MAn2bwAnG0wxGyp+l3vNKqja3NEttO =gD8+ -----END PGP SIGNATURE----- From miclark at redhat.com Mon Mar 12 14:45:40 2012 From: miclark at redhat.com (Mike Clark) Date: Mon, 12 Mar 2012 09:45:40 -0500 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5E09C6.6060809@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> <4F5DDA8B.6020300@redhat.com> <4F5DF369.60405@redhat.com> <4F5E0565.3090609@redhat.com> <4F5E09C6.6060809@redhat.com> Message-ID: <4F5E0C14.4040306@redhat.com> Hey folks, On 03/12/2012 09:35 AM, Andrew Hecox wrote: > On 03/12/2012 10:17 AM, Bryn M. Reeves wrote: > On 03/12/2012 01:00 PM, Andrew Hecox wrote: > >>>> tar archives provide a journal/manifest of the names of files, > >>>> and where they are. Because tar is a well known file format, > >>>> there are already language bindings in place for everything > >>>> anyone would want to do. > > Unfortunately this does not solve the problem; as I mentioned before > the tar structure should be considered an ABI. It is subject to change > and is not discoverable in a way that will support upcoming changes in > a robust way. > But, a manifest that just lists the file contents doesn't' provide any benefit over what we already have. What I'm interested in is what "upcoming changes" you envision. That should be driving the need of a manifest. I.e., we don't need a manifest now, we only need it when we have those changes and, thus, the manifest becomes part of those changes, not a separate thing. > > I'd guess to most consumers, the logical layout & content is an API > found through tar bindings for whatever your specific languages are. > > For instance, sos will soon need to have the ability to collect the > same configuration file from multiple locations and to record this > information in the archive. While it would be possible to search all > archives for all possible paths this is inefficient and prone to > failure. For instance, if we change the set of command line options > used to run a program the path in the tarball will changes as might > the content present at that path. > > > what are these "same configuration file from multiple locations" > use-cases? > > > -Ah > > This also cannot address more complex manipulations of the data than a > simple "read this data from this path". In looking at the tools that > consume sos data there is huge overlap and commonality in the > information that they retrieve and the processing that they carry out. > Reinventing this in each project that has overlapping requirements is > not maintainable long-term and will hinder sos's ability to adapt to > meet changing requirements (since there will be pressure to not make > changes that would affect downstream consumers). > Here, too, what are these use-cases? I think that we need a fairly high bar to do any processing beyond "read this data from this path." I.e., nearly all analysis should happen server side. Commonality can happen on the server-side analysis libraries. Also, I'm I also am interested in how opaque you are thinking the report should be. The implication here is that it is completely opaque and should only be accessed through an API. Is that the thought? (If so, I have many reservations.) Cheers, Mike C. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 554 bytes Desc: OpenPGP digital signature URL: From bmr at redhat.com Mon Mar 12 14:59:11 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Mon, 12 Mar 2012 14:59:11 +0000 Subject: [sos-devel] sosreport manifest? In-Reply-To: <4F5E0C14.4040306@redhat.com> References: <4F59E70C.50704@redhat.com> <4F59FCC9.60004@canonical.com> <4F5A05AF.8060302@redhat.com> <20120309201058.GB4168@localhost.localdomain> <4F5A68C3.4000105@redhat.com> <4F5DDA8B.6020300@redhat.com> <4F5DF369.60405@redhat.com> <4F5E0565.3090609@redhat.com> <4F5E09C6.6060809@redhat.com> <4F5E0C14.4040306@redhat.com> Message-ID: <4F5E0F3F.1030409@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/12/2012 02:45 PM, Mike Clark wrote: > But, a manifest that just lists the file contents doesn't' provide > any benefit over what we already have. What I'm interested in is > what If it just listed file content there wouldn't be much point in having a discussion of what it should contain. > "upcoming changes" you envision. That should be driving the need > of a manifest. I.e., we don't need a manifest now, we only need it > when we have those changes and, thus, the manifest becomes part of > those changes, not a separate thing. I disagree. We already have a problem with how consumers access the information in an sosreport tarball. This is about not making that problem worse over time as we make changes and add features rather than supporting any specific additions. One specific change that will drive changes here is the need to have multiple instances of the same configuration file as I mentioned earlier. > Here, too, what are these use-cases? I think that we need a fairly > high bar to do any processing beyond "read this data from this > path." I.e., nearly all analysis should happen server side. > Commonality can happen on the server-side analysis libraries. What server are you talking about? Note that any changes or additions sos makes here do not preclude interested parties from creating their own abstractions however looking at what is out there today that doesn't really seem to be happening (I spent about 20 minutes last week and found three or four different pieces of code to determine the architecture of an sosreport. That is insane). Think about use-cases such as answering questions like: "How many nodes are in this cluster?" "Is this system part of a RHEV environment?" "Is this system virtualized?" "What are the WWIDs of all visible storage devices?" "What is the total set of PCI IDs visible to this host?" "What is the topology of the PCI busses in this system?" "What is the value of property $foo in configuration file $bar?" "How many instances of httpd are configured to run on this host?" "Enumerate the configuration file paths of the configured httpd instances" Etc. > Also, I'm I also am interested in how opaque you are thinking the > report should be. The implication here is that it is completely > opaque and should only be accessed through an API. Is that the > thought? (If so, I have many reservations.) Not at all; downstream users are free to use the data any way that they want. BUT as with any structured data format that does not have a rigid specification consumers that chose to invent their own access methods may be directly exposed to changes that they would have been insulated from by using an API. This is something intended to benefit and simplify the lives of consumers of the data. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9eDz8ACgkQ6YSQoMYUY94OSQCg1y0BxCfnGfWU2cX+IkoSS8HJ jHcAn1ENDNVdewjJI/kcXaDu1Rysws4C =hTwt -----END PGP SIGNATURE----- From jjaggars at redhat.com Tue Mar 13 16:10:32 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Tue, 13 Mar 2012 11:10:32 -0500 Subject: [sos-devel] addCopySpecLimit Message-ID: <20120313161032.GD8869@localhost.localdomain> There is a plugin method called addCopySpecLimit with the following signature: def addCopySpecLimit(self, fname, sizelimit=None, sub=None) It's a fairly complicated function, as far as I can tell it does about three things: 1. If fname is a single file it will attempt to copy the file. - if the file is under sizelimit it adds it via addCopySpec - if the file is over sizelimit it tails the last sizelimit bytes and writes that as an extCommand (via shelling out to tail) 2. If fname is a glob it will attempt to copy (via addCopySpec) all the files, stopping before it gets to sizelimit bytes total. 3. If fname is a glob, but the first file it tries to copy is larger than sizelimit bytes then nothing is added via addCopySpec. There may be some problems with the above (especially #3) so I wanted to start a thread on what the expectations for this function are and what folks think this thing ought to do. Here are my ideas: * control the sorting Currently files found via glob.glob are sorted alphabetically, which is probably not what you want in some cases. It would be nice to be able to sort the files however you wanted, or at least some parameter of os.stat. * trim the first file in a glob if it is too big Rather than copying nothing, it might be better to try to copy the first file (post sort) according to the 'only one file passed in rule'. Both of these can be implemented in the plugin code so these changes aren't top priority I don't suppose. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From bmr at redhat.com Tue Mar 13 16:20:32 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Tue, 13 Mar 2012 16:20:32 +0000 Subject: [sos-devel] addCopySpecLimit In-Reply-To: <20120313161032.GD8869@localhost.localdomain> References: <20120313161032.GD8869@localhost.localdomain> Message-ID: <4F5F73D0.4010305@redhat.com> On 03/13/2012 04:10 PM, Jesse Jaggars wrote: > There is a plugin method called addCopySpecLimit with the following > signature: > > def addCopySpecLimit(self, fname, sizelimit=None, sub=None) > > It's a fairly complicated function, as far as I can tell it does about > three things: Right; it's not such a bad interface but the implementation is lacking at the moment. The intent is clear: copy files matching the copySpec up to the limit but as you noticed things aren't quite written that way today. > 1. If fname is a single file it will attempt to copy the file. > - if the file is under sizelimit it adds it via addCopySpec > - if the file is over sizelimit it tails the last sizelimit bytes and > writes that as an extCommand (via shelling out to tail) > > 2. If fname is a glob it will attempt to copy (via addCopySpec) all the > files, stopping before it gets to sizelimit bytes total. > > 3. If fname is a glob, but the first file it tries to copy is larger > than sizelimit bytes then nothing is added via addCopySpec. > > There may be some problems with the above (especially #3) so I wanted to > start a thread on what the expectations for this function are and what > folks think this thing ought to do. I think it's also worth considering whether addCopySpecLimit() is the best way to approach this. It complicates things even more but we would get more robust behaviour if we kept track of how much data we've accumulated and checked against a limit (either global or per copySpec). > Here are my ideas: > > * control the sorting > > Currently files found via glob.glob are sorted alphabetically, which is > probably not what you want in some cases. It would be nice to be able to > sort the files however you wanted, or at least some parameter of > os.stat. There are some complicated-ish rules to think about if we do something like this. For e.g. determining what is the "best" set of data to capture if we can't get everything. For log files it's not too hard - always get the latest (and they are the only current user of limits). > * trim the first file in a glob if it is too big > > Rather than copying nothing, it might be better to try to copy the first > file (post sort) according to the 'only one file passed in rule'. The current code should attempt this: # Truncate the first file (others would likely be # compressed), # ensuring we get at least some logs if flog == files[0] and limit_reached: self.collectExtOutput("tail -c%d %s" % (sizelimit, flog), "tail_" + os.path.basename(flog)) It works OK in my testing. Regards, Bryn. From jjaggars at redhat.com Tue Mar 13 16:32:43 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Tue, 13 Mar 2012 11:32:43 -0500 Subject: [sos-devel] addCopySpecLimit In-Reply-To: <4F5F73D0.4010305@redhat.com> References: <20120313161032.GD8869@localhost.localdomain> <4F5F73D0.4010305@redhat.com> Message-ID: <20120313163243.GE8869@localhost.localdomain> On Tue, Mar 13, 2012 at 04:20:32PM +0000, Bryn M. Reeves wrote: > On 03/13/2012 04:10 PM, Jesse Jaggars wrote: > > There is a plugin method called addCopySpecLimit with the following > > signature: > > > > def addCopySpecLimit(self, fname, sizelimit=None, sub=None) > > > > It's a fairly complicated function, as far as I can tell it does about > > three things: > > Right; it's not such a bad interface but the implementation is lacking > at the moment. The intent is clear: copy files matching the copySpec up > to the limit but as you noticed things aren't quite written that way today. > > > 1. If fname is a single file it will attempt to copy the file. > > - if the file is under sizelimit it adds it via addCopySpec > > - if the file is over sizelimit it tails the last sizelimit bytes and > > writes that as an extCommand (via shelling out to tail) > > > > 2. If fname is a glob it will attempt to copy (via addCopySpec) all the > > files, stopping before it gets to sizelimit bytes total. > > > > 3. If fname is a glob, but the first file it tries to copy is larger > > than sizelimit bytes then nothing is added via addCopySpec. > > > > There may be some problems with the above (especially #3) so I wanted to > > start a thread on what the expectations for this function are and what > > folks think this thing ought to do. > > I think it's also worth considering whether addCopySpecLimit() is the > best way to approach this. It complicates things even more but we would > get more robust behaviour if we kept track of how much data we've > accumulated and checked against a limit (either global or per copySpec). > This is a good point, and a tough thing to attempt to tackle. We could run into issues where we don't know what is most important (like you talked about with sorting), but then a global option would be configurable and optional. In the worst case we could ask the person generating the report to send along what we lost (at the cost of asking them to do something else.) > > Here are my ideas: > > > > * control the sorting > > > > Currently files found via glob.glob are sorted alphabetically, which is > > probably not what you want in some cases. It would be nice to be able to > > sort the files however you wanted, or at least some parameter of > > os.stat. > > There are some complicated-ish rules to think about if we do something > like this. For e.g. determining what is the "best" set of data to > capture if we can't get everything. For log files it's not too hard - > always get the latest (and they are the only current user of limits). I really mean to propose allowing the plugin author to sort the files how they like. They [should] know best which files are most important. A simple implementation of this would be to allow the caller to pass in a key function to be passed directly to sort(). This, of course, grows the function signature on new optional parameter. > > > * trim the first file in a glob if it is too big > > > > Rather than copying nothing, it might be better to try to copy the first > > file (post sort) according to the 'only one file passed in rule'. > > The current code should attempt this: > > # Truncate the first file (others would likely be > # compressed), > # ensuring we get at least some logs > if flog == files[0] and limit_reached: > self.collectExtOutput("tail -c%d %s" % (sizelimit, flog), > "tail_" + os.path.basename(flog)) > > It works OK in my testing. Yes, you are right. If the first file is too big the function will truncate it. Ignore my earlier complaint. > > Regards, > Bryn. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From dkutalek at redhat.com Tue Mar 13 16:49:29 2012 From: dkutalek at redhat.com (David Kutalek) Date: Tue, 13 Mar 2012 17:49:29 +0100 Subject: [sos-devel] addCopySpecLimit In-Reply-To: <20120313163243.GE8869@localhost.localdomain> References: <20120313161032.GD8869@localhost.localdomain> <4F5F73D0.4010305@redhat.com> <20120313163243.GE8869@localhost.localdomain> Message-ID: <4F5F7A99.8030907@redhat.com> On 03/13/2012 05:32 PM, Jesse Jaggars wrote: > On Tue, Mar 13, 2012 at 04:20:32PM +0000, Bryn M. Reeves wrote: >> On 03/13/2012 04:10 PM, Jesse Jaggars wrote: >>> There is a plugin method called addCopySpecLimit with the following >>> signature: >>> >>> def addCopySpecLimit(self, fname, sizelimit=None, sub=None) >>> >>> It's a fairly complicated function, as far as I can tell it does about >>> three things: >> >> Right; it's not such a bad interface but the implementation is lacking >> at the moment. The intent is clear: copy files matching the copySpec up >> to the limit but as you noticed things aren't quite written that way today. >> >>> 1. If fname is a single file it will attempt to copy the file. >>> - if the file is under sizelimit it adds it via addCopySpec >>> - if the file is over sizelimit it tails the last sizelimit bytes and >>> writes that as an extCommand (via shelling out to tail) >>> >>> 2. If fname is a glob it will attempt to copy (via addCopySpec) all the >>> files, stopping before it gets to sizelimit bytes total. >>> >>> 3. If fname is a glob, but the first file it tries to copy is larger >>> than sizelimit bytes then nothing is added via addCopySpec. >>> >>> There may be some problems with the above (especially #3) so I wanted to >>> start a thread on what the expectations for this function are and what >>> folks think this thing ought to do. >> >> I think it's also worth considering whether addCopySpecLimit() is the >> best way to approach this. It complicates things even more but we would >> get more robust behaviour if we kept track of how much data we've >> accumulated and checked against a limit (either global or per copySpec). >> > > This is a good point, and a tough thing to attempt to tackle. We could > run into issues where we don't know what is most important (like you > talked about with sorting), but then a global option would be > configurable and optional. In the worst case we could ask the person > generating the report to send along what we lost (at the cost of asking > them to do something else.) Seems like all callers of addCopySpecLimit are already setting limit according to some configurable option (on my rhel 6.2 box). I would consider also: - including a simple function to tell what is meaningful to tail - a mode to capture all tail-able files first, with limits to capture all of them and tailing as little as possible of them David Kutalek > >>> Here are my ideas: >>> >>> * control the sorting >>> >>> Currently files found via glob.glob are sorted alphabetically, which is >>> probably not what you want in some cases. It would be nice to be able to >>> sort the files however you wanted, or at least some parameter of >>> os.stat. >> >> There are some complicated-ish rules to think about if we do something >> like this. For e.g. determining what is the "best" set of data to >> capture if we can't get everything. For log files it's not too hard - >> always get the latest (and they are the only current user of limits). > > I really mean to propose allowing the plugin author to sort the files > how they like. They [should] know best which files are most important. A > simple implementation of this would be to allow the caller to pass in a > key function to be passed directly to sort(). This, of course, grows the > function signature on new optional parameter. > >> >>> * trim the first file in a glob if it is too big >>> >>> Rather than copying nothing, it might be better to try to copy the first >>> file (post sort) according to the 'only one file passed in rule'. >> >> The current code should attempt this: >> >> # Truncate the first file (others would likely be >> # compressed), >> # ensuring we get at least some logs >> if flog == files[0] and limit_reached: >> self.collectExtOutput("tail -c%d %s" % (sizelimit, flog), >> "tail_" + os.path.basename(flog)) >> >> It works OK in my testing. > > Yes, you are right. If the first file is too big the function will > truncate it. Ignore my earlier complaint. > >> >> Regards, >> Bryn. > > Jesse > > > > _______________________________________________ > Sos-devel mailing list > Sos-devel at redhat.com > https://www.redhat.com/mailman/listinfo/sos-devel From adam.stokes at canonical.com Mon Mar 19 13:51:53 2012 From: adam.stokes at canonical.com (Adam Stokes) Date: Mon, 19 Mar 2012 09:51:53 -0400 Subject: [sos-devel] sosreport Message-ID: <4F6739F9.5050803@canonical.com> Teodor, I am forwarding your inquiry to sos-devel where a range of upstream developers may answer your question. To better understand the issue could you give us a list of what modules may be hanging? Also I've done some research into a possible way to enable timeouts in subprocess[1]. Once I get some feedback on it and a possible green light we'll have some implemented upstream. However, on the flip side we should probably dig into the modules and see what may be causing the hang. Hello, My name is Teodor Vizirov and I am in the Red Hat GSS team in Brno. I would like to ask you something regarding the sosreport. I tried all knowledge base solutions, but there is no real solution there. My question: Sometimes sosreport hangs at different modules. The workaround is to exclude the module and the sosreport will finish ok. However I have a customer who would like not to exclude the module but to run all modules. Can you give me a hint on what to do to fix the module hanging and to make the sosreport to complete successfully. Thanks in advance. 1: https://github.com/sosreport/sosreport/issues/14 From bmr at redhat.com Mon Mar 19 17:17:43 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Mon, 19 Mar 2012 17:17:43 +0000 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available Message-ID: <4F676A37.6080503@redhat.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Folks, So I thought out a proposal for a "timeout" command to use to address our long-standing problem with commands that hang on us during data collection. Turns out (thanks Eric Blake! ;) that coreutils already introduced this in 7.0 so it's available in all modern distros and should be an easy backport to older stuff. I think this probably hits 80% of the problems we see with "sosreport hangs" reports. The rest is harder but this moves us in the right direction (at the expense of some extra fork/exec overhead). Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9najcACgkQ6YSQoMYUY97qvQCgiKV2y8rcVAiVYuSUqHxqwXXg aLMAn3Pw282YNf3ffGgvS3/J4T4X0FlJ =XbB/ -----END PGP SIGNATURE----- From adam.stokes at canonical.com Tue Mar 20 17:44:25 2012 From: adam.stokes at canonical.com (Adam Stokes) Date: Tue, 20 Mar 2012 13:44:25 -0400 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <4F676A37.6080503@redhat.com> References: <4F676A37.6080503@redhat.com> Message-ID: <4F68C1F9.8030306@canonical.com> On 03/19/2012 01:17 PM, Bryn M. Reeves wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Folks, > > So I thought out a proposal for a "timeout" command to use to address > our long-standing problem with commands that hang on us during data > collection. > > Turns out (thanks Eric Blake! ;) that coreutils already introduced > this in 7.0 so it's available in all modern distros and should be an > easy backport to older stuff. > > I think this probably hits 80% of the problems we see with "sosreport > hangs" reports. The rest is harder but this moves us in the right > direction (at the expense of some extra fork/exec overhead). > > Regards, > Bryn. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAk9najcACgkQ6YSQoMYUY97qvQCgiKV2y8rcVAiVYuSUqHxqwXXg > aLMAn3Pw282YNf3ffGgvS3/J4T4X0FlJ > =XbB/ > -----END PGP SIGNATURE----- > > _______________________________________________ > Sos-devel mailing list > Sos-devel at redhat.com > https://www.redhat.com/mailman/listinfo/sos-devel > If timeout is set with this application does it give control back to sosreport if sending a TERM fails to stop the process? If not, we would need to investigate some asynchronous way to handle those cases in which case my proposed solution may resolve that. Thoughts? Thanks Adam From bmr at redhat.com Tue Mar 20 17:49:14 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Tue, 20 Mar 2012 17:49:14 +0000 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <4F68C1F9.8030306@canonical.com> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> Message-ID: <4F68C31A.3020702@redhat.com> On 03/20/2012 05:44 PM, Adam Stokes wrote: > If timeout is set with this application does it give control back to > sosreport if sending a TERM fails to stop the process? If not, we would > need to investigate some asynchronous way to handle those cases in which > case my proposed solution may resolve that. Yes - timeout effectively runs the child as a daemon which detaches from the controlling process/terminal and then allows the parent to escape the clutches of whatever dragged the child into the swamp. Regards, Bryn. From jjaggars at redhat.com Tue Mar 20 18:21:39 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Tue, 20 Mar 2012 13:21:39 -0500 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <4F68C31A.3020702@redhat.com> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> <4F68C31A.3020702@redhat.com> Message-ID: <20120320182139.GA27103@localhost.localdomain> On Tue, Mar 20, 2012 at 05:49:14PM +0000, Bryn M. Reeves wrote: > On 03/20/2012 05:44 PM, Adam Stokes wrote: > > If timeout is set with this application does it give control back to > > sosreport if sending a TERM fails to stop the process? If not, we would > > need to investigate some asynchronous way to handle those cases in which > > case my proposed solution may resolve that. > > Yes - timeout effectively runs the child as a daemon which detaches from > the controlling process/terminal and then allows the parent to escape > the clutches of whatever dragged the child into the swamp. > > Regards, > Bryn. > Using `timeout` is great for our situation since it allows us to offer a shell to plugins (for easy pipelines). Also, there isn't much extra overhead since we are forking from python anyways. The only major downside is that we will rely on the binary to implement the timeout mechanism for us. This means that we either have to implement alternative timeout methods for platforms that lack `timeout` or fail to supply a timeout on those platforms. This is no worse than we do today, so it's not much of a downside I don't suppose. Given that we will likely want to try to provide an alternative timeout method we should probably plan to plug in different implementations somehow. Currently the method to fork a subprocess and do some work is just a thin wrapper around subprocess.Popen. It would make sense to allow a Policy object to override this implementation, though that might be overkill for now since we can see if `timeout` exists and use it or just do what we do now and hope for the best. Just some thoughts... Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From bmr at redhat.com Tue Mar 20 18:32:25 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Tue, 20 Mar 2012 18:32:25 +0000 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <20120320182139.GA27103@localhost.localdomain> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> <4F68C31A.3020702@redhat.com> <20120320182139.GA27103@localhost.localdomain> Message-ID: <4F68CD39.5060807@redhat.com> On 03/20/2012 06:21 PM, Jesse Jaggars wrote: > Using `timeout` is great for our situation since it allows us to offer a > shell to plugins (for easy pipelines). Also, there isn't much extra overhead > since we are forking from python anyways. That's true actually - I'd not thought of that extra side effect but it's neat. > The only major downside is that we will rely on the binary to implement > the timeout mechanism for us. This means that we either have to > implement alternative timeout methods for platforms that lack `timeout` > or fail to supply a timeout on those platforms. This is no worse than we > do today, so it's not much of a downside I don't suppose. Also true.. I think the fallback approach sounds like a good idea. This would also let us work on distributions that ship pre-7.x coreutils. Although I'd like to get timeout backported into the older Red Hat stuff we can still run into machines with outdated packages. > Given that we will likely want to try to provide an alternative timeout > method we should probably plan to plug in different implementations > somehow. Currently the method to fork a subprocess and do some work is > just a thin wrapper around subprocess.Popen. It would make sense to > allow a Policy object to override this implementation, though that might > be overkill for now since we can see if `timeout` exists and use it or > just do what we do now and hope for the best. If we're going to allow falling back on Linux hosts too I reckon this might be the way to go - policies can then look for a usable binary and decide which way to go. Or maybe it's better in the general plugin support code.. dunno - I think I'd need to see how it looked. Cheers, Bryn. From jjaggars at redhat.com Tue Mar 20 18:39:35 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Tue, 20 Mar 2012 13:39:35 -0500 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <4F68CD39.5060807@redhat.com> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> <4F68C31A.3020702@redhat.com> <20120320182139.GA27103@localhost.localdomain> <4F68CD39.5060807@redhat.com> Message-ID: <20120320183935.GB27103@localhost.localdomain> On Tue, Mar 20, 2012 at 06:32:25PM +0000, Bryn M. Reeves wrote: > On 03/20/2012 06:21 PM, Jesse Jaggars wrote: > > Using `timeout` is great for our situation since it allows us to offer a > > shell to plugins (for easy pipelines). Also, there isn't much extra overhead > > since we are forking from python anyways. > > That's true actually - I'd not thought of that extra side effect but > it's neat. > > > The only major downside is that we will rely on the binary to implement > > the timeout mechanism for us. This means that we either have to > > implement alternative timeout methods for platforms that lack `timeout` > > or fail to supply a timeout on those platforms. This is no worse than we > > do today, so it's not much of a downside I don't suppose. > > Also true.. I think the fallback approach sounds like a good idea. This > would also let us work on distributions that ship pre-7.x coreutils. > Although I'd like to get timeout backported into the older Red Hat stuff > we can still run into machines with outdated packages. > > > Given that we will likely want to try to provide an alternative timeout > > method we should probably plan to plug in different implementations > > somehow. Currently the method to fork a subprocess and do some work is > > just a thin wrapper around subprocess.Popen. It would make sense to > > allow a Policy object to override this implementation, though that might > > be overkill for now since we can see if `timeout` exists and use it or > > just do what we do now and hope for the best. > > If we're going to allow falling back on Linux hosts too I reckon this > might be the way to go - policies can then look for a usable binary and > decide which way to go. Or maybe it's better in the general plugin > support code.. dunno - I think I'd need to see how it looked. > > Cheers, > Bryn. > I think, in the simplest form, this commit shows how we can do this: https://github.com/jhjaggars/sosreport/commit/2300c2e0d61ad6a241e82ffbcfcb72822f1f70ee This patch offers no policy-level customization, but that might not matter right away. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: From adam.stokes at canonical.com Tue Mar 20 18:41:18 2012 From: adam.stokes at canonical.com (Adam Stokes) Date: Tue, 20 Mar 2012 14:41:18 -0400 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <20120320183935.GB27103@localhost.localdomain> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> <4F68C31A.3020702@redhat.com> <20120320182139.GA27103@localhost.localdomain> <4F68CD39.5060807@redhat.com> <20120320183935.GB27103@localhost.localdomain> Message-ID: <4F68CF4E.8050002@canonical.com> On 03/20/2012 02:39 PM, Jesse Jaggars wrote: > On Tue, Mar 20, 2012 at 06:32:25PM +0000, Bryn M. Reeves wrote: >> On 03/20/2012 06:21 PM, Jesse Jaggars wrote: >>> Using `timeout` is great for our situation since it allows us to offer a >>> shell to plugins (for easy pipelines). Also, there isn't much extra overhead >>> since we are forking from python anyways. >> That's true actually - I'd not thought of that extra side effect but >> it's neat. >> >>> The only major downside is that we will rely on the binary to implement >>> the timeout mechanism for us. This means that we either have to >>> implement alternative timeout methods for platforms that lack `timeout` >>> or fail to supply a timeout on those platforms. This is no worse than we >>> do today, so it's not much of a downside I don't suppose. >> Also true.. I think the fallback approach sounds like a good idea. This >> would also let us work on distributions that ship pre-7.x coreutils. >> Although I'd like to get timeout backported into the older Red Hat stuff >> we can still run into machines with outdated packages. >> >>> Given that we will likely want to try to provide an alternative timeout >>> method we should probably plan to plug in different implementations >>> somehow. Currently the method to fork a subprocess and do some work is >>> just a thin wrapper around subprocess.Popen. It would make sense to >>> allow a Policy object to override this implementation, though that might >>> be overkill for now since we can see if `timeout` exists and use it or >>> just do what we do now and hope for the best. >> If we're going to allow falling back on Linux hosts too I reckon this >> might be the way to go - policies can then look for a usable binary and >> decide which way to go. Or maybe it's better in the general plugin >> support code.. dunno - I think I'd need to see how it looked. >> >> Cheers, >> Bryn. >> > I think, in the simplest form, this commit shows how we can do this: > > https://github.com/jhjaggars/sosreport/commit/2300c2e0d61ad6a241e82ffbcfcb72822f1f70ee > > This patch offers no policy-level customization, but that might not > matter right away. > > Jesse looks nice and simple to me :) From adam.stokes at canonical.com Tue Mar 20 18:42:25 2012 From: adam.stokes at canonical.com (Adam Stokes) Date: Tue, 20 Mar 2012 14:42:25 -0400 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <20120320183935.GB27103@localhost.localdomain> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> <4F68C31A.3020702@redhat.com> <20120320182139.GA27103@localhost.localdomain> <4F68CD39.5060807@redhat.com> <20120320183935.GB27103@localhost.localdomain> Message-ID: <4F68CF91.9000509@canonical.com> On 03/20/2012 02:39 PM, Jesse Jaggars wrote: > On Tue, Mar 20, 2012 at 06:32:25PM +0000, Bryn M. Reeves wrote: >> On 03/20/2012 06:21 PM, Jesse Jaggars wrote: >>> Using `timeout` is great for our situation since it allows us to offer a >>> shell to plugins (for easy pipelines). Also, there isn't much extra overhead >>> since we are forking from python anyways. >> That's true actually - I'd not thought of that extra side effect but >> it's neat. >> >>> The only major downside is that we will rely on the binary to implement >>> the timeout mechanism for us. This means that we either have to >>> implement alternative timeout methods for platforms that lack `timeout` >>> or fail to supply a timeout on those platforms. This is no worse than we >>> do today, so it's not much of a downside I don't suppose. >> Also true.. I think the fallback approach sounds like a good idea. This >> would also let us work on distributions that ship pre-7.x coreutils. >> Although I'd like to get timeout backported into the older Red Hat stuff >> we can still run into machines with outdated packages. >> >>> Given that we will likely want to try to provide an alternative timeout >>> method we should probably plan to plug in different implementations >>> somehow. Currently the method to fork a subprocess and do some work is >>> just a thin wrapper around subprocess.Popen. It would make sense to >>> allow a Policy object to override this implementation, though that might >>> be overkill for now since we can see if `timeout` exists and use it or >>> just do what we do now and hope for the best. >> If we're going to allow falling back on Linux hosts too I reckon this >> might be the way to go - policies can then look for a usable binary and >> decide which way to go. Or maybe it's better in the general plugin >> support code.. dunno - I think I'd need to see how it looked. >> >> Cheers, >> Bryn. >> > I think, in the simplest form, this commit shows how we can do this: > > https://github.com/jhjaggars/sosreport/commit/2300c2e0d61ad6a241e82ffbcfcb72822f1f70ee > > This patch offers no policy-level customization, but that might not > matter right away. > > Jesse One thing, maybe add a default timeout param somewhere From bmr at redhat.com Tue Mar 20 18:47:37 2012 From: bmr at redhat.com (Bryn M. Reeves) Date: Tue, 20 Mar 2012 18:47:37 +0000 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <4F68CF4E.8050002@canonical.com> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> <4F68C31A.3020702@redhat.com> <20120320182139.GA27103@localhost.localdomain> <4F68CD39.5060807@redhat.com> <20120320183935.GB27103@localhost.localdomain> <4F68CF4E.8050002@canonical.com> Message-ID: <4F68D0C9.3050007@redhat.com> On 03/20/2012 06:41 PM, Adam Stokes wrote: > On 03/20/2012 02:39 PM, Jesse Jaggars wrote: >> I think, in the simplest form, this commit shows how we can do this: >> >> https://github.com/jhjaggars/sosreport/commit/2300c2e0d61ad6a241e82ffbcfcb72822f1f70ee >> >> This patch offers no policy-level customization, but that might not >> matter right away. >> >> Jesse > looks nice and simple to me :) Ack :) Bryn. From jjaggars at redhat.com Tue Mar 20 18:48:21 2012 From: jjaggars at redhat.com (Jesse Jaggars) Date: Tue, 20 Mar 2012 13:48:21 -0500 Subject: [sos-devel] [RFC] wrap commands in /usr/bin/timeout when available In-Reply-To: <4F68CF91.9000509@canonical.com> References: <4F676A37.6080503@redhat.com> <4F68C1F9.8030306@canonical.com> <4F68C31A.3020702@redhat.com> <20120320182139.GA27103@localhost.localdomain> <4F68CD39.5060807@redhat.com> <20120320183935.GB27103@localhost.localdomain> <4F68CF91.9000509@canonical.com> Message-ID: <20120320184821.GC27103@localhost.localdomain> On Tue, Mar 20, 2012 at 02:42:25PM -0400, Adam Stokes wrote: > On 03/20/2012 02:39 PM, Jesse Jaggars wrote: > >On Tue, Mar 20, 2012 at 06:32:25PM +0000, Bryn M. Reeves wrote: > >>On 03/20/2012 06:21 PM, Jesse Jaggars wrote: > >>>Using `timeout` is great for our situation since it allows us to offer a > >>>shell to plugins (for easy pipelines). Also, there isn't much extra overhead > >>>since we are forking from python anyways. > >>That's true actually - I'd not thought of that extra side effect but > >>it's neat. > >> > >>>The only major downside is that we will rely on the binary to implement > >>>the timeout mechanism for us. This means that we either have to > >>>implement alternative timeout methods for platforms that lack `timeout` > >>>or fail to supply a timeout on those platforms. This is no worse than we > >>>do today, so it's not much of a downside I don't suppose. > >>Also true.. I think the fallback approach sounds like a good idea. This > >>would also let us work on distributions that ship pre-7.x coreutils. > >>Although I'd like to get timeout backported into the older Red Hat stuff > >>we can still run into machines with outdated packages. > >> > >>>Given that we will likely want to try to provide an alternative timeout > >>>method we should probably plan to plug in different implementations > >>>somehow. Currently the method to fork a subprocess and do some work is > >>>just a thin wrapper around subprocess.Popen. It would make sense to > >>>allow a Policy object to override this implementation, though that might > >>>be overkill for now since we can see if `timeout` exists and use it or > >>>just do what we do now and hope for the best. > >>If we're going to allow falling back on Linux hosts too I reckon this > >>might be the way to go - policies can then look for a usable binary and > >>decide which way to go. Or maybe it's better in the general plugin > >>support code.. dunno - I think I'd need to see how it looked. > >> > >>Cheers, > >>Bryn. > >> > >I think, in the simplest form, this commit shows how we can do this: > > > >https://github.com/jhjaggars/sosreport/commit/2300c2e0d61ad6a241e82ffbcfcb72822f1f70ee > > > >This patch offers no policy-level customization, but that might not > >matter right away. > > > >Jesse > One thing, maybe add a default timeout param somewhere I think a default of 300 seconds is set everywhere, execept the (useless IMO) callExtProg. I've added a passthrough for there as well. Jesse -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available URL: