From jspray at redhat.com Sun Oct 2 19:53:42 2016 From: jspray at redhat.com (John Spray) Date: Sun, 2 Oct 2016 20:53:42 +0100 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: On Fri, Sep 30, 2016 at 9:53 PM, Ju Lim wrote: > Hi John: > > Sorry for the slow response ? I was tied up in offsite meetings in the > later part of this week. > > I appreciate all the feedback and have taken some time to think about the > IA a little more. I can?t think of strong enough use cases/user stories to > have snapshots and quotas at the 2nd level seeing that the user would have > other places to trigger related actions/workflows ? specifically from when > they are ?standing? on the Pool/RBD/File Share (storage top-level entity) > itself as well as within the Cluster object details itself. It?s always > in-context of the storage top-level entity or Cluster. The only exception > is for reporting-type use cases across all clusters being managed, but even > that is probably more applicable for quotas for things like > chargebacks/showback, which is probably not a core use case for Tendrl. > I?m glad you and Nishant raised this concern as I think some of the initial > underlying assumptions have evolved especially if there?s a higher level > management tool that leverages what Tendrl collects. > > I've made updates to the document UX Design Approach and Information > Architecture > > to reflect what I stated above (i.e. removal of snapshots and quotas from > the 2nd level nav). > > > Regarding whether it?s worth rethinking using the whole first level and > possibly bringing up to the top-level CephFS, RBD, RGW, etc. This was > something I had thought about early on but lumped everything under Storage > as putting them at the top level might get unwieldy as more things are > added. If we were going to remain static at only supporting Ceph and > Gluster, then it would be okay. However, to support future things that get > added, ideally you don?t want to keep adding them to the top as there > wouldn?t be room to grow and also it might be ?surprising? when the > top-level menu keeps changing and expanding, and eventually wrapping when > we run out of room. I see what you mean, I think this depends on a lot on how multiple storage systems are handled. If we had an install-time sense of which type of system was being managed (e.g. installing particular named module package for the storage system), then it would be reasonable to put everything at the top level as presumably users wouldn't install anything they didn't need. If the console didn't know that at install time, then you'd need to avoid having too much at the top level (although I'm not sure it's that much better to have an unbounded list of subsystems under Storage instead of at the top level). So I guess it would actually be useful to know for context, are users going to see a UI that is configured at install time to show/hide the ceph/gluster specific pieces? Any comments from engineers on this? As a Ceph person I would really like it if we could give Ceph users something that didn't show them any un-needed Gluster stuff. > With regards to user identities, I believe it?s cluster-specific, so that > would get triggered from within the cluster object details, as it wasn?t > also a frequent use case and I assumed typically performed at initial setup > time. This could be handled through an Edit Cluster, and/or optional step > in the Create Cluster workflow, and/or an optional step in the Create RBD / > Pool workflow, and/or a dedicated workflow optimized for configuration of > user identities. Does this make sense? Yes, I think it will naturally pop up in multiple places. Keys belong to a cluster, but creating one is usually contextual to a particular subsystem (crafting the right kind of key for rbd, cephfs, rgw depends which pool, filesystem, etc you want to access). This would actually be a situation where having some metadata in ceph to mark which subsystem a key is intended for would be handy (we don't currently support any "tagging" mechanism for keys, but it has been discussed), so I'd encourage anyone working on key management to think about that early so that we can look at putting anything needed into ceph. I wonder how it was concluded that it wasn't a frequent use case? Don't all users need to create keys to access their cluster? Hopefully folks testing this solution are not all just using their "client.admin" key for everything (akin to logging in as root). John > > Thoughts? > > Thanks, > Ju > > On Wed, Sep 28, 2016 at 7:15 PM, John Spray wrote: > >> On Wed, Sep 28, 2016 at 8:41 PM, Ju Lim wrote: >> > Hi Nishant: >> > >> > Thanks for reviewing. With regards how the dashboard will look when Ceph >> > and Gluster is in the picture: >> > >> > When there is a single storage subsystem present, the single dashboard >> for >> > the single storage subsystem is presented by default. >> > >> > When there are multiple storage subsystems present, the Dashboard would >> > present each dashboard in its own tab. Basically, we'd have a Ceph tab >> and >> > a Gluster tab. >> > >> > Ideally if the default dashboard tab can be specified on a per user >> basis, >> > e.g. in a user?s profile/setting -- initially configured by the >> > Administrator user. It's a nice-to-have at this point, and if folks >> agree, >> > it should be added to the backlog. >> > >> > With regards to unifying concepts such as quotas, file shares, etc., you >> > raise a fair concern. There's multiple approaches to it. For the quotas, >> > snapshots, my current thinking is that when we get to the Quotas or >> > Snapshots section, we would either have a unified view with filtering >> > capabilities (for the different types), or tabbed views, or a 3rd level >> > navigation, or some other combination. This will be determined at a >> later >> > date when the feature gets added. Going with a tabbed approach or 3rd >> level >> > navigation could potentially allow for a plug-in based approach whereby >> > specific storage subsystem capabilities can be exposed as needed. >> > >> > For File Shares, one approach I suggested was to handle it via the user's >> > workflow. Another approach could be to handle it via the navigation which >> > potentially increases complexity and may impact navigation by introducing >> > more levels of navigation. Our goal is not to go deeper than 3 levels of >> > navigation as it will become an unwieldy user experience if things are >> > buried too deeply. That being said, whether it's handled via the >> workflow, >> > navigation, or some other means, using a plug-in based approach should >> still >> > work. >> >> Regarding navigation depth, I wonder if it's worth rethinking using >> the whole first level just to get into "Storage" or "Clusters" -- >> isn't everything we do about storage clusters? In my mind, things >> like cephfs, rbd, rgw are really top-level items. >> >> Regarding unifying quotas and snapshots, I think it would be natural >> for a user to go cephfs->quotas rather than quotas->cephfs, because at >> any given moment they are probably concentrating on one particular >> subsystem. I think the cephfs quotas and cephfs snapshots have more >> in common (and belong closer together) than e.g. cephfs quotas and >> gluster quotas, or cephfs snapshots and rbd snapshots. In a cephfs >> UI, we would probably hope to see a tree view that showed directories >> in the filesystem and enabled things like setting layouts and quotas >> on directories -- picturing that, I don't see a sane way for cephfs >> quotas to live under a top-level Quotas subsystem unless the whole >> widget was duplicated inside the CephFS subsystem as well. >> >> Not trying to entirely derail this into a discussion of CephFS, but I >> think the best way to see if this design is a good fit is to work >> through some of the more challenging cases like the N types of quota >> that will ultimately need managing (not just those supported in the >> first release of the software), and see if what comes out makes sense. >> >> By the way, I was just taking another look at slide 12, and I don't >> see user identities[1] (i.e. the identities that we use for access >> control of Ceph clients, not the Tendrl users). Those should probably >> be in there somewhere. They intersect both the "Cluster" section >> (ceph keys are managed by the mons) and the RBD/RGW/CephFS parts (to >> generate a key with meaningful capabilities you need to know which >> subsystem it's going to be using). >> >> John >> >> 1. http://docs.ceph.com/docs/hammer/rados/operations/user-management/ >> >> >> > I've made updates to the document UX Design Approach and Information >> > Architecture to incorporate the points raised by both yours and John's >> > feedback. >> > >> > Thank you, >> > Ju >> > >> > >> > >> > On Wed, Sep 28, 2016 at 3:55 AM, Nishanth Thomas >> wrote: >> >> >> >> Its good that we are not going to unify the dashboard pieces. With this >> in >> >> mind how the main dashboard looks if we have a deployment with ceph and >> >> Gluster? >> >> Also I have a concern with respect to unifying 'File shares' 'quotas' >> etc, >> >> as it might complicate the navigation. Rather we could look at a plug-in >> >> based approach where file system specific things could be kept >> separately >> >> and installed on a need basis. >> >> >> >> On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: >> >>> >> >>> Hi John: >> >>> >> >>> Thank you for taking the time to review and sharing your feedback. >> >>> >> >>> WRT terminology, I think that's a tricky one and I can understand and >> >>> appreciate the confusion with RBD (vs. Block Device) and File Share. >> The >> >>> original decision to go with RBD was that it was very Ceph-specific in >> terms >> >>> of the implementation and varies from LVM and other block technologies. >> >>> Having said that, I would agree that File Share deviates from this >> initial >> >>> premise since it could mean a Gluster volume or a CephFS fileshare. >> >>> >> >>> The current thinking regarding File Share is to use it for both Ceph >> and >> >>> Gluster, and have a way to qualify it through the notion of the >> workload. >> >>> Part of the rationale was to try to make it so users did not have to >> think >> >>> should I use Ceph or Gluster, but rather let the System make an >> >>> "intelligent" choice for the mere mortal user based on just enough >> >>> information that user provides. This way, the user thinks of it as >> Red Hat >> >>> Storage providing this capability, and if we swapped out Ceph or >> Gluster for >> >>> something in the future, it would allow for that. In conjunction, >> there >> >>> would still be a way for a user to qualify it for Ceph or Gluster as >> well, >> >>> so it's not just for mere mortal but also a more knowledge / ninja >> user. >> >>> >> >>> Having said that, RGW (and Swift) are definitely in mind too, so this >> is >> >>> not limited to just File Shares but applies to pretty much block, >> object, >> >>> and file services. >> >>> >> >>> With regards to quotas and snapshots, I do agree that there are many >> >>> different kinds. It did not seem prudent to create a Pool Snapshot, >> RBD >> >>> Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, >> Pool >> >>> Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not >> >>> scalable (and would get very crowded very quickly), and would create a >> >>> fairly complex experience for the user. My current thinking is that >> when we >> >>> get to the Quotas or Snapshots section, we would either have a unified >> view >> >>> with filtering capabilities (for the different types), or tabbed >> views, or a >> >>> 3rd level navigation, or some other combination. Since quotas and >> snapshots >> >>> are not in the immediate release plans, I've deferred this to the >> detailed >> >>> design stage when we tackle those topics. >> >>> >> >>> I hope this addresses your concerns raised. I'll update the slidedeck >> >>> with these comments before it gets published out more formally. >> >>> >> >>> Thanks again, >> >>> Ju >> >>> >> >>> >> >>> >> >>> On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: >> >>>> >> >>>> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: >> >>>> > Hi. I've started documenting the UX Design Approach and Information >> >>>> > Architecture (JIRA TEN-40). It includes some of our guiding >> >>>> > principles and >> >>>> > decisions that we've made in our design. >> >>>> > >> >>>> > Here's an initial draft that you can take a look at: >> >>>> > >> >>>> > https://docs.google.com/a/redhat.com/presentation/d/ >> 1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing >> >>>> > >> >>>> > Note: For now it's open to be comments by anyone with a Red Hat >> >>>> > account. I >> >>>> > plan to publish this to Jira and/or GitHub before the end of Sprint >> 1 >> >>>> > (Oct >> >>>> > 4), but I'd like to open this up to review, comments, and discussion >> >>>> > in the >> >>>> > meantime. >> >>>> >> >>>> Thanks for posting! >> >>>> >> >>>> My main piece of feedback is on terminology -- I think it's going to >> >>>> get confusing to use a mixture of qualified names (e.g. "RBD" instead >> >>>> of "block device") and unqualified names (e.g. "File Share" when we >> >>>> really mean "Gluster"). >> >>>> >> >>>> I have a special attachment to use of "File Share" because when Tendrl >> >>>> gets support for CephFS, everywhere the term is used is going to need >> >>>> qualifying with whether we're talking about Ceph or Gluster. I don't >> >>>> think anyone is working on adding Tendrl support for CephFS at the >> >>>> moment, but it should be part of the design process, along with RGW. >> >>>> >> >>>> Terms like "Quotas" are also dangerous, because we have so many >> >>>> different kinds. In Ceph alone, we have Quotas on pools, and then a >> >>>> totally different type of quota on directories in cephfs. Same for >> >>>> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. >> >>>> >> >>>> John >> >>>> >> >>>> > Thank you, >> >>>> > Ju Lim >> >>>> > >> >>>> > >> >>>> > _______________________________________________ >> >>>> > Tendrl-devel mailing list >> >>>> > Tendrl-devel at redhat.com >> >>>> > https://www.redhat.com/mailman/listinfo/tendrl-devel >> >>>> > >> >>>> >> >>>> _______________________________________________ >> >>>> Tendrl-devel mailing list >> >>>> Tendrl-devel at redhat.com >> >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Ju Lim >> >>> Red Hat >> >>> Office: 978-399-0422 >> >>> Mobile: 781-507-1323 >> >>> Email: julim at redhat.com >> >>> IRC: julim >> >>> >> >>> >> >>> _______________________________________________ >> >>> Tendrl-devel mailing list >> >>> Tendrl-devel at redhat.com >> >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >>> >> >> >> >> >> >> _______________________________________________ >> >> Tendrl-devel mailing list >> >> Tendrl-devel at redhat.com >> >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> >> > >> > >> > >> > -- >> > Ju Lim >> > Red Hat >> > Office: 978-399-0422 >> > Mobile: 781-507-1323 >> > Email: julim at redhat.com >> > IRC: julim >> > >> > >> > _______________________________________________ >> > Tendrl-devel mailing list >> > Tendrl-devel at redhat.com >> > https://www.redhat.com/mailman/listinfo/tendrl-devel >> > >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > -- > Ju Lim > Red Hat > Office: 978-399-0422 > Mobile: 781-507-1323 > Email: julim at redhat.com > IRC: julim > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From jspray at redhat.com Tue Oct 4 06:08:05 2016 From: jspray at redhat.com (John Spray) Date: Tue, 4 Oct 2016 07:08:05 +0100 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: On Sun, Oct 2, 2016 at 8:53 PM, John Spray wrote: > On Fri, Sep 30, 2016 at 9:53 PM, Ju Lim wrote: >> Hi John: >> >> Sorry for the slow response ? I was tied up in offsite meetings in the >> later part of this week. >> >> I appreciate all the feedback and have taken some time to think about the >> IA a little more. I can?t think of strong enough use cases/user stories to >> have snapshots and quotas at the 2nd level seeing that the user would have >> other places to trigger related actions/workflows ? specifically from when >> they are ?standing? on the Pool/RBD/File Share (storage top-level entity) >> itself as well as within the Cluster object details itself. It?s always >> in-context of the storage top-level entity or Cluster. The only exception >> is for reporting-type use cases across all clusters being managed, but even >> that is probably more applicable for quotas for things like >> chargebacks/showback, which is probably not a core use case for Tendrl. >> I?m glad you and Nishant raised this concern as I think some of the initial >> underlying assumptions have evolved especially if there?s a higher level >> management tool that leverages what Tendrl collects. >> >> I've made updates to the document UX Design Approach and Information >> Architecture >> >> to reflect what I stated above (i.e. removal of snapshots and quotas from >> the 2nd level nav). >> >> >> Regarding whether it?s worth rethinking using the whole first level and >> possibly bringing up to the top-level CephFS, RBD, RGW, etc. This was >> something I had thought about early on but lumped everything under Storage >> as putting them at the top level might get unwieldy as more things are >> added. If we were going to remain static at only supporting Ceph and >> Gluster, then it would be okay. However, to support future things that get >> added, ideally you don?t want to keep adding them to the top as there >> wouldn?t be room to grow and also it might be ?surprising? when the >> top-level menu keeps changing and expanding, and eventually wrapping when >> we run out of room. > > I see what you mean, I think this depends on a lot on how multiple > storage systems are handled. If we had an install-time sense of which > type of system was being managed (e.g. installing particular named > module package for the storage system), then it would be reasonable to > put everything at the top level as presumably users wouldn't install > anything they didn't need. If the console didn't know that at install > time, then you'd need to avoid having too much at the top level > (although I'm not sure it's that much better to have an unbounded list > of subsystems under Storage instead of at the top level). > > So I guess it would actually be useful to know for context, are users > going to see a UI that is configured at install time to show/hide the > ceph/gluster specific pieces? Any comments from engineers on this? > As a Ceph person I would really like it if we could give Ceph users > something that didn't show them any un-needed Gluster stuff. *bump* can anyone tell me the answer? John >> With regards to user identities, I believe it?s cluster-specific, so that >> would get triggered from within the cluster object details, as it wasn?t >> also a frequent use case and I assumed typically performed at initial setup >> time. This could be handled through an Edit Cluster, and/or optional step >> in the Create Cluster workflow, and/or an optional step in the Create RBD / >> Pool workflow, and/or a dedicated workflow optimized for configuration of >> user identities. Does this make sense? > > Yes, I think it will naturally pop up in multiple places. Keys belong > to a cluster, but creating one is usually contextual to a particular > subsystem (crafting the right kind of key for rbd, cephfs, rgw depends > which pool, filesystem, etc you want to access). This would actually > be a situation where having some metadata in ceph to mark which > subsystem a key is intended for would be handy (we don't currently > support any "tagging" mechanism for keys, but it has been discussed), > so I'd encourage anyone working on key management to think about that > early so that we can look at putting anything needed into ceph. > > I wonder how it was concluded that it wasn't a frequent use case? > Don't all users need to create keys to access their cluster? > Hopefully folks testing this solution are not all just using their > "client.admin" key for everything (akin to logging in as root). > > John > >> >> Thoughts? >> >> Thanks, >> Ju >> >> On Wed, Sep 28, 2016 at 7:15 PM, John Spray wrote: >> >>> On Wed, Sep 28, 2016 at 8:41 PM, Ju Lim wrote: >>> > Hi Nishant: >>> > >>> > Thanks for reviewing. With regards how the dashboard will look when Ceph >>> > and Gluster is in the picture: >>> > >>> > When there is a single storage subsystem present, the single dashboard >>> for >>> > the single storage subsystem is presented by default. >>> > >>> > When there are multiple storage subsystems present, the Dashboard would >>> > present each dashboard in its own tab. Basically, we'd have a Ceph tab >>> and >>> > a Gluster tab. >>> > >>> > Ideally if the default dashboard tab can be specified on a per user >>> basis, >>> > e.g. in a user?s profile/setting -- initially configured by the >>> > Administrator user. It's a nice-to-have at this point, and if folks >>> agree, >>> > it should be added to the backlog. >>> > >>> > With regards to unifying concepts such as quotas, file shares, etc., you >>> > raise a fair concern. There's multiple approaches to it. For the quotas, >>> > snapshots, my current thinking is that when we get to the Quotas or >>> > Snapshots section, we would either have a unified view with filtering >>> > capabilities (for the different types), or tabbed views, or a 3rd level >>> > navigation, or some other combination. This will be determined at a >>> later >>> > date when the feature gets added. Going with a tabbed approach or 3rd >>> level >>> > navigation could potentially allow for a plug-in based approach whereby >>> > specific storage subsystem capabilities can be exposed as needed. >>> > >>> > For File Shares, one approach I suggested was to handle it via the user's >>> > workflow. Another approach could be to handle it via the navigation which >>> > potentially increases complexity and may impact navigation by introducing >>> > more levels of navigation. Our goal is not to go deeper than 3 levels of >>> > navigation as it will become an unwieldy user experience if things are >>> > buried too deeply. That being said, whether it's handled via the >>> workflow, >>> > navigation, or some other means, using a plug-in based approach should >>> still >>> > work. >>> >>> Regarding navigation depth, I wonder if it's worth rethinking using >>> the whole first level just to get into "Storage" or "Clusters" -- >>> isn't everything we do about storage clusters? In my mind, things >>> like cephfs, rbd, rgw are really top-level items. >>> >>> Regarding unifying quotas and snapshots, I think it would be natural >>> for a user to go cephfs->quotas rather than quotas->cephfs, because at >>> any given moment they are probably concentrating on one particular >>> subsystem. I think the cephfs quotas and cephfs snapshots have more >>> in common (and belong closer together) than e.g. cephfs quotas and >>> gluster quotas, or cephfs snapshots and rbd snapshots. In a cephfs >>> UI, we would probably hope to see a tree view that showed directories >>> in the filesystem and enabled things like setting layouts and quotas >>> on directories -- picturing that, I don't see a sane way for cephfs >>> quotas to live under a top-level Quotas subsystem unless the whole >>> widget was duplicated inside the CephFS subsystem as well. >>> >>> Not trying to entirely derail this into a discussion of CephFS, but I >>> think the best way to see if this design is a good fit is to work >>> through some of the more challenging cases like the N types of quota >>> that will ultimately need managing (not just those supported in the >>> first release of the software), and see if what comes out makes sense. >>> >>> By the way, I was just taking another look at slide 12, and I don't >>> see user identities[1] (i.e. the identities that we use for access >>> control of Ceph clients, not the Tendrl users). Those should probably >>> be in there somewhere. They intersect both the "Cluster" section >>> (ceph keys are managed by the mons) and the RBD/RGW/CephFS parts (to >>> generate a key with meaningful capabilities you need to know which >>> subsystem it's going to be using). >>> >>> John >>> >>> 1. http://docs.ceph.com/docs/hammer/rados/operations/user-management/ >>> >>> >>> > I've made updates to the document UX Design Approach and Information >>> > Architecture to incorporate the points raised by both yours and John's >>> > feedback. >>> > >>> > Thank you, >>> > Ju >>> > >>> > >>> > >>> > On Wed, Sep 28, 2016 at 3:55 AM, Nishanth Thomas >>> wrote: >>> >> >>> >> Its good that we are not going to unify the dashboard pieces. With this >>> in >>> >> mind how the main dashboard looks if we have a deployment with ceph and >>> >> Gluster? >>> >> Also I have a concern with respect to unifying 'File shares' 'quotas' >>> etc, >>> >> as it might complicate the navigation. Rather we could look at a plug-in >>> >> based approach where file system specific things could be kept >>> separately >>> >> and installed on a need basis. >>> >> >>> >> On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: >>> >>> >>> >>> Hi John: >>> >>> >>> >>> Thank you for taking the time to review and sharing your feedback. >>> >>> >>> >>> WRT terminology, I think that's a tricky one and I can understand and >>> >>> appreciate the confusion with RBD (vs. Block Device) and File Share. >>> The >>> >>> original decision to go with RBD was that it was very Ceph-specific in >>> terms >>> >>> of the implementation and varies from LVM and other block technologies. >>> >>> Having said that, I would agree that File Share deviates from this >>> initial >>> >>> premise since it could mean a Gluster volume or a CephFS fileshare. >>> >>> >>> >>> The current thinking regarding File Share is to use it for both Ceph >>> and >>> >>> Gluster, and have a way to qualify it through the notion of the >>> workload. >>> >>> Part of the rationale was to try to make it so users did not have to >>> think >>> >>> should I use Ceph or Gluster, but rather let the System make an >>> >>> "intelligent" choice for the mere mortal user based on just enough >>> >>> information that user provides. This way, the user thinks of it as >>> Red Hat >>> >>> Storage providing this capability, and if we swapped out Ceph or >>> Gluster for >>> >>> something in the future, it would allow for that. In conjunction, >>> there >>> >>> would still be a way for a user to qualify it for Ceph or Gluster as >>> well, >>> >>> so it's not just for mere mortal but also a more knowledge / ninja >>> user. >>> >>> >>> >>> Having said that, RGW (and Swift) are definitely in mind too, so this >>> is >>> >>> not limited to just File Shares but applies to pretty much block, >>> object, >>> >>> and file services. >>> >>> >>> >>> With regards to quotas and snapshots, I do agree that there are many >>> >>> different kinds. It did not seem prudent to create a Pool Snapshot, >>> RBD >>> >>> Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, >>> Pool >>> >>> Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not >>> >>> scalable (and would get very crowded very quickly), and would create a >>> >>> fairly complex experience for the user. My current thinking is that >>> when we >>> >>> get to the Quotas or Snapshots section, we would either have a unified >>> view >>> >>> with filtering capabilities (for the different types), or tabbed >>> views, or a >>> >>> 3rd level navigation, or some other combination. Since quotas and >>> snapshots >>> >>> are not in the immediate release plans, I've deferred this to the >>> detailed >>> >>> design stage when we tackle those topics. >>> >>> >>> >>> I hope this addresses your concerns raised. I'll update the slidedeck >>> >>> with these comments before it gets published out more formally. >>> >>> >>> >>> Thanks again, >>> >>> Ju >>> >>> >>> >>> >>> >>> >>> >>> On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: >>> >>>> >>> >>>> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: >>> >>>> > Hi. I've started documenting the UX Design Approach and Information >>> >>>> > Architecture (JIRA TEN-40). It includes some of our guiding >>> >>>> > principles and >>> >>>> > decisions that we've made in our design. >>> >>>> > >>> >>>> > Here's an initial draft that you can take a look at: >>> >>>> > >>> >>>> > https://docs.google.com/a/redhat.com/presentation/d/ >>> 1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing >>> >>>> > >>> >>>> > Note: For now it's open to be comments by anyone with a Red Hat >>> >>>> > account. I >>> >>>> > plan to publish this to Jira and/or GitHub before the end of Sprint >>> 1 >>> >>>> > (Oct >>> >>>> > 4), but I'd like to open this up to review, comments, and discussion >>> >>>> > in the >>> >>>> > meantime. >>> >>>> >>> >>>> Thanks for posting! >>> >>>> >>> >>>> My main piece of feedback is on terminology -- I think it's going to >>> >>>> get confusing to use a mixture of qualified names (e.g. "RBD" instead >>> >>>> of "block device") and unqualified names (e.g. "File Share" when we >>> >>>> really mean "Gluster"). >>> >>>> >>> >>>> I have a special attachment to use of "File Share" because when Tendrl >>> >>>> gets support for CephFS, everywhere the term is used is going to need >>> >>>> qualifying with whether we're talking about Ceph or Gluster. I don't >>> >>>> think anyone is working on adding Tendrl support for CephFS at the >>> >>>> moment, but it should be part of the design process, along with RGW. >>> >>>> >>> >>>> Terms like "Quotas" are also dangerous, because we have so many >>> >>>> different kinds. In Ceph alone, we have Quotas on pools, and then a >>> >>>> totally different type of quota on directories in cephfs. Same for >>> >>>> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. >>> >>>> >>> >>>> John >>> >>>> >>> >>>> > Thank you, >>> >>>> > Ju Lim >>> >>>> > >>> >>>> > >>> >>>> > _______________________________________________ >>> >>>> > Tendrl-devel mailing list >>> >>>> > Tendrl-devel at redhat.com >>> >>>> > https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >>>> > >>> >>>> >>> >>>> _______________________________________________ >>> >>>> Tendrl-devel mailing list >>> >>>> Tendrl-devel at redhat.com >>> >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Ju Lim >>> >>> Red Hat >>> >>> Office: 978-399-0422 >>> >>> Mobile: 781-507-1323 >>> >>> Email: julim at redhat.com >>> >>> IRC: julim >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> Tendrl-devel mailing list >>> >>> Tendrl-devel at redhat.com >>> >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >>> >>> >> >>> >> >>> >> _______________________________________________ >>> >> Tendrl-devel mailing list >>> >> Tendrl-devel at redhat.com >>> >> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >> >>> > >>> > >>> > >>> > -- >>> > Ju Lim >>> > Red Hat >>> > Office: 978-399-0422 >>> > Mobile: 781-507-1323 >>> > Email: julim at redhat.com >>> > IRC: julim >>> > >>> > >>> > _______________________________________________ >>> > Tendrl-devel mailing list >>> > Tendrl-devel at redhat.com >>> > https://www.redhat.com/mailman/listinfo/tendrl-devel >>> > >>> >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >> >> >> >> -- >> Ju Lim >> Red Hat >> Office: 978-399-0422 >> Mobile: 781-507-1323 >> Email: julim at redhat.com >> IRC: julim >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel From mrugesh at brainfunked.org Tue Oct 4 07:30:09 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 4 Oct 2016 13:00:09 +0530 Subject: [Tendrl-devel] Installing Ceph via Tendrl In-Reply-To: References: Message-ID: On 1 October 2016 at 02:31, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > > On Sat, Oct 1, 2016 at 12:53 AM, Alfredo Deza wrote: > > All of that will have to be done again in Tendrl. Can that design be > > explained a bit? How is this generalized approach going to tackle > > these issues? How is it going to approach tackling any other Ansible > > playbook or role? > > > > How has the ceph-installer failed here? When the project started, it > > did so with a "documentation first" approach, and we presented the API > > first so that the consumers of it could have an immediate idea on the > > HTTP API proposed. We did it this way because we thought it was > > imperative to present the ideas while the application itself was being > > crafted to improve the throughput on communication. > > It is a bit late/early here - the detail about the provisioning flow > is (as I remember) part of this sprint or, a coming one and is being > worked on. Perhaps we could provide an interim update and then take > this conversation ahead. I'm working on this in the current sprint. Updates will be made to the issue at https://github.com/Tendrl/documentation/issues/29 Also, Alfredo, do you think we can have a quick conversation somewhere towards the end of this week? Thanks, Mrugesh From rwheeler at redhat.com Tue Oct 4 07:36:00 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Tue, 4 Oct 2016 08:36:00 +0100 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> On 10/04/2016 07:08 AM, John Spray wrote: > On Sun, Oct 2, 2016 at 8:53 PM, John Spray wrote: >> On Fri, Sep 30, 2016 at 9:53 PM, Ju Lim wrote: >>> Hi John: >>> >>> Sorry for the slow response ? I was tied up in offsite meetings in the >>> later part of this week. >>> >>> I appreciate all the feedback and have taken some time to think about the >>> IA a little more. I can?t think of strong enough use cases/user stories to >>> have snapshots and quotas at the 2nd level seeing that the user would have >>> other places to trigger related actions/workflows ? specifically from when >>> they are ?standing? on the Pool/RBD/File Share (storage top-level entity) >>> itself as well as within the Cluster object details itself. It?s always >>> in-context of the storage top-level entity or Cluster. The only exception >>> is for reporting-type use cases across all clusters being managed, but even >>> that is probably more applicable for quotas for things like >>> chargebacks/showback, which is probably not a core use case for Tendrl. >>> I?m glad you and Nishant raised this concern as I think some of the initial >>> underlying assumptions have evolved especially if there?s a higher level >>> management tool that leverages what Tendrl collects. >>> >>> I've made updates to the document UX Design Approach and Information >>> Architecture >>> >>> to reflect what I stated above (i.e. removal of snapshots and quotas from >>> the 2nd level nav). >>> >>> >>> Regarding whether it?s worth rethinking using the whole first level and >>> possibly bringing up to the top-level CephFS, RBD, RGW, etc. This was >>> something I had thought about early on but lumped everything under Storage >>> as putting them at the top level might get unwieldy as more things are >>> added. If we were going to remain static at only supporting Ceph and >>> Gluster, then it would be okay. However, to support future things that get >>> added, ideally you don?t want to keep adding them to the top as there >>> wouldn?t be room to grow and also it might be ?surprising? when the >>> top-level menu keeps changing and expanding, and eventually wrapping when >>> we run out of room. >> I see what you mean, I think this depends on a lot on how multiple >> storage systems are handled. If we had an install-time sense of which >> type of system was being managed (e.g. installing particular named >> module package for the storage system), then it would be reasonable to >> put everything at the top level as presumably users wouldn't install >> anything they didn't need. If the console didn't know that at install >> time, then you'd need to avoid having too much at the top level >> (although I'm not sure it's that much better to have an unbounded list >> of subsystems under Storage instead of at the top level). >> >> So I guess it would actually be useful to know for context, are users >> going to see a UI that is configured at install time to show/hide the >> ceph/gluster specific pieces? Any comments from engineers on this? >> As a Ceph person I would really like it if we could give Ceph users >> something that didn't show them any un-needed Gluster stuff. > *bump* can anyone tell me the answer? > > John Hi John, For the UX bits that Red Hat developers are working on, I don't see us having different top levels for gluster and ceph. Not something certainly that we tried for in the current management application. Other projects of course can build something ceph (or gluster) specific for their UX top bits though. Regards, Ric > >>> With regards to user identities, I believe it?s cluster-specific, so that >>> would get triggered from within the cluster object details, as it wasn?t >>> also a frequent use case and I assumed typically performed at initial setup >>> time. This could be handled through an Edit Cluster, and/or optional step >>> in the Create Cluster workflow, and/or an optional step in the Create RBD / >>> Pool workflow, and/or a dedicated workflow optimized for configuration of >>> user identities. Does this make sense? >> Yes, I think it will naturally pop up in multiple places. Keys belong >> to a cluster, but creating one is usually contextual to a particular >> subsystem (crafting the right kind of key for rbd, cephfs, rgw depends >> which pool, filesystem, etc you want to access). This would actually >> be a situation where having some metadata in ceph to mark which >> subsystem a key is intended for would be handy (we don't currently >> support any "tagging" mechanism for keys, but it has been discussed), >> so I'd encourage anyone working on key management to think about that >> early so that we can look at putting anything needed into ceph. >> >> I wonder how it was concluded that it wasn't a frequent use case? >> Don't all users need to create keys to access their cluster? >> Hopefully folks testing this solution are not all just using their >> "client.admin" key for everything (akin to logging in as root). >> >> John >> >>> Thoughts? >>> >>> Thanks, >>> Ju >>> >>> On Wed, Sep 28, 2016 at 7:15 PM, John Spray wrote: >>> >>>> On Wed, Sep 28, 2016 at 8:41 PM, Ju Lim wrote: >>>>> Hi Nishant: >>>>> >>>>> Thanks for reviewing. With regards how the dashboard will look when Ceph >>>>> and Gluster is in the picture: >>>>> >>>>> When there is a single storage subsystem present, the single dashboard >>>> for >>>>> the single storage subsystem is presented by default. >>>>> >>>>> When there are multiple storage subsystems present, the Dashboard would >>>>> present each dashboard in its own tab. Basically, we'd have a Ceph tab >>>> and >>>>> a Gluster tab. >>>>> >>>>> Ideally if the default dashboard tab can be specified on a per user >>>> basis, >>>>> e.g. in a user?s profile/setting -- initially configured by the >>>>> Administrator user. It's a nice-to-have at this point, and if folks >>>> agree, >>>>> it should be added to the backlog. >>>>> >>>>> With regards to unifying concepts such as quotas, file shares, etc., you >>>>> raise a fair concern. There's multiple approaches to it. For the quotas, >>>>> snapshots, my current thinking is that when we get to the Quotas or >>>>> Snapshots section, we would either have a unified view with filtering >>>>> capabilities (for the different types), or tabbed views, or a 3rd level >>>>> navigation, or some other combination. This will be determined at a >>>> later >>>>> date when the feature gets added. Going with a tabbed approach or 3rd >>>> level >>>>> navigation could potentially allow for a plug-in based approach whereby >>>>> specific storage subsystem capabilities can be exposed as needed. >>>>> >>>>> For File Shares, one approach I suggested was to handle it via the user's >>>>> workflow. Another approach could be to handle it via the navigation which >>>>> potentially increases complexity and may impact navigation by introducing >>>>> more levels of navigation. Our goal is not to go deeper than 3 levels of >>>>> navigation as it will become an unwieldy user experience if things are >>>>> buried too deeply. That being said, whether it's handled via the >>>> workflow, >>>>> navigation, or some other means, using a plug-in based approach should >>>> still >>>>> work. >>>> Regarding navigation depth, I wonder if it's worth rethinking using >>>> the whole first level just to get into "Storage" or "Clusters" -- >>>> isn't everything we do about storage clusters? In my mind, things >>>> like cephfs, rbd, rgw are really top-level items. >>>> >>>> Regarding unifying quotas and snapshots, I think it would be natural >>>> for a user to go cephfs->quotas rather than quotas->cephfs, because at >>>> any given moment they are probably concentrating on one particular >>>> subsystem. I think the cephfs quotas and cephfs snapshots have more >>>> in common (and belong closer together) than e.g. cephfs quotas and >>>> gluster quotas, or cephfs snapshots and rbd snapshots. In a cephfs >>>> UI, we would probably hope to see a tree view that showed directories >>>> in the filesystem and enabled things like setting layouts and quotas >>>> on directories -- picturing that, I don't see a sane way for cephfs >>>> quotas to live under a top-level Quotas subsystem unless the whole >>>> widget was duplicated inside the CephFS subsystem as well. >>>> >>>> Not trying to entirely derail this into a discussion of CephFS, but I >>>> think the best way to see if this design is a good fit is to work >>>> through some of the more challenging cases like the N types of quota >>>> that will ultimately need managing (not just those supported in the >>>> first release of the software), and see if what comes out makes sense. >>>> >>>> By the way, I was just taking another look at slide 12, and I don't >>>> see user identities[1] (i.e. the identities that we use for access >>>> control of Ceph clients, not the Tendrl users). Those should probably >>>> be in there somewhere. They intersect both the "Cluster" section >>>> (ceph keys are managed by the mons) and the RBD/RGW/CephFS parts (to >>>> generate a key with meaningful capabilities you need to know which >>>> subsystem it's going to be using). >>>> >>>> John >>>> >>>> 1. http://docs.ceph.com/docs/hammer/rados/operations/user-management/ >>>> >>>> >>>>> I've made updates to the document UX Design Approach and Information >>>>> Architecture to incorporate the points raised by both yours and John's >>>>> feedback. >>>>> >>>>> Thank you, >>>>> Ju >>>>> >>>>> >>>>> >>>>> On Wed, Sep 28, 2016 at 3:55 AM, Nishanth Thomas >>>> wrote: >>>>>> Its good that we are not going to unify the dashboard pieces. With this >>>> in >>>>>> mind how the main dashboard looks if we have a deployment with ceph and >>>>>> Gluster? >>>>>> Also I have a concern with respect to unifying 'File shares' 'quotas' >>>> etc, >>>>>> as it might complicate the navigation. Rather we could look at a plug-in >>>>>> based approach where file system specific things could be kept >>>> separately >>>>>> and installed on a need basis. >>>>>> >>>>>> On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: >>>>>>> Hi John: >>>>>>> >>>>>>> Thank you for taking the time to review and sharing your feedback. >>>>>>> >>>>>>> WRT terminology, I think that's a tricky one and I can understand and >>>>>>> appreciate the confusion with RBD (vs. Block Device) and File Share. >>>> The >>>>>>> original decision to go with RBD was that it was very Ceph-specific in >>>> terms >>>>>>> of the implementation and varies from LVM and other block technologies. >>>>>>> Having said that, I would agree that File Share deviates from this >>>> initial >>>>>>> premise since it could mean a Gluster volume or a CephFS fileshare. >>>>>>> >>>>>>> The current thinking regarding File Share is to use it for both Ceph >>>> and >>>>>>> Gluster, and have a way to qualify it through the notion of the >>>> workload. >>>>>>> Part of the rationale was to try to make it so users did not have to >>>> think >>>>>>> should I use Ceph or Gluster, but rather let the System make an >>>>>>> "intelligent" choice for the mere mortal user based on just enough >>>>>>> information that user provides. This way, the user thinks of it as >>>> Red Hat >>>>>>> Storage providing this capability, and if we swapped out Ceph or >>>> Gluster for >>>>>>> something in the future, it would allow for that. In conjunction, >>>> there >>>>>>> would still be a way for a user to qualify it for Ceph or Gluster as >>>> well, >>>>>>> so it's not just for mere mortal but also a more knowledge / ninja >>>> user. >>>>>>> Having said that, RGW (and Swift) are definitely in mind too, so this >>>> is >>>>>>> not limited to just File Shares but applies to pretty much block, >>>> object, >>>>>>> and file services. >>>>>>> >>>>>>> With regards to quotas and snapshots, I do agree that there are many >>>>>>> different kinds. It did not seem prudent to create a Pool Snapshot, >>>> RBD >>>>>>> Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, >>>> Pool >>>>>>> Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not >>>>>>> scalable (and would get very crowded very quickly), and would create a >>>>>>> fairly complex experience for the user. My current thinking is that >>>> when we >>>>>>> get to the Quotas or Snapshots section, we would either have a unified >>>> view >>>>>>> with filtering capabilities (for the different types), or tabbed >>>> views, or a >>>>>>> 3rd level navigation, or some other combination. Since quotas and >>>> snapshots >>>>>>> are not in the immediate release plans, I've deferred this to the >>>> detailed >>>>>>> design stage when we tackle those topics. >>>>>>> >>>>>>> I hope this addresses your concerns raised. I'll update the slidedeck >>>>>>> with these comments before it gets published out more formally. >>>>>>> >>>>>>> Thanks again, >>>>>>> Ju >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: >>>>>>>> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: >>>>>>>>> Hi. I've started documenting the UX Design Approach and Information >>>>>>>>> Architecture (JIRA TEN-40). It includes some of our guiding >>>>>>>>> principles and >>>>>>>>> decisions that we've made in our design. >>>>>>>>> >>>>>>>>> Here's an initial draft that you can take a look at: >>>>>>>>> >>>>>>>>> https://docs.google.com/a/redhat.com/presentation/d/ >>>> 1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing >>>>>>>>> Note: For now it's open to be comments by anyone with a Red Hat >>>>>>>>> account. I >>>>>>>>> plan to publish this to Jira and/or GitHub before the end of Sprint >>>> 1 >>>>>>>>> (Oct >>>>>>>>> 4), but I'd like to open this up to review, comments, and discussion >>>>>>>>> in the >>>>>>>>> meantime. >>>>>>>> Thanks for posting! >>>>>>>> >>>>>>>> My main piece of feedback is on terminology -- I think it's going to >>>>>>>> get confusing to use a mixture of qualified names (e.g. "RBD" instead >>>>>>>> of "block device") and unqualified names (e.g. "File Share" when we >>>>>>>> really mean "Gluster"). >>>>>>>> >>>>>>>> I have a special attachment to use of "File Share" because when Tendrl >>>>>>>> gets support for CephFS, everywhere the term is used is going to need >>>>>>>> qualifying with whether we're talking about Ceph or Gluster. I don't >>>>>>>> think anyone is working on adding Tendrl support for CephFS at the >>>>>>>> moment, but it should be part of the design process, along with RGW. >>>>>>>> >>>>>>>> Terms like "Quotas" are also dangerous, because we have so many >>>>>>>> different kinds. In Ceph alone, we have Quotas on pools, and then a >>>>>>>> totally different type of quota on directories in cephfs. Same for >>>>>>>> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. >>>>>>>> >>>>>>>> John >>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Ju Lim >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Tendrl-devel mailing list >>>>>>>>> Tendrl-devel at redhat.com >>>>>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Tendrl-devel mailing list >>>>>>>> Tendrl-devel at redhat.com >>>>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Ju Lim >>>>>>> Red Hat >>>>>>> Office: 978-399-0422 >>>>>>> Mobile: 781-507-1323 >>>>>>> Email: julim at redhat.com >>>>>>> IRC: julim >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Tendrl-devel mailing list >>>>>>> Tendrl-devel at redhat.com >>>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Tendrl-devel mailing list >>>>>> Tendrl-devel at redhat.com >>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>> >>>>> >>>>> >>>>> -- >>>>> Ju Lim >>>>> Red Hat >>>>> Office: 978-399-0422 >>>>> Mobile: 781-507-1323 >>>>> Email: julim at redhat.com >>>>> IRC: julim >>>>> >>>>> >>>>> _______________________________________________ >>>>> Tendrl-devel mailing list >>>>> Tendrl-devel at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>> >>>> _______________________________________________ >>>> Tendrl-devel mailing list >>>> Tendrl-devel at redhat.com >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>> >>> >>> >>> -- >>> Ju Lim >>> Red Hat >>> Office: 978-399-0422 >>> Mobile: 781-507-1323 >>> Email: julim at redhat.com >>> IRC: julim >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From jspray at redhat.com Tue Oct 4 07:47:34 2016 From: jspray at redhat.com (John Spray) Date: Tue, 4 Oct 2016 08:47:34 +0100 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> References: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> Message-ID: On Tue, Oct 4, 2016 at 8:36 AM, Ric Wheeler wrote: > On 10/04/2016 07:08 AM, John Spray wrote: >> >> On Sun, Oct 2, 2016 at 8:53 PM, John Spray wrote: >>> >>> On Fri, Sep 30, 2016 at 9:53 PM, Ju Lim wrote: >>>> >>>> Hi John: >>>> >>>> Sorry for the slow response ? I was tied up in offsite meetings in the >>>> later part of this week. >>>> >>>> I appreciate all the feedback and have taken some time to think about >>>> the >>>> IA a little more. I can?t think of strong enough use cases/user stories >>>> to >>>> have snapshots and quotas at the 2nd level seeing that the user would >>>> have >>>> other places to trigger related actions/workflows ? specifically from >>>> when >>>> they are ?standing? on the Pool/RBD/File Share (storage top-level >>>> entity) >>>> itself as well as within the Cluster object details itself. It?s always >>>> in-context of the storage top-level entity or Cluster. The only >>>> exception >>>> is for reporting-type use cases across all clusters being managed, but >>>> even >>>> that is probably more applicable for quotas for things like >>>> chargebacks/showback, which is probably not a core use case for Tendrl. >>>> I?m glad you and Nishant raised this concern as I think some of the >>>> initial >>>> underlying assumptions have evolved especially if there?s a higher level >>>> management tool that leverages what Tendrl collects. >>>> >>>> I've made updates to the document UX Design Approach and Information >>>> Architecture >>>> >>>> >>>> to reflect what I stated above (i.e. removal of snapshots and quotas >>>> from >>>> the 2nd level nav). >>>> >>>> >>>> Regarding whether it?s worth rethinking using the whole first level and >>>> possibly bringing up to the top-level CephFS, RBD, RGW, etc. This was >>>> something I had thought about early on but lumped everything under >>>> Storage >>>> as putting them at the top level might get unwieldy as more things are >>>> added. If we were going to remain static at only supporting Ceph and >>>> Gluster, then it would be okay. However, to support future things that >>>> get >>>> added, ideally you don?t want to keep adding them to the top as there >>>> wouldn?t be room to grow and also it might be ?surprising? when the >>>> top-level menu keeps changing and expanding, and eventually wrapping >>>> when >>>> we run out of room. >>> >>> I see what you mean, I think this depends on a lot on how multiple >>> storage systems are handled. If we had an install-time sense of which >>> type of system was being managed (e.g. installing particular named >>> module package for the storage system), then it would be reasonable to >>> put everything at the top level as presumably users wouldn't install >>> anything they didn't need. If the console didn't know that at install >>> time, then you'd need to avoid having too much at the top level >>> (although I'm not sure it's that much better to have an unbounded list >>> of subsystems under Storage instead of at the top level). >>> >>> So I guess it would actually be useful to know for context, are users >>> going to see a UI that is configured at install time to show/hide the >>> ceph/gluster specific pieces? Any comments from engineers on this? >>> As a Ceph person I would really like it if we could give Ceph users >>> something that didn't show them any un-needed Gluster stuff. >> >> *bump* can anyone tell me the answer? >> >> John > > > Hi John, > > For the UX bits that Red Hat developers are working on, I don't see us > having different top levels for gluster and ceph. Not something certainly > that we tried for in the current management application. What I was really asking was whether at install time the console will have any knowledge of whether it's going to manage Ceph or Gluster, so that we can hide parts that aren't needed. I'm wondering if we're going to see e.g. a "tendrl-ceph" and "tendrl-gluster" package so that people can get just the bits they need. The question of what items belong at the top level would depend on that. Given that (I assert) the majority of users do not mix up ceph and gluster in one system, it seems like a big usability win if they can have a Gluster UI without Ceph bits in it, or vice versa. John > Other projects of course can build something ceph (or gluster) specific for > their UX top bits though. > > Regards, > Ric > > >> >>>> With regards to user identities, I believe it?s cluster-specific, so >>>> that >>>> would get triggered from within the cluster object details, as it wasn?t >>>> also a frequent use case and I assumed typically performed at initial >>>> setup >>>> time. This could be handled through an Edit Cluster, and/or optional >>>> step >>>> in the Create Cluster workflow, and/or an optional step in the Create >>>> RBD / >>>> Pool workflow, and/or a dedicated workflow optimized for configuration >>>> of >>>> user identities. Does this make sense? >>> >>> Yes, I think it will naturally pop up in multiple places. Keys belong >>> to a cluster, but creating one is usually contextual to a particular >>> subsystem (crafting the right kind of key for rbd, cephfs, rgw depends >>> which pool, filesystem, etc you want to access). This would actually >>> be a situation where having some metadata in ceph to mark which >>> subsystem a key is intended for would be handy (we don't currently >>> support any "tagging" mechanism for keys, but it has been discussed), >>> so I'd encourage anyone working on key management to think about that >>> early so that we can look at putting anything needed into ceph. >>> >>> I wonder how it was concluded that it wasn't a frequent use case? >>> Don't all users need to create keys to access their cluster? >>> Hopefully folks testing this solution are not all just using their >>> "client.admin" key for everything (akin to logging in as root). >>> >>> John >>> >>>> Thoughts? >>>> >>>> Thanks, >>>> Ju >>>> >>>> On Wed, Sep 28, 2016 at 7:15 PM, John Spray wrote: >>>> >>>>> On Wed, Sep 28, 2016 at 8:41 PM, Ju Lim wrote: >>>>>> >>>>>> Hi Nishant: >>>>>> >>>>>> Thanks for reviewing. With regards how the dashboard will look when >>>>>> Ceph >>>>>> and Gluster is in the picture: >>>>>> >>>>>> When there is a single storage subsystem present, the single dashboard >>>>> >>>>> for >>>>>> >>>>>> the single storage subsystem is presented by default. >>>>>> >>>>>> When there are multiple storage subsystems present, the Dashboard >>>>>> would >>>>>> present each dashboard in its own tab. Basically, we'd have a Ceph >>>>>> tab >>>>> >>>>> and >>>>>> >>>>>> a Gluster tab. >>>>>> >>>>>> Ideally if the default dashboard tab can be specified on a per user >>>>> >>>>> basis, >>>>>> >>>>>> e.g. in a user?s profile/setting -- initially configured by the >>>>>> Administrator user. It's a nice-to-have at this point, and if folks >>>>> >>>>> agree, >>>>>> >>>>>> it should be added to the backlog. >>>>>> >>>>>> With regards to unifying concepts such as quotas, file shares, etc., >>>>>> you >>>>>> raise a fair concern. There's multiple approaches to it. For the >>>>>> quotas, >>>>>> snapshots, my current thinking is that when we get to the Quotas or >>>>>> Snapshots section, we would either have a unified view with filtering >>>>>> capabilities (for the different types), or tabbed views, or a 3rd >>>>>> level >>>>>> navigation, or some other combination. This will be determined at a >>>>> >>>>> later >>>>>> >>>>>> date when the feature gets added. Going with a tabbed approach or 3rd >>>>> >>>>> level >>>>>> >>>>>> navigation could potentially allow for a plug-in based approach >>>>>> whereby >>>>>> specific storage subsystem capabilities can be exposed as needed. >>>>>> >>>>>> For File Shares, one approach I suggested was to handle it via the >>>>>> user's >>>>>> workflow. Another approach could be to handle it via the navigation >>>>>> which >>>>>> potentially increases complexity and may impact navigation by >>>>>> introducing >>>>>> more levels of navigation. Our goal is not to go deeper than 3 levels >>>>>> of >>>>>> navigation as it will become an unwieldy user experience if things are >>>>>> buried too deeply. That being said, whether it's handled via the >>>>> >>>>> workflow, >>>>>> >>>>>> navigation, or some other means, using a plug-in based approach should >>>>> >>>>> still >>>>>> >>>>>> work. >>>>> >>>>> Regarding navigation depth, I wonder if it's worth rethinking using >>>>> the whole first level just to get into "Storage" or "Clusters" -- >>>>> isn't everything we do about storage clusters? In my mind, things >>>>> like cephfs, rbd, rgw are really top-level items. >>>>> >>>>> Regarding unifying quotas and snapshots, I think it would be natural >>>>> for a user to go cephfs->quotas rather than quotas->cephfs, because at >>>>> any given moment they are probably concentrating on one particular >>>>> subsystem. I think the cephfs quotas and cephfs snapshots have more >>>>> in common (and belong closer together) than e.g. cephfs quotas and >>>>> gluster quotas, or cephfs snapshots and rbd snapshots. In a cephfs >>>>> UI, we would probably hope to see a tree view that showed directories >>>>> in the filesystem and enabled things like setting layouts and quotas >>>>> on directories -- picturing that, I don't see a sane way for cephfs >>>>> quotas to live under a top-level Quotas subsystem unless the whole >>>>> widget was duplicated inside the CephFS subsystem as well. >>>>> >>>>> Not trying to entirely derail this into a discussion of CephFS, but I >>>>> think the best way to see if this design is a good fit is to work >>>>> through some of the more challenging cases like the N types of quota >>>>> that will ultimately need managing (not just those supported in the >>>>> first release of the software), and see if what comes out makes sense. >>>>> >>>>> By the way, I was just taking another look at slide 12, and I don't >>>>> see user identities[1] (i.e. the identities that we use for access >>>>> control of Ceph clients, not the Tendrl users). Those should probably >>>>> be in there somewhere. They intersect both the "Cluster" section >>>>> (ceph keys are managed by the mons) and the RBD/RGW/CephFS parts (to >>>>> generate a key with meaningful capabilities you need to know which >>>>> subsystem it's going to be using). >>>>> >>>>> John >>>>> >>>>> 1. http://docs.ceph.com/docs/hammer/rados/operations/user-management/ >>>>> >>>>> >>>>>> I've made updates to the document UX Design Approach and Information >>>>>> Architecture to incorporate the points raised by both yours and John's >>>>>> feedback. >>>>>> >>>>>> Thank you, >>>>>> Ju >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Sep 28, 2016 at 3:55 AM, Nishanth Thomas >>>>> >>>>> wrote: >>>>>>> >>>>>>> Its good that we are not going to unify the dashboard pieces. With >>>>>>> this >>>>> >>>>> in >>>>>>> >>>>>>> mind how the main dashboard looks if we have a deployment with ceph >>>>>>> and >>>>>>> Gluster? >>>>>>> Also I have a concern with respect to unifying 'File shares' 'quotas' >>>>> >>>>> etc, >>>>>>> >>>>>>> as it might complicate the navigation. Rather we could look at a >>>>>>> plug-in >>>>>>> based approach where file system specific things could be kept >>>>> >>>>> separately >>>>>>> >>>>>>> and installed on a need basis. >>>>>>> >>>>>>> On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: >>>>>>>> >>>>>>>> Hi John: >>>>>>>> >>>>>>>> Thank you for taking the time to review and sharing your feedback. >>>>>>>> >>>>>>>> WRT terminology, I think that's a tricky one and I can understand >>>>>>>> and >>>>>>>> appreciate the confusion with RBD (vs. Block Device) and File Share. >>>>> >>>>> The >>>>>>>> >>>>>>>> original decision to go with RBD was that it was very Ceph-specific >>>>>>>> in >>>>> >>>>> terms >>>>>>>> >>>>>>>> of the implementation and varies from LVM and other block >>>>>>>> technologies. >>>>>>>> Having said that, I would agree that File Share deviates from this >>>>> >>>>> initial >>>>>>>> >>>>>>>> premise since it could mean a Gluster volume or a CephFS fileshare. >>>>>>>> >>>>>>>> The current thinking regarding File Share is to use it for both Ceph >>>>> >>>>> and >>>>>>>> >>>>>>>> Gluster, and have a way to qualify it through the notion of the >>>>> >>>>> workload. >>>>>>>> >>>>>>>> Part of the rationale was to try to make it so users did not have to >>>>> >>>>> think >>>>>>>> >>>>>>>> should I use Ceph or Gluster, but rather let the System make an >>>>>>>> "intelligent" choice for the mere mortal user based on just enough >>>>>>>> information that user provides. This way, the user thinks of it as >>>>> >>>>> Red Hat >>>>>>>> >>>>>>>> Storage providing this capability, and if we swapped out Ceph or >>>>> >>>>> Gluster for >>>>>>>> >>>>>>>> something in the future, it would allow for that. In conjunction, >>>>> >>>>> there >>>>>>>> >>>>>>>> would still be a way for a user to qualify it for Ceph or Gluster as >>>>> >>>>> well, >>>>>>>> >>>>>>>> so it's not just for mere mortal but also a more knowledge / ninja >>>>> >>>>> user. >>>>>>>> >>>>>>>> Having said that, RGW (and Swift) are definitely in mind too, so >>>>>>>> this >>>>> >>>>> is >>>>>>>> >>>>>>>> not limited to just File Shares but applies to pretty much block, >>>>> >>>>> object, >>>>>>>> >>>>>>>> and file services. >>>>>>>> >>>>>>>> With regards to quotas and snapshots, I do agree that there are many >>>>>>>> different kinds. It did not seem prudent to create a Pool Snapshot, >>>>> >>>>> RBD >>>>>>>> >>>>>>>> Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, >>>>> >>>>> Pool >>>>>>>> >>>>>>>> Quotas, Directory Quotas, etc. at the 2nd level navigation as it is >>>>>>>> not >>>>>>>> scalable (and would get very crowded very quickly), and would create >>>>>>>> a >>>>>>>> fairly complex experience for the user. My current thinking is that >>>>> >>>>> when we >>>>>>>> >>>>>>>> get to the Quotas or Snapshots section, we would either have a >>>>>>>> unified >>>>> >>>>> view >>>>>>>> >>>>>>>> with filtering capabilities (for the different types), or tabbed >>>>> >>>>> views, or a >>>>>>>> >>>>>>>> 3rd level navigation, or some other combination. Since quotas and >>>>> >>>>> snapshots >>>>>>>> >>>>>>>> are not in the immediate release plans, I've deferred this to the >>>>> >>>>> detailed >>>>>>>> >>>>>>>> design stage when we tackle those topics. >>>>>>>> >>>>>>>> I hope this addresses your concerns raised. I'll update the >>>>>>>> slidedeck >>>>>>>> with these comments before it gets published out more formally. >>>>>>>> >>>>>>>> Thanks again, >>>>>>>> Ju >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Sep 26, 2016 at 5:34 AM, John Spray >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: >>>>>>>>>> >>>>>>>>>> Hi. I've started documenting the UX Design Approach and >>>>>>>>>> Information >>>>>>>>>> Architecture (JIRA TEN-40). It includes some of our guiding >>>>>>>>>> principles and >>>>>>>>>> decisions that we've made in our design. >>>>>>>>>> >>>>>>>>>> Here's an initial draft that you can take a look at: >>>>>>>>>> >>>>>>>>>> https://docs.google.com/a/redhat.com/presentation/d/ >>>>> >>>>> 1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing >>>>>>>>>> >>>>>>>>>> Note: For now it's open to be comments by anyone with a Red Hat >>>>>>>>>> account. I >>>>>>>>>> plan to publish this to Jira and/or GitHub before the end of >>>>>>>>>> Sprint >>>>> >>>>> 1 >>>>>>>>>> >>>>>>>>>> (Oct >>>>>>>>>> 4), but I'd like to open this up to review, comments, and >>>>>>>>>> discussion >>>>>>>>>> in the >>>>>>>>>> meantime. >>>>>>>>> >>>>>>>>> Thanks for posting! >>>>>>>>> >>>>>>>>> My main piece of feedback is on terminology -- I think it's going >>>>>>>>> to >>>>>>>>> get confusing to use a mixture of qualified names (e.g. "RBD" >>>>>>>>> instead >>>>>>>>> of "block device") and unqualified names (e.g. "File Share" when we >>>>>>>>> really mean "Gluster"). >>>>>>>>> >>>>>>>>> I have a special attachment to use of "File Share" because when >>>>>>>>> Tendrl >>>>>>>>> gets support for CephFS, everywhere the term is used is going to >>>>>>>>> need >>>>>>>>> qualifying with whether we're talking about Ceph or Gluster. I >>>>>>>>> don't >>>>>>>>> think anyone is working on adding Tendrl support for CephFS at the >>>>>>>>> moment, but it should be part of the design process, along with >>>>>>>>> RGW. >>>>>>>>> >>>>>>>>> Terms like "Quotas" are also dangerous, because we have so many >>>>>>>>> different kinds. In Ceph alone, we have Quotas on pools, and then >>>>>>>>> a >>>>>>>>> totally different type of quota on directories in cephfs. Same for >>>>>>>>> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. >>>>>>>>> >>>>>>>>> John >>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Ju Lim >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Tendrl-devel mailing list >>>>>>>>>> Tendrl-devel at redhat.com >>>>>>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Tendrl-devel mailing list >>>>>>>>> Tendrl-devel at redhat.com >>>>>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Ju Lim >>>>>>>> Red Hat >>>>>>>> Office: 978-399-0422 >>>>>>>> Mobile: 781-507-1323 >>>>>>>> Email: julim at redhat.com >>>>>>>> IRC: julim >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Tendrl-devel mailing list >>>>>>>> Tendrl-devel at redhat.com >>>>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Tendrl-devel mailing list >>>>>>> Tendrl-devel at redhat.com >>>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ju Lim >>>>>> Red Hat >>>>>> Office: 978-399-0422 >>>>>> Mobile: 781-507-1323 >>>>>> Email: julim at redhat.com >>>>>> IRC: julim >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Tendrl-devel mailing list >>>>>> Tendrl-devel at redhat.com >>>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>>> >>>>> _______________________________________________ >>>>> Tendrl-devel mailing list >>>>> Tendrl-devel at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>> >>>> >>>> >>>> -- >>>> Ju Lim >>>> Red Hat >>>> Office: 978-399-0422 >>>> Mobile: 781-507-1323 >>>> Email: julim at redhat.com >>>> IRC: julim >>>> _______________________________________________ >>>> Tendrl-devel mailing list >>>> Tendrl-devel at redhat.com >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From rwheeler at redhat.com Tue Oct 4 08:03:30 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Tue, 4 Oct 2016 09:03:30 +0100 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> Message-ID: <72ddc331-57f3-d628-d432-c158df24380a@redhat.com> On 10/04/2016 08:47 AM, John Spray wrote: >> >Hi John, >> > >> >For the UX bits that Red Hat developers are working on, I don't see us >> >having different top levels for gluster and ceph. Not something certainly >> >that we tried for in the current management application. > What I was really asking was whether at install time the console will > have any knowledge of whether it's going to manage Ceph or Gluster, so > that we can hide parts that aren't needed. I'm wondering if we're > going to see e.g. a "tendrl-ceph" and "tendrl-gluster" package so that > people can get just the bits they need. The question of what items > belong at the top level would depend on that. > > Given that (I assert) the majority of users do not mix up ceph and > gluster in one system, it seems like a big usability win if they can > have a Gluster UI without Ceph bits in it, or vice versa. > > John > I do see that it is nice not to have visible elements for things you are not using, we might (for the RH downstream product), still opt to see them there. Of course, patches would be welcome to allow for that kind of flexibility, it does make sense to me for an upstream thing. For how a Red Hat specific, downstream product looks and what gets exposed there, we will have that debate internally with the UXD and product managers. Ric From mrugesh at brainfunked.org Tue Oct 4 09:14:15 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 4 Oct 2016 14:44:15 +0530 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: On 29 September 2016 at 01:11, Ju Lim wrote: > > Hi Nishant: > > Thanks for reviewing. With regards how the dashboard will look when Ceph and Gluster is in the picture: > > When there is a single storage subsystem present, the single dashboard for the single storage subsystem is presented by default. > > When there are multiple storage subsystems present, the Dashboard would present each dashboard in its own tab. Basically, we'd have a Ceph tab and a Gluster tab. I would suggest that we stop thinking in terms of types (ceph, gluster etc.), and instead think in terms of cluster instances. It is possible that Tendrl would support multiple versions of the same type. The context for the UI should be set based on which instance the user is under. Depending upon the differences between versions and the implementation of the corresponding bridge*, it is possible that two cluster instances of the same type allow different actions on the same object. * This means that it's possible, though not necessary, that different versions of the bridge would have to be maintained for different versions of a type. A ready example of this is the difference between the bridge implementation for ceph pre and post ceph-mgr. -- Mrugesh From mrugesh at brainfunked.org Tue Oct 4 09:54:51 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 4 Oct 2016 15:24:51 +0530 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> Message-ID: On 4 October 2016 at 13:17, John Spray wrote: > What I was really asking was whether at install time the console will > have any knowledge of whether it's going to manage Ceph or Gluster, so > that we can hide parts that aren't needed. I'm wondering if we're > going to see e.g. a "tendrl-ceph" and "tendrl-gluster" package so that > people can get just the bits they need. The question of what items > belong at the top level would depend on that. > > Given that (I assert) the majority of users do not mix up ceph and > gluster in one system, it seems like a big usability win if they can > have a Gluster UI without Ceph bits in it, or vice versa. ceph_bridge and gluster_bridge do exist as separate packages with no dependencies on each other. Specifically wrt the UI bit, this question would be answered in a future development sprint. -- Mrugesh From mrugesh at brainfunked.org Tue Oct 4 10:00:33 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 4 Oct 2016 15:30:33 +0530 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> Message-ID: On 4 October 2016 at 13:17, John Spray wrote: > What I was really asking was whether at install time the console will > have any knowledge of whether it's going to manage Ceph or Gluster, so > that we can hide parts that aren't needed. Console does need to understand which provisioning flow to invoke at install time. Currently, the provisioning framework is being worked out and I need to have a discussion with Alfredo as well. -- Mrugesh From adeza at redhat.com Tue Oct 4 11:08:01 2016 From: adeza at redhat.com (Alfredo Deza) Date: Tue, 4 Oct 2016 07:08:01 -0400 Subject: [Tendrl-devel] Installing Ceph via Tendrl In-Reply-To: References: Message-ID: On Tue, Oct 4, 2016 at 3:30 AM, Mrugesh Karnik wrote: > On 1 October 2016 at 02:31, Sankarshan Mukhopadhyay < > sankarshan.mukhopadhyay at gmail.com> wrote: >> >> On Sat, Oct 1, 2016 at 12:53 AM, Alfredo Deza wrote: >> > All of that will have to be done again in Tendrl. Can that design be >> > explained a bit? How is this generalized approach going to tackle >> > these issues? How is it going to approach tackling any other Ansible >> > playbook or role? >> > >> > How has the ceph-installer failed here? When the project started, it >> > did so with a "documentation first" approach, and we presented the API >> > first so that the consumers of it could have an immediate idea on the >> > HTTP API proposed. We did it this way because we thought it was >> > imperative to present the ideas while the application itself was being >> > crafted to improve the throughput on communication. >> >> It is a bit late/early here - the detail about the provisioning flow >> is (as I remember) part of this sprint or, a coming one and is being >> worked on. Perhaps we could provide an interim update and then take >> this conversation ahead. > > I'm working on this in the current sprint. Updates will be made to the > issue at https://github.com/Tendrl/documentation/issues/29 > > Also, Alfredo, do you think we can have a quick conversation somewhere > towards the end of this week? Of course. Any time! > > Thanks, > Mrugesh > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From deb at redhat.com Tue Oct 4 11:43:20 2016 From: deb at redhat.com (Soumya Deb) Date: Tue, 4 Oct 2016 17:13:20 +0530 Subject: [Tendrl-devel] Frontend tool stack In-Reply-To: References: Message-ID: FYI, the repository names has been changed (thanks Sankarshan). You can find the issue at https://github.com/Tendrl/tendrl_frontend/issues/5 Thanks! On 27 September 2016 at 16:26, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Tue, Sep 27, 2016 at 4:24 PM, Soumya Deb wrote: > > We're tracking all the tools selections > > (https://github.com/Tendrl/ui/labels/selection) made for the frontend > > development in https://github.com/Tendrl/ui/issues/5 > > Was there not a conversation to keep the ui repository as is and > instead use a 'frontend' or, similarly named repository to have this > work in? > > > -- > sankarshan mukhopadhyay > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From japplewh at redhat.com Tue Oct 4 12:19:56 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 4 Oct 2016 08:19:56 -0400 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> Message-ID: One thought I had was that Tendrl could "remember" the last used storage backend (in the case where you have Ceph and Gluster) and simply default to displaying whatever that was. This is easier on the client side with cookies than server side I suppose. In the case where it has only installed or imported either Ceph or Gluster it should not confuse the user with extraneous information related to other SDS technologies. Tendrl knows what it's scope is here. Is this doable? On Tue, Oct 4, 2016 at 6:00 AM, Mrugesh Karnik wrote: > On 4 October 2016 at 13:17, John Spray wrote: > > What I was really asking was whether at install time the console will > > have any knowledge of whether it's going to manage Ceph or Gluster, so > > that we can hide parts that aren't needed. > > Console does need to understand which provisioning flow to invoke at > install time. Currently, the provisioning framework is being worked out and > I need to have a discussion with Alfredo as well. > > -- > Mrugesh > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From kanagaraj.ktr at gmail.com Tue Oct 4 12:45:58 2016 From: kanagaraj.ktr at gmail.com (Kanagaraj M) Date: Tue, 4 Oct 2016 18:15:58 +0530 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> Message-ID: Though it might be possible, using cookies for storing this kind of information is not good. The content of the app will vary based on the user browser. The same user will see different content on a different browser. Also its not a single user who will be using Tendrl to manage the cluster. If at all this kind of information needs to be stored, It should be a setting in the application which can be done either during installation or later. On Tue, Oct 4, 2016 at 5:49 PM, Jeff Applewhite wrote: > One thought I had was that Tendrl could "remember" the last used storage > backend (in the case where you have Ceph and Gluster) and simply default to > displaying whatever that was. This is easier on the client side with > cookies than server side I suppose. > > In the case where it has only installed or imported either Ceph or Gluster > it should not confuse the user with extraneous information related to other > SDS technologies. Tendrl knows what it's scope is here. > > Is this doable? > > On Tue, Oct 4, 2016 at 6:00 AM, Mrugesh Karnik > wrote: > > > On 4 October 2016 at 13:17, John Spray wrote: > > > What I was really asking was whether at install time the console will > > > have any knowledge of whether it's going to manage Ceph or Gluster, so > > > that we can hide parts that aren't needed. > > > > Console does need to understand which provisioning flow to invoke at > > install time. Currently, the provisioning framework is being worked out > and > > I need to have a discussion with Alfredo as well. > > > > -- > > Mrugesh > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > -- > Jeff Applewhite > Principal Product Manager > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From nlevine at redhat.com Tue Oct 4 17:12:42 2016 From: nlevine at redhat.com (Neil Levine) Date: Tue, 4 Oct 2016 10:12:42 -0700 Subject: [Tendrl-devel] Architectural questions - need clear pros & cons for the integration options In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> <571db346-0ca1-894c-2cfe-0cac5ade920b@redhat.com> Message-ID: Did this email go out? On Thu, Sep 29, 2016 at 9:43 AM, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Thu, Sep 29, 2016 at 8:11 PM, Sayan Saha wrote: > > This is not an easy read. Can you please send a clear email outlining > the 3 > > options (that I have heard about) for RHScon 3 + RHCS integration with a > > pros & cons analysis for each. This will greatly help expedite decision > > making. > > I'll do a quick summary from a meeting hosted by Jeff and which had > Gregory, John, Mrugesh, Rohan, Nishanth, Ric and myself as > participants. > > The decision arrived at is that it is of benefit to work with the > Calamari developers to (a) assess the specific improvements to be > built in (b) scope the time taken to undertake such work. Nishanth and > Rohan would compile the detail leading to (a) and post to ceph-devel@ > list (by 04Oct/Tue) in order to initiate the discussions. > > > > -- > sankarshan mukhopadhyay > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From gmeno at redhat.com Tue Oct 4 18:21:27 2016 From: gmeno at redhat.com (Gregory Meno) Date: Tue, 4 Oct 2016 11:21:27 -0700 Subject: [Tendrl-devel] Architectural questions - need clear pros & cons for the integration options In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> <571db346-0ca1-894c-2cfe-0cac5ade920b@redhat.com> Message-ID: I have not seen it. On Tuesday, October 4, 2016, Neil Levine wrote: > Did this email go out? > > On Thu, Sep 29, 2016 at 9:43 AM, Sankarshan Mukhopadhyay < > sankarshan.mukhopadhyay at gmail.com > wrote: > > > On Thu, Sep 29, 2016 at 8:11 PM, Sayan Saha > wrote: > > > This is not an easy read. Can you please send a clear email outlining > > the 3 > > > options (that I have heard about) for RHScon 3 + RHCS integration with > a > > > pros & cons analysis for each. This will greatly help expedite decision > > > making. > > > > I'll do a quick summary from a meeting hosted by Jeff and which had > > Gregory, John, Mrugesh, Rohan, Nishanth, Ric and myself as > > participants. > > > > The decision arrived at is that it is of benefit to work with the > > Calamari developers to (a) assess the specific improvements to be > > built in (b) scope the time taken to undertake such work. Nishanth and > > Rohan would compile the detail leading to (a) and post to ceph-devel@ > > list (by 04Oct/Tue) in order to initiate the discussions. > > > > > > > > -- > > sankarshan mukhopadhyay > > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From julim at redhat.com Wed Oct 5 02:40:05 2016 From: julim at redhat.com (Ju Lim) Date: Tue, 4 Oct 2016 22:40:05 -0400 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) Message-ID: Team: The demo recording from Sprint 1 demo is now available along with the final agenda and notes (feedback and Q&A). They can be found at Sprint 001 Demo Agenda & Notes . If folks have any comments and/or feedback on the demo(s), please reply all to this email thread. Going forward, all Sprint goals, demo agenda, demo recording, and notes may be found at the Tendrl Sprints Landing Page , which is an index of all sprints to-date. Thank you, Ju From rwheeler at redhat.com Wed Oct 5 06:22:05 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Wed, 5 Oct 2016 07:22:05 +0100 Subject: [Tendrl-devel] Architectural questions - need clear pros & cons for the integration options In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> <571db346-0ca1-894c-2cfe-0cac5ade920b@redhat.com> Message-ID: Sankarshan - in the email quoted below - stated the quick summary as: > I'll do a quick summary from a meeting hosted by Jeff and which had > Gregory, John, Mrugesh, Rohan, Nishanth, Ric and myself as > participants. > > The decision arrived at is that it is of benefit to work with the > Calamari developers to (a) assess the specific improvements to be > built in (b) scope the time taken to undertake such work. Nishanth and > Rohan would compile the detail leading to (a) and post to ceph-devel@ > list (by 04Oct/Tue) in order to initiate the discussions. > The decision was pretty clear to those of us in the meeting: * Gregory would be on the hook for turning the existing fork of calamari that used librados into an API that tendrl can consume. * NIsanth and Rohan need to provide the specific detail needed to help Gregory and team start that work. That detail needs to be posted here. * The calamari usage will be deprecated in favor of using ceph-mgr once that is ready for production We do want the details promised from Nisanth and Rohan, but I think that they might be in transit to Berlin for LinuxCon this week. Regards, Ric On 10/04/2016 07:21 PM, Gregory Meno wrote: > I have not seen it. > > On Tuesday, October 4, 2016, Neil Levine wrote: > >> Did this email go out? >> >> On Thu, Sep 29, 2016 at 9:43 AM, Sankarshan Mukhopadhyay < >> sankarshan.mukhopadhyay at gmail.com > wrote: >> >>> On Thu, Sep 29, 2016 at 8:11 PM, Sayan Saha > > wrote: >>>> This is not an easy read. Can you please send a clear email outlining >>> the 3 >>>> options (that I have heard about) for RHScon 3 + RHCS integration with >> a >>>> pros & cons analysis for each. This will greatly help expedite decision >>>> making. >>> I'll do a quick summary from a meeting hosted by Jeff and which had >>> Gregory, John, Mrugesh, Rohan, Nishanth, Ric and myself as >>> participants. >>> >>> The decision arrived at is that it is of benefit to work with the >>> Calamari developers to (a) assess the specific improvements to be >>> built in (b) scope the time taken to undertake such work. Nishanth and >>> Rohan would compile the detail leading to (a) and post to ceph-devel@ >>> list (by 04Oct/Tue) in order to initiate the discussions. >>> From jspray at redhat.com Wed Oct 5 12:00:43 2016 From: jspray at redhat.com (John Spray) Date: Wed, 5 Oct 2016 14:00:43 +0200 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: On Wed, Oct 5, 2016 at 4:40 AM, Ju Lim wrote: > Team: > > The demo recording from Sprint 1 demo is now available along with the final > agenda and notes (feedback and Q&A). They can be found at Sprint 001 Demo > Agenda & Notes > . This looks to be behind a login page? (I've avoided having an account on that particular site so far) John > If folks have any comments and/or feedback on the demo(s), please reply all > to this email thread. > > Going forward, all Sprint goals, demo agenda, demo recording, and notes may > be found at the Tendrl Sprints Landing Page > , which is an index > of all sprints to-date. > > Thank you, > Ju > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From rkanade at redhat.com Wed Oct 5 12:57:28 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Wed, 5 Oct 2016 18:27:28 +0530 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge Message-ID: Calamari is an important component in the Ceph management/ops process. In the past, The Skyrings [0] project was able to utilize Calamari via the Calamari REST API, the integration did ran into some problems as mentioned below. - Calamari-server would often be a single point of failure as it is installed on the ceph-mon nodes - Querying Calamari REST API gave inconsistent data about the ceph cluster state, this might be happening because of sync issues occurring due to ceph-mons not being in quorum etc. - Calamari required to be installed on only one of the ceph-mons, This introduces very common scalability issues. - Calamari relies on older version of saltstack for messaging and other communication, this introduces packaging and performance overheads These issues must be re-verified and fixed if they still persist in Calamari. The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried to address these issues by: - Distributing the tendrl ceph bridge to talk to all ceph-mons in the cluster. - Gathering data (3 second poll interval) from ceph-mons only when they are in quorum and directly talking to ceph-mons via librados and using ceph admin sockets to query valid cluster maps and other ceph data. - Tendrl ceph bridge tries to address the single point of failure issue by having multiple instances of the bridge for each ceph-mon in the cluster. - Tendrl ceph bridge has centralized communications via an etcd cluster which removes need to maintain saltstack . Moving forward, we would like Calamari to accommodate these improvements back into the calamari code base, Below are the detailed requirements: - Need Calamari REST API for querying ceph admin sockets [2], More details: - Need Calamari REST API for providing all cluster maps (monitor, osd, crush, mds, pg, health, config etc) - Calamari REST API must represent the ceph cluster state correctly, no mismatch should occur between the True state of the cluster and the state represented by the Calamari REST API - Calamari REST API should provide access to monitoring actions like "ceph -w", "ceph tell" etc - Calamari REST API should provide access to operational actions via tools like "rbd" etc. - Calamari REST API should be able to service and perform well on 2-4 second polling intervals on the REST API These requirements can be run through existing calamari functional and performance test cases and we should make additions to these test cases to accommodate above requirements. These requirements would be further expanded into individual BZs or tracking items in Calamari [0]: https://github.com/skyrings [1]: https://github.com/tendrl/ceph_bridge [2]: http://docs.ceph.com/docs/jewel/rados/operations/monitoring/#using-the-admin-socket From jspray at redhat.com Wed Oct 5 13:14:56 2016 From: jspray at redhat.com (John Spray) Date: Wed, 5 Oct 2016 15:14:56 +0200 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: On Wed, Oct 5, 2016 at 2:57 PM, Rohan Kanade wrote: > Calamari is an important component in the Ceph management/ops process. In > the past, The Skyrings [0] project was able to utilize Calamari via the > Calamari REST API, the integration did ran into some problems as mentioned > below. > > - Calamari-server would often be a single point of failure as it is > installed on the ceph-mon nodes > - Querying Calamari REST API gave inconsistent data about the ceph cluster > state, this might be happening because of sync issues occurring due to > ceph-mons not being in quorum etc. > - Calamari required to be installed on only one of the ceph-mons, This > introduces very common scalability issues. > - Calamari relies on older version of saltstack for messaging and other > communication, this introduces packaging and performance overheads > > These issues must be re-verified and fixed if they still persist in > Calamari. > > > > > The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried to > address these issues by: > > - Distributing the tendrl ceph bridge to talk to all ceph-mons in the > cluster. > - Gathering data (3 second poll interval) from ceph-mons only when they are > in quorum and directly talking to ceph-mons via librados and using ceph > admin sockets to query valid cluster maps and other ceph data. > - Tendrl ceph bridge tries to address the single point of failure issue by > having multiple instances of the bridge for each ceph-mon in the cluster. > - Tendrl ceph bridge has centralized communications via an etcd cluster > which removes need to maintain saltstack . > I'll leave the detailed responses to folks working on Calamari, but I wanted to note that where you ask for admin socket or `tell` funtionality, you're asking for a pass-through rather than telling us what it is that you need access to. You may well get better answers if you can say specifically what you're looking to get out of the system. John > > Moving forward, we would like Calamari to accommodate these improvements > back into the calamari code base, Below are the detailed requirements: > > - Need Calamari REST API for querying ceph admin sockets [2], More details: > - Need Calamari REST API for providing all cluster maps (monitor, osd, > crush, mds, pg, health, config etc) > - Calamari REST API must represent the ceph cluster state correctly, no > mismatch should occur between the True state of the cluster and the state > represented by the Calamari REST API > - Calamari REST API should provide access to monitoring actions like "ceph > -w", "ceph tell" etc > - Calamari REST API should provide access to operational actions via tools > like "rbd" etc. > - Calamari REST API should be able to service and perform well on 2-4 > second polling intervals on the REST API > > > These requirements can be run through existing calamari functional and > performance test cases and we should make additions to these test cases to > accommodate above requirements. > > These requirements would be further expanded into individual BZs or > tracking items in Calamari > > > [0]: https://github.com/skyrings > [1]: https://github.com/tendrl/ceph_bridge > [2]: > http://docs.ceph.com/docs/jewel/rados/operations/monitoring/#using-the-admin-socket > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mbukatov at redhat.com Thu Oct 6 09:14:07 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 6 Oct 2016 11:14:07 +0200 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: <41fdc94a-148a-ecd9-970d-e36479f32f36@redhat.com> On 09/19/2016 01:38 PM, Soumya Deb wrote: > This is just the beginning, and would expand to all the tool choices we > would eventually make. We'll move the contents to github-wiki for the > frontend repo as their final home (slideshow may not be the best > solution for the stuff in hand), but opening the thread up to gather > feedback around the choices made (or even better - how it's made, or can > be done more efficiently). Big +1 to use the wiki as a *single* place for all tendrl related devel proposals, notes or other documents. Using google docs for this is quite terrible and should be discouraged. -- Martin Bukatovic USM QE team From mrugesh at brainfunked.org Thu Oct 6 11:11:56 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Thu, 6 Oct 2016 16:41:56 +0530 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: On 5 October 2016 at 17:30, John Spray wrote: > > On Wed, Oct 5, 2016 at 4:40 AM, Ju Lim wrote: > > Team: > > > > The demo recording from Sprint 1 demo is now available along with the final > > agenda and notes (feedback and Q&A). They can be found at Sprint 001 Demo > > Agenda & Notes > > . > > This looks to be behind a login page? (I've avoided having an account > on that particular site so far) Our jira should be publicly read-only, without the requirement to logic with an account. However, the one time I tried to change this setting myself, I was rather overwhelmed by the complexity. Jeff, would you be able to make this change in the configuration? Thanks, Mrugesh From japplewh at redhat.com Thu Oct 6 11:24:08 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Thu, 6 Oct 2016 07:24:08 -0400 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: On Wed, Oct 5, 2016 at 8:00 AM, John Spray wrote: > On Wed, Oct 5, 2016 at 4:40 AM, Ju Lim wrote: > > Team: > > > > The demo recording from Sprint 1 demo is now available along with the > final > > agenda and notes (feedback and Q&A). They can be found at Sprint 001 > Demo > > Agenda & Notes > > >. > > This looks to be behind a login page? (I've avoided having an account > on that particular site so far) > ?Hi John I think I found the cause - there are separate settings for Confluence and Jira - I had opened up the Jira but not the Confluence - now they are both setup for anonymous access now. Let me know if this is not fixed. Thanks Jeff? > > John > > > If folks have any comments and/or feedback on the demo(s), please reply > all > > to this email thread. > > > > Going forward, all Sprint goals, demo agenda, demo recording, and notes > may > > be found at the Tendrl Sprints Landing Page > > , which is an > index > > of all sprints to-date. > > > > Thank you, > > Ju > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From mbukatov at redhat.com Thu Oct 6 11:25:06 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 6 Oct 2016 13:25:06 +0200 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: <451b685c-e9f7-485d-3869-9fc1b87dc201@redhat.com> On 09/22/2016 06:31 AM, Soumya Deb wrote: > 0. Why node.js? Why not Python (or Ruby, or X)? > To answer simply, Node.js is the leanest environment for a frontend > stack, without compromising on features, while being nice to available > resources. Also to note, for frontend coding, JS is unavoidable. The > "adding yet another language" argument doesn't hold here, as JS is the > only one we're trying to stick to. In fact, a python framework for > serving frontend would actually be "yet another language" scenario. > Please do not confuse this stack as the main backend stack by any means > - Node.js will be used to serve the frontend only, and won't be used in > the backend (tendrl common/core). They'll be built into different > packages as well. Let's consider the use case for tendrl first. Tendrl is a storage management tool, with web ui and rest api. This means that the main user group are storage administrators, who will install it on either virtual or hw management machine in their data center. The machine tendrl will be installed on will run some stable linux distribution, such as RHEL/CentOS, Debian stable or Ubuntu LTS. >From the upstream community perspective, most contributors would come from the people which deals with storage technologies in some way. Their main concern would be implementation of storage management tasks. We should evaluate the technologies we use in tendrl with this in mind. Which means we should ask questions like: * Are technologies we are going to build tendrl with commonly available in stable linux distributions? * Are technologies we are going to build tendrl with already well established in the group of expected contributors? * Would a particular technology increase cost of tendrl machine installation or maintenance? We are not talking about some simple CRUD web app running in a cloud environment, which could easily use the latest technology without any problems, as it doesn't have to integrate with anything else and doesn't need to be much stable (from the technology point of view), as it can be easily updated on the fly. We are talking about the stable storage management system, which will be deployed in a data center. So for these reasons, I don't agree with statements like: * The "adding yet another language" argument doesn't hold here * a python framework for serving front end would actually be "yet another language" scenario While technology like node.js would be great for a regular web project, in our case I'm not that sure because of the points I listed above. We can't propose a technology without a clear understanding of the cost of maintenance and increased complexity related to this decision (we can't expect that node.js is already installed on storage management machine). While you may be thinking that it's a best choice for your group, you need to consider impact of this decision to the other people of the project. For the same reason, I would suggest to keep the number of languages and technologies used in tendrl project low. -- Martin Bukatovic USM QE team From mbukatov at redhat.com Thu Oct 6 11:26:51 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 6 Oct 2016 13:26:51 +0200 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On 09/19/2016 01:54 PM, John Spray wrote: > I'm concerned about adopting node.js on the server side. We already > have a C++/Python stack in both Ceph and Gluster -- I think there > needs to be a strong case for how node is going to be better than > Python for this use case, before adding another language to the > server-side stack. Consider the developer who wants to spin up this > GUI in his development environment on top of a local Ceph cluster -- > folks are much more likely to get involved if they don't have to > install a whole extra language runtime and pile of dependencies. +1 -- Martin Bukatovic USM QE team From mbukatov at redhat.com Thu Oct 6 11:51:01 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 6 Oct 2016 13:51:01 +0200 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: <8ebb81cd-9ad1-6945-0ddb-3f091a6a1ba6@redhat.com> On 09/23/2016 01:49 PM, Soumya Deb wrote: > However, there will be other SDSs to support, and some might be > developed in Go, Java, C++ or Rust (or whatever might be fitting for > that particular product's team). Do you consider that should dictate > the choice of language on the upper layers, or should they choose the > ones that works best for them (as in, Tendrl bridge and Agent is on > Python, but most SDS aren't written in Python - should they consider > writting it in C++?). In that case, when all the other parts are pretty > homogeneous and based on C++, should we consider the frontend > to also base on C++ platform? This gets pretty far-fetched there on. That's not where the problem is. It's not about enforcing a language choice based on particular storage solution. I guess that most software defined storage systems are written in c or c++ nowadays and nobody proposes that we should use c++ for a web app. The problem is in: * considering cost of maintenance, support and increased complexity by using additional runtime which is not commonly used in our target enviroment * considering language/technology familiarity among people who would be interested in contributing in the project > In my understanding, as long as each modular part/layer uses best > technology available for them, and interoperates with other parts in > the stack this shouldn't cause any headache for the other layers. I don't have a problem with using different languages best suited for a particular job for different pieces of the tendrl. But I don't agree that node.js would be a best technology for our use case. So let's put in a different way: is node.js so superior to other (also very popular) ways of web development, so that it would outweight the cost bringing in the whole node.js stack itself? Who runs node.js on storage machines it theirs datacenter nowadays? I would guess that nobody. Is node.js easily available on stable linux distributions, so that we don't have to worry about it? I don't think so. Which means that if we bring that dependency it, we need to account for increased complexity and the maintenance of whole node.js runtime. -- Martin Bukatovic USM QE team From gmeno at redhat.com Thu Oct 6 13:56:22 2016 From: gmeno at redhat.com (Gregory Meno) Date: Thu, 6 Oct 2016 06:56:22 -0700 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: On Wed, Oct 5, 2016 at 5:57 AM, Rohan Kanade wrote: > Calamari is an important component in the Ceph management/ops process. In > the past, The Skyrings [0] project was able to utilize Calamari via the > Calamari REST API, the integration did ran into some problems as mentioned > below. > > - Calamari-server would often be a single point of failure as it is > installed on the ceph-mon nodes Consider that monitor nodes are the right place to run and there are usually 3 or 5 monitors. Is that enough? I think we are at a point now where this would be reasonable to run on each monitor. What is more interesting to me is how you see the interaction happening. Does each process instance track the mon leader and proxy requests to the instance that is on the leader? John: have we considered what the ceph-mgr answer to this is? Do you expect to have perfect replication of state related to requests to change the ceph cluster? How do you want to discover available endpoints? > - Querying Calamari REST API gave inconsistent data about the ceph cluster > state, this might be happening because of sync issues occurring due to > ceph-mons not being in quorum etc. Specific tickets would go a long way here. > - Calamari required to be installed on only one of the ceph-mons, This > introduces very common scalability issues. I think we can relax this requirement if we can agree on expectations above. > - Calamari relies on older version of saltstack for messaging and other > communication, this introduces packaging and performance overheads This is not the case. We only use salt to send events to tendrl and it's something I consider to be a mistake. I think salt can be taken out of the picture in future releases > > These issues must be re-verified and fixed if they still persist in > Calamari. > > > > > The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried to > address these issues by: Did they succeed? > > - Distributing the tendrl ceph bridge to talk to all ceph-mons in the > cluster. > - Gathering data (3 second poll interval) from ceph-mons only when they are > in quorum and directly talking to ceph-mons via librados and using ceph > admin sockets to query valid cluster maps and other ceph data. Since ceph_bridge is taken directly from calamari it's fair to say that we posses the same capabilities > - Tendrl ceph bridge tries to address the single point of failure issue by > having multiple instances of the bridge for each ceph-mon in the cluster. Would you please show me where ceph_bridge is aware of other instances? Or are these multiple copies sharing nothing? > - Tendrl ceph bridge has centralized communications via an etcd cluster > which removes need to maintain saltstack . > > > > Moving forward, we would like Calamari to accommodate these improvements > back into the calamari code base, Below are the detailed requirements: > > - Need Calamari REST API for querying ceph admin sockets [2], More details: Let's not expose this. I would consider it to be a challenge to your goal of having multiple instances as you're talking to a specific socket. Really what you get is info about clusters that are being setup or severly damaged this should be built into whatever service is responsible for identifying clusters and talking to them e.g. calamari and in future ceph-mgr. > - Need Calamari REST API for providing all cluster maps (monitor, osd, > crush, mds, pg, health, config etc) This seems like a strange request. Why do you want access to the raw data? You'll have to parse it to make sense of it the same way we already do in calamari and in future ceph-mgr. > - Calamari REST API must represent the ceph cluster state correctly, no > mismatch should occur between the True state of the cluster and the state > represented by the Calamari REST API Happy to work to fix specific dependencies. Tickets are best. > - Calamari REST API should provide access to monitoring actions like "ceph > -w", "ceph tell" etc Help me understand the specifics of what you want to accomplish here. the output of ceph -w can be synthesized from looking at the cluster maps. and ceph tell is typically used for troubleshooting specific instances where you want to set new config with out changing ceph.conf and restarting. That means it applies to a single instance, not cluster wide. > - Calamari REST API should provide access to operational actions via tools > like "rbd" etc. Agreed. > - Calamari REST API should be able to service and perform well on 2-4 second > polling intervals on the REST API Yes I agree here too. > > > These requirements can be run through existing calamari functional and > performance test cases and we should make additions to these test cases to > accommodate above requirements. > > These requirements would be further expanded into individual BZs or tracking > items in Calamari > > > [0]: https://github.com/skyrings > [1]: https://github.com/tendrl/ceph_bridge > [2]: > http://docs.ceph.com/docs/jewel/rados/operations/monitoring/#using-the-admin-socket From jspray at redhat.com Thu Oct 6 14:22:35 2016 From: jspray at redhat.com (John Spray) Date: Thu, 6 Oct 2016 16:22:35 +0200 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: On Thu, Oct 6, 2016 at 3:56 PM, Gregory Meno wrote: > On Wed, Oct 5, 2016 at 5:57 AM, Rohan Kanade wrote: >> Calamari is an important component in the Ceph management/ops process. In >> the past, The Skyrings [0] project was able to utilize Calamari via the >> Calamari REST API, the integration did ran into some problems as mentioned >> below. >> >> - Calamari-server would often be a single point of failure as it is >> installed on the ceph-mon nodes > > Consider that monitor nodes are the right place to run and there are > usually 3 or 5 monitors. > Is that enough? I think we are at a point now where this would be > reasonable to run on each monitor. > What is more interesting to me is how you see the interaction happening. > > Does each process instance track the mon leader and proxy requests to > the instance that is on the leader? > John: have we considered what the ceph-mgr answer to this is? ceph-mgr does HA by running multiple daemons (usually one per mon), where one is active at any moment. The active mgr is chosen by the mons (MgrMonitor class). It essentially mirrors how MDSs are managed in this respect. It doesn't do anything special to try to talk to a particular mon, just acts like any other client (i.e. picks one at random). John > > Do you expect to have perfect replication of state related to requests > to change the ceph cluster? > How do you want to discover available endpoints? > >> - Querying Calamari REST API gave inconsistent data about the ceph cluster >> state, this might be happening because of sync issues occurring due to >> ceph-mons not being in quorum etc. > > Specific tickets would go a long way here. > >> - Calamari required to be installed on only one of the ceph-mons, This >> introduces very common scalability issues. > > I think we can relax this requirement if we can agree on expectations above. > > >> - Calamari relies on older version of saltstack for messaging and other >> communication, this introduces packaging and performance overheads > > This is not the case. We only use salt to send events to tendrl and > it's something I consider to be a mistake. > I think salt can be taken out of the picture in future releases > >> >> These issues must be re-verified and fixed if they still persist in >> Calamari. >> >> >> >> >> The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried to >> address these issues by: > > Did they succeed? > >> >> - Distributing the tendrl ceph bridge to talk to all ceph-mons in the >> cluster. >> - Gathering data (3 second poll interval) from ceph-mons only when they are >> in quorum and directly talking to ceph-mons via librados and using ceph >> admin sockets to query valid cluster maps and other ceph data. > > Since ceph_bridge is taken directly from calamari it's fair to say > that we posses the same capabilities > >> - Tendrl ceph bridge tries to address the single point of failure issue by >> having multiple instances of the bridge for each ceph-mon in the cluster. > > Would you please show me where ceph_bridge is aware of other > instances? Or are these multiple copies sharing nothing? > >> - Tendrl ceph bridge has centralized communications via an etcd cluster >> which removes need to maintain saltstack . >> >> >> >> Moving forward, we would like Calamari to accommodate these improvements >> back into the calamari code base, Below are the detailed requirements: >> >> - Need Calamari REST API for querying ceph admin sockets [2], More details: > > Let's not expose this. I would consider it to be a challenge to your > goal of having multiple instances > as you're talking to a specific socket. Really what you get is info > about clusters that are being setup or severly damaged this should be > built into whatever service is responsible for identifying clusters > and talking to them e.g. calamari and in future ceph-mgr. > >> - Need Calamari REST API for providing all cluster maps (monitor, osd, >> crush, mds, pg, health, config etc) > > This seems like a strange request. Why do you want access to the raw > data? You'll have to parse it to make sense of it the same way we > already do in calamari and in future ceph-mgr. > >> - Calamari REST API must represent the ceph cluster state correctly, no >> mismatch should occur between the True state of the cluster and the state >> represented by the Calamari REST API > > Happy to work to fix specific dependencies. Tickets are best. > >> - Calamari REST API should provide access to monitoring actions like "ceph >> -w", "ceph tell" etc > > Help me understand the specifics of what you want to accomplish here. > the output of ceph -w can be synthesized from looking at the cluster > maps. and ceph tell is typically used for troubleshooting specific > instances where you want to set new config with out changing ceph.conf > and restarting. That means it applies to a single instance, not > cluster wide. > >> - Calamari REST API should provide access to operational actions via tools >> like "rbd" etc. > > Agreed. > >> - Calamari REST API should be able to service and perform well on 2-4 second >> polling intervals on the REST API > > Yes I agree here too. > >> >> >> These requirements can be run through existing calamari functional and >> performance test cases and we should make additions to these test cases to >> accommodate above requirements. >> >> These requirements would be further expanded into individual BZs or tracking >> items in Calamari >> >> >> [0]: https://github.com/skyrings >> [1]: https://github.com/tendrl/ceph_bridge >> [2]: >> http://docs.ceph.com/docs/jewel/rados/operations/monitoring/#using-the-admin-socket > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From jspray at redhat.com Thu Oct 6 14:44:19 2016 From: jspray at redhat.com (John Spray) Date: Thu, 6 Oct 2016 16:44:19 +0200 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: On Thu, Oct 6, 2016 at 1:24 PM, Jeff Applewhite wrote: > On Wed, Oct 5, 2016 at 8:00 AM, John Spray wrote: > >> On Wed, Oct 5, 2016 at 4:40 AM, Ju Lim wrote: >> > Team: >> > >> > The demo recording from Sprint 1 demo is now available along with the >> final >> > agenda and notes (feedback and Q&A). They can be found at Sprint 001 >> Demo >> > Agenda & Notes >> > > >. >> >> This looks to be behind a login page? (I've avoided having an account >> on that particular site so far) >> > > Hi John > > I think I found the cause - there are separate settings for Confluence and > Jira - I had opened up the Jira but not the Confluence - now they are both > setup for anonymous access now. Let me know if this is not fixed. It's still asking for a login here. BTW when modifying settings you can check if anonymous access works by opening an incognito browser window and trying to access that URL. John > Thanks > > Jeff > > >> >> John >> >> > If folks have any comments and/or feedback on the demo(s), please reply >> all >> > to this email thread. >> > >> > Going forward, all Sprint goals, demo agenda, demo recording, and notes >> may >> > be found at the Tendrl Sprints Landing Page >> > , which is an >> index >> > of all sprints to-date. >> > >> > Thank you, >> > Ju >> > _______________________________________________ >> > Tendrl-devel mailing list >> > Tendrl-devel at redhat.com >> > https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > -- > Jeff Applewhite > Principal Product Manager > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From julim at redhat.com Thu Oct 6 14:52:49 2016 From: julim at redhat.com (Ju Lim) Date: Thu, 6 Oct 2016 10:52:49 -0400 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: John: Jeff A. kindly gave me permissions in Jira, and I've fixed it so the Confluence space is accessible to all (view only) without login perms. So, you can now view https://tendrl.atlassian.net/wiki/spaces/TEN and all other wiki pages linked from there including the $SUBJECT. We still have to fix it for the Kanban board. I'll look into fixing that too soon. Thanks for your patience, Ju On Thu, Oct 6, 2016 at 10:44 AM, John Spray wrote: > On Thu, Oct 6, 2016 at 1:24 PM, Jeff Applewhite > wrote: > > On Wed, Oct 5, 2016 at 8:00 AM, John Spray wrote: > > > >> On Wed, Oct 5, 2016 at 4:40 AM, Ju Lim wrote: > >> > Team: > >> > > >> > The demo recording from Sprint 1 demo is now available along with the > >> final > >> > agenda and notes (feedback and Q&A). They can be found at Sprint 001 > >> Demo > >> > Agenda & Notes > >> > pageId=7733261 > >> >. > >> > >> This looks to be behind a login page? (I've avoided having an account > >> on that particular site so far) > >> > > > > Hi John > > > > I think I found the cause - there are separate settings for Confluence > and > > Jira - I had opened up the Jira but not the Confluence - now they are > both > > setup for anonymous access now. Let me know if this is not fixed. > > It's still asking for a login here. BTW when modifying settings you > can check if anonymous access works by opening an incognito browser > window and trying to access that URL. > > John > > > Thanks > > > > Jeff > > > > > >> > >> John > >> > >> > If folks have any comments and/or feedback on the demo(s), please > reply > >> all > >> > to this email thread. > >> > > >> > Going forward, all Sprint goals, demo agenda, demo recording, and > notes > >> may > >> > be found at the Tendrl Sprints Landing Page > >> > , which is an > >> index > >> > of all sprints to-date. > >> > > >> > Thank you, > >> > Ju > >> > _______________________________________________ > >> > Tendrl-devel mailing list > >> > Tendrl-devel at redhat.com > >> > https://www.redhat.com/mailman/listinfo/tendrl-devel > >> > >> _______________________________________________ > >> Tendrl-devel mailing list > >> Tendrl-devel at redhat.com > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > >> > > > > > > > > -- > > Jeff Applewhite > > Principal Product Manager > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From mcarrano at redhat.com Thu Oct 6 14:54:21 2016 From: mcarrano at redhat.com (Matt Carrano) Date: Thu, 6 Oct 2016 10:54:21 -0400 Subject: [Tendrl-devel] Create and Import Cluster Ui Review Recording Message-ID: For those of you who missed the Create and Import Gluster Cluster UI review meeting today, a recording link can be found here: https://tendrl.atlassian.net/wiki/display/TEN/UX+Designs+and+Design+Reviews If you have additional comments or feedback on these designs, please post your comments directly to the InVision document (you'll find the links on the wiki page above). Ju and I will review and respond to all comments in InVision. Once again, thanks to those who attended for a good meeting. Regards, Matt -- Matt Carrano Sr. Interaction Designer Red Hat, Inc. mcarrano at redhat.com From japplewh at redhat.com Thu Oct 6 14:57:49 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Thu, 6 Oct 2016 10:57:49 -0400 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: It's fixed now..and verified.. On Thu, Oct 6, 2016 at 10:44 AM, John Spray wrote: > On Thu, Oct 6, 2016 at 1:24 PM, Jeff Applewhite > wrote: > > On Wed, Oct 5, 2016 at 8:00 AM, John Spray wrote: > > > >> On Wed, Oct 5, 2016 at 4:40 AM, Ju Lim wrote: > >> > Team: > >> > > >> > The demo recording from Sprint 1 demo is now available along with the > >> final > >> > agenda and notes (feedback and Q&A). They can be found at Sprint 001 > >> Demo > >> > Agenda & Notes > >> > pageId=7733261 > >> >. > >> > >> This looks to be behind a login page? (I've avoided having an account > >> on that particular site so far) > >> > > > > Hi John > > > > I think I found the cause - there are separate settings for Confluence > and > > Jira - I had opened up the Jira but not the Confluence - now they are > both > > setup for anonymous access now. Let me know if this is not fixed. > > It's still asking for a login here. BTW when modifying settings you > can check if anonymous access works by opening an incognito browser > window and trying to access that URL. > ?yes ?- I know - just didn't bother.. > > John > > > Thanks > > > > Jeff > > > > > >> > >> John > >> > >> > If folks have any comments and/or feedback on the demo(s), please > reply > >> all > >> > to this email thread. > >> > > >> > Going forward, all Sprint goals, demo agenda, demo recording, and > notes > >> may > >> > be found at the Tendrl Sprints Landing Page > >> > , which is an > >> index > >> > of all sprints to-date. > >> > > >> > Thank you, > >> > Ju > >> > _______________________________________________ > >> > Tendrl-devel mailing list > >> > Tendrl-devel at redhat.com > >> > https://www.redhat.com/mailman/listinfo/tendrl-devel > >> > >> _______________________________________________ > >> Tendrl-devel mailing list > >> Tendrl-devel at redhat.com > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > >> > > > > > > > > -- > > Jeff Applewhite > > Principal Product Manager > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From julim at redhat.com Thu Oct 6 15:07:00 2016 From: julim at redhat.com (Ju Lim) Date: Thu, 6 Oct 2016 11:07:00 -0400 Subject: [Tendrl-devel] Tendrl Sprint 1 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: Jeff: The Kanban board cannot be accessed publicly because of how JIRA Software license works. See the following snippet from https://answers.atlassian.com/questions/33135097/is-it-possible-to-show-my-kanban-board-for-anonymous-users : "Since the migration to JIRA 7's licensing it's now necessary to have a 'JIRA Software' license in order to access JIRA Boards. As a result anonymous users cannot view Boards any longer. Have a read of https://confluence.atlassian.com/migration/jira-7/cloud_jira+agile_product-changes for more details on the change, specifically there's a table further down that describes the differences between each license/role." However, through https://tendrl.atlassian.net/wiki/spaces/TEN, you can link to see all issues anonymously. Regards, Ju On Thu, Oct 6, 2016 at 10:57 AM, Jeff Applewhite wrote: > It's fixed now..and verified.. > > On Thu, Oct 6, 2016 at 10:44 AM, John Spray wrote: > > > On Thu, Oct 6, 2016 at 1:24 PM, Jeff Applewhite > > wrote: > > > On Wed, Oct 5, 2016 at 8:00 AM, John Spray wrote: > > > > > >> On Wed, Oct 5, 2016 at 4:40 AM, Ju Lim wrote: > > >> > Team: > > >> > > > >> > The demo recording from Sprint 1 demo is now available along with > the > > >> final > > >> > agenda and notes (feedback and Q&A). They can be found at Sprint > 001 > > >> Demo > > >> > Agenda & Notes > > >> > > pageId=7733261 > > >> >. > > >> > > >> This looks to be behind a login page? (I've avoided having an account > > >> on that particular site so far) > > >> > > > > > > Hi John > > > > > > I think I found the cause - there are separate settings for Confluence > > and > > > Jira - I had opened up the Jira but not the Confluence - now they are > > both > > > setup for anonymous access now. Let me know if this is not fixed. > > > > It's still asking for a login here. BTW when modifying settings you > > can check if anonymous access works by opening an incognito browser > > window and trying to access that URL. > > > > ?yes ?- I know - just didn't bother.. > > > > > John > > > > > Thanks > > > > > > Jeff > > > > > > > > >> > > >> John > > >> > > >> > If folks have any comments and/or feedback on the demo(s), please > > reply > > >> all > > >> > to this email thread. > > >> > > > >> > Going forward, all Sprint goals, demo agenda, demo recording, and > > notes > > >> may > > >> > be found at the Tendrl Sprints Landing Page > > >> > , which is > an > > >> index > > >> > of all sprints to-date. > > >> > > > >> > Thank you, > > >> > Ju > > >> > _______________________________________________ > > >> > Tendrl-devel mailing list > > >> > Tendrl-devel at redhat.com > > >> > https://www.redhat.com/mailman/listinfo/tendrl-devel > > >> > > >> _______________________________________________ > > >> Tendrl-devel mailing list > > >> Tendrl-devel at redhat.com > > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > >> > > > > > > > > > > > > -- > > > Jeff Applewhite > > > Principal Product Manager > > > _______________________________________________ > > > Tendrl-devel mailing list > > > Tendrl-devel at redhat.com > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > -- > Jeff Applewhite > Principal Product Manager > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From mbukatov at redhat.com Thu Oct 6 16:02:06 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 6 Oct 2016 18:02:06 +0200 Subject: [Tendrl-devel] purpose of Tendrl/tendrl repo Message-ID: Dear all, Looking at the https://github.com/Tendrl/tendrl repo, I'm a little puzzled. What is the purpose of this repo and what is expected to be there? We have separate repositories for all components, ui, documentation and a web site, so no clear meaning of this repository comes into my mind. Thanks for the explanation. -- Martin Bukatovic USM QE team From mbukatov at redhat.com Thu Oct 6 16:21:41 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 6 Oct 2016 18:21:41 +0200 Subject: [Tendrl-devel] system specific terms Message-ID: <9a27b831-0625-b34a-fd1a-154c7de511f7@redhat.com> Dear all, in Tendrl Architectural Guidelines[1] I see the following statement: > The interface should convey the state of the Storage Systems > using System-specific terms and representations. Considering GlusterFS project[2] refers to set of nodes hosting data as "trusted storage pool", I see a conflict with current design which uses term "cluster" for "trusted storage pool" and "host" for a gluster peer. While I guess that the current design makes more sense, we would need to decide where to put the line and update the statement from tendrl guidelines, otherwise it will be ignored in practice. Or would it make sense to make system specific terms available in the ui as well somehow, eg. in a tool tip? [1] https://github.com/Tendrl/documentation/blob/master/tendrl-architectural-guidelines.adoc [2] https://gluster.readthedocs.io/en/latest/Administrator%20Guide/glossary/ -- Martin Bukatovic USM QE team From jspray at redhat.com Fri Oct 7 10:28:21 2016 From: jspray at redhat.com (John Spray) Date: Fri, 7 Oct 2016 12:28:21 +0200 Subject: [Tendrl-devel] Deployment/dependency questions Message-ID: Based on the code I see here: https://github.com/Tendrl/tendrl/pull/5/files It seems that ruby and etcd are going to be requirements. Ruby: is Tendrl only going to use dependencies that are already present in major distros, or are you guys going to be packaging (rpm/deb) additional ruby modules that you depend on? Are you going to depend on a particular version of ruby? (some distros have 1.9.x, some have 2.x, I'm not an expert on any differences). etcd: are you expecting customers to deploy three nodes to run Tendrl on, or are you expecting to co-exist with Ceph mons? If on the mons, will etcd be expected to share drives with the mons, or go somewhere else? John From mbukatov at redhat.com Fri Oct 7 12:58:24 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Fri, 7 Oct 2016 14:58:24 +0200 Subject: [Tendrl-devel] Deployment/dependency questions In-Reply-To: References: Message-ID: On 10/07/2016 12:28 PM, John Spray wrote: > Ruby: is Tendrl only going to use dependencies that are already > present in major distros, or are you guys going to be packaging > (rpm/deb) additional ruby modules that you depend on? Are you going > to depend on a particular version of ruby? (some distros have 1.9.x, > some have 2.x, I'm not an expert on any differences). I have a related question: What component is considered to be implemented in ruby? -- Martin Bukatovic USM QE team From julim at redhat.com Fri Oct 7 16:56:15 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 7 Oct 2016 12:56:15 -0400 Subject: [Tendrl-devel] Create and Import Cluster Ui Review Recording In-Reply-To: References: Message-ID: Thanks Matt for publishing the recording. On Thu, Oct 6, 2016 at 10:54 AM, Matt Carrano wrote: > For those of you who missed the Create and Import Gluster Cluster UI review > meeting today, a recording link can be found here: > https://tendrl.atlassian.net/wiki/display/TEN/UX+Designs+ > and+Design+Reviews > > If you have additional comments or feedback on these designs, please post > your comments directly to the InVision document (you'll find the links on > the wiki page above). Ju and I will review and respond to all comments in > InVision. > > Once again, thanks to those who attended for a good meeting. > > Regards, > > Matt > > -- > Matt Carrano > Sr. Interaction Designer > Red Hat, Inc. > mcarrano at redhat.com > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From julim at redhat.com Fri Oct 7 19:25:12 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 7 Oct 2016 15:25:12 -0400 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: <571c31f3-fb69-08d7-f88d-62f511b35e6a@redhat.com> Message-ID: Team: Based on all the feedback received, I've explored a few different alternatives, and have updated UX Design Approach and Information Architecture to reflect the latest recommendation. Slides updated are as follow: 9, 10, 13-16, 18, 21. If you're interested in seeing alternate options considered/explored, please look at slides 29-33. To summarize the latest recommendation, we still have 2-level of navigation, but for the storage entities (e.g. Pools, RBDs, File Shares), they are no longer grouped under a "Storage" section and appear in the 1st level navigation. The only item with 2-levels of navigation in the Navigation Bar is the "Admin" section, which will expand (and collapse) within the single-panel navigation (column). We will not expand to the right for the 2nd-level navigation as was done previously -- this should save on a lot of wasted footprint and hopefully improve the navigation experience. The impact of bringing all the storage entities to the top-level (1st level navigation) does affect the first-time (getting started) experience. Specifically, as we don't want to unpleasantly surprise the user with a menu structure that changes*, we will enhance the first-time experience to be that of a guided experience to walk user through some basic steps to get started before "landing" in the Tendrl UI. This is only applicable for the initial setup, and once a cluster exists, user will no longer see this first-time guided experience/workflow. * the menu structure will vary based on various deployment scenarios: (1) ceph deployment only (2) gluster deployment only (3) ceph + gluster deployment John -- wrt user identities, what I mean by infrequent use case is that it typically is setup at ?install and configuration? time and not reset on a day-to-day basis vs. monitoring or troubleshooting type operations. Please do review this latest proposal and feel free to comment. Thank you, Ju On Tue, Oct 4, 2016 at 8:45 AM, Kanagaraj M wrote: > Though it might be possible, using cookies for storing this kind of > information is not good. The content of the app will vary based on the > user browser. The same user will see different content on a different > browser. Also its not a single user who will be using Tendrl to manage the > cluster. > > If at all this kind of information needs to be stored, It should be a > setting in the application which can be done either during installation or > later. > > On Tue, Oct 4, 2016 at 5:49 PM, Jeff Applewhite > wrote: > > > One thought I had was that Tendrl could "remember" the last used storage > > backend (in the case where you have Ceph and Gluster) and simply default > to > > displaying whatever that was. This is easier on the client side with > > cookies than server side I suppose. > > > > In the case where it has only installed or imported either Ceph or > Gluster > > it should not confuse the user with extraneous information related to > other > > SDS technologies. Tendrl knows what it's scope is here. > > > > Is this doable? > > > > On Tue, Oct 4, 2016 at 6:00 AM, Mrugesh Karnik > > wrote: > > > > > On 4 October 2016 at 13:17, John Spray wrote: > > > > What I was really asking was whether at install time the console will > > > > have any knowledge of whether it's going to manage Ceph or Gluster, > so > > > > that we can hide parts that aren't needed. > > > > > > Console does need to understand which provisioning flow to invoke at > > > install time. Currently, the provisioning framework is being worked out > > and > > > I need to have a discussion with Alfredo as well. > > > > > > -- > > > Mrugesh > > > _______________________________________________ > > > Tendrl-devel mailing list > > > Tendrl-devel at redhat.com > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > > > > > > -- > > Jeff Applewhite > > Principal Product Manager > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From adeza at redhat.com Fri Oct 7 20:23:29 2016 From: adeza at redhat.com (Alfredo Deza) Date: Fri, 7 Oct 2016 16:23:29 -0400 Subject: [Tendrl-devel] Installing Ceph via Tendrl In-Reply-To: References: Message-ID: On Tue, Oct 4, 2016 at 7:08 AM, Alfredo Deza wrote: > On Tue, Oct 4, 2016 at 3:30 AM, Mrugesh Karnik wrote: >> On 1 October 2016 at 02:31, Sankarshan Mukhopadhyay < >> sankarshan.mukhopadhyay at gmail.com> wrote: >>> >>> On Sat, Oct 1, 2016 at 12:53 AM, Alfredo Deza wrote: >>> > All of that will have to be done again in Tendrl. Can that design be >>> > explained a bit? How is this generalized approach going to tackle >>> > these issues? How is it going to approach tackling any other Ansible >>> > playbook or role? >>> > >>> > How has the ceph-installer failed here? When the project started, it >>> > did so with a "documentation first" approach, and we presented the API >>> > first so that the consumers of it could have an immediate idea on the >>> > HTTP API proposed. We did it this way because we thought it was >>> > imperative to present the ideas while the application itself was being >>> > crafted to improve the throughput on communication. >>> >>> It is a bit late/early here - the detail about the provisioning flow >>> is (as I remember) part of this sprint or, a coming one and is being >>> worked on. Perhaps we could provide an interim update and then take >>> this conversation ahead. >> >> I'm working on this in the current sprint. Updates will be made to the >> issue at https://github.com/Tendrl/documentation/issues/29 >> >> Also, Alfredo, do you think we can have a quick conversation somewhere >> towards the end of this week? I think it is still important to define and narrow the uncertainty of my questions in this thread. The github issue (https://github.com/Tendrl/documentation/issues/29) looks empty and we didn't get to meet this week. Please find some time and let me know when that can be to address the concerns we have. > > Of course. Any time! >> >> Thanks, >> Mrugesh >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel From shtripat at redhat.com Wed Oct 12 04:00:02 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Wed, 12 Oct 2016 09:30:02 +0530 Subject: [Tendrl-devel] Deployment/dependency questions In-Reply-To: References: Message-ID: <3c7cba28-22c0-1af7-6a33-af1917f32ac6@redhat.com> On 10/07/2016 06:28 PM, Martin Bukatovic wrote: > On 10/07/2016 12:28 PM, John Spray wrote: >> Ruby: is Tendrl only going to use dependencies that are already >> present in major distros, or are you guys going to be packaging >> (rpm/deb) additional ruby modules that you depend on? Are you going >> to depend on a particular version of ruby? (some distros have 1.9.x, >> some have 2.x, I'm not an expert on any differences). > I have a related question: What component is considered to > be implemented in ruby? > Martin, I understand the REST (tendrl app) part is to be developed in Ruby. May be Anoop can confirm the same. Regards, Shubhendu From mrugesh at brainfunked.org Wed Oct 12 09:22:43 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Wed, 12 Oct 2016 14:52:43 +0530 Subject: [Tendrl-devel] purpose of Tendrl/tendrl repo In-Reply-To: References: Message-ID: On 6 October 2016 at 21:32, Martin Bukatovic wrote: > > Dear all, > > Looking at the https://github.com/Tendrl/tendrl repo, I'm a little > puzzled. What is the purpose of this repo and what is expected to > be there? We have separate repositories for all components, ui, > documentation and a web site, so no clear meaning of this repository > comes into my mind. This repository is to host the codebase for the API and the logic for parsing the definition YAML files. -- Mrugesh From mbukatov at redhat.com Wed Oct 12 09:23:43 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 12 Oct 2016 11:23:43 +0200 Subject: [Tendrl-devel] Deployment/dependency questions In-Reply-To: <3c7cba28-22c0-1af7-6a33-af1917f32ac6@redhat.com> References: <3c7cba28-22c0-1af7-6a33-af1917f32ac6@redhat.com> Message-ID: On 10/12/2016 06:00 AM, Shubhendu Tripathi wrote: > I understand the REST (tendrl app) part is to be developed in Ruby. > May be Anoop can confirm the same. Thank for the information, I wasn't aware of that. What is the reason for this? At first sight, this seems strange to me as other components are written in python. -- Martin Bukatovic USM QE team From mrugesh at brainfunked.org Wed Oct 12 09:30:35 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Wed, 12 Oct 2016 15:00:35 +0530 Subject: [Tendrl-devel] system specific terms In-Reply-To: <9a27b831-0625-b34a-fd1a-154c7de511f7@redhat.com> References: <9a27b831-0625-b34a-fd1a-154c7de511f7@redhat.com> Message-ID: On 6 October 2016 at 21:51, Martin Bukatovic wrote: > > Dear all, > > in Tendrl Architectural Guidelines[1] I see the following > statement: > > > The interface should convey the state of the Storage Systems > > using System-specific terms and representations. > > Considering GlusterFS project[2] refers to set of nodes hosting > data as "trusted storage pool", I see a conflict with current > design which uses term "cluster" for "trusted storage pool" > and "host" for a gluster peer. > > While I guess that the current design makes more sense, we would > need to decide where to put the line and update the statement from > tendrl guidelines, otherwise it will be ignored in practice. > > Or would it make sense to make system specific terms available > in the ui as well somehow, eg. in a tool tip? The intent is to convey the information to the administrator in the same terminology as the storage system itself. As an administrator, I should be able to read gluster documentation and be able to find my way around the tendrl interface to accomplish my tasks. Internally, there are certain global entities that tendrl needs to understand. Both hosts and clusters are such entities. They may be referred to by some other term by the storage systems, but as part of the tendrl integration, these entities need to be pointed out to tendrl, as such. Apart from such specifically `hard-coded' entities, any terminology specific to a storage system will be reused. A simple example is the gluster terminology of volumes and bricks as opposed to ceph's pools and osds. In either of these cases, tendrl doesn't need to understand what that entity means, it simply needs to be able to convey it's status or enable defined actions to be invoked on them. These system specific terms will be carried over to the administrative interface. -- Mrugesh From rkanade at redhat.com Wed Oct 12 10:31:02 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Wed, 12 Oct 2016 16:01:02 +0530 Subject: [Tendrl-devel] Heketi and Tendrl Gluster 1st release Message-ID: Hi, Over the past few months, we discussed ways in which Tendrl and Heketi can leverage each other's capabilities. We came up with couple of ways in which Tendrl-Heketi can be beneficial to the end user 1) Tendrl-Heketi together provide a completely managed (meaning: install Tendrl-Heketi on your storage nodes and let Tendrl-Heketi manage your storage functions) 2) Install Tendrl on your storage nodes and let your storage/system administrators manage storage resource bootstrapping and scaling/allocation In the long run, we would like both these options to be available for any given cluster for our end users. However for Tendrl 1st release we would like to present a more basic approach (option 2 above) which would enable the customers and their administrators closer control of their storage clusters and underlying storage resources. Regards, Rohan Kanade From mbukatov at redhat.com Wed Oct 12 11:45:03 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 12 Oct 2016 13:45:03 +0200 Subject: [Tendrl-devel] system specific terms In-Reply-To: References: <9a27b831-0625-b34a-fd1a-154c7de511f7@redhat.com> Message-ID: On 10/12/2016 11:30 AM, Mrugesh Karnik wrote: > The intent is to convey the information to the administrator in the same > terminology as the storage system itself. As an administrator, I should be > able to read gluster documentation and be able to find my way around the > tendrl interface to accomplish my tasks. > > Internally, there are certain global entities that tendrl needs to > understand. Both hosts and clusters are such entities. They may be referred > to by some other term by the storage systems, but as part of the tendrl > integration, these entities need to be pointed out to tendrl, as such. > Apart from such specifically `hard-coded' entities, any terminology > specific to a storage system will be reused. A simple example is the > gluster terminology of volumes and bricks as opposed to ceph's pools and > osds. In either of these cases, tendrl doesn't need to understand what that > entity means, it simply needs to be able to convey it's status or enable > defined actions to be invoked on them. These system specific terms will be > carried over to the administrative interface. Thanks for verification of this case. Would it make sense to use your explanation and add it to the tendrl architecture guide[1]? https://github.com/Tendrl/documentation/blob/master/tendrl-architectural-guidelines.adoc -- Martin Bukatovic USM QE team From mbukatov at redhat.com Wed Oct 12 14:34:12 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 12 Oct 2016 16:34:12 +0200 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 Message-ID: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> Dear all, if I understand the design of tendrl right, gluster_bridge would be installed on the storage machines, in this case gluster storage machines. Since glusterfs supports rhel/centos 6 as well, we would need to do the same for gluster_bridge, right? For gluster_bridge to run on el6, I see the following issues: * gluster_bridge need to run on python 2.6 * there are no etcd packages for epel 6 (BZ 1217518) Is my understanding correct? -- Martin Bukatovic USM QE team From japplewh at redhat.com Wed Oct 12 14:50:36 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Wed, 12 Oct 2016 10:50:36 -0400 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 In-Reply-To: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> References: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> Message-ID: Yes it sounds correct with the exception that containerizing the gluster_bridge for rhel 6 would solve these problems by bundling rhel 7 libraries for the dependencies. This is under consideration as I understand it. On Wed, Oct 12, 2016 at 10:34 AM, Martin Bukatovic wrote: > Dear all, > > if I understand the design of tendrl right, gluster_bridge would > be installed on the storage machines, in this case gluster storage > machines. Since glusterfs supports rhel/centos 6 as well, we would > need to do the same for gluster_bridge, right? > > For gluster_bridge to run on el6, I see the following issues: > > * gluster_bridge need to run on python 2.6 > * there are no etcd packages for epel 6 (BZ 1217518) > > Is my understanding correct? > > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From mbukatov at redhat.com Wed Oct 12 15:23:04 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 12 Oct 2016 17:23:04 +0200 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 In-Reply-To: References: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> Message-ID: On 10/12/2016 04:50 PM, Jeff Applewhite wrote: > Yes it sounds correct with the exception that containerizing the > gluster_bridge for rhel 6 would solve these problems by bundling rhel 7 > libraries for the dependencies. This is under consideration as I understand > it. As long as GlusterFS project supports this kind of deployment, this looks like an interesting solution. Thanks for the update. -- Martin Bukatovic USM QE team From kdreyer at redhat.com Wed Oct 12 17:47:24 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Wed, 12 Oct 2016 11:47:24 -0600 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 In-Reply-To: References: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> Message-ID: On Wed, Oct 12, 2016 at 8:50 AM, Jeff Applewhite wrote: > Yes it sounds correct with the exception that containerizing the > gluster_bridge for rhel 6 would solve these problems by bundling rhel 7 > libraries for the dependencies. This is under consideration as I understand > it. Does RHEL 6 support containers? - Ken From rhartman at redhat.com Wed Oct 12 18:02:35 2016 From: rhartman at redhat.com (Ryan Hartman) Date: Wed, 12 Oct 2016 14:02:35 -0400 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 In-Reply-To: References: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> Message-ID: On Wed, Oct 12, 2016 at 1:47 PM, Ken Dreyer wrote: > On Wed, Oct 12, 2016 at 8:50 AM, Jeff Applewhite wrote: >> Yes it sounds correct with the exception that containerizing the >> gluster_bridge for rhel 6 would solve these problems by bundling rhel 7 >> libraries for the dependencies. This is under consideration as I understand >> it. > > Does RHEL 6 support containers? No, we only ship docker for rhel7 > > - Ken > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From rwheeler at redhat.com Wed Oct 12 19:45:05 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Wed, 12 Oct 2016 13:45:05 -0600 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 In-Reply-To: References: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> Message-ID: <51103234-0ae6-606e-513b-a434f1d29493@redhat.com> On 10/12/2016 12:02 PM, Ryan Hartman wrote: > On Wed, Oct 12, 2016 at 1:47 PM, Ken Dreyer wrote: >> On Wed, Oct 12, 2016 at 8:50 AM, Jeff Applewhite wrote: >>> Yes it sounds correct with the exception that containerizing the >>> gluster_bridge for rhel 6 would solve these problems by bundling rhel 7 >>> libraries for the dependencies. This is under consideration as I understand >>> it. >> Does RHEL 6 support containers? > No, we only ship docker for rhel7 This is supposed to be an upstream list - not a Red Hat product list. What I suggest is that we focus on doing the right thing for the current upstream new targets and let Red Hat (or others) worry about what their distros ship or not. Regards, Ric From mbukatov at redhat.com Thu Oct 13 08:29:50 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 13 Oct 2016 10:29:50 +0200 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 In-Reply-To: <51103234-0ae6-606e-513b-a434f1d29493@redhat.com> References: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> <51103234-0ae6-606e-513b-a434f1d29493@redhat.com> Message-ID: On 10/12/2016 09:45 PM, Ric Wheeler wrote: > This is supposed to be an upstream list - not a Red Hat product list. > > What I suggest is that we focus on doing the right thing for the current > upstream new targets and let Red Hat (or others) worry about what their > distros ship or not. You are right. But since the tendrl project expects it's components to be installed on machines running software defined storage and such machines are usually running some long term stable linux distros, tendrl project needs to take this into account during some decisions in upstream as well. Which is why I asked about the upstream plan for gluster_bridge on epel 6. And so far, it seems that the upstream direction here is to ignore this issue, stay with python 2.7 and other dependencies as it is now. Is my impression on this correct? -- Martin Bukatovic USM QE team From amukherj at redhat.com Thu Oct 13 09:10:43 2016 From: amukherj at redhat.com (Atin Mukherjee) Date: Thu, 13 Oct 2016 14:40:43 +0530 Subject: [Tendrl-devel] gluster_bridge on rhel/centos 6 In-Reply-To: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> References: <83c0089f-3cc5-3fc1-a0a4-13bc002e060c@redhat.com> Message-ID: On Wed, Oct 12, 2016 at 8:04 PM, Martin Bukatovic wrote: > Dear all, > > if I understand the design of tendrl right, gluster_bridge would > be installed on the storage machines, in this case gluster storage > machines. Since glusterfs supports rhel/centos 6 as well, we would > need to do the same for gluster_bridge, right? > > For gluster_bridge to run on el6, I see the following issues: > > * gluster_bridge need to run on python 2.6 > * there are no etcd packages for epel 6 (BZ 1217518) > As per my understanding of the arch, gluster bridge will not spawn etcd on the storage nodes rather it will be maintained by USM engine which can be el7 specific. > Is my understanding correct? > > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- --Atin From rkanade at redhat.com Thu Oct 13 11:18:17 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Thu, 13 Oct 2016 16:48:17 +0530 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: On Thu, Oct 6, 2016 at 7:26 PM, Gregory Meno wrote: > > On Wed, Oct 5, 2016 at 5:57 AM, Rohan Kanade wrote: > > Calamari is an important component in the Ceph management/ops process. In > > the past, The Skyrings [0] project was able to utilize Calamari via the > > Calamari REST API, the integration did ran into some problems as mentioned > > below. > > > > - Calamari-server would often be a single point of failure as it is > > installed on the ceph-mon nodes > > Consider that monitor nodes are the right place to run and there are > usually 3 or 5 monitors. > Is that enough? I think we are at a point now where this would be > reasonable to run on each monitor. > What is more interesting to me is how you see the interaction happening. > > Does each process instance track the mon leader and proxy requests to > the instance that is on the leader? Tendrl calamari interaction will happen via calamari REST API, Tendrl wont be tracking calamari processes in any part of the cluster > John: have we considered what the ceph-mgr answer to this is? > > Do you expect to have perfect replication of state related to requests > to change the ceph cluster? eg: If we call GET /calamari/osds , it should provide a list of osds up until that point in time > How do you want to discover available endpoints? On a new cluster or an existing cluster, Tendrl will assume calamari is pre-installed and functional (this will be part of Tendrl pre-requisites) Then, while installing Tendrl itself, it should be provided with the calamari REST API endpoint either via conf files or user input. > > > - Querying Calamari REST API gave inconsistent data about the ceph cluster > > state, this might be happening because of sync issues occurring due to > > ceph-mons not being in quorum etc. > > Specific tickets would go a long way here. > > > - Calamari required to be installed on only one of the ceph-mons, This > > introduces very common scalability issues. > > I think we can relax this requirement if we can agree on expectations above. > > > > - Calamari relies on older version of saltstack for messaging and other > > communication, this introduces packaging and performance overheads > > This is not the case. We only use salt to send events to tendrl and > it's something I consider to be a mistake. > I think salt can be taken out of the picture in future releases Ok, removal of salt works for Tendrl > > > > > These issues must be re-verified and fixed if they still persist in > > Calamari. > > > > > > > > > > The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried to > > address these issues by: > > Did they succeed? Yes > > > > > - Distributing the tendrl ceph bridge to talk to all ceph-mons in the > > cluster. > > - Gathering data (3 second poll interval) from ceph-mons only when they are > > in quorum and directly talking to ceph-mons via librados and using ceph > > admin sockets to query valid cluster maps and other ceph data. > > Since ceph_bridge is taken directly from calamari it's fair to say > that we posses the same capabilities Please check the forked repository for changes in capabilities > > > - Tendrl ceph bridge tries to address the single point of failure issue by > > having multiple instances of the bridge for each ceph-mon in the cluster. > > Would you please show me where ceph_bridge is aware of other > instances? Or are these multiple copies sharing nothing? For READ of cluster state, we only pull data out of the cluster if all mons are in quorum. We have a single copy of the state which is refreshed at regular intervals as well as per specific events. > > > - Tendrl ceph bridge has centralized communications via an etcd cluster > > which removes need to maintain saltstack . > > > > > > > > Moving forward, we would like Calamari to accommodate these improvements > > back into the calamari code base, Below are the detailed requirements: > > > > - Need Calamari REST API for querying ceph admin sockets [2], More details: > > Let's not expose this. I would consider it to be a challenge to your > goal of having multiple instances > as you're talking to a specific socket. Really what you get is info > about clusters that are being setup or severly damaged this should be > built into whatever service is responsible for identifying clusters > and talking to them e.g. calamari and in future ceph-mgr. This is required specifically to verify correctness of cluster state reported by other instances. And as i mentioned we only pull data from sockets if all mons are in quorum. > > > - Need Calamari REST API for providing all cluster maps (monitor, osd, > > crush, mds, pg, health, config etc) > > This seems like a strange request. Why do you want access to the raw > data? You'll have to parse it to make sense of it the same way we > already do in calamari and in future ceph-mgr. Calamari already provides this via REST API http://calamari.readthedocs.io/en/latest/calamari_rest/resources/api_example_api_v2_cluster__fsid__sync_object.html Tendrl is simply asking this to be supported in future. > > > - Calamari REST API must represent the ceph cluster state correctly, no > > mismatch should occur between the True state of the cluster and the state > > represented by the Calamari REST API > > Happy to work to fix specific dependencies. Tickets are best. > > > - Calamari REST API should provide access to monitoring actions like "ceph > > -w", "ceph tell" etc > > Help me understand the specifics of what you want to accomplish here. > the output of ceph -w can be synthesized from looking at the cluster > maps. and ceph tell is typically used for troubleshooting specific > instances where you want to set new config with out changing ceph.conf > and restarting. That means it applies to a single instance, not > cluster wide. ceph -w will be used to generate events based on the output. Computing such events from cluster maps is doable too but is more work. Tendrl works not only on a cluster context but also at a node context, ceph tell is required for operations which pertain to specific nodes. > > > - Calamari REST API should provide access to operational actions via tools > > like "rbd" etc. > > Agreed. > > > - Calamari REST API should be able to service and perform well on 2-4 second > > polling intervals on the REST API > > Yes I agree here too. > > > > > > > These requirements can be run through existing calamari functional and > > performance test cases and we should make additions to these test cases to > > accommodate above requirements. > > > > These requirements would be further expanded into individual BZs or tracking > > items in Calamari > > > > > > [0]: https://github.com/skyrings > > [1]: https://github.com/tendrl/ceph_bridge > > [2]: > > http://docs.ceph.com/docs/jewel/rados/operations/monitoring/#using-the-admin-socket From deb at redhat.com Thu Oct 13 12:38:19 2016 From: deb at redhat.com (Soumya Deb) Date: Thu, 13 Oct 2016 18:08:19 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: <8ebb81cd-9ad1-6945-0ddb-3f091a6a1ba6@redhat.com> References: <8ebb81cd-9ad1-6945-0ddb-3f091a6a1ba6@redhat.com> Message-ID: There has been some changes to the choices made regarding Node.js usage. Before explaining that, I'll try answering some of the previous questions. Please bear in mind, the recent changes invalidates some of the questions raised, but I'll answer them anyways. On 6 October 2016 at 17:21, Martin Bukatovic wrote: > So let's put in a different way: is node.js so superior to other > (also very popular) ways of web development, so that it would outweight > the cost bringing in the whole node.js stack itself? > Yes. > Who runs node.js on storage machines it theirs datacenter nowadays? I > would guess that nobody. I have no data about how many people are running node.js on storage machies. But node.js is one of the most popular and first class platform available as package/image/cartridge in (statutory: almost) all IaaS/PaaS. Is node.js easily available on stable linux distributions, so that we don't > have to worry about it? I don't think so. https://nodejs.org/en/download/package-manager/ Which means that if we bring that dependency it, we need to account > for increased complexity and the maintenance of whole node.js runtime. > False assumption (as per previous link), at least for our platform priorities. Please correct me if I'm wrong, but there seems to be some amount of discomfort regarding Node.js, without giving enough effort to accommodate it for our purposes, even when it might be appropriate. There also seems to be some assumptions affecting the decisions, without being factually right. ----- With that said, there has been some changes in how we will use Node.js for Tendrl frontend. In short: - For deployment: there won't be any Node.js dependency to run the frontend package, built from the code (as it will be a static webapp) - For development, building & tests: node.js will be used for various purposes Hope that helps and is a sufficient balance of feasibility. From shtripat at redhat.com Fri Oct 14 06:24:51 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Fri, 14 Oct 2016 11:54:51 +0530 Subject: [Tendrl-devel] Issues while setting up tendrl with gluster bridge and executing a volume creation Message-ID: <79cf6a91-caa3-246c-4c82-3cf3a1064025@redhat.com> Hi, I followed the below steps and running into few issues as mentioned down. 1. Installed etcd on fedora24 node and its up and running. Able to submit jobs to it. 2. Created 2 gluster nodes (centos7) and setup bridge_common and gluster_bridge on these nodes. I had to build gluster locally from https://github.com/samikshan/glusterfs/tree/getstatecli as command "gluster get-state" is not yet available in nightly builds. 3. While starting tendrl-gluster-bridge ran into an issue where command "gluster get-state" is not correct in tendrl/gluster_bridge/manager/manager.py. Actually it should be mentioned as "gluster get-state glusterd odir " and not "gluster daemon get-state odir " as available in master of gluster-bridge. Corrected the same locally in my setup and then tendrl-gluster-bridge starts fine and sarts creating glusterd state files. 4. Now I ran sample volume creation test program tendrl/gluster_bridge/etc/create_api_job.py which submits a job to etcd. Had to do change etcd node and port reference. It does create the volume creation job in etcd and I can see the job using URL "http://:2379/v2/keys/api_job_queue/" with status as "processing". Now after step-4 actual issues I face is that tendrl-gluster-bridge hangs and stops creating glusterd state files, and the volume creation job remains in processing state itself. Also I dont see a volume created on the gluster cluster (checking using "gluster volume status all"). Need support to resolve this. @Rohan, I had scheduled a BJ session to debug the same at 11:30 AM IST today, but as you dont seem to be online, can you suggest me a new time when we can debug and close the issue? Thanks and Regards, Shubhendu From amukherj at redhat.com Fri Oct 14 06:39:39 2016 From: amukherj at redhat.com (Atin Mukherjee) Date: Fri, 14 Oct 2016 12:09:39 +0530 Subject: [Tendrl-devel] Issues while setting up tendrl with gluster bridge and executing a volume creation In-Reply-To: <79cf6a91-caa3-246c-4c82-3cf3a1064025@redhat.com> References: <79cf6a91-caa3-246c-4c82-3cf3a1064025@redhat.com> Message-ID: On Fri, Oct 14, 2016 at 11:54 AM, Shubhendu Tripathi wrote: > Hi, > > I followed the below steps and running into few issues as mentioned down. > > 1. Installed etcd on fedora24 node and its up and running. Able to submit > jobs to it. > 2. Created 2 gluster nodes (centos7) and setup bridge_common and > gluster_bridge on these nodes. I had to build gluster locally from > https://github.com/samikshan/glusterfs/tree/getstatecli as command > "gluster get-state" is not yet available in nightly builds. > Which Gluster version are you using? 3.9rc should have this change and the rpms are available at [1] 3. While starting tendrl-gluster-bridge ran into an issue where command > "gluster get-state" is not correct in tendrl/gluster_bridge/manager/manager.py. > Actually it should be mentioned as "gluster get-state glusterd odir " > and not "gluster daemon get-state odir " as available in master of > gluster-bridge. Corrected the same locally in my setup and then > tendrl-gluster-bridge starts fine and sarts creating glusterd state files. > I suspect this issue is because of Samikshan's github is no in sync with the changes which went into the GlusterFS mainline branch. > 4. Now I ran sample volume creation test program > tendrl/gluster_bridge/etc/create_api_job.py which submits a job to etcd. > Had to do change etcd node and port reference. It does create the volume > creation job in etcd and I can see the job using URL "http:// ip>:2379/v2/keys/api_job_queue/" with status as "processing". > > Now after step-4 actual issues I face is that tendrl-gluster-bridge hangs > and stops creating glusterd state files, and the volume creation job > remains in processing state itself. > Also I dont see a volume created on the gluster cluster (checking using > "gluster volume status all"). > > Need support to resolve this. > @Rohan, I had scheduled a BJ session to debug the same at 11:30 AM IST > today, but as you dont seem to be online, can you suggest me a new time > when we can debug and close the issue? > > Thanks and Regards, > Shubhendu > [1] https://cbs.centos.org/koji/taskinfo?taskID=111151 > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- --Atin From jspray at redhat.com Fri Oct 14 07:52:41 2016 From: jspray at redhat.com (John Spray) Date: Fri, 14 Oct 2016 08:52:41 +0100 Subject: [Tendrl-devel] Deployment/dependency questions In-Reply-To: References: Message-ID: On Fri, Oct 7, 2016 at 11:28 AM, John Spray wrote: > Based on the code I see here: > https://github.com/Tendrl/tendrl/pull/5/files > > It seems that ruby and etcd are going to be requirements. > > Ruby: is Tendrl only going to use dependencies that are already > present in major distros, or are you guys going to be packaging > (rpm/deb) additional ruby modules that you depend on? Are you going > to depend on a particular version of ruby? (some distros have 1.9.x, > some have 2.x, I'm not an expert on any differences). > > etcd: are you expecting customers to deploy three nodes to run Tendrl > on, or are you expecting to co-exist with Ceph mons? If on the mons, > will etcd be expected to share drives with the mons, or go somewhere > else? *bump* Would love to hear more about this. I'm coming at this from a "what will go on the mons?" perspective. Especially resource demands on the mons. If the intention is to running two paxos-like stores (etcd and ceph-mon) on the same triplet of nodes needs thought. The impact of the Ruby thing is more a function of how much extra packaging effort you guys are doing to support it, and which distros it's going to work on (i.e. if it is to run on the same nodes as ceph services, will it be available on the same set of distros that ceph is?) John From deb at redhat.com Fri Oct 14 09:16:43 2016 From: deb at redhat.com (Soumya Deb) Date: Fri, 14 Oct 2016 14:46:43 +0530 Subject: [Tendrl-devel] Moving code from Skyrings/kitoon to Tendrl/tendrl_frontend Message-ID: As you might imagine, that this process is technically trivial, but has some weight on the copyright and licensing side (or it doesn't - depending on license). Consulting Apache License v2.0 (used in Kitoon), it appears that it should be fine to copy, rebrand/modify and redistribute codes under this license (as long as the same license is retained). However, I propose mentioning the change as putting down the origin (Skyrings/kitoon) of Tendrl/tendrl_frontend in its README prominently, and also add a note to the README of Skyrings/kitoon about "active development has been moved to" information. However, if this isn't the right way to do it, or my understanding is wrong - please raise it ASAP. If nobody has any objection, we will move (copy over) the code to the new repo on Monday 2016/10/17 at UTC 12:00. Thanks, Deb From nigelb at redhat.com Fri Oct 14 12:15:09 2016 From: nigelb at redhat.com (Nigel Babu) Date: Fri, 14 Oct 2016 17:45:09 +0530 Subject: [Tendrl-devel] CI Plans Message-ID: <20161014121509.GA32384@gmail.com> Hello, Nishant, Mrugesh, and I have had a chat with Brian Stinson from the Centos team about consuming CentOS CI for Tendrl. I've been working on a good pattern for Gluster[1] and helping adopt this pattern for Tendrl as well. As a first step would, we will setup nightly rpm builds for Tendrl. In the future, we can consume these rpms for running the test framework. As an example of how this will work, here's where Gluster is headed in the next few weeks. QE is upstreaming Glusto, a generic framework to run tests written in python test frameworks (PyTest, nose, UnitTest) over a network. We will request 5 nodes from Centos CI, the first one will be the management node and the next 4 will be part of the cluster. We configure the nodes, install the nightly RPMs on there, and run our tests against this build. Once this passes, it gives us nightly builds, which are verified to be working. Once the test framework for Tendrl is ready and setup, we can proceed in doing more than packaging. Let me know if anyone has questions. [1]: https://github.com/nigelbabu/centos-ci-sample -- nigelb From kanagaraj.ktr at gmail.com Fri Oct 14 12:27:35 2016 From: kanagaraj.ktr at gmail.com (Kanagaraj M) Date: Fri, 14 Oct 2016 17:57:35 +0530 Subject: [Tendrl-devel] Moving code from Skyrings/kitoon to Tendrl/tendrl_frontend In-Reply-To: References: Message-ID: Should be alright. Since tendrl_frontend repo is already present, you might need to drop it and fork from kitoon. I am not sure if github provides a way to fork and merge. Thanks, Kanagaraj On Fri, Oct 14, 2016 at 2:46 PM, Soumya Deb wrote: > As you might imagine, that this process is technically trivial, but has > some weight on the copyright and licensing side (or it doesn't - depending > on license). > > Consulting Apache License v2.0 (used in Kitoon), it appears that it should > be fine to copy, rebrand/modify and redistribute codes under this license > (as long as the same license is retained). However, I propose mentioning > the change as putting down the origin (Skyrings/kitoon) of > Tendrl/tendrl_frontend in its README prominently, and also add a note to > the README of Skyrings/kitoon about "active development has been moved to" > information. > > However, if this isn't the right way to do it, or my understanding is wrong > - please raise it ASAP. > > If nobody has any objection, we will move (copy over) the code to the new > repo on Monday 2016/10/17 at UTC 12:00. > > Thanks, > Deb > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From deb at redhat.com Fri Oct 14 12:31:00 2016 From: deb at redhat.com (Soumya Deb) Date: Fri, 14 Oct 2016 18:01:00 +0530 Subject: [Tendrl-devel] GitHub access rights Message-ID: Do we have a plan around who should be having which tier of access rights on Tendrl? So far it seems very informal & inter-personal (which is okay to start with), but going forward, we probably should have a properly defined plan about requesting access, acceptance criteria, accepting body etc. at least at a basic level. Tendrl being an opensource project, I don't see why its core parts (codebase management being one) can't be more federated. --- The reason to have to write this email, is because I'm trying to get some service integration done for the Tendrl frontend - which requires admin/owner access to the repo. I'm not sure who all hold this access right - but at least Sankarshan & Rohan does, that I'm aware of. I had asked Rohan if he'd be willing to help me out, and he agreed without hesitation - much appreciated. Now, I am part for 10+ GitHub orgs (admin of 5+) and have done plenty of GitHub housekeeping in those ones that I'm pretty confident about being able to manage those chores on my own. So, if I were to request access, estimating that I am capable, how do I proceed? Please note, by no way I'm saying "I should be" having access; rather, if I, for one, was interested to share hands (still subject to actually passing the acceptance criteria) - then how do I proceed? I'm sure just emailing Jeff/Sankarshan about this would've been easy, but hey... who wants easy? :D From deb at redhat.com Fri Oct 14 12:37:26 2016 From: deb at redhat.com (Soumya Deb) Date: Fri, 14 Oct 2016 18:07:26 +0530 Subject: [Tendrl-devel] Moving code from Skyrings/kitoon to Tendrl/tendrl_frontend In-Reply-To: References: Message-ID: On 14 October 2016 at 17:57, Kanagaraj M wrote: > > Since tendrl_frontend repo is already present, you might need to drop it > and fork from kitoon. I am not sure if github provides a way to fork and > merge. > It's possible to push kitoon's code with detached head for tendrl_frontend as the master, and rename the existing master branch to something else (or drop it), at which point it'll contain the same history as of the previous repo. As I mentioned the technical challenge isn't that much, and can be easily overcome. Thanks for pointing this out though, I avoided detailing them in the original email. From julim at redhat.com Fri Oct 14 12:38:45 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 14 Oct 2016 08:38:45 -0400 Subject: [Tendrl-devel] GitHub access rights In-Reply-To: References: Message-ID: Soumya: +1 to needing a plan. As I don't believe such a plan exists yet, would you be willing to draft a proposal and share it? Thanks, Ju On Fri, Oct 14, 2016 at 8:31 AM, Soumya Deb wrote: > Do we have a plan around who should be having which tier of access rights > on Tendrl? > > So far it seems very informal & inter-personal (which is okay to start > with), but going forward, we probably should have a properly defined plan > about requesting access, acceptance criteria, accepting body etc. at least > at a basic level. > > Tendrl being an opensource project, I don't see why its core parts > (codebase management being one) can't be more federated. > > --- > > The reason to have to write this email, is because I'm trying to get some > service integration done for the Tendrl frontend - which requires > admin/owner access to the repo. I'm not sure who all hold this access right > - but at least Sankarshan & Rohan does, that I'm aware of. I had asked > Rohan if he'd be willing to help me out, and he agreed without hesitation - > much appreciated. > > Now, I am part for 10+ GitHub orgs (admin of 5+) and have done plenty of > GitHub housekeeping in those ones that I'm pretty confident about being > able to manage those chores on my own. So, if I were to request access, > estimating that I am capable, how do I proceed? Please note, by no way I'm > saying "I should be" having access; rather, if I, for one, was interested > to share hands (still subject to actually passing the acceptance criteria) > - then how do I proceed? > > I'm sure just emailing Jeff/Sankarshan about this would've been easy, but > hey... who wants easy? :D > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From deb at redhat.com Fri Oct 14 12:53:39 2016 From: deb at redhat.com (Soumya Deb) Date: Fri, 14 Oct 2016 18:23:39 +0530 Subject: [Tendrl-devel] GitHub access rights In-Reply-To: References: Message-ID: Gmail doesn't support enough emoji, but I'd be glad to draft one. Needed something interesting to do for the weekend anyway. :D On 14 October 2016 at 18:08, Ju Lim wrote: > Soumya: > > +1 to needing a plan. As I don't believe such a plan exists yet, would > you be willing to draft a proposal and share it? > > Thanks, > Ju > > On Fri, Oct 14, 2016 at 8:31 AM, Soumya Deb wrote: > >> Do we have a plan around who should be having which tier of access rights >> on Tendrl? >> >> So far it seems very informal & inter-personal (which is okay to start >> with), but going forward, we probably should have a properly defined plan >> about requesting access, acceptance criteria, accepting body etc. at least >> at a basic level. >> >> Tendrl being an opensource project, I don't see why its core parts >> (codebase management being one) can't be more federated. >> >> --- >> >> The reason to have to write this email, is because I'm trying to get some >> service integration done for the Tendrl frontend - which requires >> admin/owner access to the repo. I'm not sure who all hold this access >> right >> - but at least Sankarshan & Rohan does, that I'm aware of. I had asked >> Rohan if he'd be willing to help me out, and he agreed without hesitation >> - >> much appreciated. >> >> Now, I am part for 10+ GitHub orgs (admin of 5+) and have done plenty of >> GitHub housekeeping in those ones that I'm pretty confident about being >> able to manage those chores on my own. So, if I were to request access, >> estimating that I am capable, how do I proceed? Please note, by no way I'm >> saying "I should be" having access; rather, if I, for one, was interested >> to share hands (still subject to actually passing the acceptance criteria) >> - then how do I proceed? >> >> I'm sure just emailing Jeff/Sankarshan about this would've been easy, but >> hey... who wants easy? :D >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > -- > Ju Lim > Red Hat > Office: 978-399-0422 > Mobile: 781-507-1323 > Email: julim at redhat.com > IRC: julim > > From julim at redhat.com Fri Oct 14 12:56:20 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 14 Oct 2016 08:56:20 -0400 Subject: [Tendrl-devel] GitHub access rights In-Reply-To: References: Message-ID: Thanks Soumya. I appreciate you bringing up this point as I too have struggled with how we've set this up. I look forward to seeing your proposal! Regards, Ju On Fri, Oct 14, 2016 at 8:53 AM, Soumya Deb wrote: > Gmail doesn't support enough emoji, but I'd be glad to draft one. > Needed something interesting to do for the weekend anyway. :D > > > On 14 October 2016 at 18:08, Ju Lim wrote: > > > Soumya: > > > > +1 to needing a plan. As I don't believe such a plan exists yet, would > > you be willing to draft a proposal and share it? > > > > Thanks, > > Ju > > > > On Fri, Oct 14, 2016 at 8:31 AM, Soumya Deb wrote: > > > >> Do we have a plan around who should be having which tier of access > rights > >> on Tendrl? > >> > >> So far it seems very informal & inter-personal (which is okay to start > >> with), but going forward, we probably should have a properly defined > plan > >> about requesting access, acceptance criteria, accepting body etc. at > least > >> at a basic level. > >> > >> Tendrl being an opensource project, I don't see why its core parts > >> (codebase management being one) can't be more federated. > >> > >> --- > >> > >> The reason to have to write this email, is because I'm trying to get > some > >> service integration done for the Tendrl frontend - which requires > >> admin/owner access to the repo. I'm not sure who all hold this access > >> right > >> - but at least Sankarshan & Rohan does, that I'm aware of. I had asked > >> Rohan if he'd be willing to help me out, and he agreed without > hesitation > >> - > >> much appreciated. > >> > >> Now, I am part for 10+ GitHub orgs (admin of 5+) and have done plenty of > >> GitHub housekeeping in those ones that I'm pretty confident about being > >> able to manage those chores on my own. So, if I were to request access, > >> estimating that I am capable, how do I proceed? Please note, by no way > I'm > >> saying "I should be" having access; rather, if I, for one, was > interested > >> to share hands (still subject to actually passing the acceptance > criteria) > >> - then how do I proceed? > >> > >> I'm sure just emailing Jeff/Sankarshan about this would've been easy, > but > >> hey... who wants easy? :D > >> _______________________________________________ > >> Tendrl-devel mailing list > >> Tendrl-devel at redhat.com > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > >> > > > > > > > > -- > > Ju Lim > > Red Hat > > Office: 978-399-0422 > > Mobile: 781-507-1323 > > Email: julim at redhat.com > > IRC: julim > > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From japplewh at redhat.com Fri Oct 14 13:58:11 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Fri, 14 Oct 2016 09:58:11 -0400 Subject: [Tendrl-devel] GitHub access rights In-Reply-To: References: Message-ID: Yes agreed - we should have a plan for how we will accommodate contributors before things get unwieldy. It is fairly monolithic at this point but perhaps we can break down the ownership a bit. You sound like the perfect person to suggest this plan Soumya. Look forward to hearing it! On Fri, Oct 14, 2016 at 8:56 AM, Ju Lim wrote: > Thanks Soumya. I appreciate you bringing up this point as I too have > struggled with how we've set this up. > > I look forward to seeing your proposal! > > Regards, > Ju > > On Fri, Oct 14, 2016 at 8:53 AM, Soumya Deb wrote: > > > Gmail doesn't support enough emoji, but I'd be glad to draft one. > > Needed something interesting to do for the weekend anyway. :D > > > > > > On 14 October 2016 at 18:08, Ju Lim wrote: > > > > > Soumya: > > > > > > +1 to needing a plan. As I don't believe such a plan exists yet, would > > > you be willing to draft a proposal and share it? > > > > > > Thanks, > > > Ju > > > > > > On Fri, Oct 14, 2016 at 8:31 AM, Soumya Deb wrote: > > > > > >> Do we have a plan around who should be having which tier of access > > rights > > >> on Tendrl? > > >> > > >> So far it seems very informal & inter-personal (which is okay to start > > >> with), but going forward, we probably should have a properly defined > > plan > > >> about requesting access, acceptance criteria, accepting body etc. at > > least > > >> at a basic level. > > >> > > >> Tendrl being an opensource project, I don't see why its core parts > > >> (codebase management being one) can't be more federated. > > >> > > >> --- > > >> > > >> The reason to have to write this email, is because I'm trying to get > > some > > >> service integration done for the Tendrl frontend - which requires > > >> admin/owner access to the repo. I'm not sure who all hold this access > > >> right > > >> - but at least Sankarshan & Rohan does, that I'm aware of. I had asked > > >> Rohan if he'd be willing to help me out, and he agreed without > > hesitation > > >> - > > >> much appreciated. > > >> > > >> Now, I am part for 10+ GitHub orgs (admin of 5+) and have done plenty > of > > >> GitHub housekeeping in those ones that I'm pretty confident about > being > > >> able to manage those chores on my own. So, if I were to request > access, > > >> estimating that I am capable, how do I proceed? Please note, by no way > > I'm > > >> saying "I should be" having access; rather, if I, for one, was > > interested > > >> to share hands (still subject to actually passing the acceptance > > criteria) > > >> - then how do I proceed? > > >> > > >> I'm sure just emailing Jeff/Sankarshan about this would've been easy, > > but > > >> hey... who wants easy? :D > > >> _______________________________________________ > > >> Tendrl-devel mailing list > > >> Tendrl-devel at redhat.com > > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > >> > > > > > > > > > > > > -- > > > Ju Lim > > > Red Hat > > > Office: 978-399-0422 > > > Mobile: 781-507-1323 > > > Email: julim at redhat.com > > > IRC: julim > > > > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > -- > Ju Lim > Red Hat > Office: 978-399-0422 > Mobile: 781-507-1323 > Email: julim at redhat.com > IRC: julim > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From julim at redhat.com Fri Oct 14 14:47:53 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 14 Oct 2016 10:47:53 -0400 Subject: [Tendrl-devel] Upcoming Tendrl Sprint 2 Demo on Tuesday, Oct 18 2016 Message-ID: Team: Just a heads-up and reminder that the upcoming sprint 1 demo is next Tuesday, 18 Oct 2016. The proposed Tendrl Sprint 2 Demo Agenda may be found at: Sprint 002 Demo Agenda & Notes What's listed in the link above is the tentative agenda. Items that have a strike-through will not be demonstrated until they have satisfied the Definition of Done (DoD) . For tasks (e.g. items with SPIKE, UX or have a suffix of task) that will be demoed, they must satisfy the acceptance criteria. If there are any demos that I've missed in the above link, please add to / edit the wiki page. For folks who are planning to demo something, please refer to the Sprint Demo Guidelines to help ensure a smooth and successful demo. As soon as the Sprint demo is over, we'll have a retrospective. Thank you, Ju From japplewh at redhat.com Fri Oct 14 18:24:01 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Fri, 14 Oct 2016 14:24:01 -0400 Subject: [Tendrl-devel] Progress Update Message-ID: Hi All I thought I would highlight some of the good things that are going on with Tendrl and thank you all for your contributions. I really see that we are set up for great success going forward. Consider this.. We have: - Launched this Tendrl upstream project! - Obtained and launched new tooling (JIRA) with a free unlimited open source project based license to manage our development work - Implemented new processes that will help drive quality and limit risk to integration points with other upstream projects - agreed on a really solid definition of done - started work on a full continuous integration (CI) process - implemented unit tests - implemented code reviews - started work on full RPM builds for integrated testing - Are in the process of finalizing a new and exciting architecture that gives us: - High availability with excellent performance. Why is this important? Because in the new era of private cloud, provisioning is now a critical piece of cloud infrastructure. It can't go down without impacting the wider enterprise. - Flexibility to evolve with new component technologies as they arise - High level APIs to expose to other tools, technologies, or internal processes - A level of abstraction that allows for widely divergent backend storage technologies - Continuing forward many of the successful strategies and pieces of work derived from our predecessor Skyrings project. We learned a lot in this and are channeling this excellent work into Tendrl. - Done some excellent planning for the UX and also decided on key technologies - Developed a framework for ongoing automated test development I'm please to see all the great work that looks like it's coming in the demo next week - thanks Ju for the summary. We are definitely hitting our stride now. We still have a little more to do on the Tendrl side of things - the web site is nearing completion and we will be showing a demo soon for feedback. The code is on Tendrl github now if you would like to contribute. And now, looking forward to our goal of a Tendrl 1 release in a couple more sprints here are the high level goals we set out: - Create and Document all APIs - Import Ceph and Gluster clusters - Create frameworks for Logging, Jobs/Flows, Events, Alerts, Metrics/Time series data - CI Test infrastructure / PEP 8 compliance / Automation /etc. - Demo Create via API: (Ceph) Pools, Volumes, RBDs (Gluster) Bricks, Volumes - List views - Cluster (Ceph/Gluster), Nodes, Pools, File Shares, RBDs - Finalize architectural decisions ?Let's all find our best way to contribute to get this over the line. Thank you all for your hard work thus far!!!? Jeff From julim at redhat.com Fri Oct 14 19:17:52 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 14 Oct 2016 15:17:52 -0400 Subject: [Tendrl-devel] Progress Update In-Reply-To: References: Message-ID: +1 Thanks Jeff for highlighting some of the progress the team has been making. It's good to see this update so everyone is aware of all the work going on and our progress. We've still a ways to go, but with everyones putting their best foot forward, I'm sure we'll get there. Thanks again, Ju On Fri, Oct 14, 2016 at 2:24 PM, Jeff Applewhite wrote: > Hi All > > I thought I would highlight some of the good things that are going on with > Tendrl and thank you all for your contributions. I really see that we are > set up for great success going forward. > > Consider this.. > > We have: > > - Launched this Tendrl upstream project! > - Obtained and launched new tooling (JIRA) with a free unlimited open > source project based license to manage our development work > - Implemented new processes that will help drive quality and limit risk > to integration points with other upstream projects > - agreed on a really solid definition of done > - started work on a full continuous integration (CI) process > - implemented unit tests > - implemented code reviews > - started work on full RPM builds for integrated testing > - Are in the process of finalizing a new and exciting architecture that > gives us: > - High availability with excellent performance. Why is this > important? Because in the new era of private cloud, provisioning is > now a > critical piece of cloud infrastructure. It can't go down without > impacting > the wider enterprise. > - Flexibility to evolve with new component technologies as they arise > - High level APIs to expose to other tools, technologies, or internal > processes > - A level of abstraction that allows for widely divergent backend > storage technologies > - Continuing forward many of the successful strategies and pieces of > work derived from our predecessor Skyrings project. We learned a lot in > this and are channeling this excellent work into Tendrl. > - Done some excellent planning for the UX and also decided on key > technologies > - Developed a framework for ongoing automated test development > > I'm please to see all the great work that looks like it's coming in the > demo next week - thanks Ju for the summary. We are definitely hitting our > stride now. > > We still have a little more to do on the Tendrl side of things - the web > site is nearing completion and we will be showing a demo soon for feedback. > The code is on Tendrl github now if you would like to contribute. > > And now, looking forward to our goal of a Tendrl 1 release in a couple more > sprints here are the high level goals we set out: > > > - Create and Document all APIs > - Import Ceph and Gluster clusters > - Create frameworks for Logging, Jobs/Flows, Events, Alerts, > Metrics/Time series data > - CI Test infrastructure / PEP 8 compliance / Automation /etc. > - Demo Create via API: (Ceph) Pools, Volumes, RBDs (Gluster) Bricks, > Volumes > - List views - Cluster (Ceph/Gluster), Nodes, Pools, File Shares, RBDs > - Finalize architectural decisions > > ?Let's all find our best way to contribute to get this over the line. > > Thank you all for your hard work thus far!!!? > > > Jeff > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From kdreyer at redhat.com Fri Oct 14 19:20:11 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Fri, 14 Oct 2016 13:20:11 -0600 Subject: [Tendrl-devel] CI Plans In-Reply-To: <20161014121509.GA32384@gmail.com> References: <20161014121509.GA32384@gmail.com> Message-ID: Hi Nigel, Will you be hosting the Tendrl dist-git repositories at https://github.com/CentOS-Storage-SIG ? Or somewhere else? I'm in the process of building out the CI system for packaging https://github.com/ceph/ceph-installer and I'd like to eventually get it going in CentOS's CI system. - Ken On Fri, Oct 14, 2016 at 6:15 AM, Nigel Babu wrote: > Hello, > > Nishant, Mrugesh, and I have had a chat with Brian Stinson from the Centos team > about consuming CentOS CI for Tendrl. I've been working on a good pattern for > Gluster[1] and helping adopt this pattern for Tendrl as well. > > As a first step would, we will setup nightly rpm builds for Tendrl. In the > future, we can consume these rpms for running the test framework. > > As an example of how this will work, here's where Gluster is headed in the next > few weeks. QE is upstreaming Glusto, a generic framework to run tests written > in python test frameworks (PyTest, nose, UnitTest) over a network. We will > request 5 nodes from Centos CI, the first one will be the management node and > the next 4 will be part of the cluster. We configure the nodes, install the > nightly RPMs on there, and run our tests against this build. Once this passes, > it gives us nightly builds, which are verified to be working. > > Once the test framework for Tendrl is ready and setup, we can proceed in doing > more than packaging. Let me know if anyone has questions. > > [1]: https://github.com/nigelbabu/centos-ci-sample > > -- > nigelb > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From nigelb at redhat.com Sat Oct 15 13:05:48 2016 From: nigelb at redhat.com (Nigel Babu) Date: Sat, 15 Oct 2016 18:35:48 +0530 Subject: [Tendrl-devel] CI Plans In-Reply-To: References: <20161014121509.GA32384@gmail.com> Message-ID: <20161015130548.GA23962@gmail.com> Hi Ken, Our access to Centos CI is through the Storage SIG, so we will be contributing CentOS packages via the Centos SIG once Tendrl has releases. (I'm guessing one of the developers will be taking on the role. I'm only advicing on building out the CI architecture.) On Fri, Oct 14, 2016 at 01:20:11PM -0600, Ken Dreyer wrote: > Hi Nigel, > > Will you be hosting the Tendrl dist-git repositories at > https://github.com/CentOS-Storage-SIG ? Or somewhere else? > > I'm in the process of building out the CI system for packaging > https://github.com/ceph/ceph-installer and I'd like to eventually get > it going in CentOS's CI system. > > - Ken -- nigelb From mkudlej at redhat.com Mon Oct 17 12:19:22 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Mon, 17 Oct 2016 14:19:22 +0200 Subject: [Tendrl-devel] Sprint demo and DoD Message-ID: <355534a5-5259-15d5-ee74-2515a738f5fe@redhat.com> Hi all, I see that there are couple of tasks which are still "In Progress" state and they are proposed for demo. I think that tasks proposed for demo should be in "Accepted" column or at least in "Complete" one if we know that DoD will be completed till demo. Do I understand wrong our workflow and DoD? Thank you for your answer! -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From japplewh at redhat.com Mon Oct 17 12:26:53 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Mon, 17 Oct 2016 08:26:53 -0400 Subject: [Tendrl-devel] Sprint demo and DoD In-Reply-To: <355534a5-5259-15d5-ee74-2515a738f5fe@redhat.com> References: <355534a5-5259-15d5-ee74-2515a738f5fe@redhat.com> Message-ID: On Mon, Oct 17, 2016 at 8:19 AM, Martin Kudlej wrote: > Hi all, > > I see that there are couple of tasks which are still "In Progress" state > and they are proposed for demo. I think that tasks proposed for demo should > be in "Accepted" column or at least in "Complete" one if we know that DoD > will be completed till demo. > ?Accepted should happen after the demo except for items that are not demo-able (research spikes primarily, tasks outside development/QE, etc.).? ?The correct state prior to demo would be Complete.? ? > Do I understand wrong our workflow and DoD? > > Thank you for your answer! > > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, > #usm-meeting @ redhat > #tendrl-devel @ freenode > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From mkudlej at redhat.com Mon Oct 17 13:32:51 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Mon, 17 Oct 2016 15:32:51 +0200 Subject: [Tendrl-devel] API availability Message-ID: <659291cb-32c1-a166-836d-ece0a15358b4@redhat.com> Hi Anup, I understand that you work on API in Tendrl(If you don't work on API, could you please point me to proper person, please?) There are couple tasks in Jira which implement some API functions, for example https://tendrl.atlassian.net/browse/TEN-43 https://tendrl.atlassian.net/browse/TEN-95 How can we deploy API so we can test tasks which implement API functions? Thank you for answer! -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From anivargi at redhat.com Mon Oct 17 13:36:16 2016 From: anivargi at redhat.com (Anup Nivargi) Date: Mon, 17 Oct 2016 19:06:16 +0530 Subject: [Tendrl-devel] API availability In-Reply-To: <659291cb-32c1-a166-836d-ece0a15358b4@redhat.com> References: <659291cb-32c1-a166-836d-ece0a15358b4@redhat.com> Message-ID: <84953872-A89C-404C-A06A-56E660D93389@redhat.com> Hi, Yeah, I am working on the API, and I will have the deployment documentation in place very soon. > On 17-Oct-2016, at 19:02, Martin Kudlej wrote: > > Hi Anup, > > I understand that you work on API in Tendrl(If you don't work on API, could you please point me to proper person, please?) > > There are couple tasks in Jira which implement some API functions, for example > https://tendrl.atlassian.net/browse/TEN-43 > https://tendrl.atlassian.net/browse/TEN-95 > > How can we deploy API so we can test tasks which implement API functions? > > Thank you for answer! > > > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat > #tendrl-devel @ freenode -- Anup Nivargi From deb at redhat.com Tue Oct 18 07:51:44 2016 From: deb at redhat.com (Soumya Deb) Date: Tue, 18 Oct 2016 13:21:44 +0530 Subject: [Tendrl-devel] Sprint demo and DoD In-Reply-To: References: <355534a5-5259-15d5-ee74-2515a738f5fe@redhat.com> Message-ID: I'm sorry, but there's no state currently as Complete. Do you mean "Done" (which again confuses me about the DoD then)? On 17 October 2016 at 17:56, Jeff Applewhite wrote: > On Mon, Oct 17, 2016 at 8:19 AM, Martin Kudlej wrote: > > > Hi all, > > > > I see that there are couple of tasks which are still "In Progress" state > > and they are proposed for demo. I think that tasks proposed for demo > should > > be in "Accepted" column or at least in "Complete" one if we know that DoD > > will be completed till demo. > > > > ?Accepted should happen after the demo except for items that are not > demo-able (research spikes primarily, tasks outside development/QE, etc.).? > > ?The correct state prior to demo would be Complete.? > ? > > > > Do I understand wrong our workflow and DoD? > > > > Thank you for your answer! > > > > -- > > Best Regards, > > Martin Kudlej. > > RHSC/USM Senior Quality Assurance Engineer > > Red Hat Czech s.r.o. > > > > Phone: +420 532 294 155 > > E-mail:mkudlej at redhat.com > > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, > > #usm-meeting @ redhat > > #tendrl-devel @ freenode > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > -- > Jeff Applewhite > Principal Product Manager > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From mkudlej at redhat.com Tue Oct 18 08:19:18 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Tue, 18 Oct 2016 10:19:18 +0200 Subject: [Tendrl-devel] Sprint demo and DoD In-Reply-To: References: <355534a5-5259-15d5-ee74-2515a738f5fe@redhat.com> Message-ID: <40a40a40-3dfd-629c-d374-4007cf593017@redhat.com> Hi Soumya, On 10/18/2016 09:51 AM, Soumya Deb wrote: > I'm sorry, but there's no state currently as Complete. > Do you mean "Done" (which again confuses me about the DoD then)? You are partially right. "Done" in task UI means "Complete" in Kanban board. DoD doesn't refer to "Done"/"Complete" state of task. It refers to "Accepted"(task UI and also Kanban board) state of task. It will be great if admin(Jeff?) will clean and unify these states to avoid confuses. -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From rkanade at redhat.com Tue Oct 18 12:11:58 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Tue, 18 Oct 2016 17:41:58 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) Message-ID: Tendrl is going through its first release cycle and we have covered some ground w.r.t Basic plumbing, inter component messaging, central store (etcd), tendrl API, Ceph/Gluster ops etc. We still need to cover significant ground in these areas before we can be ready for our first release I'd like to propose that the Tendrl contributors get together with a flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and collaborate on various pending/upcoming items. Dates: 8, 9, 10 Nov 2016 Location: Red Hat, Bangalore, India Please share your ideas for this proposal so we can get a rough agenda in place soon. Regards, Rohan Kanade From mrugesh at brainfunked.org Tue Oct 18 12:28:36 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 18 Oct 2016 17:58:36 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: I whole-heartedly second this! On 18 October 2016 at 17:41, Rohan Kanade wrote: > > Tendrl is going through its first release cycle and we have covered some > ground w.r.t Basic plumbing, inter component messaging, central store > (etcd), tendrl API, Ceph/Gluster ops etc. > > We still need to cover significant ground in these areas before we can be > ready for our first release > > I'd like to propose that the Tendrl contributors get together with a > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and > collaborate on various pending/upcoming items. > > Dates: 8, 9, 10 Nov 2016 > Location: Red Hat, Bangalore, India > > Please share your ideas for this proposal so we can get a rough agenda in > place soon. > > Regards, > Rohan Kanade > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From japplewh at redhat.com Tue Oct 18 14:30:15 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 18 Oct 2016 10:30:15 -0400 Subject: [Tendrl-devel] TripleO blueprint Message-ID: Hi All An OpenStack blueprint was added today by John Fulton "? Integrate TripleO with Tendrl for External Storage Deployment/Management ?"? Please take a look: https://review.openstack.org/#/c/387631 ?I'll be adding this to the Jira backlog items from the Tendrl perspective. The work will revolved around developing and integrating an API service for flows that will include create job, initial install, updates, upgrades, node replacement, node scaling, cluster import, and cluster export for Ceph. Thanks!? Jeff From japplewh at redhat.com Tue Oct 18 16:15:52 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 18 Oct 2016 12:15:52 -0400 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: I think it's a great idea! +1 But given that we are releasing the first version of Tendrl on Nov 8 it seems like maybe we should target the 2nd-4th of November. That would give us more time for QE and acceptance of user stories Thoughts? On Tue, Oct 18, 2016 at 8:28 AM, Mrugesh Karnik wrote: > I whole-heartedly second this! > > On 18 October 2016 at 17:41, Rohan Kanade wrote: > > > > Tendrl is going through its first release cycle and we have covered some > > ground w.r.t Basic plumbing, inter component messaging, central store > > (etcd), tendrl API, Ceph/Gluster ops etc. > > > > We still need to cover significant ground in these areas before we can be > > ready for our first release > > > > I'd like to propose that the Tendrl contributors get together with a > > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and > > collaborate on various pending/upcoming items. > > > > Dates: 8, 9, 10 Nov 2016 > > Location: Red Hat, Bangalore, India > > > > Please share your ideas for this proposal so we can get a rough agenda in > > place soon. > > > > Regards, > > Rohan Kanade > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From amukherj at redhat.com Tue Oct 18 16:26:25 2016 From: amukherj at redhat.com (Atin Mukherjee) Date: Tue, 18 Oct 2016 21:56:25 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: This is fantastic! From gluster point of view, I'd like to see a basic end to end workflow (probably volume create to start with) between Tendrl gluster bridge and Gluster get established and how an event can be consumed and processed by the tendrl layer. This would immensly benefit different stakeholders to understand the integration of tendrl and gluster. On Tuesday 18 October 2016, Rohan Kanade wrote: > Tendrl is going through its first release cycle and we have covered some > ground w.r.t Basic plumbing, inter component messaging, central store > (etcd), tendrl API, Ceph/Gluster ops etc. > > We still need to cover significant ground in these areas before we can be > ready for our first release > > I'd like to propose that the Tendrl contributors get together with a > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and > collaborate on various pending/upcoming items. > > Dates: 8, 9, 10 Nov 2016 > Location: Red Hat, Bangalore, India > > Please share your ideas for this proposal so we can get a rough agenda in > place soon. > > Regards, > Rohan Kanade > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- --Atin From julim at redhat.com Tue Oct 18 16:32:15 2016 From: julim at redhat.com (Ju Lim) Date: Tue, 18 Oct 2016 12:32:15 -0400 Subject: [Tendrl-devel] Tendrl Sprint 2 Demo Details (Recording & Notes) Message-ID: Team: The demo recording from the Sprint 2 demo is now available along with the final agenda and notes (feedback and Q&A). They can be found at Sprint 002 Demo Agenda & Notes . If folks have any comments and/or feedback on the demo(s), please reply all to this email thread. As always, all Sprint goals, demo agenda, demo recording, and notes may be found at the Tendrl Sprints Landing Page , which is an index of all sprints to-date. Thank you, Ju From fulton at redhat.com Tue Oct 18 16:35:13 2016 From: fulton at redhat.com (John Fulton) Date: Tue, 18 Oct 2016 12:35:13 -0400 (EDT) Subject: [Tendrl-devel] TripleO blueprint In-Reply-To: References: Message-ID: <1769912770.2196280.1476808513412.JavaMail.zimbra@redhat.com> Thanks Jeff, Also, I started a thread on it on the OpenStack dev list in case anyone is interested: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105951.html John ----- Original Message ----- From: "Jeff Applewhite" To: "Mailing list for the contributors to the Tendrl project" Cc: "John Fulton" Sent: Tuesday, October 18, 2016 10:30:15 AM Subject: TripleO blueprint Hi All An OpenStack blueprint was added today by John Fulton "? Integrate TripleO with Tendrl for External Storage Deployment/Management ?"? Please take a look: https://review.openstack.org/#/c/387631 ?I'll be adding this to the Jira backlog items from the Tendrl perspective. The work will revolved around developing and integrating an API service for flows that will include create job, initial install, updates, upgrades, node replacement, node scaling, cluster import, and cluster export for Ceph. Thanks!? Jeff From jspray at redhat.com Tue Oct 18 17:15:10 2016 From: jspray at redhat.com (John Spray) Date: Tue, 18 Oct 2016 18:15:10 +0100 Subject: [Tendrl-devel] Tendrl Sprint 2 Demo Details (Recording & Notes) In-Reply-To: References: Message-ID: On Tue, Oct 18, 2016 at 5:32 PM, Ju Lim wrote: > Team: > > The demo recording from the Sprint 2 demo is now available along with the > final agenda and notes (feedback and Q&A). They can be found at Sprint 002 > Demo Agenda & Notes > . > > If folks have any comments and/or feedback on the demo(s), please reply all > to this email thread. > > As always, all Sprint goals, demo agenda, demo recording, and notes may be > found at the Tendrl Sprints Landing Page > , which is an index > of all sprints to-date. I'm liking the new top-level navigation with the ceph/gluster-specific bits pushed up a level -- big +1 on that. Martin's comment about use of "File Share" struck a chord with me -- I think it's important to distinguish between file storage itself, and the act of sharing it over the network using a particular protocol (NFS/CIFS). If the storage itself is referred to as a "file share", then it will be important to avoid any use of the word "share" when describing NFS and Samba exports (which I personally tend to think of as shares). This has historically been a difficult area -- e.g. Manila uses "share" to describe any allocation of storage, as it merges the storage concept with the sharing concept (i.e. a piece of storage is NFS or Samba, but not both). Googling around a bit for what the proprietary systems do, NetApp appear to use "volume" to describe file storage, and then "NFS exports" to describe sharing it[1]. John 1. https://kb.netapp.com/support/s/article/what-are-the-rules-for-exporting-volumes-and-directories-in-network-file-system-nfs?language=en_US > Thank you, > Ju > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From nthomas at redhat.com Wed Oct 19 04:24:45 2016 From: nthomas at redhat.com (Nishanth Thomas) Date: Wed, 19 Oct 2016 09:54:45 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: +1, This will be ideal On Tue, Oct 18, 2016 at 9:56 PM, Atin Mukherjee wrote: > This is fantastic! From gluster point of view, I'd like to see a basic end > to end workflow (probably volume create to start with) between Tendrl > gluster bridge and Gluster get established and how an event can be consumed > and processed by the tendrl layer. This would immensly benefit different > stakeholders to understand the integration of tendrl and gluster. > > On Tuesday 18 October 2016, Rohan Kanade wrote: > > > Tendrl is going through its first release cycle and we have covered some > > ground w.r.t Basic plumbing, inter component messaging, central store > > (etcd), tendrl API, Ceph/Gluster ops etc. > > > > We still need to cover significant ground in these areas before we can be > > ready for our first release > > > > I'd like to propose that the Tendrl contributors get together with a > > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and > > collaborate on various pending/upcoming items. > > > > Dates: 8, 9, 10 Nov 2016 > > Location: Red Hat, Bangalore, India > > > > Please share your ideas for this proposal so we can get a rough agenda in > > place soon. > > > > Regards, > > Rohan Kanade > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > -- > --Atin > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From shtripat at redhat.com Wed Oct 19 04:28:49 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Wed, 19 Oct 2016 09:58:49 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: +1 This would be really helpful for team. Regards, Shubhendu On 10/18/2016 05:41 PM, Rohan Kanade wrote: > Tendrl is going through its first release cycle and we have covered some > ground w.r.t Basic plumbing, inter component messaging, central store > (etcd), tendrl API, Ceph/Gluster ops etc. > > We still need to cover significant ground in these areas before we can be > ready for our first release > > I'd like to propose that the Tendrl contributors get together with a > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and > collaborate on various pending/upcoming items. > > Dates: 8, 9, 10 Nov 2016 > Location: Red Hat, Bangalore, India > > Please share your ideas for this proposal so we can get a rough agenda in > place soon. > > Regards, > Rohan Kanade > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From mkudlej at redhat.com Wed Oct 19 09:03:46 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Wed, 19 Oct 2016 11:03:46 +0200 Subject: [Tendrl-devel] CI Plans In-Reply-To: <20161014121509.GA32384@gmail.com> References: <20161014121509.GA32384@gmail.com> Message-ID: <7f41d167-d6e9-e960-bf5e-aba73ef703ef@redhat.com> Hi Nigel, I've filed ticket for creating account in CentoOS CI: https://bugs.centos.org/view.php?id=12097 On 10/14/2016 02:15 PM, Nigel Babu wrote: > As a first step would, we will setup nightly rpm builds for Tendrl. In the > future, we can consume these rpms for running the test framework. I agree and we(QE) would like to help with this work. We already done similar work with almost same tools like CentOS CI for different project. > As an example of how this will work, here's where Gluster is headed in the next > few weeks. QE is upstreaming Glusto, a generic framework to run tests written > in python test frameworks (PyTest, nose, UnitTest) over a network. We will > request 5 nodes from Centos CI, the first one will be the management node and > the next 4 will be part of the cluster. We configure the nodes, install the > nightly RPMs on there, and run our tests against this build. Once this passes, > it gives us nightly builds, which are verified to be working. I think it will be great to schedule meeting and discuss this in more details. Thank you for helping us to build upstream CI! -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From rkanade at redhat.com Wed Oct 19 11:22:08 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Wed, 19 Oct 2016 16:52:08 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: Jeff, I wont be available during 30 Oct to 4th Nov (Important Indian Festival time). If other team members are available from 2nd-4th Nov, we can chalk up the agenda and continue on these dates. On Tue, Oct 18, 2016 at 9:45 PM, Jeff Applewhite wrote: > I think it's a great idea! +1 > > But given that we are releasing the first version of Tendrl on Nov 8 it > seems like maybe we should target the 2nd-4th of November. > That would give us more time for QE and acceptance of user stories > > Thoughts? > > On Tue, Oct 18, 2016 at 8:28 AM, Mrugesh Karnik > wrote: > > > I whole-heartedly second this! > > > > On 18 October 2016 at 17:41, Rohan Kanade wrote: > > > > > > Tendrl is going through its first release cycle and we have covered > some > > > ground w.r.t Basic plumbing, inter component messaging, central store > > > (etcd), tendrl API, Ceph/Gluster ops etc. > > > > > > We still need to cover significant ground in these areas before we can > be > > > ready for our first release > > > > > > I'd like to propose that the Tendrl contributors get together with a > > > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and > > > collaborate on various pending/upcoming items. > > > > > > Dates: 8, 9, 10 Nov 2016 > > > Location: Red Hat, Bangalore, India > > > > > > Please share your ideas for this proposal so we can get a rough agenda > in > > > place soon. > > > > > > Regards, > > > Rohan Kanade > > > _______________________________________________ > > > Tendrl-devel mailing list > > > Tendrl-devel at redhat.com > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > -- > Jeff Applewhite > Principal Product Manager > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From nthomas at redhat.com Wed Oct 19 11:27:48 2016 From: nthomas at redhat.com (Nishanth Thomas) Date: Wed, 19 Oct 2016 16:57:48 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: I think its important to have Rohan here for hack days On Wed, Oct 19, 2016 at 4:52 PM, Rohan Kanade wrote: > Jeff, > > I wont be available during 30 Oct to 4th Nov (Important Indian Festival > time). If other team members are available from 2nd-4th Nov, we can chalk > up the agenda and continue on these dates. > > On Tue, Oct 18, 2016 at 9:45 PM, Jeff Applewhite > wrote: > > > I think it's a great idea! +1 > > > > But given that we are releasing the first version of Tendrl on Nov 8 it > > seems like maybe we should target the 2nd-4th of November. > > That would give us more time for QE and acceptance of user stories > > > > Thoughts? > > > > On Tue, Oct 18, 2016 at 8:28 AM, Mrugesh Karnik > > > wrote: > > > > > I whole-heartedly second this! > > > > > > On 18 October 2016 at 17:41, Rohan Kanade wrote: > > > > > > > > Tendrl is going through its first release cycle and we have covered > > some > > > > ground w.r.t Basic plumbing, inter component messaging, central store > > > > (etcd), tendrl API, Ceph/Gluster ops etc. > > > > > > > > We still need to cover significant ground in these areas before we > can > > be > > > > ready for our first release > > > > > > > > I'd like to propose that the Tendrl contributors get together with a > > > > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) and > > > > collaborate on various pending/upcoming items. > > > > > > > > Dates: 8, 9, 10 Nov 2016 > > > > Location: Red Hat, Bangalore, India > > > > > > > > Please share your ideas for this proposal so we can get a rough > agenda > > in > > > > place soon. > > > > > > > > Regards, > > > > Rohan Kanade > > > > _______________________________________________ > > > > Tendrl-devel mailing list > > > > Tendrl-devel at redhat.com > > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > _______________________________________________ > > > Tendrl-devel mailing list > > > Tendrl-devel at redhat.com > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > > > > > > -- > > Jeff Applewhite > > Principal Product Manager > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From japplewh at redhat.com Wed Oct 19 11:40:02 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Wed, 19 Oct 2016 07:40:02 -0400 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: ?Agreed - let's stick with the original plan? On Wed, Oct 19, 2016 at 7:27 AM, Nishanth Thomas wrote: > I think its important to have Rohan here for hack days > > On Wed, Oct 19, 2016 at 4:52 PM, Rohan Kanade wrote: > > > Jeff, > > > > I wont be available during 30 Oct to 4th Nov (Important Indian Festival > > time). If other team members are available from 2nd-4th Nov, we can chalk > > up the agenda and continue on these dates. > > > > On Tue, Oct 18, 2016 at 9:45 PM, Jeff Applewhite > > wrote: > > > > > I think it's a great idea! +1 > > > > > > But given that we are releasing the first version of Tendrl on Nov 8 it > > > seems like maybe we should target the 2nd-4th of November. > > > That would give us more time for QE and acceptance of user stories > > > > > > Thoughts? > > > > > > On Tue, Oct 18, 2016 at 8:28 AM, Mrugesh Karnik < > mrugesh at brainfunked.org > > > > > > wrote: > > > > > > > I whole-heartedly second this! > > > > > > > > On 18 October 2016 at 17:41, Rohan Kanade > wrote: > > > > > > > > > > Tendrl is going through its first release cycle and we have covered > > > some > > > > > ground w.r.t Basic plumbing, inter component messaging, central > store > > > > > (etcd), tendrl API, Ceph/Gluster ops etc. > > > > > > > > > > We still need to cover significant ground in these areas before we > > can > > > be > > > > > ready for our first release > > > > > > > > > > I'd like to propose that the Tendrl contributors get together with > a > > > > > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) > and > > > > > collaborate on various pending/upcoming items. > > > > > > > > > > Dates: 8, 9, 10 Nov 2016 > > > > > Location: Red Hat, Bangalore, India > > > > > > > > > > Please share your ideas for this proposal so we can get a rough > > agenda > > > in > > > > > place soon. > > > > > > > > > > Regards, > > > > > Rohan Kanade > > > > > _______________________________________________ > > > > > Tendrl-devel mailing list > > > > > Tendrl-devel at redhat.com > > > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > _______________________________________________ > > > > Tendrl-devel mailing list > > > > Tendrl-devel at redhat.com > > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > > > > > > > > > > > -- > > > Jeff Applewhite > > > Principal Product Manager > > > _______________________________________________ > > > Tendrl-devel mailing list > > > Tendrl-devel at redhat.com > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From rkanade at redhat.com Wed Oct 19 11:56:45 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Wed, 19 Oct 2016 17:26:45 +0530 Subject: [Tendrl-devel] Moving code from Skyrings/kitoon to Tendrl/tendrl_frontend In-Reply-To: References: Message-ID: Should have replied early, 1. All the Tendrl repos should be under LGPL 2.1. 2. For LGPL project (tendrl_frontend) that copies in some source code from upstream Apache License 2.0 projects (such as Kitoon): Make sure you are compliant with the Apache License - this generally means including a copy of the Apache License 2.0 and keeping intact any upstream license notices or other legal notices including copyright/license notices in individual source files. On Fri, Oct 14, 2016 at 6:07 PM, Soumya Deb wrote: > > On 14 October 2016 at 17:57, Kanagaraj M wrote: > > > > > Since tendrl_frontend repo is already present, you might need to drop it > > and fork from kitoon. I am not sure if github provides a way to fork and > > merge. > > > > It's possible to push kitoon's code with detached head for tendrl_frontend > as the master, and rename the existing master branch to something else (or > drop it), at which point it'll contain the same history as of the previous > repo. > As I mentioned the technical challenge isn't that much, and can be easily > overcome. > > Thanks for pointing this out though, I avoided detailing them in the > original email. > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From deb at redhat.com Wed Oct 19 13:46:36 2016 From: deb at redhat.com (Soumya Deb) Date: Wed, 19 Oct 2016 19:16:36 +0530 Subject: [Tendrl-devel] Moving code from Skyrings/kitoon to Tendrl/tendrl_frontend In-Reply-To: References: Message-ID: Hi Rohan, Thanks for the suggestion, I'll keep them in mind. Ric had prescribed explaining the situation to Richard Fontana and take his learned opinion on this matter just to be very sure. Just a few hours back I have received feedback regarding this and he's also positive that we're good to go with this. I will see this done by tomorrow. On 19 October 2016 at 17:26, Rohan Kanade wrote: > Should have replied early, > > 1. All the Tendrl repos should be under LGPL 2.1. > > 2. For LGPL project (tendrl_frontend) that copies in some source code from > upstream > Apache License 2.0 projects (such as Kitoon): > > Make sure you are compliant with the Apache License - this generally > means including a copy of the Apache License 2.0 and keeping intact any > upstream > license notices or other legal notices including copyright/license > notices in individual source files. > > > > > On Fri, Oct 14, 2016 at 6:07 PM, Soumya Deb wrote: > > > > On 14 October 2016 at 17:57, Kanagaraj M > wrote: > > > > > > > > Since tendrl_frontend repo is already present, you might need to drop > it > > > and fork from kitoon. I am not sure if github provides a way to fork > and > > > merge. > > > > > > > It's possible to push kitoon's code with detached head for > tendrl_frontend > > as the master, and rename the existing master branch to something else > (or > > drop it), at which point it'll contain the same history as of the > previous > > repo. > > As I mentioned the technical challenge isn't that much, and can be easily > > overcome. > > > > Thanks for pointing this out though, I avoided detailing them in the > > original email. > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > From sankarshan.mukhopadhyay at gmail.com Wed Oct 19 13:54:14 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Wed, 19 Oct 2016 19:24:14 +0530 Subject: [Tendrl-devel] GitHub access rights In-Reply-To: References: Message-ID: On Fri, Oct 14, 2016 at 7:28 PM, Jeff Applewhite wrote: > Yes agreed - we should have a plan for how we will accommodate contributors > before things get unwieldy. It is fairly monolithic at this point but > perhaps we can break down the ownership a bit. You sound like the perfect > person to suggest this plan Soumya. Look forward to hearing it! > Here's how I see this - I think it is good to have a baseline proposal on how the organization would like to have access rights and such in place. It provides a baseline idea to prospect participants (individuals and organizations) around this topic. I would hesitate to categorize this activity as "urgent" and rather to me it is the "good to have" which we can pick-up once we have a couple of releases underway. I am proposing trading the "perfect approach" against "just good enough to start with". I prefer the latter. The path to contributions and participation needs to be complemented with artefacts to contribute to - the latter is of higher priority. > On Fri, Oct 14, 2016 at 8:56 AM, Ju Lim wrote: > >> Thanks Soumya. I appreciate you bringing up this point as I too have >> struggled with how we've set this up. >> >> I look forward to seeing your proposal! >> >> Regards, >> Ju >> >> On Fri, Oct 14, 2016 at 8:53 AM, Soumya Deb wrote: >> >> > Gmail doesn't support enough emoji, but I'd be glad to draft one. >> > Needed something interesting to do for the weekend anyway. :D >> > >> > >> > On 14 October 2016 at 18:08, Ju Lim wrote: >> > >> > > Soumya: >> > > >> > > +1 to needing a plan. As I don't believe such a plan exists yet, would >> > > you be willing to draft a proposal and share it? >> > > >> > > Thanks, >> > > Ju >> > > >> > > On Fri, Oct 14, 2016 at 8:31 AM, Soumya Deb wrote: >> > > >> > >> Do we have a plan around who should be having which tier of access >> > rights >> > >> on Tendrl? >> > >> >> > >> So far it seems very informal & inter-personal (which is okay to start >> > >> with), but going forward, we probably should have a properly defined >> > plan >> > >> about requesting access, acceptance criteria, accepting body etc. at >> > least >> > >> at a basic level. >> > >> >> > >> Tendrl being an opensource project, I don't see why its core parts >> > >> (codebase management being one) can't be more federated. >> > >> >> > >> --- >> > >> >> > >> The reason to have to write this email, is because I'm trying to get >> > some >> > >> service integration done for the Tendrl frontend - which requires >> > >> admin/owner access to the repo. I'm not sure who all hold this access >> > >> right >> > >> - but at least Sankarshan & Rohan does, that I'm aware of. I had asked >> > >> Rohan if he'd be willing to help me out, and he agreed without >> > hesitation >> > >> - >> > >> much appreciated. >> > >> >> > >> Now, I am part for 10+ GitHub orgs (admin of 5+) and have done plenty >> of >> > >> GitHub housekeeping in those ones that I'm pretty confident about >> being >> > >> able to manage those chores on my own. So, if I were to request >> access, >> > >> estimating that I am capable, how do I proceed? Please note, by no way >> > I'm >> > >> saying "I should be" having access; rather, if I, for one, was >> > interested >> > >> to share hands (still subject to actually passing the acceptance >> > criteria) >> > >> - then how do I proceed? >> > >> >> > >> I'm sure just emailing Jeff/Sankarshan about this would've been easy, >> > but >> > >> hey... who wants easy? :D -- sankarshan mukhopadhyay From gmeno at redhat.com Wed Oct 19 19:52:17 2016 From: gmeno at redhat.com (Gregory Meno) Date: Wed, 19 Oct 2016 12:52:17 -0700 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: On Thu, Oct 13, 2016 at 4:18 AM, Rohan Kanade wrote: > > > On Thu, Oct 6, 2016 at 7:26 PM, Gregory Meno wrote: >> >> On Wed, Oct 5, 2016 at 5:57 AM, Rohan Kanade wrote: >> > Calamari is an important component in the Ceph management/ops process. >> > In >> > the past, The Skyrings [0] project was able to utilize Calamari via the >> > Calamari REST API, the integration did ran into some problems as >> > mentioned >> > below. >> > >> > - Calamari-server would often be a single point of failure as it is >> > installed on the ceph-mon nodes >> >> Consider that monitor nodes are the right place to run and there are >> usually 3 or 5 monitors. >> Is that enough? I think we are at a point now where this would be >> reasonable to run on each monitor. >> What is more interesting to me is how you see the interaction happening. >> >> Does each process instance track the mon leader and proxy requests to >> the instance that is on the leader? > > Tendrl calamari interaction will happen via calamari REST API, Tendrl wont > be tracking calamari processes in any part of the cluster > >> John: have we considered what the ceph-mgr answer to this is? >> >> Do you expect to have perfect replication of state related to requests >> to change the ceph cluster? > > eg: If we call GET /calamari/osds , it should provide a list of osds up > until that point in time > >> How do you want to discover available endpoints? > > On a new cluster or an existing cluster, Tendrl will assume calamari is > pre-installed and functional (this will be part of Tendrl pre-requisites) > Then, while installing Tendrl itself, it should be provided with the > calamari REST API endpoint either via conf files or user input. > >> >> > - Querying Calamari REST API gave inconsistent data about the ceph >> > cluster >> > state, this might be happening because of sync issues occurring due to >> > ceph-mons not being in quorum etc. >> >> Specific tickets would go a long way here. >> >> > - Calamari required to be installed on only one of the ceph-mons, This >> > introduces very common scalability issues. >> >> I think we can relax this requirement if we can agree on expectations >> above. >> >> >> > - Calamari relies on older version of saltstack for messaging and other >> > communication, this introduces packaging and performance overheads >> >> This is not the case. We only use salt to send events to tendrl and >> it's something I consider to be a mistake. >> I think salt can be taken out of the picture in future releases > > Ok, removal of salt works for Tendrl > >> >> > >> > These issues must be re-verified and fixed if they still persist in >> > Calamari. >> > >> > >> > >> > >> > The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried to >> > address these issues by: >> >> Did they succeed? > > Yes >> >> > >> > - Distributing the tendrl ceph bridge to talk to all ceph-mons in the >> > cluster. >> > - Gathering data (3 second poll interval) from ceph-mons only when they >> > are >> > in quorum and directly talking to ceph-mons via librados and using ceph >> > admin sockets to query valid cluster maps and other ceph data. >> >> Since ceph_bridge is taken directly from calamari it's fair to say >> that we posses the same capabilities > > Please check the forked repository for changes in capabilities > >> >> > - Tendrl ceph bridge tries to address the single point of failure issue >> > by >> > having multiple instances of the bridge for each ceph-mon in the >> > cluster. >> >> Would you please show me where ceph_bridge is aware of other >> instances? Or are these multiple copies sharing nothing? > > For READ of cluster state, we only pull data out of the cluster if all mons > are in quorum. We have a single copy of the state which is refreshed at > regular intervals as well as per specific events. > >> >> > - Tendrl ceph bridge has centralized communications via an etcd cluster >> > which removes need to maintain saltstack . >> > >> > >> > >> > Moving forward, we would like Calamari to accommodate these improvements >> > back into the calamari code base, Below are the detailed requirements: >> > >> > - Need Calamari REST API for querying ceph admin sockets [2], More >> > details: >> >> Let's not expose this. I would consider it to be a challenge to your >> goal of having multiple instances >> as you're talking to a specific socket. Really what you get is info >> about clusters that are being setup or severly damaged this should be >> built into whatever service is responsible for identifying clusters >> and talking to them e.g. calamari and in future ceph-mgr. > > This is required specifically to verify correctness of cluster state > reported by other instances. And as i mentioned we only pull data from > sockets if all mons are in quorum. > >> >> > - Need Calamari REST API for providing all cluster maps (monitor, osd, >> > crush, mds, pg, health, config etc) >> >> This seems like a strange request. Why do you want access to the raw >> data? You'll have to parse it to make sense of it the same way we >> already do in calamari and in future ceph-mgr. > > Calamari already provides this via REST API > http://calamari.readthedocs.io/en/latest/calamari_rest/resources/api_example_api_v2_cluster__fsid__sync_object.html > Tendrl is simply asking this to be supported in future. > >> >> > - Calamari REST API must represent the ceph cluster state correctly, no >> > mismatch should occur between the True state of the cluster and the >> > state >> > represented by the Calamari REST API >> >> Happy to work to fix specific dependencies. Tickets are best. >> >> > - Calamari REST API should provide access to monitoring actions like >> > "ceph >> > -w", "ceph tell" etc >> >> Help me understand the specifics of what you want to accomplish here. >> the output of ceph -w can be synthesized from looking at the cluster >> maps. and ceph tell is typically used for troubleshooting specific >> instances where you want to set new config with out changing ceph.conf >> and restarting. That means it applies to a single instance, not >> cluster wide. > > ceph -w will be used to generate events based on the output. Computing such > events from cluster maps is doable too but is more work. > Tendrl works not only on a cluster context but also at a node context, ceph > tell is required for operations which pertain to specific nodes. > >> >> > - Calamari REST API should provide access to operational actions via >> > tools >> > like "rbd" etc. >> >> Agreed. >> >> > - Calamari REST API should be able to service and perform well on 2-4 >> > second >> > polling intervals on the REST API >> >> Yes I agree here too. >> >> > >> > >> > These requirements can be run through existing calamari functional and >> > performance test cases and we should make additions to these test cases >> > to >> > accommodate above requirements. >> > >> > These requirements would be further expanded into individual BZs or >> > tracking >> > items in Calamari >> > >> > >> > [0]: https://github.com/skyrings >> > [1]: https://github.com/tendrl/ceph_bridge >> > [2]: >> > >> > http://docs.ceph.com/docs/jewel/rados/operations/monitoring/#using-the-admin-socket I've read through this thread again and I'm trying to capture all the items where we agree on the next steps: * add support for RBD * get rid of salt * run on all monitor nodes With eventually consistent results for READ operations * leave the sync_object API in place Items where I need more detail to agree to: * access to admin sockets to "verify correctness of cluster state". Can we not trust the mon_map? If we cannot it needs either better docs or bug fixes. I think this state needs to be exposed correctly by ceph. Does that make sense? * perform well. Would you please file specific tickets for what endpoints are not responding promptly or causing high load? Items I don't see implementing before ceph-mgr transition and maybe not then: * access to ceph -w and ceph tell Ceph -w seems like a poor source of events that would require long-polling or eventing to surface, and if we did you'd be scraping a data-source that may change. I'm not saying no to eventing, but calamari is definatly the wrong place for this to live. ceph tell is an implementation detail for changing config at runtime. I agree that we might want to alter configuration values in rare instances. We should either convince Ceph to set better defaults or provide write access to a /config endpoint. I would say that if you are looking to troubleshoot a cluster and want ephemeral config values for specific daemons that we are a long way off doing this correctly. I want to work with you and help Tendrl get what it needs AND I'm still trying to figure out just what those specifics are. cheers, G From julim at redhat.com Wed Oct 19 21:07:48 2016 From: julim at redhat.com (Ju Lim) Date: Wed, 19 Oct 2016 17:07:48 -0400 Subject: [Tendrl-devel] Tendrl Sprint 3 (19 Oct 2016 - 2 Nov 2016) Message-ID: Team: Just a quick recap from today's Sprint 3 Planning. The goals and the recording may be found at https://tendrl.atlassian.net/wiki/display/TEN/Sprints. Please note that based on offline discussion, we've decided to keep our Sprints at the 2 week cycle for Sprint 3 (vs. extending out to include the upcoming Hackathon) so that we can ensure that there's a demo planned. So the Sprint 3 will end on Wed, 2 Nov 2016 (instead of Tues, 1 Nov 2016) -- this was adjusted to accommodate the holidays in India on 31 Oct - 1 Nov. Sprint 4 will begin as soon as Sprint 3 ends. On another note regarding demos, if your work satisfies the Definition of Done (DoD) long before the Sprint ends, it would be very much welcomed by all if you would like to put together a recorded demo and share it -- specifically, adding the link to the recorded demo to the JIRA User Story/Task and publishing it to the Tendrl-Devel mailing list. The following were action items captured today: - Logging - Nishant to assign (TEN-67) - Nishant & Shubhendu to discuss / break down further TEN-35 (Provide management access to Gluster (CRUD ops on cluster objects) - User stories would be created by Rohan for the following: - Ancilary tasks for cluster import - Persistence of jobs and flows - Stablelizing and freezing of YML schema - API Docs (if it doesn't exist) - Clarification and scope of what logging, jobs, events, and alerts for Tendrl1 (and rest of the releases) needs to be sorted out (AI for Jeff) - Hackathon - everyone who will participate is asked to comment on the email thread - UX and Dev team to meet tomorrow on various topics -- scheduled on the shared calendar As a reminder, if folks have upcoming OOO/PTO, please add it to the shared calendar. If you have work that others are depending on you to complete, please ensure you've completed the work before you go on OOO/PTO or have someone to take it over during your absence. If I've missed anything, please feel free to add/edit. Thanks, Ju From japplewh at redhat.com Thu Oct 20 11:13:58 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Thu, 20 Oct 2016 07:13:58 -0400 Subject: [Tendrl-devel] logging Message-ID: Hi All Here is a snip from an email with Peter Portante regarding best practices on logging: --Thanks Peter this pertains to ten-67 in Jira "Hi Jeff, Before all else, ensure the time stamps on log entires are either UTC or always include an accurate timezone. And ensure the environment in which the app or system runs has an accurate notion of time. Biggest problem to make sure this is solved properly. The next is how logs are emitted by an application. Using JSON is one of the best ways to get logs emitted with rich metadata that enables further use when they are analyzed and correlated to other logs, metrics, and configuration data. If one cannot use JSON, is to make sure the logs that are emitted are uniform, easily described, and machine parse-able. The next is ensure that the metadata is named following that namespace document you reference. There is a project on GitHub called ViaQ where we work with practical delivery of documented namespaces, see https://github.com/ViaQ/elasticsearch-templates/ Lastly, visualize early and often. =) That is, work with an environment like Elasticsearch & Kibana to ensure when you emit for logs can be placed into a dashboard that shows in aggregate the various pieces of metadata associated with the logs, and that those visualizations are useful. ?"? -- Jeff Applewhite Principal Product Manager From rkanade at redhat.com Thu Oct 20 12:27:38 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Thu, 20 Oct 2016 17:57:38 +0530 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: Lets start working on what we have mutually agreed upon till now. - Continue the sync_object API in place (http://calamari.readthedocs.i o/en/latest/calamari_rest/resources/api_example_api_v2_clust er__fsid__sync_object__sync_type_.html) - get rid of salt dependency completely - provide single REST API endpoint (Tendrl does not need to worry on what mons calamari runs this API) consumable by Tendrl and ensure scenario like below works as described eg: Assuming that both below actions are taken at the same time. 1. GET api/v2/cluster//pool should return same list of pools as step 2. 2 ceph osd lspools - Add support for RBD - In a pre-defined environment, calamari should be stress tested for a refresh interval of 2-3 seconds per API call. We would need to add this to calamari QE test cases maybe. These are the minimum requirements Tendrl has from calamari. If we are agreeing on these, we can file detailed BZ's. On Thu, Oct 20, 2016 at 1:22 AM, Gregory Meno wrote: > On Thu, Oct 13, 2016 at 4:18 AM, Rohan Kanade wrote: > > > > > > On Thu, Oct 6, 2016 at 7:26 PM, Gregory Meno wrote: > >> > >> On Wed, Oct 5, 2016 at 5:57 AM, Rohan Kanade > wrote: > >> > Calamari is an important component in the Ceph management/ops process. > >> > In > >> > the past, The Skyrings [0] project was able to utilize Calamari via > the > >> > Calamari REST API, the integration did ran into some problems as > >> > mentioned > >> > below. > >> > > >> > - Calamari-server would often be a single point of failure as it is > >> > installed on the ceph-mon nodes > >> > >> Consider that monitor nodes are the right place to run and there are > >> usually 3 or 5 monitors. > >> Is that enough? I think we are at a point now where this would be > >> reasonable to run on each monitor. > >> What is more interesting to me is how you see the interaction happening. > >> > >> Does each process instance track the mon leader and proxy requests to > >> the instance that is on the leader? > > > > Tendrl calamari interaction will happen via calamari REST API, Tendrl > wont > > be tracking calamari processes in any part of the cluster > > > >> John: have we considered what the ceph-mgr answer to this is? > >> > >> Do you expect to have perfect replication of state related to requests > >> to change the ceph cluster? > > > > eg: If we call GET /calamari/osds , it should provide a list of osds up > > until that point in time > > > >> How do you want to discover available endpoints? > > > > On a new cluster or an existing cluster, Tendrl will assume calamari is > > pre-installed and functional (this will be part of Tendrl > pre-requisites) > > Then, while installing Tendrl itself, it should be provided with the > > calamari REST API endpoint either via conf files or user input. > > > >> > >> > - Querying Calamari REST API gave inconsistent data about the ceph > >> > cluster > >> > state, this might be happening because of sync issues occurring due to > >> > ceph-mons not being in quorum etc. > >> > >> Specific tickets would go a long way here. > >> > >> > - Calamari required to be installed on only one of the ceph-mons, This > >> > introduces very common scalability issues. > >> > >> I think we can relax this requirement if we can agree on expectations > >> above. > >> > >> > >> > - Calamari relies on older version of saltstack for messaging and > other > >> > communication, this introduces packaging and performance overheads > >> > >> This is not the case. We only use salt to send events to tendrl and > >> it's something I consider to be a mistake. > >> I think salt can be taken out of the picture in future releases > > > > Ok, removal of salt works for Tendrl > > > >> > >> > > >> > These issues must be re-verified and fixed if they still persist in > >> > Calamari. > >> > > >> > > >> > > >> > > >> > The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried > to > >> > address these issues by: > >> > >> Did they succeed? > > > > Yes > >> > >> > > >> > - Distributing the tendrl ceph bridge to talk to all ceph-mons in the > >> > cluster. > >> > - Gathering data (3 second poll interval) from ceph-mons only when > they > >> > are > >> > in quorum and directly talking to ceph-mons via librados and using > ceph > >> > admin sockets to query valid cluster maps and other ceph data. > >> > >> Since ceph_bridge is taken directly from calamari it's fair to say > >> that we posses the same capabilities > > > > Please check the forked repository for changes in capabilities > > > >> > >> > - Tendrl ceph bridge tries to address the single point of failure > issue > >> > by > >> > having multiple instances of the bridge for each ceph-mon in the > >> > cluster. > >> > >> Would you please show me where ceph_bridge is aware of other > >> instances? Or are these multiple copies sharing nothing? > > > > For READ of cluster state, we only pull data out of the cluster if all > mons > > are in quorum. We have a single copy of the state which is refreshed at > > regular intervals as well as per specific events. > > > >> > >> > - Tendrl ceph bridge has centralized communications via an etcd > cluster > >> > which removes need to maintain saltstack . > >> > > >> > > >> > > >> > Moving forward, we would like Calamari to accommodate these > improvements > >> > back into the calamari code base, Below are the detailed requirements: > >> > > >> > - Need Calamari REST API for querying ceph admin sockets [2], More > >> > details: > >> > >> Let's not expose this. I would consider it to be a challenge to your > >> goal of having multiple instances > >> as you're talking to a specific socket. Really what you get is info > >> about clusters that are being setup or severly damaged this should be > >> built into whatever service is responsible for identifying clusters > >> and talking to them e.g. calamari and in future ceph-mgr. > > > > This is required specifically to verify correctness of cluster state > > reported by other instances. And as i mentioned we only pull data from > > sockets if all mons are in quorum. > > > >> > >> > - Need Calamari REST API for providing all cluster maps (monitor, osd, > >> > crush, mds, pg, health, config etc) > >> > >> This seems like a strange request. Why do you want access to the raw > >> data? You'll have to parse it to make sense of it the same way we > >> already do in calamari and in future ceph-mgr. > > > > Calamari already provides this via REST API > > http://calamari.readthedocs.io/en/latest/calamari_rest/ > resources/api_example_api_v2_cluster__fsid__sync_object.html > > Tendrl is simply asking this to be supported in future. > > > >> > >> > - Calamari REST API must represent the ceph cluster state correctly, > no > >> > mismatch should occur between the True state of the cluster and the > >> > state > >> > represented by the Calamari REST API > >> > >> Happy to work to fix specific dependencies. Tickets are best. > >> > >> > - Calamari REST API should provide access to monitoring actions like > >> > "ceph > >> > -w", "ceph tell" etc > >> > >> Help me understand the specifics of what you want to accomplish here. > >> the output of ceph -w can be synthesized from looking at the cluster > >> maps. and ceph tell is typically used for troubleshooting specific > >> instances where you want to set new config with out changing ceph.conf > >> and restarting. That means it applies to a single instance, not > >> cluster wide. > > > > ceph -w will be used to generate events based on the output. Computing > such > > events from cluster maps is doable too but is more work. > > Tendrl works not only on a cluster context but also at a node context, > ceph > > tell is required for operations which pertain to specific nodes. > > > >> > >> > - Calamari REST API should provide access to operational actions via > >> > tools > >> > like "rbd" etc. > >> > >> Agreed. > >> > >> > - Calamari REST API should be able to service and perform well on 2-4 > >> > second > >> > polling intervals on the REST API > >> > >> Yes I agree here too. > >> > >> > > >> > > >> > These requirements can be run through existing calamari functional and > >> > performance test cases and we should make additions to these test > cases > >> > to > >> > accommodate above requirements. > >> > > >> > These requirements would be further expanded into individual BZs or > >> > tracking > >> > items in Calamari > >> > > >> > > >> > [0]: https://github.com/skyrings > >> > [1]: https://github.com/tendrl/ceph_bridge > >> > [2]: > >> > > >> > http://docs.ceph.com/docs/jewel/rados/operations/ > monitoring/#using-the-admin-socket > > > I've read through this thread again and I'm trying to capture all the > items where we agree on the next steps: > * add support for RBD > * get rid of salt > * run on all monitor nodes With eventually consistent results for READ > operations > * leave the sync_object API in place > > Items where I need more detail to agree to: > * access to admin sockets to "verify correctness of cluster state". > Can we not trust the mon_map? If we cannot it needs either better docs > or bug fixes. I think this state needs to be exposed correctly by > ceph. Does that make sense? > * perform well. Would you please file specific tickets for what > endpoints are not responding promptly or causing high load? > > > Items I don't see implementing before ceph-mgr transition and maybe not > then: > * access to ceph -w and ceph tell > Ceph -w seems like a poor source of events that would require > long-polling or eventing to surface, and if we did you'd be scraping a > data-source that may change. I'm not saying no to eventing, but > calamari is definatly the wrong place for this to live. > > ceph tell is an implementation detail for changing config at runtime. > I agree that we might want to alter configuration values in rare > instances. We should either convince Ceph to set better defaults or > provide write access to a /config endpoint. > I would say that if you are looking to troubleshoot a cluster and want > ephemeral config values for specific daemons that we are a long way > off doing this correctly. > > I want to work with you and help Tendrl get what it needs AND I'm > still trying to figure out just what those specifics are. > > cheers, > G > From japplewh at redhat.com Thu Oct 20 19:25:21 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Thu, 20 Oct 2016 15:25:21 -0400 Subject: [Tendrl-devel] TripleO blueprint In-Reply-To: <1769912770.2196280.1476808513412.JavaMail.zimbra@redhat.com> References: <1769912770.2196280.1476808513412.JavaMail.zimbra@redhat.com> Message-ID: After reading through this I think it would be a good idea if we read it carefully and comment where appropriate before the discussion next week at OpenStack summit (Thursday?). There are specifics around communication mechanisms and other things that ought to be verified imo. Thanks Jeff On Tue, Oct 18, 2016 at 12:35 PM, John Fulton wrote: > Thanks Jeff, > > Also, I started a thread on it on the OpenStack dev list in case anyone is > interested: > > http://lists.openstack.org/pipermail/openstack-dev/2016- > October/105951.html > > John > > > ----- Original Message ----- > From: "Jeff Applewhite" > To: "Mailing list for the contributors to the Tendrl project" < > tendrl-devel at redhat.com> > Cc: "John Fulton" > Sent: Tuesday, October 18, 2016 10:30:15 AM > Subject: TripleO blueprint > > Hi All > > An OpenStack blueprint was added today by John Fulton > > "? > Integrate TripleO with Tendrl for External Storage Deployment/Management > ?"? > > > > Please take a look: https://review.openstack.org/#/c/387631 > > ?I'll be adding this to the Jira backlog items from the Tendrl perspective. > The work will revolved around developing and integrating an API > service for flows > that will include create job, initial install, updates, upgrades, node > replacement, node scaling, cluster import, and cluster export for Ceph. > > Thanks!? > > > Jeff > -- Jeff Applewhite Principal Product Manager From jefbrown at redhat.com Thu Oct 20 21:32:33 2016 From: jefbrown at redhat.com (Jeff Brown) Date: Thu, 20 Oct 2016 17:32:33 -0400 Subject: [Tendrl-devel] TripleO blueprint In-Reply-To: References: <1769912770.2196280.1476808513412.JavaMail.zimbra@redhat.com> Message-ID: Let's talk in the morning. Thanks On Thursday, October 20, 2016, Jeff Applewhite wrote: > After reading through this I think it would be a good idea if we read it > carefully and comment where appropriate before the discussion next week at > OpenStack summit (Thursday?). There are specifics around communication > mechanisms and other things that ought to be verified imo. > > > Thanks > > Jeff > > On Tue, Oct 18, 2016 at 12:35 PM, John Fulton > wrote: > >> Thanks Jeff, >> >> Also, I started a thread on it on the OpenStack dev list in case anyone >> is interested: >> >> http://lists.openstack.org/pipermail/openstack-dev/2016-Oct >> ober/105951.html >> >> John >> >> >> ----- Original Message ----- >> From: "Jeff Applewhite" > > >> To: "Mailing list for the contributors to the Tendrl project" < >> tendrl-devel at redhat.com >> > >> Cc: "John Fulton" > > >> Sent: Tuesday, October 18, 2016 10:30:15 AM >> Subject: TripleO blueprint >> >> Hi All >> >> An OpenStack blueprint was added today by John Fulton >> >> "? >> Integrate TripleO with Tendrl for External Storage Deployment/Management >> ?"? >> >> >> >> Please take a look: https://review.openstack.org/#/c/387631 >> >> ?I'll be adding this to the Jira backlog items from the Tendrl >> perspective. >> The work will revolved around developing and integrating an API >> service for flows >> that will include create job, initial install, updates, upgrades, node >> replacement, node scaling, cluster import, and cluster export for Ceph. >> >> Thanks!? >> >> >> Jeff >> > > > > -- > Jeff Applewhite > Principal Product Manager > > From kdreyer at redhat.com Fri Oct 21 01:02:17 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Thu, 20 Oct 2016 19:02:17 -0600 Subject: [Tendrl-devel] Tendrl and ceph-installer in CentOS SIG's Koji Message-ID: Hi folks, I'd like to get Tendrl and ceph-installer packaged in the CentOS Storage SIG. Looking at the list of Koji tags they've set up previously [1], I think it would be appropriate to set up a "tendrl-1" tag. (Assuming our first version of Tendrl is going to be "1"). storage7-tendrl-1-{candidate,testing,release} storage7-tendrl-1-el7-build Any objections? - Ken https://wiki.centos.org/SpecialInterestGroup/Storage/sigworkflow From nthomas at redhat.com Fri Oct 21 03:02:04 2016 From: nthomas at redhat.com (Nishanth Thomas) Date: Fri, 21 Oct 2016 08:32:04 +0530 Subject: [Tendrl-devel] Tendrl and ceph-installer in CentOS SIG's Koji In-Reply-To: References: Message-ID: Hi Ken, Looks good to me. Are you planing to work on writing the spec files for tendrl packages? Thanks, Nishanth On Fri, Oct 21, 2016 at 6:32 AM, Ken Dreyer wrote: > Hi folks, > > I'd like to get Tendrl and ceph-installer packaged in the CentOS Storage > SIG. > > Looking at the list of Koji tags they've set up previously [1], I > think it would be appropriate to set up a "tendrl-1" tag. (Assuming > our first version of Tendrl is going to be "1"). > > storage7-tendrl-1-{candidate,testing,release} > storage7-tendrl-1-el7-build > > Any objections? > > - Ken > > https://wiki.centos.org/SpecialInterestGroup/Storage/sigworkflow > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From shtripat at redhat.com Fri Oct 21 10:26:03 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Fri, 21 Oct 2016 15:56:03 +0530 Subject: [Tendrl-devel] logging In-Reply-To: References: Message-ID: Hi Jeff, Today I have tried out few things around logging framework in tendrl. The summary of the same goes as below ------------------------------------------------- 1. Added a configuration /etc/tendrl/logging.json (can be ported as yaml as well) with below content { "version": 1, "disable_existing_loggers": false, "formatters": { "simple": { "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s", "datefmt": "%Y-%m-%dT%H:%M:%S%z" } }, "handlers": { "console": { "class": "logging.StreamHandler", "level": "DEBUG", "formatter": "simple", "stream": "ext://sys.stdout" }, "info_file_handler": { "class": "logging.handlers.SysLogHandler", "level": "INFO", "formatter": "simple" }, "error_file_handler": { "class": "logging.handlers.RotatingFileHandler", "level": "ERROR", "formatter": "simple", "filename": "/var/log/tendrl/errors.log", "maxBytes": 10485760, "backupCount": 20, "encoding": "utf8" } }, "loggers": { "my_module": { "level": "ERROR", "handlers": ["console"], "propagate": "no" } }, "root": { "level": "INFO", "handlers": ["console", "info_file_handler", "error_file_handler"] } } 2. Did small changes in tendrl gluster bridge logger to start logger with above configuration. ----------------------------------- Below are few facts figured out as part of this exercise - 1. There are different handlers like StreamHandler, SysLogHandler, RotatingFileHandler, FileHandler and JournalHandler (JournalHandler is provided by systemd-python package and not available with default python logging framework as such) which can be configured as log handler above 2. Different handlers can be configured for console, info and error files. So we can decide something like - all the "errors" are logged to journald/syslog and "info" messages to a rotating file. 3. There are formatters which could be configured to define the format of the log record. Date format can be decided using "datefmt" field in formatter as shown above (I have defaulted it to UTC for testing) 4. Individual modules can decide what all handlers defined to be used while logging 5. With RotationFileHandler we can set maximum no of bytes allowed in log file, backupCount to decide on how many backups to be kept. Havent tried time based rotation at the moment though ! With this I have tried rotational, file based, syslog based and journald based logging and find it working. Just wanted a heads up on this, if this meets kind of our expectation for tendrl. If so I can raise PR with changes soon :) Thanks and Regards, Shubhendu On 10/20/2016 04:43 PM, Jeff Applewhite wrote: > Hi All > > Here is a snip from an email with Peter Portante regarding best practices > on logging: --Thanks Peter > > this pertains to ten-67 in Jira > > "Hi Jeff, > > Before all else, ensure the time stamps on log entires are either UTC > or always include an accurate timezone. And ensure the environment in > which the app or system runs has an accurate notion of time. Biggest > problem to make sure this is solved properly. > > The next is how logs are emitted by an application. Using JSON is one > of the best ways to get logs emitted with rich metadata that enables > further use when they are analyzed and correlated to other logs, > metrics, and configuration data. If one cannot use JSON, is to make > sure the logs that are emitted are uniform, easily described, and > machine parse-able. > > The next is ensure that the metadata is named following that namespace > document you reference. There is a project on GitHub called ViaQ > where we work with practical delivery of documented namespaces, see > https://github.com/ViaQ/elasticsearch-templates/ > > Lastly, visualize early and often. =) That is, work with an > environment like Elasticsearch & Kibana to ensure when you emit for > logs can be placed into a dashboard that shows in aggregate the > various pieces of metadata associated with the logs, and that those > visualizations are useful. > ?"? > From shtripat at redhat.com Fri Oct 21 11:35:16 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Fri, 21 Oct 2016 17:05:16 +0530 Subject: [Tendrl-devel] logging In-Reply-To: References: Message-ID: <11c74ecd-e5cf-e72a-8cd5-92612d9689f4@redhat.com> Tried TimedRotatingFileHandler as well and works fine. Different possible intervals could be Seconds, Minutes, Hours, Days, Weekdays (0-6), 'midnight'. Regards, Shubhendu On 10/21/2016 03:56 PM, Shubhendu Tripathi wrote: > Hi Jeff, > > Today I have tried out few things around logging framework in tendrl. > The summary of the same goes as below > > ------------------------------------------------- > 1. Added a configuration /etc/tendrl/logging.json (can be ported as > yaml as well) with below content > > { > "version": 1, > "disable_existing_loggers": false, > "formatters": { > "simple": { > "format": "%(asctime)s - %(name)s - %(levelname)s - > %(message)s", > "datefmt": "%Y-%m-%dT%H:%M:%S%z" > } > }, > > "handlers": { > "console": { > "class": "logging.StreamHandler", > "level": "DEBUG", > "formatter": "simple", > "stream": "ext://sys.stdout" > }, > > "info_file_handler": { > "class": "logging.handlers.SysLogHandler", > "level": "INFO", > "formatter": "simple" > }, > > "error_file_handler": { > "class": "logging.handlers.RotatingFileHandler", > "level": "ERROR", > "formatter": "simple", > "filename": "/var/log/tendrl/errors.log", > "maxBytes": 10485760, > "backupCount": 20, > "encoding": "utf8" > } > }, > > "loggers": { > "my_module": { > "level": "ERROR", > "handlers": ["console"], > "propagate": "no" > } > }, > > "root": { > "level": "INFO", > "handlers": ["console", "info_file_handler", "error_file_handler"] > } > } > > 2. Did small changes in tendrl gluster bridge logger to start logger > with above configuration. > > ----------------------------------- > > Below are few facts figured out as part of this exercise - > > 1. There are different handlers like StreamHandler, SysLogHandler, > RotatingFileHandler, FileHandler and JournalHandler (JournalHandler is > provided by systemd-python package and not available with default > python logging framework as such) which can be configured as log > handler above > 2. Different handlers can be configured for console, info and error > files. So we can decide something like - all the "errors" are logged > to journald/syslog and "info" messages to a rotating file. > 3. There are formatters which could be configured to define the format > of the log record. Date format can be decided using "datefmt" field in > formatter as shown above (I have defaulted it to UTC for testing) > 4. Individual modules can decide what all handlers defined to be used > while logging > 5. With RotationFileHandler we can set maximum no of bytes allowed in > log file, backupCount to decide on how many backups to be kept. Havent > tried time based rotation at the moment though ! > > With this I have tried rotational, file based, syslog based and > journald based logging and find it working. > Just wanted a heads up on this, if this meets kind of our expectation > for tendrl. If so I can raise PR with changes soon :) > > Thanks and Regards, > Shubhendu > > On 10/20/2016 04:43 PM, Jeff Applewhite wrote: >> Hi All >> >> Here is a snip from an email with Peter Portante regarding best practices >> on logging: --Thanks Peter >> >> this pertains to ten-67 in Jira >> >> "Hi Jeff, >> >> Before all else, ensure the time stamps on log entires are either UTC >> or always include an accurate timezone. And ensure the environment in >> which the app or system runs has an accurate notion of time. Biggest >> problem to make sure this is solved properly. >> >> The next is how logs are emitted by an application. Using JSON is one >> of the best ways to get logs emitted with rich metadata that enables >> further use when they are analyzed and correlated to other logs, >> metrics, and configuration data. If one cannot use JSON, is to make >> sure the logs that are emitted are uniform, easily described, and >> machine parse-able. >> >> The next is ensure that the metadata is named following that namespace >> document you reference. There is a project on GitHub called ViaQ >> where we work with practical delivery of documented namespaces, see >> https://github.com/ViaQ/elasticsearch-templates/ >> >> Lastly, visualize early and often. =) That is, work with an >> environment like Elasticsearch & Kibana to ensure when you emit for >> logs can be placed into a dashboard that shows in aggregate the >> various pieces of metadata associated with the logs, and that those >> visualizations are useful. >> ?"? >> > From avishwan at redhat.com Fri Oct 21 12:06:19 2016 From: avishwan at redhat.com (Aravinda) Date: Fri, 21 Oct 2016 17:36:19 +0530 Subject: [Tendrl-devel] logging In-Reply-To: <11c74ecd-e5cf-e72a-8cd5-92612d9689f4@redhat.com> References: <11c74ecd-e5cf-e72a-8cd5-92612d9689f4@redhat.com> Message-ID: In Gluster Geo-replication we are using "WatchedFileHandler", so that running application need not worry if the log files are rotated externally using `logrotate` tools. https://docs.python.org/2/library/logging.handlers.html regards Aravinda On Friday 21 October 2016 05:05 PM, Shubhendu Tripathi wrote: > Tried TimedRotatingFileHandler as well and works fine. > Different possible intervals could be Seconds, Minutes, Hours, Days, > Weekdays (0-6), 'midnight'. > > Regards, > Shubhendu > > > On 10/21/2016 03:56 PM, Shubhendu Tripathi wrote: >> Hi Jeff, >> >> Today I have tried out few things around logging framework in tendrl. >> The summary of the same goes as below >> >> ------------------------------------------------- >> 1. Added a configuration /etc/tendrl/logging.json (can be ported as >> yaml as well) with below content >> >> { >> "version": 1, >> "disable_existing_loggers": false, >> "formatters": { >> "simple": { >> "format": "%(asctime)s - %(name)s - %(levelname)s - >> %(message)s", >> "datefmt": "%Y-%m-%dT%H:%M:%S%z" >> } >> }, >> >> "handlers": { >> "console": { >> "class": "logging.StreamHandler", >> "level": "DEBUG", >> "formatter": "simple", >> "stream": "ext://sys.stdout" >> }, >> >> "info_file_handler": { >> "class": "logging.handlers.SysLogHandler", >> "level": "INFO", >> "formatter": "simple" >> }, >> >> "error_file_handler": { >> "class": "logging.handlers.RotatingFileHandler", >> "level": "ERROR", >> "formatter": "simple", >> "filename": "/var/log/tendrl/errors.log", >> "maxBytes": 10485760, >> "backupCount": 20, >> "encoding": "utf8" >> } >> }, >> >> "loggers": { >> "my_module": { >> "level": "ERROR", >> "handlers": ["console"], >> "propagate": "no" >> } >> }, >> >> "root": { >> "level": "INFO", >> "handlers": ["console", "info_file_handler", >> "error_file_handler"] >> } >> } >> >> 2. Did small changes in tendrl gluster bridge logger to start logger >> with above configuration. >> >> ----------------------------------- >> >> Below are few facts figured out as part of this exercise - >> >> 1. There are different handlers like StreamHandler, SysLogHandler, >> RotatingFileHandler, FileHandler and JournalHandler (JournalHandler >> is provided by systemd-python package and not available with default >> python logging framework as such) which can be configured as log >> handler above >> 2. Different handlers can be configured for console, info and error >> files. So we can decide something like - all the "errors" are logged >> to journald/syslog and "info" messages to a rotating file. >> 3. There are formatters which could be configured to define the >> format of the log record. Date format can be decided using "datefmt" >> field in formatter as shown above (I have defaulted it to UTC for >> testing) >> 4. Individual modules can decide what all handlers defined to be used >> while logging >> 5. With RotationFileHandler we can set maximum no of bytes allowed in >> log file, backupCount to decide on how many backups to be kept. >> Havent tried time based rotation at the moment though ! >> >> With this I have tried rotational, file based, syslog based and >> journald based logging and find it working. >> Just wanted a heads up on this, if this meets kind of our expectation >> for tendrl. If so I can raise PR with changes soon :) >> >> Thanks and Regards, >> Shubhendu >> >> On 10/20/2016 04:43 PM, Jeff Applewhite wrote: >>> Hi All >>> >>> Here is a snip from an email with Peter Portante regarding best >>> practices >>> on logging: --Thanks Peter >>> >>> this pertains to ten-67 in Jira >>> >>> "Hi Jeff, >>> >>> Before all else, ensure the time stamps on log entires are either UTC >>> or always include an accurate timezone. And ensure the environment in >>> which the app or system runs has an accurate notion of time. Biggest >>> problem to make sure this is solved properly. >>> >>> The next is how logs are emitted by an application. Using JSON is one >>> of the best ways to get logs emitted with rich metadata that enables >>> further use when they are analyzed and correlated to other logs, >>> metrics, and configuration data. If one cannot use JSON, is to make >>> sure the logs that are emitted are uniform, easily described, and >>> machine parse-able. >>> >>> The next is ensure that the metadata is named following that namespace >>> document you reference. There is a project on GitHub called ViaQ >>> where we work with practical delivery of documented namespaces, see >>> https://github.com/ViaQ/elasticsearch-templates/ >>> >>> Lastly, visualize early and often. =) That is, work with an >>> environment like Elasticsearch & Kibana to ensure when you emit for >>> logs can be placed into a dashboard that shows in aggregate the >>> various pieces of metadata associated with the logs, and that those >>> visualizations are useful. >>> ?"? >>> >> > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From shtripat at redhat.com Fri Oct 21 12:25:15 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Fri, 21 Oct 2016 17:55:15 +0530 Subject: [Tendrl-devel] logging In-Reply-To: References: <11c74ecd-e5cf-e72a-8cd5-92612d9689f4@redhat.com> Message-ID: <416aad4f-7888-6b2d-7c74-010b581a7837@redhat.com> Thanks Aravinda for guidelines. I appreciate this. Yes for sure this could be an option for handlers. All we need is making it configurable and even support for syslog/journald etc should be easy to configure. Regards, Shubhendu On 10/21/2016 05:36 PM, Aravinda wrote: > In Gluster Geo-replication we are using "WatchedFileHandler", so that > running application need not worry if the log files are rotated > externally using `logrotate` tools. > https://docs.python.org/2/library/logging.handlers.html > > regards > Aravinda > > On Friday 21 October 2016 05:05 PM, Shubhendu Tripathi wrote: >> Tried TimedRotatingFileHandler as well and works fine. >> Different possible intervals could be Seconds, Minutes, Hours, Days, >> Weekdays (0-6), 'midnight'. >> >> Regards, >> Shubhendu >> >> >> On 10/21/2016 03:56 PM, Shubhendu Tripathi wrote: >>> Hi Jeff, >>> >>> Today I have tried out few things around logging framework in >>> tendrl. The summary of the same goes as below >>> >>> ------------------------------------------------- >>> 1. Added a configuration /etc/tendrl/logging.json (can be ported as >>> yaml as well) with below content >>> >>> { >>> "version": 1, >>> "disable_existing_loggers": false, >>> "formatters": { >>> "simple": { >>> "format": "%(asctime)s - %(name)s - %(levelname)s - >>> %(message)s", >>> "datefmt": "%Y-%m-%dT%H:%M:%S%z" >>> } >>> }, >>> >>> "handlers": { >>> "console": { >>> "class": "logging.StreamHandler", >>> "level": "DEBUG", >>> "formatter": "simple", >>> "stream": "ext://sys.stdout" >>> }, >>> >>> "info_file_handler": { >>> "class": "logging.handlers.SysLogHandler", >>> "level": "INFO", >>> "formatter": "simple" >>> }, >>> >>> "error_file_handler": { >>> "class": "logging.handlers.RotatingFileHandler", >>> "level": "ERROR", >>> "formatter": "simple", >>> "filename": "/var/log/tendrl/errors.log", >>> "maxBytes": 10485760, >>> "backupCount": 20, >>> "encoding": "utf8" >>> } >>> }, >>> >>> "loggers": { >>> "my_module": { >>> "level": "ERROR", >>> "handlers": ["console"], >>> "propagate": "no" >>> } >>> }, >>> >>> "root": { >>> "level": "INFO", >>> "handlers": ["console", "info_file_handler", >>> "error_file_handler"] >>> } >>> } >>> >>> 2. Did small changes in tendrl gluster bridge logger to start logger >>> with above configuration. >>> >>> ----------------------------------- >>> >>> Below are few facts figured out as part of this exercise - >>> >>> 1. There are different handlers like StreamHandler, SysLogHandler, >>> RotatingFileHandler, FileHandler and JournalHandler (JournalHandler >>> is provided by systemd-python package and not available with default >>> python logging framework as such) which can be configured as log >>> handler above >>> 2. Different handlers can be configured for console, info and error >>> files. So we can decide something like - all the "errors" are logged >>> to journald/syslog and "info" messages to a rotating file. >>> 3. There are formatters which could be configured to define the >>> format of the log record. Date format can be decided using "datefmt" >>> field in formatter as shown above (I have defaulted it to UTC for >>> testing) >>> 4. Individual modules can decide what all handlers defined to be >>> used while logging >>> 5. With RotationFileHandler we can set maximum no of bytes allowed >>> in log file, backupCount to decide on how many backups to be kept. >>> Havent tried time based rotation at the moment though ! >>> >>> With this I have tried rotational, file based, syslog based and >>> journald based logging and find it working. >>> Just wanted a heads up on this, if this meets kind of our >>> expectation for tendrl. If so I can raise PR with changes soon :) >>> >>> Thanks and Regards, >>> Shubhendu >>> >>> On 10/20/2016 04:43 PM, Jeff Applewhite wrote: >>>> Hi All >>>> >>>> Here is a snip from an email with Peter Portante regarding best >>>> practices >>>> on logging: --Thanks Peter >>>> >>>> this pertains to ten-67 in Jira >>>> >>>> "Hi Jeff, >>>> >>>> Before all else, ensure the time stamps on log entires are either UTC >>>> or always include an accurate timezone. And ensure the environment in >>>> which the app or system runs has an accurate notion of time. Biggest >>>> problem to make sure this is solved properly. >>>> >>>> The next is how logs are emitted by an application. Using JSON is one >>>> of the best ways to get logs emitted with rich metadata that enables >>>> further use when they are analyzed and correlated to other logs, >>>> metrics, and configuration data. If one cannot use JSON, is to make >>>> sure the logs that are emitted are uniform, easily described, and >>>> machine parse-able. >>>> >>>> The next is ensure that the metadata is named following that namespace >>>> document you reference. There is a project on GitHub called ViaQ >>>> where we work with practical delivery of documented namespaces, see >>>> https://github.com/ViaQ/elasticsearch-templates/ >>>> >>>> Lastly, visualize early and often. =) That is, work with an >>>> environment like Elasticsearch & Kibana to ensure when you emit for >>>> logs can be placed into a dashboard that shows in aggregate the >>>> various pieces of metadata associated with the logs, and that those >>>> visualizations are useful. >>>> ?"? >>>> >>> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From japplewh at redhat.com Fri Oct 21 19:57:03 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Fri, 21 Oct 2016 15:57:03 -0400 Subject: [Tendrl-devel] logging In-Reply-To: <416aad4f-7888-6b2d-7c74-010b581a7837@redhat.com> References: <11c74ecd-e5cf-e72a-8cd5-92612d9689f4@redhat.com> <416aad4f-7888-6b2d-7c74-010b581a7837@redhat.com> Message-ID: Thanks all - I have updated the acceptance criteria. see https://tendrl.atlassian.net/browse/TEN-67 If there is any disagreement or gap between what is possible/practical please highlight. It looks like maybe I have more logging levels - any problem with more including DEBUG, INFO, WARN, ERROR and FATAL? On Fri, Oct 21, 2016 at 8:25 AM, Shubhendu Tripathi wrote: > Thanks Aravinda for guidelines. I appreciate this. > Yes for sure this could be an option for handlers. All we need is making > it configurable and even support for syslog/journald etc should be easy to > configure. > > Regards, > Shubhendu > > > > On 10/21/2016 05:36 PM, Aravinda wrote: > > In Gluster Geo-replication we are using "WatchedFileHandler", so that > running application need not worry if the log files are rotated externally > using `logrotate` tools. > https://docs.python.org/2/library/logging.handlers.html > > regards > Aravinda > > On Friday 21 October 2016 05:05 PM, Shubhendu Tripathi wrote: > > Tried TimedRotatingFileHandler as well and works fine. > Different possible intervals could be Seconds, Minutes, Hours, Days, > Weekdays (0-6), 'midnight'. > > Regards, > Shubhendu > > > On 10/21/2016 03:56 PM, Shubhendu Tripathi wrote: > > Hi Jeff, > > Today I have tried out few things around logging framework in tendrl. The > summary of the same goes as below > > ------------------------------------------------- > 1. Added a configuration /etc/tendrl/logging.json (can be ported as yaml > as well) with below content > > { > "version": 1, > "disable_existing_loggers": false, > "formatters": { > "simple": { > "format": "%(asctime)s - %(name)s - %(levelname)s - > %(message)s", > "datefmt": "%Y-%m-%dT%H:%M:%S%z" > } > }, > > "handlers": { > "console": { > "class": "logging.StreamHandler", > "level": "DEBUG", > "formatter": "simple", > "stream": "ext://sys.stdout" > }, > > "info_file_handler": { > "class": "logging.handlers.SysLogHandler", > "level": "INFO", > "formatter": "simple" > }, > > "error_file_handler": { > "class": "logging.handlers.RotatingFileHandler", > "level": "ERROR", > "formatter": "simple", > "filename": "/var/log/tendrl/errors.log", > "maxBytes": 10485760, > "backupCount": 20, > "encoding": "utf8" > } > }, > > "loggers": { > "my_module": { > "level": "ERROR", > "handlers": ["console"], > "propagate": "no" > } > }, > > "root": { > "level": "INFO", > "handlers": ["console", "info_file_handler", "error_file_handler"] > } > } > > 2. Did small changes in tendrl gluster bridge logger to start logger with > above configuration. > > ----------------------------------- > > Below are few facts figured out as part of this exercise - > > 1. There are different handlers like StreamHandler, SysLogHandler, > RotatingFileHandler, FileHandler and JournalHandler (JournalHandler is > provided by systemd-python package and not available with default python > logging framework as such) which can be configured as log handler above > 2. Different handlers can be configured for console, info and error files. > So we can decide something like - all the "errors" are logged to > journald/syslog and "info" messages to a rotating file. > 3. There are formatters which could be configured to define the format of > the log record. Date format can be decided using "datefmt" field in > formatter as shown above (I have defaulted it to UTC for testing) > 4. Individual modules can decide what all handlers defined to be used > while logging > 5. With RotationFileHandler we can set maximum no of bytes allowed in log > file, backupCount to decide on how many backups to be kept. Havent tried > time based rotation at the moment though ! > > With this I have tried rotational, file based, syslog based and journald > based logging and find it working. > Just wanted a heads up on this, if this meets kind of our expectation for > tendrl. If so I can raise PR with changes soon :) > > Thanks and Regards, > Shubhendu > > On 10/20/2016 04:43 PM, Jeff Applewhite wrote: > > Hi All > > Here is a snip from an email with Peter Portante regarding best practices > on logging: --Thanks Peter > > this pertains to ten-67 in Jira > > "Hi Jeff, > > Before all else, ensure the time stamps on log entires are either UTC > or always include an accurate timezone. And ensure the environment in > which the app or system runs has an accurate notion of time. Biggest > problem to make sure this is solved properly. > > The next is how logs are emitted by an application. Using JSON is one > of the best ways to get logs emitted with rich metadata that enables > further use when they are analyzed and correlated to other logs, > metrics, and configuration data. If one cannot use JSON, is to make > sure the logs that are emitted are uniform, easily described, and > machine parse-able. > > The next is ensure that the metadata is named following that namespace > document you reference. There is a project on GitHub called ViaQ > where we work with practical delivery of documented namespaces, see > https://github.com/ViaQ/elasticsearch-templates/ > > Lastly, visualize early and often. =) That is, work with an > environment like Elasticsearch & Kibana to ensure when you emit for > logs can be placed into a dashboard that shows in aggregate the > various pieces of metadata associated with the logs, and that those > visualizations are useful. > ?"? > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > -- Jeff Applewhite Principal Product Manager From kdreyer at redhat.com Fri Oct 21 20:57:42 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Fri, 21 Oct 2016 14:57:42 -0600 Subject: [Tendrl-devel] Tendrl and ceph-installer in CentOS SIG's Koji In-Reply-To: References: Message-ID: Thanks Nishanth. I was not planning to create the .spec files for Tendrl at this time. Just to be absolutely sure: we are saying this is "version 1" of Tendrl? - Ken On Thu, Oct 20, 2016 at 9:02 PM, Nishanth Thomas wrote: > Hi Ken, > > Looks good to me. > Are you planing to work on writing the spec files for tendrl packages? > > Thanks, > Nishanth > > On Fri, Oct 21, 2016 at 6:32 AM, Ken Dreyer wrote: > >> Hi folks, >> >> I'd like to get Tendrl and ceph-installer packaged in the CentOS Storage >> SIG. >> >> Looking at the list of Koji tags they've set up previously [1], I >> think it would be appropriate to set up a "tendrl-1" tag. (Assuming >> our first version of Tendrl is going to be "1"). >> >> storage7-tendrl-1-{candidate,testing,release} >> storage7-tendrl-1-el7-build >> >> Any objections? >> >> - Ken >> >> https://wiki.centos.org/SpecialInterestGroup/Storage/sigworkflow >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mcarrano at redhat.com Fri Oct 21 21:37:37 2016 From: mcarrano at redhat.com (Matt Carrano) Date: Fri, 21 Oct 2016 17:37:37 -0400 Subject: [Tendrl-devel] Preliminary UI design published for Tendrl 1 Message-ID: We are publishing several UI designs/flows to enable UI development work on the Tendrl 1 release currently underway. There are all in InVision and links are given below. Please feel free to comment directly in InVision using the commenting tool. For more information about commenting in InVision see [1]. Please note that all of these designs are still works in progress. We are publishing them early in order to give development a head start on implementation to meet Tendrl 1 goals. Further refinements are likely in the future. I will schedule a review next week where Ju and I can answer questions. List Views (including Clusters, Hosts, Pools, RBDs, and File Shares): https://redhat.invisionapp.com/share/BR8JDCGSQ First Use Experience (initial landing page leading to first-time cluster creation): https://redhat.invisionapp.com/share/Q78YMAVDJ Import Cluster (updated to support SSH keys): https://redhat.invisionapp.com/share/R88EUSGJK Add File Share (Volume creation wizard): https://redhat.invisionapp.com/share/Q78YMAVDJ Let me know if you have any questions or problems accessing these files. Regards, Matt [1] https://support.invisionapp.com/hc/en-us/articles/209192426-How-do-I-comment-on-a-prototype- -- Matt Carrano Sr. Interaction Designer Red Hat, Inc. mcarrano at redhat.com From japplewh at redhat.com Sat Oct 22 00:48:37 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Fri, 21 Oct 2016 20:48:37 -0400 Subject: [Tendrl-devel] Tendrl and ceph-installer in CentOS SIG's Koji In-Reply-To: References: Message-ID: On Friday, October 21, 2016, Ken Dreyer wrote: > Thanks Nishanth. I was not planning to create the .spec files for > Tendrl at this time. > > Just to be absolutely sure: we are saying this is "version 1" of Tendrl? Yes November Tendrl 1 Jan Tendrl 2 March Tendrl 3 > > - Ken > > On Thu, Oct 20, 2016 at 9:02 PM, Nishanth Thomas > wrote: > > Hi Ken, > > > > Looks good to me. > > Are you planing to work on writing the spec files for tendrl packages? > > > > Thanks, > > Nishanth > > > > On Fri, Oct 21, 2016 at 6:32 AM, Ken Dreyer > wrote: > > > >> Hi folks, > >> > >> I'd like to get Tendrl and ceph-installer packaged in the CentOS Storage > >> SIG. > >> > >> Looking at the list of Koji tags they've set up previously [1], I > >> think it would be appropriate to set up a "tendrl-1" tag. (Assuming > >> our first version of Tendrl is going to be "1"). > >> > >> storage7-tendrl-1-{candidate,testing,release} > >> storage7-tendrl-1-el7-build > >> > >> Any objections? > >> > >> - Ken > >> > >> https://wiki.centos.org/SpecialInterestGroup/Storage/sigworkflow > >> > >> _______________________________________________ > >> Tendrl-devel mailing list > >> Tendrl-devel at redhat.com > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > >> > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From shtripat at redhat.com Mon Oct 24 06:45:30 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Mon, 24 Oct 2016 12:15:30 +0530 Subject: [Tendrl-devel] logging In-Reply-To: References: <11c74ecd-e5cf-e72a-8cd5-92612d9689f4@redhat.com> <416aad4f-7888-6b2d-7c74-010b581a7837@redhat.com> Message-ID: Jeff, The framework here is default logging available with standard python and it does support DEBUG, INFO, WARN, ERROR and CRITICAL. The log level CRITICAL could be actually mapped as FATAL mentioned by you I feel. The above ones are the supported ones at the moment by framework. I also wanted to use this space to discuss the default handlers to be used in tendrl. We have options like /Rotational, TimedRotational, SysLog, JournalD/ and /Watched/ as popularly used ones. Looks like gluster geo-replication uses /Watched/ one as it provides more flexibility for rotation options and it can use external logrotateD configurations to rotate the log files. @Aravinda can provide more insight on this :) Request suggestions from people here to decide the default log handlers for tendrl. Refer PRs [1] and [2] raised for gluster and ceph tendrl bridges._ _ Thanks and Regards, Shubhendu [1] https://github.com/Tendrl/gluster_bridge/pull/32 [2] https://github.com/Tendrl/ceph_bridge/pull/16 _Note: Setting handlers is change in configuration and does not have any impact on source code. _ On 10/22/2016 01:27 AM, Jeff Applewhite wrote: > Thanks all - I have updated the acceptance criteria. > > see https://tendrl.atlassian.net/browse/TEN-67 > > If there is any disagreement or gap between what is > possible/practical please highlight. > > It looks like maybe I have more logging levels - any problem with > more including DEBUG, INFO, WARN, ERROR and FATAL? > > On Fri, Oct 21, 2016 at 8:25 AM, Shubhendu Tripathi > > wrote: > > Thanks Aravinda for guidelines. I appreciate this. > Yes for sure this could be an option for handlers. All we need is > making it configurable and even support for syslog/journald etc > should be easy to configure. > > Regards, > Shubhendu > > > > On 10/21/2016 05:36 PM, Aravinda wrote: >> In Gluster Geo-replication we are using "WatchedFileHandler", so >> that running application need not worry if the log files are >> rotated externally using `logrotate` tools. >> https://docs.python.org/2/library/logging.handlers.html >> >> >> regards >> Aravinda >> >> On Friday 21 October 2016 05:05 PM, Shubhendu Tripathi wrote: >>> Tried TimedRotatingFileHandler as well and works fine. >>> Different possible intervals could be Seconds, Minutes, Hours, >>> Days, Weekdays (0-6), 'midnight'. >>> >>> Regards, >>> Shubhendu >>> >>> >>> On 10/21/2016 03:56 PM, Shubhendu Tripathi wrote: >>>> Hi Jeff, >>>> >>>> Today I have tried out few things around logging framework in >>>> tendrl. The summary of the same goes as below >>>> >>>> ------------------------------------------------- >>>> 1. Added a configuration /etc/tendrl/logging.json (can be >>>> ported as yaml as well) with below content >>>> >>>> { >>>> "version": 1, >>>> "disable_existing_loggers": false, >>>> "formatters": { >>>> "simple": { >>>> "format": "%(asctime)s - %(name)s - %(levelname)s - >>>> %(message)s", >>>> "datefmt": "%Y-%m-%dT%H:%M:%S%z" >>>> } >>>> }, >>>> >>>> "handlers": { >>>> "console": { >>>> "class": "logging.StreamHandler", >>>> "level": "DEBUG", >>>> "formatter": "simple", >>>> "stream": "ext://sys.stdout" >>>> }, >>>> >>>> "info_file_handler": { >>>> "class": "logging.handlers.SysLogHandler", >>>> "level": "INFO", >>>> "formatter": "simple" >>>> }, >>>> >>>> "error_file_handler": { >>>> "class": "logging.handlers.RotatingFileHandler", >>>> "level": "ERROR", >>>> "formatter": "simple", >>>> "filename": "/var/log/tendrl/errors.log", >>>> "maxBytes": 10485760, >>>> "backupCount": 20, >>>> "encoding": "utf8" >>>> } >>>> }, >>>> >>>> "loggers": { >>>> "my_module": { >>>> "level": "ERROR", >>>> "handlers": ["console"], >>>> "propagate": "no" >>>> } >>>> }, >>>> >>>> "root": { >>>> "level": "INFO", >>>> "handlers": ["console", "info_file_handler", >>>> "error_file_handler"] >>>> } >>>> } >>>> >>>> 2. Did small changes in tendrl gluster bridge logger to start >>>> logger with above configuration. >>>> >>>> ----------------------------------- >>>> >>>> Below are few facts figured out as part of this exercise - >>>> >>>> 1. There are different handlers like StreamHandler, >>>> SysLogHandler, RotatingFileHandler, FileHandler and >>>> JournalHandler (JournalHandler is provided by systemd-python >>>> package and not available with default python logging framework >>>> as such) which can be configured as log handler above >>>> 2. Different handlers can be configured for console, info and >>>> error files. So we can decide something like - all the "errors" >>>> are logged to journald/syslog and "info" messages to a rotating >>>> file. >>>> 3. There are formatters which could be configured to define the >>>> format of the log record. Date format can be decided using >>>> "datefmt" field in formatter as shown above (I have defaulted >>>> it to UTC for testing) >>>> 4. Individual modules can decide what all handlers defined to >>>> be used while logging >>>> 5. With RotationFileHandler we can set maximum no of bytes >>>> allowed in log file, backupCount to decide on how many backups >>>> to be kept. Havent tried time based rotation at the moment >>>> though ! >>>> >>>> With this I have tried rotational, file based, syslog based and >>>> journald based logging and find it working. >>>> Just wanted a heads up on this, if this meets kind of our >>>> expectation for tendrl. If so I can raise PR with changes soon :) >>>> >>>> Thanks and Regards, >>>> Shubhendu >>>> >>>> On 10/20/2016 04:43 PM, Jeff Applewhite wrote: >>>>> Hi All >>>>> >>>>> Here is a snip from an email with Peter Portante regarding >>>>> best practices >>>>> on logging: --Thanks Peter >>>>> >>>>> this pertains to ten-67 in Jira >>>>> >>>>> "Hi Jeff, >>>>> >>>>> Before all else, ensure the time stamps on log entires are >>>>> either UTC >>>>> or always include an accurate timezone. And ensure the >>>>> environment in >>>>> which the app or system runs has an accurate notion of time. >>>>> Biggest >>>>> problem to make sure this is solved properly. >>>>> >>>>> The next is how logs are emitted by an application. Using >>>>> JSON is one >>>>> of the best ways to get logs emitted with rich metadata that >>>>> enables >>>>> further use when they are analyzed and correlated to other logs, >>>>> metrics, and configuration data. If one cannot use JSON, is to >>>>> make >>>>> sure the logs that are emitted are uniform, easily described, and >>>>> machine parse-able. >>>>> >>>>> The next is ensure that the metadata is named following that >>>>> namespace >>>>> document you reference. There is a project on GitHub called ViaQ >>>>> where we work with practical delivery of documented >>>>> namespaces, see >>>>> https://github.com/ViaQ/elasticsearch-templates/ >>>>> >>>>> >>>>> Lastly, visualize early and often. =) That is, work with an >>>>> environment like Elasticsearch & Kibana to ensure when you >>>>> emit for >>>>> logs can be placed into a dashboard that shows in aggregate the >>>>> various pieces of metadata associated with the logs, and that >>>>> those >>>>> visualizations are useful. >>>>> ?"? >>>>> >>>> >>> >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > > -- > Jeff Applewhite > Principal Product Manager > From negupta at redhat.com Mon Oct 24 10:44:33 2016 From: negupta at redhat.com (Neha Gupta) Date: Mon, 24 Oct 2016 06:44:33 -0400 (EDT) Subject: [Tendrl-devel] Recording - demo of UI dynamic generation for create volume workflow In-Reply-To: <1907330076.19175419.1477304408762.JavaMail.zimbra@redhat.com> Message-ID: <1268550853.19185166.1477305873570.JavaMail.zimbra@redhat.com> Hi Team, The recording for Tendrl UI Discussion's meeting can be found at - https://bluejeans.com/s/38i at i/. It includes the demo of dynamic generation of UI for create volume workflow with basic UI design. Thanks, Neha Gupta From japplewh at redhat.com Mon Oct 24 15:49:17 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Mon, 24 Oct 2016 11:49:17 -0400 Subject: [Tendrl-devel] logging In-Reply-To: References: <11c74ecd-e5cf-e72a-8cd5-92612d9689f4@redhat.com> <416aad4f-7888-6b2d-7c74-010b581a7837@redhat.com> Message-ID: On Mon, Oct 24, 2016 at 2:45 AM, Shubhendu Tripathi wrote: > Jeff, > > The framework here is default logging available with standard python and > it does support DEBUG, INFO, WARN, ERROR and CRITICAL. > The log level CRITICAL could be actually mapped as FATAL mentioned by you > I feel. The above ones are the supported ones at the moment by framework. > ?No let's go with the default handler's schema for this. I don't want to create extra work. I will update the acceptance criteria. ? > > I also wanted to use this space to discuss the default handlers to be used > in tendrl. > We have options like *Rotational, TimedRotational, SysLog, JournalD* and > *Watched* as popularly used ones. > > Looks like gluster geo-replication uses *Watched* one as it provides more > flexibility for rotation options and it can use external logrotateD > configurations to rotate the log files. > @Aravinda can provide more insight on this :) > > Request suggestions from people here to decide the default log handlers > for tendrl. > > Refer PRs [1] and [2] raised for gluster and ceph tendrl bridges. > > Thanks and Regards, > Shubhendu > > [1] https://github.com/Tendrl/gluster_bridge/pull/32 > [2] https://github.com/Tendrl/ceph_bridge/pull/16 > > *Note: Setting handlers is change in configuration and does not have any > impact on source code. * > > > On 10/22/2016 01:27 AM, Jeff Applewhite wrote: > > Thanks all - I have updated the acceptance criteria. > > see https://tendrl.atlassian.net/browse/TEN-67 > > If there is any disagreement or gap between what is possible/practical > please highlight. > > It looks like maybe I have more logging levels - any problem with more > including DEBUG, INFO, WARN, ERROR and FATAL? > > On Fri, Oct 21, 2016 at 8:25 AM, Shubhendu Tripathi > wrote: > >> Thanks Aravinda for guidelines. I appreciate this. >> Yes for sure this could be an option for handlers. All we need is making >> it configurable and even support for syslog/journald etc should be easy to >> configure. >> >> Regards, >> Shubhendu >> >> >> >> On 10/21/2016 05:36 PM, Aravinda wrote: >> >> In Gluster Geo-replication we are using "WatchedFileHandler", so that >> running application need not worry if the log files are rotated externally >> using `logrotate` tools. >> https://docs.python.org/2/library/logging.handlers.html >> >> regards >> Aravinda >> >> On Friday 21 October 2016 05:05 PM, Shubhendu Tripathi wrote: >> >> Tried TimedRotatingFileHandler as well and works fine. >> Different possible intervals could be Seconds, Minutes, Hours, Days, >> Weekdays (0-6), 'midnight'. >> >> Regards, >> Shubhendu >> >> >> On 10/21/2016 03:56 PM, Shubhendu Tripathi wrote: >> >> Hi Jeff, >> >> Today I have tried out few things around logging framework in tendrl. The >> summary of the same goes as below >> >> ------------------------------------------------- >> 1. Added a configuration /etc/tendrl/logging.json (can be ported as yaml >> as well) with below content >> >> { >> "version": 1, >> "disable_existing_loggers": false, >> "formatters": { >> "simple": { >> "format": "%(asctime)s - %(name)s - %(levelname)s - >> %(message)s", >> "datefmt": "%Y-%m-%dT%H:%M:%S%z" >> } >> }, >> >> "handlers": { >> "console": { >> "class": "logging.StreamHandler", >> "level": "DEBUG", >> "formatter": "simple", >> "stream": "ext://sys.stdout" >> }, >> >> "info_file_handler": { >> "class": "logging.handlers.SysLogHandler", >> "level": "INFO", >> "formatter": "simple" >> }, >> >> "error_file_handler": { >> "class": "logging.handlers.RotatingFileHandler", >> "level": "ERROR", >> "formatter": "simple", >> "filename": "/var/log/tendrl/errors.log", >> "maxBytes": 10485760, >> "backupCount": 20, >> "encoding": "utf8" >> } >> }, >> >> "loggers": { >> "my_module": { >> "level": "ERROR", >> "handlers": ["console"], >> "propagate": "no" >> } >> }, >> >> "root": { >> "level": "INFO", >> "handlers": ["console", "info_file_handler", >> "error_file_handler"] >> } >> } >> >> 2. Did small changes in tendrl gluster bridge logger to start logger with >> above configuration. >> >> ----------------------------------- >> >> Below are few facts figured out as part of this exercise - >> >> 1. There are different handlers like StreamHandler, SysLogHandler, >> RotatingFileHandler, FileHandler and JournalHandler (JournalHandler is >> provided by systemd-python package and not available with default python >> logging framework as such) which can be configured as log handler above >> 2. Different handlers can be configured for console, info and error >> files. So we can decide something like - all the "errors" are logged to >> journald/syslog and "info" messages to a rotating file. >> 3. There are formatters which could be configured to define the format of >> the log record. Date format can be decided using "datefmt" field in >> formatter as shown above (I have defaulted it to UTC for testing) >> 4. Individual modules can decide what all handlers defined to be used >> while logging >> 5. With RotationFileHandler we can set maximum no of bytes allowed in log >> file, backupCount to decide on how many backups to be kept. Havent tried >> time based rotation at the moment though ! >> >> With this I have tried rotational, file based, syslog based and journald >> based logging and find it working. >> Just wanted a heads up on this, if this meets kind of our expectation for >> tendrl. If so I can raise PR with changes soon :) >> >> Thanks and Regards, >> Shubhendu >> >> On 10/20/2016 04:43 PM, Jeff Applewhite wrote: >> >> Hi All >> >> Here is a snip from an email with Peter Portante regarding best practices >> on logging: --Thanks Peter >> >> this pertains to ten-67 in Jira >> >> "Hi Jeff, >> >> Before all else, ensure the time stamps on log entires are either UTC >> or always include an accurate timezone. And ensure the environment in >> which the app or system runs has an accurate notion of time. Biggest >> problem to make sure this is solved properly. >> >> The next is how logs are emitted by an application. Using JSON is one >> of the best ways to get logs emitted with rich metadata that enables >> further use when they are analyzed and correlated to other logs, >> metrics, and configuration data. If one cannot use JSON, is to make >> sure the logs that are emitted are uniform, easily described, and >> machine parse-able. >> >> The next is ensure that the metadata is named following that namespace >> document you reference. There is a project on GitHub called ViaQ >> where we work with practical delivery of documented namespaces, see >> https://github.com/ViaQ/elasticsearch-templates/ >> >> Lastly, visualize early and often. =) That is, work with an >> environment like Elasticsearch & Kibana to ensure when you emit for >> logs can be placed into a dashboard that shows in aggregate the >> various pieces of metadata associated with the logs, and that those >> visualizations are useful. >> ?"? >> >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> >> > > > -- > Jeff Applewhite > Principal Product Manager > > > -- Jeff Applewhite Principal Product Manager From japplewh at redhat.com Mon Oct 24 15:52:42 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Mon, 24 Oct 2016 11:52:42 -0400 Subject: [Tendrl-devel] Architecture meeting recording from today Message-ID: Containerization as per OpenStack integration, installation, Heketi, among the topics discussed. https://bluejeans.com/s/wV4Pa/ -- Jeff Applewhite Principal Product Manager From rkanade at redhat.com Tue Oct 25 02:33:18 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Tue, 25 Oct 2016 08:03:18 +0530 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: Any updates on this Gregory? On 20-Oct-2016 17:57, "Rohan Kanade" wrote: > Lets start working on what we have mutually agreed upon till now. > > - Continue the sync_object API in place (http://calamari.readthedocs.i > o/en/latest/calamari_rest/resources/api_example_api_v2_clust > er__fsid__sync_object__sync_type_.html) > - get rid of salt dependency completely > - provide single REST API endpoint (Tendrl does not need to worry on what > mons calamari runs this API) consumable by Tendrl and ensure scenario like > below works as described > > eg: Assuming that both below actions are taken at the same time. > 1. GET api/v2/cluster//pool should return same list of pools as > step 2. > 2 ceph osd lspools > > - Add support for RBD > - In a pre-defined environment, calamari should be stress tested for a > refresh interval of 2-3 seconds per API call. We would need to add this to > calamari QE test cases maybe. > > These are the minimum requirements Tendrl has from calamari. If we are > agreeing on these, we can file detailed BZ's. > > On Thu, Oct 20, 2016 at 1:22 AM, Gregory Meno wrote: > >> On Thu, Oct 13, 2016 at 4:18 AM, Rohan Kanade wrote: >> > >> > >> > On Thu, Oct 6, 2016 at 7:26 PM, Gregory Meno wrote: >> >> >> >> On Wed, Oct 5, 2016 at 5:57 AM, Rohan Kanade >> wrote: >> >> > Calamari is an important component in the Ceph management/ops >> process. >> >> > In >> >> > the past, The Skyrings [0] project was able to utilize Calamari via >> the >> >> > Calamari REST API, the integration did ran into some problems as >> >> > mentioned >> >> > below. >> >> > >> >> > - Calamari-server would often be a single point of failure as it is >> >> > installed on the ceph-mon nodes >> >> >> >> Consider that monitor nodes are the right place to run and there are >> >> usually 3 or 5 monitors. >> >> Is that enough? I think we are at a point now where this would be >> >> reasonable to run on each monitor. >> >> What is more interesting to me is how you see the interaction >> happening. >> >> >> >> Does each process instance track the mon leader and proxy requests to >> >> the instance that is on the leader? >> > >> > Tendrl calamari interaction will happen via calamari REST API, Tendrl >> wont >> > be tracking calamari processes in any part of the cluster >> > >> >> John: have we considered what the ceph-mgr answer to this is? >> >> >> >> Do you expect to have perfect replication of state related to requests >> >> to change the ceph cluster? >> > >> > eg: If we call GET /calamari/osds , it should provide a list of osds up >> > until that point in time >> > >> >> How do you want to discover available endpoints? >> > >> > On a new cluster or an existing cluster, Tendrl will assume calamari is >> > pre-installed and functional (this will be part of Tendrl >> pre-requisites) >> > Then, while installing Tendrl itself, it should be provided with the >> > calamari REST API endpoint either via conf files or user input. >> > >> >> >> >> > - Querying Calamari REST API gave inconsistent data about the ceph >> >> > cluster >> >> > state, this might be happening because of sync issues occurring due >> to >> >> > ceph-mons not being in quorum etc. >> >> >> >> Specific tickets would go a long way here. >> >> >> >> > - Calamari required to be installed on only one of the ceph-mons, >> This >> >> > introduces very common scalability issues. >> >> >> >> I think we can relax this requirement if we can agree on expectations >> >> above. >> >> >> >> >> >> > - Calamari relies on older version of saltstack for messaging and >> other >> >> > communication, this introduces packaging and performance overheads >> >> >> >> This is not the case. We only use salt to send events to tendrl and >> >> it's something I consider to be a mistake. >> >> I think salt can be taken out of the picture in future releases >> > >> > Ok, removal of salt works for Tendrl >> > >> >> >> >> > >> >> > These issues must be re-verified and fixed if they still persist in >> >> > Calamari. >> >> > >> >> > >> >> > >> >> > >> >> > The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried >> to >> >> > address these issues by: >> >> >> >> Did they succeed? >> > >> > Yes >> >> >> >> > >> >> > - Distributing the tendrl ceph bridge to talk to all ceph-mons in the >> >> > cluster. >> >> > - Gathering data (3 second poll interval) from ceph-mons only when >> they >> >> > are >> >> > in quorum and directly talking to ceph-mons via librados and using >> ceph >> >> > admin sockets to query valid cluster maps and other ceph data. >> >> >> >> Since ceph_bridge is taken directly from calamari it's fair to say >> >> that we posses the same capabilities >> > >> > Please check the forked repository for changes in capabilities >> > >> >> >> >> > - Tendrl ceph bridge tries to address the single point of failure >> issue >> >> > by >> >> > having multiple instances of the bridge for each ceph-mon in the >> >> > cluster. >> >> >> >> Would you please show me where ceph_bridge is aware of other >> >> instances? Or are these multiple copies sharing nothing? >> > >> > For READ of cluster state, we only pull data out of the cluster if all >> mons >> > are in quorum. We have a single copy of the state which is refreshed at >> > regular intervals as well as per specific events. >> > >> >> >> >> > - Tendrl ceph bridge has centralized communications via an etcd >> cluster >> >> > which removes need to maintain saltstack . >> >> > >> >> > >> >> > >> >> > Moving forward, we would like Calamari to accommodate these >> improvements >> >> > back into the calamari code base, Below are the detailed >> requirements: >> >> > >> >> > - Need Calamari REST API for querying ceph admin sockets [2], More >> >> > details: >> >> >> >> Let's not expose this. I would consider it to be a challenge to your >> >> goal of having multiple instances >> >> as you're talking to a specific socket. Really what you get is info >> >> about clusters that are being setup or severly damaged this should be >> >> built into whatever service is responsible for identifying clusters >> >> and talking to them e.g. calamari and in future ceph-mgr. >> > >> > This is required specifically to verify correctness of cluster state >> > reported by other instances. And as i mentioned we only pull data from >> > sockets if all mons are in quorum. >> > >> >> >> >> > - Need Calamari REST API for providing all cluster maps (monitor, >> osd, >> >> > crush, mds, pg, health, config etc) >> >> >> >> This seems like a strange request. Why do you want access to the raw >> >> data? You'll have to parse it to make sense of it the same way we >> >> already do in calamari and in future ceph-mgr. >> > >> > Calamari already provides this via REST API >> > http://calamari.readthedocs.io/en/latest/calamari_rest/resou >> rces/api_example_api_v2_cluster__fsid__sync_object.html >> > Tendrl is simply asking this to be supported in future. >> > >> >> >> >> > - Calamari REST API must represent the ceph cluster state >> correctly, no >> >> > mismatch should occur between the True state of the cluster and the >> >> > state >> >> > represented by the Calamari REST API >> >> >> >> Happy to work to fix specific dependencies. Tickets are best. >> >> >> >> > - Calamari REST API should provide access to monitoring actions like >> >> > "ceph >> >> > -w", "ceph tell" etc >> >> >> >> Help me understand the specifics of what you want to accomplish here. >> >> the output of ceph -w can be synthesized from looking at the cluster >> >> maps. and ceph tell is typically used for troubleshooting specific >> >> instances where you want to set new config with out changing ceph.conf >> >> and restarting. That means it applies to a single instance, not >> >> cluster wide. >> > >> > ceph -w will be used to generate events based on the output. Computing >> such >> > events from cluster maps is doable too but is more work. >> > Tendrl works not only on a cluster context but also at a node context, >> ceph >> > tell is required for operations which pertain to specific nodes. >> > >> >> >> >> > - Calamari REST API should provide access to operational actions via >> >> > tools >> >> > like "rbd" etc. >> >> >> >> Agreed. >> >> >> >> > - Calamari REST API should be able to service and perform well on 2-4 >> >> > second >> >> > polling intervals on the REST API >> >> >> >> Yes I agree here too. >> >> >> >> > >> >> > >> >> > These requirements can be run through existing calamari functional >> and >> >> > performance test cases and we should make additions to these test >> cases >> >> > to >> >> > accommodate above requirements. >> >> > >> >> > These requirements would be further expanded into individual BZs or >> >> > tracking >> >> > items in Calamari >> >> > >> >> > >> >> > [0]: https://github.com/skyrings >> >> > [1]: https://github.com/tendrl/ceph_bridge >> >> > [2]: >> >> > >> >> > http://docs.ceph.com/docs/jewel/rados/operations/monitoring/ >> #using-the-admin-socket >> >> >> I've read through this thread again and I'm trying to capture all the >> items where we agree on the next steps: >> * add support for RBD >> * get rid of salt >> * run on all monitor nodes With eventually consistent results for READ >> operations >> * leave the sync_object API in place >> >> Items where I need more detail to agree to: >> * access to admin sockets to "verify correctness of cluster state". >> Can we not trust the mon_map? If we cannot it needs either better docs >> or bug fixes. I think this state needs to be exposed correctly by >> ceph. Does that make sense? >> * perform well. Would you please file specific tickets for what >> endpoints are not responding promptly or causing high load? >> >> >> Items I don't see implementing before ceph-mgr transition and maybe not >> then: >> * access to ceph -w and ceph tell >> Ceph -w seems like a poor source of events that would require >> long-polling or eventing to surface, and if we did you'd be scraping a >> data-source that may change. I'm not saying no to eventing, but >> calamari is definatly the wrong place for this to live. >> >> ceph tell is an implementation detail for changing config at runtime. >> I agree that we might want to alter configuration values in rare >> instances. We should either convince Ceph to set better defaults or >> provide write access to a /config endpoint. >> I would say that if you are looking to troubleshoot a cluster and want >> ephemeral config values for specific daemons that we are a long way >> off doing this correctly. >> >> I want to work with you and help Tendrl get what it needs AND I'm >> still trying to figure out just what those specifics are. >> >> cheers, >> G >> > > From rkanade at redhat.com Tue Oct 25 02:48:17 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Tue, 25 Oct 2016 08:18:17 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: Based on the very little inputs for the Hack Days agenda. I propose the following Hack Day pre-requisites (before 8 Nov): 1. Developers should have working knowledge of codebase Tendrl/bridge_common, Tendrl/ceph_bridge, Tendrl/gluster_bridge, Tendrl/node_agent, Tendrl/tendrl repositories on GitHub. 2. Devs should have clear understanding of common pieces like the etcd persistence layer, Tendrl definition yamls, Tendrl and gevent, Tendrl inter component communication via api job queue, Tendrl flows and atoms etc. 3. Have your own Dev and Test Environment handy. Which includes minimum 2 node Ceph and Gluster clusters and with Tendrl components installed and running. Hack Days Agenda 1. Complete current backlog and bring it up to the Definition of done with priority on the Tendrl 1 and Tendrl 2 (see Tendrl mvp for details) 2. Finish packaging tasks. 3. Implement/Deliver all provisioning flows. I request the team to be available throughout the Hack Days. If any change of plans. Please notify in advance. On 19-Oct-2016 17:10, "Jeff Applewhite" wrote: > ?Agreed - let's stick with the original plan? > > On Wed, Oct 19, 2016 at 7:27 AM, Nishanth Thomas > wrote: > > > I think its important to have Rohan here for hack days > > > > On Wed, Oct 19, 2016 at 4:52 PM, Rohan Kanade > wrote: > > > > > Jeff, > > > > > > I wont be available during 30 Oct to 4th Nov (Important Indian Festival > > > time). If other team members are available from 2nd-4th Nov, we can > chalk > > > up the agenda and continue on these dates. > > > > > > On Tue, Oct 18, 2016 at 9:45 PM, Jeff Applewhite > > > wrote: > > > > > > > I think it's a great idea! +1 > > > > > > > > But given that we are releasing the first version of Tendrl on Nov 8 > it > > > > seems like maybe we should target the 2nd-4th of November. > > > > That would give us more time for QE and acceptance of user stories > > > > > > > > Thoughts? > > > > > > > > On Tue, Oct 18, 2016 at 8:28 AM, Mrugesh Karnik < > > mrugesh at brainfunked.org > > > > > > > > wrote: > > > > > > > > > I whole-heartedly second this! > > > > > > > > > > On 18 October 2016 at 17:41, Rohan Kanade > > wrote: > > > > > > > > > > > > Tendrl is going through its first release cycle and we have > covered > > > > some > > > > > > ground w.r.t Basic plumbing, inter component messaging, central > > store > > > > > > (etcd), tendrl API, Ceph/Gluster ops etc. > > > > > > > > > > > > We still need to cover significant ground in these areas before > we > > > can > > > > be > > > > > > ready for our first release > > > > > > > > > > > > I'd like to propose that the Tendrl contributors get together > with > > a > > > > > > flexible agenda for 3 Hack days (8, 9, 10 Nov, Bangalore, India) > > and > > > > > > collaborate on various pending/upcoming items. > > > > > > > > > > > > Dates: 8, 9, 10 Nov 2016 > > > > > > Location: Red Hat, Bangalore, India > > > > > > > > > > > > Please share your ideas for this proposal so we can get a rough > > > agenda > > > > in > > > > > > place soon. > > > > > > > > > > > > Regards, > > > > > > Rohan Kanade > > > > > > _______________________________________________ > > > > > > Tendrl-devel mailing list > > > > > > Tendrl-devel at redhat.com > > > > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > _______________________________________________ > > > > > Tendrl-devel mailing list > > > > > Tendrl-devel at redhat.com > > > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > > > > > > > > > > > > > > > > -- > > > > Jeff Applewhite > > > > Principal Product Manager > > > > _______________________________________________ > > > > Tendrl-devel mailing list > > > > Tendrl-devel at redhat.com > > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > > _______________________________________________ > > > Tendrl-devel mailing list > > > Tendrl-devel at redhat.com > > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > -- > Jeff Applewhite > Principal Product Manager > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From mike.hulsman at proxy.nl Tue Oct 25 06:21:05 2016 From: mike.hulsman at proxy.nl (Mike Hulsman) Date: Tue, 25 Oct 2016 08:21:05 +0200 (CEST) Subject: [Tendrl-devel] Integration with ManageIQ/CloudForms Message-ID: <31782482.848675.1477376465292.JavaMail.zimbra@proxy.nl> Hi, Are there any plans to integrate Tendrl with RedHat project ManageIQ/CloudForms ? ManageIQ/CloudForms is the tool to manage Virtualisation and Cloud environments. Would be nice if we can also manage SDS with ManageIQ by using Tendrl as a provider. I will also ask around in the ManageIQ community if they heard of Tendrl and want to integrate. Kind regards, Mike Hulsman Proxy Managed Services B.V. | www.proxy.nl | Enterprise IT-Infra, Open Source and Cloud Technology Delftweg 128 3043 NB Rotterdam The Netherlands | +31 10 307 0965 From kchidamb at redhat.com Tue Oct 25 07:21:24 2016 From: kchidamb at redhat.com (Karnan Chidambarakani) Date: Tue, 25 Oct 2016 03:21:24 -0400 (EDT) Subject: [Tendrl-devel] queries on dynamic UI generation In-Reply-To: <1268550853.19185166.1477305873570.JavaMail.zimbra@redhat.com> References: <1268550853.19185166.1477305873570.JavaMail.zimbra@redhat.com> Message-ID: <1064340566.4456113.1477380084427.JavaMail.zimbra@redhat.com> Hi Team, My understanding from the demo [demo of UI dynamic generation for create volume workflow] The server is generating Json from Yaml spec, which has the meta information about fields in the UI. Any addition/modification to fields on UI requires changes only on backend. i have few questions on this dynamic generation. - addition/removal of fields might(mostly) requires tweaks on styling/padding on UI.How can we handle this? - How much info about styling will be provided from server? - Does Unit test cases and mock data used for testing have to be updated for every change ? - Will changes to common render view might impact the other pages? - if server is deciding what is to be displayed like php/jsp, where will the business logic reside (server or client)? - Can this be used for generating wizards and dashboards? Thanks, Karnan ----- Original Message ----- From: "Neha Gupta" To: tendrl-devel at redhat.com Sent: Monday, October 24, 2016 4:14:33 PM Subject: [Tendrl-devel] Recording - demo of UI dynamic generation for create volume workflow Hi Team, The recording for Tendrl UI Discussion's meeting can be found at - https://bluejeans.com/s/38i at i/. It includes the demo of dynamic generation of UI for create volume workflow with basic UI design. Thanks, Neha Gupta _______________________________________________ Tendrl-devel mailing list Tendrl-devel at redhat.com https://www.redhat.com/mailman/listinfo/tendrl-devel From deb at redhat.com Tue Oct 25 11:34:09 2016 From: deb at redhat.com (Soumya Deb) Date: Tue, 25 Oct 2016 17:04:09 +0530 Subject: [Tendrl-devel] GitHub access rights In-Reply-To: References: Message-ID: [Top posting to match most replies in the thread - sorry for inconsistency] Agreed, that making a solid project baseline is essential and is more of a priority than defining contribution pathways - but the later doesn't have to be blocked by the former. It can have less priority, but I really hope we're not using the term to euphemize ignoring it. Making some progresses on the project's ecosystem (having a website, defined licenses, access rights etc), in parallel, should be helpful in the long run (if we are to have a long run). I've drafted something in free times, and I am requesting reviews from y'all on this draft: https://github.com/Tendrl/documentation/pull/48 Thanks Sankarshan for already being through the doc & adding thought. On 19 October 2016 at 19:24, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Fri, Oct 14, 2016 at 7:28 PM, Jeff Applewhite > wrote: > > Yes agreed - we should have a plan for how we will accommodate > contributors > > before things get unwieldy. It is fairly monolithic at this point but > > perhaps we can break down the ownership a bit. You sound like the perfect > > person to suggest this plan Soumya. Look forward to hearing it! > > > > Here's how I see this - > > I think it is good to have a baseline proposal on how the organization > would like to have access rights and such in place. It provides a > baseline idea to prospect participants (individuals and organizations) > around this topic. > > I would hesitate to categorize this activity as "urgent" and rather to > me it is the "good to have" which we can pick-up once we have a couple > of releases underway. I am proposing trading the "perfect approach" > against "just good enough to start with". I prefer the latter. > > The path to contributions and participation needs to be complemented > with artefacts to contribute to - the latter is of higher priority. > > > On Fri, Oct 14, 2016 at 8:56 AM, Ju Lim wrote: > > > >> Thanks Soumya. I appreciate you bringing up this point as I too have > >> struggled with how we've set this up. > >> > >> I look forward to seeing your proposal! > >> > >> Regards, > >> Ju > >> > >> On Fri, Oct 14, 2016 at 8:53 AM, Soumya Deb wrote: > >> > >> > Gmail doesn't support enough emoji, but I'd be glad to draft one. > >> > Needed something interesting to do for the weekend anyway. :D > >> > > >> > > >> > On 14 October 2016 at 18:08, Ju Lim wrote: > >> > > >> > > Soumya: > >> > > > >> > > +1 to needing a plan. As I don't believe such a plan exists yet, > would > >> > > you be willing to draft a proposal and share it? > >> > > > >> > > Thanks, > >> > > Ju > >> > > > >> > > On Fri, Oct 14, 2016 at 8:31 AM, Soumya Deb wrote: > >> > > > >> > >> Do we have a plan around who should be having which tier of access > >> > rights > >> > >> on Tendrl? > >> > >> > >> > >> So far it seems very informal & inter-personal (which is okay to > start > >> > >> with), but going forward, we probably should have a properly > defined > >> > plan > >> > >> about requesting access, acceptance criteria, accepting body etc. > at > >> > least > >> > >> at a basic level. > >> > >> > >> > >> Tendrl being an opensource project, I don't see why its core parts > >> > >> (codebase management being one) can't be more federated. > >> > >> > >> > >> --- > >> > >> > >> > >> The reason to have to write this email, is because I'm trying to > get > >> > some > >> > >> service integration done for the Tendrl frontend - which requires > >> > >> admin/owner access to the repo. I'm not sure who all hold this > access > >> > >> right > >> > >> - but at least Sankarshan & Rohan does, that I'm aware of. I had > asked > >> > >> Rohan if he'd be willing to help me out, and he agreed without > >> > hesitation > >> > >> - > >> > >> much appreciated. > >> > >> > >> > >> Now, I am part for 10+ GitHub orgs (admin of 5+) and have done > plenty > >> of > >> > >> GitHub housekeeping in those ones that I'm pretty confident about > >> being > >> > >> able to manage those chores on my own. So, if I were to request > >> access, > >> > >> estimating that I am capable, how do I proceed? Please note, by no > way > >> > I'm > >> > >> saying "I should be" having access; rather, if I, for one, was > >> > interested > >> > >> to share hands (still subject to actually passing the acceptance > >> > criteria) > >> > >> - then how do I proceed? > >> > >> > >> > >> I'm sure just emailing Jeff/Sankarshan about this would've been > easy, > >> > but > >> > >> hey... who wants easy? :D > > > > > -- > sankarshan mukhopadhyay > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From sankarshan.mukhopadhyay at gmail.com Tue Oct 25 12:36:31 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 25 Oct 2016 18:06:31 +0530 Subject: [Tendrl-devel] GitHub access rights In-Reply-To: References: Message-ID: On Tue, Oct 25, 2016 at 5:04 PM, Soumya Deb wrote: > It can have less priority, but I really hope we're not using the term to > euphemize ignoring it. Making some progresses on the project's > ecosystem (having a website, defined licenses, access rights etc), > in parallel, should be helpful in the long run (if we are to have a long > run). It would be proper to re-iterate - crawl, walk, run would help us getting this issue going. The compilation of a structure in the form of a proposal (which we now have) is a good first step. -- sankarshan mukhopadhyay From japplewh at redhat.com Tue Oct 25 14:29:48 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 25 Oct 2016 10:29:48 -0400 Subject: [Tendrl-devel] Integration with ManageIQ/CloudForms In-Reply-To: <31782482.848675.1477376465292.JavaMail.zimbra@proxy.nl> References: <31782482.848675.1477376465292.JavaMail.zimbra@proxy.nl> Message-ID: Hi Mike Yes there are plans and we have had some discussions along these lines. It would make an excellent extension to Tendrl capabilities because ultimately having management of your virtual and cloud infrastructure along with storage has great benefit to operators. Please let us know what you find out from the ManageIQ community. I think the advantage we provide is a single integration point and a management backplane for SDS that means potentially one provider for Ceph, Gluster and future storage technologies. The initial challenge is to get ManageIQ schemas updated to accommodate data from Tendrl APIs related to Ceph and Gluster. Mrugesh and Anup have done some work around this already so perhaps they can share what they have found. Thanks Jeff On Tue, Oct 25, 2016 at 2:21 AM, Mike Hulsman wrote: > Hi, > > Are there any plans to integrate Tendrl with RedHat project > ManageIQ/CloudForms ? > ManageIQ/CloudForms is the tool to manage Virtualisation and Cloud > environments. > > Would be nice if we can also manage SDS with ManageIQ by using Tendrl as a > provider. > I will also ask around in the ManageIQ community if they heard of Tendrl > and want to integrate. > > Kind regards, > > Mike Hulsman > > Proxy Managed Services B.V. | www.proxy.nl | Enterprise IT-Infra, Open > Source and Cloud Technology > Delftweg 128 3043 NB Rotterdam The Netherlands | +31 10 307 0965 > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From mike.hulsman at proxy.nl Tue Oct 25 15:05:53 2016 From: mike.hulsman at proxy.nl (Mike Hulsman) Date: Tue, 25 Oct 2016 17:05:53 +0200 (CEST) Subject: [Tendrl-devel] Integration with ManageIQ/CloudForms In-Reply-To: References: <31782482.848675.1477376465292.JavaMail.zimbra@proxy.nl> Message-ID: <884750544.859338.1477407953214.JavaMail.zimbra@proxy.nl> Hi Jeff, The discussion on the MIQ community I started can be found at http://talk.manageiq.org/t/tendlr-sds-integration/1824/1 Regards, Mike Hulsman Proxy Managed Services B.V. | www.proxy.nl | Enterprise IT-Infra, Open Source and Cloud Technology Delftweg 128 3043 NB Rotterdam The Netherlands | +31 10 307 0965 ----- Original Message ----- > From: "Jeff Applewhite" > To: "tendrl-devel" > Sent: Tuesday, October 25, 2016 4:29:48 PM > Subject: Re: [Tendrl-devel] Integration with ManageIQ/CloudForms > Hi Mike > > Yes there are plans and we have had some discussions along these lines. It > would make an excellent extension to Tendrl capabilities because ultimately > having management of your virtual and cloud infrastructure along with > storage has great benefit to operators. Please let us know what you find > out from the ManageIQ community. I think the advantage we provide is a > single integration point and a management backplane for SDS that means > potentially one provider for Ceph, Gluster and future storage technologies. > > The initial challenge is to get ManageIQ schemas updated to accommodate > data from Tendrl APIs related to Ceph and Gluster. Mrugesh and Anup have > done some work around this already so perhaps they can share what they have > found. > > Thanks > > Jeff > > On Tue, Oct 25, 2016 at 2:21 AM, Mike Hulsman wrote: > >> Hi, >> >> Are there any plans to integrate Tendrl with RedHat project >> ManageIQ/CloudForms ? >> ManageIQ/CloudForms is the tool to manage Virtualisation and Cloud >> environments. >> >> Would be nice if we can also manage SDS with ManageIQ by using Tendrl as a >> provider. >> I will also ask around in the ManageIQ community if they heard of Tendrl >> and want to integrate. >> >> Kind regards, >> >> Mike Hulsman >> >> Proxy Managed Services B.V. | www.proxy.nl | Enterprise IT-Infra, Open >> Source and Cloud Technology >> Delftweg 128 3043 NB Rotterdam The Netherlands | +31 10 307 0965 >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > -- > Jeff Applewhite > Principal Product Manager > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mrugesh at brainfunked.org Tue Oct 25 15:40:35 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 25 Oct 2016 21:10:35 +0530 Subject: [Tendrl-devel] Installing Ceph via Tendrl In-Reply-To: References: Message-ID: I've updated https://github.com/Tendrl/documentation/issues/29 to reference https://github.com/Tendrl/documentation/pull/50 which contains the relevant documentation. On 8 October 2016 at 01:53, Alfredo Deza wrote: > > On Tue, Oct 4, 2016 at 7:08 AM, Alfredo Deza wrote: > > On Tue, Oct 4, 2016 at 3:30 AM, Mrugesh Karnik wrote: > >> On 1 October 2016 at 02:31, Sankarshan Mukhopadhyay < > >> sankarshan.mukhopadhyay at gmail.com> wrote: > >>> > >>> On Sat, Oct 1, 2016 at 12:53 AM, Alfredo Deza wrote: > >>> > All of that will have to be done again in Tendrl. Can that design be > >>> > explained a bit? How is this generalized approach going to tackle > >>> > these issues? How is it going to approach tackling any other Ansible > >>> > playbook or role? > >>> > > >>> > How has the ceph-installer failed here? When the project started, it > >>> > did so with a "documentation first" approach, and we presented the API > >>> > first so that the consumers of it could have an immediate idea on the > >>> > HTTP API proposed. We did it this way because we thought it was > >>> > imperative to present the ideas while the application itself was being > >>> > crafted to improve the throughput on communication. > >>> > >>> It is a bit late/early here - the detail about the provisioning flow > >>> is (as I remember) part of this sprint or, a coming one and is being > >>> worked on. Perhaps we could provide an interim update and then take > >>> this conversation ahead. > >> > >> I'm working on this in the current sprint. Updates will be made to the > >> issue at https://github.com/Tendrl/documentation/issues/29 > >> > >> Also, Alfredo, do you think we can have a quick conversation somewhere > >> towards the end of this week? > > I think it is still important to define and narrow the uncertainty of > my questions in this thread. > > The github issue (https://github.com/Tendrl/documentation/issues/29) > looks empty and we didn't get > to meet this week. > > Please find some time and let me know when that can be to address the > concerns we have. > > > > Of course. Any time! > >> > >> Thanks, > >> Mrugesh > >> _______________________________________________ > >> Tendrl-devel mailing list > >> Tendrl-devel at redhat.com > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mrugesh at brainfunked.org Tue Oct 25 15:43:50 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 25 Oct 2016 21:13:50 +0530 Subject: [Tendrl-devel] Provisioning system design has been posted.. Message-ID: ..in the pull request https://github.com/Tendrl/documentation/pull/50 Rohan will be contributing the YAML definition files for the workflows discussed soon, which will impart more clarity to the description in the document. -- Mrugesh From kdreyer at redhat.com Tue Oct 25 15:58:05 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Tue, 25 Oct 2016 09:58:05 -0600 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: On Mon, Oct 24, 2016 at 8:48 PM, Rohan Kanade wrote: > with Tendrl components installed and running. Would you please point me at the step-by-step instructions for how to do this? (Install all Tendrl components and get them running) - Ken From mbukatov at redhat.com Tue Oct 25 16:18:02 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Tue, 25 Oct 2016 18:18:02 +0200 Subject: [Tendrl-devel] usmqe integration tests for tendrl project Message-ID: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> Dear all, the decision has been made to maintain integration tests for tendrl project upstream. By integration tests I mean tests of the whole solution running on real hardware or virtual machines, checking user scenarios based on requirements/supported use cases. Those tests are created and maintained by our usmqe team. Now the question is where we should keep them. Our team could either create an usmqe group and maintain our qe repositories there or we could ask for creation and full admin access to few qe repositories in tendrl group on github. What do you think make most sense? Having separate qe group would be easier for our team to manage, but I acknowledge the value of having it together with other repositories. -- Martin Bukatovic USM QE team From sankarshan.mukhopadhyay at gmail.com Tue Oct 25 16:24:11 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 25 Oct 2016 21:54:11 +0530 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> Message-ID: On 25-Oct-2016 21:48, "Martin Bukatovic" wrote: > > Dear all, > > the decision has been made to maintain integration tests for > tendrl project upstream. By integration tests I mean tests of > the whole solution running on real hardware or virtual machines, > checking user scenarios based on requirements/supported use cases. > Those tests are created and maintained by our usmqe team. > > Now the question is where we should keep them. Our team could > either create an usmqe group and maintain our qe repositories > there or we could ask for creation and full admin access to > few qe repositories in tendrl group on github. > Unless you intend to design, develop and maintain a new/bespoke test framework for Tendrl, it would perhaps be convenient to maintain the tests within the Tendrl organization with specific group and access rights. If you use the GitHub workflows, the reviews and such would be widely and publicly available. > What do you think make most sense? Having separate qe group > would be easier for our team to manage, but I acknowledge > the value of having it together with other repositories. > > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From japplewh at redhat.com Tue Oct 25 16:36:28 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 25 Oct 2016 12:36:28 -0400 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> Message-ID: I would vote for "full admin access to few qe repositories in tendrl group on github ?" and I'm happy to set that up quickly if you elect to go this route. Tell me the repo name and the group members (and if all are admins or just a subset) and I will make it happen. Thanks Jeff ? On Tue, Oct 25, 2016 at 12:24 PM, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On 25-Oct-2016 21:48, "Martin Bukatovic" wrote: > > > > Dear all, > > > > the decision has been made to maintain integration tests for > > tendrl project upstream. By integration tests I mean tests of > > the whole solution running on real hardware or virtual machines, > > checking user scenarios based on requirements/supported use cases. > > Those tests are created and maintained by our usmqe team. > > > > Now the question is where we should keep them. Our team could > > either create an usmqe group and maintain our qe repositories > > there or we could ask for creation and full admin access to > > few qe repositories in tendrl group on github. > > > > Unless you intend to design, develop and maintain a new/bespoke test > framework for Tendrl, it would perhaps be convenient to maintain the tests > within the Tendrl organization with specific group and access rights. > > If you use the GitHub workflows, the reviews and such would be widely and > publicly available. > > > What do you think make most sense? Having separate qe group > > would be easier for our team to manage, but I acknowledge > > the value of having it together with other repositories. > > > > -- > > Martin Bukatovic > > USM QE team > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From mbukatov at redhat.com Tue Oct 25 16:43:12 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Tue, 25 Oct 2016 18:43:12 +0200 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> Message-ID: On 10/25/2016 06:24 PM, Sankarshan Mukhopadhyay wrote: > Unless you intend to design, develop and maintain a new/bespoke test > framework for Tendrl, it would perhaps be convenient to maintain the tests > within the Tendrl organization with specific group and access rights. The plan is to use pytest framework as described here: https://tendrl.atlassian.net/browse/TEN-16 > If you use the GitHub workflows, the reviews and such would be widely and > publicly available. The same applies if we decide to create our own group either on gilab.com or on github.com. So the question here boils down to this: could our qe team ask for a new repository to be created in tendrl github.com group, with full admin access (so that we would own the repository, could configure and enforce workflow of our choice and so on)? -- Martin Bukatovic USM QE team From sankarshan.mukhopadhyay at gmail.com Tue Oct 25 16:46:04 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 25 Oct 2016 22:16:04 +0530 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> Message-ID: On 25-Oct-2016 22:13, "Martin Bukatovic" wrote: > > On 10/25/2016 06:24 PM, Sankarshan Mukhopadhyay wrote: > > Unless you intend to design, develop and maintain a new/bespoke test > > framework for Tendrl, it would perhaps be convenient to maintain the tests > > within the Tendrl organization with specific group and access rights. > > The plan is to use pytest framework as described here: > > https://tendrl.atlassian.net/browse/TEN-16 > > > If you use the GitHub workflows, the reviews and such would be widely and > > publicly available. > > The same applies if we decide to create our own group either on > gilab.com or on github.com. So the question here boils down to > this: could our qe team ask for a new repository to be created in > tendrl github.com group, with full admin access (so that we would > own the repository, could configure and enforce workflow of our > choice and so on)? > I would think that you definitely should. Perhaps this request needs to have an issue filed (I think that's useful to do) and it should be sorted. > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mbukatov at redhat.com Tue Oct 25 16:55:32 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Tue, 25 Oct 2016 18:55:32 +0200 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> Message-ID: <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> On 10/25/2016 06:36 PM, Jeff Applewhite wrote: > I would vote for "full admin access to few qe repositories in tendrl group > on github > ?" and I'm happy to set that up quickly if you elect to go this route. > > Tell me the repo name and the group members (and if all are admins or just > a subset) > and I will make it happen. Great, so let's try it this way. We would like the following repositories to be created: * usmqe-tests (for automated test cases) * usmqe-setup (qe ansible playbooks for initial and test setup) So that the new empty repositories are managed by: * https://github.com/mbukatov * https://github.com/dahorak * https://github.com/ltrilety * https://github.com/mkudlej Thank you. -- Martin Bukatovic USM QE team From mbukatov at redhat.com Tue Oct 25 17:04:42 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Tue, 25 Oct 2016 19:04:42 +0200 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> Message-ID: <27f067c4-f7ee-b30d-780c-d2db11eb2f38@redhat.com> On 10/25/2016 06:46 PM, Sankarshan Mukhopadhyay wrote: > I would think that you definitely should. Perhaps this request needs to > have an issue filed (I think that's useful to do) and it should be sorted. Good, so we have an agreement on this across the teams. Now we can let Jeff to make it happen. -- Martin Bukatovic USM QE team From japplewh at redhat.com Tue Oct 25 17:11:13 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 25 Oct 2016 13:11:13 -0400 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> Message-ID: one suggestion - how about I drop the usm string (or replace it with tendrl)? Also - are we going with LGPL license? On Tue, Oct 25, 2016 at 12:55 PM, Martin Bukatovic wrote: > On 10/25/2016 06:36 PM, Jeff Applewhite wrote: > > I would vote for "full admin access to few qe repositories in tendrl > group > > on github > > ?" and I'm happy to set that up quickly if you elect to go this route. > > > > Tell me the repo name and the group members (and if all are admins or > just > > a subset) > > and I will make it happen. > > Great, so let's try it this way. We would like the following > repositories to be created: > > * usmqe-tests (for automated test cases) > * usmqe-setup (qe ansible playbooks for initial and test setup) > > So that the new empty repositories are managed by: > > * https://github.com/mbukatov > * https://github.com/dahorak > * https://github.com/ltrilety > * https://github.com/mkudlej > > Thank you. > > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From rkanade at redhat.com Wed Oct 26 06:49:32 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Wed, 26 Oct 2016 12:19:32 +0530 Subject: [Tendrl-devel] Tendrl mid-cycle Hack Days Proposal (8, 9, 10 Nov 2016, Bangalore, India) In-Reply-To: References: Message-ID: Ken, https://github.com/Tendrl/ceph_bridge/blob/master/doc/source/installation.rst You can find similar doc for other tendrl repos. On Tue, Oct 25, 2016 at 9:28 PM, Ken Dreyer wrote: > On Mon, Oct 24, 2016 at 8:48 PM, Rohan Kanade wrote: > > with Tendrl components installed and running. > > Would you please point me at the step-by-step instructions for how to > do this? (Install all Tendrl components and get them running) > > - Ken > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From negupta at redhat.com Wed Oct 26 07:23:49 2016 From: negupta at redhat.com (Neha Gupta) Date: Wed, 26 Oct 2016 03:23:49 -0400 (EDT) Subject: [Tendrl-devel] queries on dynamic UI generation In-Reply-To: <1064340566.4456113.1477380084427.JavaMail.zimbra@redhat.com> References: <1268550853.19185166.1477305873570.JavaMail.zimbra@redhat.com> <1064340566.4456113.1477380084427.JavaMail.zimbra@redhat.com> Message-ID: <1381289821.28989055.1477466629423.JavaMail.zimbra@redhat.com> @Mrugesh, @Anup: Can you please reply to it or we can schedule a call for it, whichever way you are fine. Thanks, Neha Gupta ----- Original Message ----- From: "Karnan Chidambarakani" To: tendrl-devel at redhat.com Sent: Tuesday, October 25, 2016 12:51:24 PM Subject: [Tendrl-devel] queries on dynamic UI generation Hi Team, My understanding from the demo [demo of UI dynamic generation for create volume workflow] The server is generating Json from Yaml spec, which has the meta information about fields in the UI. Any addition/modification to fields on UI requires changes only on backend. i have few questions on this dynamic generation. - addition/removal of fields might(mostly) requires tweaks on styling/padding on UI.How can we handle this? - How much info about styling will be provided from server? - Does Unit test cases and mock data used for testing have to be updated for every change ? - Will changes to common render view might impact the other pages? - if server is deciding what is to be displayed like php/jsp, where will the business logic reside (server or client)? - Can this be used for generating wizards and dashboards? Thanks, Karnan ----- Original Message ----- From: "Neha Gupta" To: tendrl-devel at redhat.com Sent: Monday, October 24, 2016 4:14:33 PM Subject: [Tendrl-devel] Recording - demo of UI dynamic generation for create volume workflow Hi Team, The recording for Tendrl UI Discussion's meeting can be found at - https://bluejeans.com/s/38i at i/. It includes the demo of dynamic generation of UI for create volume workflow with basic UI design. Thanks, Neha Gupta _______________________________________________ Tendrl-devel mailing list Tendrl-devel at redhat.com https://www.redhat.com/mailman/listinfo/tendrl-devel _______________________________________________ Tendrl-devel mailing list Tendrl-devel at redhat.com https://www.redhat.com/mailman/listinfo/tendrl-devel From mkudlej at redhat.com Wed Oct 26 09:05:09 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Wed, 26 Oct 2016 11:05:09 +0200 Subject: [Tendrl-devel] Record of meeting QE and Anup about how to set up API Message-ID: <5143fe30-45df-5c87-752d-1f5812811b0e@redhat.com> https://bluejeans.com/s/CXLD5/ -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From mkudlej at redhat.com Wed Oct 26 09:06:01 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Wed, 26 Oct 2016 11:06:01 +0200 Subject: [Tendrl-devel] Record of meeting QE and Anup about how to set up API In-Reply-To: <5143fe30-45df-5c87-752d-1f5812811b0e@redhat.com> References: <5143fe30-45df-5c87-752d-1f5812811b0e@redhat.com> Message-ID: <8f67e42a-4b8d-cbae-d417-d28e0dd84893@redhat.com> Sorry, this url should work https://bluejeans.com/s/NY1QS/ On 10/26/2016 11:05 AM, Martin Kudlej wrote: > https://bluejeans.com/s/CXLD5/ > -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From mbukatov at redhat.com Wed Oct 26 09:38:39 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 26 Oct 2016 11:38:39 +0200 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> Message-ID: On 10/25/2016 07:11 PM, Jeff Applewhite wrote: > one suggestion - how about I drop the usm string (or replace it with > tendrl)? I have chosen "usmqe" prefix for repositories based on the name of our qe team. This would make it clear that those repositories are owned by usmqe team. While I understand the idea of having the name aligned, based on our current experiences of naming the projects both upstream and downstream, I would rather not rename our repositories, modules or any other code. > Also - are we going with LGPL license? Approach of our team is to default to GPLv3 license, unless we have a specific reason to license particular component with a different license. I'm checking the details right now to make sure that it would work for us as intended, so we may change the default if I find some good reason to do so. -- Martin Bukatovic USM QE team From deb at redhat.com Wed Oct 26 09:54:28 2016 From: deb at redhat.com (Soumya Deb) Date: Wed, 26 Oct 2016 15:24:28 +0530 Subject: [Tendrl-devel] Record of meeting QE and Anup about how to set up API In-Reply-To: <8f67e42a-4b8d-cbae-d417-d28e0dd84893@redhat.com> References: <5143fe30-45df-5c87-752d-1f5812811b0e@redhat.com> <8f67e42a-4b8d-cbae-d417-d28e0dd84893@redhat.com> Message-ID: Thanks Martin. Request to Anup: can you please jot down a API deployment guide? I understand and appreciate you taking time explaining this through the screenshare but I (and may be some other) have trouble following video guide, and can follow better if it's in text format (call me uncool... i already know, right). Also, I can't copy-paste from vids. :P I was looking for something like this: https://github.com/debloper/tendrl-documentation/blob/master/frontend/deployment.adoc - use it as a template if you want. P.S: I'll ping you for help, setting up a demo/common API server for the frontend team to use. On 26 October 2016 at 14:36, Martin Kudlej wrote: > Sorry, > this url should work https://bluejeans.com/s/NY1QS/ > > > On 10/26/2016 11:05 AM, Martin Kudlej wrote: > >> https://bluejeans.com/s/CXLD5/ >> >> > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, > #usm-meeting @ redhat > #tendrl-devel @ freenode > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From sankarshan.mukhopadhyay at gmail.com Wed Oct 26 10:03:14 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Wed, 26 Oct 2016 15:33:14 +0530 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> Message-ID: On Wed, Oct 26, 2016 at 3:08 PM, Martin Bukatovic wrote: > On 10/25/2016 07:11 PM, Jeff Applewhite wrote: >> one suggestion - how about I drop the usm string (or replace it with >> tendrl)? > > I have chosen "usmqe" prefix for repositories based on the name of > our qe team. This would make it clear that those repositories are > owned by usmqe team. > > While I understand the idea of having the name aligned, based on our > current experiences of naming the projects both upstream and downstream, > I would rather not rename our repositories, modules or any other code. > Tendrl does aspire to be the Unified Storage Manager. Hence the usage of 'usm' moniker in the name might not be intriguing or, awkward. >> Also - are we going with LGPL license? > > Approach of our team is to default to GPLv3 license, unless we have > a specific reason to license particular component with a different > license. I'm checking the details right now to make sure that it would > work for us as intended, so we may change the default if I find some > good reason to do so. > Wonderful! Would wait further update on this. -- sankarshan mukhopadhyay From rwheeler at redhat.com Wed Oct 26 10:20:20 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Wed, 26 Oct 2016 06:20:20 -0400 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> Message-ID: On 10/26/2016 06:03 AM, Sankarshan Mukhopadhyay wrote: > On Wed, Oct 26, 2016 at 3:08 PM, Martin Bukatovic wrote: >> On 10/25/2016 07:11 PM, Jeff Applewhite wrote: >>> one suggestion - how about I drop the usm string (or replace it with >>> tendrl)? >> I have chosen "usmqe" prefix for repositories based on the name of >> our qe team. This would make it clear that those repositories are >> owned by usmqe team. >> >> While I understand the idea of having the name aligned, based on our >> current experiences of naming the projects both upstream and downstream, >> I would rather not rename our repositories, modules or any other code. >> > Tendrl does aspire to be the Unified Storage Manager. Hence the usage > of 'usm' moniker in the name might not be intriguing or, awkward. > >>> Also - are we going with LGPL license? >> Approach of our team is to default to GPLv3 license, unless we have >> a specific reason to license particular component with a different >> license. I'm checking the details right now to make sure that it would >> work for us as intended, so we may change the default if I find some >> good reason to do so. >> > Wonderful! Would wait further update on this. > > We need to use an lgplv3 license (not GPLV3) - so that we can link into non-gpled projects... Regards, Ric From rkanade at redhat.com Wed Oct 26 10:23:10 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Wed, 26 Oct 2016 15:53:10 +0530 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> Message-ID: For the current Tendrl repos we are using LGPL v2.1 We can upgrade that to LGPL V3 though, if it is a hard requirement for the first tendrl release On Wed, Oct 26, 2016 at 3:50 PM, Ric Wheeler wrote: > On 10/26/2016 06:03 AM, Sankarshan Mukhopadhyay wrote: > >> On Wed, Oct 26, 2016 at 3:08 PM, Martin Bukatovic >> wrote: >> >>> On 10/25/2016 07:11 PM, Jeff Applewhite wrote: >>> >>>> one suggestion - how about I drop the usm string (or replace it with >>>> tendrl)? >>>> >>> I have chosen "usmqe" prefix for repositories based on the name of >>> our qe team. This would make it clear that those repositories are >>> owned by usmqe team. >>> >>> While I understand the idea of having the name aligned, based on our >>> current experiences of naming the projects both upstream and downstream, >>> I would rather not rename our repositories, modules or any other code. >>> >>> Tendrl does aspire to be the Unified Storage Manager. Hence the usage >> of 'usm' moniker in the name might not be intriguing or, awkward. >> >> Also - are we going with LGPL license? >>>> >>> Approach of our team is to default to GPLv3 license, unless we have >>> a specific reason to license particular component with a different >>> license. I'm checking the details right now to make sure that it would >>> work for us as intended, so we may change the default if I find some >>> good reason to do so. >>> >>> Wonderful! Would wait further update on this. >> >> >> We need to use an lgplv3 license (not GPLV3) - so that we can link into > non-gpled projects... > > Regards, > > Ric > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From deb at redhat.com Wed Oct 26 10:26:41 2016 From: deb at redhat.com (Soumya Deb) Date: Wed, 26 Oct 2016 15:56:41 +0530 Subject: [Tendrl-devel] Record of meeting QE and Anup about how to set up API In-Reply-To: References: <5143fe30-45df-5c87-752d-1f5812811b0e@redhat.com> <8f67e42a-4b8d-cbae-d417-d28e0dd84893@redhat.com> Message-ID: Update: found a slim version from your own repo's working branch: https://github.com/anivargi/tendrl/tree/gluster-create-volume-api Thanks! This needs to be more visible/available & more concise. On 26 October 2016 at 15:24, Soumya Deb wrote: > Thanks Martin. > > Request to Anup: can you please jot down a API deployment guide? > > I understand and appreciate you taking time explaining this through the > screenshare but I (and may be some other) have trouble following video > guide, and can follow better if it's in text format (call me uncool... i > already know, right). Also, I can't copy-paste from vids. :P > > I was looking for something like this: https://github.com/debloper/ > tendrl-documentation/blob/master/frontend/deployment.adoc - use it as a > template if you want. > > P.S: I'll ping you for help, setting up a demo/common API server for the > frontend team to use. > > > On 26 October 2016 at 14:36, Martin Kudlej wrote: > >> Sorry, >> this url should work https://bluejeans.com/s/NY1QS/ >> >> >> On 10/26/2016 11:05 AM, Martin Kudlej wrote: >> >>> https://bluejeans.com/s/CXLD5/ >>> >>> >> -- >> Best Regards, >> Martin Kudlej. >> RHSC/USM Senior Quality Assurance Engineer >> Red Hat Czech s.r.o. >> >> Phone: +420 532 294 155 >> E-mail:mkudlej at redhat.com >> IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, >> #usm-meeting @ redhat >> #tendrl-devel @ freenode >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > From japplewh at redhat.com Wed Oct 26 12:49:45 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Wed, 26 Oct 2016 08:49:45 -0400 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> Message-ID: Ok - I will go with the proposed usmqe naming and LGPLv3 if there are no objections On Wed, Oct 26, 2016 at 6:23 AM, Rohan Kanade wrote: > For the current Tendrl repos we are using LGPL v2.1 > > We can upgrade that to LGPL V3 though, if it is a hard requirement for the > first tendrl release > > On Wed, Oct 26, 2016 at 3:50 PM, Ric Wheeler wrote: > > > On 10/26/2016 06:03 AM, Sankarshan Mukhopadhyay wrote: > > > >> On Wed, Oct 26, 2016 at 3:08 PM, Martin Bukatovic > >> wrote: > >> > >>> On 10/25/2016 07:11 PM, Jeff Applewhite wrote: > >>> > >>>> one suggestion - how about I drop the usm string (or replace it with > >>>> tendrl)? > >>>> > >>> I have chosen "usmqe" prefix for repositories based on the name of > >>> our qe team. This would make it clear that those repositories are > >>> owned by usmqe team. > >>> > >>> While I understand the idea of having the name aligned, based on our > >>> current experiences of naming the projects both upstream and > downstream, > >>> I would rather not rename our repositories, modules or any other code. > >>> > >>> Tendrl does aspire to be the Unified Storage Manager. Hence the usage > >> of 'usm' moniker in the name might not be intriguing or, awkward. > >> > >> Also - are we going with LGPL license? > >>>> > >>> Approach of our team is to default to GPLv3 license, unless we have > >>> a specific reason to license particular component with a different > >>> license. I'm checking the details right now to make sure that it would > >>> work for us as intended, so we may change the default if I find some > >>> good reason to do so. > >>> > >>> Wonderful! Would wait further update on this. > >> > >> > >> We need to use an lgplv3 license (not GPLV3) - so that we can link into > > non-gpled projects... > > > > Regards, > > > > Ric > > > > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From deb at redhat.com Wed Oct 26 14:26:55 2016 From: deb at redhat.com (Soumya Deb) Date: Wed, 26 Oct 2016 19:56:55 +0530 Subject: [Tendrl-devel] Tendrl API demo server has been set up Message-ID: TL;DR: 10.70.41.164:9292 Purpose: for the frontend developers to avoid going through the PTSD of setting up the whole Tendrl stack locally (in order to work on frontend stuff only). Example: http://10.70.41.164:9292/1.0/cluster/3969b68f-e927-45da-84d6-004c67974f07/volume/actions Related: - http://xkcd.com/754/ - http://xkcd.com/1579/ Situation Report: caffeine rush, tunnel vision, 100x open tabs, buzzing noises, rabbit holes, tip of icebergs, late for home etc. On the bright side, the server is working 10/10. kthxbai From mrugesh at brainfunked.org Wed Oct 26 15:39:21 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Wed, 26 Oct 2016 21:09:21 +0530 Subject: [Tendrl-devel] Provisioning Integration for Ceph and Gluster Message-ID: Hi, We had a discussion with Alfredo Deza (ceph-installer) and Sachidanand (gdeploy) wrt the integration of their respective projects with tendrl's provisioning framework for provisioning ceph and gluster. Here are the takeaways from the discussion. ceph-installer vs ceph-ansible * ceph-installer provides a wrapper over ceph-ansible with normalised output and allows for a more granular execution of the provisioning flows. * The tendrl team has prior experience with integrating with ceph-installer in skyrings. * Developing a wrapper over ceph-ansible would take some development effort. However, ceph-installer already addresses some of the problems that would solved by such a wrapper. We've decided to go ahead with ceph-installer integration because of these reasons. There are some existing bugs that have been filed against ceph-installer which need to be fixed. tendrl integration * tendrl's provisioning design enables ansible based systems by handling user account creation and ssh access. Both ceph-installer and gdeploy depend upon this functionality. tendrl would ship this functionality as part of it's Native Execution Scenario. * Wrapper scripts of small ansible playbooks would be required to allow tendrl to deploy ceph-installer and gdeploy on it's Provisioning Control Node as part of the Internal Execution Scenario. * The wrapper scripts, along with the system-specific YAML definition files would enable tendrl to handle provisioning flows. Together these would serve as the Provisioner Interface Modules. Plan of Action: * Nishanth would provide the list of bugs to be fixed in ceph-installer to Alfredo. * Alfredo would scope out the development effort required to implement the fixes to the aforementioned bugs. * Mrugesh will work upon the YAML flow definitions, 2nd November onwards. * Sachidanand, Alfredo and Mrugesh will work together to scope out the exact details of the implementation required to have a working tendrl provisioning integration based on the YAML flows. * Sachidanand and Mrugesh would work upon the integration during the hack days in Bangalore in the week of 7th November. * Based on the details that would emerge in the week of 7th November, feature requests would be filed against ceph-installer and gdeploy for implementation. -- Mrugesh From mbukatov at redhat.com Wed Oct 26 19:04:18 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 26 Oct 2016 21:04:18 +0200 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> Message-ID: <1e17fcb1-c37e-5e62-db7f-4c861bb547b4@redhat.com> On 10/26/2016 02:49 PM, Jeff Applewhite wrote: > Ok - I will go with the proposed usmqe naming and LGPLv3 if there are no > objections First, thanks for creating the repositories in the tendrl group. The main difference (based on my rough understanding) between LGPLv3 and GPLv3 is that the former allows to use the original software as a library in another project which doesn't have to be GPL compatible. Since our code is not a library and is not planned to be used in this way (actually, the usmqe module would be quite useless without the code of the test cases), we see no benefit in using LGPLv3 instead of GPLv3. Actually using LGPLv3 would prevent us from reusing projects licensed with "GPLv2 or any later version". This means that using LGPLv3 would actually made our situation more complicated without any benefit. See gnu licensing guides for more details (I reread them today to validate my understanding): https://www.gnu.org/licenses/license-recommendations.html https://www.gnu.org/licenses/quick-guide-gplv3.html Based on this, I would stay with GPLv3 as the default. That said, I may be wrong (I'm not a lawyer after all). Which brings me to an additional request: I already have a usmqe-tests repository with initial structure in it (readme, license file, empty tree structure with explanation what goes where) and without any actual code. And I would like to force push this repository into the Tendrl/usmqe-tests on github. Could you make it possible for me to do that? -- Martin Bukatovic USM QE team From japplewh at redhat.com Wed Oct 26 20:15:05 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Wed, 26 Oct 2016 16:15:05 -0400 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: <1e17fcb1-c37e-5e62-db7f-4c861bb547b4@redhat.com> References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> <1e17fcb1-c37e-5e62-db7f-4c861bb547b4@redhat.com> Message-ID: The QE team has write access now to both repos. On Wed, Oct 26, 2016 at 3:04 PM, Martin Bukatovic wrote: > On 10/26/2016 02:49 PM, Jeff Applewhite wrote: > > Ok - I will go with the proposed usmqe naming and LGPLv3 if there are no > > objections > > First, thanks for creating the repositories in the tendrl group. > > The main difference (based on my rough understanding) between > LGPLv3 and GPLv3 is that the former allows to use the original > software as a library in another project which doesn't have to > be GPL compatible. Since our code is not a library and is not > planned to be used in this way (actually, the usmqe module would > be quite useless without the code of the test cases), we see no > benefit in using LGPLv3 instead of GPLv3. Actually using LGPLv3 > would prevent us from reusing projects licensed with "GPLv2 or > any later version". This means that using LGPLv3 would actually > made our situation more complicated without any benefit. > > See gnu licensing guides for more details (I reread them today > to validate my understanding): > > https://www.gnu.org/licenses/license-recommendations.html > https://www.gnu.org/licenses/quick-guide-gplv3.html > > Based on this, I would stay with GPLv3 as the default. That > said, I may be wrong (I'm not a lawyer after all). > > Which brings me to an additional request: I already have > a usmqe-tests repository with initial structure in it > (readme, license file, empty tree structure with explanation > what goes where) and without any actual code. And I would like > to force push this repository into the Tendrl/usmqe-tests > on github. Could you make it possible for me to do that? > > > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From nthomas at redhat.com Thu Oct 27 03:45:24 2016 From: nthomas at redhat.com (Nishanth Thomas) Date: Thu, 27 Oct 2016 09:15:24 +0530 Subject: [Tendrl-devel] Provisioning Integration for Ceph and Gluster In-Reply-To: References: Message-ID: Hi Alfredo, Here is the first cut: https://bugzilla.redhat.com/show_bug.cgi?id=1319856 - ceph-installer executes installation of packages in sequences for different nodes https://bugzilla.redhat.com/show_bug.cgi?id=1380628 - Task status is not presented in consumable manner and not updated periodically https://bugzilla.redhat.com/show_bug.cgi?id=1313935 - properly handle ansible-playbook processes that are hung https://bugzilla.redhat.com/show_bug.cgi?id=1305269 - Shrink a ceph cluster by removing nodes or services using the installer API https://bugzilla.redhat.com/show_bug.cgi?id=1309415 - Allow callback URLs on API endpoints Please have look. We can further discuss on the expectations from our side which will help you to scope out the work. Thanks, Nishanth On Wed, Oct 26, 2016 at 9:09 PM, Mrugesh Karnik wrote: > Hi, > > We had a discussion with Alfredo Deza (ceph-installer) and Sachidanand > (gdeploy) wrt the integration of their respective projects with tendrl's > provisioning framework for provisioning ceph and gluster. Here are the > takeaways from the discussion. > > > ceph-installer vs ceph-ansible > > * ceph-installer provides a wrapper over ceph-ansible with normalised > output and allows for a more granular execution of the provisioning flows. > * The tendrl team has prior experience with integrating with ceph-installer > in skyrings. > * Developing a wrapper over ceph-ansible would take some development > effort. However, ceph-installer already addresses some of the problems that > would solved by such a wrapper. > > We've decided to go ahead with ceph-installer integration because of these > reasons. There are some existing bugs that have been filed against > ceph-installer which need to be fixed. > > > tendrl integration > > * tendrl's provisioning design enables ansible based systems by handling > user account creation and ssh access. Both ceph-installer and gdeploy > depend upon this functionality. tendrl would ship this functionality as > part of it's Native Execution Scenario. > * Wrapper scripts of small ansible playbooks would be required to allow > tendrl to deploy ceph-installer and gdeploy on it's Provisioning Control > Node as part of the Internal Execution Scenario. > * The wrapper scripts, along with the system-specific YAML definition files > would enable tendrl to handle provisioning flows. Together these would > serve as the Provisioner Interface Modules. > > > Plan of Action: > > * Nishanth would provide the list of bugs to be fixed in ceph-installer to > Alfredo. > * Alfredo would scope out the development effort required to implement the > fixes to the aforementioned bugs. > * Mrugesh will work upon the YAML flow definitions, 2nd November onwards. > * Sachidanand, Alfredo and Mrugesh will work together to scope out the > exact details of the implementation required to have a working tendrl > provisioning integration based on the YAML flows. > * Sachidanand and Mrugesh would work upon the integration during the hack > days in Bangalore in the week of 7th November. > * Based on the details that would emerge in the week of 7th November, > feature requests would be filed against ceph-installer and gdeploy for > implementation. > > -- > Mrugesh > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From sankarshan.mukhopadhyay at gmail.com Thu Oct 27 05:08:16 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 27 Oct 2016 10:38:16 +0530 Subject: [Tendrl-devel] Tendrl API demo server has been set up In-Reply-To: References: Message-ID: On Wed, Oct 26, 2016 at 7:56 PM, Soumya Deb wrote: > Purpose: for the frontend developers to avoid going through the PTSD of > setting up the whole Tendrl stack locally (in order to work on frontend > stuff only). I was not aware that installation of the Tendrl stack was causing trauma and stress. That is certainly not what one would desire to inflict on the prospective users. -- sankarshan mukhopadhyay From mbukatov at redhat.com Thu Oct 27 08:00:16 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Thu, 27 Oct 2016 10:00:16 +0200 Subject: [Tendrl-devel] usmqe integration tests for tendrl project In-Reply-To: References: <51e5f6e3-8e6b-a5d2-493a-8571e8e14e2e@redhat.com> <3bee1877-f3f3-8dba-d40b-cbcb85b3e3f1@redhat.com> <1e17fcb1-c37e-5e62-db7f-4c861bb547b4@redhat.com> Message-ID: On 10/26/2016 10:15 PM, Jeff Applewhite wrote: > The QE team has write access now to both repos. Thank you, I just pushed the initial version (just the skeleton) of usmqe-tests repo. -- Martin Bukatovic USM QE team From kdreyer at redhat.com Fri Oct 28 00:27:43 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Thu, 27 Oct 2016 18:27:43 -0600 Subject: [Tendrl-devel] Fwd: CentOS Storage SIG meeting on Friday 28-Oct-2016 at 13:00 UTC In-Reply-To: <20161027165114.GG2936@ndevos-x240.usersys.redhat.com> References: <20161026185118.GD2936@ndevos-x240.usersys.redhat.com> <20161027165114.GG2936@ndevos-x240.usersys.redhat.com> Message-ID: Forwarding to tendrl-devel. It would be great to discuss Tendrl RPM packaging and CentOS's Community Build System (Koji) during this meeting. - Ken ---------- Forwarded message ---------- From: Niels de Vos Date: Thu, Oct 27, 2016 at 10:51 AM Subject: CentOS Storage SIG meeting on Friday 28-Oct-2016 at 13:00 UTC To: centos-devel at lists.centos.org Cc: Ken Dreyer , Jose Rivera , Karanbir Singh , Brian Stinson , Francois Cami , Nishanth Thomas Hi all, We did not have a Storage SIG meeting for a while. Now we have interest from additional participants to join the SIG, so it would be good to have a chat again. The (old) time that was used did not seem practical for most attendees, so tomorrow we're planning to have the meeting a little earlier, namely at 13:00 UTC. run "date -d '13:00 UTC'" in your terminal to convert to $LOCALTIME The venue will be #centos-devel on Freenode :) The agenda will mostly be done ad-hoc, with at least the following topics: - roll call - current state of Ceph packaging and testing - current state of Gluster packaging and testing - plans for Samba inclusion - plans for Tendrl inclusion - any other business Thanks, Niels From japplewh at redhat.com Fri Oct 28 03:09:28 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Thu, 27 Oct 2016 23:09:28 -0400 Subject: [Tendrl-devel] Tendrl site Message-ID: Hi All We have a basic site up for viewing and feedback. Please take a look at https://tendrl.github.io/ Once we are happy with the final product we can flip the DNS switch and make tendrl.org live. Thanks to Tuomas Kuosmanen for all his work on this! -- Jeff Applewhite Principal Product Manager From sankarshan.mukhopadhyay at gmail.com Fri Oct 28 04:13:11 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 28 Oct 2016 09:43:11 +0530 Subject: [Tendrl-devel] Tendrl site In-Reply-To: References: Message-ID: On Fri, Oct 28, 2016 at 8:39 AM, Jeff Applewhite wrote: > We have a basic site up for viewing and feedback. Please take a look at > https://tendrl.github.io/ > > Once we are happy with the final product we can flip the DNS switch and > make tendrl.org live. > > Thanks to Tuomas Kuosmanen for all his work on this! Indeed. Thank you, Tuomas! -- sankarshan mukhopadhyay From shtripat at redhat.com Fri Oct 28 09:29:26 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Fri, 28 Oct 2016 14:59:26 +0530 Subject: [Tendrl-devel] !! Wish you all a very happy and prosperous Diwali !! Message-ID: <261da8bd-98a9-76f9-955e-8924afdc81e7@redhat.com> From sankarshan.mukhopadhyay at gmail.com Fri Oct 28 09:58:26 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 28 Oct 2016 15:28:26 +0530 Subject: [Tendrl-devel] Fwd: CentOS Storage SIG meeting on Friday 28-Oct-2016 at 13:00 UTC In-Reply-To: References: <20161026185118.GD2936@ndevos-x240.usersys.redhat.com> <20161027165114.GG2936@ndevos-x240.usersys.redhat.com> Message-ID: On 28-Oct-2016 05:57, "Ken Dreyer" wrote: > > Forwarding to tendrl-devel. It would be great to discuss Tendrl RPM > packaging and CentOS's Community Build System (Koji) during this > meeting. > It would indeed be nice. And thank you for forwarding the notification. Sadly, 1830 on a Friday evening at the start of a long weekend is probably not the best time this week for most of the intended audience around Tendrl. > > > ---------- Forwarded message ---------- > From: Niels de Vos > Date: Thu, Oct 27, 2016 at 10:51 AM > Subject: CentOS Storage SIG meeting on Friday 28-Oct-2016 at 13:00 UTC > To: centos-devel at lists.centos.org > Cc: Ken Dreyer , Jose Rivera , > Karanbir Singh , Brian Stinson > , Francois Cami , Nishanth > Thomas > > > Hi all, > > We did not have a Storage SIG meeting for a while. Now we have interest > from additional participants to join the SIG, so it would be good to > have a chat again. > > The (old) time that was used did not seem practical for most attendees, > so tomorrow we're planning to have the meeting a little earlier, namely > at 13:00 UTC. > > run "date -d '13:00 UTC'" in your terminal to convert to $LOCALTIME > > The venue will be #centos-devel on Freenode :) > > The agenda will mostly be done ad-hoc, with at least the following > topics: > > - roll call > - current state of Ceph packaging and testing > - current state of Gluster packaging and testing > - plans for Samba inclusion > - plans for Tendrl inclusion > - any other business > > Thanks, > Niels > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From dnarayan at redhat.com Fri Oct 28 13:22:05 2016 From: dnarayan at redhat.com (Darshan Narayana Murthy) Date: Fri, 28 Oct 2016 09:22:05 -0400 (EDT) Subject: [Tendrl-devel] Gluster brick creation via tendrl In-Reply-To: <546529559.4814067.1477659968440.JavaMail.zimbra@redhat.com> Message-ID: <1119188463.4817432.1477660925724.JavaMail.zimbra@redhat.com> Hi all, This is regarding the approach to be taken by tendrl for automating the process of gluster brick creation. I have investigated some of the tools available like pyudev, storaged, udisks, blivet for this purpose. >From the analysis it seems to me that blivet would be the better choice for our use case. I have tracked the details of this analysis in a github issue[1]. Please, have a look and provide your inputs/comments. [1] https://github.com/Tendrl/documentation/issues/49 Thanks, Darshan N From dnarayan at redhat.com Fri Oct 28 13:54:43 2016 From: dnarayan at redhat.com (Darshan Narayana Murthy) Date: Fri, 28 Oct 2016 09:54:43 -0400 (EDT) Subject: [Tendrl-devel] Gluster brick creation via tendrl Message-ID: <559141840.4823645.1477662883806.JavaMail.zimbra@redhat.com> Hi all, This is regarding the approach to be taken by tendrl for automating the process of gluster brick creation. I have investigated some of the tools available like pyudev, storaged, udisks, blivet for this purpose. >From the analysis it seems to me that blivet would be the better choice for our use case. I have tracked the details of this analysis in a github issue[1]. Please, have a look and provide your inputs/comments. [1] https://github.com/Tendrl/documentation/issues/49 Thanks, Darshan N From gmeno at redhat.com Fri Oct 28 17:36:25 2016 From: gmeno at redhat.com (Gregory Meno) Date: Fri, 28 Oct 2016 10:36:25 -0700 Subject: [Tendrl-devel] Calamari and the Tendrl Ceph Bridge In-Reply-To: References: Message-ID: Looks good to me. On Thu, Oct 20, 2016 at 5:27 AM, Rohan Kanade wrote: > Lets start working on what we have mutually agreed upon till now. > > - Continue the sync_object API in place > (http://calamari.readthedocs.io/en/latest/calamari_rest/resources/api_example_api_v2_cluster__fsid__sync_object__sync_type_.html) > - get rid of salt dependency completely > - provide single REST API endpoint (Tendrl does not need to worry on what > mons calamari runs this API) consumable by Tendrl and ensure scenario like > below works as described > > eg: Assuming that both below actions are taken at the same time. > 1. GET api/v2/cluster//pool should return same list of pools as step > 2. > 2 ceph osd lspools > > - Add support for RBD > - In a pre-defined environment, calamari should be stress tested for a > refresh interval of 2-3 seconds per API call. We would need to add this to > calamari QE test cases maybe. > > These are the minimum requirements Tendrl has from calamari. If we are > agreeing on these, we can file detailed BZ's. > > On Thu, Oct 20, 2016 at 1:22 AM, Gregory Meno wrote: >> >> On Thu, Oct 13, 2016 at 4:18 AM, Rohan Kanade wrote: >> > >> > >> > On Thu, Oct 6, 2016 at 7:26 PM, Gregory Meno wrote: >> >> >> >> On Wed, Oct 5, 2016 at 5:57 AM, Rohan Kanade >> >> wrote: >> >> > Calamari is an important component in the Ceph management/ops >> >> > process. >> >> > In >> >> > the past, The Skyrings [0] project was able to utilize Calamari via >> >> > the >> >> > Calamari REST API, the integration did ran into some problems as >> >> > mentioned >> >> > below. >> >> > >> >> > - Calamari-server would often be a single point of failure as it is >> >> > installed on the ceph-mon nodes >> >> >> >> Consider that monitor nodes are the right place to run and there are >> >> usually 3 or 5 monitors. >> >> Is that enough? I think we are at a point now where this would be >> >> reasonable to run on each monitor. >> >> What is more interesting to me is how you see the interaction >> >> happening. >> >> >> >> Does each process instance track the mon leader and proxy requests to >> >> the instance that is on the leader? >> > >> > Tendrl calamari interaction will happen via calamari REST API, Tendrl >> > wont >> > be tracking calamari processes in any part of the cluster >> > >> >> John: have we considered what the ceph-mgr answer to this is? >> >> >> >> Do you expect to have perfect replication of state related to requests >> >> to change the ceph cluster? >> > >> > eg: If we call GET /calamari/osds , it should provide a list of osds up >> > until that point in time >> > >> >> How do you want to discover available endpoints? >> > >> > On a new cluster or an existing cluster, Tendrl will assume calamari is >> > pre-installed and functional (this will be part of Tendrl >> > pre-requisites) >> > Then, while installing Tendrl itself, it should be provided with the >> > calamari REST API endpoint either via conf files or user input. >> > >> >> >> >> > - Querying Calamari REST API gave inconsistent data about the ceph >> >> > cluster >> >> > state, this might be happening because of sync issues occurring due >> >> > to >> >> > ceph-mons not being in quorum etc. >> >> >> >> Specific tickets would go a long way here. >> >> >> >> > - Calamari required to be installed on only one of the ceph-mons, >> >> > This >> >> > introduces very common scalability issues. >> >> >> >> I think we can relax this requirement if we can agree on expectations >> >> above. >> >> >> >> >> >> > - Calamari relies on older version of saltstack for messaging and >> >> > other >> >> > communication, this introduces packaging and performance overheads >> >> >> >> This is not the case. We only use salt to send events to tendrl and >> >> it's something I consider to be a mistake. >> >> I think salt can be taken out of the picture in future releases >> > >> > Ok, removal of salt works for Tendrl >> > >> >> >> >> > >> >> > These issues must be re-verified and fixed if they still persist in >> >> > Calamari. >> >> > >> >> > >> >> > >> >> > >> >> > The Tendrl ceph bridge [1], which is a fork of Calamari/cthulu tried >> >> > to >> >> > address these issues by: >> >> >> >> Did they succeed? >> > >> > Yes >> >> >> >> > >> >> > - Distributing the tendrl ceph bridge to talk to all ceph-mons in the >> >> > cluster. >> >> > - Gathering data (3 second poll interval) from ceph-mons only when >> >> > they >> >> > are >> >> > in quorum and directly talking to ceph-mons via librados and using >> >> > ceph >> >> > admin sockets to query valid cluster maps and other ceph data. >> >> >> >> Since ceph_bridge is taken directly from calamari it's fair to say >> >> that we posses the same capabilities >> > >> > Please check the forked repository for changes in capabilities >> > >> >> >> >> > - Tendrl ceph bridge tries to address the single point of failure >> >> > issue >> >> > by >> >> > having multiple instances of the bridge for each ceph-mon in the >> >> > cluster. >> >> >> >> Would you please show me where ceph_bridge is aware of other >> >> instances? Or are these multiple copies sharing nothing? >> > >> > For READ of cluster state, we only pull data out of the cluster if all >> > mons >> > are in quorum. We have a single copy of the state which is refreshed at >> > regular intervals as well as per specific events. >> > >> >> >> >> > - Tendrl ceph bridge has centralized communications via an etcd >> >> > cluster >> >> > which removes need to maintain saltstack . >> >> > >> >> > >> >> > >> >> > Moving forward, we would like Calamari to accommodate these >> >> > improvements >> >> > back into the calamari code base, Below are the detailed >> >> > requirements: >> >> > >> >> > - Need Calamari REST API for querying ceph admin sockets [2], More >> >> > details: >> >> >> >> Let's not expose this. I would consider it to be a challenge to your >> >> goal of having multiple instances >> >> as you're talking to a specific socket. Really what you get is info >> >> about clusters that are being setup or severly damaged this should be >> >> built into whatever service is responsible for identifying clusters >> >> and talking to them e.g. calamari and in future ceph-mgr. >> > >> > This is required specifically to verify correctness of cluster state >> > reported by other instances. And as i mentioned we only pull data from >> > sockets if all mons are in quorum. >> > >> >> >> >> > - Need Calamari REST API for providing all cluster maps (monitor, >> >> > osd, >> >> > crush, mds, pg, health, config etc) >> >> >> >> This seems like a strange request. Why do you want access to the raw >> >> data? You'll have to parse it to make sense of it the same way we >> >> already do in calamari and in future ceph-mgr. >> > >> > Calamari already provides this via REST API >> > >> > http://calamari.readthedocs.io/en/latest/calamari_rest/resources/api_example_api_v2_cluster__fsid__sync_object.html >> > Tendrl is simply asking this to be supported in future. >> > >> >> >> >> > - Calamari REST API must represent the ceph cluster state correctly, >> >> > no >> >> > mismatch should occur between the True state of the cluster and the >> >> > state >> >> > represented by the Calamari REST API >> >> >> >> Happy to work to fix specific dependencies. Tickets are best. >> >> >> >> > - Calamari REST API should provide access to monitoring actions like >> >> > "ceph >> >> > -w", "ceph tell" etc >> >> >> >> Help me understand the specifics of what you want to accomplish here. >> >> the output of ceph -w can be synthesized from looking at the cluster >> >> maps. and ceph tell is typically used for troubleshooting specific >> >> instances where you want to set new config with out changing ceph.conf >> >> and restarting. That means it applies to a single instance, not >> >> cluster wide. >> > >> > ceph -w will be used to generate events based on the output. Computing >> > such >> > events from cluster maps is doable too but is more work. >> > Tendrl works not only on a cluster context but also at a node context, >> > ceph >> > tell is required for operations which pertain to specific nodes. >> > >> >> >> >> > - Calamari REST API should provide access to operational actions via >> >> > tools >> >> > like "rbd" etc. >> >> >> >> Agreed. >> >> >> >> > - Calamari REST API should be able to service and perform well on 2-4 >> >> > second >> >> > polling intervals on the REST API >> >> >> >> Yes I agree here too. >> >> >> >> > >> >> > >> >> > These requirements can be run through existing calamari functional >> >> > and >> >> > performance test cases and we should make additions to these test >> >> > cases >> >> > to >> >> > accommodate above requirements. >> >> > >> >> > These requirements would be further expanded into individual BZs or >> >> > tracking >> >> > items in Calamari >> >> > >> >> > >> >> > [0]: https://github.com/skyrings >> >> > [1]: https://github.com/tendrl/ceph_bridge >> >> > [2]: >> >> > >> >> > >> >> > http://docs.ceph.com/docs/jewel/rados/operations/monitoring/#using-the-admin-socket >> >> >> I've read through this thread again and I'm trying to capture all the >> items where we agree on the next steps: >> * add support for RBD >> * get rid of salt >> * run on all monitor nodes With eventually consistent results for READ >> operations >> * leave the sync_object API in place >> >> Items where I need more detail to agree to: >> * access to admin sockets to "verify correctness of cluster state". >> Can we not trust the mon_map? If we cannot it needs either better docs >> or bug fixes. I think this state needs to be exposed correctly by >> ceph. Does that make sense? >> * perform well. Would you please file specific tickets for what >> endpoints are not responding promptly or causing high load? >> >> >> Items I don't see implementing before ceph-mgr transition and maybe not >> then: >> * access to ceph -w and ceph tell >> Ceph -w seems like a poor source of events that would require >> long-polling or eventing to surface, and if we did you'd be scraping a >> data-source that may change. I'm not saying no to eventing, but >> calamari is definatly the wrong place for this to live. >> >> ceph tell is an implementation detail for changing config at runtime. >> I agree that we might want to alter configuration values in rare >> instances. We should either convince Ceph to set better defaults or >> provide write access to a /config endpoint. >> I would say that if you are looking to troubleshoot a cluster and want >> ephemeral config values for specific daemons that we are a long way >> off doing this correctly. >> >> I want to work with you and help Tendrl get what it needs AND I'm >> still trying to figure out just what those specifics are. >> >> cheers, >> G > > From rwheeler at redhat.com Sun Oct 30 13:36:52 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Sun, 30 Oct 2016 09:36:52 -0400 Subject: [Tendrl-devel] Gluster brick creation via tendrl In-Reply-To: <1119188463.4817432.1477660925724.JavaMail.zimbra@redhat.com> References: <1119188463.4817432.1477660925724.JavaMail.zimbra@redhat.com> Message-ID: On 10/28/2016 09:22 AM, Darshan Narayana Murthy wrote: > Hi all, > > This is regarding the approach to be taken by tendrl for automating > the process of gluster brick creation. I have investigated some of > the tools available like pyudev, storaged, udisks, blivet for this > purpose. > > >From the analysis it seems to me that blivet would be the better > choice for our use case. I have tracked the details of this analysis > in a github issue[1]. > > Please, have a look and provide your inputs/comments. > > [1] https://github.com/Tendrl/documentation/issues/49 > > Thanks, > Darshan N Hi Darshan, I would suggest syncing up with David Lehman on the choice of tools - he lead the blivet team and is leading the new Springfield effort. We should make sure to check that we are not moving to something with a short life span :) thanks! Ric From sankarshan.mukhopadhyay at gmail.com Sun Oct 30 15:08:12 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Sun, 30 Oct 2016 20:38:12 +0530 Subject: [Tendrl-devel] Gluster brick creation via tendrl In-Reply-To: <559141840.4823645.1477662883806.JavaMail.zimbra@redhat.com> References: <559141840.4823645.1477662883806.JavaMail.zimbra@redhat.com> Message-ID: On Fri, Oct 28, 2016 at 7:24 PM, Darshan Narayana Murthy wrote: > This is regarding the approach to be taken by tendrl for automating > the process of gluster brick creation. I have investigated some of > the tools available like pyudev, storaged, udisks, blivet for this > purpose. > > >From the analysis it seems to me that blivet would be the better > choice for our use case. I have tracked the details of this analysis > in a github issue[1]. > > Please, have a look and provide your inputs/comments. > > [1] https://github.com/Tendrl/documentation/issues/49 Thanks for putting together the assessment in an issue. It would be good to reach out to the developers (of blivet) to scope out any known gaps/issues as well as endorsing this choice. We could look to a discussion on the issue itself. Also, you conclude "None of these have capability to detect hardware raid volume related details. I guess good number of customers might be using this. Raid specific details like RAID leval, stripe count, disk count are requirement to provision the bricks considering best practices wrt performance." I believe these are important and evolving data to gather in order to be able to create a profile/shape of the underlying hardware enabling the storage system. Additionally, might be something worth taking a quick look at. -- sankarshan mukhopadhyay From ssaha at redhat.com Mon Oct 31 13:05:16 2016 From: ssaha at redhat.com (Sayan Saha) Date: Mon, 31 Oct 2016 09:05:16 -0400 Subject: [Tendrl-devel] Gluster brick creation via tendrl In-Reply-To: References: <559141840.4823645.1477662883806.JavaMail.zimbra@redhat.com> Message-ID: On 10/30/16 11:08 AM, Sankarshan Mukhopadhyay wrote: > On Fri, Oct 28, 2016 at 7:24 PM, Darshan Narayana Murthy > wrote: >> This is regarding the approach to be taken by tendrl for automating >> the process of gluster brick creation. I have investigated some of >> the tools available like pyudev, storaged, udisks, blivet for this >> purpose. >> >> >From the analysis it seems to me that blivet would be the better >> choice for our use case. I have tracked the details of this analysis >> in a github issue[1]. blivet is what we use today with RHGS-C. The RHGS-C team worked with the blivet team to make it happen. So we have knowledge about how to leverage it within the team. Thanks Sayan >> >> Please, have a look and provide your inputs/comments. >> >> [1] https://github.com/Tendrl/documentation/issues/49 > Thanks for putting together the assessment in an issue. It would be > good to reach out to the developers (of blivet) to scope out any known > gaps/issues as well as endorsing this choice. We could look to a > discussion on the issue itself. > > Also, you conclude "None of these have capability to detect hardware > raid volume related details. I guess good number of customers might be > using this. Raid specific details like RAID leval, stripe count, disk > count are requirement to provision the bricks considering best > practices wrt performance." I believe these are important and evolving > data to gather in order to be able to create a profile/shape of the > underlying hardware enabling the storage system. > > Additionally, might be > something worth taking a quick look at. > > From rwheeler at redhat.com Mon Oct 31 13:23:55 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Mon, 31 Oct 2016 09:23:55 -0400 Subject: [Tendrl-devel] Gluster brick creation via tendrl In-Reply-To: References: <559141840.4823645.1477662883806.JavaMail.zimbra@redhat.com> Message-ID: <5bd7de81-7ce1-b7bf-3aed-ba074e5c110d@redhat.com> On 10/31/2016 09:05 AM, Sayan Saha wrote: > On 10/30/16 11:08 AM, Sankarshan Mukhopadhyay wrote: >> On Fri, Oct 28, 2016 at 7:24 PM, Darshan Narayana Murthy >> wrote: >>> This is regarding the approach to be taken by tendrl for automating >>> the process of gluster brick creation. I have investigated some of >>> the tools available like pyudev, storaged, udisks, blivet for this >>> purpose. >>> >>> >From the analysis it seems to me that blivet would be the better >>> choice for our use case. I have tracked the details of this analysis >>> in a github issue[1]. > > blivet is what we use today with RHGS-C. The RHGS-C team worked with the > blivet team to make it happen. So we have knowledge about how to leverage it > within the team. > > Thanks > Sayan As I mentioned early in the thread, we need to sync up with David who lead the blivet team about its future versus the new projects (Springfield) which I think is their future direction. Ric > >>> >>> Please, have a look and provide your inputs/comments. >>> >>> [1] https://github.com/Tendrl/documentation/issues/49 >> Thanks for putting together the assessment in an issue. It would be >> good to reach out to the developers (of blivet) to scope out any known >> gaps/issues as well as endorsing this choice. We could look to a >> discussion on the issue itself. >> >> Also, you conclude "None of these have capability to detect hardware >> raid volume related details. I guess good number of customers might be >> using this. Raid specific details like RAID leval, stripe count, disk >> count are requirement to provision the bricks considering best >> practices wrt performance." I believe these are important and evolving >> data to gather in order to be able to create a profile/shape of the >> underlying hardware enabling the storage system. >> >> Additionally, might be >> something worth taking a quick look at. >> >> > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From adeza at redhat.com Mon Oct 31 14:08:15 2016 From: adeza at redhat.com (Alfredo Deza) Date: Mon, 31 Oct 2016 10:08:15 -0400 Subject: [Tendrl-devel] Provisioning Integration for Ceph and Gluster In-Reply-To: References: Message-ID: On Wed, Oct 26, 2016 at 11:45 PM, Nishanth Thomas wrote: > Hi Alfredo, > > Here is the first cut: > > > https://bugzilla.redhat.com/show_bug.cgi?id=1319856 - ceph-installer > executes installation of packages in sequences for different nodes > https://bugzilla.redhat.com/show_bug.cgi?id=1380628 - Task status is not > presented in consumable manner and not updated periodically > https://bugzilla.redhat.com/show_bug.cgi?id=1313935 - properly handle > ansible-playbook processes that are hung > https://bugzilla.redhat.com/show_bug.cgi?id=1305269 - > Shrink a ceph cluster by removing nodes or services using the installer API > https://bugzilla.redhat.com/show_bug.cgi?id=1309415 - > Allow callback URLs on API endpoints > > Please have look. We can further discuss on the expectations from our side > which will help you to scope out the work. What is the timeframe that we are looking at this before code-freeze? > > Thanks, > Nishanth > > On Wed, Oct 26, 2016 at 9:09 PM, Mrugesh Karnik > wrote: > >> Hi, >> >> We had a discussion with Alfredo Deza (ceph-installer) and Sachidanand >> (gdeploy) wrt the integration of their respective projects with tendrl's >> provisioning framework for provisioning ceph and gluster. Here are the >> takeaways from the discussion. >> >> >> ceph-installer vs ceph-ansible >> >> * ceph-installer provides a wrapper over ceph-ansible with normalised >> output and allows for a more granular execution of the provisioning flows. >> * The tendrl team has prior experience with integrating with ceph-installer >> in skyrings. >> * Developing a wrapper over ceph-ansible would take some development >> effort. However, ceph-installer already addresses some of the problems that >> would solved by such a wrapper. >> >> We've decided to go ahead with ceph-installer integration because of these >> reasons. There are some existing bugs that have been filed against >> ceph-installer which need to be fixed. >> >> >> tendrl integration >> >> * tendrl's provisioning design enables ansible based systems by handling >> user account creation and ssh access. Both ceph-installer and gdeploy >> depend upon this functionality. tendrl would ship this functionality as >> part of it's Native Execution Scenario. >> * Wrapper scripts of small ansible playbooks would be required to allow >> tendrl to deploy ceph-installer and gdeploy on it's Provisioning Control >> Node as part of the Internal Execution Scenario. >> * The wrapper scripts, along with the system-specific YAML definition files >> would enable tendrl to handle provisioning flows. Together these would >> serve as the Provisioner Interface Modules. >> >> >> Plan of Action: >> >> * Nishanth would provide the list of bugs to be fixed in ceph-installer to >> Alfredo. >> * Alfredo would scope out the development effort required to implement the >> fixes to the aforementioned bugs. >> * Mrugesh will work upon the YAML flow definitions, 2nd November onwards. >> * Sachidanand, Alfredo and Mrugesh will work together to scope out the >> exact details of the implementation required to have a working tendrl >> provisioning integration based on the YAML flows. >> * Sachidanand and Mrugesh would work upon the integration during the hack >> days in Bangalore in the week of 7th November. >> * Based on the details that would emerge in the week of 7th November, >> feature requests would be filed against ceph-installer and gdeploy for >> implementation. >> >> -- >> Mrugesh >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From kdreyer at redhat.com Mon Oct 31 16:20:20 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Mon, 31 Oct 2016 10:20:20 -0600 Subject: [Tendrl-devel] CentOS Storage SIG meeting on Friday 28-Oct-2016 at 13:00 UTC In-Reply-To: References: <20161026185118.GD2936@ndevos-x240.usersys.redhat.com> <20161027165114.GG2936@ndevos-x240.usersys.redhat.com> Message-ID: Hi folks, Here are the minutes from the #centos-devel meeting last Friday. https://www.centos.org/minutes/2016/october/centos-devel.2016-10-28-13.05.html full logs: https://www.centos.org/minutes/2016/october/centos-devel.2016-10-28-13.05.log.html - Ken On Thu, Oct 27, 2016 at 6:27 PM, Ken Dreyer wrote: > Forwarding to tendrl-devel. It would be great to discuss Tendrl RPM > packaging and CentOS's Community Build System (Koji) during this > meeting. > > - Ken > > > ---------- Forwarded message ---------- > From: Niels de Vos > Date: Thu, Oct 27, 2016 at 10:51 AM > Subject: CentOS Storage SIG meeting on Friday 28-Oct-2016 at 13:00 UTC > To: centos-devel at lists.centos.org > Cc: Ken Dreyer , Jose Rivera , > Karanbir Singh , Brian Stinson > , Francois Cami , Nishanth > Thomas > > > Hi all, > > We did not have a Storage SIG meeting for a while. Now we have interest > from additional participants to join the SIG, so it would be good to > have a chat again. > > The (old) time that was used did not seem practical for most attendees, > so tomorrow we're planning to have the meeting a little earlier, namely > at 13:00 UTC. > > run "date -d '13:00 UTC'" in your terminal to convert to $LOCALTIME > > The venue will be #centos-devel on Freenode :) > > The agenda will mostly be done ad-hoc, with at least the following > topics: > > - roll call > - current state of Ceph packaging and testing > - current state of Gluster packaging and testing > - plans for Samba inclusion > - plans for Tendrl inclusion > - any other business > > Thanks, > Niels