From shtripat at redhat.com Thu Sep 1 07:20:22 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Thu, 1 Sep 2016 12:50:22 +0530 Subject: [Tendrl-devel] Few questions/clarification regarding tendrl In-Reply-To: References: <822cc76f-2dd5-f31e-bd4d-fa69290dafde@redhat.com> Message-ID: <88adcd4c-8844-be13-eac4-8ffb489b6398@redhat.com> Same would be true for cthulu code as well if I am not wrong. On 08/31/2016 06:13 PM, Rohan Kanade wrote: > Weird, must have slipped out during the beginning. > > Thanks for pointing out, fixed. > > On Wed, Aug 31, 2016 at 3:18 PM, John Spray > wrote: > > On Wed, Aug 31, 2016 at 10:28 AM, Shubhendu Tripathi > > wrote: > > On 08/31/2016 02:33 PM, Rohan Kanade wrote: > > > > On 31-Aug-2016 14:20, "Shubhendu Tripathi" > wrote: > >> > >> Hi All, > >> > >> I got few questions in my mind which I wanted to clarify > regarding tendrl > >> new architecture > >> > >> 1. As per my understanding cthulu module from calamari is more > or less > >> used as ceph-bridge with certain changes may be. My question > is, why not use > >> cthulu as layered product and use as library only? Do we > really need to > >> copy code as a separate project? Also even if we don't use > cthulu as > >> library, due to LGPL should we retain the copyright notices of > the original > >> sources as is in tendrl's ceph-bridge code ? > >> > > > > Cthulu is not used as is and there's various changes like removal of > > saltstack as a dependency, changing its persistence layer for > etcd instead > > of Relational etc. It is difficult to maintain these features > directly in > > cthulu. > > > > Tendrl is LGPL2 and we should definitely run this through Legal > just to be > > sure > > > >> 2. The etcdobj of bridge-common looks taken from > >> https://github.com/ashcrow/etcdobj > . Wouldn't this come with > additional > >> responsibility of maintaining this project and we might miss > out of updates > >> as well. Also if its intentional copy, should we retain the actual > >> copyright/license message? Can we think of using this project > as library? > >> > > > > Etcdobj is a very small codebase. The original repo is hasn't > seen any > > activity since 4months. Etcd is a central part of Tendrl and we > need finer > > control over this piece and it's lifecycle. > > It looks like (I hope accidentally) someone has stripped the original > header from that code. > > If you're going to copy it, at least respect the conditions from the > original, including the condition that source copies must reproduce > the copyright notice and list of conditions. > https://github.com/ashcrow/etcdobj/blob/master/src/etcdobj/fields.py > > > John > > > > > Just a suggestion, why not contribute to the original project > and make it > > mature enough and use as library. At least replication of effort > is reduced. > > > > Regarding the licensing issue. We should consult with Legal and > rectify any > > noncompliance > >> These are just my points of view and I was not clear so thought of > >> clarifying with broader audience. > >> > >> Thanks and Regards, > >> Shubhendu > >> > >> PS: https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html > > >> > >> _______________________________________________ > >> Tendrl-devel mailing list > >> Tendrl-devel at redhat.com > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > >> > > > > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shtripat at redhat.com Thu Sep 1 10:29:16 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Thu, 1 Sep 2016 15:59:16 +0530 Subject: [Tendrl-devel] Few questions/clarification regarding tendrl In-Reply-To: <88adcd4c-8844-be13-eac4-8ffb489b6398@redhat.com> References: <822cc76f-2dd5-f31e-bd4d-fa69290dafde@redhat.com> <88adcd4c-8844-be13-eac4-8ffb489b6398@redhat.com> Message-ID: Based on discussion over IRC tendrl-devel, John has a valid point. I feel, it would be better to fork out from cthulu and then append/update the required changes as per our needs ----------------------------------------------------- shubhendu, k4n0: the interesting thing about the ceph_bridge repository is there doesn't seem to be any copyright notice at all. there's just a license. compare to e.g. calamari: https://github.com/ceph/calamari/blob/master/COPYING however that Calamari notice isn't strictly correct either as not all contributions are copyright Red Hat. the thing is that in existing Calamari and Ceph repositories, the git history gives you the record of who contributed what when you just lift the code, you lose all that. this only needs addressing if anyone is planning to keep using that Calamari code in Tendrl I got the impression that there was an intention to drop it last time I spoke to mrugesh a month ago, but I haven't heard anything since jcsp, I am not sure this last part regarding dropping it. Yes I am in total agreement on losing out on contributors list if just lifted the code ----------------------------------------------------- John, do you want to add/suggest something ?? Regards, Shubhendu On 09/01/2016 12:50 PM, Shubhendu Tripathi wrote: > Same would be true for cthulu code as well if I am not wrong. > > > On 08/31/2016 06:13 PM, Rohan Kanade wrote: >> Weird, must have slipped out during the beginning. >> >> Thanks for pointing out, fixed. >> >> On Wed, Aug 31, 2016 at 3:18 PM, John Spray > > wrote: >> >> On Wed, Aug 31, 2016 at 10:28 AM, Shubhendu Tripathi >> > wrote: >> > On 08/31/2016 02:33 PM, Rohan Kanade wrote: >> > >> > On 31-Aug-2016 14:20, "Shubhendu Tripathi" > > wrote: >> >> >> >> Hi All, >> >> >> >> I got few questions in my mind which I wanted to clarify >> regarding tendrl >> >> new architecture >> >> >> >> 1. As per my understanding cthulu module from calamari is more >> or less >> >> used as ceph-bridge with certain changes may be. My question >> is, why not use >> >> cthulu as layered product and use as library only? Do we >> really need to >> >> copy code as a separate project? Also even if we don't use >> cthulu as >> >> library, due to LGPL should we retain the copyright notices of >> the original >> >> sources as is in tendrl's ceph-bridge code ? >> >> >> > >> > Cthulu is not used as is and there's various changes like >> removal of >> > saltstack as a dependency, changing its persistence layer for >> etcd instead >> > of Relational etc. It is difficult to maintain these features >> directly in >> > cthulu. >> > >> > Tendrl is LGPL2 and we should definitely run this through Legal >> just to be >> > sure >> > >> >> 2. The etcdobj of bridge-common looks taken from >> >> https://github.com/ashcrow/etcdobj >> . Wouldn't this come with >> additional >> >> responsibility of maintaining this project and we might miss >> out of updates >> >> as well. Also if its intentional copy, should we retain the actual >> >> copyright/license message? Can we think of using this project >> as library? >> >> >> > >> > Etcdobj is a very small codebase. The original repo is hasn't >> seen any >> > activity since 4months. Etcd is a central part of Tendrl and >> we need finer >> > control over this piece and it's lifecycle. >> >> It looks like (I hope accidentally) someone has stripped the original >> header from that code. >> >> If you're going to copy it, at least respect the conditions from the >> original, including the condition that source copies must reproduce >> the copyright notice and list of conditions. >> https://github.com/ashcrow/etcdobj/blob/master/src/etcdobj/fields.py >> >> >> John >> >> > >> > Just a suggestion, why not contribute to the original project >> and make it >> > mature enough and use as library. At least replication of >> effort is reduced. >> > >> > Regarding the licensing issue. We should consult with Legal and >> rectify any >> > noncompliance >> >> These are just my points of view and I was not clear so thought of >> >> clarifying with broader audience. >> >> >> >> Thanks and Regards, >> >> Shubhendu >> >> >> >> PS: https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html >> >> >> >> >> _______________________________________________ >> >> Tendrl-devel mailing list >> >> Tendrl-devel at redhat.com >> >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> >> >> > >> > >> > >> > _______________________________________________ >> > Tendrl-devel mailing list >> > Tendrl-devel at redhat.com >> > https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> > >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Thu Sep 1 10:38:44 2016 From: jspray at redhat.com (John Spray) Date: Thu, 1 Sep 2016 11:38:44 +0100 Subject: [Tendrl-devel] Few questions/clarification regarding tendrl In-Reply-To: References: <822cc76f-2dd5-f31e-bd4d-fa69290dafde@redhat.com> <88adcd4c-8844-be13-eac4-8ffb489b6398@redhat.com> Message-ID: On Thu, Sep 1, 2016 at 11:29 AM, Shubhendu Tripathi wrote: > Based on discussion over IRC tendrl-devel, John has a valid point. > I feel, it would be better to fork out from cthulu and then append/update > the required changes as per our needs > > ----------------------------------------------------- > shubhendu, k4n0: the interesting thing about the ceph_bridge > repository is there doesn't seem to be any copyright notice at all. > there's just a license. > compare to e.g. calamari: > https://github.com/ceph/calamari/blob/master/COPYING > however that Calamari notice isn't strictly correct either as not all > contributions are copyright Red Hat. > the thing is that in existing Calamari and Ceph repositories, the git > history gives you the record of who contributed what > when you just lift the code, you lose all that. > this only needs addressing if anyone is planning to keep using that > Calamari code in Tendrl > I got the impression that there was an intention to drop it last time I > spoke to mrugesh a month ago, but I haven't heard anything since > jcsp, I am not sure this last part regarding dropping it. Yes I > am in total agreement on losing out on contributors list if just lifted the > code > ----------------------------------------------------- > > John, do you want to add/suggest something ?? Just that we really need to hear from the people driving this project about what their plans are. John > Regards, > Shubhendu > > > > On 09/01/2016 12:50 PM, Shubhendu Tripathi wrote: > > Same would be true for cthulu code as well if I am not wrong. > > > On 08/31/2016 06:13 PM, Rohan Kanade wrote: > > Weird, must have slipped out during the beginning. > > Thanks for pointing out, fixed. > > On Wed, Aug 31, 2016 at 3:18 PM, John Spray wrote: >> >> On Wed, Aug 31, 2016 at 10:28 AM, Shubhendu Tripathi >> wrote: >> > On 08/31/2016 02:33 PM, Rohan Kanade wrote: >> > >> > On 31-Aug-2016 14:20, "Shubhendu Tripathi" wrote: >> >> >> >> Hi All, >> >> >> >> I got few questions in my mind which I wanted to clarify regarding >> >> tendrl >> >> new architecture >> >> >> >> 1. As per my understanding cthulu module from calamari is more or less >> >> used as ceph-bridge with certain changes may be. My question is, why >> >> not use >> >> cthulu as layered product and use as library only? Do we really need >> >> to >> >> copy code as a separate project? Also even if we don't use cthulu as >> >> library, due to LGPL should we retain the copyright notices of the >> >> original >> >> sources as is in tendrl's ceph-bridge code ? >> >> >> > >> > Cthulu is not used as is and there's various changes like removal of >> > saltstack as a dependency, changing its persistence layer for etcd >> > instead >> > of Relational etc. It is difficult to maintain these features directly >> > in >> > cthulu. >> > >> > Tendrl is LGPL2 and we should definitely run this through Legal just to >> > be >> > sure >> > >> >> 2. The etcdobj of bridge-common looks taken from >> >> https://github.com/ashcrow/etcdobj. Wouldn't this come with additional >> >> responsibility of maintaining this project and we might miss out of >> >> updates >> >> as well. Also if its intentional copy, should we retain the actual >> >> copyright/license message? Can we think of using this project as >> >> library? >> >> >> > >> > Etcdobj is a very small codebase. The original repo is hasn't seen any >> > activity since 4months. Etcd is a central part of Tendrl and we need >> > finer >> > control over this piece and it's lifecycle. >> >> It looks like (I hope accidentally) someone has stripped the original >> header from that code. >> >> If you're going to copy it, at least respect the conditions from the >> original, including the condition that source copies must reproduce >> the copyright notice and list of conditions. >> https://github.com/ashcrow/etcdobj/blob/master/src/etcdobj/fields.py >> >> John >> >> > >> > Just a suggestion, why not contribute to the original project and make >> > it >> > mature enough and use as library. At least replication of effort is >> > reduced. >> > >> > Regarding the licensing issue. We should consult with Legal and rectify >> > any >> > noncompliance >> >> These are just my points of view and I was not clear so thought of >> >> clarifying with broader audience. >> >> >> >> Thanks and Regards, >> >> Shubhendu >> >> >> >> PS: https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html >> >> >> >> _______________________________________________ >> >> Tendrl-devel mailing list >> >> Tendrl-devel at redhat.com >> >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> >> > >> > >> > >> > _______________________________________________ >> > Tendrl-devel mailing list >> > Tendrl-devel at redhat.com >> > https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > > From avishwan at redhat.com Thu Sep 1 13:55:55 2016 From: avishwan at redhat.com (Aravinda) Date: Thu, 1 Sep 2016 19:25:55 +0530 Subject: [Tendrl-devel] Few questions/clarification regarding tendrl In-Reply-To: References: <822cc76f-2dd5-f31e-bd4d-fa69290dafde@redhat.com> <88adcd4c-8844-be13-eac4-8ffb489b6398@redhat.com> Message-ID: <8250b005-a5c5-5617-a56f-66dd14a15597@redhat.com> +1 We should retain git history when a project forked from another project. regards Aravinda On Thursday 01 September 2016 04:08 PM, John Spray wrote: > On Thu, Sep 1, 2016 at 11:29 AM, Shubhendu Tripathi wrote: >> Based on discussion over IRC tendrl-devel, John has a valid point. >> I feel, it would be better to fork out from cthulu and then append/update >> the required changes as per our needs >> >> ----------------------------------------------------- >> shubhendu, k4n0: the interesting thing about the ceph_bridge >> repository is there doesn't seem to be any copyright notice at all. >> there's just a license. >> compare to e.g. calamari: >> https://github.com/ceph/calamari/blob/master/COPYING >> however that Calamari notice isn't strictly correct either as not all >> contributions are copyright Red Hat. >> the thing is that in existing Calamari and Ceph repositories, the git >> history gives you the record of who contributed what >> when you just lift the code, you lose all that. >> this only needs addressing if anyone is planning to keep using that >> Calamari code in Tendrl >> I got the impression that there was an intention to drop it last time I >> spoke to mrugesh a month ago, but I haven't heard anything since >> jcsp, I am not sure this last part regarding dropping it. Yes I >> am in total agreement on losing out on contributors list if just lifted the >> code >> ----------------------------------------------------- >> >> John, do you want to add/suggest something ?? > Just that we really need to hear from the people driving this project > about what their plans are. > > John > >> Regards, >> Shubhendu >> >> >> >> On 09/01/2016 12:50 PM, Shubhendu Tripathi wrote: >> >> Same would be true for cthulu code as well if I am not wrong. >> >> >> On 08/31/2016 06:13 PM, Rohan Kanade wrote: >> >> Weird, must have slipped out during the beginning. >> >> Thanks for pointing out, fixed. >> >> On Wed, Aug 31, 2016 at 3:18 PM, John Spray wrote: >>> On Wed, Aug 31, 2016 at 10:28 AM, Shubhendu Tripathi >>> wrote: >>>> On 08/31/2016 02:33 PM, Rohan Kanade wrote: >>>> >>>> On 31-Aug-2016 14:20, "Shubhendu Tripathi" wrote: >>>>> Hi All, >>>>> >>>>> I got few questions in my mind which I wanted to clarify regarding >>>>> tendrl >>>>> new architecture >>>>> >>>>> 1. As per my understanding cthulu module from calamari is more or less >>>>> used as ceph-bridge with certain changes may be. My question is, why >>>>> not use >>>>> cthulu as layered product and use as library only? Do we really need >>>>> to >>>>> copy code as a separate project? Also even if we don't use cthulu as >>>>> library, due to LGPL should we retain the copyright notices of the >>>>> original >>>>> sources as is in tendrl's ceph-bridge code ? >>>>> >>>> Cthulu is not used as is and there's various changes like removal of >>>> saltstack as a dependency, changing its persistence layer for etcd >>>> instead >>>> of Relational etc. It is difficult to maintain these features directly >>>> in >>>> cthulu. >>>> >>>> Tendrl is LGPL2 and we should definitely run this through Legal just to >>>> be >>>> sure >>>> >>>>> 2. The etcdobj of bridge-common looks taken from >>>>> https://github.com/ashcrow/etcdobj. Wouldn't this come with additional >>>>> responsibility of maintaining this project and we might miss out of >>>>> updates >>>>> as well. Also if its intentional copy, should we retain the actual >>>>> copyright/license message? Can we think of using this project as >>>>> library? >>>>> >>>> Etcdobj is a very small codebase. The original repo is hasn't seen any >>>> activity since 4months. Etcd is a central part of Tendrl and we need >>>> finer >>>> control over this piece and it's lifecycle. >>> It looks like (I hope accidentally) someone has stripped the original >>> header from that code. >>> >>> If you're going to copy it, at least respect the conditions from the >>> original, including the condition that source copies must reproduce >>> the copyright notice and list of conditions. >>> https://github.com/ashcrow/etcdobj/blob/master/src/etcdobj/fields.py >>> >>> John >>> >>>> Just a suggestion, why not contribute to the original project and make >>>> it >>>> mature enough and use as library. At least replication of effort is >>>> reduced. >>>> >>>> Regarding the licensing issue. We should consult with Legal and rectify >>>> any >>>> noncompliance >>>>> These are just my points of view and I was not clear so thought of >>>>> clarifying with broader audience. >>>>> >>>>> Thanks and Regards, >>>>> Shubhendu >>>>> >>>>> PS: https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html >>>>> >>>>> _______________________________________________ >>>>> Tendrl-devel mailing list >>>>> Tendrl-devel at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Tendrl-devel mailing list >>>> Tendrl-devel at redhat.com >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>> >> >> >> > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From kdreyer at redhat.com Thu Sep 1 21:45:12 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Thu, 1 Sep 2016 15:45:12 -0600 Subject: [Tendrl-devel] ceph-installer in tendrl? Message-ID: Hi folks, Are you planning to use https://github.com/ceph/ceph-installer with Tendrl? What changes will you need? - Ken From shtripat at redhat.com Fri Sep 2 04:35:47 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Fri, 2 Sep 2016 10:05:47 +0530 Subject: [Tendrl-devel] ceph-installer in tendrl? In-Reply-To: References: Message-ID: Thanks Ken for bringing this up. In addition to this I have set of other questions as below 1. Currently RHS-Console uses salt mechanism for node discovery and then accept them. What would be mechanism to replace this in tendrl? Current work-flow is driven from getting new machines and bootstrap them so that they are discovered by RHS-Console and steps follow for cluster creation process. Are we still planning to retain this workflow ? I understand it should be seamless from UI perspective. 2. What http server do we plan to use for deploying tendrl REST apis ? Will the tendrl UI be also hosted under the same? 3. Do we rule out use of calamari at all? If so what do we plan for ceph events handling in tendrl? 4. In case there is no salt involved, how we plan to handle events from storage nodes to tendrl ? Do we need to implement a different event handling mechanism afresh in tendrl? 5. Are we still going to have collectd and graphite combination for monitoring purpose in tendrl (in case its used as tendrl apis + tendrl UI combination)? Thanks and Regards, Shubhendu On 09/02/2016 03:15 AM, Ken Dreyer wrote: > Hi folks, > > Are you planning to use https://github.com/ceph/ceph-installer with Tendrl? > > What changes will you need? > > - Ken > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From japplewh at redhat.com Fri Sep 2 16:35:20 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Fri, 2 Sep 2016 12:35:20 -0400 Subject: [Tendrl-devel] Tendrl Upstream Jira: Open Source Free license received Message-ID: Hi All I have received confirmation of our free open source license for Jira hosted and the project board is now up, though in a very early stage. If you would like an account and have not received one please let me know. If you just want to browse the project you can do so anonymously. The site url is: https://tendrl.atlassian.net/ Most of the current information is in "issues" (which is an odd term for "user stories", but nevertheless quite a nice piece of software). Over the next few days there will be more information posted by Mrugesh on the Github site on architectural progress, getting started, etc. The actual html site is in progress (thanks Patrick McGarry!) and we will keep you posted on the progress around that. Thanks! -- Jeff Applewhite Principal Product Manager 919-381-7406 m -------------- next part -------------- An HTML attachment was scrubbed... URL: From mbukatov at redhat.com Tue Sep 6 10:46:37 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Tue, 6 Sep 2016 12:46:37 +0200 Subject: [Tendrl-devel] Few questions/clarification regarding tendrl In-Reply-To: References: Message-ID: On 08/31/2016 11:14 AM, John Spray wrote: > I would also like to hear more about how Tendrl will interface to Ceph > -- I'm sure others on ceph-devel would be interested too. I've been > given the impression off-list that the copied Calamari code is mainly > a placeholder, and I've pointed out that ceph-mgr is coming soon and > is intended for exactly this kind of thing. > > Remember that Tendrl is not the only project that's interested in > managing Ceph, so it's important to work with others in the community > (such as OpenAttic) to find areas of overlapping interest. +1 from me. Does anyone have any updates on this? -- Martin Bukatovic USM QE team From mkudlej at redhat.com Tue Sep 6 15:24:46 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Tue, 6 Sep 2016 17:24:46 +0200 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy Message-ID: Hello all Tendrl fans, I would like to present our view of CI and how we would like to build our CI infrastructure. Workflow schema: 1) there are some git repositories 2) all unit tests run 3) there are packages 4) all package sanity tests run 5) all machines for integration testing are ready 6) packages are installed on proper machines 7) all integration tests run You can replace "packages" to any other distribution package like QCow2 image or container. There are some git repositories where source code of all Tendrl components is stored. If trigger is fired all unit tests will run. Trigger can be for example daily cron, patch is merged to master, during package building or any other event. It's easy to change this in CI tool. If all unit tests pass packages are built. Once packages are successfully built they are checked by package sanity tests. If packages are OK then required machines for integration testing are deployed and packages are installed on them. If installation is successful all integration tests run. There are some tasks in Jira for making this workflow happened. We need to move our integration tests to upstream so everybody is able to do 7) and 4). It will be great if anybody can see what we are testing. There are tasks which depends on environment(5 and 6). Because we have no resources to maintain publicly available environment, deployment scripts will be private for now. Sanity of git repositories, code there and building packages (1, 2, 3) are domains of software developers and details depend mostly on developers. If we can we would like to help with those tasks too. Results from testing will be publicly available somewhere(should be decided, I expect tendrl.org). Please share with us your comments about our plans. Thank you! -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From jspray at redhat.com Tue Sep 6 15:38:41 2016 From: jspray at redhat.com (John Spray) Date: Tue, 6 Sep 2016 16:38:41 +0100 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: On Tue, Sep 6, 2016 at 4:24 PM, Martin Kudlej wrote: > Hello all Tendrl fans, > > I would like to present our view of CI and how we would like to build our CI > infrastructure. > Workflow schema: > > 1) there are some git repositories > 2) all unit tests run > 3) there are packages > 4) all package sanity tests run > 5) all machines for integration testing are ready > 6) packages are installed on proper machines > 7) all integration tests run > > You can replace "packages" to any other distribution package like QCow2 > image or container. > > There are some git repositories where source code of all Tendrl components > is stored. If trigger is fired all unit tests will run. Trigger can be for > example daily cron, patch is merged to master, during package building or > any other event. It's easy to change this in CI tool. If all unit tests pass > packages are built. Once packages are successfully built they are checked by > package sanity tests. If packages are OK then required machines for > integration testing are deployed and packages are installed on them. If > installation is successful all integration tests run. > > There are some tasks in Jira for making this workflow happened. We need to > move our integration tests to upstream so everybody is able to do 7) and 4). > It will be great if anybody can see what we are testing. > > There are tasks which depends on environment(5 and 6). Because we have no > resources to maintain publicly available environment, deployment scripts > will be private for now. This point would be a good reason to consider using the existing Ceph test framework (teuthology) -- we have a public environment. You could for example have a private teuthology environment if you wanted, but when folks wanted to run their tests in a public upstream environment they could either ask for resources in the "sepia" Ceph environment, or they could use teuthology's openstack mode to run on an openstack cloud. John > Sanity of git repositories, code there and building packages (1, 2, 3) are > domains of software developers and details depend mostly on developers. If > we can we would like to help with those tasks too. > > Results from testing will be publicly available somewhere(should be decided, > I expect tendrl.org). > > Please share with us your comments about our plans. Thank you! > > > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting > @ redhat > #tendrl-devel @ freenode > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mbukatov at redhat.com Tue Sep 6 16:54:20 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Tue, 6 Sep 2016 18:54:20 +0200 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: On 09/06/2016 05:38 PM, John Spray wrote: >> There are tasks which depends on environment(5 and 6). Because we have no >> resources to maintain publicly available environment, deployment scripts >> will be private for now. > > This point would be a good reason to consider using the existing Ceph > test framework (teuthology) -- we have a public environment. You > could for example have a private teuthology environment if you wanted, > but when folks wanted to run their tests in a public upstream > environment they could either ask for resources in the "sepia" Ceph > environment, or they could use teuthology's openstack mode to run on > an openstack cloud. I don't think we could use teuthology directly. But maybe we could at least create some wrapper which would run our tests in teuthology environment. Could you point me to a description of how teuthology describes machines of a cluster? -- Martin Bukatovic USM QE team From jspray at redhat.com Tue Sep 6 17:26:05 2016 From: jspray at redhat.com (John Spray) Date: Tue, 6 Sep 2016 18:26:05 +0100 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: On Tue, Sep 6, 2016 at 5:54 PM, Martin Bukatovic wrote: > On 09/06/2016 05:38 PM, John Spray wrote: >>> There are tasks which depends on environment(5 and 6). Because we have no >>> resources to maintain publicly available environment, deployment scripts >>> will be private for now. >> >> This point would be a good reason to consider using the existing Ceph >> test framework (teuthology) -- we have a public environment. You >> could for example have a private teuthology environment if you wanted, >> but when folks wanted to run their tests in a public upstream >> environment they could either ask for resources in the "sepia" Ceph >> environment, or they could use teuthology's openstack mode to run on >> an openstack cloud. > > I don't think we could use teuthology directly. But maybe we could at > least create some wrapper which would run our tests in teuthology > environment. Can you elaborate on why you can't use teuthology? It seems like testing a management console on top of a Ceph cluster would have extremely similar environmental requirements to just testing the Ceph cluster. Yes, at some stage you need to be able to add Gluster support too, but that's not a reason to start from scratch. > Could you point me to a description of how teuthology describes > machines of a cluster? http://docs.ceph.com/teuthology/docs/detailed_test_config.html#test-configuration Machines themselves are generally identical within a particular cluster so there's no description as such. CCing the sepia list which has bigger teuthology experts than myself. John > > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mrugesh at brainfunked.org Wed Sep 7 08:59:48 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Wed, 7 Sep 2016 14:29:48 +0530 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: John, Thanks for pointing out teuthology. I took a cursory look at the documentation. The primary question that I currently have is whether the assumption, that this'll be a ceph specific environment, correct. Tendrl's integration tests would also need to be run against gluster. On 6 September 2016 at 22:56, John Spray wrote: > > On Tue, Sep 6, 2016 at 5:54 PM, Martin Bukatovic wrote: > > On 09/06/2016 05:38 PM, John Spray wrote: > >>> There are tasks which depends on environment(5 and 6). Because we have no > >>> resources to maintain publicly available environment, deployment scripts > >>> will be private for now. > >> > >> This point would be a good reason to consider using the existing Ceph > >> test framework (teuthology) -- we have a public environment. You > >> could for example have a private teuthology environment if you wanted, > >> but when folks wanted to run their tests in a public upstream > >> environment they could either ask for resources in the "sepia" Ceph > >> environment, or they could use teuthology's openstack mode to run on > >> an openstack cloud. > > > > I don't think we could use teuthology directly. But maybe we could at > > least create some wrapper which would run our tests in teuthology > > environment. > > Can you elaborate on why you can't use teuthology? It seems like > testing a management console on top of a Ceph cluster would have > extremely similar environmental requirements to just testing the Ceph > cluster. Yes, at some stage you need to be able to add Gluster > support too, but that's not a reason to start from scratch. > > > Could you point me to a description of how teuthology describes > > machines of a cluster? > > http://docs.ceph.com/teuthology/docs/detailed_test_config.html#test-configuration > > Machines themselves are generally identical within a particular > cluster so there's no description as such. > > CCing the sepia list which has bigger teuthology experts than myself. > > John > > > > > -- > > Martin Bukatovic > > USM QE team > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Wed Sep 7 09:09:15 2016 From: jspray at redhat.com (John Spray) Date: Wed, 7 Sep 2016 10:09:15 +0100 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 9:59 AM, Mrugesh Karnik wrote: > John, > > Thanks for pointing out teuthology. I took a cursory look at the > documentation. The primary question that I currently have is whether the > assumption, that this'll be a ceph specific environment, correct. Tendrl's > integration tests would also need to be run against gluster. It was built for Ceph, but there is a fairly clear separation between the test infrastructure (teuthology) and the ceph-specific parts (ceph-qa-suite). Even so, it's a lot less work to adapt an existing test framework to include another storage system than it is to try and write one from scratch. As anyone who has worked on Ceph QA can attest, these things are not trivial. What I'd suggest is that anyone making decisions about test environments for Tendrl should first get a strong understanding of how Ceph testing is done today, and only then consider what changes they may need. Creating something new should be a last resort. John > > On 6 September 2016 at 22:56, John Spray wrote: >> >> On Tue, Sep 6, 2016 at 5:54 PM, Martin Bukatovic >> wrote: >> > On 09/06/2016 05:38 PM, John Spray wrote: >> >>> There are tasks which depends on environment(5 and 6). Because we have >> >>> no >> >>> resources to maintain publicly available environment, deployment >> >>> scripts >> >>> will be private for now. >> >> >> >> This point would be a good reason to consider using the existing Ceph >> >> test framework (teuthology) -- we have a public environment. You >> >> could for example have a private teuthology environment if you wanted, >> >> but when folks wanted to run their tests in a public upstream >> >> environment they could either ask for resources in the "sepia" Ceph >> >> environment, or they could use teuthology's openstack mode to run on >> >> an openstack cloud. >> > >> > I don't think we could use teuthology directly. But maybe we could at >> > least create some wrapper which would run our tests in teuthology >> > environment. >> >> Can you elaborate on why you can't use teuthology? It seems like >> testing a management console on top of a Ceph cluster would have >> extremely similar environmental requirements to just testing the Ceph >> cluster. Yes, at some stage you need to be able to add Gluster >> support too, but that's not a reason to start from scratch. >> >> > Could you point me to a description of how teuthology describes >> > machines of a cluster? >> >> >> http://docs.ceph.com/teuthology/docs/detailed_test_config.html#test-configuration >> >> Machines themselves are generally identical within a particular >> cluster so there's no description as such. >> >> CCing the sepia list which has bigger teuthology experts than myself. >> >> John >> >> > >> > -- >> > Martin Bukatovic >> > USM QE team >> > >> > _______________________________________________ >> > Tendrl-devel mailing list >> > Tendrl-devel at redhat.com >> > https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel From mbukatov at redhat.com Wed Sep 7 09:23:12 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 7 Sep 2016 11:23:12 +0200 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: <1343a8ab-04ea-1738-6050-96134ce75f7f@redhat.com> On 09/06/2016 07:26 PM, John Spray wrote: > On Tue, Sep 6, 2016 at 5:54 PM, Martin Bukatovic wrote: >> On 09/06/2016 05:38 PM, John Spray wrote: >>> This point would be a good reason to consider using the existing Ceph >>> test framework (teuthology) -- we have a public environment. You >>> could for example have a private teuthology environment if you wanted, >>> but when folks wanted to run their tests in a public upstream >>> environment they could either ask for resources in the "sepia" Ceph >>> environment, or they could use teuthology's openstack mode to run on >>> an openstack cloud. >> >> I don't think we could use teuthology directly. But maybe we could at >> least create some wrapper which would run our tests in teuthology >> environment. > > Can you elaborate on why you can't use teuthology? It seems like > testing a management console on top of a Ceph cluster would have > extremely similar environmental requirements to just testing the Ceph > cluster. Yes, at some stage you need to be able to add Gluster > support too, but that's not a reason to start from scratch. We need to work with both gluster and ceph teams. Here the problem is that while we integrate with both of the teams, we don't actually have the same testing strategy, we don't need to retest all possible details of ceph or gluster use cases, we will test just use cases of the console assuming that the ceph and gluster works. We use Selenium based automation clicking in the web interface or scripting REST API requests and in both cases when some operation is started, we need to check details on the actual machines if it was done as expected (eg. volume created). So we have a centralized testing model, when the tests runs on a machine from which we access the console via HTTP and the other nodes via ssh (when necessary). And since we need to work with 2 different teams with 2 completely different testing strategies and automation, it wouldn't make sense for us (at least at first sight) to just pick either ceph or gluster qe framework, as we would need to extend it to: * be able test web ui/REST api with it while running other test cases * use the other test framework we didn't pick * provide a way for both gluster and ceph teams to run our tests (eg. to setup a cluster via tendrl) It would make more sense if we could reuse particular components from both teams when necessary. But here the problem is that both teuthology[1] and distaf[2] looks quite monolithic/all in one solutions. There is also new ceph qe tool glusto[3] which doesn't look that monolithic. [1] http://docs.ceph.com/teuthology/docs/README.html [2] https://github.com/gluster/distaf [3] https://glusto.readthedocs.io/en/latest/ That said, maybe I'm missing few important details and my impression is wrong, but at least at this point, it seem to me that it would be better for our team to try to have modular test automation, which would allow to reuse particular components with other teams. And for this reason, our current plan is to use pytest framework for our testing and ansible for setup/configuration. Our current plan ================ Our plan looks like this: * use pytest framework as a base * our test code is python 3 only, while libraries/extensions we would maintain will be python2/3 * write few pytest plugins/extensions to get functionality we need * use some pytest extensions from other teams * use ansible for setup * use ansible inventory file for both ansible itself and our tests (to describe machines and roles in the cluster) We would like to get inspiration from ManageIQ QE team, which has a similar scenario: they test a management server which provides both web UI and REST API, and controls/manages (to some degree) several other machines (virtual machines running in cloud in ManageIQ case, we would manage storage machines in our case). Theirs repo is here: https://github.com/ManageIQ/integration_tests It would be great if we are able to cooperate with them and get some modules out so that both teams could use them without taking in the whole framework and the tests with it. When it comes to the management of the machines, we would like to be able utilize any kind of machines, from real hw storage team has in the lab to libvirt or open stack virtual machines. To describe the machines of a particular cluster, we use ansible inventory file. For example: this is how mbukatov-usm2.hosts file looks like (it's generated by our deployment automation service): ~~~ [usm_nodes] mbukatov-usm2-mon1.example.redhat.com mbukatov-usm2-mon2.example.redhat.com mbukatov-usm2-mon3.example.redhat.com mbukatov-usm2-node1.example.redhat.com mbukatov-usm2-node2.example.redhat.com mbukatov-usm2-node3.example.redhat.com mbukatov-usm2-node4.example.redhat.com mbukatov-usm2-node5.example.redhat.com mbukatov-usm2-node6.example.redhat.com [ceph_mon] mbukatov-usm2-mon1.example.redhat.com mbukatov-usm2-mon2.example.redhat.com mbukatov-usm2-mon3.example.redhat.com [ceph_osd] mbukatov-usm2-node1.example.redhat.com mbukatov-usm2-node2.example.redhat.com mbukatov-usm2-node3.example.redhat.com mbukatov-usm2-node4.example.redhat.com mbukatov-usm2-node5.example.redhat.com mbukatov-usm2-node6.example.redhat.com [usm_server] mbukatov-usm2-server.example.redhat.com ~~~ This file contains all machines of the particular test cluster (mbukatov-usm2), and groups them into roles (from tendrl perspective). The inventory is then used during: * run of our qe ansible playbooks both during deployment and during setup (for a particular test of set of tests - eg. one playbook would install stress package on given roles, other would configure firewall based on the documentation ... and such playbook could be used for both manual testing and run during automated test run) * when we need to check something ad hoc (from the commanline) * from or test code to do some operation for every machine of a particular group (eg. ceph_mon nodes) This way, we have a unified elementary description of the cluster machines. >> Could you point me to a description of how teuthology describes >> machines of a cluster? > > http://docs.ceph.com/teuthology/docs/detailed_test_config.html#test-configuration > > Machines themselves are generally identical within a particular > cluster so there's no description as such. > > CCing the sepia list which has bigger teuthology experts than myself. That looks interesting, thanks for the pointer. We would need to study the teuthology in more detail anyway, so I just created a ticket to track it here: https://tendrl.atlassian.net/browse/TEN-23 -- Martin Bukatovic USM QE team From mbukatov at redhat.com Wed Sep 7 09:31:49 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 7 Sep 2016 11:31:49 +0200 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: On 09/07/2016 11:09 AM, John Spray wrote: > What I'd suggest is that anyone making decisions about test > environments for Tendrl should first get a strong understanding of how > Ceph testing is done today, and only then consider what changes they > may need. Creating something new should be a last resort. I agree. We just need to do the same for gluster test framework and ManageIQ test framework as well. -- Martin Bukatovic USM QE team From mkudlej at redhat.com Wed Sep 7 09:34:11 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Wed, 7 Sep 2016 11:34:11 +0200 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: Hi John, On 09/07/2016 11:09 AM, John Spray wrote: > On Wed, Sep 7, 2016 at 9:59 AM, Mrugesh Karnik wrote: >> John, >> >> Thanks for pointing out teuthology. I took a cursory look at the >> documentation. The primary question that I currently have is whether the >> assumption, that this'll be a ceph specific environment, correct. Tendrl's >> integration tests would also need to be run against gluster. > > It was built for Ceph, but there is a fairly clear separation between > the test infrastructure (teuthology) and the ceph-specific parts > (ceph-qa-suite). Even so, it's a lot less work to adapt an existing > test framework to include another storage system than it is to try and > write one from scratch. As anyone who has worked on Ceph QA can > attest, these things are not trivial. Yes, I agree that use existing testing framework is better than create new one. After quick look at Teuthology I'm not sure if it is right framework for us. It seems to me too Ceph specific. Also there is testing framework for Gluster https://github.com/gluster/distaf which I expect is also too Gluster specific. Our workflow is very similar to ManageIQ. We have to test Web UI and also REST API like ManageIQ. We can consume a lot of Pytest plugins from them. And also they've hired Pytest contributor, so it can help us too. https://github.com/ManageIQ/integration_tests And final bonus is that we can easily integrate with ManageIQ testing if USM upstream will decide to integrate with it. On the other hand we would like to use as many as possible from Gluster and Ceph tests. Because it is waste of time to rewrite existing tests. For example I've talked with one Ceph contributor about tests for simulating disk failures. > > What I'd suggest is that anyone making decisions about test > environments for Tendrl should first get a strong understanding of how > Ceph testing is done today, and only then consider what changes they > may need. Creating something new should be a last resort. > > John > >> >> On 6 September 2016 at 22:56, John Spray wrote: >>> >>> On Tue, Sep 6, 2016 at 5:54 PM, Martin Bukatovic >>> wrote: >>>> On 09/06/2016 05:38 PM, John Spray wrote: >>>>>> There are tasks which depends on environment(5 and 6). Because we have >>>>>> no >>>>>> resources to maintain publicly available environment, deployment >>>>>> scripts >>>>>> will be private for now. >>>>> >>>>> This point would be a good reason to consider using the existing Ceph >>>>> test framework (teuthology) -- we have a public environment. You >>>>> could for example have a private teuthology environment if you wanted, >>>>> but when folks wanted to run their tests in a public upstream >>>>> environment they could either ask for resources in the "sepia" Ceph >>>>> environment, or they could use teuthology's openstack mode to run on >>>>> an openstack cloud. >>>> >>>> I don't think we could use teuthology directly. But maybe we could at >>>> least create some wrapper which would run our tests in teuthology >>>> environment. >>> >>> Can you elaborate on why you can't use teuthology? It seems like >>> testing a management console on top of a Ceph cluster would have >>> extremely similar environmental requirements to just testing the Ceph >>> cluster. Yes, at some stage you need to be able to add Gluster >>> support too, but that's not a reason to start from scratch. >>> >>>> Could you point me to a description of how teuthology describes >>>> machines of a cluster? >>> >>> >>> http://docs.ceph.com/teuthology/docs/detailed_test_config.html#test-configuration >>> >>> Machines themselves are generally identical within a particular >>> cluster so there's no description as such. >>> >>> CCing the sepia list which has bigger teuthology experts than myself. >>> >>> John >>> >>>> >>>> -- >>>> Martin Bukatovic >>>> USM QE team >>>> >>>> _______________________________________________ >>>> Tendrl-devel mailing list >>>> Tendrl-devel at redhat.com >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From jspray at redhat.com Wed Sep 7 10:10:22 2016 From: jspray at redhat.com (John Spray) Date: Wed, 7 Sep 2016 11:10:22 +0100 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: References: Message-ID: On Wed, Sep 7, 2016 at 10:34 AM, Martin Kudlej wrote: > Hi John, > > On 09/07/2016 11:09 AM, John Spray wrote: >> >> On Wed, Sep 7, 2016 at 9:59 AM, Mrugesh Karnik >> wrote: >>> >>> John, >>> >>> Thanks for pointing out teuthology. I took a cursory look at the >>> documentation. The primary question that I currently have is whether the >>> assumption, that this'll be a ceph specific environment, correct. >>> Tendrl's >>> integration tests would also need to be run against gluster. >> >> >> It was built for Ceph, but there is a fairly clear separation between >> the test infrastructure (teuthology) and the ceph-specific parts >> (ceph-qa-suite). Even so, it's a lot less work to adapt an existing >> test framework to include another storage system than it is to try and >> write one from scratch. As anyone who has worked on Ceph QA can >> attest, these things are not trivial. > > > Yes, I agree that use existing testing framework is better than create new > one. After quick look at Teuthology I'm not sure if it is right framework > for us. It seems to me too Ceph specific. Also there is testing framework > for Gluster https://github.com/gluster/distaf which I expect is also too > Gluster specific. > Our workflow is very similar to ManageIQ. We have to test Web UI and also > REST API like ManageIQ. We can consume a lot of Pytest plugins from them. > And also they've hired Pytest contributor, so it can help us too. > https://github.com/ManageIQ/integration_tests > And final bonus is that we can easily integrate with ManageIQ testing if USM > upstream will decide to integrate with it. > > On the other hand we would like to use as many as possible from Gluster and > Ceph tests. Because it is waste of time to rewrite existing tests. For > example I've talked with one Ceph contributor about tests for simulating > disk failures. OK -- at this stage we probably come back to the big unknown of what Tendrl is actually going to do. If it was going to have a lot of Ceph-related features then I think there would be a good argument to be made for tight integration with the Ceph testing, but if it's mainly doing higher-level manageiq-like functionality then it's a different story. The main thing I needed reassurance of was that you weren't writing a whole new teuthology-like think from scratch, so thanks :-) John > >> >> What I'd suggest is that anyone making decisions about test >> environments for Tendrl should first get a strong understanding of how >> Ceph testing is done today, and only then consider what changes they >> may need. Creating something new should be a last resort. >> >> John >> >>> >>> On 6 September 2016 at 22:56, John Spray wrote: >>>> >>>> >>>> On Tue, Sep 6, 2016 at 5:54 PM, Martin Bukatovic >>>> wrote: >>>>> >>>>> On 09/06/2016 05:38 PM, John Spray wrote: >>>>>>> >>>>>>> There are tasks which depends on environment(5 and 6). Because we >>>>>>> have >>>>>>> no >>>>>>> resources to maintain publicly available environment, deployment >>>>>>> scripts >>>>>>> will be private for now. >>>>>> >>>>>> >>>>>> This point would be a good reason to consider using the existing Ceph >>>>>> test framework (teuthology) -- we have a public environment. You >>>>>> could for example have a private teuthology environment if you wanted, >>>>>> but when folks wanted to run their tests in a public upstream >>>>>> environment they could either ask for resources in the "sepia" Ceph >>>>>> environment, or they could use teuthology's openstack mode to run on >>>>>> an openstack cloud. >>>>> >>>>> >>>>> I don't think we could use teuthology directly. But maybe we could at >>>>> least create some wrapper which would run our tests in teuthology >>>>> environment. >>>> >>>> >>>> Can you elaborate on why you can't use teuthology? It seems like >>>> testing a management console on top of a Ceph cluster would have >>>> extremely similar environmental requirements to just testing the Ceph >>>> cluster. Yes, at some stage you need to be able to add Gluster >>>> support too, but that's not a reason to start from scratch. >>>> >>>>> Could you point me to a description of how teuthology describes >>>>> machines of a cluster? >>>> >>>> >>>> >>>> >>>> http://docs.ceph.com/teuthology/docs/detailed_test_config.html#test-configuration >>>> >>>> Machines themselves are generally identical within a particular >>>> cluster so there's no description as such. >>>> >>>> CCing the sepia list which has bigger teuthology experts than myself. >>>> >>>> John >>>> >>>>> >>>>> -- >>>>> Martin Bukatovic >>>>> USM QE team >>>>> >>>>> _______________________________________________ >>>>> Tendrl-devel mailing list >>>>> Tendrl-devel at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>>> >>>> >>>> _______________________________________________ >>>> Tendrl-devel mailing list >>>> Tendrl-devel at redhat.com >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting > @ redhat > #tendrl-devel @ freenode From adeza at redhat.com Wed Sep 7 11:30:07 2016 From: adeza at redhat.com (Alfredo Deza) Date: Wed, 7 Sep 2016 07:30:07 -0400 Subject: [Tendrl-devel] Former Console 2.0 QE upstream CI strategy In-Reply-To: <1343a8ab-04ea-1738-6050-96134ce75f7f@redhat.com> References: <1343a8ab-04ea-1738-6050-96134ce75f7f@redhat.com> Message-ID: On Wed, Sep 7, 2016 at 5:23 AM, Martin Bukatovic wrote: > On 09/06/2016 07:26 PM, John Spray wrote: >> On Tue, Sep 6, 2016 at 5:54 PM, Martin Bukatovic wrote: >>> On 09/06/2016 05:38 PM, John Spray wrote: >>>> This point would be a good reason to consider using the existing Ceph >>>> test framework (teuthology) -- we have a public environment. You >>>> could for example have a private teuthology environment if you wanted, >>>> but when folks wanted to run their tests in a public upstream >>>> environment they could either ask for resources in the "sepia" Ceph >>>> environment, or they could use teuthology's openstack mode to run on >>>> an openstack cloud. >>> >>> I don't think we could use teuthology directly. But maybe we could at >>> least create some wrapper which would run our tests in teuthology >>> environment. >> >> Can you elaborate on why you can't use teuthology? It seems like >> testing a management console on top of a Ceph cluster would have >> extremely similar environmental requirements to just testing the Ceph >> cluster. Yes, at some stage you need to be able to add Gluster >> support too, but that's not a reason to start from scratch. > > We need to work with both gluster and ceph teams. Here the > problem is that while we integrate with both of the teams, we don't > actually have the same testing strategy, we don't need to retest all > possible details of ceph or gluster use cases, we will test just use > cases of the console assuming that the ceph and gluster works. > > We use Selenium based automation clicking in the web interface or > scripting REST API requests and in both cases when some operation is > started, we need to check details on the actual machines if it was > done as expected (eg. volume created). So we have a centralized testing > model, when the tests runs on a machine from which we access the > console via HTTP and the other nodes via ssh (when necessary). > > And since we need to work with 2 different teams with 2 completely > different testing strategies and automation, it wouldn't make sense > for us (at least at first sight) to just pick either ceph or gluster > qe framework, as we would need to extend it to: > > * be able test web ui/REST api with it while running > other test cases > * use the other test framework we didn't pick > * provide a way for both gluster and ceph teams to run our tests > (eg. to setup a cluster via tendrl) > > It would make more sense if we could reuse particular components > from both teams when necessary. But here the problem is that both > teuthology[1] and distaf[2] looks quite monolithic/all in one > solutions. There is also new ceph qe tool glusto[3] which doesn't > look that monolithic. > > [1] http://docs.ceph.com/teuthology/docs/README.html > [2] https://github.com/gluster/distaf > [3] https://glusto.readthedocs.io/en/latest/ > > That said, maybe I'm missing few important details and my impression > is wrong, but at least at this point, it seem to me that it would be > better for our team to try to have modular test automation, which would > allow to reuse particular components with other teams. And for this > reason, our current plan is to use pytest framework for our > testing and ansible for setup/configuration. > > Our current plan > ================ > > Our plan looks like this: > > * use pytest framework as a base > * our test code is python 3 only, while libraries/extensions we > would maintain will be python2/3 > * write few pytest plugins/extensions to get functionality we need > * use some pytest extensions from other teams py.test is a great choice here for unittests and some functional testing. It is very easy to extend into non-standard needs (e.g. driving selenium [0]) or doing remote assertions. When the time comes to make assertions of remote state of nodes in a cluster, this will not be that hard to do. I would be more than happy to provide any help/direction when it comes to py.test [0] https://pypi.python.org/pypi/pytest-selenium > * use ansible for setup > * use ansible inventory file for both ansible itself and our tests > (to describe machines and roles in the cluster) > > We would like to get inspiration from ManageIQ QE team, which has > a similar scenario: they test a management server which provides > both web UI and REST API, and controls/manages (to some degree) > several other machines (virtual machines running in cloud in ManageIQ > case, we would manage storage machines in our case). Theirs repo is > here: > > https://github.com/ManageIQ/integration_tests > > It would be great if we are able to cooperate with them and get some > modules out so that both teams could use them without taking in the > whole framework and the tests with it. > > When it comes to the management of the machines, we would like to > be able utilize any kind of machines, from real hw storage team > has in the lab to libvirt or open stack virtual machines. To describe > the machines of a particular cluster, we use ansible inventory file. > > For example: this is how mbukatov-usm2.hosts file looks like (it's > generated by our deployment automation service): > > ~~~ > [usm_nodes] > mbukatov-usm2-mon1.example.redhat.com > mbukatov-usm2-mon2.example.redhat.com > mbukatov-usm2-mon3.example.redhat.com > mbukatov-usm2-node1.example.redhat.com > mbukatov-usm2-node2.example.redhat.com > mbukatov-usm2-node3.example.redhat.com > mbukatov-usm2-node4.example.redhat.com > mbukatov-usm2-node5.example.redhat.com > mbukatov-usm2-node6.example.redhat.com > > [ceph_mon] > mbukatov-usm2-mon1.example.redhat.com > mbukatov-usm2-mon2.example.redhat.com > mbukatov-usm2-mon3.example.redhat.com > > [ceph_osd] > mbukatov-usm2-node1.example.redhat.com > mbukatov-usm2-node2.example.redhat.com > mbukatov-usm2-node3.example.redhat.com > mbukatov-usm2-node4.example.redhat.com > mbukatov-usm2-node5.example.redhat.com > mbukatov-usm2-node6.example.redhat.com > > [usm_server] > mbukatov-usm2-server.example.redhat.com > ~~~ > > This file contains all machines of the particular test cluster > (mbukatov-usm2), and groups them into roles (from tendrl perspective). > > The inventory is then used during: > > * run of our qe ansible playbooks both during deployment and during > setup (for a particular test of set of tests - eg. one playbook would > install stress package on given roles, > other would configure firewall based on the documentation ... and > such playbook could be used for both manual testing and run during > automated test run) > * when we need to check something ad hoc (from the commanline) > * from or test code to do some operation for every machine of a > particular group (eg. ceph_mon nodes) > > This way, we have a unified elementary description of the cluster > machines. > >>> Could you point me to a description of how teuthology describes >>> machines of a cluster? >> >> http://docs.ceph.com/teuthology/docs/detailed_test_config.html#test-configuration >> >> Machines themselves are generally identical within a particular >> cluster so there's no description as such. >> >> CCing the sepia list which has bigger teuthology experts than myself. > > That looks interesting, thanks for the pointer. > > We would need to study the teuthology in more detail anyway, so I > just created a ticket to track it here: > > https://tendrl.atlassian.net/browse/TEN-23 > > -- > Martin Bukatovic > USM QE team > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mbukatov at redhat.com Wed Sep 14 09:50:51 2016 From: mbukatov at redhat.com (Martin Bukatovic) Date: Wed, 14 Sep 2016 11:50:51 +0200 Subject: [Tendrl-devel] ceph-mgr and tendrl Message-ID: <678aa95f-73de-f309-c821-495c446e25f4@redhat.com> Dear all, we may need to check the current plans of the ceph community to introduce a new component of the ceph stack: ceph-mgr The original idea has been shared some time ago here: * https://www.spinics.net/lists/ceph-devel/msg28047.html * http://pad.ceph.com/p/ceph-mgr While this service seems to be concerned with additional features which may not be the best fit for adding into the MON daemons and as such there seems to be no or little overlap with what tendrl will to be doing, it's clear that both tendrl and ceph-mgr components need to cooperate, and so the dev teams would benefit from discussing their ideas from the early on. -- Martin Bukatovic USM QE team From jlemmons at redhat.com Wed Sep 14 16:11:56 2016 From: jlemmons at redhat.com (Joseph Lemmons) Date: Wed, 14 Sep 2016 12:11:56 -0400 Subject: [Tendrl-devel] Mailing list delivery test - Please ignore Message-ID: Hello, This is a delivery test and no action is necessary on your part. If you have any questions, please contact the Service Desk by creating a ticket at help.redhat.com, and requesting that your ticket be assigned to IT-Mail-Services. Thank you, Joseph Lemmons Red Hat IT-Productivity Systems Administrator Zimbra Service Owner Email - jlemmons at redhat.com Phone (External): 1 (919) 890.8016 Phone (Internal): 81 48016 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Thu Sep 15 00:56:23 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 15 Sep 2016 06:26:23 +0530 Subject: [Tendrl-devel] CentOS Storage SIG, Tendrl and CI Message-ID: [The list is not configured to receive emails from non-subscribers. Nigel, you would need to subscribe from Nigel, in light of your recent conversations around the CentOS CI system for Gluster.org, can you help Tendrl understand the set of things which need to be put together before we start running jobs off the CentOS CI? I am, in effect, asking about the application build and test pipeline. /s -- sankarshan mukhopadhyay From nigelb at redhat.com Thu Sep 15 02:59:36 2016 From: nigelb at redhat.com (Nigel Babu) Date: Thu, 15 Sep 2016 08:29:36 +0530 Subject: [Tendrl-devel] CentOS Storage SIG, Tendrl and CI In-Reply-To: References: Message-ID: <20160915025936.GA19390@gmail.com> On Thu, Sep 15, 2016 at 06:26:23AM +0530, Sankarshan Mukhopadhyay wrote: > [The list is not configured to receive emails from non-subscribers. > Nigel, you would need to subscribe from > response> > > Nigel, in light of your recent conversations around the CentOS CI > system for Gluster.org, can you help Tendrl understand the set of > things which need to be put together before we start running jobs off > the CentOS CI? I am, in effect, asking about the application build and > test pipeline. > > /s > Centos CI is already configured to work with Github. So you should be able to get started very easily. I would recommend using Jenkins Job Builder[1] for configuration so it can be easily code-reviewed. I'd say to begin with you want at least the following: 1. A job to test pull requests before merge that they don't break master. 2. If these tests are fast, a post-merge run as well. Otherwise a periodic run on master will also do. 3. You'd want to do a nightly build which will produce artifacts for testing. 4. You'd want to run integration tests using the nightly builds. If you folks need help with code review for JJB jobs, happy to help. I'm working on converting our Centos CI jobs to JJB format as well. [1]: http://docs.openstack.org/infra/jenkins-job-builder/ -- nigelb From mrugesh at brainfunked.org Thu Sep 15 04:38:52 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Thu, 15 Sep 2016 10:08:52 +0530 Subject: [Tendrl-devel] Architecture Documentation. Message-ID: Hello everyone, The documentation repository[1] has now been setup with some architecture related documentation. The README[2] has navigation that points to the appropriate documents. The documents together should be able to provide a clear understanding of the architecture and the role of the components therein. However, should you feel that something requires a better explanation, please initiate a discussion here or file issues[3] directly on the documentation repository. [1] https://github.com/Tendrl/documentation [2] https://github.com/Tendrl/documentation/blob/master/README.adoc [3] https://github.com/Tendrl/documentation/issues Thanks, Mrugesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkudlej at redhat.com Thu Sep 15 05:41:59 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Thu, 15 Sep 2016 07:41:59 +0200 Subject: [Tendrl-devel] Missing installation instructions Message-ID: Hi all, what are installation instructions, please? For tendrl/gluster_bridge and tendrl/ceph_bridge, you can do "python setup.py install" and configure /etc/tendrl/tendrl.conf and run cmd "tendrl_gluster_bridge" or "tendrl_ceph_bridge". Tendrl-api and tendrl-node-agents are not in working condition yet. Are there any other requirements which should be done for installation? Installation of any packages? Do we already need to install for example Etcd? Thank you for answers! -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From adeza at redhat.com Thu Sep 15 06:08:08 2016 From: adeza at redhat.com (Alfredo Deza) Date: Thu, 15 Sep 2016 11:38:08 +0530 Subject: [Tendrl-devel] Missing installation instructions In-Reply-To: References: Message-ID: On Thu, Sep 15, 2016 at 11:11 AM, Martin Kudlej wrote: > Hi all, > > what are installation instructions, please? > > For tendrl/gluster_bridge and tendrl/ceph_bridge, you can do "python > setup.py install" and configure /etc/tendrl/tendrl.conf and run cmd > "tendrl_gluster_bridge" or "tendrl_ceph_bridge". > Tendrl-api and tendrl-node-agents are not in working condition yet. > That type of interacting with python installation is going to cause some problems. What a raw call to `python setup.py install` will do is install the Python package and its dependencies globally. This will cause issues when developing, as it can be very tricky to remove/reinstall Python packages and dependencies. > > Are there any other requirements which should be done for installation? > Installation of any packages? Do we already need to install for example > Etcd? If the Python code is not standalone and requires system dependencies it will be far easier to start producing distro packages and a system that can get them out when developing. > > Thank you for answers! > > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting > @ redhat > #tendrl-devel @ freenode > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From mrugesh at brainfunked.org Thu Sep 15 06:18:17 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Thu, 15 Sep 2016 11:48:17 +0530 Subject: [Tendrl-devel] Missing installation instructions In-Reply-To: References: Message-ID: On 15 September 2016 at 11:11, Martin Kudlej wrote: > > Hi all, > > what are installation instructions, please? > > For tendrl/gluster_bridge and tendrl/ceph_bridge, you can do "python setup.py install" and configure /etc/tendrl/tendrl.conf and run cmd "tendrl_gluster_bridge" or "tendrl_ceph_bridge". > Tendrl-api and tendrl-node-agents are not in working condition yet. > > > Are there any other requirements which should be done for installation? Installation of any packages? Do we already need to install for example Etcd? > > Thank you for answers! Would you mind opening issues on both the ceph_bridge and the gluster_bridge repositories to address this? We need to have the appropriate documentation in place to ensure that the existing code can be run and tried out. Mrugesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkudlej at redhat.com Thu Sep 15 06:33:48 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Thu, 15 Sep 2016 08:33:48 +0200 Subject: [Tendrl-devel] Missing installation instructions In-Reply-To: References: Message-ID: Hi all, On 09/15/2016 08:18 AM, Mrugesh Karnik wrote: > On 15 September 2016 at 11:11, Martin Kudlej > wrote: >> >> Hi all, >> >> what are installation instructions, please? >> >> For tendrl/gluster_bridge and tendrl/ceph_bridge, you can do "python setup.py install" and configure /etc/tendrl/tendrl.conf and run cmd "tendrl_gluster_bridge" or "tendrl_ceph_bridge". >> Tendrl-api and tendrl-node-agents are not in working condition yet. >> >> >> Are there any other requirements which should be done for installation? Installation of any packages? Do we already need to install for example Etcd? >> >> Thank you for answers! > > Would you mind opening issues on both the ceph_bridge and the gluster_bridge repositories to address > this? We need to have the appropriate documentation in place to ensure that the existing code can be > run and tried out. > > Mrugesh filed in upstream Jira https://tendrl.atlassian.net/browse/TEN-36 -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From sankarshan.mukhopadhyay at gmail.com Thu Sep 15 14:12:46 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 15 Sep 2016 19:42:46 +0530 Subject: [Tendrl-devel] first pass at a definition of "done" Message-ID: is a first pass which attempts to note down the qualities and characteristics which can lead to a definition of done. Note: (a) this is a first pass to initiate conversation (b) the final form can and will emerge out of conversations (c) a good part of the list is 'aspirational' and won't fit the first few builds and releases -- sankarshan mukhopadhyay From sankarshan.mukhopadhyay at gmail.com Thu Sep 15 16:45:29 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 15 Sep 2016 22:15:29 +0530 Subject: [Tendrl-devel] Missing installation instructions In-Reply-To: References: Message-ID: On Thu, Sep 15, 2016 at 11:48 AM, Mrugesh Karnik wrote: > On 15 September 2016 at 11:11, Martin Kudlej wrote: >> >> Hi all, >> >> what are installation instructions, please? >> >> For tendrl/gluster_bridge and tendrl/ceph_bridge, you can do "python >> setup.py install" and configure /etc/tendrl/tendrl.conf and run cmd >> "tendrl_gluster_bridge" or "tendrl_ceph_bridge". >> Tendrl-api and tendrl-node-agents are not in working condition yet. >> >> >> Are there any other requirements which should be done for installation? >> Installation of any packages? Do we already need to install for example >> Etcd? >> >> Thank you for answers! > > Would you mind opening issues on both the ceph_bridge and the gluster_bridge > repositories to address this? We need to have the appropriate documentation > in place to ensure that the existing code can be run and tried out. > Now tracked via -- sankarshan mukhopadhyay From kdreyer at redhat.com Thu Sep 15 16:50:29 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Thu, 15 Sep 2016 10:50:29 -0600 Subject: [Tendrl-devel] GitHub vs Jira Message-ID: We have GitHub issues and Jira issues. When should I use one or the other? - Ken From sankarshan.mukhopadhyay at gmail.com Thu Sep 15 16:56:00 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 15 Sep 2016 22:26:00 +0530 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: Message-ID: On Thu, Sep 15, 2016 at 10:20 PM, Ken Dreyer wrote: > We have GitHub issues and Jira issues. When should I use one or the other? vide Github Issues - This is the primary ?repository? for tracking all the proposed changes to any of the sub-projects' codebases. Every Pull Request must have an associated Issue. -- sankarshan mukhopadhyay From japplewh at redhat.com Thu Sep 15 21:00:20 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Thu, 15 Sep 2016 17:00:20 -0400 Subject: [Tendrl-devel] first pass at a definition of "done" In-Reply-To: References: Message-ID: Looks excellent - commented on github On Thu, Sep 15, 2016 at 10:12 AM, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > is a first pass > which attempts to note down the qualities and characteristics which > can lead to a definition of done. > > Note: > > (a) this is a first pass to initiate conversation > (b) the final form can and will emerge out of conversations > (c) a good part of the list is 'aspirational' and won't fit the first > few builds and releases > > > -- > sankarshan mukhopadhyay > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager -------------- next part -------------- An HTML attachment was scrubbed... URL: From japplewh at redhat.com Thu Sep 15 21:05:44 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Thu, 15 Sep 2016 17:05:44 -0400 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: Message-ID: the "issues" name in Jira is somewhat misleading because it really encompasses a user story (as a child component of an epic) and tasks can be a yet smaller work item. Jira has a long history in the service of ticketing and whatnot so the naming makes sense to them. It doesn't make sense from the standpoint of describing a feature or unit of work so well. How's this? Github issues = issues Jira issues = user stories On Thu, Sep 15, 2016 at 12:56 PM, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Thu, Sep 15, 2016 at 10:20 PM, Ken Dreyer wrote: > > We have GitHub issues and Jira issues. When should I use one or the > other? > > vide development-workflow.adoc> > > Github Issues - This is the primary ?repository? for tracking all the > proposed changes to any of the sub-projects' codebases. Every Pull > Request must have an associated Issue. > > > > -- > sankarshan mukhopadhyay > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager -------------- next part -------------- An HTML attachment was scrubbed... URL: From kdreyer at redhat.com Thu Sep 15 21:17:18 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Thu, 15 Sep 2016 15:17:18 -0600 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: Message-ID: On Thu, Sep 15, 2016 at 3:05 PM, Jeff Applewhite wrote: > Github issues = issues > Jira issues = user stories Thanks Jeff, that makes it clearer. - Ken From julim at redhat.com Thu Sep 15 21:39:32 2016 From: julim at redhat.com (Ju Lim) Date: Thu, 15 Sep 2016 17:39:32 -0400 Subject: [Tendrl-devel] first pass at a definition of "done" In-Reply-To: References: Message-ID: Looks very promising and I will be cautiously optimistic! I've added my comments on github. On Thu, Sep 15, 2016 at 5:00 PM, Jeff Applewhite wrote: > Looks excellent - commented on github > > On Thu, Sep 15, 2016 at 10:12 AM, Sankarshan Mukhopadhyay < > sankarshan.mukhopadhyay at gmail.com> wrote: > >> is a first pass >> which attempts to note down the qualities and characteristics which >> can lead to a definition of done. >> >> Note: >> >> (a) this is a first pass to initiate conversation >> (b) the final form can and will emerge out of conversations >> (c) a good part of the list is 'aspirational' and won't fit the first >> few builds and releases >> >> >> -- >> sankarshan mukhopadhyay >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > -- > Jeff Applewhite > Principal Product Manager > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkudlej at redhat.com Fri Sep 16 11:59:47 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Fri, 16 Sep 2016 13:59:47 +0200 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: Message-ID: <9918bb42-b216-5bb3-cab3-b01d02edeca3@redhat.com> Hi Jeff, all, [comments inline] On 09/15/2016 11:05 PM, Jeff Applewhite wrote: > the "issues" name in Jira is somewhat misleading because it really encompasses a user story (as a > child component of an epic) and tasks can be a yet smaller work item. > > Jira has a long history in the service of ticketing and whatnot so the naming makes sense to them. > It doesn't make sense from the standpoint of describing a feature or unit of work so well. > > How's this? > > Github issues = issues > Jira issues = user stories > > On Thu, Sep 15, 2016 at 12:56 PM, Sankarshan Mukhopadhyay > wrote: > > On Thu, Sep 15, 2016 at 10:20 PM, Ken Dreyer > wrote: > > We have GitHub issues and Jira issues. When should I use one or the other? > > vide > > > Github Issues - This is the primary ?repository? for tracking all the > proposed changes to any of the sub-projects' codebases. Every Pull > Request must have an associated Issue. I would like to ask what are benefits to have 2 tracking system for one project? From my perspective 2 tracking systems are good when we would like to separate planning and issue tracking and there is potential plan to change planning system. Is there any (potential) plan to change planning system? On the other hand I see from my perspective some reasons why to use only one tracking system: 1) If we use only one tracking system there is only one place for planning discussion and at same time technical discussion. There is only one place where contributors can search for answers. 2) Document about current workflow for tracking issues/stories is OK from my perspective but I see some weak place there. For example how will we deal with splitting of imported issues from Github because it cannot be done in one sprint? I expect that synchronization is just one way from Github to Jira so we can hit couple of situations like this because of one way synchronization. 3) Is SCRUM Master aware that we will use 2 tracking systems? I expect that SCRUM Master will be responsible for synchronizing and cleaning both tracking systems according workflow described in https://github.com/Tendrl/documentation/blob/master/development-workflow.adoc Could you please explain me what are benefits of using 2 tracking systems? Will be this workflow effective? If it will be effective, why will be effective? -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From sankarshan.mukhopadhyay at gmail.com Fri Sep 16 12:10:40 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 16 Sep 2016 17:40:40 +0530 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: <9918bb42-b216-5bb3-cab3-b01d02edeca3@redhat.com> References: <9918bb42-b216-5bb3-cab3-b01d02edeca3@redhat.com> Message-ID: On Fri, Sep 16, 2016 at 5:29 PM, Martin Kudlej wrote: > Could you please explain me what are benefits of using 2 tracking systems? > Will be this workflow effective? If it will be effective, why will be > effective? Do you propose an alternative workflow for the developers? -- sankarshan mukhopadhyay From japplewh at redhat.com Fri Sep 16 12:53:40 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Fri, 16 Sep 2016 08:53:40 -0400 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: <9918bb42-b216-5bb3-cab3-b01d02edeca3@redhat.com> Message-ID: As I understand it the intention now is to use Jira for Epics/User Stories/Tasks and to use Github issues for upstream bugs. So in essence the project planning for features is in Jira but bugs are logged "closer to the code" in Github. I think the primary goal for upstream WRT bugs is to have them in a tool that is most likely to be used. As of now the tendrl.org site is in progress but there is not a good clearing house of information on the project. We hope to have an alpha site up next week, but in any case (even if we had the site up with a pointer to Jira) the question remains: where will developers and interested parties using the upstream code most likely want to log a defect. My bet is Github. But please put forward other issues that should be considered. ?/jeff ? On Fri, Sep 16, 2016 at 8:10 AM, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Fri, Sep 16, 2016 at 5:29 PM, Martin Kudlej wrote: > > Could you please explain me what are benefits of using 2 tracking > systems? > > Will be this workflow effective? If it will be effective, why will be > > effective? > > Do you propose an alternative workflow for the developers? > > > -- > sankarshan mukhopadhyay > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager -------------- next part -------------- An HTML attachment was scrubbed... URL: From kdreyer at redhat.com Fri Sep 16 16:17:50 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Fri, 16 Sep 2016 10:17:50 -0600 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: <9918bb42-b216-5bb3-cab3-b01d02edeca3@redhat.com> Message-ID: On Fri, Sep 16, 2016 at 6:10 AM, Sankarshan Mukhopadhyay wrote: > On Fri, Sep 16, 2016 at 5:29 PM, Martin Kudlej wrote: >> Could you please explain me what are benefits of using 2 tracking systems? >> Will be this workflow effective? If it will be effective, why will be >> effective? > > Do you propose an alternative workflow for the developers? > One idea: Use GitHub issues for everything (and use the labeling system to track "epics" specifically) - Ken From gmeno at redhat.com Fri Sep 16 17:10:51 2016 From: gmeno at redhat.com (Gregory Meno) Date: Fri, 16 Sep 2016 10:10:51 -0700 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: <9918bb42-b216-5bb3-cab3-b01d02edeca3@redhat.com> Message-ID: On Fri, Sep 16, 2016 at 9:17 AM, Ken Dreyer wrote: > On Fri, Sep 16, 2016 at 6:10 AM, Sankarshan Mukhopadhyay > wrote: >> On Fri, Sep 16, 2016 at 5:29 PM, Martin Kudlej wrote: >>> Could you please explain me what are benefits of using 2 tracking systems? >>> Will be this workflow effective? If it will be effective, why will be >>> effective? >> >> Do you propose an alternative workflow for the developers? >> > > One idea: > > Use GitHub issues for everything (and use the labeling system to track > "epics" specifically) > > - Ken > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel Alternatively Jira for everything. It's got the features Jeff would need to support his work-flows AND it's not that bad for us engineers. cheers, Gregory From kdreyer at redhat.com Fri Sep 16 17:32:55 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Fri, 16 Sep 2016 11:32:55 -0600 Subject: [Tendrl-devel] GitHub vs Jira In-Reply-To: References: <9918bb42-b216-5bb3-cab3-b01d02edeca3@redhat.com> Message-ID: On Fri, Sep 16, 2016 at 11:10 AM, Gregory Meno wrote: > On Fri, Sep 16, 2016 at 9:17 AM, Ken Dreyer wrote: >> On Fri, Sep 16, 2016 at 6:10 AM, Sankarshan Mukhopadhyay >> wrote: >>> On Fri, Sep 16, 2016 at 5:29 PM, Martin Kudlej wrote: >>>> Could you please explain me what are benefits of using 2 tracking systems? >>>> Will be this workflow effective? If it will be effective, why will be >>>> effective? >>> >>> Do you propose an alternative workflow for the developers? >>> >> >> One idea: >> >> Use GitHub issues for everything (and use the labeling system to track >> "epics" specifically) >> >> - Ken >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > Alternatively Jira for everything. It's got the features Jeff would > need to support his work-flows AND it's not that bad for us engineers. Yep, I'd be fine with that as well. The Libcloud project does this (use Jira for everything, and contributors prefix each GitHub PR by "LIBCLOUD-123" (or whatever the ticket is). They have a bot that updates Jira when the PR gets created. They've disabled GitHub's built-in issue tracker, similar to what Ceph's done for the Ceph GitHub org. - Ken From deb at redhat.com Mon Sep 19 11:38:29 2016 From: deb at redhat.com (Soumya Deb) Date: Mon, 19 Sep 2016 17:08:29 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made Message-ID: Hi, Karnan, Neha and I, spent some time last week to re-evaluate the necessary tool stack and set up the base for the Tendrl frontend development. Most of the time we tend to skip this step in favor of more "pressing" issues, and reach quick resolution by saying, "oh I know... I'll use X cause I already know how to use it, it's popular, and I've used it in my last project too". Breaking free of that custom is harder than it seems, but I think it's for the best of the product. It's not always done-in-one-go process as well. But we took a first stab, and tried documenting it: https://docs.google.com/presentation/d/1t9hiYvvEHbE1QzZbfYamkkGILJWs9LoGa_jJ6PUv8XM/edit This is just the beginning, and would expand to all the tool choices we would eventually make. We'll move the contents to github-wiki for the frontend repo as their final home (slideshow may not be the best solution for the stuff in hand), but opening the thread up to gather feedback around the choices made (or even better - how it's made, or can be done more efficiently). Thanks! P.S: for now, it's open to be commented by anyone with a Red Hat account. Which is a shortcoming - I understand and sorry for that. Till we move to github wiki, please use this thread to raise your thoughts, if you don't have a Red Hat ID. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Mon Sep 19 11:54:57 2016 From: jspray at redhat.com (John Spray) Date: Mon, 19 Sep 2016 12:54:57 +0100 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On Mon, Sep 19, 2016 at 5:08 PM, Soumya Deb wrote: > Hi, > > Karnan, Neha and I, spent some time last week to re-evaluate the necessary > tool stack and set up the base for the Tendrl frontend development. > > Most of the time we tend to skip this step in favor of more "pressing" > issues, and reach quick resolution by saying, "oh I know... I'll use X cause > I already know how to use it, it's popular, and I've used it in my last > project too". > > Breaking free of that custom is harder than it seems, but I think it's for > the best of the product. It's not always done-in-one-go process as well. But > we took a first stab, and tried documenting it: > https://docs.google.com/presentation/d/1t9hiYvvEHbE1QzZbfYamkkGILJWs9LoGa_jJ6PUv8XM/edit > > This is just the beginning, and would expand to all the tool choices we > would eventually make. We'll move the contents to github-wiki for the > frontend repo as their final home (slideshow may not be the best solution > for the stuff in hand), but opening the thread up to gather feedback around > the choices made (or even better - how it's made, or can be done more > efficiently). I'm concerned about adopting node.js on the server side. We already have a C++/Python stack in both Ceph and Gluster -- I think there needs to be a strong case for how node is going to be better than Python for this use case, before adding another language to the server-side stack. Consider the developer who wants to spin up this GUI in his development environment on top of a local Ceph cluster -- folks are much more likely to get involved if they don't have to install a whole extra language runtime and pile of dependencies. Other key things I would like to see considered: * how the technologies chosen will handle keeping the UI in sync with the backend state. For example, a flow chart of how a Ceph OSD state change gets passed up into the GUI would be useful. * how non-specialists can expose storage functionality in the front end: it would be useful to show an example of what it would look like to add a simple page to the UI that exposes some particular knob or piece of information from the underlying storage cluster. John > Thanks! > > P.S: for now, it's open to be commented by anyone with a Red Hat account. > Which is a shortcoming - I understand and sorry for that. Till we move to > github wiki, please use this thread to raise your thoughts, if you don't > have a Red Hat ID. > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From kdreyer at redhat.com Mon Sep 19 15:11:30 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Mon, 19 Sep 2016 09:11:30 -0600 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On Mon, Sep 19, 2016 at 5:54 AM, John Spray wrote: > Consider the developer who wants to spin up this > GUI in his development environment on top of a local Ceph cluster -- > folks are much more likely to get involved if they don't have to > install a whole extra language runtime and pile of dependencies. I agree with this. I had a hard time testing contributions to the Skyring project earlier because the node.js system was unfamiliar to me, and I failed to install the dependencies in my local environment. With the Python Skyring bits it was easier to jump in and fix things. Python has several great backend frameworks, and I imagine Python would encourage Tendrl community adoption in the communities that are already heavily invested in Python (Ceph, OpenStack, etc) - Ken From kdreyer at redhat.com Mon Sep 19 15:15:25 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Mon, 19 Sep 2016 09:15:25 -0600 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On Mon, Sep 19, 2016 at 9:11 AM, Ken Dreyer wrote: > On Mon, Sep 19, 2016 at 5:54 AM, John Spray wrote: >> Consider the developer who wants to spin up this >> GUI in his development environment on top of a local Ceph cluster -- >> folks are much more likely to get involved if they don't have to >> install a whole extra language runtime and pile of dependencies. > > I agree with this. I had a hard time testing contributions to the > Skyring project earlier because the node.js system was unfamiliar to > me, and I failed to install the dependencies in my local environment. > With the Python Skyring bits it was easier to jump in and fix things. Whoops, I meant Skyring's Go system here, for the bits I was thinking about earlier. I didn't even try the node stuff in Kitoon. - Ken From japplewh at redhat.com Mon Sep 19 21:03:57 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Mon, 19 Sep 2016 17:03:57 -0400 Subject: [Tendrl-devel] Kanban board created Message-ID: Hi All I have created a Kanban board here with columns mapping to Ju Lim's proposed categories as per our process discussions ?Thanks ? Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From shtripat at redhat.com Tue Sep 20 04:30:58 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Tue, 20 Sep 2016 10:00:58 +0530 Subject: [Tendrl-devel] Kanban board created In-Reply-To: References: Message-ID: Jeff, I dont have access to the Kanban board here. Tried with kerberos id and it does not allow in. Can you please provide required access to the same? Thanks and Regards, Shubhendu On 09/20/2016 02:33 AM, Jeff Applewhite wrote: > Hi All > > I have created a Kanban board here > with > columns mapping to Ju Lim's proposed categories as per our process > discussions > > ?Thanks > ? > > Jeff > > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Tue Sep 20 04:44:10 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 20 Sep 2016 10:14:10 +0530 Subject: [Tendrl-devel] Kanban board created In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 10:00 AM, Shubhendu Tripathi wrote: > I dont have access to the Kanban board here. > Tried with kerberos id and it does not allow in. > > Can you please provide required access to the same? It is likely that your account needs to be created for the Jira instance which Jeff refers to. The instance has no integration with any Kerberos systems or, other OAuth* providers. -- sankarshan mukhopadhyay From shtripat at redhat.com Tue Sep 20 04:50:27 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Tue, 20 Sep 2016 10:20:27 +0530 Subject: [Tendrl-devel] Kanban board created In-Reply-To: References: Message-ID: <6a6c7794-7726-5d05-bb27-5b576c01bde5@redhat.com> Jeff, I got a mail from Atlassin system that a user is created with "shubendu" and I reset the password fro my account. But still not able to login with credentials :( Regards, Shubhendu On 09/20/2016 10:14 AM, Sankarshan Mukhopadhyay wrote: > On Tue, Sep 20, 2016 at 10:00 AM, Shubhendu Tripathi > wrote: >> I dont have access to the Kanban board here. >> Tried with kerberos id and it does not allow in. >> >> Can you please provide required access to the same? > It is likely that your account needs to be created for the Jira > instance which Jeff refers to. The instance has no integration with > any Kerberos systems or, other OAuth* providers. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Tue Sep 20 04:54:07 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 20 Sep 2016 10:24:07 +0530 Subject: [Tendrl-devel] Kanban board created In-Reply-To: <6a6c7794-7726-5d05-bb27-5b576c01bde5@redhat.com> References: <6a6c7794-7726-5d05-bb27-5b576c01bde5@redhat.com> Message-ID: On Tue, Sep 20, 2016 at 10:20 AM, Shubhendu Tripathi wrote: > I got a mail from Atlassin system that a user is created with "shubendu" and > I reset the password fro my account. > But still not able to login with credentials :( Yikes. That might just be me, rather than Jeff. I was trying to add an account and the system froze on me. Sorry about this. Would you like me to try again, or, prefer that Jeff sort this out? > On 09/20/2016 10:14 AM, Sankarshan Mukhopadhyay wrote: > > On Tue, Sep 20, 2016 at 10:00 AM, Shubhendu Tripathi > wrote: > > I dont have access to the Kanban board here. > Tried with kerberos id and it does not allow in. > > Can you please provide required access to the same? > > It is likely that your account needs to be created for the Jira > instance which Jeff refers to. The instance has no integration with > any Kerberos systems or, other OAuth* providers. > > > -- sankarshan mukhopadhyay From vsarmila at redhat.com Tue Sep 20 04:59:13 2016 From: vsarmila at redhat.com (Sharmilla Abhilash) Date: Tue, 20 Sep 2016 10:29:13 +0530 Subject: [Tendrl-devel] Kanban board created In-Reply-To: References: <6a6c7794-7726-5d05-bb27-5b576c01bde5@redhat.com> Message-ID: On Tue, Sep 20, 2016 at 10:24 AM, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Tue, Sep 20, 2016 at 10:20 AM, Shubhendu Tripathi > wrote: > > > I got a mail from Atlassin system that a user is created with "shubendu" > and > > I reset the password fro my account. > > But still not able to login with credentials :( > > Yikes. That might just be me, rather than Jeff. I was trying to add an > account and the system froze on me. Sorry about this. > > Would you like me to try again, or, prefer that Jeff sort this out? > I have sorted I guess. Will wait for confirmation from Shubhendu. > > > On 09/20/2016 10:14 AM, Sankarshan Mukhopadhyay wrote: > > > > On Tue, Sep 20, 2016 at 10:00 AM, Shubhendu Tripathi > > wrote: > > > > I dont have access to the Kanban board here. > > Tried with kerberos id and it does not allow in. > > > > Can you please provide required access to the same? > > > > It is likely that your account needs to be created for the Jira > > instance which Jeff refers to. The instance has no integration with > > any Kerberos systems or, other OAuth* providers. > > > > > > > > > > -- > sankarshan mukhopadhyay > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Sharmilla Abhilash PgM, Storage -------------- next part -------------- An HTML attachment was scrubbed... URL: From shtripat at redhat.com Tue Sep 20 05:39:49 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Tue, 20 Sep 2016 11:09:49 +0530 Subject: [Tendrl-devel] Kanban board created In-Reply-To: References: <6a6c7794-7726-5d05-bb27-5b576c01bde5@redhat.com> Message-ID: <0ac48f5f-363c-0cad-2447-44550a193fba@redhat.com> On 09/20/2016 10:29 AM, Sharmilla Abhilash wrote: > > > On Tue, Sep 20, 2016 at 10:24 AM, Sankarshan Mukhopadhyay > > wrote: > > On Tue, Sep 20, 2016 at 10:20 AM, Shubhendu Tripathi > > wrote: > > > I got a mail from Atlassin system that a user is created with > "shubendu" and > > I reset the password fro my account. > > But still not able to login with credentials :( > > Yikes. That might just be me, rather than Jeff. I was trying to add an > account and the system froze on me. Sorry about this. > > Would you like me to try again, or, prefer that Jeff sort this out? > > > I have sorted I guess. Will wait for confirmation from Shubhendu. Thanks Sharmilla. Able to login now. > > > On 09/20/2016 10:14 AM, Sankarshan Mukhopadhyay wrote: > > > > On Tue, Sep 20, 2016 at 10:00 AM, Shubhendu Tripathi > > > wrote: > > > > I dont have access to the Kanban board here. > > Tried with kerberos id and it does not allow in. > > > > Can you please provide required access to the same? > > > > It is likely that your account needs to be created for the Jira > > instance which Jeff refers to. The instance has no integration with > > any Kerberos systems or, other OAuth* providers. > > > > > > > > > > -- > sankarshan mukhopadhyay > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > -- > Sharmilla Abhilash > PgM, Storage > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaverma at redhat.com Tue Sep 20 06:31:43 2016 From: kaverma at redhat.com (Kamlesh Verma) Date: Tue, 20 Sep 2016 02:31:43 -0400 (EDT) Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: <130120826.586764.1474353103264.JavaMail.zimbra@redhat.com> Hi, I went through shared slide and discussion. I am concerned about upgrading from angular-1.x to angular-2.0 for the front-end development. We already using angular1.x and typescript, so going forward with angular-2.0, i don't think it will create any problem . and also now angular-2.0 is in final release with stable version. going ahead with this is safe for production. and major key of using 2.0 will be making plugin-play kind of thing will be very easy with angular-2.0's component. I consider this as a major upgrade from Angular 1.x. Angular 2 has very powerful routes. The Angular 2 Router will only load components when it absolutely needs them. Kind of partial loading which is a great feature I think. Angular 2 is 5 times faster as compared to Angular 1.x. i am sharing some link for advantage of using angular-2.0 over 1.x http://www.algoworks.com/blog/benefits-of-angularjs-2-0/ Thanks and Regards Kamlesh kumar verma ----- Original Message ----- From: "Soumya Deb" To: tendrl-devel at redhat.com Sent: Monday, September 19, 2016 5:08:29 PM Subject: [Tendrl-devel] Frontend tool stack: choices made Hi, Karnan, Neha and I, spent some time last week to re-evaluate the necessary tool stack and set up the base for the Tendrl frontend development. Most of the time we tend to skip this step in favor of more "pressing" issues, and reach quick resolution by saying, "oh I know... I'll use X cause I already know how to use it, it's popular, and I've used it in my last project too". Breaking free of that custom is harder than it seems, but I think it's for the best of the product. It's not always done-in-one-go process as well. But we took a first stab, and tried documenting it: https://docs.google.com/presentation/d/1t9hiYvvEHbE1QzZbfYamkkGILJWs9LoGa_jJ6PUv8XM/edit This is just the beginning, and would expand to all the tool choices we would eventually make. We'll move the contents to github-wiki for the frontend repo as their final home (slideshow may not be the best solution for the stuff in hand), but opening the thread up to gather feedback around the choices made (or even better - how it's made, or can be done more efficiently). Thanks! P.S: for now, it's open to be commented by anyone with a Red Hat account. Which is a shortcoming - I understand and sorry for that. Till we move to github wiki, please use this thread to raise your thoughts, if you don't have a Red Hat ID. _______________________________________________ Tendrl-devel mailing list Tendrl-devel at redhat.com https://www.redhat.com/mailman/listinfo/tendrl-devel From japplewh at redhat.com Tue Sep 20 20:43:01 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 20 Sep 2016 16:43:01 -0400 Subject: [Tendrl-devel] bugs Message-ID: Hi All I'd like to gage people's thoughts on the proper place for bugs in Tendrl. Among the options are: ------------------------------------- *Pros/Cons* - *Jira* Pros: close to the project management side of things, good history for this type of function in Jira, ? Cons: further from the code, medium feature capability, ? - *Github issues* Pros: closer to the code (and the developers), ? Cons: quite simple, lack of features, ? - *Bugzilla** Pros: full featured, well known, eases the pain for downstream (duplicative reduction/no sync needed), ? Cons: Further from the code, not integrated into Jira, ? ------------------------------------- I want to hear people's honest opinion of the pros/cons of each so we make a good choice. Please add to or correct the pros/cons list. (*In the Bugzilla case the proposal is to do something similar to upstream OpenShift Origin where there is an upstream component within Bugzilla specifically for the Tendrl codebase, *not the downstream release.) Thanks Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From kdreyer at redhat.com Tue Sep 20 20:54:08 2016 From: kdreyer at redhat.com (Ken Dreyer) Date: Tue, 20 Sep 2016 14:54:08 -0600 Subject: [Tendrl-devel] bugs In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 2:43 PM, Jeff Applewhite wrote: > Pros: closer to the code (and the developers), ? > Cons: quite simple, lack of features, ? GitHub gets my vote for being simple, faster than Bugzilla, and one less account to manage compared to Jira. Large projects like Rails use it. It integrates easily with Trello and has advanced features like labels, milestones, and templates. https://guides.github.com/features/issues/ - Ken From nlevine at redhat.com Tue Sep 20 21:05:41 2016 From: nlevine at redhat.com (Neil Levine) Date: Tue, 20 Sep 2016 14:05:41 -0700 Subject: [Tendrl-devel] bugs In-Reply-To: References: Message-ID: If we want external developer interaction, my vote is for Github as this is where all the are developers already. The least friction to interaction and participation should be the priority. Jira will force people to sign up to a new account while Bugzilla is well...bugzilla :) On Tue, Sep 20, 2016 at 1:54 PM, Ken Dreyer wrote: > On Tue, Sep 20, 2016 at 2:43 PM, Jeff Applewhite > wrote: > > Pros: closer to the code (and the developers), ? > > Cons: quite simple, lack of features, ? > > GitHub gets my vote for being simple, faster than Bugzilla, and one > less account to manage compared to Jira. > > Large projects like Rails use it. It integrates easily with Trello and > has advanced features like labels, milestones, and templates. > > https://guides.github.com/features/issues/ > > - Ken > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From julim at redhat.com Tue Sep 20 21:53:29 2016 From: julim at redhat.com (Ju Lim) Date: Tue, 20 Sep 2016 17:53:29 -0400 Subject: [Tendrl-devel] bugs In-Reply-To: References: Message-ID: +1 on GitHub as that's also how openshift, manageiq, and cockpit deal with bugs. On Sep 20, 2016 5:06 PM, "Neil Levine" wrote: > If we want external developer interaction, my vote is for Github as this > is where all the are developers already. The least friction to interaction > and participation should be the priority. Jira will force people to sign up > to a new account while Bugzilla is well...bugzilla :) > > On Tue, Sep 20, 2016 at 1:54 PM, Ken Dreyer wrote: > >> On Tue, Sep 20, 2016 at 2:43 PM, Jeff Applewhite >> wrote: >> > Pros: closer to the code (and the developers), ? >> > Cons: quite simple, lack of features, ? >> >> GitHub gets my vote for being simple, faster than Bugzilla, and one >> less account to manage compared to Jira. >> >> Large projects like Rails use it. It integrates easily with Trello and >> has advanced features like labels, milestones, and templates. >> >> https://guides.github.com/features/issues/ >> >> - Ken >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cblum at redhat.com Wed Sep 21 03:34:50 2016 From: cblum at redhat.com (Christopher Blum) Date: Wed, 21 Sep 2016 05:34:50 +0200 Subject: [Tendrl-devel] bugs In-Reply-To: References: Message-ID: In case my vote counts...i vote for github as well ;) Would Jira even be a viable product for an open source community? Who would pay for the license? Additionally github just released a major product management update for its platform: https://techcrunch.com/2016/09/14/github-gets-built-in-project-management-tools-and-support-for-reviews - Chris On Sep 20, 2016 23:53, "Ju Lim" wrote: > +1 on GitHub as that's also how openshift, manageiq, and cockpit deal with > bugs. > > On Sep 20, 2016 5:06 PM, "Neil Levine" wrote: > >> If we want external developer interaction, my vote is for Github as this >> is where all the are developers already. The least friction to interaction >> and participation should be the priority. Jira will force people to sign up >> to a new account while Bugzilla is well...bugzilla :) >> >> On Tue, Sep 20, 2016 at 1:54 PM, Ken Dreyer wrote: >> >>> On Tue, Sep 20, 2016 at 2:43 PM, Jeff Applewhite >>> wrote: >>> > Pros: closer to the code (and the developers), ? >>> > Cons: quite simple, lack of features, ? >>> >>> GitHub gets my vote for being simple, faster than Bugzilla, and one >>> less account to manage compared to Jira. >>> >>> Large projects like Rails use it. It integrates easily with Trello and >>> has advanced features like labels, milestones, and templates. >>> >>> https://guides.github.com/features/issues/ >>> >>> - Ken >>> >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From avishwan at redhat.com Wed Sep 21 06:42:09 2016 From: avishwan at redhat.com (Aravinda) Date: Wed, 21 Sep 2016 12:12:09 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: "Static vs Dynamic" is not making sense here, I think it should be "Client side rendering" vs "Server side rendering", Please correct me if I understood it wrong.From USMv2 project repo(https://github.com/skyrings/kitoon/), I understand that it is using Angular JS for rendering required content based on data. If the UI is already driven by data, how it can be called as Static? Even with the use of dynamic/scripting language at server side, we are not avoiding Client side templates or rendering because we need not refresh the page every time to get new content from server side. Server side should be as light as possible, dealing with data, authentication and pushing notifications to client/UI via Websockets. regards Aravinda On Monday 19 September 2016 05:08 PM, Soumya Deb wrote: > Hi, > > Karnan, Neha and I, spent some time last week to re-evaluate the > necessary tool stack and set up the base for the Tendrl frontend > development. > > Most of the time we tend to skip this step in favor of more "pressing" > issues, and reach quick resolution by saying, "oh I know... I'll use X > cause I already know how to use it, it's popular, and I've used it in > my last project too". > > Breaking free of that custom is harder than it seems, but I think it's > for the best of the product. It's not always done-in-one-go process as > well. But we took a first stab, and tried documenting it: > https://docs.google.com/presentation/d/1t9hiYvvEHbE1QzZbfYamkkGILJWs9LoGa_jJ6PUv8XM/edit > > This is just the beginning, and would expand to all the tool choices > we would eventually make. We'll move the contents to github-wiki for > the frontend repo as their final home (slideshow may not be the best > solution for the stuff in hand), but opening the thread up to gather > feedback around the choices made (or even better - how it's made, or > can be done more efficiently). > > Thanks! > > P.S: for now, it's open to be commented by anyone with a Red Hat > account. Which is a shortcoming - I understand and sorry for that. > Till we move to github wiki, please use this thread to raise your > thoughts, if you don't have a Red Hat ID. > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Wed Sep 21 07:31:57 2016 From: jspray at redhat.com (John Spray) Date: Wed, 21 Sep 2016 08:31:57 +0100 Subject: [Tendrl-devel] bugs In-Reply-To: References: Message-ID: On Tue, Sep 20, 2016 at 9:43 PM, Jeff Applewhite wrote: > Hi All > > I'd like to gage people's thoughts on the proper place for bugs in Tendrl. > Among the options are: > ------------------------------------- > > Pros/Cons > > Jira > > Pros: close to the project management side of things, good history for this > type of function in Jira, ? > Cons: further from the code, medium feature capability, ? > > Github issues > > Pros: closer to the code (and the developers), ? > Cons: quite simple, lack of features, ? > > Bugzilla* > > Pros: full featured, well known, eases the pain for downstream (duplicative > reduction/no sync needed), ? > Cons: Further from the code, not integrated into Jira, ? > > ------------------------------------- > > I want to hear people's honest opinion of the pros/cons of each so we make a > good choice. Please add to or correct the pros/cons list. I don't mind which tool, as long as you don't have all three! If github or bugzilla is used for bug reports, then I don't think it's going to be helpful to have Jira in the mix as well. Even if it has nice project management features, it is not worth having the same ticket potentially in three trackers (bugzilla for Red Hat, Jira for Tendrl planning, and github for writing code). IMHO github issues or jira would both be reasonable candidates for elimination from the picture. Eliminating Jira would probably save the most effort as it's an entirely separate platform. John > (*In the Bugzilla case the proposal is to do something similar to upstream > OpenShift Origin where there is an upstream component within Bugzilla > specifically for the Tendrl codebase, *not the downstream release.) > > > Thanks > > Jeff > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From mkudlej at redhat.com Wed Sep 21 08:21:15 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Wed, 21 Sep 2016 10:21:15 +0200 Subject: [Tendrl-devel] bugs In-Reply-To: References: Message-ID: <90c54738-03f9-8159-23e7-2ccdfa5a4ed8@redhat.com> Hi all, Jira: pros: it scales better than Trello, planning features are much better than in Github and Bugzilla cons: it is not close to code Bugzilla: Well here I'm not that objective because I'm used to work with BZ and I can do there planning and issue tracking. pros: it's full featured issue tracking system, there are couple of tools integrated with it cons: it's not close to code and it's hard to do planning there Github: pros: close to code cons: lack of planning features for project like Tendrl. need to synchronize with external tools. I vote for Jira as single point for storing everything about Tendrl. I think planning and issue tracking should be on one place because synchronization between 2 or more tools can generate mistakes, mess and additional effort. -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From julim at redhat.com Wed Sep 21 15:22:36 2016 From: julim at redhat.com (Ju Lim) Date: Wed, 21 Sep 2016 11:22:36 -0400 Subject: [Tendrl-devel] Tendrl: Sprint 1 Planning - Brief Summary and Action Items Message-ID: Team: Just a quick summary from today's Tendrl Sprint 001 Planning Meeting: - Sprint #: 001 - Sprint Duration: 21 Sept 2016 - 4 Oct 2016 - Sprint Goal: Solidify framework with showcasing one basic flow, i.e. create basic Gluster volume using API, create Ceph pool using API Action Items: - Nishant and Team will work on assigning tasks and stories/tasks will be assigned before Thu, Sept 22 Check-in - Martin will create a user story to capture the GitHub issues related to installation instructions for the components, and will be added to the Next column in the Tendrl Kanban Board . - All work being performed in this Sprint/cycle should be visible in the Tendrl Kanban Board . - As folks start working on the user stories/tasks, please ensure it's assigned (to you), has an acceptance criteria (if it doesn't, please raise this as an issue), move it to "In Progress" column. - For any demos planned and/or discussions that should be known to others, please ensure they appear on the RHSC shared/public calendar . - If you are blocked on any of your work, please raise the issue immediately, i.e. email, IRC, etc. A couple things to note: - I've added "prefixes" to certain user stories/tasks labels (in addition to the labels being used) to make it easier to identify SPIKE, UX, and QE specific tasks in the Kanban board. - IRC ?Walk-in? Hours (Timezone Translator ): 8am-9am US/ET, 5:30pm-6:30pm IST, 2pm-3pm CST - Upstream: #tendrl-devel on Freenode - Downstream: #usm-meeting If folks have any questions/concerns, please feel free to reach out. Thank you all for your participation, and I look forward to seeing the progress we make this Sprint! Thank you, Ju P.S. The attachment is a snapshot of the Kanban board for this Sprint and all the epics/user stories/tasks currently captured. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Tendrl Sprint 1 20160921.png Type: image/png Size: 358103 bytes Desc: not available URL: From japplewh at redhat.com Wed Sep 21 19:45:04 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Wed, 21 Sep 2016 15:45:04 -0400 Subject: [Tendrl-devel] bugs In-Reply-To: <90c54738-03f9-8159-23e7-2ccdfa5a4ed8@redhat.com> References: <90c54738-03f9-8159-23e7-2ccdfa5a4ed8@redhat.com> Message-ID: Hi All Just a couple points of clarification. First of all the Jira hosting was obtained free of charge through Atlassian. I applied for their Open Source license request[1] and it was granted. So they demonstrate support of Open Source development at least in this fashion. Their source is not open (but then neither is Github itself). I am not aware of any Github policy WRT open source projects and free hosting for project management functionality (but then it is quite new). Secondly there is no requirement to have a Jira account to *view epics, stories, tasks, bugs, etc. in Jira because that is one of the requirements for participating in the open source program. They require that you modify your project to be open to anonymous viewers and I have done that. If you want to *participate in the development, testing, or be assigned to a story, etc. you will need an account and those accounts are unlimited with the licensing we have. If that is a significant barrier to developer participation then I tend to doubt the eagerness of said developer to actually contribute. It literally takes an email to this list and one of the 4 or 5 admins we have will create the account. They will get an email, set their password, and login. Simple as that. On bugs.. It seems the preponderance of commenters prefer Github issues for speed and developer friendliness. Let software continue to eat the world.. Let's call it case closed on that. Thanks Jeff [1] https://www.atlassian.com/software/views/open-source-license-request On Wed, Sep 21, 2016 at 4:21 AM, Martin Kudlej wrote: > Hi all, > > Jira: > pros: it scales better than Trello, planning features are much better than > in Github and Bugzilla > cons: it is not close to code > > Bugzilla: > Well here I'm not that objective because I'm used to work with BZ and I > can do there planning and issue tracking. > pros: it's full featured issue tracking system, there are couple of tools > integrated with it > cons: it's not close to code and it's hard to do planning there > > Github: > pros: close to code > cons: lack of planning features for project like Tendrl. need to > synchronize with external tools. > > I vote for Jira as single point for storing everything about Tendrl. I > think planning and issue tracking should be on one place because > synchronization between 2 or more tools can generate mistakes, mess and > additional effort. > > > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, > #usm-meeting @ redhat > #tendrl-devel @ freenode > -- Jeff Applewhite Principal Product Manager -------------- next part -------------- An HTML attachment was scrubbed... URL: From deb at redhat.com Thu Sep 22 04:31:32 2016 From: deb at redhat.com (Soumya Deb) Date: Thu, 22 Sep 2016 10:01:32 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: Hi all, Took me a while to get back; mostly to figure out how to take the discussion forward. First thing first, in future, we'll not use google docs to host information like this, but somewhere more accessible, like in our GitHub repos' issues. More ripe and resolved discussions can become part of the documentation repo itself. I also find mono-threaded email-lists a bit primitive tool for branching out conversations (where there's a need for it), but that's out of the scope to discuss at the moment. To get back to the thread, and try addressing the points made (thanks for chipping in, btw), here are the topics that came out (to avoid multiple inline replies): 0. Why node.js? Why not Python (or Ruby, or X)? To answer simply, Node.js is the leanest environment for a frontend stack, without compromising on features, while being nice to available resources. Also to note, for frontend coding, JS is unavoidable. The "adding yet another language" argument doesn't hold here, as JS is the only one we're trying to stick to. In fact, a python framework for serving frontend would actually be "yet another language" scenario. Please do not confuse this stack as the main backend stack by any means - Node.js will be used to serve the frontend only, and won't be used in the backend (tendrl common/core). They'll be built into different packages as well. 1. I am concerned about upgrading from angular-1.x to angular-2... Hold on, using/continuing with angular (as it's mentioned) hasn't been confirmed. Which version of angular will be used heavily relies on the fact - whether we'll actually use angular.js. This will be figured out in a week or so. Also, you didn't specify which concern you have with the version update (in fact, your points sound pretty positive in favor of it). 2. Other key (architecture and API related) things I would like to see considered... Those discussion resides in their rightful places (hopefully). Mrugesh should be able to provide more lights about them (at any point), if you have specific concerns. The part which needs frontend team's understanding and inputs, we'll be happy to provide, as required. 3. "Static vs Dynamic" is not making sense here... Let's not confuse between DHTML and dynamic server responses. Let's also not equate serving USM 2.0 frontend and usage of Node.js/npm therein, with this tendrl 1.0. They're gonna be quite different. We'll be moving onto github issues (https://github.com/Tendrl/ui repo, likely to be renamed to tendrl_frontend repo) for further discussions on particular points. We're gonna put the issues up there shortly. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Thu Sep 22 04:52:39 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 22 Sep 2016 10:22:39 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On Thu, Sep 22, 2016 at 10:01 AM, Soumya Deb wrote: > We'll be moving onto github issues (https://github.com/Tendrl/ui repo, > likely to be renamed to tendrl_frontend repo) for further discussions on > particular points. We're gonna put the issues up there shortly. Is this something which is decided upon? I think I could make the repo-name-change bits happen right away. -- sankarshan mukhopadhyay From deb at redhat.com Thu Sep 22 04:58:51 2016 From: deb at redhat.com (Soumya Deb) Date: Thu, 22 Sep 2016 10:28:51 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On 22 September 2016 at 10:22, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Thu, Sep 22, 2016 at 10:01 AM, Soumya Deb wrote: > > We'll be moving onto github issues (https://github.com/Tendrl/ui repo, > > likely to be renamed to tendrl_frontend repo) for further discussions on > > particular points. We're gonna put the issues up there shortly. > > Is this something which is decided upon? I think I could make the > repo-name-change bits happen right away. > https://github.com/Tendrl/ui/issues/1 captures this. Unless we're blocked on some particular go/no-go, this change seems agreeable to the frontend team at least. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Thu Sep 22 09:01:44 2016 From: jspray at redhat.com (John Spray) Date: Thu, 22 Sep 2016 10:01:44 +0100 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On Thu, Sep 22, 2016 at 5:31 AM, Soumya Deb wrote: > Hi all, > > Took me a while to get back; mostly to figure out how to take the discussion > forward. > > First thing first, in future, we'll not use google docs to host information > like this, but somewhere more accessible, like in our GitHub repos' issues. > More ripe and resolved discussions can become part of the documentation repo > itself. I also find mono-threaded email-lists a bit primitive tool for > branching out conversations (where there's a need for it), but that's out of > the scope to discuss at the moment. > > To get back to the thread, and try addressing the points made (thanks for > chipping in, btw), here are the topics that came out (to avoid multiple > inline replies): > > 0. Why node.js? Why not Python (or Ruby, or X)? > To answer simply, Node.js is the leanest environment for a frontend stack, > without compromising on features, while being nice to available resources. > Also to note, for frontend coding, JS is unavoidable. The "adding yet > another language" argument doesn't hold here, as JS is the only one we're > trying to stick to. In fact, a python framework for serving frontend would > actually be "yet another language" scenario. Please do not confuse this > stack as the main backend stack by any means - Node.js will be used to serve > the frontend only, and won't be used in the backend (tendrl common/core). > They'll be built into different packages as well. The key point here is the added *runtime*. For frontend javascript you already have a runtime built into the browser. On the backend, you're going to be imposing a requirement for node and all its dependencies. Frontend/backend also involves different tools etc (I know how to do right click->inspect, I do not know all the command line equivalents). I'm not terribly convinced by "node is the leanest". A lot of people have written a lot of web application backends in python, and continue to do so. However, I am not actually clear on what your backend is going to do, other than serving up the assets to the UI. Are you implementing a specialised REST API for your client-side code? Something else? This comes back to the fact that we're having a conversation about the "how", without having really seen a description of the "what". It would be helpful for you to describe what you envision the backend code doing. John > > 1. I am concerned about upgrading from angular-1.x to angular-2... > Hold on, using/continuing with angular (as it's mentioned) hasn't been > confirmed. Which version of angular will be used heavily relies on the fact > - whether we'll actually use angular.js. This will be figured out in a week > or so. Also, you didn't specify which concern you have with the version > update (in fact, your points sound pretty positive in favor of it). > > 2. Other key (architecture and API related) things I would like to see > considered... > Those discussion resides in their rightful places (hopefully). Mrugesh > should be able to provide more lights about them (at any point), if you have > specific concerns. The part which needs frontend team's understanding and > inputs, we'll be happy to provide, as required. > > 3. "Static vs Dynamic" is not making sense here... > Let's not confuse between DHTML and dynamic server responses. Let's also not > equate serving USM 2.0 frontend and usage of Node.js/npm therein, with this > tendrl 1.0. They're gonna be quite different. > > > We'll be moving onto github issues (https://github.com/Tendrl/ui repo, > likely to be renamed to tendrl_frontend repo) for further discussions on > particular points. We're gonna put the issues up there shortly. > > Thanks! > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From deb at redhat.com Thu Sep 22 15:06:55 2016 From: deb at redhat.com (Soumya Deb) Date: Thu, 22 Sep 2016 20:36:55 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On 22 September 2016 at 14:31, John Spray wrote: > > The key point here is the added *runtime*. For frontend javascript > you already have a runtime built into the browser. On the backend, > you're going to be imposing a requirement for node and all its > dependencies. Frontend/backend also involves different tools etc (I > know how to do right click->inspect, I do not know all the command > line equivalents). > > I'm not terribly convinced by "node is the leanest". A lot of people > have written a lot of web application backends in python, and continue > to do so. > > However, I am not actually clear on what your backend is going to do, > other than serving up the assets to the UI. Are you implementing a > specialised REST API for your client-side code? Something else? This > comes back to the fact that we're having a conversation about the > "how", without having really seen a description of the "what". It > would be helpful for you to describe what you envision the backend > code doing. > Hey John, Yes, you're correct about the terminology; we were indeed talking about the runtime (along with the platform). However, using node for the frontend deployment should now impose any restriction to the backend at all. They communicate over HTTP(S), and that should be same no matter which platform/runtime we use. The Node.js stack in our discussion is something that stands between the user's client/browser and the API server (the upper layer of backend server that provides data), as long as it makes things easy. In cases when it doesn't, it allows the client directly communicating to the API server. Please check https://github.com/Tendrl/ui/issues/7 for some more contexts on this. Now, about Node.js itself... it runs pretty smooth on a 256mb VM with single core and can serve hundreds of average requests per minute. I'm pretty sure that's as good as it gets when compared for being graceful to system, against other competing php, python or ruby frameworks. It's pretty lean for the features and richness it provides. For the sake of argument, we can compare it with "camping" on Ruby or "bottle" on Python - but we know how limited those alternatives are. To address what are we about to do with it; yes you've made some correct guesses there. Our frontend implemented on REST for the browser/client to communicate with the server, we will have sophisticated routes handling in the server side as well, many of the non-generic error/failure handling can be implemented more easily (in cases where tendrl's backend provides machine readable errors, but we can process that into multilanguage supported humanly understood error-msg/warnings etc), abstracting out sensitive backend endpoints, implementing more login strategies more efficiently, mock APIs for building UI while the backend API isn't ready or to test features and many more reasons that come into play why having a server side scripting environment will come in handy. We've set up our discussion process on GitHub, and requesting all to follow https://github.com/Tendrl/ui/issues/5 going forward. https://github.com/Tendrl/ui/issues/9 in particular talks about the selection of server runtime - to take the discussion forward. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Thu Sep 22 16:58:24 2016 From: jspray at redhat.com (John Spray) Date: Thu, 22 Sep 2016 17:58:24 +0100 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On Thu, Sep 22, 2016 at 4:06 PM, Soumya Deb wrote: > On 22 September 2016 at 14:31, John Spray wrote: >> >> >> The key point here is the added *runtime*. For frontend javascript >> you already have a runtime built into the browser. On the backend, >> you're going to be imposing a requirement for node and all its >> dependencies. Frontend/backend also involves different tools etc (I >> know how to do right click->inspect, I do not know all the command >> line equivalents). >> >> I'm not terribly convinced by "node is the leanest". A lot of people >> have written a lot of web application backends in python, and continue >> to do so. >> >> However, I am not actually clear on what your backend is going to do, >> other than serving up the assets to the UI. Are you implementing a >> specialised REST API for your client-side code? Something else? This >> comes back to the fact that we're having a conversation about the >> "how", without having really seen a description of the "what". It >> would be helpful for you to describe what you envision the backend >> code doing. > > > Hey John, > > Yes, you're correct about the terminology; we were indeed talking about the > runtime (along with the platform). > > However, using node for the frontend deployment should now impose any > restriction to the backend at all. They communicate over HTTP(S), and that > should be same no matter which platform/runtime we use. The Node.js stack in > our discussion is something that stands between the user's client/browser > and the API server (the upper layer of backend server that provides data), > as long as it makes things easy. In cases when it doesn't, it allows the > client directly communicating to the API server. Please check > https://github.com/Tendrl/ui/issues/7 for some more contexts on this. > > Now, about Node.js itself... it runs pretty smooth on a 256mb VM with single > core and can serve hundreds of average requests per minute. I'm pretty sure > that's as good as it gets when compared for being graceful to system, > against other competing php, python or ruby frameworks. It's pretty lean for > the features and richness it provides. For the sake of argument, we can > compare it with "camping" on Ruby or "bottle" on Python - but we know how > limited those alternatives are. > > To address what are we about to do with it; yes you've made some correct > guesses there. Our frontend implemented on REST for the browser/client to > communicate with the server, we will have sophisticated routes handling in > the server side as well, many of the non-generic error/failure handling can > be implemented more easily (in cases where tendrl's backend provides machine > readable errors, but we can process that into multilanguage supported > humanly understood error-msg/warnings etc), abstracting out sensitive > backend endpoints, implementing more login strategies more efficiently, mock > APIs for building UI while the backend API isn't ready or to test features > and many more reasons that come into play why having a server side scripting > environment will come in handy. So to put this into context with [1], it seems like the stack looks like: UI code (Javascript, running in browser) || v Server-side UI support code (Javascript, running in node.js on a Tendrl server node) || v Server-side Tendrl Application Instance (Unknown language) || v Tendrl Central Store (Unknown service, probably etcd?) || v Tendrl Node Agent (Python, running on Ceph storage node) || v Tendrl SDS Bridge (Python, running on Ceph storage node) || v Ceph (Python/C++) Can you see why I'm concerned about the number of moving parts here? It's not so much a question of the merits of any one of them, as it is the resulting total complexity. It's not always easy to identify pieces to cut out, but the server side UI support code certainly looks like a tempting candidate, given that it could presumably be folded into whatever language/framework is in use in the Tendrl Application Instance. I wonder if you've thought about how to package server-side javascript for linux distributions, and how you will cope with the varying availability of different versions of packages across different distros. If you go forward with using node.js on the server, I advise you to do full packaging from day 1, so that you have early visibility of any issues. Anyway, I think I've made my point enough here -- this is not my decision, but I want to put this into context of the overall stack that Ceph developers like me see. The Tendrl design already has many pieces, and we should be keeping an eye out for anything non-essential to cut out and reduce complexity. John 1. https://github.com/Tendrl/documentation/blob/master/tendrl-core-components-overview.adoc > > We've set up our discussion process on GitHub, and requesting all to follow > https://github.com/Tendrl/ui/issues/5 going forward. > > https://github.com/Tendrl/ui/issues/9 in particular talks about the > selection of server runtime - to take the discussion forward. From anivargi at redhat.com Fri Sep 23 10:39:40 2016 From: anivargi at redhat.com (Anup Nivargi) Date: Fri, 23 Sep 2016 16:09:40 +0530 Subject: [Tendrl-devel] Initial API documentation and Gluster create volume example Message-ID: <7B879C23-EB16-4356-B083-74D70CC79605@redhat.com> Hi, I have created a pull request[1] with the initial API documentation and a Gluster create volume example. Please add you feedback or suggestions in the comments section of the pull request. Note that this is the initial proposal and things might change as we move along, so please don't start building anything around it as yet. [1] https://github.com/Tendrl/documentation/pull/37 -- Anup Nivargi -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From deb at redhat.com Fri Sep 23 11:49:59 2016 From: deb at redhat.com (Soumya Deb) Date: Fri, 23 Sep 2016 17:19:59 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On 22 September 2016 at 22:28, John Spray wrote: > On Thu, Sep 22, 2016 at 4:06 PM, Soumya Deb wrote: > > On 22 September 2016 at 14:31, John Spray wrote: > >> > >> The key point here is the added *runtime*. For frontend javascript > >> you already have a runtime built into the browser. On the backend, > >> you're going to be imposing a requirement for node and all its > >> dependencies. Frontend/backend also involves different tools etc (I > >> know how to do right click->inspect, I do not know all the command > >> line equivalents). > > > > [SNIPPED] > > > > To address what are we about to do with it; yes you've made some correct > > guesses there. Our frontend implemented on REST for the browser/client to > > communicate with the server, we will have sophisticated routes handling > in > > the server side as well, many of the non-generic error/failure handling > can > > be implemented more easily (in cases where tendrl's backend provides > machine > > readable errors, but we can process that into multilanguage supported > > humanly understood error-msg/warnings etc), abstracting out sensitive > > backend endpoints, implementing more login strategies more efficiently, > mock > > APIs for building UI while the backend API isn't ready or to test > features > > and many more reasons that come into play why having a server side > scripting > > environment will come in handy. > > So to put this into context with [1], it seems like the stack looks like: > > UI code (Javascript, running in browser) > || > v > Server-side UI support code (Javascript, running in node.js on a > Tendrl server node) > || > v > Server-side Tendrl Application Instance (Unknown language) > || > v > Tendrl Central Store (Unknown service, probably etcd?) > || > v > Tendrl Node Agent (Python, running on Ceph storage node) > || > v > Tendrl SDS Bridge (Python, running on Ceph storage node) > || > v > Ceph (Python/C++) > > Can you see why I'm concerned about the number of moving parts here? > It's not so much a question of the merits of any one of them, as it is > the resulting total complexity. It's not always easy to identify > pieces to cut out, but the server side UI support code certainly looks > like a tempting candidate, given that it could presumably be folded > into whatever language/framework is in use in the Tendrl Application > Instance. > Your text-chart of the stack looks pretty correct for a linear and simple for the representation. Thanks for taking your time with it. Now, to address the points you brought (about varying tech-stack in each layer vs having consistency), brings some solid argument. Especially, when we see it under the perspective of one SDS. And in this case Python does look dominant. However, there will be other SDSs to support, and some might be developed in Go, Java, C++ or Rust (or whatever might be fitting for that particular product's team). Do you consider that should dictate the choice of language on the upper layers, or should they choose the ones that works best for them (as in, Tendrl bridge and Agent is on Python, but most SDS aren't written in Python - should they consider writting it in C++?). In that case, when all the other parts are pretty homogeneous and based on C++, should we consider the frontend to also base on C++ platform? This gets pretty far-fetched there on. In my understanding, as long as each modular part/layer uses best technology available for them, and interoperates with other parts in the stack this shouldn't cause any headache for the other layers. I wonder if you've thought about how to package server-side javascript > for linux distributions, and how you will cope with the varying > availability of different versions of packages across different > distros. If you go forward with using node.js on the server, I advise > you to do full packaging from day 1, so that you have early visibility > of any issues. > About packaging - packaging, deploying and maintaining Node.js app is among the most hassle free option there is. It has its own package manager for the dependencies it uses, is smarter than pip in almost all the ways, doesn't require a venv to contain itself and not mess with the base system, node dependencies can have their own dependencies of same product's different versions and neither the system or the developer has to utter a word/write a line of extra code to avoid mess. Suggestion taken for doing full packaging from day 1. We already have the intention to automate the entire process anyway. So, from building, testing, packaging, deploying should ideally be taken care of after each successful merge/push into the upstream/master. -------------- next part -------------- An HTML attachment was scrubbed... URL: From adeza at redhat.com Fri Sep 23 12:03:11 2016 From: adeza at redhat.com (Alfredo Deza) Date: Fri, 23 Sep 2016 08:03:11 -0400 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: On Fri, Sep 23, 2016 at 7:49 AM, Soumya Deb wrote: > > > On 22 September 2016 at 22:28, John Spray wrote: >> >> On Thu, Sep 22, 2016 at 4:06 PM, Soumya Deb wrote: >> > On 22 September 2016 at 14:31, John Spray wrote: >> >> >> >> The key point here is the added *runtime*. For frontend javascript >> >> you already have a runtime built into the browser. On the backend, >> >> you're going to be imposing a requirement for node and all its >> >> dependencies. Frontend/backend also involves different tools etc (I >> >> know how to do right click->inspect, I do not know all the command >> >> line equivalents). >> > >> > [SNIPPED] >> > >> > To address what are we about to do with it; yes you've made some correct >> > guesses there. Our frontend implemented on REST for the browser/client >> > to >> > communicate with the server, we will have sophisticated routes handling >> > in >> > the server side as well, many of the non-generic error/failure handling >> > can >> > be implemented more easily (in cases where tendrl's backend provides >> > machine >> > readable errors, but we can process that into multilanguage supported >> > humanly understood error-msg/warnings etc), abstracting out sensitive >> > backend endpoints, implementing more login strategies more efficiently, >> > mock >> > APIs for building UI while the backend API isn't ready or to test >> > features >> > and many more reasons that come into play why having a server side >> > scripting >> > environment will come in handy. >> >> So to put this into context with [1], it seems like the stack looks like: >> >> UI code (Javascript, running in browser) >> || >> v >> Server-side UI support code (Javascript, running in node.js on a >> Tendrl server node) >> || >> v >> Server-side Tendrl Application Instance (Unknown language) >> || >> v >> Tendrl Central Store (Unknown service, probably etcd?) >> || >> v >> Tendrl Node Agent (Python, running on Ceph storage node) >> || >> v >> Tendrl SDS Bridge (Python, running on Ceph storage node) >> || >> v >> Ceph (Python/C++) >> >> Can you see why I'm concerned about the number of moving parts here? >> It's not so much a question of the merits of any one of them, as it is >> the resulting total complexity. It's not always easy to identify >> pieces to cut out, but the server side UI support code certainly looks >> like a tempting candidate, given that it could presumably be folded >> into whatever language/framework is in use in the Tendrl Application >> Instance. > > > Your text-chart of the stack looks pretty correct for a linear and simple > for the representation. Thanks for taking your time with it. > > Now, to address the points you brought (about varying tech-stack in each > layer vs having consistency), brings some solid argument. Especially, > when we see it under the perspective of one SDS. And in this case > Python does look dominant. > > However, there will be other SDSs to support, and some might be > developed in Go, Java, C++ or Rust (or whatever might be fitting for > that particular product's team). Do you consider that should dictate > the choice of language on the upper layers, or should they choose the > ones that works best for them (as in, Tendrl bridge and Agent is on > Python, but most SDS aren't written in Python - should they consider > writting it in C++?). In that case, when all the other parts are pretty > homogeneous and based on C++, should we consider the frontend > to also base on C++ platform? This gets pretty far-fetched there on. > > In my understanding, as long as each modular part/layer uses best > technology available for them, and interoperates with other parts in > the stack this shouldn't cause any headache for the other layers. > > >> I wonder if you've thought about how to package server-side javascript >> for linux distributions, and how you will cope with the varying >> availability of different versions of packages across different >> distros. If you go forward with using node.js on the server, I advise >> you to do full packaging from day 1, so that you have early visibility >> of any issues. > > > About packaging - packaging, deploying and maintaining Node.js > app is among the most hassle free option there is. It has its own > package manager for the dependencies it uses, is smarter than pip > in almost all the ways, doesn't require a venv to contain itself and > not mess with the base system, node dependencies can have their > own dependencies of same product's different versions and neither > the system or the developer has to utter a word/write a line of extra > code to avoid mess. I don't think John meant packaging using the community/language support, but rather using system packages. The Ceph project builds binaries for a few distributions, and packages some Python tools that come included. This has nothing to do with pip, virtualenv, or how (easy?) it may be with Node.js, the point here is that it might be very tricky to do this with JS dependencies compared to say Python, because Python packages are more readily available. > > Suggestion taken for doing full packaging from day 1. We already > have the intention to automate the entire process anyway. So, from > building, testing, packaging, deploying should ideally be taken care > of after each successful merge/push into the upstream/master. I would love to see what the plan for that is here. For Ceph we build for several different distros and distro versions, and sometimes for different architectures. The current system produces RPMs, DEBs and their corresponding repositories. This is a very non-trivial system that does build from upstream changes in most branches too "packaging from day 1" means here that, there should be a plan for the community to be able to consume these binaries and repositories for various different distros (is there a list of supported distros somewhere? or what the plan is for those?) > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From nigelb at redhat.com Fri Sep 23 13:29:32 2016 From: nigelb at redhat.com (Nigel Babu) Date: Fri, 23 Sep 2016 18:59:32 +0530 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: References: Message-ID: <20160923132932.GA15486@gmail.com> On Fri, Sep 23, 2016 at 08:03:11AM -0400, Alfredo Deza wrote: > I don't think John meant packaging using the community/language > support, but rather using system packages. > > The Ceph project builds binaries for a few distributions, and packages > some Python tools that come included. > > This has nothing to do with pip, virtualenv, or how (easy?) it may be > with Node.js, the point here is that it might be > very tricky to do this with JS dependencies compared to say Python, > because Python packages are more readily available. > > > > > Suggestion taken for doing full packaging from day 1. We already > > have the intention to automate the entire process anyway. So, from > > building, testing, packaging, deploying should ideally be taken care > > of after each successful merge/push into the upstream/master. > > I would love to see what the plan for that is here. For Ceph we build > for several different distros and distro versions, and sometimes > for different architectures. The current system produces RPMs, DEBs > and their corresponding repositories. > > This is a very non-trivial system that does build from upstream > changes in most branches too > > "packaging from day 1" means here that, there should be a plan for the > community to be able to consume these binaries and repositories > for various different distros (is there a list of supported distros > somewhere? or what the plan is for those?) Nishanth has been working with me to find the best way forward for tests and builds. I've recommended that we follow best practices, like defining jobs with Jenkins Job Builder. I do not want to see Tendrl playing catch with CI infra like we are in Gluster-land. I believe the current plan is to start with Centos CI for automated tests and builds. We can easily automate CentOS-based packages and builds from there. As Alfredo said, this is something we need to think of in the context of this discussion. Installing dependencies using npm in not how we ease installation for clients and users. Most people (sysadmins?) would prefer installing a package and have it work. Assuming we're going to support CentOS6 and CentOS7. How easy is to get everything installed for node.js dependencies using rpm? Does node work on CentOS5? Because, remember, it still gets security fixes. If we decide we're only going to support CentOS7 and above, remember CentOS7 is supported until 2024. Can we make sure everything works until then? -- nigelb From rwheeler at redhat.com Fri Sep 23 13:41:07 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Fri, 23 Sep 2016 16:41:07 +0300 Subject: [Tendrl-devel] Frontend tool stack: choices made In-Reply-To: <20160923132932.GA15486@gmail.com> References: <20160923132932.GA15486@gmail.com> Message-ID: <0db9b47b-85d6-7dea-8160-99e8b97b669c@redhat.com> On 09/23/2016 04:29 PM, Nigel Babu wrote: > On Fri, Sep 23, 2016 at 08:03:11AM -0400, Alfredo Deza wrote: >> I don't think John meant packaging using the community/language >> support, but rather using system packages. >> >> The Ceph project builds binaries for a few distributions, and packages >> some Python tools that come included. >> >> This has nothing to do with pip, virtualenv, or how (easy?) it may be >> with Node.js, the point here is that it might be >> very tricky to do this with JS dependencies compared to say Python, >> because Python packages are more readily available. >> >>> Suggestion taken for doing full packaging from day 1. We already >>> have the intention to automate the entire process anyway. So, from >>> building, testing, packaging, deploying should ideally be taken care >>> of after each successful merge/push into the upstream/master. >> I would love to see what the plan for that is here. For Ceph we build >> for several different distros and distro versions, and sometimes >> for different architectures. The current system produces RPMs, DEBs >> and their corresponding repositories. >> >> This is a very non-trivial system that does build from upstream >> changes in most branches too >> >> "packaging from day 1" means here that, there should be a plan for the >> community to be able to consume these binaries and repositories >> for various different distros (is there a list of supported distros >> somewhere? or what the plan is for those?) > Nishanth has been working with me to find the best way forward for tests and > builds. I've recommended that we follow best practices, like defining jobs with > Jenkins Job Builder. I do not want to see Tendrl playing catch with CI infra > like we are in Gluster-land. > > I believe the current plan is to start with Centos CI for automated tests and > builds. We can easily automate CentOS-based packages and builds from there. > > As Alfredo said, this is something we need to think of in the context of this > discussion. Installing dependencies using npm in not how we ease installation > for clients and users. Most people (sysadmins?) would prefer installing > a package and have it work. > > Assuming we're going to support CentOS6 and CentOS7. How easy is to get > everything installed for node.js dependencies using rpm? Does node work on > CentOS5? Because, remember, it still gets security fixes. If we decide we're > only going to support CentOS7 and above, remember CentOS7 is supported until > 2024. Can we make sure everything works until then? > > > -- > nigelb Hi Nigel, I don't see a need for us to back port to older platforms new versions of code for community users. If someone cares enough, they can do the backport for a downstream product or something. Similar to how the kernel people handle backports for example to stable series kernels. As for Red Hat products, what is supported on what platforms is a product management call and will be discussed elsewhere. thanks! Ric From julim at redhat.com Fri Sep 23 18:03:02 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 23 Sep 2016 14:03:02 -0400 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture Message-ID: Hi. I've started documenting the UX Design Approach and Information Architecture (JIRA TEN-40 ). It includes some of our guiding principles and decisions that we've made in our design. Here's an initial draft that you can take a look at: https://docs.google.com/a/redhat.com/presentation/d/1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing Note: For now it's open to be comments by anyone with a Red Hat account. I plan to publish this to Jira and/or GitHub before the end of Sprint 1 (Oct 4), but I'd like to open this up to review, comments, and discussion in the meantime. Thank you, Ju Lim -------------- next part -------------- An HTML attachment was scrubbed... URL: From shtripat at redhat.com Mon Sep 26 09:12:06 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Mon, 26 Sep 2016 14:42:06 +0530 Subject: [Tendrl-devel] Help regarding ceph-mgr In-Reply-To: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> References: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> Message-ID: <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> Hi John, With a fresh clone I am able to run the command "./bin/ceph tell mgr osd perf" and see the output. But still not able to access to rest apis. Please let me know a comfortable time when I can have BJ session with you so that can figure out the issue and close this. Regards, Shubhendu On 09/22/2016 10:39 PM, Shubhendu Tripathi wrote: > Hi John, > > I was able to resolve all the dependencies and now ceph comes up fine > without any errors in out/mgr.x.log (attached the log file for reference). > I see ceph-mgr running in the processes list as shown below > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > [shubhendu at dhcp35-52 build]$ ps -ef | grep ceph > shubhen+ 8279 1 0 22:20 pts/0 00:00:02 > /home/shubhendu/work/ceph/build/bin/ceph-mon -i a -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 8461 1 0 22:20 ? 00:00:01 > /home/shubhendu/work/ceph/build/bin/ceph-osd -i 0 -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 8654 1 0 22:20 ? 00:00:01 > /home/shubhendu/work/ceph/build/bin/ceph-osd -i 1 -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 8846 1 0 22:20 ? 00:00:01 > /home/shubhendu/work/ceph/build/bin/ceph-osd -i 2 -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 9055 1 0 22:20 ? 00:00:00 > /home/shubhendu/work/ceph/build/bin/ceph-mds -i a -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 9139 1 0 22:20 ? 00:00:00 > /home/shubhendu/work/ceph/build/bin/ceph-mgr -i x > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > But still not able to access http://localhost:8002/api/v2/cluster and > it shows the error "This site can't be reached". > Not sure what is missing here. > > Kindly help out here. > > Also help out with some sample "ceph tell mgr" commands so that I can > verify in the setup. > > Thanks and Regards, > Shubhendu -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Mon Sep 26 09:34:01 2016 From: jspray at redhat.com (John Spray) Date: Mon, 26 Sep 2016 10:34:01 +0100 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: > Hi. I've started documenting the UX Design Approach and Information > Architecture (JIRA TEN-40). It includes some of our guiding principles and > decisions that we've made in our design. > > Here's an initial draft that you can take a look at: > https://docs.google.com/a/redhat.com/presentation/d/1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing > > Note: For now it's open to be comments by anyone with a Red Hat account. I > plan to publish this to Jira and/or GitHub before the end of Sprint 1 (Oct > 4), but I'd like to open this up to review, comments, and discussion in the > meantime. Thanks for posting! My main piece of feedback is on terminology -- I think it's going to get confusing to use a mixture of qualified names (e.g. "RBD" instead of "block device") and unqualified names (e.g. "File Share" when we really mean "Gluster"). I have a special attachment to use of "File Share" because when Tendrl gets support for CephFS, everywhere the term is used is going to need qualifying with whether we're talking about Ceph or Gluster. I don't think anyone is working on adding Tendrl support for CephFS at the moment, but it should be part of the design process, along with RGW. Terms like "Quotas" are also dangerous, because we have so many different kinds. In Ceph alone, we have Quotas on pools, and then a totally different type of quota on directories in cephfs. Same for snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. John > Thank you, > Ju Lim > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From shtripat at redhat.com Mon Sep 26 11:07:38 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Mon, 26 Sep 2016 16:37:38 +0530 Subject: [Tendrl-devel] Help regarding ceph-mgr In-Reply-To: <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> References: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> Message-ID: <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> Thanks John, It was really helpful BJ session we had. I am now able to access REST apis. Would continue to verify the set of apis available and what is missing as per our needs. In the meantime, I remember you talking about other protocols support for talking to ceph-mgr. Is it like these protocols needs to be implemented or anything else already available. May be I would need another round of BJ session with you sometime to understand the architecture better :) Thanks and Regards, Shubhendu On 09/26/2016 02:42 PM, Shubhendu Tripathi wrote: > Hi John, > > With a fresh clone I am able to run the command "./bin/ceph tell mgr > osd perf" and see the output. > But still not able to access to rest apis. > > Please let me know a comfortable time when I can have BJ session with > you so that can figure out the issue and close this. > > Regards, > Shubhendu > > > > On 09/22/2016 10:39 PM, Shubhendu Tripathi wrote: >> Hi John, >> >> I was able to resolve all the dependencies and now ceph comes up fine >> without any errors in out/mgr.x.log (attached the log file for >> reference). >> I see ceph-mgr running in the processes list as shown below >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> [shubhendu at dhcp35-52 build]$ ps -ef | grep ceph >> shubhen+ 8279 1 0 22:20 pts/0 00:00:02 >> /home/shubhendu/work/ceph/build/bin/ceph-mon -i a -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 8461 1 0 22:20 ? 00:00:01 >> /home/shubhendu/work/ceph/build/bin/ceph-osd -i 0 -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 8654 1 0 22:20 ? 00:00:01 >> /home/shubhendu/work/ceph/build/bin/ceph-osd -i 1 -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 8846 1 0 22:20 ? 00:00:01 >> /home/shubhendu/work/ceph/build/bin/ceph-osd -i 2 -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 9055 1 0 22:20 ? 00:00:00 >> /home/shubhendu/work/ceph/build/bin/ceph-mds -i a -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 9139 1 0 22:20 ? 00:00:00 >> /home/shubhendu/work/ceph/build/bin/ceph-mgr -i x >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> But still not able to access http://localhost:8002/api/v2/cluster and >> it shows the error "This site can't be reached". >> Not sure what is missing here. >> >> Kindly help out here. >> >> Also help out with some sample "ceph tell mgr" commands so that I can >> verify in the setup. >> >> Thanks and Regards, >> Shubhendu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rwheeler at redhat.com Mon Sep 26 11:11:08 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Mon, 26 Sep 2016 14:11:08 +0300 Subject: [Tendrl-devel] Help regarding ceph-mgr In-Reply-To: <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> References: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> Message-ID: <6fb42871-eba6-b7db-367a-25157dce9527@redhat.com> On 09/26/2016 02:07 PM, Shubhendu Tripathi wrote: > Thanks John, > > It was really helpful BJ session we had. > I am now able to access REST apis. > > Would continue to verify the set of apis available and what is missing as per > our needs. > > In the meantime, I remember you talking about other protocols support for > talking to ceph-mgr. > Is it like these protocols needs to be implemented or anything else already > available. > > May be I would need another round of BJ session with you sometime to > understand the architecture better :) > > Thanks and Regards, > Shubhendu Great to see you jumping in with this! Do we have a list of things that we would like to see implemented in ceph-mgr (above and beyond the current things supported in our USM project)? It would be great to make sure we put out those ideas, we might get a lot of help and good feedback as well on the whole effort. Regards, Ric From jspray at redhat.com Mon Sep 26 11:41:42 2016 From: jspray at redhat.com (John Spray) Date: Mon, 26 Sep 2016 12:41:42 +0100 Subject: [Tendrl-devel] Help regarding ceph-mgr In-Reply-To: <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> References: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> Message-ID: On Mon, Sep 26, 2016 at 12:07 PM, Shubhendu Tripathi wrote: > Thanks John, > > It was really helpful BJ session we had. > I am now able to access REST apis. > > Would continue to verify the set of apis available and what is missing as > per our needs. > > In the meantime, I remember you talking about other protocols support for > talking to ceph-mgr. > Is it like these protocols needs to be implemented or anything else already > available. The key idea is that the service loads and runs whatever python modules you want, so anyone who wants different protocols or functionality can write such a module. ceph-mgr is about enabling people doing integration work to create whatever they need. What you get "out of the box" is a port of the Calamari REST API, and then any module can expose CLI commands (accessed via "ceph tell mgr") if they don't want to implement their own network protocol. John > May be I would need another round of BJ session with you sometime to > understand the architecture better :) > > Thanks and Regards, > Shubhendu > > > > On 09/26/2016 02:42 PM, Shubhendu Tripathi wrote: > > Hi John, > > With a fresh clone I am able to run the command "./bin/ceph tell mgr osd > perf" and see the output. > But still not able to access to rest apis. > > Please let me know a comfortable time when I can have BJ session with you so > that can figure out the issue and close this. > > Regards, > Shubhendu > > > > On 09/22/2016 10:39 PM, Shubhendu Tripathi wrote: > > Hi John, > > I was able to resolve all the dependencies and now ceph comes up fine > without any errors in out/mgr.x.log (attached the log file for reference). > I see ceph-mgr running in the processes list as shown below > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > [shubhendu at dhcp35-52 build]$ ps -ef | grep ceph > shubhen+ 8279 1 0 22:20 pts/0 00:00:02 > /home/shubhendu/work/ceph/build/bin/ceph-mon -i a -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 8461 1 0 22:20 ? 00:00:01 > /home/shubhendu/work/ceph/build/bin/ceph-osd -i 0 -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 8654 1 0 22:20 ? 00:00:01 > /home/shubhendu/work/ceph/build/bin/ceph-osd -i 1 -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 8846 1 0 22:20 ? 00:00:01 > /home/shubhendu/work/ceph/build/bin/ceph-osd -i 2 -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 9055 1 0 22:20 ? 00:00:00 > /home/shubhendu/work/ceph/build/bin/ceph-mds -i a -c > /home/shubhendu/work/ceph/build/ceph.conf > shubhen+ 9139 1 0 22:20 ? 00:00:00 > /home/shubhendu/work/ceph/build/bin/ceph-mgr -i x > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > But still not able to access http://localhost:8002/api/v2/cluster and it > shows the error "This site can't be reached". > Not sure what is missing here. > > Kindly help out here. > > Also help out with some sample "ceph tell mgr" commands so that I can verify > in the setup. > > Thanks and Regards, > Shubhendu > > > From shtripat at redhat.com Mon Sep 26 11:55:26 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Mon, 26 Sep 2016 17:25:26 +0530 Subject: [Tendrl-devel] Help regarding ceph-mgr In-Reply-To: References: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> Message-ID: On 09/26/2016 05:11 PM, John Spray wrote: > On Mon, Sep 26, 2016 at 12:07 PM, Shubhendu Tripathi > wrote: >> Thanks John, >> >> It was really helpful BJ session we had. >> I am now able to access REST apis. >> >> Would continue to verify the set of apis available and what is missing as >> per our needs. >> >> In the meantime, I remember you talking about other protocols support for >> talking to ceph-mgr. >> Is it like these protocols needs to be implemented or anything else already >> available. > The key idea is that the service loads and runs whatever python > modules you want, so anyone who wants different protocols or > functionality can write such a module. ceph-mgr is about enabling > people doing integration work to create whatever they need. > > What you get "out of the box" is a port of the Calamari REST API, and > then any module can expose CLI commands (accessed via "ceph tell mgr") > if they don't want to implement their own network protocol. Thanks John, that makes clearer to me. Thanks a lot. > > John > >> May be I would need another round of BJ session with you sometime to >> understand the architecture better :) >> >> Thanks and Regards, >> Shubhendu >> >> >> >> On 09/26/2016 02:42 PM, Shubhendu Tripathi wrote: >> >> Hi John, >> >> With a fresh clone I am able to run the command "./bin/ceph tell mgr osd >> perf" and see the output. >> But still not able to access to rest apis. >> >> Please let me know a comfortable time when I can have BJ session with you so >> that can figure out the issue and close this. >> >> Regards, >> Shubhendu >> >> >> >> On 09/22/2016 10:39 PM, Shubhendu Tripathi wrote: >> >> Hi John, >> >> I was able to resolve all the dependencies and now ceph comes up fine >> without any errors in out/mgr.x.log (attached the log file for reference). >> I see ceph-mgr running in the processes list as shown below >> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> [shubhendu at dhcp35-52 build]$ ps -ef | grep ceph >> shubhen+ 8279 1 0 22:20 pts/0 00:00:02 >> /home/shubhendu/work/ceph/build/bin/ceph-mon -i a -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 8461 1 0 22:20 ? 00:00:01 >> /home/shubhendu/work/ceph/build/bin/ceph-osd -i 0 -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 8654 1 0 22:20 ? 00:00:01 >> /home/shubhendu/work/ceph/build/bin/ceph-osd -i 1 -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 8846 1 0 22:20 ? 00:00:01 >> /home/shubhendu/work/ceph/build/bin/ceph-osd -i 2 -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 9055 1 0 22:20 ? 00:00:00 >> /home/shubhendu/work/ceph/build/bin/ceph-mds -i a -c >> /home/shubhendu/work/ceph/build/ceph.conf >> shubhen+ 9139 1 0 22:20 ? 00:00:00 >> /home/shubhendu/work/ceph/build/bin/ceph-mgr -i x >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> But still not able to access http://localhost:8002/api/v2/cluster and it >> shows the error "This site can't be reached". >> Not sure what is missing here. >> >> Kindly help out here. >> >> Also help out with some sample "ceph tell mgr" commands so that I can verify >> in the setup. >> >> Thanks and Regards, >> Shubhendu >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From shtripat at redhat.com Mon Sep 26 12:45:54 2016 From: shtripat at redhat.com (Shubhendu Tripathi) Date: Mon, 26 Sep 2016 18:15:54 +0530 Subject: [Tendrl-devel] Help regarding ceph-mgr In-Reply-To: <6fb42871-eba6-b7db-367a-25157dce9527@redhat.com> References: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> <6fb42871-eba6-b7db-367a-25157dce9527@redhat.com> Message-ID: <143de23e-fe29-77d0-3310-0a306cf461f0@redhat.com> On 09/26/2016 04:41 PM, Ric Wheeler wrote: > On 09/26/2016 02:07 PM, Shubhendu Tripathi wrote: >> Thanks John, >> >> It was really helpful BJ session we had. >> I am now able to access REST apis. >> >> Would continue to verify the set of apis available and what is >> missing as per our needs. >> >> In the meantime, I remember you talking about other protocols support >> for talking to ceph-mgr. >> Is it like these protocols needs to be implemented or anything else >> already available. >> >> May be I would need another round of BJ session with you sometime to >> understand the architecture better :) >> >> Thanks and Regards, >> Shubhendu > > Great to see you jumping in with this! > > Do we have a list of things that we would like to see implemented in > ceph-mgr (above and beyond the current things supported in our USM > project)? It would be great to make sure we put out those ideas, we > might get a lot of help and good feedback as well on the whole effort. Hi Ric, Just started looking at ceph-mgr and its available features/api. Soon I should have a list of already supported and needed features over tendrl github. Regards, Shubhendu > > Regards, > Ric > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Mon Sep 26 12:57:54 2016 From: jspray at redhat.com (John Spray) Date: Mon, 26 Sep 2016 13:57:54 +0100 Subject: [Tendrl-devel] Help regarding ceph-mgr In-Reply-To: <143de23e-fe29-77d0-3310-0a306cf461f0@redhat.com> References: <6298ced7-c927-6c0e-5886-dde1d92f73e7@redhat.com> <797869cf-e571-6ae8-6312-13133a7af695@redhat.com> <2c77b968-37e6-5ec6-5151-d09b0ab41184@redhat.com> <6fb42871-eba6-b7db-367a-25157dce9527@redhat.com> <143de23e-fe29-77d0-3310-0a306cf461f0@redhat.com> Message-ID: On Mon, Sep 26, 2016 at 1:45 PM, Shubhendu Tripathi wrote: > On 09/26/2016 04:41 PM, Ric Wheeler wrote: > > On 09/26/2016 02:07 PM, Shubhendu Tripathi wrote: > > Thanks John, > > It was really helpful BJ session we had. > I am now able to access REST apis. > > Would continue to verify the set of apis available and what is missing as > per our needs. > > In the meantime, I remember you talking about other protocols support for > talking to ceph-mgr. > Is it like these protocols needs to be implemented or anything else already > available. > > May be I would need another round of BJ session with you sometime to > understand the architecture better :) > > Thanks and Regards, > Shubhendu > > > Great to see you jumping in with this! > > Do we have a list of things that we would like to see implemented in > ceph-mgr (above and beyond the current things supported in our USM project)? > It would be great to make sure we put out those ideas, we might get a lot of > help and good feedback as well on the whole effort. > > > Hi Ric, > > Just started looking at ceph-mgr and its available features/api. > Soon I should have a list of already supported and needed features over > tendrl github. To keep the communications clear, let's make a separation between: A Things wanted from the REST module B Things wanted from ceph-mgr itself (i.e. gaps in the interface for python modules) John > Regards, > Shubhendu > > > Regards, > Ric > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > From deb at redhat.com Tue Sep 27 10:54:38 2016 From: deb at redhat.com (Soumya Deb) Date: Tue, 27 Sep 2016 16:24:38 +0530 Subject: [Tendrl-devel] Frontend tool stack Message-ID: Hi, We're tracking all the tools selections ( https://github.com/Tendrl/ui/labels/selection) made for the frontend development in https://github.com/Tendrl/ui/issues/5 As per present sprint deliverable, we've made almost all the choices needed to set up the basic developer environment. Any new choices that we'd be making will be appended and tracked by the same issue for easier discovery (so, it'll not be closed any time very soon). Please post your observations about the particular issues in their relative links, and use this issue to discuss the meta/process of it. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Tue Sep 27 10:56:45 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 27 Sep 2016 16:26:45 +0530 Subject: [Tendrl-devel] Frontend tool stack In-Reply-To: References: Message-ID: On Tue, Sep 27, 2016 at 4:24 PM, Soumya Deb wrote: > We're tracking all the tools selections > (https://github.com/Tendrl/ui/labels/selection) made for the frontend > development in https://github.com/Tendrl/ui/issues/5 Was there not a conversation to keep the ui repository as is and instead use a 'frontend' or, similarly named repository to have this work in? -- sankarshan mukhopadhyay From julim at redhat.com Tue Sep 27 11:31:30 2016 From: julim at redhat.com (Ju Lim) Date: Tue, 27 Sep 2016 07:31:30 -0400 Subject: [Tendrl-devel] Weekly Check-in for Tuesday, Sept 27 Message-ID: Team: A reminder to folks who were not able to be at today's check-in (because you;re attending Jim Whitehurst's Town Hall), please add your updates to the Tendrl Sprint Check-in Rolling Notes . Please note that tomorrow, Sept 28 is a Czech public holiday. Thank you, Ju -------------- next part -------------- An HTML attachment was scrubbed... URL: From mkudlej at redhat.com Tue Sep 27 13:27:54 2016 From: mkudlej at redhat.com (Martin Kudlej) Date: Tue, 27 Sep 2016 15:27:54 +0200 Subject: [Tendrl-devel] installation instructions for Tendrl components Message-ID: <4f52ef4d-34b9-c6b9-c8f9-91184dbb5bd3@redhat.com> Hi Rohan, I write email because there are common comments for more tickets(TEN-53, TEN-54, TEN-55, TEN-57): 1) I would like to start discussion about environment supported by Tendrl. I think it is required to define install instructions for supported OSes. I expect that there will be many common parts and differences will be only because of different packaging systems. What OSes would we like to support for Tendrl 1.0? I've tried to define install instructions for CentOS 7 minimal install. 2) Common install instructions for all Tendrl components for CentOS 7. I've installed only Git, Epel repo and Pip rpm -iUvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm yum install git yum install python-pip Is there any software required for running Tendrl components? I expect that I should install for example Etcd, should I? 3) bridge_common: git clone https://github.com/Tendrl/bridge_common.git try to install all requirements from requirements.txt(many of requirements are common with other Tendrl components): yum install python-pbr pip install python-etcd yum install python-dateutil.noarch yum install python-gevent yum install pytz yum install gcc pip install oslo.log python setup.py install 4) ceph_bridge: git clone https://github.com/Tendrl/ceph_bridge.git try to install all requirements: yum install python2-msgpack python setup.py install There are unsatisfied imports. I've tried to run command installed to /usr/bin: $ /usr/bin/tendrl_ceph_bridge Traceback (most recent call last): File "/usr/bin/tendrl_ceph_bridge", line 6, in from ceph_bridge.manager.manager import main File "/usr/lib/python2.7/site-packages/ceph_bridge/manager/manager.py", line 14, in from ceph_bridge.manager.cluster_monitor import ClusterMonitor File "/usr/lib/python2.7/site-packages/ceph_bridge/manager/cluster_monitor.py", line 13, in from ceph_bridge.manager.crush_node_request_factory \ File "/usr/lib/python2.7/site-packages/ceph_bridge/manager/crush_node_request_factory.py", line 5, in from ceph_bridge.manager.server_monitor import ServiceId File "/usr/lib/python2.7/site-packages/ceph_bridge/manager/server_monitor.py", line 24, in from ceph_bridge.log import LOG as tendrl_log ImportError: cannot import name LOG 5) gluster_bridge: git clone https://github.com/Tendrl/gluster_bridge.git python setup.py install I've tried to run tendrl_gluster_bridge: $ /usr/bin/tendrl_gluster_bridge Traceback (most recent call last): File "/usr/bin/tendrl_gluster_bridge", line 6, in from gluster_bridge.manager.manager import main File "/usr/lib/python2.7/site-packages/gluster_bridge/manager/manager.py", line 17, in from gluster_bridge.persistence.persister import Persister File "/usr/lib/python2.7/site-packages/gluster_bridge/persistence/persister.py", line 1, in from etcdobj import Server as etcd_server ImportError: No module named etcdobj Should I install Etcd and/or any Python modules? 6) node_agent: git clone https://github.com/Tendrl/node_agent.git python setup.py install Could I try to create pull request to Tendrl documentation with these instructions? I think there are common part, so they can be shared to another Tendrl repos. I move all TEN-53, TEN-54, TEN-55, TEN-57 back to "In Progress" because tasks are not completed from my perspective. Thank you for fixing these issues. -- Best Regards, Martin Kudlej. RHSC/USM Senior Quality Assurance Engineer Red Hat Czech s.r.o. Phone: +420 532 294 155 E-mail:mkudlej at redhat.com IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting @ redhat #tendrl-devel @ freenode From adeza at redhat.com Tue Sep 27 13:42:52 2016 From: adeza at redhat.com (Alfredo Deza) Date: Tue, 27 Sep 2016 09:42:52 -0400 Subject: [Tendrl-devel] installation instructions for Tendrl components In-Reply-To: <4f52ef4d-34b9-c6b9-c8f9-91184dbb5bd3@redhat.com> References: <4f52ef4d-34b9-c6b9-c8f9-91184dbb5bd3@redhat.com> Message-ID: On Tue, Sep 27, 2016 at 9:27 AM, Martin Kudlej wrote: > Hi Rohan, > > I write email because there are common comments for more tickets(TEN-53, > TEN-54, TEN-55, TEN-57): > 1) I would like to start discussion about environment supported by Tendrl. I > think it is required to define install instructions for supported OSes. I > expect that there will be many common parts and differences will be only > because of different packaging systems. What OSes would we like to support > for Tendrl 1.0? > I've tried to define install instructions for CentOS 7 minimal install. > > 2) Common install instructions for all Tendrl components for CentOS 7. I've > installed only Git, Epel repo and Pip > rpm -iUvh > http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm > yum install git > yum install python-pip > > Is there any software required for running Tendrl components? I expect that > I should install for example Etcd, should I? > > 3) bridge_common: > git clone https://github.com/Tendrl/bridge_common.git > > try to install all requirements from requirements.txt(many of requirements > are common with other Tendrl components): > yum install python-pbr > pip install python-etcd > yum install python-dateutil.noarch > yum install python-gevent > yum install pytz > yum install gcc > pip install oslo.log > > python setup.py install > > 4) ceph_bridge: > git clone https://github.com/Tendrl/ceph_bridge.git > > try to install all requirements: > yum install python2-msgpack > > python setup.py install > > There are unsatisfied imports. I've tried to run command installed to > /usr/bin: > $ /usr/bin/tendrl_ceph_bridge > Traceback (most recent call last): > File "/usr/bin/tendrl_ceph_bridge", line 6, in > from ceph_bridge.manager.manager import main > File "/usr/lib/python2.7/site-packages/ceph_bridge/manager/manager.py", > line 14, in > from ceph_bridge.manager.cluster_monitor import ClusterMonitor > File > "/usr/lib/python2.7/site-packages/ceph_bridge/manager/cluster_monitor.py", > line 13, in > from ceph_bridge.manager.crush_node_request_factory \ > File > "/usr/lib/python2.7/site-packages/ceph_bridge/manager/crush_node_request_factory.py", > line 5, in > from ceph_bridge.manager.server_monitor import ServiceId > File > "/usr/lib/python2.7/site-packages/ceph_bridge/manager/server_monitor.py", > line 24, in > from ceph_bridge.log import LOG as tendrl_log > ImportError: cannot import name LOG > > 5) gluster_bridge: > git clone https://github.com/Tendrl/gluster_bridge.git > python setup.py install > > I've tried to run tendrl_gluster_bridge: > $ /usr/bin/tendrl_gluster_bridge > Traceback (most recent call last): > File "/usr/bin/tendrl_gluster_bridge", line 6, in > from gluster_bridge.manager.manager import main > File "/usr/lib/python2.7/site-packages/gluster_bridge/manager/manager.py", > line 17, in > from gluster_bridge.persistence.persister import Persister > File > "/usr/lib/python2.7/site-packages/gluster_bridge/persistence/persister.py", > line 1, in > from etcdobj import Server as etcd_server > ImportError: No module named etcdobj > > Should I install Etcd and/or any Python modules? > > 6) node_agent: > git clone https://github.com/Tendrl/node_agent.git > python setup.py install > > Could I try to create pull request to Tendrl documentation with these > instructions? I think there are common part, so they can be shared to > another Tendrl repos. Setting up a project like Tendrl will run into issues like this even if documented (not that it shouldn't be documented :)). For the ceph-installer and almost all applications that support ceph we use ansible for setting everything up. I highly recommend to start automating this because it will provide: * fully repeatable deployment process * immediate exposure to issues * easier to impact a wide audience when fixes come in (e.g. new dependency is added) * easier to port "manual" installation steps (e.g. `python setup.py install`) to actual production installs with binaries (yum/apt-get) and have then live alongside each other. > > I move all TEN-53, TEN-54, TEN-55, TEN-57 back to "In Progress" because > tasks are not completed from my perspective. > Thank you for fixing these issues. > > -- > Best Regards, > Martin Kudlej. > RHSC/USM Senior Quality Assurance Engineer > Red Hat Czech s.r.o. > > Phone: +420 532 294 155 > E-mail:mkudlej at redhat.com > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, #usm-meeting > @ redhat > #tendrl-devel @ freenode > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From japplewh at redhat.com Tue Sep 27 13:52:11 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Tue, 27 Sep 2016 09:52:11 -0400 Subject: [Tendrl-devel] Architectural questions Message-ID: Hi All I have reviewed the documentation and I have some questions on basic strategies that I wanted to get an update on. I know these are complex topics but let's air the issues and get updates on where we are and input on how to bridge the gaps. I will break these down by storage technology. *Ceph*: How will be be installing Ceph via Tendrl? How will we be provisioning volumes/pools/rbds/etc in Tendrl? *Gluster*: How will we be installing Gluster? What is our plan for provisioning volumes/bricks/etc? It sounds like the Heketi adoption means that we would not be able to allow users of Gluster to issue gluster commands and they would instead have to issue Heketi command line statements. This seems to be an unacceptable tradeoff to me. *Core:* How will we store and display time series data? How will we establish a secure channel to the managed Ceph or Gluster nodes? Thanks, Jeff -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrugesh at brainfunked.org Tue Sep 27 14:39:46 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Tue, 27 Sep 2016 20:09:46 +0530 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: Message-ID: On 27 September 2016 at 19:22, Jeff Applewhite wrote: > > Hi All > > Ceph: > How will be be installing Ceph via Tendrl? I'm investigating this. Two choices currently exist: ceph-installer and ceph-ansible. I'm prioritising on ceph-ansible since adding ansible support naturally opens up the provisioning framework for systems other than ceph to be supported through. > How will we be provisioning volumes/pools/rbds/etc in Tendrl? The ceph bridge uses rados calls to get this done. The actual flows are to be translated over from skyring. Nishanth is leading this effort. > Gluster: > How will we be installing Gluster? gdeploy is a candidate. However, as part of the ansible integration evaluation for ceph-ansible, future directions will be known. > What is our plan for provisioning volumes/bricks/etc? > It sounds like the Heketi adoption means that we would not be able to allow users of Gluster to issue gluster commands and they would instead have to issue Heketi command line statements. This seems to be an unacceptable tradeoff to me. Heketi has either already implemented or is planning on implementing the volume and brick handling functionality that tendrl would eventually need to. Hence the best course of action would be to understand if and how heketi's functionality could be leveraged by tendrl, cleanly. Failing this, it's possible to implement the logic in tendrl. We'll have a decision on this issue by Friday, after a discussion with Luis. > Core: > How will we store and display time series data? This is being looked at in the current sprint. > How will we establish a secure channel to the managed Ceph or Gluster nodes? This is currently an open issue, which will be picked up in a latter sprint, once the basic node agent functionality has been implemented. -- Mrugesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Tue Sep 27 15:02:01 2016 From: jspray at redhat.com (John Spray) Date: Tue, 27 Sep 2016 16:02:01 +0100 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: Message-ID: On Tue, Sep 27, 2016 at 3:39 PM, Mrugesh Karnik wrote: > On 27 September 2016 at 19:22, Jeff Applewhite wrote: >> >> Hi All >> >> Ceph: >> How will be be installing Ceph via Tendrl? > > I'm investigating this. Two choices currently exist: ceph-installer and > ceph-ansible. I'm prioritising on ceph-ansible since adding ansible support > naturally opens up the provisioning framework for systems other than ceph to > be supported through. > > >> How will we be provisioning volumes/pools/rbds/etc in Tendrl? > > The ceph bridge uses rados calls to get this done. The actual flows are to > be translated over from skyring. Nishanth is leading this effort. This sounds ominously like you've gone back to intending to use librados calls directly to the mon from tendrl :-/ I thought when I asked about this two weeks ago you said the ceph_bridge use of librados was just a prototype? John > >> Gluster: >> How will we be installing Gluster? > > gdeploy is a candidate. However, as part of the ansible integration > evaluation for ceph-ansible, future directions will be known. > > >> What is our plan for provisioning volumes/bricks/etc? >> It sounds like the Heketi adoption means that we would not be able to >> allow users of Gluster to issue gluster commands and they would instead have >> to issue Heketi command line statements. This seems to be an unacceptable >> tradeoff to me. > > Heketi has either already implemented or is planning on implementing the > volume and brick handling functionality that tendrl would eventually need > to. Hence the best course of action would be to understand if and how > heketi's functionality could be leveraged by tendrl, cleanly. Failing this, > it's possible to implement the logic in tendrl. > > We'll have a decision on this issue by Friday, after a discussion with Luis. > > >> Core: >> How will we store and display time series data? > > This is being looked at in the current sprint. > >> How will we establish a secure channel to the managed Ceph or Gluster >> nodes? > > This is currently an open issue, which will be picked up in a latter sprint, > once the basic node agent functionality has been implemented. > > > > -- > Mrugesh > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From khartsoe at redhat.com Tue Sep 27 18:11:37 2016 From: khartsoe at redhat.com (Kenneth Hartsoe) Date: Tue, 27 Sep 2016 14:11:37 -0400 (EDT) Subject: [Tendrl-devel] Kanban board created In-Reply-To: References: Message-ID: <1234277149.5071756.1474999897771.JavaMail.zimbra@redhat.com> Hi Bobb, Piyush, RHS-C 3 sprint planning is tracked via the Kanban board linked below. I believe you use your JIRA ID/account to login. can you check your access? Thanks. Ken Hartsoe Content Strategist Red Hat Storage Documentation khartsoe at redhat.com; IRC: khartsoe Office: 919 754 4770; Internal: 814 4770 ----- Original Message ----- | Hi All | | I have created a Kanban board here | with | columns mapping to Ju Lim's proposed categories as per our process | discussions | | ?Thanks | ? | | Jeff | | _______________________________________________ | Tendrl-devel mailing list | Tendrl-devel at redhat.com | https://www.redhat.com/mailman/listinfo/tendrl-devel | From rwheeler at redhat.com Tue Sep 27 19:18:25 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Tue, 27 Sep 2016 22:18:25 +0300 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: Message-ID: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> On 09/27/2016 06:02 PM, John Spray wrote: > On Tue, Sep 27, 2016 at 3:39 PM, Mrugesh Karnik wrote: >> On 27 September 2016 at 19:22, Jeff Applewhite wrote: >>> Hi All >>> >>> Ceph: >>> How will be be installing Ceph via Tendrl? >> I'm investigating this. Two choices currently exist: ceph-installer and >> ceph-ansible. I'm prioritising on ceph-ansible since adding ansible support >> naturally opens up the provisioning framework for systems other than ceph to >> be supported through. >> >> >>> How will we be provisioning volumes/pools/rbds/etc in Tendrl? >> The ceph bridge uses rados calls to get this done. The actual flows are to >> be translated over from skyring. Nishanth is leading this effort. > This sounds ominously like you've gone back to intending to use > librados calls directly to the mon from tendrl :-/ > > I thought when I asked about this two weeks ago you said the > ceph_bridge use of librados was just a prototype? > > John Hi John, My understanding is that we are clear that we intend to move to ceph-mgr as soon as ceph-mgr is ready for use. No backtracking on this, it is really important that we do this to keep the ceph community on board and enable them to contribute (and us to ride other teams' work) as we go forward. Not if we make the switch, but rather *when* we can make that switch. Ric > > > >>> Gluster: >>> How will we be installing Gluster? >> gdeploy is a candidate. However, as part of the ansible integration >> evaluation for ceph-ansible, future directions will be known. >> >> >>> What is our plan for provisioning volumes/bricks/etc? >>> It sounds like the Heketi adoption means that we would not be able to >>> allow users of Gluster to issue gluster commands and they would instead have >>> to issue Heketi command line statements. This seems to be an unacceptable >>> tradeoff to me. >> Heketi has either already implemented or is planning on implementing the >> volume and brick handling functionality that tendrl would eventually need >> to. Hence the best course of action would be to understand if and how >> heketi's functionality could be leveraged by tendrl, cleanly. Failing this, >> it's possible to implement the logic in tendrl. >> >> We'll have a decision on this issue by Friday, after a discussion with Luis. >> >> >>> Core: >>> How will we store and display time series data? >> This is being looked at in the current sprint. >> >>> How will we establish a secure channel to the managed Ceph or Gluster >>> nodes? >> This is currently an open issue, which will be picked up in a latter sprint, >> once the basic node agent functionality has been implemented. >> >> >> >> -- >> Mrugesh >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From julim at redhat.com Tue Sep 27 20:42:07 2016 From: julim at redhat.com (Ju Lim) Date: Tue, 27 Sep 2016 16:42:07 -0400 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: Hi John: Thank you for taking the time to review and sharing your feedback. WRT terminology, I think that's a tricky one and I can understand and appreciate the confusion with RBD (vs. Block Device) and File Share. The original decision to go with RBD was that it was very Ceph-specific in terms of the implementation and varies from LVM and other block technologies. Having said that, I would agree that File Share deviates from this initial premise since it could mean a Gluster volume or a CephFS fileshare. The current thinking regarding File Share is to use it for both Ceph and Gluster, and have a way to qualify it through the notion of the workload. Part of the rationale was to try to make it so users did not have to think should I use Ceph or Gluster, but rather let the System make an "intelligent" choice for the mere mortal user based on just enough information that user provides. This way, the user thinks of it as Red Hat Storage providing this capability, and if we swapped out Ceph or Gluster for something in the future, it would allow for that. In conjunction, there would still be a way for a user to qualify it for Ceph or Gluster as well, so it's not just for mere mortal but also a more knowledge / ninja user. Having said that, RGW (and Swift) are definitely in mind too, so this is not limited to just File Shares but applies to pretty much block, object, and file services. With regards to quotas and snapshots, I do agree that there are many different kinds. It did not seem prudent to create a Pool Snapshot, RBD Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, Pool Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not scalable (and would get very crowded very quickly), and would create a fairly complex experience for the user. My current thinking is that when we get to the Quotas or Snapshots section, we would either have a unified view with filtering capabilities (for the different types), or tabbed views, or a 3rd level navigation, or some other combination. Since quotas and snapshots are not in the immediate release plans, I've deferred this to the detailed design stage when we tackle those topics. I hope this addresses your concerns raised. I'll update the slidedeck with these comments before it gets published out more formally. Thanks again, Ju On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: > On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: > > Hi. I've started documenting the UX Design Approach and Information > > Architecture (JIRA TEN-40). It includes some of our guiding principles > and > > decisions that we've made in our design. > > > > Here's an initial draft that you can take a look at: > > https://docs.google.com/a/redhat.com/presentation/d/ > 1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing > > > > Note: For now it's open to be comments by anyone with a Red Hat > account. I > > plan to publish this to Jira and/or GitHub before the end of Sprint 1 > (Oct > > 4), but I'd like to open this up to review, comments, and discussion in > the > > meantime. > > Thanks for posting! > > My main piece of feedback is on terminology -- I think it's going to > get confusing to use a mixture of qualified names (e.g. "RBD" instead > of "block device") and unqualified names (e.g. "File Share" when we > really mean "Gluster"). > > I have a special attachment to use of "File Share" because when Tendrl > gets support for CephFS, everywhere the term is used is going to need > qualifying with whether we're talking about Ceph or Gluster. I don't > think anyone is working on adding Tendrl support for CephFS at the > moment, but it should be part of the design process, along with RGW. > > Terms like "Quotas" are also dangerous, because we have so many > different kinds. In Ceph alone, we have Quotas on pools, and then a > totally different type of quota on directories in cephfs. Same for > snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. > > John > > > Thank you, > > Ju Lim > > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim -------------- next part -------------- An HTML attachment was scrubbed... URL: From nlevine at redhat.com Tue Sep 27 21:27:49 2016 From: nlevine at redhat.com (Neil Levine) Date: Tue, 27 Sep 2016 14:27:49 -0700 Subject: [Tendrl-devel] Architectural questions In-Reply-To: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> Message-ID: Ric, This is not about ceph-mgr. It's an implementation detail (not to dimiss the great effort that John is putting in!). Everyone not involved in core Ceph engineering can forget about ceph-mgr really. It will just be a different way for showing the same RESTful API we have for management currently. This is about defining what the boundaries are between Ceph and consumers of it. We have a RESTful API for installation (ceph-installer) and an RESTful API for management (calamari). These APIs are there to create an easier abstraction for the ecosystem to develop with and to ensure separation of responsibility between development teams. They are also in use by the current version of the product. If we are changing the Tendrl architecture to NOT consume these APIs, which is what is being proposed, we need to hear what the reasons are and *get buy in for them* as the current consensus seems to be it will a) slow down velocity b) create technical debt and c) not provide any value to the end user. Almost all management products that interface with 3rd party storage systems accept an API boundary presented to them. Just because Ceph is open source and allows other lower-level access mechanisms, it doesn't mean Tendrl should use them. Neil On Tue, Sep 27, 2016 at 12:18 PM, Ric Wheeler wrote: > On 09/27/2016 06:02 PM, John Spray wrote: > >> On Tue, Sep 27, 2016 at 3:39 PM, Mrugesh Karnik >> wrote: >> >>> On 27 September 2016 at 19:22, Jeff Applewhite >>> wrote: >>> >>>> Hi All >>>> >>>> Ceph: >>>> How will be be installing Ceph via Tendrl? >>>> >>> I'm investigating this. Two choices currently exist: ceph-installer and >>> ceph-ansible. I'm prioritising on ceph-ansible since adding ansible >>> support >>> naturally opens up the provisioning framework for systems other than >>> ceph to >>> be supported through. >>> >>> >>> How will we be provisioning volumes/pools/rbds/etc in Tendrl? >>>> >>> The ceph bridge uses rados calls to get this done. The actual flows are >>> to >>> be translated over from skyring. Nishanth is leading this effort. >>> >> This sounds ominously like you've gone back to intending to use >> librados calls directly to the mon from tendrl :-/ >> >> I thought when I asked about this two weeks ago you said the >> ceph_bridge use of librados was just a prototype? >> >> John >> > > Hi John, > > My understanding is that we are clear that we intend to move to ceph-mgr > as soon as ceph-mgr is ready for use. > > No backtracking on this, it is really important that we do this to keep > the ceph community on board and enable them to contribute (and us to ride > other teams' work) as we go forward. > > Not if we make the switch, but rather *when* we can make that switch. > > Ric > > > >> >> >> Gluster: >>>> How will we be installing Gluster? >>>> >>> gdeploy is a candidate. However, as part of the ansible integration >>> evaluation for ceph-ansible, future directions will be known. >>> >>> >>> What is our plan for provisioning volumes/bricks/etc? >>>> It sounds like the Heketi adoption means that we would not be able to >>>> allow users of Gluster to issue gluster commands and they would instead >>>> have >>>> to issue Heketi command line statements. This seems to be an >>>> unacceptable >>>> tradeoff to me. >>>> >>> Heketi has either already implemented or is planning on implementing the >>> volume and brick handling functionality that tendrl would eventually need >>> to. Hence the best course of action would be to understand if and how >>> heketi's functionality could be leveraged by tendrl, cleanly. Failing >>> this, >>> it's possible to implement the logic in tendrl. >>> >>> We'll have a decision on this issue by Friday, after a discussion with >>> Luis. >>> >>> >>> Core: >>>> How will we store and display time series data? >>>> >>> This is being looked at in the current sprint. >>> >>> How will we establish a secure channel to the managed Ceph or Gluster >>>> nodes? >>>> >>> This is currently an open issue, which will be picked up in a latter >>> sprint, >>> once the basic node agent functionality has been implemented. >>> >>> >>> >>> -- >>> Mrugesh >>> >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >>> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Wed Sep 28 01:04:43 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Wed, 28 Sep 2016 06:34:43 +0530 Subject: [Tendrl-devel] [FYI] New email list for Heketi Message-ID: The developers working on heketi now have a new mailing list at Prior to this, all heketi related conversations were on the gluster-devel list. -- sankarshan mukhopadhyay From nthomas at redhat.com Wed Sep 28 07:55:48 2016 From: nthomas at redhat.com (Nishanth Thomas) Date: Wed, 28 Sep 2016 13:25:48 +0530 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: Its good that we are not going to unify the dashboard pieces. With this in mind how the main dashboard looks if we have a deployment with ceph and Gluster? Also I have a concern with respect to unifying 'File shares' 'quotas' etc, as it might complicate the navigation. Rather we could look at a plug-in based approach where file system specific things could be kept separately and installed on a need basis. On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: > Hi John: > > Thank you for taking the time to review and sharing your feedback. > > WRT terminology, I think that's a tricky one and I can understand and > appreciate the confusion with RBD (vs. Block Device) and File Share. The > original decision to go with RBD was that it was very Ceph-specific in > terms of the implementation and varies from LVM and other block > technologies. Having said that, I would agree that File Share deviates > from this initial premise since it could mean a Gluster volume or a CephFS > fileshare. > > The current thinking regarding File Share is to use it for both Ceph and > Gluster, and have a way to qualify it through the notion of the workload. > Part of the rationale was to try to make it so users did not have to think > should I use Ceph or Gluster, but rather let the System make an > "intelligent" choice for the mere mortal user based on just enough > information that user provides. This way, the user thinks of it as Red Hat > Storage providing this capability, and if we swapped out Ceph or Gluster > for something in the future, it would allow for that. In conjunction, > there would still be a way for a user to qualify it for Ceph or Gluster as > well, so it's not just for mere mortal but also a more knowledge / ninja > user. > > Having said that, RGW (and Swift) are definitely in mind too, so this is > not limited to just File Shares but applies to pretty much block, object, > and file services. > > With regards to quotas and snapshots, I do agree that there are many > different kinds. It did not seem prudent to create a Pool Snapshot, RBD > Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, Pool > Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not > scalable (and would get very crowded very quickly), and would create a > fairly complex experience for the user. My current thinking is that when > we get to the Quotas or Snapshots section, we would either have a unified > view with filtering capabilities (for the different types), or tabbed > views, or a 3rd level navigation, or some other combination. Since quotas > and snapshots are not in the immediate release plans, I've deferred this to > the detailed design stage when we tackle those topics. > > I hope this addresses your concerns raised. I'll update the slidedeck > with these comments before it gets published out more formally. > > Thanks again, > Ju > > > > On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: > >> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: >> > Hi. I've started documenting the UX Design Approach and Information >> > Architecture (JIRA TEN-40). It includes some of our guiding principles >> and >> > decisions that we've made in our design. >> > >> > Here's an initial draft that you can take a look at: >> > https://docs.google.com/a/redhat.com/presentation/d/1P4ejy0Q >> 7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing >> > >> > Note: For now it's open to be comments by anyone with a Red Hat >> account. I >> > plan to publish this to Jira and/or GitHub before the end of Sprint 1 >> (Oct >> > 4), but I'd like to open this up to review, comments, and discussion in >> the >> > meantime. >> >> Thanks for posting! >> >> My main piece of feedback is on terminology -- I think it's going to >> get confusing to use a mixture of qualified names (e.g. "RBD" instead >> of "block device") and unqualified names (e.g. "File Share" when we >> really mean "Gluster"). >> >> I have a special attachment to use of "File Share" because when Tendrl >> gets support for CephFS, everywhere the term is used is going to need >> qualifying with whether we're talking about Ceph or Gluster. I don't >> think anyone is working on adding Tendrl support for CephFS at the >> moment, but it should be part of the design process, along with RGW. >> >> Terms like "Quotas" are also dangerous, because we have so many >> different kinds. In Ceph alone, we have Quotas on pools, and then a >> totally different type of quota on directories in cephfs. Same for >> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. >> >> John >> >> > Thank you, >> > Ju Lim >> > >> > >> > _______________________________________________ >> > Tendrl-devel mailing list >> > Tendrl-devel at redhat.com >> > https://www.redhat.com/mailman/listinfo/tendrl-devel >> > >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > -- > Ju Lim > Red Hat > Office: 978-399-0422 > Mobile: 781-507-1323 > Email: julim at redhat.com > IRC: julim > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rwheeler at redhat.com Wed Sep 28 13:12:31 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Wed, 28 Sep 2016 09:12:31 -0400 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> Message-ID: <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> On 09/27/2016 05:27 PM, Neil Levine wrote: > Ric, > > This is not about ceph-mgr. It's an implementation detail (not to dimiss the > great effort that John is putting in!). Everyone not involved in core Ceph > engineering can forget about ceph-mgr really. It will just be a different way > for showing the same RESTful API we have for management currently. > > This is about defining what the boundaries are between Ceph and consumers of > it. We have a RESTful API for installation (ceph-installer) and an RESTful API > for management (calamari). These APIs are there to create an easier > abstraction for the ecosystem to develop with and to ensure separation of > responsibility between development teams. They are also in use by the current > version of the product. > > If we are changing the Tendrl architecture to NOT consume these APIs, which is > what is being proposed, we need to hear what the reasons are and *get buy in > for them* as the current consensus seems to be it will a) slow down velocity > b) create technical debt and c) not provide any value to the end user. > > Almost all management products that interface with 3rd party storage systems > accept an API boundary presented to them. Just because Ceph is open source and > allows other lower-level access mechanisms, it doesn't mean Tendrl should use > them. > > Neil > > Are you saying that ceph-mgr is not the source of the restful API's? That is the plan that we are working towards. There is no intention to go around those external API's once ceph-mgr is ready to step up. Ric From jspray at redhat.com Wed Sep 28 13:27:08 2016 From: jspray at redhat.com (John Spray) Date: Wed, 28 Sep 2016 14:27:08 +0100 Subject: [Tendrl-devel] Architectural questions In-Reply-To: <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: On Wed, Sep 28, 2016 at 2:12 PM, Ric Wheeler wrote: > On 09/27/2016 05:27 PM, Neil Levine wrote: >> >> Ric, >> >> This is not about ceph-mgr. It's an implementation detail (not to dimiss >> the great effort that John is putting in!). Everyone not involved in core >> Ceph engineering can forget about ceph-mgr really. It will just be a >> different way for showing the same RESTful API we have for management >> currently. >> >> This is about defining what the boundaries are between Ceph and consumers >> of it. We have a RESTful API for installation (ceph-installer) and an >> RESTful API for management (calamari). These APIs are there to create an >> easier abstraction for the ecosystem to develop with and to ensure >> separation of responsibility between development teams. They are also in use >> by the current version of the product. >> >> If we are changing the Tendrl architecture to NOT consume these APIs, >> which is what is being proposed, we need to hear what the reasons are and >> *get buy in for them* as the current consensus seems to be it will a) slow >> down velocity b) create technical debt and c) not provide any value to the >> end user. >> >> Almost all management products that interface with 3rd party storage >> systems accept an API boundary presented to them. Just because Ceph is open >> source and allows other lower-level access mechanisms, it doesn't mean >> Tendrl should use them. >> >> Neil >> >> > Are you saying that ceph-mgr is not the source of the restful API's? That is > the plan that we are working towards. > > There is no intention to go around those external API's once ceph-mgr is > ready to step up. I think we've veered a bit from the specific technical issue I was trying to get at. My concern is that there is code out there in ceph_bridge that uses librados to talk to Ceph mons directly. That's undesirable, because I don't want it to treat e.g. the format of the JSON dumps from Ceph commands as a stable interface. It's also a wasteful distraction from the work to use a long term supportable interface (either ceph-mgr python modules, or a REST API). The immediate engineering issue here is that we're going to have two versions of Tendrl (the pre-Luminous one and the post-Luminous one) with two different versions of Ceph (Jewel and Luminous). Calling the Tendrl versions T1 and T2 respectively, we have the following cases: T1->Jewel T2->Jewel (assuming Tendrl remains backwards compatible) T2->Luminous What we need from the Tendrl engineering folks is a clear statement about which interfaces they intend to us. For example, this would be reasonable if the Tendrl folks want to consume REST. T1->J Calamari REST API T2->J Calamari REST API T2->L ceph-mgr REST API ...or this if they want to consume REST transitionally but then move to a native ceph-mgr module in some future T3 version: T1->J Calamari REST API T2->J Calamari REST API T2->L ceph-mgr REST API T3+->L ceph-mgr Tendrl module ...or this if they want to move to a native ceph-mgr module as soon as Luminous is supported: T1->J Calamari REST API T2->J Calamari REST API T2->L ceph-mgr Tendrl module What I'm afraid of is that someone is intending to do this: T1->J raw librados commands T2->J raw librados commands T2->L raw librados commands T3+->L ceph-mgr Tendrl module John > Ric > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From nlevine at redhat.com Wed Sep 28 16:55:40 2016 From: nlevine at redhat.com (Neil Levine) Date: Wed, 28 Sep 2016 09:55:40 -0700 Subject: [Tendrl-devel] Architectural questions In-Reply-To: <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: Ric: ceph-mgr is a low-level, Ceph native control plane that uses Ceph protocols to pass information to and from Ceph components. It can then be extended via plugins to display information in any way you want (REST, SNMP, ascii art, blinking LEDs). For its first release, it will have a plugin (called rest.py i think) that will offer the *same* API that is currently provided by the Calamari code. Once ceph-mgr stabilizes, the Calamari implementation will disappear and ceph-mgr + rest.py will become the default way to access the Ceph REST API. The key point is that Ceph has now and will continue to have a stable RESTful API. On Wed, Sep 28, 2016 at 6:12 AM, Ric Wheeler wrote: > On 09/27/2016 05:27 PM, Neil Levine wrote: > >> Ric, >> >> This is not about ceph-mgr. It's an implementation detail (not to dimiss >> the great effort that John is putting in!). Everyone not involved in core >> Ceph engineering can forget about ceph-mgr really. It will just be a >> different way for showing the same RESTful API we have for management >> currently. >> >> This is about defining what the boundaries are between Ceph and consumers >> of it. We have a RESTful API for installation (ceph-installer) and an >> RESTful API for management (calamari). These APIs are there to create an >> easier abstraction for the ecosystem to develop with and to ensure >> separation of responsibility between development teams. They are also in >> use by the current version of the product. >> >> If we are changing the Tendrl architecture to NOT consume these APIs, >> which is what is being proposed, we need to hear what the reasons are and >> *get buy in for them* as the current consensus seems to be it will a) slow >> down velocity b) create technical debt and c) not provide any value to the >> end user. >> >> Almost all management products that interface with 3rd party storage >> systems accept an API boundary presented to them. Just because Ceph is open >> source and allows other lower-level access mechanisms, it doesn't mean >> Tendrl should use them. >> >> Neil >> >> >> Are you saying that ceph-mgr is not the source of the restful API's? That > is the plan that we are working towards. > > There is no intention to go around those external API's once ceph-mgr is > ready to step up. > > Ric > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rwheeler at redhat.com Wed Sep 28 17:03:03 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Wed, 28 Sep 2016 13:03:03 -0400 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: <5cea1a66-f7ac-97f3-127c-262eb97fbf6b@redhat.com> On 09/28/2016 12:55 PM, Neil Levine wrote: > Ric: ceph-mgr is a low-level, Ceph native control plane that uses Ceph > protocols to pass information to and from Ceph components. It can then be > extended via plugins to display information in any way you want (REST, SNMP, > ascii art, blinking LEDs). > > For its first release, it will have a plugin (called rest.py i think) that > will offer the *same* API that is currently provided by the Calamari code. > Once ceph-mgr stabilizes, the Calamari implementation will disappear and > ceph-mgr + rest.py will become the default way to access the Ceph REST API. > > The key point is that Ceph has now and will continue to have a stable RESTful API. That is exactly what I was stating below. Their is no difference of approach here that I can see, there is a concern over timing and possible duplication of work (throw away code if we need to invest too much using the pre-ceph-mgr path). Ric > > > > On Wed, Sep 28, 2016 at 6:12 AM, Ric Wheeler > wrote: > > On 09/27/2016 05:27 PM, Neil Levine wrote: > > Ric, > > This is not about ceph-mgr. It's an implementation detail (not to > dimiss the great effort that John is putting in!). Everyone not > involved in core Ceph engineering can forget about ceph-mgr really. It > will just be a different way for showing the same RESTful API we have > for management currently. > > This is about defining what the boundaries are between Ceph and > consumers of it. We have a RESTful API for installation > (ceph-installer) and an RESTful API for management (calamari). These > APIs are there to create an easier abstraction for the ecosystem to > develop with and to ensure separation of responsibility between > development teams. They are also in use by the current version of the > product. > > If we are changing the Tendrl architecture to NOT consume these APIs, > which is what is being proposed, we need to hear what the reasons are > and *get buy in for them* as the current consensus seems to be it will > a) slow down velocity b) create technical debt and c) not provide any > value to the end user. > > Almost all management products that interface with 3rd party storage > systems accept an API boundary presented to them. Just because Ceph is > open source and allows other lower-level access mechanisms, it doesn't > mean Tendrl should use them. > > Neil > > > Are you saying that ceph-mgr is not the source of the restful API's? That > is the plan that we are working towards. > > There is no intention to go around those external API's once ceph-mgr is > ready to step up. > > Ric > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From japplewh at redhat.com Wed Sep 28 18:19:56 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Wed, 28 Sep 2016 14:19:56 -0400 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: On Wed, Sep 28, 2016 at 9:27 AM, John Spray wrote: > On Wed, Sep 28, 2016 at 2:12 PM, Ric Wheeler wrote: > > On 09/27/2016 05:27 PM, Neil Levine wrote: > >> > >> Ric, > >> > >> This is not about ceph-mgr. It's an implementation detail (not to > dimiss > >> the great effort that John is putting in!). Everyone not involved in > core > >> Ceph engineering can forget about ceph-mgr really. It will just be a > >> different way for showing the same RESTful API we have for management > >> currently. > >> > >> This is about defining what the boundaries are between Ceph and > consumers > >> of it. We have a RESTful API for installation (ceph-installer) and an > >> RESTful API for management (calamari). These APIs are there to create an > >> easier abstraction for the ecosystem to develop with and to ensure > >> separation of responsibility between development teams. They are also > in use > >> by the current version of the product. > >> > >> If we are changing the Tendrl architecture to NOT consume these APIs, > >> which is what is being proposed, we need to hear what the reasons are > and > >> *get buy in for them* as the current consensus seems to be it will a) > slow > >> down velocity b) create technical debt and c) not provide any value to > the > >> end user. > >> > >> Almost all management products that interface with 3rd party storage > >> systems accept an API boundary presented to them. Just because Ceph is > open > >> source and allows other lower-level access mechanisms, it doesn't mean > >> Tendrl should use them. > >> > >> Neil > >> > >> > > Are you saying that ceph-mgr is not the source of the restful API's? > That is > > the plan that we are working towards. > > > > There is no intention to go around those external API's once ceph-mgr is > > ready to step up. > > I think we've veered a bit from the specific technical issue I was > trying to get at. > > My concern is that there is code out there in ceph_bridge that uses > librados to talk to Ceph mons directly. That's undesirable, because I > don't want it to treat e.g. the format of the JSON dumps from Ceph > commands as a stable interface. It's also a wasteful distraction from > the work to use a long term supportable interface (either ceph-mgr > python modules, or a REST API). > > The immediate engineering issue here is that we're going to have two > versions of Tendrl (the pre-Luminous one and the post-Luminous one) > with two different versions of Ceph (Jewel and Luminous). Calling the > Tendrl versions T1 and T2 respectively, we have the following cases: > > T1->Jewel > T2->Jewel (assuming Tendrl remains backwards compatible) > T2->Luminous > > What we need from the Tendrl engineering folks is a clear statement > about which interfaces they intend to us. For example, this would be > reasonable if the Tendrl folks want to consume REST. > > T1->J Calamari REST API > T2->J Calamari REST API > T2->L ceph-mgr REST API > > ...or this if they want to consume REST transitionally but then move > to a native ceph-mgr module in some future T3 version: > > T1->J Calamari REST API > T2->J Calamari REST API > T2->L ceph-mgr REST API > T3+->L ceph-mgr Tendrl module > > ...or this if they want to move to a native ceph-mgr module as soon as > Luminous is supported: > > T1->J Calamari REST API > T2->J Calamari REST API > T2->L ceph-mgr Tendrl module > > What I'm afraid of is that someone is intending to do this: > > T1->J raw librados commands > T2->J raw librados commands > T2->L raw librados commands > T3+->L ceph-mgr Tendrl module > ?John this is not what the current thinking is AFAIK. First of all I don't want to speak up too loudly for those on the architecture side (they will chime in tomorrow mid day IST) but my understanding is that we are drawing a bright red line between operations that create change (creates, updates, deletes of ceph objects) and those operations that rapidly and efficiently *query* the current cluster map in a read only fashion. The former should take place through an approved API firewall - I think we are agreed on this. The latter are considered necessary to solve the sync state problem (updating UI components efficiently and in one pass rather than piecemeal). Performance is a big concern here as well as scalability of the API interface. But let's not debate the merits of what I have said just yet before others can revise and extend it. John > > > > > Ric > > > > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager -------------- next part -------------- An HTML attachment was scrubbed... URL: From julim at redhat.com Wed Sep 28 19:41:53 2016 From: julim at redhat.com (Ju Lim) Date: Wed, 28 Sep 2016 15:41:53 -0400 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: Hi Nishant: Thanks for reviewing. With regards how the dashboard will look when Ceph and Gluster is in the picture: - When there is a single storage subsystem present, the single dashboard for the single storage subsystem is presented by default. - When there are multiple storage subsystems present, the Dashboard would present each dashboard in its own tab. Basically, we'd have a Ceph tab and a Gluster tab. - Ideally if the default dashboard tab can be specified on a per user basis, e.g. in a user?s profile/setting -- initially configured by the Administrator user. It's a nice-to-have at this point, and if folks agree, it should be added to the backlog. With regards to unifying concepts such as quotas, file shares, etc., you raise a fair concern. There's multiple approaches to it. For the quotas, snapshots, my current thinking is that when we get to the Quotas or Snapshots section, we would either have a unified view with filtering capabilities (for the different types), or tabbed views, or a 3rd level navigation, or some other combination. This will be determined at a later date when the feature gets added. Going with a tabbed approach or 3rd level navigation could potentially allow for a plug-in based approach whereby specific storage subsystem capabilities can be exposed as needed. For File Shares, one approach I suggested was to handle it via the user's workflow. Another approach could be to handle it via the navigation which potentially increases complexity and may impact navigation by introducing more levels of navigation. Our goal is not to go deeper than 3 levels of navigation as it will become an unwieldy user experience if things are buried too deeply. That being said, whether it's handled via the workflow, navigation, or some other means, using a plug-in based approach should still work. I've made updates to the document UX Design Approach and Information Architecture to incorporate the points raised by both yours and John's feedback. Thank you, Ju On Wed, Sep 28, 2016 at 3:55 AM, Nishanth Thomas wrote: > Its good that we are not going to unify the dashboard pieces. With this in > mind how the main dashboard looks if we have a deployment with ceph and > Gluster? > Also I have a concern with respect to unifying 'File shares' 'quotas' etc, > as it might complicate the navigation. Rather we could look at a plug-in > based approach where file system specific things could be kept separately > and installed on a need basis. > > On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: > >> Hi John: >> >> Thank you for taking the time to review and sharing your feedback. >> >> WRT terminology, I think that's a tricky one and I can understand and >> appreciate the confusion with RBD (vs. Block Device) and File Share. The >> original decision to go with RBD was that it was very Ceph-specific in >> terms of the implementation and varies from LVM and other block >> technologies. Having said that, I would agree that File Share deviates >> from this initial premise since it could mean a Gluster volume or a CephFS >> fileshare. >> >> The current thinking regarding File Share is to use it for both Ceph and >> Gluster, and have a way to qualify it through the notion of the workload. >> Part of the rationale was to try to make it so users did not have to think >> should I use Ceph or Gluster, but rather let the System make an >> "intelligent" choice for the mere mortal user based on just enough >> information that user provides. This way, the user thinks of it as Red Hat >> Storage providing this capability, and if we swapped out Ceph or Gluster >> for something in the future, it would allow for that. In conjunction, >> there would still be a way for a user to qualify it for Ceph or Gluster as >> well, so it's not just for mere mortal but also a more knowledge / ninja >> user. >> >> Having said that, RGW (and Swift) are definitely in mind too, so this is >> not limited to just File Shares but applies to pretty much block, object, >> and file services. >> >> With regards to quotas and snapshots, I do agree that there are many >> different kinds. It did not seem prudent to create a Pool Snapshot, RBD >> Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, Pool >> Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not >> scalable (and would get very crowded very quickly), and would create a >> fairly complex experience for the user. My current thinking is that when >> we get to the Quotas or Snapshots section, we would either have a unified >> view with filtering capabilities (for the different types), or tabbed >> views, or a 3rd level navigation, or some other combination. Since quotas >> and snapshots are not in the immediate release plans, I've deferred this to >> the detailed design stage when we tackle those topics. >> >> I hope this addresses your concerns raised. I'll update the slidedeck >> with these comments before it gets published out more formally. >> >> Thanks again, >> Ju >> >> >> >> On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: >> >>> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: >>> > Hi. I've started documenting the UX Design Approach and Information >>> > Architecture (JIRA TEN-40). It includes some of our guiding >>> principles and >>> > decisions that we've made in our design. >>> > >>> > Here's an initial draft that you can take a look at: >>> > https://docs.google.com/a/redhat.com/presentation/d/1P4ejy0Q >>> 7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing >>> > >>> > Note: For now it's open to be comments by anyone with a Red Hat >>> account. I >>> > plan to publish this to Jira and/or GitHub before the end of Sprint 1 >>> (Oct >>> > 4), but I'd like to open this up to review, comments, and discussion >>> in the >>> > meantime. >>> >>> Thanks for posting! >>> >>> My main piece of feedback is on terminology -- I think it's going to >>> get confusing to use a mixture of qualified names (e.g. "RBD" instead >>> of "block device") and unqualified names (e.g. "File Share" when we >>> really mean "Gluster"). >>> >>> I have a special attachment to use of "File Share" because when Tendrl >>> gets support for CephFS, everywhere the term is used is going to need >>> qualifying with whether we're talking about Ceph or Gluster. I don't >>> think anyone is working on adding Tendrl support for CephFS at the >>> moment, but it should be part of the design process, along with RGW. >>> >>> Terms like "Quotas" are also dangerous, because we have so many >>> different kinds. In Ceph alone, we have Quotas on pools, and then a >>> totally different type of quota on directories in cephfs. Same for >>> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. >>> >>> John >>> >>> > Thank you, >>> > Ju Lim >>> > >>> > >>> > _______________________________________________ >>> > Tendrl-devel mailing list >>> > Tendrl-devel at redhat.com >>> > https://www.redhat.com/mailman/listinfo/tendrl-devel >>> > >>> >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >> >> >> >> -- >> Ju Lim >> Red Hat >> Office: 978-399-0422 >> Mobile: 781-507-1323 >> Email: julim at redhat.com >> IRC: julim >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> >> > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Wed Sep 28 23:15:51 2016 From: jspray at redhat.com (John Spray) Date: Thu, 29 Sep 2016 00:15:51 +0100 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: On Wed, Sep 28, 2016 at 8:41 PM, Ju Lim wrote: > Hi Nishant: > > Thanks for reviewing. With regards how the dashboard will look when Ceph > and Gluster is in the picture: > > When there is a single storage subsystem present, the single dashboard for > the single storage subsystem is presented by default. > > When there are multiple storage subsystems present, the Dashboard would > present each dashboard in its own tab. Basically, we'd have a Ceph tab and > a Gluster tab. > > Ideally if the default dashboard tab can be specified on a per user basis, > e.g. in a user?s profile/setting -- initially configured by the > Administrator user. It's a nice-to-have at this point, and if folks agree, > it should be added to the backlog. > > With regards to unifying concepts such as quotas, file shares, etc., you > raise a fair concern. There's multiple approaches to it. For the quotas, > snapshots, my current thinking is that when we get to the Quotas or > Snapshots section, we would either have a unified view with filtering > capabilities (for the different types), or tabbed views, or a 3rd level > navigation, or some other combination. This will be determined at a later > date when the feature gets added. Going with a tabbed approach or 3rd level > navigation could potentially allow for a plug-in based approach whereby > specific storage subsystem capabilities can be exposed as needed. > > For File Shares, one approach I suggested was to handle it via the user's > workflow. Another approach could be to handle it via the navigation which > potentially increases complexity and may impact navigation by introducing > more levels of navigation. Our goal is not to go deeper than 3 levels of > navigation as it will become an unwieldy user experience if things are > buried too deeply. That being said, whether it's handled via the workflow, > navigation, or some other means, using a plug-in based approach should still > work. Regarding navigation depth, I wonder if it's worth rethinking using the whole first level just to get into "Storage" or "Clusters" -- isn't everything we do about storage clusters? In my mind, things like cephfs, rbd, rgw are really top-level items. Regarding unifying quotas and snapshots, I think it would be natural for a user to go cephfs->quotas rather than quotas->cephfs, because at any given moment they are probably concentrating on one particular subsystem. I think the cephfs quotas and cephfs snapshots have more in common (and belong closer together) than e.g. cephfs quotas and gluster quotas, or cephfs snapshots and rbd snapshots. In a cephfs UI, we would probably hope to see a tree view that showed directories in the filesystem and enabled things like setting layouts and quotas on directories -- picturing that, I don't see a sane way for cephfs quotas to live under a top-level Quotas subsystem unless the whole widget was duplicated inside the CephFS subsystem as well. Not trying to entirely derail this into a discussion of CephFS, but I think the best way to see if this design is a good fit is to work through some of the more challenging cases like the N types of quota that will ultimately need managing (not just those supported in the first release of the software), and see if what comes out makes sense. By the way, I was just taking another look at slide 12, and I don't see user identities[1] (i.e. the identities that we use for access control of Ceph clients, not the Tendrl users). Those should probably be in there somewhere. They intersect both the "Cluster" section (ceph keys are managed by the mons) and the RBD/RGW/CephFS parts (to generate a key with meaningful capabilities you need to know which subsystem it's going to be using). John 1. http://docs.ceph.com/docs/hammer/rados/operations/user-management/ > I've made updates to the document UX Design Approach and Information > Architecture to incorporate the points raised by both yours and John's > feedback. > > Thank you, > Ju > > > > On Wed, Sep 28, 2016 at 3:55 AM, Nishanth Thomas wrote: >> >> Its good that we are not going to unify the dashboard pieces. With this in >> mind how the main dashboard looks if we have a deployment with ceph and >> Gluster? >> Also I have a concern with respect to unifying 'File shares' 'quotas' etc, >> as it might complicate the navigation. Rather we could look at a plug-in >> based approach where file system specific things could be kept separately >> and installed on a need basis. >> >> On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: >>> >>> Hi John: >>> >>> Thank you for taking the time to review and sharing your feedback. >>> >>> WRT terminology, I think that's a tricky one and I can understand and >>> appreciate the confusion with RBD (vs. Block Device) and File Share. The >>> original decision to go with RBD was that it was very Ceph-specific in terms >>> of the implementation and varies from LVM and other block technologies. >>> Having said that, I would agree that File Share deviates from this initial >>> premise since it could mean a Gluster volume or a CephFS fileshare. >>> >>> The current thinking regarding File Share is to use it for both Ceph and >>> Gluster, and have a way to qualify it through the notion of the workload. >>> Part of the rationale was to try to make it so users did not have to think >>> should I use Ceph or Gluster, but rather let the System make an >>> "intelligent" choice for the mere mortal user based on just enough >>> information that user provides. This way, the user thinks of it as Red Hat >>> Storage providing this capability, and if we swapped out Ceph or Gluster for >>> something in the future, it would allow for that. In conjunction, there >>> would still be a way for a user to qualify it for Ceph or Gluster as well, >>> so it's not just for mere mortal but also a more knowledge / ninja user. >>> >>> Having said that, RGW (and Swift) are definitely in mind too, so this is >>> not limited to just File Shares but applies to pretty much block, object, >>> and file services. >>> >>> With regards to quotas and snapshots, I do agree that there are many >>> different kinds. It did not seem prudent to create a Pool Snapshot, RBD >>> Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, Pool >>> Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not >>> scalable (and would get very crowded very quickly), and would create a >>> fairly complex experience for the user. My current thinking is that when we >>> get to the Quotas or Snapshots section, we would either have a unified view >>> with filtering capabilities (for the different types), or tabbed views, or a >>> 3rd level navigation, or some other combination. Since quotas and snapshots >>> are not in the immediate release plans, I've deferred this to the detailed >>> design stage when we tackle those topics. >>> >>> I hope this addresses your concerns raised. I'll update the slidedeck >>> with these comments before it gets published out more formally. >>> >>> Thanks again, >>> Ju >>> >>> >>> >>> On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: >>>> >>>> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: >>>> > Hi. I've started documenting the UX Design Approach and Information >>>> > Architecture (JIRA TEN-40). It includes some of our guiding >>>> > principles and >>>> > decisions that we've made in our design. >>>> > >>>> > Here's an initial draft that you can take a look at: >>>> > >>>> > https://docs.google.com/a/redhat.com/presentation/d/1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing >>>> > >>>> > Note: For now it's open to be comments by anyone with a Red Hat >>>> > account. I >>>> > plan to publish this to Jira and/or GitHub before the end of Sprint 1 >>>> > (Oct >>>> > 4), but I'd like to open this up to review, comments, and discussion >>>> > in the >>>> > meantime. >>>> >>>> Thanks for posting! >>>> >>>> My main piece of feedback is on terminology -- I think it's going to >>>> get confusing to use a mixture of qualified names (e.g. "RBD" instead >>>> of "block device") and unqualified names (e.g. "File Share" when we >>>> really mean "Gluster"). >>>> >>>> I have a special attachment to use of "File Share" because when Tendrl >>>> gets support for CephFS, everywhere the term is used is going to need >>>> qualifying with whether we're talking about Ceph or Gluster. I don't >>>> think anyone is working on adding Tendrl support for CephFS at the >>>> moment, but it should be part of the design process, along with RGW. >>>> >>>> Terms like "Quotas" are also dangerous, because we have so many >>>> different kinds. In Ceph alone, we have Quotas on pools, and then a >>>> totally different type of quota on directories in cephfs. Same for >>>> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. >>>> >>>> John >>>> >>>> > Thank you, >>>> > Ju Lim >>>> > >>>> > >>>> > _______________________________________________ >>>> > Tendrl-devel mailing list >>>> > Tendrl-devel at redhat.com >>>> > https://www.redhat.com/mailman/listinfo/tendrl-devel >>>> > >>>> >>>> _______________________________________________ >>>> Tendrl-devel mailing list >>>> Tendrl-devel at redhat.com >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >>> >>> >>> >>> -- >>> Ju Lim >>> Red Hat >>> Office: 978-399-0422 >>> Mobile: 781-507-1323 >>> Email: julim at redhat.com >>> IRC: julim >>> >>> >>> _______________________________________________ >>> Tendrl-devel mailing list >>> Tendrl-devel at redhat.com >>> https://www.redhat.com/mailman/listinfo/tendrl-devel >>> >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel >> > > > > -- > Ju Lim > Red Hat > Office: 978-399-0422 > Mobile: 781-507-1323 > Email: julim at redhat.com > IRC: julim > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From rkanade at redhat.com Thu Sep 29 10:59:33 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Thu, 29 Sep 2016 16:29:33 +0530 Subject: [Tendrl-devel] installation instructions for Tendrl components In-Reply-To: References: <4f52ef4d-34b9-c6b9-c8f9-91184dbb5bd3@redhat.com> Message-ID: The import errors have been fixed, the install instructions will now work as the should. Please try to reproduce the same steps as above. Regards, Rohan Kanade On Tue, Sep 27, 2016 at 7:12 PM, Alfredo Deza wrote: > On Tue, Sep 27, 2016 at 9:27 AM, Martin Kudlej wrote: > > Hi Rohan, > > > > I write email because there are common comments for more tickets(TEN-53, > > TEN-54, TEN-55, TEN-57): > > 1) I would like to start discussion about environment supported by > Tendrl. I > > think it is required to define install instructions for supported OSes. I > > expect that there will be many common parts and differences will be only > > because of different packaging systems. What OSes would we like to > support > > for Tendrl 1.0? > > I've tried to define install instructions for CentOS 7 minimal install. > > > > 2) Common install instructions for all Tendrl components for CentOS 7. > I've > > installed only Git, Epel repo and Pip > > rpm -iUvh > > http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel- > release-7-8.noarch.rpm > > yum install git > > yum install python-pip > > > > Is there any software required for running Tendrl components? I expect > that > > I should install for example Etcd, should I? > > > > 3) bridge_common: > > git clone https://github.com/Tendrl/bridge_common.git > > > > try to install all requirements from requirements.txt(many of > requirements > > are common with other Tendrl components): > > yum install python-pbr > > pip install python-etcd > > yum install python-dateutil.noarch > > yum install python-gevent > > yum install pytz > > yum install gcc > > pip install oslo.log > > > > python setup.py install > > > > 4) ceph_bridge: > > git clone https://github.com/Tendrl/ceph_bridge.git > > > > try to install all requirements: > > yum install python2-msgpack > > > > python setup.py install > > > > There are unsatisfied imports. I've tried to run command installed to > > /usr/bin: > > $ /usr/bin/tendrl_ceph_bridge > > Traceback (most recent call last): > > File "/usr/bin/tendrl_ceph_bridge", line 6, in > > from ceph_bridge.manager.manager import main > > File "/usr/lib/python2.7/site-packages/ceph_bridge/manager/ > manager.py", > > line 14, in > > from ceph_bridge.manager.cluster_monitor import ClusterMonitor > > File > > "/usr/lib/python2.7/site-packages/ceph_bridge/manager/ > cluster_monitor.py", > > line 13, in > > from ceph_bridge.manager.crush_node_request_factory \ > > File > > "/usr/lib/python2.7/site-packages/ceph_bridge/manager/ > crush_node_request_factory.py", > > line 5, in > > from ceph_bridge.manager.server_monitor import ServiceId > > File > > "/usr/lib/python2.7/site-packages/ceph_bridge/manager/ > server_monitor.py", > > line 24, in > > from ceph_bridge.log import LOG as tendrl_log > > ImportError: cannot import name LOG > > > > 5) gluster_bridge: > > git clone https://github.com/Tendrl/gluster_bridge.git > > python setup.py install > > > > I've tried to run tendrl_gluster_bridge: > > $ /usr/bin/tendrl_gluster_bridge > > Traceback (most recent call last): > > File "/usr/bin/tendrl_gluster_bridge", line 6, in > > from gluster_bridge.manager.manager import main > > File "/usr/lib/python2.7/site-packages/gluster_bridge/ > manager/manager.py", > > line 17, in > > from gluster_bridge.persistence.persister import Persister > > File > > "/usr/lib/python2.7/site-packages/gluster_bridge/ > persistence/persister.py", > > line 1, in > > from etcdobj import Server as etcd_server > > ImportError: No module named etcdobj > > > > Should I install Etcd and/or any Python modules? > > > > 6) node_agent: > > git clone https://github.com/Tendrl/node_agent.git > > python setup.py install > > > > Could I try to create pull request to Tendrl documentation with these > > instructions? I think there are common part, so they can be shared to > > another Tendrl repos. > > Setting up a project like Tendrl will run into issues like this even > if documented (not that it shouldn't be documented :)). > > For the ceph-installer and almost all applications that support ceph > we use ansible for setting everything up. I highly recommend to start > automating this because it will provide: > > * fully repeatable deployment process > * immediate exposure to issues > * easier to impact a wide audience when fixes come in (e.g. new > dependency is added) > * easier to port "manual" installation steps (e.g. `python setup.py > install`) to actual production installs with binaries (yum/apt-get) > and have > then live alongside each other. > > > > > > I move all TEN-53, TEN-54, TEN-55, TEN-57 back to "In Progress" because > > tasks are not completed from my perspective. > > Thank you for fixing these issues. > > > > -- > > Best Regards, > > Martin Kudlej. > > RHSC/USM Senior Quality Assurance Engineer > > Red Hat Czech s.r.o. > > > > Phone: +420 532 294 155 > > E-mail:mkudlej at redhat.com > > IRC: mkudlej at #brno, #gluster, #storage-qa, #rhs, #rh-ceph, > #usm-meeting > > @ redhat > > #tendrl-devel @ freenode > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrugesh at brainfunked.org Thu Sep 29 11:42:08 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Thu, 29 Sep 2016 17:12:08 +0530 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: Long response below, but here's the TL;DR: * Using rest apis for state information gathering does not work for tendrl. * Invoking rest apis for operations is doable, but with reservations as to what precise benefit it brings, because: * Since calamari lite uses librados, wouldn't the problem with librados' json output affect it and in turn tendrl too? On 28 September 2016 at 18:57, John Spray wrote: > I think we've veered a bit from the specific technical issue I was > trying to get at. > > My concern is that there is code out there in ceph_bridge that uses > librados to talk to Ceph mons directly. That's undesirable, because I > don't want it to treat e.g. the format of the JSON dumps from Ceph > commands as a stable interface. Ceph architecture documentation points out librados as an interface directly consumable by apps[1]. From my understanding, projects such as openattic and calamari lite use librados. Is this incorrect? > It's also a wasteful distraction from > the work to use a long term supportable interface (either ceph-mgr > python modules, or a REST API). While this point is valid, ceph-mgr isn't currently available as a stable feature in ceph. > The immediate engineering issue here is that we're going to have two > versions of Tendrl (the pre-Luminous one and the post-Luminous one) > with two different versions of Ceph (Jewel and Luminous). Calling the > Tendrl versions T1 and T2 respectively, we have the following cases: > > T1->Jewel > T2->Jewel (assuming Tendrl remains backwards compatible) > T2->Luminous > > What we need from the Tendrl engineering folks is a clear statement > about which interfaces they intend to us. For example, this would be > reasonable if the Tendrl folks want to consume REST. > > T1->J Calamari REST API > T2->J Calamari REST API > T2->L ceph-mgr REST API Tendrl has evolved from skyrings[2]. Skyrings uses the calamari apis. A major problem faced with skyring was that skyring could fall out of sync with the ceph state should any operations be carried out on the cluster directly via the ceph cli. Skyrings also functions primarily as an application. I'm going to take a detour here and explain a bit about the thought behind tendrl's state information architecture. Tendrl is not intended to support ceph exclusively. Gluster is as important to tendrl as ceph is. When a system is designed to support multiple technologies, the logical path to take is to abstract. However, in the storage space, such abstractions have been based on very specific assumptions. These lead to components from different systems to be categories as a specific abstract entity. Example of this is categorising ceph pools and gluster volumes as `distributed logical storage' eg. Skyrings does this. This approach scales to an extent, at which point you may end up looking at a schema such as the cim schema. Storage management systems have been doing this traditionally by creating management interfaces as applications. Tendrl core aims to be a framework onto which applications such as the tendrl ui can be plugged. The question we evaluated was whether it was necessary to do this at all. Our realisation was that the management interface (api, gui etc.) does not need to understand the systems it encapsulates. Doing so means hardcoded, storage system knowledge being placed in an abstract, higher level interface. Tendrl hence relies on a configuration based approach where the state and the operations from the storage system are bubbled up to tendrl's interface by describing them in a configuration file. This approach isn't a ground-breaking new approach either. Configuration management systems like puppet and chef have approached a problem in a different domain in a similar manner. They support a custom dsl via which the invocations and abstractions are defined. The implementation details are left to the components specific to the systems. Tendrl does the same thing. Instead of the ceph specific codebase being in a component on the skyring side, tendrl pushes the component towards ceph, as the ceph bridge. So coming back to the problem at hand, that of state management, the rest api based approach implies that tendrl knows precisely what to ask for. The approach tendrl takes is that it gathers everything the system knows about itself, but doesn't try to understand, modify or fit any of that information into any specific schema. This where tendrl's object model abstraction comes into the picture. Every component on the system is an object, with it's own possibly unique properties, attributes, actions, states etc. The configuration files define this for tendrl. This way, we have a glue layer which allows to look at all the systems in an abstract manner. When any additional features are added, they're exposed via the configuration files by pointing to the relevant data. Because the state information aboute the system being managed is already available, there's no need to code in support for any additional api calls that may or may not exist in the system. So, tying this together, the point I'm getting to is this: * rest apis for gathering state information does not work for tendrl because tendrl needs all the information available at any point in time. Making calls to the rest api to fetch information bit-by-bit and reassembling on tendrl's side does not work for tendrl. Essentially, we need to be able to browse a book according to it's table of contents, rather than searching it for a specific phrase that our code is expected to know about. (Developers knowing the underlying mechanics of the system is essential, but what we're talking about here is the hardcoding of that domain logic in an abstract layer here, which tendrl is looking to avoid.) For ceph specifically, this problem has been addressed in tendrl by going directly to the layer where calamari gets it's information from. A similar approach was taken for gluster in cooperation with the gluster team. Both of these approaches worked out as intended in the poc and hence were carried forward. While this addresses specifically the state information part, using calamari apis for invoking the operations, in lieu of the librados calls, isn't as big a problem. Operations themselves are also defined as configuration files, but their implementation is left upto the operations bridges. However, this does bring my earlier point into focus. If librados is seen as brittle, aren't the calamari apis affected by it too? Also, is there any benefit to using the calamari apis in this case as opposed to the librados calls? [1] http://docs.ceph.com/docs/jewel/architecture/ [2] https://github.com/skyring -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrugesh at brainfunked.org Thu Sep 29 11:52:31 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Thu, 29 Sep 2016 17:22:31 +0530 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: On 28 September 2016 at 22:25, Neil Levine wrote: > > Ric: ceph-mgr is a low-level, Ceph native control plane that uses Ceph protocols to pass information to and from Ceph components. It can then be extended via plugins to display information in any way you want (REST, SNMP, ascii art, blinking LEDs). > > For its first release, it will have a plugin (called rest.py i think) that will offer the *same* API that is currently provided by the Calamari code. Once ceph-mgr stabilizes, the Calamari implementation will disappear and ceph-mgr + rest.py will become the default way to access the Ceph REST API. > > The key point is that Ceph has now and will continue to have a stable RESTful API. I've explained in details why tendrl isn't consuming the rest apis in response to John's earlier email: https://www.redhat.com/archives/tendrl-devel/2016-September/msg00106.html The response to this particular email comes down to the fact that using rest apis brings forth the same issues we've observed in skyrings. Tendrl's current implementation is a transition between the rest api interface and the eventual ceph-mgr python module interface. > On Wed, Sep 28, 2016 at 6:12 AM, Ric Wheeler wrote: >> >> On 09/27/2016 05:27 PM, Neil Levine wrote: >>> >>> Ric, >>> >>> This is not about ceph-mgr. It's an implementation detail (not to dimiss the great effort that John is putting in!). Everyone not involved in core Ceph engineering can forget about ceph-mgr really. It will just be a different way for showing the same RESTful API we have for management currently. >>> >>> This is about defining what the boundaries are between Ceph and consumers of it. We have a RESTful API for installation (ceph-installer) and an RESTful API for management (calamari). These APIs are there to create an easier abstraction for the ecosystem to develop with and to ensure separation of responsibility between development teams. They are also in use by the current version of the product. >>> >>> If we are changing the Tendrl architecture to NOT consume these APIs, which is what is being proposed, we need to hear what the reasons are and *get buy in for them* as the current consensus seems to be it will a) slow down velocity b) create technical debt and c) not provide any value to the end user. >>> >>> Almost all management products that interface with 3rd party storage systems accept an API boundary presented to them. Just because Ceph is open source and allows other lower-level access mechanisms, it doesn't mean Tendrl should use them. >>> >>> Neil >>> >>> >> Are you saying that ceph-mgr is not the source of the restful API's? That is the plan that we are working towards. >> >> There is no intention to go around those external API's once ceph-mgr is ready to step up. >> >> Ric >> >> >> >> _______________________________________________ >> Tendrl-devel mailing list >> Tendrl-devel at redhat.com >> https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rwheeler at redhat.com Thu Sep 29 11:52:59 2016 From: rwheeler at redhat.com (Ric Wheeler) Date: Thu, 29 Sep 2016 07:52:59 -0400 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: On 09/29/2016 07:42 AM, Mrugesh Karnik wrote: > My concern is that there is code out there in ceph_bridge that uses > > librados to talk to Ceph mons directly. That's undesirable, because I > > don't want it to treat e.g. the format of the JSON dumps from Ceph > > commands as a stable interface. > > Ceph architecture documentation points out librados as an interface directly > consumable by apps[1]. From my understanding, projects such as openattic and > calamari lite use librados. Is this incorrect? > > > It's also a wasteful distraction from > > the work to use a long term supportable interface (either ceph-mgr > > python modules, or a REST API). > > While this point is valid, ceph-mgr isn't currently available as a stable > feature in ceph. Just to be 100% clear, we all agree that this is a question of when we move to ceph-mgr, not if, right? Thanks! Ric From mrugesh at brainfunked.org Thu Sep 29 11:53:50 2016 From: mrugesh at brainfunked.org (Mrugesh Karnik) Date: Thu, 29 Sep 2016 17:23:50 +0530 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: On 29 September 2016 at 17:22, Ric Wheeler wrote: > > On 09/29/2016 07:42 AM, Mrugesh Karnik wrote: >> >> My concern is that there is code out there in ceph_bridge that uses >> > librados to talk to Ceph mons directly. That's undesirable, because I >> > don't want it to treat e.g. the format of the JSON dumps from Ceph >> > commands as a stable interface. >> >> Ceph architecture documentation points out librados as an interface directly consumable by apps[1]. From my understanding, projects such as openattic and calamari lite use librados. Is this incorrect? >> >> > It's also a wasteful distraction from >> > the work to use a long term supportable interface (either ceph-mgr >> > python modules, or a REST API). >> >> While this point is valid, ceph-mgr isn't currently available as a stable feature in ceph. > > > Just to be 100% clear, we all agree that this is a question of when we move to ceph-mgr, not if, right? That is correct. -- Mrugesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jspray at redhat.com Thu Sep 29 12:53:02 2016 From: jspray at redhat.com (John Spray) Date: Thu, 29 Sep 2016 13:53:02 +0100 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: On Thu, Sep 29, 2016 at 12:42 PM, Mrugesh Karnik wrote: > Long response below, but here's the TL;DR: > * Using rest apis for state information gathering does not work for tendrl. There is no semantic difference between doing mon_command("osd dump") instead of GET api/v2/osd Either way you're sending requests and getting responses. You don't magically get a different level of consistency. > * Invoking rest apis for operations is doable, but with reservations as to > what precise benefit it brings, because: The benefit is that you will be using a supported interface and not going "off the reservation". > * Since calamari lite uses librados, wouldn't the problem with librados' > json output affect it and in turn tendrl too? Yes, which is why Calamari is going away and the rest code is being brought inside the Ceph tree. A big motivation for that is to avoid having out-of-tree consumers of the CLI functions and their returned JSON structures. We are not going to take the backwards step of having a new out-of-tree consumer of the CLI functions in the form of Tendrl. Instead of using Calamari as an example (it is not exemplary!) you should listen to the people who wrote it (that includes me). > On 28 September 2016 at 18:57, John Spray wrote: >> I think we've veered a bit from the specific technical issue I was >> trying to get at. >> >> My concern is that there is code out there in ceph_bridge that uses >> librados to talk to Ceph mons directly. That's undesirable, because I >> don't want it to treat e.g. the format of the JSON dumps from Ceph >> commands as a stable interface. > > Ceph architecture documentation points out librados as an interface directly > consumable by apps[1]. From my understanding, projects such as openattic and > calamari lite use librados. Is this incorrect? You are correct that librados is a stable API. You are not correct in thinking that means that the commands you send via mon_command() are themselves a stable API. Those commands are essentially our CLI. They are pseudo-stable in that we don't change them gratuitously, but they are not meant to be a stable API. Especially, the JSON output you get from those commands comes from myriad ::dump() methods on structures within the Ceph codebase. They are not a stable API. I have severe reservations about the rest of your comments about having the UI be unaware of Ceph details, trying to expose features in the form of configuration files, etc. However, those are ultimately up to you. What isn't up to you is which interfaces Ceph will support for you. John > >> It's also a wasteful distraction from >> the work to use a long term supportable interface (either ceph-mgr >> python modules, or a REST API). > > While this point is valid, ceph-mgr isn't currently available as a stable > feature in ceph. > >> The immediate engineering issue here is that we're going to have two >> versions of Tendrl (the pre-Luminous one and the post-Luminous one) >> with two different versions of Ceph (Jewel and Luminous). Calling the >> Tendrl versions T1 and T2 respectively, we have the following cases: >> >> T1->Jewel >> T2->Jewel (assuming Tendrl remains backwards compatible) >> T2->Luminous >> >> What we need from the Tendrl engineering folks is a clear statement >> about which interfaces they intend to us. For example, this would be >> reasonable if the Tendrl folks want to consume REST. >> >> T1->J Calamari REST API >> T2->J Calamari REST API >> T2->L ceph-mgr REST API > > Tendrl has evolved from skyrings[2]. Skyrings uses the calamari apis. A > major problem faced with skyring was that skyring could fall out of sync > with the ceph state should any operations be carried out on the cluster > directly via the ceph cli. Skyrings also functions primarily as an > application. > > I'm going to take a detour here and explain a bit about the thought behind > tendrl's state information architecture. > > Tendrl is not intended to support ceph exclusively. Gluster is as important > to tendrl as ceph is. When a system is designed to support multiple > technologies, the logical path to take is to abstract. However, in the > storage space, such abstractions have been based on very specific > assumptions. These lead to components from different systems to be > categories as a specific abstract entity. Example of this is categorising > ceph pools and gluster volumes as `distributed logical storage' eg. Skyrings > does this. This approach scales to an extent, at which point you may end up > looking at a schema such as the cim schema. > > Storage management systems have been doing this traditionally by creating > management interfaces as applications. Tendrl core aims to be a framework > onto which applications such as the tendrl ui can be plugged. The question > we evaluated was whether it was necessary to do this at all. Our realisation > was that the management interface (api, gui etc.) does not need to > understand the systems it encapsulates. Doing so means hardcoded, storage > system knowledge being placed in an abstract, higher level interface. Tendrl > hence relies on a configuration based approach where the state and the > operations from the storage system are bubbled up to tendrl's interface by > describing them in a configuration file. > > This approach isn't a ground-breaking new approach either. Configuration > management systems like puppet and chef have approached a problem in a > different domain in a similar manner. They support a custom dsl via which > the invocations and abstractions are defined. The implementation details are > left to the components specific to the systems. > > Tendrl does the same thing. Instead of the ceph specific codebase being in a > component on the skyring side, tendrl pushes the component towards ceph, as > the ceph bridge. > > So coming back to the problem at hand, that of state management, the rest > api based approach implies that tendrl knows precisely what to ask for. The > approach tendrl takes is that it gathers everything the system knows about > itself, but doesn't try to understand, modify or fit any of that information > into any specific schema. This where tendrl's object model abstraction comes > into the picture. Every component on the system is an object, with it's own > possibly unique properties, attributes, actions, states etc. The > configuration files define this for tendrl. This way, we have a glue layer > which allows to look at all the systems in an abstract manner. When any > additional features are added, they're exposed via the configuration files > by pointing to the relevant data. Because the state information aboute the > system being managed is already available, there's no need to code in > support for any additional api calls that may or may not exist in the > system. > > So, tying this together, the point I'm getting to is this: > * rest apis for gathering state information does not work for tendrl because > tendrl needs all the information available at any point in time. Making > calls to the rest api to fetch information bit-by-bit and reassembling on > tendrl's side does not work for tendrl. > > Essentially, we need to be able to browse a book according to it's table of > contents, rather than searching it for a specific phrase that our code is > expected to know about. (Developers knowing the underlying mechanics of the > system is essential, but what we're talking about here is the hardcoding of > that domain logic in an abstract layer here, which tendrl is looking to > avoid.) > > For ceph specifically, this problem has been addressed in tendrl by going > directly to the layer where calamari gets it's information from. A similar > approach was taken for gluster in cooperation with the gluster team. Both of > these approaches worked out as intended in the poc and hence were carried > forward. > > While this addresses specifically the state information part, using calamari > apis for invoking the operations, in lieu of the librados calls, isn't as > big a problem. Operations themselves are also defined as configuration > files, but their implementation is left upto the operations bridges. > However, this does bring my earlier point into focus. If librados is seen as > brittle, aren't the calamari apis affected by it too? Also, is there any > benefit to using the calamari apis in this case as opposed to the librados > calls? > > [1] http://docs.ceph.com/docs/jewel/architecture/ > [2] https://github.com/skyring > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > From rkanade at redhat.com Thu Sep 29 13:37:28 2016 From: rkanade at redhat.com (Rohan Kanade) Date: Thu, 29 Sep 2016 19:07:28 +0530 Subject: [Tendrl-devel] Architectural questions In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: On Thu, Sep 29, 2016 at 6:23 PM, John Spray wrote: > Yes, which is why Calamari is going away and the rest code is being > brought inside the Ceph tree. A big motivation for that is to avoid > having out-of-tree consumers of the CLI functions and their returned > JSON structures. We are not going to take the backwards step of > having a new out-of-tree consumer of the CLI functions in the form of > Tendrl. > Usage of these interfaces/CLI functions (librados) are well documented and encouraged by Ceph , For frameworks like Tendrl to avoid using them, appropriate deprecation warnings in Ceph/librados are required which discourage use of these interfaces/CLI functions Here's how OpenStack handle's interface/API deprecation: https://governance.openstack.org/reference/tags/assert_follows-standard-deprecation.html Reference: http://docs.ceph.com/docs/jewel/rados/api/librados/#rados_mon_command > Instead of using Calamari as an example (it is not exemplary!) you > should listen to the people who wrote it (that includes me). > > > On 28 September 2016 at 18:57, John Spray wrote: > >> I think we've veered a bit from the specific technical issue I was > >> trying to get at. > >> > >> My concern is that there is code out there in ceph_bridge that uses > >> librados to talk to Ceph mons directly. That's undesirable, because I > >> don't want it to treat e.g. the format of the JSON dumps from Ceph > >> commands as a stable interface. > > > > Ceph architecture documentation points out librados as an interface directly > > consumable by apps[1]. From my understanding, projects such as openattic and > > calamari lite use librados. Is this incorrect? > > You are correct that librados is a stable API. You are not correct in > thinking that means that the commands you send via mon_command() are > themselves a stable API. Those commands are essentially our CLI. > They are pseudo-stable in that we don't change them gratuitously, but > they are not meant to be a stable API. > > Especially, the JSON output you get from those commands comes from > myriad ::dump() methods on structures within the Ceph codebase. They > are not a stable API. > If it is not a stable API it needs to be documented by the Ceph project upstream. > I have severe reservations about the rest of your comments about > having the UI be unaware of Ceph details, trying to expose features in > the form of configuration files, etc. However, those are ultimately > up to you. What isn't up to you is which interfaces Ceph will support > for you. > > John > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssaha at redhat.com Thu Sep 29 14:41:36 2016 From: ssaha at redhat.com (Sayan Saha) Date: Thu, 29 Sep 2016 10:41:36 -0400 Subject: [Tendrl-devel] Architectural questions - need clear pros & cons for the integration options In-Reply-To: References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> Message-ID: <571db346-0ca1-894c-2cfe-0cac5ade920b@redhat.com> Hi Mrugesh, This is not an easy read. Can you please send a clear email outlining the 3 options (that I have heard about) for RHScon 3 + RHCS integration with a pros & cons analysis for each. This will greatly help expedite decision making. Thanks Sayan On 9/29/16 7:42 AM, Mrugesh Karnik wrote: > Long response below, but here's the TL;DR: > * Using rest apis for state information gathering does not work for > tendrl. > * Invoking rest apis for operations is doable, but with reservations > as to what precise benefit it brings, because: > * Since calamari lite uses librados, wouldn't the problem with > librados' json output affect it and in turn tendrl too? > > > On 28 September 2016 at 18:57, John Spray > wrote: > > I think we've veered a bit from the specific technical issue I was > > trying to get at. > > > > My concern is that there is code out there in ceph_bridge that uses > > librados to talk to Ceph mons directly. That's undesirable, because I > > don't want it to treat e.g. the format of the JSON dumps from Ceph > > commands as a stable interface. > > Ceph architecture documentation points out librados as an interface > directly consumable by apps[1]. From my understanding, projects such > as openattic and calamari lite use librados. Is this incorrect? > > > It's also a wasteful distraction from > > the work to use a long term supportable interface (either ceph-mgr > > python modules, or a REST API). > > While this point is valid, ceph-mgr isn't currently available as a > stable feature in ceph. > > > The immediate engineering issue here is that we're going to have two > > versions of Tendrl (the pre-Luminous one and the post-Luminous one) > > with two different versions of Ceph (Jewel and Luminous). Calling the > > Tendrl versions T1 and T2 respectively, we have the following cases: > > > > T1->Jewel > > T2->Jewel (assuming Tendrl remains backwards compatible) > > T2->Luminous > > > > What we need from the Tendrl engineering folks is a clear statement > > about which interfaces they intend to us. For example, this would be > > reasonable if the Tendrl folks want to consume REST. > > > > T1->J Calamari REST API > > T2->J Calamari REST API > > T2->L ceph-mgr REST API > > Tendrl has evolved from skyrings[2]. Skyrings uses the calamari apis. > A major problem faced with skyring was that skyring could fall out of > sync with the ceph state should any operations be carried out on the > cluster directly via the ceph cli. Skyrings also functions primarily > as an application. > > I'm going to take a detour here and explain a bit about the thought > behind tendrl's state information architecture. > > Tendrl is not intended to support ceph exclusively. Gluster is as > important to tendrl as ceph is. When a system is designed to support > multiple technologies, the logical path to take is to abstract. > However, in the storage space, such abstractions have been based on > very specific assumptions. These lead to components from different > systems to be categories as a specific abstract entity. Example of > this is categorising ceph pools and gluster volumes as `distributed > logical storage' eg. Skyrings does this. This approach scales to an > extent, at which point you may end up looking at a schema such as the > cim schema. > > Storage management systems have been doing this traditionally by > creating management interfaces as applications. Tendrl core aims to be > a framework onto which applications such as the tendrl ui can be > plugged. The question we evaluated was whether it was necessary to do > this at all. Our realisation was that the management interface (api, > gui etc.) does not need to understand the systems it encapsulates. > Doing so means hardcoded, storage system knowledge being placed in an > abstract, higher level interface. Tendrl hence relies on a > configuration based approach where the state and the operations from > the storage system are bubbled up to tendrl's interface by describing > them in a configuration file. > > This approach isn't a ground-breaking new approach either. > Configuration management systems like puppet and chef have approached > a problem in a different domain in a similar manner. They support a > custom dsl via which the invocations and abstractions are defined. The > implementation details are left to the components specific to the systems. > > Tendrl does the same thing. Instead of the ceph specific codebase > being in a component on the skyring side, tendrl pushes the component > towards ceph, as the ceph bridge. > > So coming back to the problem at hand, that of state management, the > rest api based approach implies that tendrl knows precisely what to > ask for. The approach tendrl takes is that it gathers everything the > system knows about itself, but doesn't try to understand, modify or > fit any of that information into any specific schema. This where > tendrl's object model abstraction comes into the picture. Every > component on the system is an object, with it's own possibly unique > properties, attributes, actions, states etc. The configuration files > define this for tendrl. This way, we have a glue layer which allows to > look at all the systems in an abstract manner. When any additional > features are added, they're exposed via the configuration files by > pointing to the relevant data. Because the state information aboute > the system being managed is already available, there's no need to code > in support for any additional api calls that may or may not exist in > the system. > > So, tying this together, the point I'm getting to is this: > * rest apis for gathering state information does not work for tendrl > because tendrl needs all the information available at any point in > time. Making calls to the rest api to fetch information bit-by-bit and > reassembling on tendrl's side does not work for tendrl. > > Essentially, we need to be able to browse a book according to it's > table of contents, rather than searching it for a specific phrase that > our code is expected to know about. (Developers knowing the underlying > mechanics of the system is essential, but what we're talking about > here is the hardcoding of that domain logic in an abstract layer here, > which tendrl is looking to avoid.) > > For ceph specifically, this problem has been addressed in tendrl by > going directly to the layer where calamari gets it's information from. > A similar approach was taken for gluster in cooperation with the > gluster team. Both of these approaches worked out as intended in the > poc and hence were carried forward. > > While this addresses specifically the state information part, using > calamari apis for invoking the operations, in lieu of the librados > calls, isn't as big a problem. Operations themselves are also defined > as configuration files, but their implementation is left upto the > operations bridges. However, this does bring my earlier point into > focus. If librados is seen as brittle, aren't the calamari apis > affected by it too? Also, is there any benefit to using the calamari > apis in this case as opposed to the librados calls? > > [1] http://docs.ceph.com/docs/jewel/architecture/ > [2] https://github.com/skyring > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel From sankarshan.mukhopadhyay at gmail.com Thu Sep 29 16:43:55 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 29 Sep 2016 22:13:55 +0530 Subject: [Tendrl-devel] Architectural questions - need clear pros & cons for the integration options In-Reply-To: <571db346-0ca1-894c-2cfe-0cac5ade920b@redhat.com> References: <74b1d421-8c2a-fc11-f691-5966df713688@redhat.com> <33e50f0d-fb85-f90a-1c04-db954d708384@redhat.com> <571db346-0ca1-894c-2cfe-0cac5ade920b@redhat.com> Message-ID: On Thu, Sep 29, 2016 at 8:11 PM, Sayan Saha wrote: > This is not an easy read. Can you please send a clear email outlining the 3 > options (that I have heard about) for RHScon 3 + RHCS integration with a > pros & cons analysis for each. This will greatly help expedite decision > making. I'll do a quick summary from a meeting hosted by Jeff and which had Gregory, John, Mrugesh, Rohan, Nishanth, Ric and myself as participants. The decision arrived at is that it is of benefit to work with the Calamari developers to (a) assess the specific improvements to be built in (b) scope the time taken to undertake such work. Nishanth and Rohan would compile the detail leading to (a) and post to ceph-devel@ list (by 04Oct/Tue) in order to initiate the discussions. -- sankarshan mukhopadhyay From sankarshan.mukhopadhyay at gmail.com Fri Sep 30 11:28:13 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 30 Sep 2016 16:58:13 +0530 Subject: [Tendrl-devel] Fwd: ceph-mgr is in master In-Reply-To: References: Message-ID: FYI. ---------- Forwarded message ---------- From: John Spray Date: Fri, Sep 30, 2016 at 4:54 PM Subject: ceph-mgr is in master To: Ceph Development Hi all, A big thank you to everyone who helped with the reviews, packaging and design discussions. This will be included in forthcoming Kraken (11.2.x) release. There is a new section in the documentation: http://docs.ceph.com/docs/master/mgr/ There is a new sub-project on tracker.ceph.com, populated with some issues by me: http://tracker.ceph.com/projects/mgr/issues Pull requests are of course welcome. At some point before Kraken I'll try to remember to send out a message to ceph-users to give people a heads up about this mysterious new daemon that's appearing on their systems. Thanks, John -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo at vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- sankarshan mukhopadhyay From mcarrano at redhat.com Fri Sep 30 17:04:25 2016 From: mcarrano at redhat.com (Matt Carrano) Date: Fri, 30 Sep 2016 13:04:25 -0400 Subject: [Tendrl-devel] Create and Import Gluster Cluster UI wireframes available for review Message-ID: Here are links to two InVision prototypes that includes the wireframes/screens for UI review. You can find them via the links below. Create Gluster Cluster: https://redhat.invisionapp.com/share/8F8PQVLHD Import Gluster Cluster: https://redhat.invisionapp.com/share/R88EUSGJK We will be using the InVision link provided above for the community to review and provide comments. For information on how to make comments in Invision, please refer to: https://support.invisionapp.com/hc/en-us/articles/209192426-How-do-I-comment-on-a-prototype- . Please review and comment in InVision by COB on Friday, October 7. We will be scheduling a meeting for follow-up discussion. Let me know if you have any questions. Regards, Matt -- Matt Carrano Sr. Interaction Designer Red Hat, Inc. mcarrano at redhat.com From japplewh at redhat.com Fri Sep 30 17:07:10 2016 From: japplewh at redhat.com (Jeff Applewhite) Date: Fri, 30 Sep 2016 13:07:10 -0400 Subject: [Tendrl-devel] Create and Import Gluster Cluster UI wireframes available for review In-Reply-To: References: Message-ID: Thanks Matt! On Fri, Sep 30, 2016 at 1:04 PM, Matt Carrano wrote: > Here are links to two InVision prototypes that includes the > wireframes/screens for UI review. You can find them via the links below. > > Create Gluster Cluster: https://redhat.invisionapp.com/share/8F8PQVLHD > > Import Gluster Cluster: https://redhat.invisionapp.com/share/R88EUSGJK > > We will be using the InVision link provided above for the community to > review and provide comments. For information on how to make comments in > Invision, please refer to: > https://support.invisionapp.com/hc/en-us/articles/ > 209192426-How-do-I-comment-on-a-prototype- > . > > Please review and comment in InVision by COB on Friday, October 7. We will > be scheduling a meeting for follow-up discussion. Let me know if you have > any questions. > > Regards, > > Matt > > > -- > Matt Carrano > Sr. Interaction Designer > Red Hat, Inc. > mcarrano at redhat.com > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Jeff Applewhite Principal Product Manager From julim at redhat.com Fri Sep 30 18:45:47 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 30 Sep 2016 14:45:47 -0400 Subject: [Tendrl-devel] Upcoming Tendrl Sprint 1 Demo on Tuesday, Oct 4 Message-ID: Team: Just a heads-up and reminder that the upcoming sprint 1 demo is next Tuesday, 4 Oct 2016. The proposed Tendrl Sprint 1 Demo Agenda may be found at: Sprint 001 Demo Agenda & Notes What's listed in the link above is the tentative agenda. Items that have a strike-through will not be demonstrated until they have satisfied the Definition of Done (DoD) . If there are any demos that I've missed in the above link, please add to the wiki page. For folks who are planning to demo something, please refer to the Sprint Demo Guidelines to help ensure a smooth and successful demo. As soon as the Sprint demo is over, we'll have a retrospective. Thank you, Ju From adeza at redhat.com Fri Sep 30 19:23:06 2016 From: adeza at redhat.com (Alfredo Deza) Date: Fri, 30 Sep 2016 15:23:06 -0400 Subject: [Tendrl-devel] Installing Ceph via Tendrl Message-ID: On the "Architectural questions" thread there was mention of the following: >> How will be be installing Ceph via Tendrl? > > I'm investigating this. Two choices currently exist: ceph-installer and ceph-ansible. I'm prioritising > on ceph-ansible since adding ansible support naturally opens up the provisioning framework for > systems other than ceph to be supported through. This sounds concerning because "adding ansible support" doesn't mean that other playbooks/modules will work the same way and it is not an easy problem to solve. The work that went into the ceph-installer will have to be replicated in many ways *regardless of what playbook is targeted*: * asynchronous calls * playbook setup, uniquely crafted to fit the variable/role scheme presented by the role (if it has roles!) * system process management: errors responses, user permissions, stderr, stdout, stdin, and other logging * error handling: missing required variables (which ones are required?), invalid input (string vs. int values etc..) All of that will have to be done again in Tendrl. Can that design be explained a bit? How is this generalized approach going to tackle these issues? How is it going to approach tackling any other Ansible playbook or role? How has the ceph-installer failed here? When the project started, it did so with a "documentation first" approach, and we presented the API first so that the consumers of it could have an immediate idea on the HTTP API proposed. We did it this way because we thought it was imperative to present the ideas while the application itself was being crafted to improve the throughput on communication. We would like to understand better what is it that we failed to provide a clear interface for Ceph deployment, and how those failures (if any?) are impossible to solve on our end. Thanks, Alfredo From julim at redhat.com Fri Sep 30 20:53:29 2016 From: julim at redhat.com (Ju Lim) Date: Fri, 30 Sep 2016 16:53:29 -0400 Subject: [Tendrl-devel] UX: Design Approach and Information Architecture In-Reply-To: References: Message-ID: Hi John: Sorry for the slow response ? I was tied up in offsite meetings in the later part of this week. I appreciate all the feedback and have taken some time to think about the IA a little more. I can?t think of strong enough use cases/user stories to have snapshots and quotas at the 2nd level seeing that the user would have other places to trigger related actions/workflows ? specifically from when they are ?standing? on the Pool/RBD/File Share (storage top-level entity) itself as well as within the Cluster object details itself. It?s always in-context of the storage top-level entity or Cluster. The only exception is for reporting-type use cases across all clusters being managed, but even that is probably more applicable for quotas for things like chargebacks/showback, which is probably not a core use case for Tendrl. I?m glad you and Nishant raised this concern as I think some of the initial underlying assumptions have evolved especially if there?s a higher level management tool that leverages what Tendrl collects. I've made updates to the document UX Design Approach and Information Architecture to reflect what I stated above (i.e. removal of snapshots and quotas from the 2nd level nav). Regarding whether it?s worth rethinking using the whole first level and possibly bringing up to the top-level CephFS, RBD, RGW, etc. This was something I had thought about early on but lumped everything under Storage as putting them at the top level might get unwieldy as more things are added. If we were going to remain static at only supporting Ceph and Gluster, then it would be okay. However, to support future things that get added, ideally you don?t want to keep adding them to the top as there wouldn?t be room to grow and also it might be ?surprising? when the top-level menu keeps changing and expanding, and eventually wrapping when we run out of room. With regards to user identities, I believe it?s cluster-specific, so that would get triggered from within the cluster object details, as it wasn?t also a frequent use case and I assumed typically performed at initial setup time. This could be handled through an Edit Cluster, and/or optional step in the Create Cluster workflow, and/or an optional step in the Create RBD / Pool workflow, and/or a dedicated workflow optimized for configuration of user identities. Does this make sense? Thoughts? Thanks, Ju On Wed, Sep 28, 2016 at 7:15 PM, John Spray wrote: > On Wed, Sep 28, 2016 at 8:41 PM, Ju Lim wrote: > > Hi Nishant: > > > > Thanks for reviewing. With regards how the dashboard will look when Ceph > > and Gluster is in the picture: > > > > When there is a single storage subsystem present, the single dashboard > for > > the single storage subsystem is presented by default. > > > > When there are multiple storage subsystems present, the Dashboard would > > present each dashboard in its own tab. Basically, we'd have a Ceph tab > and > > a Gluster tab. > > > > Ideally if the default dashboard tab can be specified on a per user > basis, > > e.g. in a user?s profile/setting -- initially configured by the > > Administrator user. It's a nice-to-have at this point, and if folks > agree, > > it should be added to the backlog. > > > > With regards to unifying concepts such as quotas, file shares, etc., you > > raise a fair concern. There's multiple approaches to it. For the quotas, > > snapshots, my current thinking is that when we get to the Quotas or > > Snapshots section, we would either have a unified view with filtering > > capabilities (for the different types), or tabbed views, or a 3rd level > > navigation, or some other combination. This will be determined at a > later > > date when the feature gets added. Going with a tabbed approach or 3rd > level > > navigation could potentially allow for a plug-in based approach whereby > > specific storage subsystem capabilities can be exposed as needed. > > > > For File Shares, one approach I suggested was to handle it via the user's > > workflow. Another approach could be to handle it via the navigation which > > potentially increases complexity and may impact navigation by introducing > > more levels of navigation. Our goal is not to go deeper than 3 levels of > > navigation as it will become an unwieldy user experience if things are > > buried too deeply. That being said, whether it's handled via the > workflow, > > navigation, or some other means, using a plug-in based approach should > still > > work. > > Regarding navigation depth, I wonder if it's worth rethinking using > the whole first level just to get into "Storage" or "Clusters" -- > isn't everything we do about storage clusters? In my mind, things > like cephfs, rbd, rgw are really top-level items. > > Regarding unifying quotas and snapshots, I think it would be natural > for a user to go cephfs->quotas rather than quotas->cephfs, because at > any given moment they are probably concentrating on one particular > subsystem. I think the cephfs quotas and cephfs snapshots have more > in common (and belong closer together) than e.g. cephfs quotas and > gluster quotas, or cephfs snapshots and rbd snapshots. In a cephfs > UI, we would probably hope to see a tree view that showed directories > in the filesystem and enabled things like setting layouts and quotas > on directories -- picturing that, I don't see a sane way for cephfs > quotas to live under a top-level Quotas subsystem unless the whole > widget was duplicated inside the CephFS subsystem as well. > > Not trying to entirely derail this into a discussion of CephFS, but I > think the best way to see if this design is a good fit is to work > through some of the more challenging cases like the N types of quota > that will ultimately need managing (not just those supported in the > first release of the software), and see if what comes out makes sense. > > By the way, I was just taking another look at slide 12, and I don't > see user identities[1] (i.e. the identities that we use for access > control of Ceph clients, not the Tendrl users). Those should probably > be in there somewhere. They intersect both the "Cluster" section > (ceph keys are managed by the mons) and the RBD/RGW/CephFS parts (to > generate a key with meaningful capabilities you need to know which > subsystem it's going to be using). > > John > > 1. http://docs.ceph.com/docs/hammer/rados/operations/user-management/ > > > > I've made updates to the document UX Design Approach and Information > > Architecture to incorporate the points raised by both yours and John's > > feedback. > > > > Thank you, > > Ju > > > > > > > > On Wed, Sep 28, 2016 at 3:55 AM, Nishanth Thomas > wrote: > >> > >> Its good that we are not going to unify the dashboard pieces. With this > in > >> mind how the main dashboard looks if we have a deployment with ceph and > >> Gluster? > >> Also I have a concern with respect to unifying 'File shares' 'quotas' > etc, > >> as it might complicate the navigation. Rather we could look at a plug-in > >> based approach where file system specific things could be kept > separately > >> and installed on a need basis. > >> > >> On Wed, Sep 28, 2016 at 2:12 AM, Ju Lim wrote: > >>> > >>> Hi John: > >>> > >>> Thank you for taking the time to review and sharing your feedback. > >>> > >>> WRT terminology, I think that's a tricky one and I can understand and > >>> appreciate the confusion with RBD (vs. Block Device) and File Share. > The > >>> original decision to go with RBD was that it was very Ceph-specific in > terms > >>> of the implementation and varies from LVM and other block technologies. > >>> Having said that, I would agree that File Share deviates from this > initial > >>> premise since it could mean a Gluster volume or a CephFS fileshare. > >>> > >>> The current thinking regarding File Share is to use it for both Ceph > and > >>> Gluster, and have a way to qualify it through the notion of the > workload. > >>> Part of the rationale was to try to make it so users did not have to > think > >>> should I use Ceph or Gluster, but rather let the System make an > >>> "intelligent" choice for the mere mortal user based on just enough > >>> information that user provides. This way, the user thinks of it as > Red Hat > >>> Storage providing this capability, and if we swapped out Ceph or > Gluster for > >>> something in the future, it would allow for that. In conjunction, > there > >>> would still be a way for a user to qualify it for Ceph or Gluster as > well, > >>> so it's not just for mere mortal but also a more knowledge / ninja > user. > >>> > >>> Having said that, RGW (and Swift) are definitely in mind too, so this > is > >>> not limited to just File Shares but applies to pretty much block, > object, > >>> and file services. > >>> > >>> With regards to quotas and snapshots, I do agree that there are many > >>> different kinds. It did not seem prudent to create a Pool Snapshot, > RBD > >>> Snapshot and Clones, Volume Snapshot and Clones, File Share Quotas, > Pool > >>> Quotas, Directory Quotas, etc. at the 2nd level navigation as it is not > >>> scalable (and would get very crowded very quickly), and would create a > >>> fairly complex experience for the user. My current thinking is that > when we > >>> get to the Quotas or Snapshots section, we would either have a unified > view > >>> with filtering capabilities (for the different types), or tabbed > views, or a > >>> 3rd level navigation, or some other combination. Since quotas and > snapshots > >>> are not in the immediate release plans, I've deferred this to the > detailed > >>> design stage when we tackle those topics. > >>> > >>> I hope this addresses your concerns raised. I'll update the slidedeck > >>> with these comments before it gets published out more formally. > >>> > >>> Thanks again, > >>> Ju > >>> > >>> > >>> > >>> On Mon, Sep 26, 2016 at 5:34 AM, John Spray wrote: > >>>> > >>>> On Fri, Sep 23, 2016 at 7:03 PM, Ju Lim wrote: > >>>> > Hi. I've started documenting the UX Design Approach and Information > >>>> > Architecture (JIRA TEN-40). It includes some of our guiding > >>>> > principles and > >>>> > decisions that we've made in our design. > >>>> > > >>>> > Here's an initial draft that you can take a look at: > >>>> > > >>>> > https://docs.google.com/a/redhat.com/presentation/d/ > 1P4ejy0Q7BT0PHa2H4wC_nFAh--Db72NpX8yT44qU3Rc/edit?usp=sharing > >>>> > > >>>> > Note: For now it's open to be comments by anyone with a Red Hat > >>>> > account. I > >>>> > plan to publish this to Jira and/or GitHub before the end of Sprint > 1 > >>>> > (Oct > >>>> > 4), but I'd like to open this up to review, comments, and discussion > >>>> > in the > >>>> > meantime. > >>>> > >>>> Thanks for posting! > >>>> > >>>> My main piece of feedback is on terminology -- I think it's going to > >>>> get confusing to use a mixture of qualified names (e.g. "RBD" instead > >>>> of "block device") and unqualified names (e.g. "File Share" when we > >>>> really mean "Gluster"). > >>>> > >>>> I have a special attachment to use of "File Share" because when Tendrl > >>>> gets support for CephFS, everywhere the term is used is going to need > >>>> qualifying with whether we're talking about Ceph or Gluster. I don't > >>>> think anyone is working on adding Tendrl support for CephFS at the > >>>> moment, but it should be part of the design process, along with RGW. > >>>> > >>>> Terms like "Quotas" are also dangerous, because we have so many > >>>> different kinds. In Ceph alone, we have Quotas on pools, and then a > >>>> totally different type of quota on directories in cephfs. Same for > >>>> snapshots, we have pool snapshots, rbd snapshots, cephfs snapshots. > >>>> > >>>> John > >>>> > >>>> > Thank you, > >>>> > Ju Lim > >>>> > > >>>> > > >>>> > _______________________________________________ > >>>> > Tendrl-devel mailing list > >>>> > Tendrl-devel at redhat.com > >>>> > https://www.redhat.com/mailman/listinfo/tendrl-devel > >>>> > > >>>> > >>>> _______________________________________________ > >>>> Tendrl-devel mailing list > >>>> Tendrl-devel at redhat.com > >>>> https://www.redhat.com/mailman/listinfo/tendrl-devel > >>> > >>> > >>> > >>> > >>> -- > >>> Ju Lim > >>> Red Hat > >>> Office: 978-399-0422 > >>> Mobile: 781-507-1323 > >>> Email: julim at redhat.com > >>> IRC: julim > >>> > >>> > >>> _______________________________________________ > >>> Tendrl-devel mailing list > >>> Tendrl-devel at redhat.com > >>> https://www.redhat.com/mailman/listinfo/tendrl-devel > >>> > >> > >> > >> _______________________________________________ > >> Tendrl-devel mailing list > >> Tendrl-devel at redhat.com > >> https://www.redhat.com/mailman/listinfo/tendrl-devel > >> > > > > > > > > -- > > Ju Lim > > Red Hat > > Office: 978-399-0422 > > Mobile: 781-507-1323 > > Email: julim at redhat.com > > IRC: julim > > > > > > _______________________________________________ > > Tendrl-devel mailing list > > Tendrl-devel at redhat.com > > https://www.redhat.com/mailman/listinfo/tendrl-devel > > > > _______________________________________________ > Tendrl-devel mailing list > Tendrl-devel at redhat.com > https://www.redhat.com/mailman/listinfo/tendrl-devel > -- Ju Lim Red Hat Office: 978-399-0422 Mobile: 781-507-1323 Email: julim at redhat.com IRC: julim From sankarshan.mukhopadhyay at gmail.com Fri Sep 30 21:01:58 2016 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Sat, 1 Oct 2016 02:31:58 +0530 Subject: [Tendrl-devel] Installing Ceph via Tendrl In-Reply-To: References: Message-ID: On Sat, Oct 1, 2016 at 12:53 AM, Alfredo Deza wrote: > All of that will have to be done again in Tendrl. Can that design be > explained a bit? How is this generalized approach going to tackle > these issues? How is it going to approach tackling any other Ansible > playbook or role? > > How has the ceph-installer failed here? When the project started, it > did so with a "documentation first" approach, and we presented the API > first so that the consumers of it could have an immediate idea on the > HTTP API proposed. We did it this way because we thought it was > imperative to present the ideas while the application itself was being > crafted to improve the throughput on communication. It is a bit late/early here - the detail about the provisioning flow is (as I remember) part of this sprint or, a coming one and is being worked on. Perhaps we could provide an interim update and then take this conversation ahead. -- sankarshan mukhopadhyay