From dtantsur at redhat.com Tue Nov 1 09:46:41 2016 From: dtantsur at redhat.com (Dmitry Tantsur) Date: Tue, 1 Nov 2016 10:46:41 +0100 Subject: [rdo-list] RDO at PTG or OpenStack Summit? In-Reply-To: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> References: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> Message-ID: On 10/31/2016 06:35 PM, Rich Bowen wrote: > As you no doubt know by now ... well, I'll just quote from another list: > > > As you probably already know, the OpenStack Foundation is restructuring > its events in 2017, splitting the activities that traditionally occurred > at the Design Summit event into work sessions (at the "Project Teams > Gathering" at the start of the cycle) and open community discussions > ("Forum" at the OpenStack Summit in the middle of the cycle). You can > learn more about this at http://www.openstack.org/ptg > > > At the last several OpenStack Summits, the RDO community has hosted some > kind of meetup, and of course last week, we had an evening gathering > that we shared with the Ceph community, where we had 215 people in > attendance. > > With OpenStack Summit splitting in two, I'm left uncertain what, if > anything, I should be planning for 2017. Will most of the RDO community > attend the PTG, or OpenStack Summit? At which one of these does it make > the most sense to try to do an RDO gathering, if at all? People refuse to agree with that, but speaking realistically, you cannot expect a huge developer's attendance on summits any more. I personally only plan to be at PTGs from now on, unless my employer has some different plans for me ;) I'd like to second the idea of having a technical meetup at PTGs. > I can see good arguments for each, including my personal travel plans > ;-) but I would like to hear from the rest of you - if I try to do > another event like what we did Tuesday evening in Barcelona, which event > makes more sense? > > Thanks. > From hbrock at redhat.com Tue Nov 1 09:51:05 2016 From: hbrock at redhat.com (Hugh Brock) Date: Tue, 1 Nov 2016 10:51:05 +0100 Subject: [rdo-list] RDO at PTG or OpenStack Summit? In-Reply-To: References: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> Message-ID: On Tue, Nov 1, 2016 at 10:46 AM, Dmitry Tantsur wrote: > On 10/31/2016 06:35 PM, Rich Bowen wrote: >> >> As you no doubt know by now ... well, I'll just quote from another list: >> >> >> As you probably already know, the OpenStack Foundation is restructuring >> its events in 2017, splitting the activities that traditionally occurred >> at the Design Summit event into work sessions (at the "Project Teams >> Gathering" at the start of the cycle) and open community discussions >> ("Forum" at the OpenStack Summit in the middle of the cycle). You can >> learn more about this at http://www.openstack.org/ptg >> >> >> At the last several OpenStack Summits, the RDO community has hosted some >> kind of meetup, and of course last week, we had an evening gathering >> that we shared with the Ceph community, where we had 215 people in >> attendance. >> >> With OpenStack Summit splitting in two, I'm left uncertain what, if >> anything, I should be planning for 2017. Will most of the RDO community >> attend the PTG, or OpenStack Summit? At which one of these does it make >> the most sense to try to do an RDO gathering, if at all? > > > People refuse to agree with that, but speaking realistically, you cannot > expect a huge developer's attendance on summits any more. I personally only > plan to be at PTGs from now on, unless my employer has some different plans > for me ;) > > I'd like to second the idea of having a technical meetup at PTGs. Is there any chance there would be room during the cross-project days (Monday and Tuesday, IIRC) for an actual RDO dev session? I know it's a bit strange to do this for a distro, but there are a lot of topics -- packaging, CI, etc. -- that would benefit from exposure to the upstream teams. --Hugh >> I can see good arguments for each, including my personal travel plans >> ;-) but I would like to hear from the rest of you - if I try to do >> another event like what we did Tuesday evening in Barcelona, which event >> makes more sense? >> >> Thanks. >> > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com -- , , | Hugh Brock, hbrock at redhat.com )-_"""_-( | Director of Engineering, OpenStack Management ./ o\ /o \. | TripleO: Install, configure, and scale OpenStack. . \__/ \__/ . | http://rdoproject.org, http://tripleo.org ... V ... | ... - - - ... | "I know that you believe you understand what you . - - . | think I said, but I'm not sure you realize that what `-.....-? | you heard is not what I meant." --Robert McCloskey "TripleOwl" From rbowen at redhat.com Tue Nov 1 13:23:45 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 1 Nov 2016 09:23:45 -0400 Subject: [rdo-list] RDO at PTG or OpenStack Summit? In-Reply-To: References: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> Message-ID: On 11/01/2016 05:51 AM, Hugh Brock wrote: > Is there any chance there would be room during the cross-project days > (Monday and Tuesday, IIRC) for an actual RDO dev session? I know it's > a bit strange to do this for a distro, but there are a lot of topics > -- packaging, CI, etc. -- that would benefit from exposure to the > upstream teams. Based on the feedback in this thread, it's looking like I probably won't be at the event in Atlanta. Can someone take point in making this happen? It's not clear to me, yet, from the information available, what the process is for requesting a room. Perhaps a broader packaging/distro topic would be more likely to get a room than a specifically RDO one? -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From rbowen at redhat.com Tue Nov 1 13:45:05 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 1 Nov 2016 09:45:05 -0400 Subject: [rdo-list] An evening with RDO and Ceph, OpenStack Summit Barcelona Message-ID: For those of you who missed the evening with Ceph and RDO in Barcelona, you can see the presentations here: https://youtu.be/Vk818DhwwJw If you want to skip to a particular presentation, the schedule was as follows: 0. Welcome and Intro (Patrick/Rich) 1. Jack Zhang, Intel -- 3D Xpoint & 3D NAND with OpenStack and Ceph 2. Ha?kel Gu?mar - RDO release status (aarch64, repos, workflow) 3. George Mihaiescu, Bioinformatics -- Openstack and Ceph used in large scale cancer research projects 4. Javier Pe?a - RDO repos overview (CBS vs Trunk, and what goes where) 5. Lars Marowsky-Br?e, SUSE -- Convergence in managing Ceph: one backend with fewer frontends 6. Gulio Fidente: RDO and Ceph 7. Allen Samuels -- Bluestore Teaser 8. Jeff Chu, ARM -- Ceph on ARM 9. Jakub Ruzicka - quick look at new rpmfactory workflow with rdopkg 10. Dan van der Ster, CERN -- How to replace several petabytes of Ceph hardware without downtime 11. Alfredo Moralej - CI in RDO - what are we testing? 12. Sage Weil - Project Update There is a *lot* of background noise, but for most of the presentations, you can hear what is being said. Unfortunately, it looks like the camera got moved early on, and so the screen isn't visible. Also, many of the speakers stood off-camera. Slides/Notes from the various presentations coming shortly, where possible. -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From apevec at redhat.com Tue Nov 1 14:19:11 2016 From: apevec at redhat.com (Alan Pevec) Date: Tue, 1 Nov 2016 15:19:11 +0100 Subject: [rdo-list] RDO at PTG or OpenStack Summit? In-Reply-To: References: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> Message-ID: > what the process is for requesting a room. openstack projects are getting rooms based on PTL proposal, similar like it was the case for Design Summits until now > Perhaps a broader packaging/distro topic would be more likely to get a > room than a specifically RDO one? That would be packaging-rpm, need to check with PTL (Haikel) what's the plan for PTG, afaik it is at early planning stages. Alan From rbowen at redhat.com Tue Nov 1 14:24:05 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 1 Nov 2016 10:24:05 -0400 Subject: [rdo-list] RDO Infrastructure meeting, OpenStack Summit, Barcelona Message-ID: On Monday afternoon at OpenStack Summit in Barcelona, a small-ish meeting was held with people interested in the infrastructure needs of the RDO project. The agenda and notes from that event are here: https://review.rdoproject.org/etherpad/p/barcelona-rdo-infra-meetup A full recording of that meeting is here: https://rdocommunity.podbean.com/mf/play/mtdrju/RDO_Infra_Meeting_Barcelona.mp3 The audio quality varies a great deal from speaker to speaker, so I would also like to encourage each of those who were present to follow up with a summary of their portion of the meeeting. Thanks! -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From rbowen at redhat.com Tue Nov 1 15:11:59 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 1 Nov 2016 11:11:59 -0400 Subject: [rdo-list] RDO at PTG or OpenStack Summit? In-Reply-To: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> References: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> Message-ID: Thanks for the feedback, everyone. I'll see what I can do about planning another RDO evening event in Boston. On 10/31/2016 01:35 PM, Rich Bowen wrote: > As you no doubt know by now ... well, I'll just quote from another list: > > > As you probably already know, the OpenStack Foundation is restructuring > its events in 2017, splitting the activities that traditionally occurred > at the Design Summit event into work sessions (at the "Project Teams > Gathering" at the start of the cycle) and open community discussions > ("Forum" at the OpenStack Summit in the middle of the cycle). You can > learn more about this at http://www.openstack.org/ptg > > > At the last several OpenStack Summits, the RDO community has hosted some > kind of meetup, and of course last week, we had an evening gathering > that we shared with the Ceph community, where we had 215 people in > attendance. > > With OpenStack Summit splitting in two, I'm left uncertain what, if > anything, I should be planning for 2017. Will most of the RDO community > attend the PTG, or OpenStack Summit? At which one of these does it make > the most sense to try to do an RDO gathering, if at all? > > I can see good arguments for each, including my personal travel plans > ;-) but I would like to hear from the rest of you - if I try to do > another event like what we did Tuesday evening in Barcelona, which event > makes more sense? > > Thanks. > -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From rbowen at redhat.com Tue Nov 1 15:30:14 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 1 Nov 2016 11:30:14 -0400 Subject: [rdo-list] Tentative test day schedule Message-ID: I've posted a tentative test day schedule at https://www.rdoproject.org/testday/ based on the Ocata development schedule. I'd appreciate it if a few of you could take a moment to look it over to see if I've put any of these days on obviously-bad times. (The one in December is kind of iffy, for sure, but it *might* work.) --Rich -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From eyalb1 at gmail.com Tue Nov 1 15:41:18 2016 From: eyalb1 at gmail.com (Eyal B) Date: Tue, 1 Nov 2016 17:41:18 +0200 Subject: [rdo-list] Packging for OpenStack Vitrage Message-ID: Hi all, I'm writing to you to let you know of a relatively-new OpenStack project: Vitrage, the official OpenStack project for Root Cause Analysis (RCA). The Vitrage service aims at organizing, analyzing and expanding OpenStack alarms & events, yielding insights regarding the root cause of problems and deducing their existence before they are directly detected. By doing so, it promotes clarity and transparency of the cloud structure and state. You can see more about Vitrage in the wiki ( https://wiki.openstack.org/wiki/Vitrage) and in the following links from the recent Barcelona summit, including the keynote address where we presented part of our project in the context of OPNFV-Doctor, which was really cool :-) https://www.openstack.org/videos/video/demo-openstack-and-opnfv-keeping-your-mobile-phone-calls-connected https://www.openstack.org/videos/video/nokia-root-cause-analysis-principles-and-practice-in-openstack-and-beyond https://www.openstack.org/videos/video/fault-management-with-openstack-congress-and-vitrage-based-on-opnfv-doctor-framework Currently there are three package reviews awaiting review in RDO project: https://bugzilla.redhat.com/show_bug.cgi?id=1342987 https://bugzilla.redhat.com/show_bug.cgi?id=1379786 https://bugzilla.redhat.com/show_bug.cgi?id=1390608 We are also in the process of pushing changes to puppet-integration and tempest https://review.openstack.org/#/c/392093/ There is already a puppet module for vitrage https://github.com/openstack/puppet-vitrage We would most appreciate your contribution and assistance in this process of adding the Vitrage project to the RDO. Thanks in advance Eyal -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrybacki at redhat.com Tue Nov 1 18:08:00 2016 From: hrybacki at redhat.com (Harry Rybacki) Date: Tue, 1 Nov 2016 14:08:00 -0400 Subject: [rdo-list] RDO-Infra Weekly Scrum Recap: Message-ID: Greetings All, Links to the RDO-Infra team's scrum etherpad[1] and video recording[2] of the meeting are below. Highlights: - Code review requests/discussion - Review the new production chain escalation process - Recap of the Summit and the coming adoption of OOOQ by tripleo-ci - Retouch the 'great unification of CI tooling' email thread - Convergence of CI reporting status/API [1] - https://review.rdoproject.org/etherpad/p/rdo-infra-scrum [2] - https://bluejeans.com/s/jJJif/4/20/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/f0ade02/logs/ /R Harry Rybacki From taisto.qvist at gmail.com Tue Nov 1 21:17:51 2016 From: taisto.qvist at gmail.com (Taisto Qvist) Date: Tue, 1 Nov 2016 22:17:51 +0100 Subject: [rdo-list] Understanding Policy.Json, for domain authorization Message-ID: Hi folks, I've run into a wall with making openstack domain auth working, and I dont know where to get help, so I am trying here. I've created a question on: https://ask.openstack.org/en/question/98429/project-specific-admin-unable-to-list-users-or-use-horizon/ ..but no-one seems to be able to help. Since I wrote that, I've gotten as far as creating a working cloud-wide admin(the policy trigger for cloud_admin matching against domain_id, didnt seem to work for the default domain...?), and that user is now working fine as super-mega-admin. But my old admin user, that has admin rights only in the default domain, admin project, cant list users, or projects, in the default domain. And sureley he should be able to, with the rules: "admin_and_matching_domain_id": "rule:admin_required and domain_id:%(domain_id)s", "identity:list_users": "rule:cloud_admin or rule:admin_and_matching_domain_id", I've tried to find comprehensive and up2date references on how to read the policy.json syntax, but no success so I am unsure on how to interpret the rule exactly though. I tried changing to: "admin_and_matching_domain_id": "rule:admin_required and domain_id:%( *target*.domain_id)s", after looking at the rule for: "identity:get_project": "rule:cloud_admin or rule:admin_and_matching_target_project_domain_id or project_id:%( target.project.id)s", But it didnt help. During the failure, I can see keystone logging: 2016-11-01 22:16:24.521 4824 INFO keystone.common.wsgi [req-46e3301f-f234-434b-a013-5aa2297b6119 admin_User admin_Prj - default default] GET http://172.16.12.100:35357/v3/projects/admin_Prj (where admin_Prj/User is the UUID's regexped) What is wrong? Where can I learn how to do this??? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ayoung at redhat.com Tue Nov 1 21:54:54 2016 From: ayoung at redhat.com (Adam Young) Date: Tue, 1 Nov 2016 17:54:54 -0400 Subject: [rdo-list] Understanding Policy.Json, for domain authorization In-Reply-To: References: Message-ID: <16598480-8b50-47a4-43a1-8a9df38dde29@redhat.com> On 11/01/2016 05:17 PM, Taisto Qvist wrote: > Hi folks, > > I've run into a wall with making openstack domain auth working, and I > dont know where to get help, so I am trying here. I've created a > question on: > > https://ask.openstack.org/en/question/98429/project-specific-admin-unable-to-list-users-or-use-horizon/ > > ..but no-one seems to be able to help. > > Since I wrote that, I've gotten as far as creating a working > cloud-wide admin(the policy trigger for cloud_admin matching against > domain_id, didnt seem to work for the default domain...?), and that > user is now working fine as super-mega-admin. Can you post what your cloud_admin rule looks like? > > But my old admin user, that has admin rights only in the default > domain, admin project, cant list users, or projects, in the default > domain. admin_and_matching_domain_id: But his domain must not be matching: If he has a domain scoped token for another domain, it will not be valid for the default. > > And sureley he should be able to, with the rules: > > "admin_and_matching_domain_id": "rule:admin_required and > domain_id:%(domain_id)s", > "identity:list_users": "rule:cloud_admin or > rule:admin_and_matching_domain_id", > > I've tried to find comprehensive and up2date references on how to read > the policy.json syntax, but no success so I am unsure on how to > interpret the rule exactly though. > I tried changing to: > > "admin_and_matching_domain_id": "rule:admin_required and > domain_id:%(/target/.domain_id)s", Have you been using the CLI to test your changes? It might greatly simplify things. I'd also recommend using pdb and actually stepping through the code executed: you can learn a lot this way. > > after looking at the rule for: > > "identity:get_project": "rule:cloud_admin or > rule:admin_and_matching_target_project_domain_id or > project_id:%(target.project.id )s", Again, in this rule, you have explicit matching. The token either needs to match the domain ID or the project ID. > > But it didnt help. During the failure, I can see keystone logging: > > 2016-11-01 22:16:24.521 4824 INFO keystone.common.wsgi > [req-46e3301f-f234-434b-a013-5aa2297b6119 admin_User > admin_Prj - default default] GET > http://172.16.12.100:35357/v3/projects/admin_Prj > > (where admin_Prj/User is the UUID's regexped) > > What is wrong? Where can I learn how to do this??? > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From mrunge at redhat.com Wed Nov 2 07:13:57 2016 From: mrunge at redhat.com (Matthias Runge) Date: Wed, 2 Nov 2016 08:13:57 +0100 Subject: [rdo-list] Packging for OpenStack Vitrage In-Reply-To: References: Message-ID: <20161102071357.gia5uic4o76cempv@sofja.berg.ol> On Tue, Nov 01, 2016 at 05:41:18PM +0200, Eyal B wrote: > Hi all, > > https://bugzilla.redhat.com/show_bug.cgi?id=1342987 > https://bugzilla.redhat.com/show_bug.cgi?id=1379786 > https://bugzilla.redhat.com/show_bug.cgi?id=1390608 > > We are also in the process of pushing changes to puppet-integration and > tempest > https://review.openstack.org/#/c/392093/ > > There is already a puppet module for vitrage > https://github.com/openstack/puppet-vitrage > Hello Eyal, thank you for reaching out. As you already know, I looked at the reviews. There is still some stuff missing, something blocking each other (etc). >From my point of view, the thing currently missing in vitrage is a sensu data source. Vitrage somehow fits into the bigger game of OpsTools (in short for operational tools)[1]. That toolchain dropped nagios a while ago (or didn't even adopt it), and it doesn't use zabbix either. > We would most appreciate your contribution and assistance in this process > of adding the Vitrage project to the RDO. As stated before, I've done a few things to help it getting into RDO myself: https://github.com/mrunge/vitrage-dashboard https://github.com/mrunge/openstack-vitrage Matthias [1] https://wiki.centos.org/SpecialInterestGroup/OpsTools -- Matthias Runge From eyalb1 at gmail.com Wed Nov 2 08:30:54 2016 From: eyalb1 at gmail.com (Eyal B) Date: Wed, 2 Nov 2016 10:30:54 +0200 Subject: [rdo-list] Packging for OpenStack Vitrage In-Reply-To: <20161102071357.gia5uic4o76cempv@sofja.berg.ol> References: <20161102071357.gia5uic4o76cempv@sofja.berg.ol> Message-ID: Hi Matthias, Thanks for the prompt response, and for your help in getting into RDO. Regarding Sensu - we looked at it in the past, and adding a datasource for it is indeed in the roadmap (though I need to check if we added a BP for it yet). It's important to note that Vitrage is highly-pluggable, and writing a new datasource can be done easily. If you (or others in RedHat) would like to take this task to speed this process up, we would be happy to assist with this, both on our IRC channel or in the OpenStack ML - use the [Vitrage] tagline. Thanks again, and have a nice day, Eyal On Wed, Nov 2, 2016 at 9:13 AM, Matthias Runge wrote: > On Tue, Nov 01, 2016 at 05:41:18PM +0200, Eyal B wrote: > > Hi all, > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1342987 > > https://bugzilla.redhat.com/show_bug.cgi?id=1379786 > > https://bugzilla.redhat.com/show_bug.cgi?id=1390608 > > > > We are also in the process of pushing changes to puppet-integration and > > tempest > > https://review.openstack.org/#/c/392093/ > > > > There is already a puppet module for vitrage > > https://github.com/openstack/puppet-vitrage > > > > Hello Eyal, > > thank you for reaching out. > > As you already know, I looked at the reviews. There is still some stuff > missing, something blocking each other (etc). > > >From my point of view, the thing currently missing in vitrage is a sensu > data source. Vitrage somehow fits into the bigger game of OpsTools (in > short for operational tools)[1]. That toolchain dropped nagios a while > ago (or didn't even adopt it), and it doesn't use zabbix either. > > > We would most appreciate your contribution and assistance in this process > > of adding the Vitrage project to the RDO. > > As stated before, I've done a few things to help it getting into RDO > myself: > > https://github.com/mrunge/vitrage-dashboard > https://github.com/mrunge/openstack-vitrage > > Matthias > > [1] https://wiki.centos.org/SpecialInterestGroup/OpsTools > -- > Matthias Runge > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taisto.qvist at gmail.com Wed Nov 2 08:35:03 2016 From: taisto.qvist at gmail.com (Taisto Qvist) Date: Wed, 2 Nov 2016 09:35:03 +0100 Subject: [rdo-list] Understanding Policy.Json, for domain authorization In-Reply-To: References: Message-ID: I've found parts of the problem, but here is my policy.json excerpt: "cloud_admin": "role:admin and (token.is_admin_project:True or domain_id:d8aaf957fa5d4ced85cf308d464e0020)", "admin_and_matching_domain_id": "rule:admin_required and domain_id:%(domain_id)s", "identity:get_domain": "rule:cloud_admin or rule:admin_and_matching_domain_id or token.project.domain.id:%( target.domain.id)s", "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_domain_id", "identity:list_user_projects": "rule:owner or rule:admin_and_matching_domain_id", "identity:list_users": "rule:cloud_admin or rule:admin_and_matching_domain_id", "identity:list_groups": "rule:cloud_admin or rule:admin_and_matching_domain_id", "identity:list_users_in_group": "rule:cloud_admin or rule:admin_and_matching_target_group_domain_id", The cause of problem was the keystonerc-file I was sourcing, but I cant help wondering wether this error should be possible to avoid, and it seems I am still not all the way home...This was my inital keystonerc_admin: export OS_PROJECT_DOMAIN_NAME=Default export OS_USER_DOMAIN_NAME=Default export OS_PROJECT_NAME=admin export OS_USERNAME=admin export OS_PASSWORD=admin00 export OS_AUTH_URL=http://172.16.12.100:35357/v3 export OS_IDENTITY_API_VERSION=3 export OS_IMAGE_API_VERSION=2 I realized from ngrepping on the keystone api at :35357, that openstack was asking for a project scoped token, (which I thought I needed since my belief was that a domain-local admin should only have admin-role on a project in the domain but that doesnt seem to be the case?), but reading the policy rule made me realise that the "admin_required and domain_id:" wanted a domain-scoped token. So I ran "openstack user list --os-domain-name default", trying to get a domain scoped token, with the admin role, since my default admin user in the default domain, has the admin role on both default domain, and the admin project in the default domain. (Which seems to be a requirement, even for a local-domain admin) However, because of (I assume) the OS_PROJECT_NAME env, the token is created with a: {"auth": {"scope": {"project": {"domain": {"name": "Default"}, "name": "admin"}}, "identity": {"password": {"user": {"domain": {"name": "Default"}, "password": "admin00", "name": "admin"}}, "methods": ["password"]}}} Now, yes, defining *both* OS_PROJECT_NAME and using --os-domain-name default is of course kinda stupid, but since I can only have EITHER project scoped OR domain scoped tokens, isnt it kind of a bug that the openstack CLI is trying to give me a project scoped token, using the DOMAIN NAME??? However, it still doesnt work 100% correctly I think. Undefining OS_PROJECT NAME, and running "openstack user list --os-domain-name Default" *Will still not work*. I have to do: "openstack user list --os-domain-name Default --domain *default* " And the reason I dont like that, is that I have to know the domain *UUID*(default), which a normal project/domain-*local* admin cant find out, because of the rule for identity:get_domain which said: "identity:get_domain": "rule:cloud_admin or rule:admin_and_matching_domain_id or token.project.domain.id:%( target.domain.id)s", The only way I can do openstack domain show , is if I already now the uuid to begin with... Running openstack user list --os-domain-name Default --domain Default openstack domain show --os-domain-name Default Default result in: "Could not find resource Default" So. Maybe I can fix this by allowing domain-names in the rule for get_domain, but since I cant fathom that I would be the first to try to list users with a domain local admin-user, I cant believe that I need to change the recently downloaded policy file. Surely the major users of Openstack have started using domains by now? Best Regards Taisto Qvist 2016-11-01 22:17 GMT+01:00 Taisto Qvist : > Hi folks, > > I've run into a wall with making openstack domain auth working, and I dont > know where to get help, so I am trying here. I've created a question on: > > https://ask.openstack.org/en/question/98429/project- > specific-admin-unable-to-list-users-or-use-horizon/ > > ..but no-one seems to be able to help. > > Since I wrote that, I've gotten as far as creating a working cloud-wide > admin(the policy trigger for cloud_admin matching against domain_id, didnt > seem to work for the default domain...?), and that user is now working fine > as super-mega-admin. > > But my old admin user, that has admin rights only in the default domain, > admin project, cant list users, or projects, in the default domain. > > And sureley he should be able to, with the rules: > > "admin_and_matching_domain_id": "rule:admin_required and > domain_id:%(domain_id)s", > "identity:list_users": "rule:cloud_admin or rule:admin_and_matching_domain_id", > > > I've tried to find comprehensive and up2date references on how to read the > policy.json syntax, but no success so I am unsure on how to interpret the > rule exactly though. > I tried changing to: > > "admin_and_matching_domain_id": "rule:admin_required and domain_id:%( > *target*.domain_id)s", > > after looking at the rule for: > > "identity:get_project": "rule:cloud_admin or rule:admin_and_matching_target_project_domain_id > or project_id:%(target.project.id)s", > > But it didnt help. During the failure, I can see keystone logging: > > 2016-11-01 22:16:24.521 4824 INFO keystone.common.wsgi > [req-46e3301f-f234-434b-a013-5aa2297b6119 admin_User > admin_Prj - default default] GET > http://172.16.12.100:35357/v3/projects/admin_Prj > > (where admin_Prj/User is the UUID's regexped) > > What is wrong? Where can I learn how to do this??? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taisto.qvist at gmail.com Wed Nov 2 09:17:35 2016 From: taisto.qvist at gmail.com (Taisto Qvist) Date: Wed, 2 Nov 2016 10:17:35 +0100 Subject: [rdo-list] Understanding Policy.Json, for domain authorization In-Reply-To: References: Message-ID: Additionally I realized that even if I did get it to work in CLI, my dashboard/horizon still is not able to list the users or projects for the admin user in the default domain, with the adminRole on both adminproject and default domain. TQ 2016-11-02 9:35 GMT+01:00 Taisto Qvist : > I've found parts of the problem, but here is my policy.json excerpt: > > "cloud_admin": "role:admin and (token.is_admin_project:True or > domain_id:d8aaf957fa5d4ced85cf308d464e0020)", > "admin_and_matching_domain_id": "rule:admin_required and > domain_id:%(domain_id)s", > "identity:get_domain": "rule:cloud_admin or rule:admin_and_matching_domain_id > or token.project.domain.id:%(target.domain.id)s", > "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_ > domain_id", > "identity:list_user_projects": "rule:owner or rule:admin_and_matching_ > domain_id", > "identity:list_users": "rule:cloud_admin or rule:admin_and_matching_ > domain_id", > "identity:list_groups": "rule:cloud_admin or rule:admin_and_matching_ > domain_id", > "identity:list_users_in_group": "rule:cloud_admin or > rule:admin_and_matching_target_group_domain_id", > > The cause of problem was the keystonerc-file I was sourcing, but I cant > help wondering wether this error should be possible to avoid, and it seems > I am still not all the way home...This was my inital keystonerc_admin: > > export OS_PROJECT_DOMAIN_NAME=Default > export OS_USER_DOMAIN_NAME=Default > export OS_PROJECT_NAME=admin > export OS_USERNAME=admin > export OS_PASSWORD=admin00 > export OS_AUTH_URL=http://172.16.12.100:35357/v3 > export OS_IDENTITY_API_VERSION=3 > export OS_IMAGE_API_VERSION=2 > > I realized from ngrepping on the keystone api at :35357, that openstack > was asking for a project scoped token, (which I thought I needed since my > belief was that a domain-local admin should only have admin-role on a > project in the domain but that doesnt seem to be the case?), but reading > the policy rule made me realise that the "admin_required and domain_id:" > wanted a domain-scoped token. > > So I ran "openstack user list --os-domain-name default", trying to get a > domain scoped token, with the admin role, since my default admin user in > the default domain, has the admin role on both default domain, and the > admin project in the default domain. (Which seems to be a requirement, even > for a local-domain admin) > > However, because of (I assume) the OS_PROJECT_NAME env, the token is > created with a: > > {"auth": {"scope": {"project": {"domain": {"name": "Default"}, "name": > "admin"}}, "identity": {"password": {"user": {"domain": {"name": > "Default"}, "password": "admin00", "name": "admin"}}, "methods": > ["password"]}}} > > Now, yes, defining *both* OS_PROJECT_NAME and using --os-domain-name > default is of course kinda stupid, but since I can only have EITHER > project scoped OR domain scoped tokens, isnt it kind of a bug that the > openstack CLI is trying to give me a project scoped token, using the DOMAIN > NAME??? > > However, it still doesnt work 100% correctly I think. Undefining > OS_PROJECT NAME, and running > > "openstack user list --os-domain-name Default" > > *Will still not work*. I have to do: > > "openstack user list --os-domain-name Default --domain *default* " > > And the reason I dont like that, is that I have to know the domain *UUID*(default), > which a normal project/domain-*local* admin cant find out, because of the > rule for identity:get_domain which said: > > "identity:get_domain": "rule:cloud_admin or rule:admin_and_matching_domain_id > or token.project.domain.id:%(target.domain.id)s", > > The only way I can do openstack domain show , is if I already now > the uuid to begin with... > > Running > > openstack user list --os-domain-name Default --domain Default > openstack domain show --os-domain-name Default Default > > result in: "Could not find resource Default" > > So. Maybe I can fix this by allowing domain-names in the rule for > get_domain, but since I cant fathom that I would be the first to try to > list users with a domain local admin-user, I cant believe that I need to > change the recently downloaded policy file. > > Surely the major users of Openstack have started using domains by now? > > Best Regards > Taisto Qvist > > > 2016-11-01 22:17 GMT+01:00 Taisto Qvist : > >> Hi folks, >> >> I've run into a wall with making openstack domain auth working, and I >> dont know where to get help, so I am trying here. I've created a question >> on: >> >> https://ask.openstack.org/en/question/98429/project-specific >> -admin-unable-to-list-users-or-use-horizon/ >> >> ..but no-one seems to be able to help. >> >> Since I wrote that, I've gotten as far as creating a working cloud-wide >> admin(the policy trigger for cloud_admin matching against domain_id, didnt >> seem to work for the default domain...?), and that user is now working fine >> as super-mega-admin. >> >> But my old admin user, that has admin rights only in the default domain, >> admin project, cant list users, or projects, in the default domain. >> >> And sureley he should be able to, with the rules: >> >> "admin_and_matching_domain_id": "rule:admin_required and >> domain_id:%(domain_id)s", >> "identity:list_users": "rule:cloud_admin or >> rule:admin_and_matching_domain_id", >> >> I've tried to find comprehensive and up2date references on how to read >> the policy.json syntax, but no success so I am unsure on how to interpret >> the rule exactly though. >> I tried changing to: >> >> "admin_and_matching_domain_id": "rule:admin_required and domain_id:%( >> *target*.domain_id)s", >> >> after looking at the rule for: >> >> "identity:get_project": "rule:cloud_admin or >> rule:admin_and_matching_target_project_domain_id or project_id:%( >> target.project.id)s", >> >> But it didnt help. During the failure, I can see keystone logging: >> >> 2016-11-01 22:16:24.521 4824 INFO keystone.common.wsgi >> [req-46e3301f-f234-434b-a013-5aa2297b6119 admin_User >> admin_Prj - default default] GET >> http://172.16.12.100:35357/v3/projects/admin_Prj >> >> (where admin_Prj/User is the UUID's regexped) >> >> What is wrong? Where can I learn how to do this??? >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cems at ebi.ac.uk Wed Nov 2 09:44:05 2016 From: cems at ebi.ac.uk (Charles Short) Date: Wed, 2 Nov 2016 09:44:05 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues Message-ID: Hi, I am running TripleO Newton stable release and am deploying on baremetal with CentOS. I have 64 nodes, and the Undercloud has plenty of resource as it is one of the nodes with 294 GB Memory and 64 CPUs. The provisioning network is 1Gbps I have tried tuning the Undercloud using this tuning section in 10.7 as a guide https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/paged/director-installation-and-usage/chapter-10-troubleshooting-director-issues My Undercloud passes validations in Clapper https://github.com/rthallisey/clapper I am deploying with Network Isolation and 3 Controllers in HA. If I create a stack with 3 Controllers and 3 compute nodes this takes about 1 hour If I create a stack with 3 Controllers and 15 compute nodes this takes about 1 hour Both stacks pass Clapper validations. During deployment I can see that the first 20 to 30 mins is using all the bandwidth available for the overcloud image deployment and them uses hardly any bandwidth whilst the rest of the configuration takes place. So I try a stack with 40 nodes. This is where I have issues. I set the timeout to 4 hours and leave it over night to deploy. It seems to timeout and fail to deploy due to the timeout every time. During the 40 node deployment the overcloud image is distributed in about 45 mins to all nodes and the all nodes appear ACTIVE and have an IP address on the deployment network. So it would appear that the rest of the low bandwidth configuration is taking well over 3 hours to complete. This seems excessive I have configured nova.conf for deployment concurrency (from the tuning link above) and configured the heat.conf 'num_engine_workers' to be 32 taking in to account this bug https://bugzilla.redhat.com/show_bug.cgi?id=1370516 So my question is how do I tune my Undercloud to speed up the deployment? Looking at htop during deployment I can see heat is using many CPUs, but the work pattern is NOT distributed. What typically happens is all the CPUs are at 0 to 1 % used apart from one which is at 50 to 100%. This one CPU id changes regularly, but there is no concurrent distributed workload across all the CPUs that the heat processes are running on. Is heat really multi-threaded, or does if have limitations so it can only really do proper work on one CPU at a time (which I am seeing in htop)? Thanks Charles -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From javier.pena at redhat.com Wed Nov 2 11:31:11 2016 From: javier.pena at redhat.com (Javier Pena) Date: Wed, 2 Nov 2016 07:31:11 -0400 (EDT) Subject: [rdo-list] Tentative test day schedule In-Reply-To: References: Message-ID: <1973810789.12062282.1478086271359.JavaMail.zimbra@redhat.com> ----- Original Message ----- > I've posted a tentative test day schedule at > https://www.rdoproject.org/testday/ based on the Ocata development > schedule. I'd appreciate it if a few of you could take a moment to look > it over to see if I've put any of these days on obviously-bad times. > (The one in December is kind of iffy, for sure, but it *might* work.) > The Ocata 1 test days happen during Thanksgiving in the US, so maybe we could move it to the following week? Javier > --Rich > > > -- > Rich Bowen - rbowen at redhat.com > RDO Community Liaison > http://rdoproject.org > @RDOCommunity > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From whayutin at redhat.com Wed Nov 2 14:15:09 2016 From: whayutin at redhat.com (Wesley Hayutin) Date: Wed, 2 Nov 2016 10:15:09 -0400 Subject: [rdo-list] Tentative test day schedule In-Reply-To: <1973810789.12062282.1478086271359.JavaMail.zimbra@redhat.com> References: <1973810789.12062282.1478086271359.JavaMail.zimbra@redhat.com> Message-ID: On Wed, Nov 2, 2016 at 7:31 AM, Javier Pena wrote: > > > ----- Original Message ----- > > I've posted a tentative test day schedule at > > https://www.rdoproject.org/testday/ based on the Ocata development > > schedule. I'd appreciate it if a few of you could take a moment to look > > it over to see if I've put any of these days on obviously-bad times. > > (The one in December is kind of iffy, for sure, but it *might* work.) > > > > The Ocata 1 test days happen during Thanksgiving in the US, so maybe we > could move it to the following week? > > Javier > - Ocata 1 ? November 24, 25, 2016 - Ocata 2 ? December 22, 23, 2016 Both these days conflict w/ holidays, to get the participation we're looking for imho it would be best to schedule around the holidays. > > > --Rich > > > > > > -- > > Rich Bowen - rbowen at redhat.com > > RDO Community Liaison > > http://rdoproject.org > > @RDOCommunity > > > > _______________________________________________ > > rdo-list mailing list > > rdo-list at redhat.com > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com > > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hguemar at fedoraproject.org Wed Nov 2 16:30:30 2016 From: hguemar at fedoraproject.org (=?UTF-8?Q?Ha=C3=AFkel?=) Date: Wed, 2 Nov 2016 17:30:30 +0100 Subject: [rdo-list] RDO at PTG or OpenStack Summit? In-Reply-To: References: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> Message-ID: 2016-11-01 15:19 GMT+01:00 Alan Pevec : >> what the process is for requesting a room. > > openstack projects are getting rooms based on PTL proposal, similar > like it was the case for Design Summits until now > >> Perhaps a broader packaging/distro topic would be more likely to get a >> room than a specifically RDO one? > > That would be packaging-rpm, need to check with PTL (Haikel) what's > the plan for PTG, afaik it is at early planning stages. > I asked for some space in PTG, potentially to be shared with Debian packaging. I personally think that discussions with other downstreams is beneficial to share our experiences, so I'll investigate the others distros about it. H. > Alan > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From lorenzetto.luca at gmail.com Wed Nov 2 16:58:25 2016 From: lorenzetto.luca at gmail.com (Luca 'remix_tj' Lorenzetto) Date: Wed, 2 Nov 2016 17:58:25 +0100 Subject: [rdo-list] RDO at PTG or OpenStack Summit? In-Reply-To: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> References: <82e2729a-4c1f-c67a-0748-35dc88cbdb73@redhat.com> Message-ID: On Mon, Oct 31, 2016 at 6:35 PM, Rich Bowen wrote: [cut] > I can see good arguments for each, including my personal travel plans > ;-) but I would like to hear from the rest of you - if I try to do > another event like what we did Tuesday evening in Barcelona, which event > makes more sense? > For what it's worth, i'll give my opinion about it: Everything depends which is the target of the event, and what you consider as community. I'm not a developer and not being involved in RDO activities. I'm an user of RDO packages and I'll try to help, when possible, with testing. This will result in no interest to join PTG and additionally my company will never allow me to join this kind of events since developing for openstack is not what they expect me to do. On the other hand joining summits instead will be more easy (in Europe for sure) since is classified as training (and funded appropriately). I think that getting in touch with developers will deliver a greater value to me because will allow to understand better some internals and will help me to move around in the projects. This will also help me to forbear a deep integration with vendor for using openstack. In my opinion using an opensource product through a vendor and without joining the community is like using closed-source software. You cannot take advantage from the openness. TL;DR: I'd like to have a meeting at summits like the one held in Barcelona (the only summit i attended) because I'd like to get in touch with the developers and with other RDO users. -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet ? la pi? grande biblioteca del mondo. Ma il problema ? che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , From cems at ebi.ac.uk Wed Nov 2 19:30:17 2016 From: cems at ebi.ac.uk (Charles Short) Date: Wed, 2 Nov 2016 19:30:17 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: Message-ID: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> Some more testing of different amounts of nodes vs time taken for successful deployments - 3 controller 3 compute = 1 hour 3 controller 15 compute = 1 hour 3 controller 25 compute = 1 hour 45 mins 3 controller 35 compute = 4 hours Charles On 02/11/2016 09:44, Charles Short wrote: > Hi, > > I am running TripleO Newton stable release and am deploying on > baremetal with CentOS. > I have 64 nodes, and the Undercloud has plenty of resource as it is > one of the nodes with 294 GB Memory and 64 CPUs. > The provisioning network is 1Gbps > > I have tried tuning the Undercloud using this tuning section in 10.7 > as a guide > > https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/paged/director-installation-and-usage/chapter-10-troubleshooting-director-issues > > > My Undercloud passes validations in Clapper > > https://github.com/rthallisey/clapper > > I am deploying with Network Isolation and 3 Controllers in HA. > > If I create a stack with 3 Controllers and 3 compute nodes this takes > about 1 hour > If I create a stack with 3 Controllers and 15 compute nodes this takes > about 1 hour > Both stacks pass Clapper validations. > > During deployment I can see that the first 20 to 30 mins is using all > the bandwidth available for the overcloud image deployment and them > uses hardly any bandwidth whilst the rest of the configuration takes > place. > > So I try a stack with 40 nodes. This is where I have issues. > I set the timeout to 4 hours and leave it over night to deploy. > It seems to timeout and fail to deploy due to the timeout every time. > > During the 40 node deployment the overcloud image is distributed in > about 45 mins to all nodes and the all nodes appear ACTIVE and have an > IP address on the deployment network. > So it would appear that the rest of the low bandwidth configuration is > taking well over 3 hours to complete. This seems excessive > I have configured nova.conf for deployment concurrency (from the > tuning link above) and configured the heat.conf 'num_engine_workers' > to be 32 taking in to account this bug > > https://bugzilla.redhat.com/show_bug.cgi?id=1370516 > > So my question is how do I tune my Undercloud to speed up the deployment? > > Looking at htop during deployment I can see heat is using many CPUs, > but the work pattern is NOT distributed. What typically happens is all > the CPUs are at 0 to 1 % used apart from one which is at 50 to 100%. > This one CPU id changes regularly, but there is no concurrent > distributed workload across all the CPUs that the heat processes are > running on. Is heat really multi-threaded, or does if have limitations > so it can only really do proper work on one CPU at a time (which I am > seeing in htop)? > > Thanks > > Charles > > > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From dms at redhat.com Wed Nov 2 20:11:16 2016 From: dms at redhat.com (David Moreau Simard) Date: Wed, 2 Nov 2016 16:11:16 -0400 Subject: [rdo-list] FYI - Updates/refactors to WeIRDO: Ansible 2.2 Message-ID: Hi rdo-list, Ansible 2.2 is out [1] and contains a sizeable changelog [2]. The two highlights of that release, at least as far as I'm concerned are: - Significant performance improvements across the board [3] - The new "include_role" task [4] I haven't really had an incentive to update WeIRDO past Ansible 2.0.x so far but these two are worth it. Where I want to get that is that I'll have to do some gymnastics in the review gate since there are inter-dependencies between the core of WeIRDO and it's roles. I'll try and make sure everything works and is properly tested before approving (or force-merging) each change -- and be particularly on the look out for issues in Jenkins jobs. But, please, point out if you see anything weird (!) around WeIRDO jobs this week and in the near future. Thanks ! [1]: https://www.ansible.com/press/ansible-22-delivers-new-automation-capabilities-for-containers-networks-and-cloud [2]: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md#22-the-battle-of-evermore---active-development [3]: https://github.com/ansible/ansible/issues/12239 [4]: https://github.com/ansible/ansible-modules-core/blob/ecc40297537e7cf3912710011a09529fa8b53560/utilities/logic/include_role.py David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] From rbowen at redhat.com Wed Nov 2 20:26:29 2016 From: rbowen at redhat.com (Rich Bowen) Date: Wed, 2 Nov 2016 16:26:29 -0400 Subject: [rdo-list] Tentative test day schedule In-Reply-To: References: Message-ID: <786d1ac4-70e5-9131-1a7e-6fa5ae2e5891@redhat.com> Based on various comments here, and in the IRC meeting this morning, I've updated the schedule at https://www.rdoproject.org/testday/ Ocata 1 ? December 1st and 2nd, 2016 Ocata 3 ? February 2nd and 3rd, 2017 Ocata Final ? March 2nd and 3rd, 2017 Ocata is a short schedule, and there's always a lot of people out during this time of year, so we have dropped the Ocata 2 test day, and moved some of other dates around. Thanks, everyone. On 11/01/2016 11:30 AM, Rich Bowen wrote: > I've posted a tentative test day schedule at > https://www.rdoproject.org/testday/ based on the Ocata development > schedule. I'd appreciate it if a few of you could take a moment to look > it over to see if I've put any of these days on obviously-bad times. > (The one in December is kind of iffy, for sure, but it *might* work.) > > --Rich > > -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From rbowen at redhat.com Wed Nov 2 20:42:54 2016 From: rbowen at redhat.com (Rich Bowen) Date: Wed, 2 Nov 2016 16:42:54 -0400 Subject: [rdo-list] OpenStack Summit blog posts Message-ID: I'm hoping to include some OpenStack Summit reports in the RDO newsletter, which I hope to send out either late this week or early next week. However, so far I've only found my own blog post, and Julien Danjou's. If you attended OpenStack Summit, and, in particular, if you attended any of the RDO events, or talks, at the Summit, please consider writing a brief blog post about the week - the highlights, important decisions, fun events, and so on. (And then let me know where it is!) Thanks! -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From ak at cloudssky.com Wed Nov 2 21:33:38 2016 From: ak at cloudssky.com (Arash Kaffamanesh) Date: Wed, 2 Nov 2016 22:33:38 +0100 Subject: [rdo-list] OpenStack DACH Day in Berlin this week on Saturday needs your support Message-ID: Hi Rich, hello together, We're hosting our OS after summit day as our 2nd OpenStack DACH Day 2016 this week on Saturday Nov. 5th in Berlin. Red Hat, HPE, DELL-EMC are our sponsors and we have currently 150 registrations as per today. We'd highly appreciate your support by tweeting or do some PR for our event to reach the 200 participants mark or more in the next two days (250 is our limit): https://www.openstack-dach.org/ The ticket is really affordable only for 25 Euros ;-) https://www.openstack-dach.org/dach-day-2016/tickets/ Many Thanks! -Arash P.S: We're going to provide our RDO session with Packstack for young staff members with the title: OpenStack fundamentals and easy deployment for young staff members (the text is provided currently only in German): https://www.openstack-dach.org/dach-day-2016/programm/openstack-fundamentals-und-die-einfache-installation-fuer-nachwuchskraefte/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From coolsvap at gmail.com Thu Nov 3 04:21:20 2016 From: coolsvap at gmail.com (Swapnil Kulkarni) Date: Thu, 3 Nov 2016 08:21:20 +0400 Subject: [rdo-list] FYI - Updates/refactors to WeIRDO: Ansible 2.2 In-Reply-To: References: Message-ID: On Thu, Nov 3, 2016 at 12:11 AM, David Moreau Simard wrote: > Hi rdo-list, > > Ansible 2.2 is out [1] and contains a sizeable changelog [2]. > > The two highlights of that release, at least as far as I'm concerned are: > - Significant performance improvements across the board [3] > - The new "include_role" task [4] > > I haven't really had an incentive to update WeIRDO past Ansible 2.0.x > so far but these two are worth it. > > Where I want to get that is that I'll have to do some gymnastics in > the review gate since there are inter-dependencies between the core of > WeIRDO and it's roles. > I'll try and make sure everything works and is properly tested before > approving (or force-merging) each change -- and be particularly on the > look out for issues in Jenkins jobs. > > But, please, point out if you see anything weird (!) around WeIRDO > jobs this week and in the near future. > > Thanks ! > > [1]: https://www.ansible.com/press/ansible-22-delivers-new-automation-capabilities-for-containers-networks-and-cloud > [2]: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md#22-the-battle-of-evermore---active-development > [3]: https://github.com/ansible/ansible/issues/12239 > [4]: https://github.com/ansible/ansible-modules-core/blob/ecc40297537e7cf3912710011a09529fa8b53560/utilities/logic/include_role.py > > David Moreau Simard > Senior Software Engineer | Openstack RDO > > dmsimard = [irc, github, twitter] > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com Sure. Thanks for the heads up. This is a good idea. Will keep an eye for the changes. ~coolsvap From javier.pena at redhat.com Thu Nov 3 10:21:56 2016 From: javier.pena at redhat.com (Javier Pena) Date: Thu, 3 Nov 2016 06:21:56 -0400 (EDT) Subject: [rdo-list] [infra][DLRN] Maintenance of master DLRN worker on November 10 In-Reply-To: <1887846418.12458327.1478168153455.JavaMail.zimbra@redhat.com> Message-ID: <468477697.12458963.1478168516004.JavaMail.zimbra@redhat.com> Hi RDO, We are planning to run a maintenance on the centos-master DLRN worker starting November 10, 9:00 AM UTC time. This maintenance is required to improve future version transitions between master and stable branches, and the expected downtime for https://trunk.rdoproject.org/centos7-master and https://trunk.rdoproject.org/centos7-ocata is about 1 hour. All other repos will remain active during the maintenance. You can find further details in https://trello.com/c/ws1EPmwp/390-improve-master-and-stable-branch-handling-in-dlrn-repos . Please note that we will also update several promotion jobs to match the new internal user name. If you have any questions, please do not hesitate to contact me. Regards, Javier From rbowen at redhat.com Thu Nov 3 12:51:03 2016 From: rbowen at redhat.com (Rich Bowen) Date: Thu, 3 Nov 2016 08:51:03 -0400 Subject: [rdo-list] OpenStack DACH Day in Berlin this week on Saturday needs your support In-Reply-To: References: Message-ID: <39f6f77c-d53a-5821-048e-7e36a51478d2@redhat.com> Will do. Thanks for the reminder. On 11/02/2016 05:33 PM, Arash Kaffamanesh wrote: > Hi Rich, > hello together, > > We're hosting our OS after summit day as our 2nd OpenStack DACH Day 2016 > this week on Saturday Nov. 5th in Berlin. > Red Hat, HPE, DELL-EMC are our sponsors and we have currently 150 > registrations as per today. > We'd highly appreciate your support by tweeting or do some PR for our > event to reach the 200 participants mark or more in the next two days > (250 is our limit): > > https://www.openstack-dach.org/ > > The ticket is really affordable only for 25 Euros ;-) > > https://www.openstack-dach.org/dach-day-2016/tickets/ > > Many Thanks! > -Arash > > P.S: We're going to provide our RDO session with Packstack for young > staff members with the title: > OpenStack fundamentals and easy deployment for young staff members (the > text is provided currently only in German): > https://www.openstack-dach.org/dach-day-2016/programm/openstack-fundamentals-und-die-einfache-installation-fuer-nachwuchskraefte/ > > -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From lorenzetto.luca at gmail.com Thu Nov 3 13:16:14 2016 From: lorenzetto.luca at gmail.com (Luca 'remix_tj' Lorenzetto) Date: Thu, 3 Nov 2016 14:16:14 +0100 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> Message-ID: On Wed, Nov 2, 2016 at 8:30 PM, Charles Short wrote: > Some more testing of different amounts of nodes vs time taken for successful > deployments - > > 3 controller 3 compute = 1 hour > 3 controller 15 compute = 1 hour > 3 controller 25 compute = 1 hour 45 mins > 3 controller 35 compute = 4 hours Hello, i'm now preparing my deployment of 3+2 nodes. I'll check what you reported and give you some feedback. Luca -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet ? la pi? grande biblioteca del mondo. Ma il problema ? che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , From jkilpatr at redhat.com Thu Nov 3 13:30:13 2016 From: jkilpatr at redhat.com (Justin Kilpatrick) Date: Thu, 3 Nov 2016 09:30:13 -0400 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> Message-ID: Hey Charles, If you want to deploy a large number of machines, I suggest you deploy a small configuration (maybe 3 controllers 1 compute) and then run the overcloud deploy command again with 2 computes, so on and so forth until you reach your full allocation Realistically you can probably do a stride of 5 computes each time, experiment with it a bit, as you get up to the full allocation of nodes you might run into a race condition bug with assigning computes to nodes and need to pin nodes (pinning is adding as an ironic property that overcloud-novacompute-0 goes here, 1 here, so on and so forth). As for actually solving the deployment issues at scale (instead of this horrible hack) I'm looking into adding some robustness at the ironic or tripleo level to these operations. It sounds like you're running more into node assignment issues rather than pxe issues though. 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto < lorenzetto.luca at gmail.com>: > On Wed, Nov 2, 2016 at 8:30 PM, Charles Short wrote: > > Some more testing of different amounts of nodes vs time taken for > successful > > deployments - > > > > 3 controller 3 compute = 1 hour > > 3 controller 15 compute = 1 hour > > 3 controller 25 compute = 1 hour 45 mins > > 3 controller 35 compute = 4 hours > > Hello, > > i'm now preparing my deployment of 3+2 nodes. I'll check what you > reported and give you some feedback. > > Luca > > > -- > "E' assurdo impiegare gli uomini di intelligenza eccellente per fare > calcoli che potrebbero essere affidati a chiunque se si usassero delle > macchine" > Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) > > "Internet ? la pi? grande biblioteca del mondo. > Ma il problema ? che i libri sono tutti sparsi sul pavimento" > John Allen Paulos, Matematico (1945-vivente) > > Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , < > lorenzetto.luca at gmail.com> > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hguemar at fedoraproject.org Thu Nov 3 14:04:34 2016 From: hguemar at fedoraproject.org (=?UTF-8?Q?Ha=C3=AFkel?=) Date: Thu, 3 Nov 2016 15:04:34 +0100 Subject: [rdo-list] [Meeting] RDO meeting (2016-11-02) Minutes Message-ID: My bad for sending this late! ============================== #rdo: RDO meeting - 2016-10-02 ============================== Meeting started by number80 at 15:00:27 UTC. The full logs are available at https://meetbot.fedoraproject.org/rdo/2016-11-02/rdo_meeting_-_2016-10-02.2016-11-02-15.00.log.html . Meeting summary --------------- * roll call (number80, 15:00:34) * vitrage packaging (number80, 15:03:27) * LINK: https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-package-to-rdo-trunk (apevec, 15:08:39) * Rename centos-ocata user in DLRN instance to simplify future stable branch creations (number80, 15:12:28) * LINK: https://trello.com/c/ws1EPmwp/390-improve-master-and-stable-branch-handling-in-dlrn-repos (jpena, 15:13:43) * jpena to prepare DLRN update on Nov, 10 + communication (number80, 15:18:23) * ACTION: jpena to coordinate and communicate change (jpena, 15:18:24) * Ocata RDO testdays proposal (number80, 15:19:03) * LINK: http://rdoproject.org/testday (rbowen, 15:21:43) * test days: dec 1/2, feb 2/3, March 2/3 (skipping M2 test days) (number80, 15:29:58) * Alternate arch status (number80, 15:31:51) * ACTION: number80 look at ceph/ppc64le with fcami and kdreyer (number80, 15:42:08) * next week chair (number80, 15:42:24) * ACTION: imcsk8 is chairing next week (number80, 15:44:04) * open floor (number80, 15:44:25) Meeting ended at 15:49:20 UTC. Action Items ------------ * jpena to coordinate and communicate change * number80 look at ceph/ppc64le with fcami and kdreyer * imcsk8 is chairing next week Action Items, by person ----------------------- * imcsk8 * imcsk8 is chairing next week * jpena * jpena to coordinate and communicate change * number80 * number80 look at ceph/ppc64le with fcami and kdreyer * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * number80 (67) * apevec (32) * rbowen (24) * leifmadsen (15) * jpena (14) * zodbot (7) * mengxd (6) * jruzicka (6) * imcsk8 (4) * weshay (3) * openstack (3) * rdogerrit (2) * openstackgerrit (1) * dmsimard (1) * myoung (1) * chandankumar (1) * bkero (1) * EmilienM (1) * jschlueter (1) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot From cems at ebi.ac.uk Thu Nov 3 14:09:07 2016 From: cems at ebi.ac.uk (Charles Short) Date: Thu, 3 Nov 2016 14:09:07 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> Message-ID: On 03/11/2016 13:30, Justin Kilpatrick wrote: > It sounds like you're running more into node assignment issues rather > than pxe issues though Hi, Thanks for the reply. This is an interesting thought. I already assign the controllers to specific nodes, but let the compute nodes get assigned at random. I will try pinning the compute nodes and see if that helps. A 50 node deployment is about to complete. It looks like it will take in the region of 6-7 hours to complete! I will then assign the remaining compute nodes explicitly and run a stack update with the extra nodes. I had previously tried the approach of lots of updates of smaller numbers of nodes, but I found that for example adding 10 nodes to a deployment of say 30 nodes would also take an age (hours and hours). I will report back after more testing Charles -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From rbowen at redhat.com Thu Nov 3 15:35:38 2016 From: rbowen at redhat.com (Rich Bowen) Date: Thu, 3 Nov 2016 11:35:38 -0400 Subject: [rdo-list] [Rdo-newsletter] November 2016 RDO Community Newsletter Message-ID: <6a07120e-6fb4-3f2a-5bac-cb636a71f34f@redhat.com> If you're having difficulty with the formatting of this message, you can read it on the website at https://www.rdoproject.org/newsletter/2016-november/ Quick links: * Quick Start * Mailing Lists * RDO release packages * Review.RDOProject.org * RDO blog * Q&A * Open Tickets * Twitter * Newton release schedule Thanks for being part of the RDO community! RDO Newton Released The RDO community is pleased to announce the general availability of the RDO build for OpenStack Newton for RPM-based distributions, CentOS Linux 7 and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds. Newton is the 14th release from the OpenStack project , which is the work of more than 2700 contributors from around the world (source ). The RDO community project curates, packages, builds, tests, and maintains a complete OpenStack component set for RHEL and CentOS Linux and is a member of the CentOS Cloud Infrastructure SIG . The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS Linux users looking to build and maintain their own on-premise, public or hybrid clouds. At latest count, RDO contains 1157 packages . All work on RDO, and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first. *Getting Started* There are three ways to get started with RDO. To spin up a proof of concept cloud, quickly, and on limited hardware, try the All-In-One Quickstart . You can run RDO on a single node to get a feel for how it works. For a production deployment of RDO, use the TripleO Quickstart and you'll be running a production cloud in short order. Finally, if you want to try out OpenStack, but don't have the time or hardware to run it yourself, visit TryStack , where you can use a free public OpenStack instance, running RDO packages, to experiment with the OpenStack management interface and API, launch instances, configure networks, and generally familiarize yourself with OpenStack OpenStack Summit Recap It's just been a few days since the OpenStack Summit in Barcelona, Spain, and some of us are still recovering. There's always far too much happening at Summit for any one person to experience, so catching up on all of the various blog posts afterwards is part of the experience. Here's some of the event recaps from our community: * Attending OpenStack Summit Ocata * OpenStack Summit, Barcelona: 2 of N * What you missed at OpenStack Summit Two highlights of OpenStack Summit for the RDO community were the RDO Infrastructure meeting, and the Evening with RDO and Ceph. On Monday afternoon, before the Summit officially started, about 20 RDO community members gathered to discuss some of the infrastructure issues around the RDO project. As RDO has grown, our infrastructure needs have grown, and various community members have brought up services to address these needs. Over time, our infrastructure footprint has become complicated enough that a beginner is likely to have trouble figuring out what goes where. Over the coming year, we're going to try to consolidate some of these services to fewer locations, preferably running on RDO-based clouds, and document these services so that it's easier for newbies to jump in. This, and other infrastructure related issues, were discussed. You can see the agenda, and notes, from that meeting, in the meeting etherpad . On Tuesday evening, RDO and Ceph together hosted a gathering at the Princess Hotel. It was an evening of food, drinks, and technical sessions about Ceph and RDO. We ended up having 215 people in attendance, and 12 speakers, covering a variety of topics around our two projects. Watch rdo-list and @rdocommunity for the slides from these presentations over the coming days. Ocata Test Day Schedule Based on the Ocata development schedule , we've posted a tentative Ocata RDO test day schedule on the RDO website. Mark your calendar. Plan to spend a few hours on those days testing the latest RDO packages, and help make RDO Ocata the best RDO yet. Over the last few test days, we've improved the test day instructions to make it easier for you to participate without having to know everything about RDO ahead of time. So don't worry if you're not an OpenStack expert. Bring a fresh CentOS VM to the party, and we'll provide the rest of the pieces. Here's the tentative schedule: * Ocata 1 ? December 1st and 2nd, 2016 * Ocata 3 ? February 2nd and 3rd, 2017 * Ocata Final ? March 2nd and 3rd, 2017 You'll notice there's only three test days listed - Ocata is a shorter than usual cycle, and falls during a time of the year where it's already difficult to find test day dates. So we're dropping the Ocata 2 test day, and moving tghe others around a little. This makes it all the more important that the community show up to test. Upcoming Events There's several events on the horizon that you should be aware of. There are some of the upcoming OpenStack Day events where we expect members of the RDO community to be present. * OpenStack DACH Tag , Berlin, November 5th * OpenStack Day France , Paris, November 22nd * OpenStack Day Canada , Montreal, November 22nd Other RDO events, including the many OpenStack meetups around the world, are always listed on the RDO events page . If you have an RDO-related event, please feel free to add it by submitting a pull request on Github . Community meetings Every Wednesday at 15:00 UTC, we have the weekly RDO community meeting on the #RDO channel on Freenode IRC. And at 15:00 UTC Thursdays, we have the CentOS Cloud SIG Meeting on #centos-devel. Keep in touch There's lots of ways to stay in in touch with what's going on in the RDO community. The best ways are ? WWW * RDO * OpenStack Q&A Mailing Lists: * rdo-list mailing list * This newsletter IRC * IRC - #rdo on Freenode.irc.net * Puppet module development - #rdo-puppet Social Media * Follow us on Twitter * Google+ * Facebook Thanks again for being part of the RDO community! -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- _______________________________________________ Rdo-newsletter mailing list Rdo-newsletter at redhat.com https://www.redhat.com/mailman/listinfo/rdo-newsletter From cems at ebi.ac.uk Fri Nov 4 14:45:15 2016 From: cems at ebi.ac.uk (Charles Short) Date: Fri, 4 Nov 2016 14:45:15 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> Message-ID: Hi, So you are implying that tripleO is not really currently able to roll out large deployments easily as it is is prone to scaling delays/errors? Is the same true for RH OSP9 (out of the box) as this also uses tripleO? I would expect exactly the same scaling issues. But surely OSP9 is designed for large enterprise Openstack installations? So if OSP9 does work well with large deployments, what are the tripleO tweaks that make this work (if any)? Many Thanks Charles On 03/11/2016 13:30, Justin Kilpatrick wrote: > Hey Charles, > > If you want to deploy a large number of machines, I suggest you deploy > a small configuration (maybe 3 controllers 1 compute) and then run the > overcloud deploy command again with 2 computes, so on and so forth > until you reach your full allocation > > Realistically you can probably do a stride of 5 computes each time, > experiment with it a bit, as you get up to the full allocation of > nodes you might run into a race condition bug with assigning computes > to nodes and need to pin nodes (pinning is adding as an ironic > property that overcloud-novacompute-0 goes here, 1 here, so on and so > forth). > > As for actually solving the deployment issues at scale (instead of > this horrible hack) I'm looking into adding some robustness at the > ironic or tripleo level to these operations. It sounds like you're > running more into node assignment issues rather than pxe issues though. > > 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto > >: > > On Wed, Nov 2, 2016 at 8:30 PM, Charles Short > wrote: > > Some more testing of different amounts of nodes vs time taken > for successful > > deployments - > > > > 3 controller 3 compute = 1 hour > > 3 controller 15 compute = 1 hour > > 3 controller 25 compute = 1 hour 45 mins > > 3 controller 35 compute = 4 hours > > Hello, > > i'm now preparing my deployment of 3+2 nodes. I'll check what you > reported and give you some feedback. > > Luca > > > -- > "E' assurdo impiegare gli uomini di intelligenza eccellente per fare > calcoli che potrebbero essere affidati a chiunque se si usassero delle > macchine" > Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) > > "Internet ? la pi? grande biblioteca del mondo. > Ma il problema ? che i libri sono tutti sparsi sul pavimento" > John Allen Paulos, Matematico (1945-vivente) > > Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , > > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > > To unsubscribe: rdo-list-unsubscribe at redhat.com > > > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkilpatr at redhat.com Fri Nov 4 15:17:06 2016 From: jkilpatr at redhat.com (Justin Kilpatrick) Date: Fri, 4 Nov 2016 11:17:06 -0400 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> Message-ID: Hey Charles, What sort of issues are you seeing now? How did node pinning work out and did a slow scale up present any more problems? Deployments tend to be disk and network limited, you don't mention what sort of disks your machines have but you do note 1g nics, which are doable but might require some timeout adjustments or other considerations to give everything time to complete. On Fri, Nov 4, 2016 at 10:45 AM, Charles Short wrote: > Hi, > > So you are implying that tripleO is not really currently able to roll out > large deployments easily as it is is prone to scaling delays/errors? > Is the same true for RH OSP9 (out of the box) as this also uses tripleO? > I would expect exactly the same scaling issues. But surely OSP9 is designed > for large enterprise Openstack installations? > So if OSP9 does work well with large deployments, what are the tripleO > tweaks that make this work (if any)? > > Many Thanks > > Charles > > On 03/11/2016 13:30, Justin Kilpatrick wrote: > > Hey Charles, > > If you want to deploy a large number of machines, I suggest you deploy a > small configuration (maybe 3 controllers 1 compute) and then run the > overcloud deploy command again with 2 computes, so on and so forth until > you reach your full allocation > > Realistically you can probably do a stride of 5 computes each time, > experiment with it a bit, as you get up to the full allocation of nodes you > might run into a race condition bug with assigning computes to nodes and > need to pin nodes (pinning is adding as an ironic property that > overcloud-novacompute-0 goes here, 1 here, so on and so forth). > > As for actually solving the deployment issues at scale (instead of this > horrible hack) I'm looking into adding some robustness at the ironic or > tripleo level to these operations. It sounds like you're running more into > node assignment issues rather than pxe issues though. > > 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto < > lorenzetto.luca at gmail.com>: > >> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short wrote: >> > Some more testing of different amounts of nodes vs time taken for >> successful >> > deployments - >> > >> > 3 controller 3 compute = 1 hour >> > 3 controller 15 compute = 1 hour >> > 3 controller 25 compute = 1 hour 45 mins >> > 3 controller 35 compute = 4 hours >> >> Hello, >> >> i'm now preparing my deployment of 3+2 nodes. I'll check what you >> reported and give you some feedback. >> >> Luca >> >> >> -- >> "E' assurdo impiegare gli uomini di intelligenza eccellente per fare >> calcoli che potrebbero essere affidati a chiunque se si usassero delle >> macchine" >> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) >> >> "Internet ? la pi? grande biblioteca del mondo. >> Ma il problema ? che i libri sono tutti sparsi sul pavimento" >> John Allen Paulos, Matematico (1945-vivente) >> >> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , < >> lorenzetto.luca at gmail.com> >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list at redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com >> > > > -- > Charles Short > Cloud Engineer > Virtualization and Cloud Team > European Bioinformatics Institute (EMBL-EBI) > Tel: +44 (0)1223 494205 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cems at ebi.ac.uk Fri Nov 4 15:31:38 2016 From: cems at ebi.ac.uk (Charles Short) Date: Fri, 4 Nov 2016 15:31:38 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> Message-ID: <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> Hi, Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. The 1Gb deployment NIC is not really causing the delay. It is very busy for the time the overcloud image is rolled out (the first 30 to 45 mins of deployment), but after that (once all the nodes are up and active with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on average for the rest of the deployment. For info the NICS in the nodes for the Overcloud networks are dual bonded 10Gbit. The deployment I mentioned before (50 nodes) actually completed in 8 hours (which is double the time it took for 35 nodes!) I am in the process of a new 3 controller 59 compute node deployment pinning all the nodes as you suggested. The initial overcloud image roll out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5 hours in and all is running (slowly). It is currently on Step2 (of 5 Steps). I would expect this deployment to take 10 hours on current speed. Regards Charles On 04/11/2016 15:17, Justin Kilpatrick wrote: > Hey Charles, > > What sort of issues are you seeing now? How did node pinning work out > and did a slow scale up present any more problems? > > Deployments tend to be disk and network limited, you don't mention > what sort of disks your machines have but you do note 1g nics, which > are doable but might require some timeout adjustments or other > considerations to give everything time to complete. > > On Fri, Nov 4, 2016 at 10:45 AM, Charles Short > wrote: > > Hi, > > So you are implying that tripleO is not really currently able to > roll out large deployments easily as it is is prone to scaling > delays/errors? > Is the same true for RH OSP9 (out of the box) as this also uses > tripleO? I would expect exactly the same scaling issues. But > surely OSP9 is designed for large enterprise Openstack installations? > So if OSP9 does work well with large deployments, what are the > tripleO tweaks that make this work (if any)? > > Many Thanks > > Charles > > On 03/11/2016 13:30, Justin Kilpatrick wrote: >> Hey Charles, >> >> If you want to deploy a large number of machines, I suggest you >> deploy a small configuration (maybe 3 controllers 1 compute) and >> then run the overcloud deploy command again with 2 computes, so >> on and so forth until you reach your full allocation >> >> Realistically you can probably do a stride of 5 computes each >> time, experiment with it a bit, as you get up to the full >> allocation of nodes you might run into a race condition bug with >> assigning computes to nodes and need to pin nodes (pinning is >> adding as an ironic property that overcloud-novacompute-0 goes >> here, 1 here, so on and so forth). >> >> As for actually solving the deployment issues at scale (instead >> of this horrible hack) I'm looking into adding some robustness at >> the ironic or tripleo level to these operations. It sounds like >> you're running more into node assignment issues rather than pxe >> issues though. >> >> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >> >: >> >> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short > > wrote: >> > Some more testing of different amounts of nodes vs time >> taken for successful >> > deployments - >> > >> > 3 controller 3 compute = 1 hour >> > 3 controller 15 compute = 1 hour >> > 3 controller 25 compute = 1 hour 45 mins >> > 3 controller 35 compute = 4 hours >> >> Hello, >> >> i'm now preparing my deployment of 3+2 nodes. I'll check what you >> reported and give you some feedback. >> >> Luca >> >> >> -- >> "E' assurdo impiegare gli uomini di intelligenza eccellente >> per fare >> calcoli che potrebbero essere affidati a chiunque se si >> usassero delle >> macchine" >> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) >> >> "Internet ? la pi? grande biblioteca del mondo. >> Ma il problema ? che i libri sono tutti sparsi sul pavimento" >> John Allen Paulos, Matematico (1945-vivente) >> >> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >> > >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list at redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com >> >> >> > > -- > Charles Short > Cloud Engineer > Virtualization and Cloud Team > European Bioinformatics Institute (EMBL-EBI) > Tel:+44 (0)1223 494205 > > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rasca at redhat.com Fri Nov 4 17:36:51 2016 From: rasca at redhat.com (Raoul Scarazzini) Date: Fri, 4 Nov 2016 18:36:51 +0100 Subject: [rdo-list] [quickstart] Status of undercloud baremetal role and future evolutions Message-ID: Hi guys, just would like to share the status around the role in the subject so to keep the one interested up to date. What we want to achieve with this role is to have a ready to use baremetal undercloud, like the virtual one produced by tripleo-quickstart. At the moment there is just one job in the RDO pipeline that uses this role [1], plan is to have more than this, because you already know that "with baremetal is never easy". I've worked to make the role integrate with the oooq evolution, trying to reuse as much as possible the existing extra roles. So, with the latest modifications the role does these basic task: 1) machine-provision: provide the machine with specific a script that needs to be passed; 2) machine-setup: create the user, add the machine to ansible inventory and basically makes virthost == undercloud; 3) undercloud-repos-conf: configure the repos based upon the release parameter passed to quickstar.sh command line; 4) overcloud-images: download the images again based upon the release parameter; So a playbook that deploys a complete undercloud baremetal env should include these roles: - tripleo-baremetal-undercloud - undercloud-scripts (from tripleo-quickstart) - undercloud-install (from tripleo-quickstart) The additional role for the baremetal preparation must be used to finish the bm setup: - overcloud-prep-baremetal And then, from this point on, everything can proceed with the usual order, so: - overcloud-prep-config - overcloud-prep-images - overcloud-prep-flavors - overcloud-prep-network - tripleo-overcloud - tripleo-inventory In addition one can specify the validation steps, in my case I'm using this [2] because of HA. Interventions were needed on these roles: - ansible-role-tripleo-overcloud-prep-network: Do not copy network-environment.yaml on baremetal [3]; - ansible-role-tripleo-overcloud-prep-baremetal: Add optional upstream ipxe installation [4]; - ansible-role-tripleo-baremetal-undercloud: Integrate role with oooq extra roles [5]; That's basically all, all this stuff was tested as working on a bm env, and basically I'm asking now for reviews and considerations. Thanks, [1] https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-newton-baremetal-undercloud-haa01-lab-float_nic_with_vlans/ [2] https://github.com/redhat-openstack/ansible-role-tripleo-overcloud-validate-ha [3] https://review.gerrithub.io/#/c/300865/ [4] https://review.gerrithub.io/#/c/300866/ [5] https://review.gerrithub.io/#/c/300869/ -- Raoul Scarazzini rasca at redhat.com From ggillies at redhat.com Sun Nov 6 23:25:29 2016 From: ggillies at redhat.com (Graeme Gillies) Date: Mon, 7 Nov 2016 09:25:29 +1000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> Message-ID: Hi Charles, This definitely looks a bit strange to me, as we do deploys around 42 nodes and it takes around 2 hours to do so, similar to your setup (1G link for provisoning, bonded 10G for everything else). Would it be possible for you to run an sosreport on your undercloud and provide it somewhere (if you are comfortable doing so). Also, can you show us the output of openstack stack list --nested And most importantly, if we can get a fully copy of the output of the overcloud deploy command, that has timestamps against when ever stack is created/finished, so we can try and narrow down where all the time is being spent. You note that you have quite a powerful undercloud (294GB of Memory and 64 cpus), and we have had issues in the past with very powerful underclouds, because the Openstack components try and tune themselves around the hardware they are running on and get it wrong for bigger servers. Are we able to get an output from "sar" or some other tool you are using to track cpu and memory usage during the deployment? I'd like to check those values look sane. Thanks in advance, Graeme On 05/11/16 01:31, Charles Short wrote: > Hi, > > Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. > The 1Gb deployment NIC is not really causing the delay. It is very busy > for the time the overcloud image is rolled out (the first 30 to 45 mins > of deployment), but after that (once all the nodes are up and active > with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on > average for the rest of the deployment. For info the NICS in the nodes > for the Overcloud networks are dual bonded 10Gbit. > > The deployment I mentioned before (50 nodes) actually completed in 8 > hours (which is double the time it took for 35 nodes!) > > I am in the process of a new 3 controller 59 compute node deployment > pinning all the nodes as you suggested. The initial overcloud image roll > out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5 > hours in and all is running (slowly). It is currently on Step2 (of 5 > Steps). I would expect this deployment to take 10 hours on current speed. > > Regards > > Charles > > On 04/11/2016 15:17, Justin Kilpatrick wrote: >> Hey Charles, >> >> What sort of issues are you seeing now? How did node pinning work out >> and did a slow scale up present any more problems? >> >> Deployments tend to be disk and network limited, you don't mention >> what sort of disks your machines have but you do note 1g nics, which >> are doable but might require some timeout adjustments or other >> considerations to give everything time to complete. >> >> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short > > wrote: >> >> Hi, >> >> So you are implying that tripleO is not really currently able to >> roll out large deployments easily as it is is prone to scaling >> delays/errors? >> Is the same true for RH OSP9 (out of the box) as this also uses >> tripleO? I would expect exactly the same scaling issues. But >> surely OSP9 is designed for large enterprise Openstack installations? >> So if OSP9 does work well with large deployments, what are the >> tripleO tweaks that make this work (if any)? >> >> Many Thanks >> >> Charles >> >> On 03/11/2016 13:30, Justin Kilpatrick wrote: >>> Hey Charles, >>> >>> If you want to deploy a large number of machines, I suggest you >>> deploy a small configuration (maybe 3 controllers 1 compute) and >>> then run the overcloud deploy command again with 2 computes, so >>> on and so forth until you reach your full allocation >>> >>> Realistically you can probably do a stride of 5 computes each >>> time, experiment with it a bit, as you get up to the full >>> allocation of nodes you might run into a race condition bug with >>> assigning computes to nodes and need to pin nodes (pinning is >>> adding as an ironic property that overcloud-novacompute-0 goes >>> here, 1 here, so on and so forth). >>> >>> As for actually solving the deployment issues at scale (instead >>> of this horrible hack) I'm looking into adding some robustness at >>> the ironic or tripleo level to these operations. It sounds like >>> you're running more into node assignment issues rather than pxe >>> issues though. >>> >>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >>> >: >>> >>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short >> > wrote: >>> > Some more testing of different amounts of nodes vs time >>> taken for successful >>> > deployments - >>> > >>> > 3 controller 3 compute = 1 hour >>> > 3 controller 15 compute = 1 hour >>> > 3 controller 25 compute = 1 hour 45 mins >>> > 3 controller 35 compute = 4 hours >>> >>> Hello, >>> >>> i'm now preparing my deployment of 3+2 nodes. I'll check what you >>> reported and give you some feedback. >>> >>> Luca >>> >>> >>> -- >>> "E' assurdo impiegare gli uomini di intelligenza eccellente >>> per fare >>> calcoli che potrebbero essere affidati a chiunque se si >>> usassero delle >>> macchine" >>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) >>> >>> "Internet ? la pi? grande biblioteca del mondo. >>> Ma il problema ? che i libri sono tutti sparsi sul pavimento" >>> John Allen Paulos, Matematico (1945-vivente) >>> >>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >>> > >>> >>> _______________________________________________ >>> rdo-list mailing list >>> rdo-list at redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> >>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>> >>> >>> >> >> -- >> Charles Short >> Cloud Engineer >> Virtualization and Cloud Team >> European Bioinformatics Institute (EMBL-EBI) >> Tel: +44 (0)1223 494205 >> >> > > -- > Charles Short > Cloud Engineer > Virtualization and Cloud Team > European Bioinformatics Institute (EMBL-EBI) > Tel: +44 (0)1223 494205 > > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > -- Graeme Gillies Principal Systems Administrator Openstack Infrastructure Red Hat Australia From whayutin at redhat.com Mon Nov 7 13:59:02 2016 From: whayutin at redhat.com (Wesley Hayutin) Date: Mon, 7 Nov 2016 08:59:02 -0500 Subject: [rdo-list] [CI] combine rdo master and tripleo status etherpads Message-ID: Greetings, I think it would make sense to use one etherpad to track the status of openstack master. There are several etherpads and it's confusing to use and get status. Is it reasonable to: - move all work to [1], maybe we need to rename it. - move any items from [2] to [1] There was quite a bit of duplicate work going on in both etherpads over the last week with memcache, and ping test issues. Thanks [1] https://etherpad.openstack.org/p/tripleo-ci-status [2] https://etherpad.openstack.org/p/delorean_master_current_issues -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbowen at redhat.com Mon Nov 7 14:21:52 2016 From: rbowen at redhat.com (Rich Bowen) Date: Mon, 7 Nov 2016 09:21:52 -0500 Subject: [rdo-list] Unanswered 'RDO' questions on ask.openstack.org Message-ID: <585f7394-ab78-bfe7-9375-2ca36b99dac6@redhat.com> 28 unanswered questions: Compute nodes are not visible after controller node rebooted https://ask.openstack.org/en/question/98654/compute-nodes-are-not-visible-after-controller-node-rebooted/ Tags: rdo Live migration failing with qemu error "does not support drive-mirror" https://ask.openstack.org/en/question/98144/live-migration-failing-with-qemu-error-does-not-support-drive-mirror/ Tags: newton, nova, qemu, kvm, block-live-migration error while launching instance in openstack 'newton' using rdo packstack https://ask.openstack.org/en/question/98113/error-while-launching-instance-in-openstack-newton-using-rdo-packstack/ Tags: rdo, newton, packstack How can I add a new region to an installed All-In-One RDO openstack? https://ask.openstack.org/en/question/98111/how-can-i-add-a-new-region-to-an-installed-all-in-one-rdo-openstack/ Tags: rdo, all-in-one, multi-region, region, add-region How to setup solr search engine for openstack swift https://ask.openstack.org/en/question/98093/how-to-setup-solr-search-engine-for-openstack-swift/ Tags: openstack, openstack-swift, newton, rdo Heat fails with 504 error. https://ask.openstack.org/en/question/98092/heat-fails-with-504-error/ Tags: rdo, tripleo, mitaka, heat Devstack and OpenvSwitch v2.3+ https://ask.openstack.org/en/question/98004/devstack-and-openvswitch-v23/ Tags: devstack, openvswitch How how does icmp packet travel across two compute nodes without br-int and br-tun running? https://ask.openstack.org/en/question/97905/how-how-does-icmp-packet-travel-across-two-compute-nodes-without-br-int-and-br-tun-running/ Tags: neutron-ovs-pktflow "Parameter outiface failed on Firewall" during installation of openstack rdo on centos 7 https://ask.openstack.org/en/question/95657/parameter-outiface-failed-on-firewall-during-installation-of-openstack-rdo-on-centos-7/ Tags: rdo, devstack#mitaka multi nodes provider network ovs config https://ask.openstack.org/en/question/95423/multi-nodes-provider-network-ovs-config/ Tags: rdo, liberty-neutron Adding additional packages to an RDO installation https://ask.openstack.org/en/question/95380/adding-additional-packages-to-an-rdo-installation/ Tags: rdo, mistral RDO TripleO Mitaka HA Overcloud Failing https://ask.openstack.org/en/question/95249/rdo-tripleo-mitaka-ha-overcloud-failing/ Tags: mitaka, tripleo, overcloud, centos7 RDO - is there any fedora package newer than puppet-4.2.1-3.fc24.noarch.rpm https://ask.openstack.org/en/question/94969/rdo-is-there-any-fedora-package-newer-than-puppet-421-3fc24noarchrpm/ Tags: rdo, puppet, install-openstack how to deploy haskell-distributed in RDO? https://ask.openstack.org/en/question/94785/how-to-deploy-haskell-distributed-in-rdo/ Tags: rdo rdo tripleO liberty undercloud install failing https://ask.openstack.org/en/question/94023/rdo-tripleo-liberty-undercloud-install-failing/ Tags: rdo, rdo-manager, liberty, undercloud, instack Liberty RDO: stack resource topology icons are pink https://ask.openstack.org/en/question/91347/liberty-rdo-stack-resource-topology-icons-are-pink/ Tags: resource, topology, dashboard, horizon, pink No handlers could be found for logger "oslo_config.cfg" while syncing the glance database https://ask.openstack.org/en/question/91169/no-handlers-could-be-found-for-logger-oslo_configcfg-while-syncing-the-glance-database/ Tags: liberty, glance, install-openstack CentOS OpenStack - compute node can't talk https://ask.openstack.org/en/question/88989/centos-openstack-compute-node-cant-talk/ Tags: rdo How to setup SWIFT_PROXY_NODE and SWIFT_STORAGE_NODEs separately on RDO Liberty ? https://ask.openstack.org/en/question/88897/how-to-setup-swift_proxy_node-and-swift_storage_nodes-separately-on-rdo-liberty/ Tags: rdo, liberty, swift, ha VM and container can't download anything from internet https://ask.openstack.org/en/question/88338/vm-and-container-cant-download-anything-from-internet/ Tags: rdo, neutron, network, connectivity Fedora22, Liberty, horizon VNC console and keymap=sv with ; and/ https://ask.openstack.org/en/question/87451/fedora22-liberty-horizon-vnc-console-and-keymapsv-with-and/ Tags: keyboard, map, keymap, vncproxy, novnc OpenStack-Docker driver failed https://ask.openstack.org/en/question/87243/openstack-docker-driver-failed/ Tags: docker, openstack, liberty Routing between two tenants https://ask.openstack.org/en/question/84645/routing-between-two-tenants/ Tags: kilo, fuel, rdo, routing openstack baremetal introspection internal server error https://ask.openstack.org/en/question/82790/openstack-baremetal-introspection-internal-server-error/ Tags: rdo, ironic-inspector, tripleo Installing openstack using packstack (rdo) failed https://ask.openstack.org/en/question/82473/installing-openstack-using-packstack-rdo-failed/ Tags: rdo, packstack, installation-error, keystone VMware Host Backend causes No valid host was found. Bug ??? https://ask.openstack.org/en/question/79738/vmware-host-backend-causes-no-valid-host-was-found-bug/ Tags: vmware, rdo Mutlinode Devstack with two interfaces https://ask.openstack.org/en/question/78615/mutlinode-devstack-with-two-interfaces/ Tags: devstack, vlan, openstack Overcloud deployment on VM fails as IP address from DHCP is not assigned https://ask.openstack.org/en/question/66272/overcloud-deployment-on-vm-fails-as-ip-address-from-dhcp-is-not-assigned/ Tags: overcloud_in_vm -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From dms at redhat.com Mon Nov 7 14:58:53 2016 From: dms at redhat.com (David Moreau Simard) Date: Mon, 7 Nov 2016 09:58:53 -0500 Subject: [rdo-list] [CI] combine rdo master and tripleo status etherpads In-Reply-To: References: Message-ID: +1 from me David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Mon, Nov 7, 2016 at 8:59 AM, Wesley Hayutin wrote: > Greetings, > > I think it would make sense to use one etherpad to track the status of > openstack master. There are several etherpads and it's confusing to use and > get status. > > Is it reasonable to: > > move all work to [1], maybe we need to rename it. > move any items from [2] to [1] > > There was quite a bit of duplicate work going on in both etherpads over the > last week with memcache, and ping test issues. > > Thanks > > [1] https://etherpad.openstack.org/p/tripleo-ci-status > [2] https://etherpad.openstack.org/p/delorean_master_current_issues From hguemar at fedoraproject.org Mon Nov 7 15:00:04 2016 From: hguemar at fedoraproject.org (hguemar at fedoraproject.org) Date: Mon, 7 Nov 2016 15:00:04 +0000 (UTC) Subject: [rdo-list] [Fedocal] Reminder meeting : RDO meeting Message-ID: <20161107150004.12A0D60A3FDC@fedocal02.phx2.fedoraproject.org> Dear all, You are kindly invited to the meeting: RDO meeting on 2016-11-09 from 15:00:00 to 16:00:00 UTC At rdo at irc.freenode.net The meeting will be about: RDO IRC meeting [Agenda at https://etherpad.openstack.org/p/RDO-Meeting ](https://etherpad.openstack.org/p/RDO-Meeting) Every Wednesday on #rdo on Freenode IRC Source: https://apps.fedoraproject.org/calendar/meeting/2017/ From amoralej at redhat.com Mon Nov 7 15:15:26 2016 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Mon, 7 Nov 2016 16:15:26 +0100 Subject: [rdo-list] [CI] combine rdo master and tripleo status etherpads In-Reply-To: References: Message-ID: I'm ok with consolidating the etherpads, but then maybe we should add some info to clearly identify what pipelines or CIs are impacted by each issue, for example issues with packstack or p-o-i impact on delorean-master-promotion but not in Tripleo CI. On Mon, Nov 7, 2016 at 2:59 PM, Wesley Hayutin wrote: > Greetings, > > I think it would make sense to use one etherpad to track the status of > openstack master. There are several etherpads and it's confusing to use and > get status. > > Is it reasonable to: > > move all work to [1], maybe we need to rename it. > move any items from [2] to [1] > > There was quite a bit of duplicate work going on in both etherpads over the > last week with memcache, and ping test issues. > > Thanks > > [1] https://etherpad.openstack.org/p/tripleo-ci-status > [2] https://etherpad.openstack.org/p/delorean_master_current_issues > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From javier.pena at redhat.com Mon Nov 7 15:56:16 2016 From: javier.pena at redhat.com (Javier Pena) Date: Mon, 7 Nov 2016 10:56:16 -0500 (EST) Subject: [rdo-list] [CI] combine rdo master and tripleo status etherpads In-Reply-To: References: Message-ID: <259340835.13517837.1478534176025.JavaMail.zimbra@redhat.com> ----- Original Message ----- > I'm ok with consolidating the etherpads, but then maybe we should add > some info to clearly identify what pipelines or CIs are impacted by > each issue, for example issues with packstack or p-o-i impact on > delorean-master-promotion but not in Tripleo CI. > +1 to this. Also, let's keep in mind that both etherpads are parsed by http://dashboards.rdoproject.org/rdo-dev, so we'll need to adapt it as well. Javier > > On Mon, Nov 7, 2016 at 2:59 PM, Wesley Hayutin wrote: > > Greetings, > > > > I think it would make sense to use one etherpad to track the status of > > openstack master. There are several etherpads and it's confusing to use > > and > > get status. > > > > Is it reasonable to: > > > > move all work to [1], maybe we need to rename it. > > move any items from [2] to [1] > > > > There was quite a bit of duplicate work going on in both etherpads over the > > last week with memcache, and ping test issues. > > > > Thanks > > > > [1] https://etherpad.openstack.org/p/tripleo-ci-status > > [2] https://etherpad.openstack.org/p/delorean_master_current_issues > > > > _______________________________________________ > > rdo-list mailing list > > rdo-list at redhat.com > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From ggillies at redhat.com Tue Nov 8 00:30:31 2016 From: ggillies at redhat.com (Graeme Gillies) Date: Tue, 8 Nov 2016 10:30:31 +1000 Subject: [rdo-list] Python-shade in RDO In-Reply-To: References: <9f38876e-eec8-b4b1-11fd-68d14f7c471d@redhat.com> <3cdfa6fc-6ca7-b99c-b2c2-2f0a59719ab8@redhat.com> <4ee88a00-980a-8e01-c25e-3fdf4a3ba28c@redhat.com> <1a036e38-1126-1a01-5ebe-1c198432e516@redhat.com> Message-ID: Apologies for top posting, trying to resurrect this thread. So if I am understanding correctly, the outcome from this email thread was that a new repo in RDO was going to be setup for packages in Openstack big tent which do not follow the "stable branch" model. Was this ever done? If so, are we in a position to put python-shade inside it? I recently rebuilt it for centos 7 in COPR https://copr.fedorainfracloud.org/coprs/ggillies/rdo-newton-extras/packages/ And it works fine. If we haven't gotten to the point of creating this repo, what do we need to do to get this done? How do we move forward from here? Regards, Graeme On 30/08/16 23:19, Ha?kel wrote: > 2016-08-30 8:23 GMT+02:00 Graeme Gillies : >> >> I would really implore you to think very carefully about splitting out >> release:independant projects into a separate repo, especially one with >> the name "extras". >> > > Naming is easy to change, could be RDO clients and SDK, RDO Cloud > users or whatever > >> Fracturing the repo setup only reduces usability, causes confusion, and >> for those projects that choose an independent release model, it makes >> them feel like second class citizens and less likely to want to care or >> be part of the RDO community. >> > > As Alan explained, we can't. > > Actually, forcing release-independent projects like shade in > release-specific repositories would make them more of a second-class > citizen. Let's say that shade requires a newer version of clients than > we can ship in Mitaka, that would force us to pin shade to an older > release and maybe fork it to backport security updates. > A separate repository would give us the flexibility to ship the latest > and greatest of those projects without worrying to break stuff that > will be installed in our cloud nodes. > >> As I've already mentioned, we already ship a release:independant project >> in the normal repo, and I fail to see from a technical level why others >> can't do the same. Simply ship their latest stable release in the >> current stable repo. >> > > That's poor workaround because we didn't have the flexibility to do otherwise. > As for shade, stable releases requirements don't match > release-dependent projects requirements and we do have a precedent: > gnocchi. > Gnocchi follows its own branching model and because of requirements, > we have to collaborate with upstream devs to map specific gnocchi > branches to an OpenStack release, though if you install gnocchi nodes > in separated nodes, it does not matter. > > >> A lot of the projects that choose release:independant are smaller, >> perhaps a bit newer, and still in a growth phase. If RDO is able to show >> that those projects can be a part of RDO like everything else, it means >> they are more likely to participate in the community, as it makes their >> software more accessible. >> > > We can provide flexibility like we did with gnocchi but here I'd > rather think of how could we provide better fits to our community by > adding a new repository to fit the needs of cloud user > >> Remember the goal here is to grow the community and have as many >> projects participating in RDO as possible. Encouraging the smaller >> projects to do so is a great way to help that. >> >> Regards, >> >> Graeme >> > > *nods* > > Regards, > H. > >>> >>>> Regards, >>>> >>>> Graeme >>>> >>>> [1] https://governance.openstack.org/reference/tags/ >>>> >>>> -- >>>> Graeme Gillies >>>> Principal Systems Administrator >>>> Openstack Infrastructure >>>> Red Hat Australia >> >> >> -- >> Graeme Gillies >> Principal Systems Administrator >> Openstack Infrastructure >> Red Hat Australia -- Graeme Gillies Principal Systems Administrator Openstack Infrastructure Red Hat Australia From whayutin at redhat.com Tue Nov 8 02:53:01 2016 From: whayutin at redhat.com (Wesley Hayutin) Date: Mon, 7 Nov 2016 21:53:01 -0500 Subject: [rdo-list] [quickstart] Status of undercloud baremetal role and future evolutions In-Reply-To: References: Message-ID: On Fri, Nov 4, 2016 at 1:36 PM, Raoul Scarazzini wrote: > Hi guys, > just would like to share the status around the role in the subject so to > keep the one interested up to date. What we want to achieve with this > role is to have a ready to use baremetal undercloud, like the virtual > one produced by tripleo-quickstart. > At the moment there is just one job in the RDO pipeline that uses this > role [1], plan is to have more than this, because you already know that > "with baremetal is never easy". > This role sets up the undercloud on a provided server directly w/o setting up a virtual guest for the undercloud. I would think the performance team would be interested in this role so that the undercloud performance metrics are not skewed by libvirt. I was also thinking it may be useful in the multinode nodepool workflow, where the undercloud is directly installed. I don't see there being an advantage in using it, but I could be missing something. I could also see ops teams taking advantage of this to deploy out to production environments. We would need to ensure the role is uncoupled from any provisioning, but I see it being valuable for sure. I've copied some other folks to get their thoughts. Thanks Raoul! > > I've worked to make the role integrate with the oooq evolution, trying > to reuse as much as possible the existing extra roles. So, with the > latest modifications the role does these basic task: > > 1) machine-provision: provide the machine with specific a script that > needs to be passed; > 2) machine-setup: create the user, add the machine to ansible inventory > and basically makes virthost == undercloud; > 3) undercloud-repos-conf: configure the repos based upon the release > parameter passed to quickstar.sh command line; > 4) overcloud-images: download the images again based upon the release > parameter; > > So a playbook that deploys a complete undercloud baremetal env should > include these roles: > > - tripleo-baremetal-undercloud > - undercloud-scripts (from tripleo-quickstart) > - undercloud-install (from tripleo-quickstart) > > The additional role for the baremetal preparation must be used to finish > the bm setup: > > - overcloud-prep-baremetal > > And then, from this point on, everything can proceed with the usual > order, so: > > - overcloud-prep-config > - overcloud-prep-images > - overcloud-prep-flavors > - overcloud-prep-network > - tripleo-overcloud > - tripleo-inventory > > In addition one can specify the validation steps, in my case I'm using > this [2] because of HA. > > Interventions were needed on these roles: > > - ansible-role-tripleo-overcloud-prep-network: Do not copy > network-environment.yaml on baremetal [3]; > - ansible-role-tripleo-overcloud-prep-baremetal: Add optional upstream > ipxe installation [4]; > - ansible-role-tripleo-baremetal-undercloud: Integrate role with oooq > extra roles [5]; > > That's basically all, all this stuff was tested as working on a bm env, > and basically I'm asking now for reviews and considerations. > > Thanks, > > [1] > https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ > tripleo-quickstart-newton-baremetal-undercloud-haa01- > lab-float_nic_with_vlans/ > [2] > https://github.com/redhat-openstack/ansible-role- > tripleo-overcloud-validate-ha > [3] https://review.gerrithub.io/#/c/300865/ > [4] https://review.gerrithub.io/#/c/300866/ > [5] https://review.gerrithub.io/#/c/300869/ > > -- > Raoul Scarazzini > rasca at redhat.com > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pabelanger at redhat.com Tue Nov 8 13:14:29 2016 From: pabelanger at redhat.com (Paul Belanger) Date: Tue, 8 Nov 2016 08:14:29 -0500 Subject: [rdo-list] Python-shade in RDO In-Reply-To: References: <9f38876e-eec8-b4b1-11fd-68d14f7c471d@redhat.com> <3cdfa6fc-6ca7-b99c-b2c2-2f0a59719ab8@redhat.com> <4ee88a00-980a-8e01-c25e-3fdf4a3ba28c@redhat.com> <1a036e38-1126-1a01-5ebe-1c198432e516@redhat.com> Message-ID: <20161108131429.GA15851@localhost.localdomain> On Tue, Nov 08, 2016 at 10:30:31AM +1000, Graeme Gillies wrote: > Apologies for top posting, trying to resurrect this thread. > > So if I am understanding correctly, the outcome from this email thread > was that a new repo in RDO was going to be setup for packages in > Openstack big tent which do not follow the "stable branch" model. > > Was this ever done? If so, are we in a position to put python-shade > inside it? I recently rebuilt it for centos 7 in COPR > > https://copr.fedorainfracloud.org/coprs/ggillies/rdo-newton-extras/packages/ > > And it works fine. > > If we haven't gotten to the point of creating this repo, what do we need > to do to get this done? How do we move forward from here? > I believe we have some upcoming changes to shade that is going to make packaging a little easier. I seen a patch from Monty recently to replace python-*client dependencies. With that in mind, it should make landing shade into EPEL a little easier. > Regards, > > Graeme > > On 30/08/16 23:19, Ha?kel wrote: > > 2016-08-30 8:23 GMT+02:00 Graeme Gillies : > >> > >> I would really implore you to think very carefully about splitting out > >> release:independant projects into a separate repo, especially one with > >> the name "extras". > >> > > > > Naming is easy to change, could be RDO clients and SDK, RDO Cloud > > users or whatever > > > >> Fracturing the repo setup only reduces usability, causes confusion, and > >> for those projects that choose an independent release model, it makes > >> them feel like second class citizens and less likely to want to care or > >> be part of the RDO community. > >> > > > > As Alan explained, we can't. > > > > Actually, forcing release-independent projects like shade in > > release-specific repositories would make them more of a second-class > > citizen. Let's say that shade requires a newer version of clients than > > we can ship in Mitaka, that would force us to pin shade to an older > > release and maybe fork it to backport security updates. > > A separate repository would give us the flexibility to ship the latest > > and greatest of those projects without worrying to break stuff that > > will be installed in our cloud nodes. > > > >> As I've already mentioned, we already ship a release:independant project > >> in the normal repo, and I fail to see from a technical level why others > >> can't do the same. Simply ship their latest stable release in the > >> current stable repo. > >> > > > > That's poor workaround because we didn't have the flexibility to do otherwise. > > As for shade, stable releases requirements don't match > > release-dependent projects requirements and we do have a precedent: > > gnocchi. > > Gnocchi follows its own branching model and because of requirements, > > we have to collaborate with upstream devs to map specific gnocchi > > branches to an OpenStack release, though if you install gnocchi nodes > > in separated nodes, it does not matter. > > > > > >> A lot of the projects that choose release:independant are smaller, > >> perhaps a bit newer, and still in a growth phase. If RDO is able to show > >> that those projects can be a part of RDO like everything else, it means > >> they are more likely to participate in the community, as it makes their > >> software more accessible. > >> > > > > We can provide flexibility like we did with gnocchi but here I'd > > rather think of how could we provide better fits to our community by > > adding a new repository to fit the needs of cloud user > > > >> Remember the goal here is to grow the community and have as many > >> projects participating in RDO as possible. Encouraging the smaller > >> projects to do so is a great way to help that. > >> > >> Regards, > >> > >> Graeme > >> > > > > *nods* > > > > Regards, > > H. > > > >>> > >>>> Regards, > >>>> > >>>> Graeme > >>>> > >>>> [1] https://governance.openstack.org/reference/tags/ > >>>> > >>>> -- > >>>> Graeme Gillies > >>>> Principal Systems Administrator > >>>> Openstack Infrastructure > >>>> Red Hat Australia > >> > >> > >> -- > >> Graeme Gillies > >> Principal Systems Administrator > >> Openstack Infrastructure > >> Red Hat Australia > > > -- > Graeme Gillies > Principal Systems Administrator > Openstack Infrastructure > Red Hat Australia From jkilpatr at redhat.com Tue Nov 8 13:45:22 2016 From: jkilpatr at redhat.com (Justin Kilpatrick) Date: Tue, 8 Nov 2016 08:45:22 -0500 Subject: [rdo-list] [quickstart] Status of undercloud baremetal role and future evolutions In-Reply-To: References: Message-ID: Nice job Raoul, I'll probably be using this for a couple of things in the coming weeks. Expect more formal overcloud deployment benchmarking soon as I'm wrapping up the alpha of a tool for it, comparing virt vs non-virt underclouds will be worthwhile I'm sure. On Mon, Nov 7, 2016 at 9:53 PM, Wesley Hayutin wrote: > > > On Fri, Nov 4, 2016 at 1:36 PM, Raoul Scarazzini wrote: > >> Hi guys, >> just would like to share the status around the role in the subject so to >> keep the one interested up to date. What we want to achieve with this >> role is to have a ready to use baremetal undercloud, like the virtual >> one produced by tripleo-quickstart. >> At the moment there is just one job in the RDO pipeline that uses this >> role [1], plan is to have more than this, because you already know that >> "with baremetal is never easy". >> > > This role sets up the undercloud on a provided server directly w/o setting > up a virtual guest for the undercloud. > I would think the performance team would be interested in this role so > that the undercloud performance metrics are not skewed by libvirt. > > I was also thinking it may be useful in the multinode nodepool workflow, > where the undercloud is directly installed. I don't see there being an > advantage in using it, but I could be missing something. > > I could also see ops teams taking advantage of this to deploy out to > production environments. We would need to ensure the role is uncoupled > from any provisioning, but I see it being valuable for sure. > > I've copied some other folks to get their thoughts. > Thanks Raoul! > >> >> I've worked to make the role integrate with the oooq evolution, trying >> to reuse as much as possible the existing extra roles. So, with the >> latest modifications the role does these basic task: >> >> 1) machine-provision: provide the machine with specific a script that >> needs to be passed; >> 2) machine-setup: create the user, add the machine to ansible inventory >> and basically makes virthost == undercloud; >> 3) undercloud-repos-conf: configure the repos based upon the release >> parameter passed to quickstar.sh command line; >> 4) overcloud-images: download the images again based upon the release >> parameter; >> >> So a playbook that deploys a complete undercloud baremetal env should >> include these roles: >> >> - tripleo-baremetal-undercloud >> - undercloud-scripts (from tripleo-quickstart) >> - undercloud-install (from tripleo-quickstart) >> >> The additional role for the baremetal preparation must be used to finish >> the bm setup: >> >> - overcloud-prep-baremetal >> >> And then, from this point on, everything can proceed with the usual >> order, so: >> >> - overcloud-prep-config >> - overcloud-prep-images >> - overcloud-prep-flavors >> - overcloud-prep-network >> - tripleo-overcloud >> - tripleo-inventory >> >> In addition one can specify the validation steps, in my case I'm using >> this [2] because of HA. >> >> Interventions were needed on these roles: >> >> - ansible-role-tripleo-overcloud-prep-network: Do not copy >> network-environment.yaml on baremetal [3]; >> - ansible-role-tripleo-overcloud-prep-baremetal: Add optional upstream >> ipxe installation [4]; >> - ansible-role-tripleo-baremetal-undercloud: Integrate role with oooq >> extra roles [5]; >> >> That's basically all, all this stuff was tested as working on a bm env, >> and basically I'm asking now for reviews and considerations. >> >> Thanks, >> >> [1] >> https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tri >> pleo-quickstart-newton-baremetal-undercloud-haa01-lab-float_ >> nic_with_vlans/ >> [2] >> https://github.com/redhat-openstack/ansible-role-tripleo- >> overcloud-validate-ha >> [3] https://review.gerrithub.io/#/c/300865/ >> [4] https://review.gerrithub.io/#/c/300866/ >> [5] https://review.gerrithub.io/#/c/300869/ >> >> -- >> Raoul Scarazzini >> rasca at redhat.com >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list at redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dms at redhat.com Tue Nov 8 13:56:06 2016 From: dms at redhat.com (David Moreau Simard) Date: Tue, 8 Nov 2016 08:56:06 -0500 Subject: [rdo-list] Python-shade in RDO In-Reply-To: <20161108131429.GA15851@localhost.localdomain> References: <9f38876e-eec8-b4b1-11fd-68d14f7c471d@redhat.com> <3cdfa6fc-6ca7-b99c-b2c2-2f0a59719ab8@redhat.com> <4ee88a00-980a-8e01-c25e-3fdf4a3ba28c@redhat.com> <1a036e38-1126-1a01-5ebe-1c198432e516@redhat.com> <20161108131429.GA15851@localhost.localdomain> Message-ID: On Tue, Nov 8, 2016 at 8:14 AM, Paul Belanger wrote: > I believe we have some upcoming changes to shade that is going to make packaging > a little easier. I seen a patch from Monty recently to replace python-*client > dependencies. Ah, indeed, I've seen that he started taking out client library usage from shade with it's own implementation so it should be fairly standalone (i.e, similar to how Tempest has it's own clients). David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] From rbowen at redhat.com Tue Nov 8 14:13:02 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 8 Nov 2016 09:13:02 -0500 Subject: [rdo-list] Meetups this week Message-ID: <88f46036-9ef4-b699-cea2-49d7046946b1@redhat.com> The following are the meetups I'm aware of in the coming week where OpenStack and/or RDO enthusiasts are likely to be present. If you know of others, please let me know, and/or add them to http://rdoproject.org/events If there's a meetup in your area, please consider attending. If you attend, please consider taking a few photos, and possibly even writing up a brief summary of what was covered. --Rich * Tuesday November 08 in Fort Collins, CO, US: Talk about the Barcelona OpenStack Meetup - http://www.meetup.com/OpenStack-Colorado/events/234485140/ * Wednesday November 09 in Reston, VA, US: How Oracle is using Puppet with OpenStack - http://www.meetup.com/OpenStack-Nova/events/234698075/ * Wednesday November 09 in Hamburg, DE: Openstack Meetup #5 powered by Windcloud & Perennial AG - http://www.meetup.com/Openstack-Hamburg/events/234752723/ * Wednesday November 09 in San Diego, CA, US: Building a Cloud - Working/Planning Session - http://www.meetup.com/OpenStack-SD/events/235335095/ * Wednesday November 09 in Charlotte, NC, US: User stories - http://www.meetup.com/Charlotte-OpenStack/events/235252386/ * Thursday November 10 in San Antonio, TX, US: Using Cloud Orchestration and Config Management on AWS and OpenStack - http://www.meetup.com/Cloud-Technology/events/235328712/ * Thursday November 10 in Sunnyvale, CA, US: Diskimage-builder & Zero to DevOps with Zuul - http://www.meetup.com/openstack/events/234949514/ * Saturday November 12 in Ha Noi, VN: Vietnam Official OpenStack Community 12th Technical Meetup - http://www.meetup.com/VietOpenStack/events/235216862/ * Saturday November 12 in Tambaram, IN: AWS Amazon public Cloud - http://www.meetup.com/CloudnLoud-Openstack-Cloud-RedHat-Opensource/events/234771635/ * Monday November 14 in Canberra, AU: OpenStack Australia Day: Government - http://australiaday?????.openstack.org.au? - http://www.meetup.com/Australian-OpenStack-User-Group/events/232940544/ -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From ajith.adapa at gmail.com Tue Nov 8 16:09:47 2016 From: ajith.adapa at gmail.com (Ajith Adapa) Date: Tue, 8 Nov 2016 08:09:47 -0800 Subject: [rdo-list] RDO kilo installation fails in mullti-node setup Message-ID: Hi, I am trying to install multi-node kilo based openstack installation and facing errors. I am following the standard steps which is mentioned in installation guides. yum install https://repos.fedorapeople.org/repos/openstack/openstack-kilo/rdo-release-kilo-2.noarch.rpm yum install openstack-packstack packstack --gen-answer-file=/root/answer.txt I have modified answer.txt for below variables CONFIG_NAGIOS_INSTALL=n CONFIG_CONTROLLER_HOST=172.16.45.44 CONFIG_COMPUTE_HOSTS=172.16.45.45, 172.16.45.46 CONFIG_NETWORK_HOSTS=172.16.45.447 CONFIG_KEYSTONE_ADMIN_PW=password CONFIG_PROVISION_DEMO=n packstack --answer-file=/root/answer.txt 172.16.45.45_neutron.pp: [ DONE ] 172.16.45.46_neutron.pp: [ DONE ] 172.16.45.44_neutron.pp: [ DONE ] 172.16.45.47_neutron.pp: [ DONE ] Applying 172.16.45.44_osclient.pp Applying 172.16.45.44_horizon.pp 172.16.45.44_osclient.pp: [ DONE ] 172.16.45.44_horizon.pp: [ ERROR ] Applying Puppet manifests [ ERROR ] ERROR : Error appeared during Puppet run: 172.16.45.44_horizon.pp Notice: /Stage[main]/Horizon/Exec[refresh_horizon_django_cache]/returns: CommandError: An error occurred during rendering /usr/share/openstack-dashboard/openstack_dashboard/templates/_stylesheets.html: /bin/sh: django_pyscss.compressor.DjangoScssFilter: command not found You will find full trace in log /var/tmp/packstack/20161108-001755-Xd4RYx/manifests/172.16.45.44_horizon.pp.log Please check log file /var/tmp/packstack/20161108-001755-Xd4RYx/openstack-setup.log for more information Additional information: * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * File /root/keystonerc_admin has been created on OpenStack client host 172.16.45.44. To use the command line tools you need to source the file. * To access the OpenStack Dashboard browse to http://172.16.45.44/dashboard . Please, find your login credentials stored in the keystonerc_admin in your home directory. Thanks for your help. Regards, Ajith From amedeo.salvati at fastweb.it Wed Nov 9 11:30:31 2016 From: amedeo.salvati at fastweb.it (Salvati Amedeo) Date: Wed, 9 Nov 2016 11:30:31 +0000 Subject: [rdo-list] OpenStack meetup - Milan 14-12-2016 Message-ID: <751EEF2DDA813143A5F20A73F5B78F3864805D1C@ex020ims.fastwebit.ofc> Hi Rich, Hi all, at Fastweb we're hosting our first OpenStack meetup on 14th December in Milan. We'd highly appreciate your support by tweeting or do some PR for this meetup to reach all stackers around Milan :D. https://www.meetup.com/it-IT/Meetup-OpenStack-Milano/events/234800766/ Many thanks! Amedeo @amedeosalvati -------------- next part -------------- An HTML attachment was scrubbed... URL: From ichavero at redhat.com Wed Nov 9 15:30:39 2016 From: ichavero at redhat.com (Ivan Chavero) Date: Wed, 9 Nov 2016 10:30:39 -0500 (EST) Subject: [rdo-list] [Meeting] RDO meeting - (2016-11-09) Minutes In-Reply-To: <1226670317.7088587.1478705417020.JavaMail.zimbra@redhat.com> Message-ID: <524583245.7088625.1478705439271.JavaMail.zimbra@redhat.com> ============================== #rdo: RDO meeting - 2016-11-09 ============================== Meeting started by imcsk8 at 15:01:30 UTC. The full logs are available at https://meetbot.fedoraproject.org/rdo/2016-11-09/rdo_meeting_-_2016-11-09.2016-11-09-15.01.log.html . Meeting summary --------------- * Updates on Tempest Plugin Entry Separation (imcsk8, 15:04:04) * LINK: https://review.rdoproject.org/r/#/q/topic:tempest-plugin-entrypoint (chandankumar, 15:05:45) * Updated RDO test day schedule: https://www.rdoproject.org/testday/ (imcsk8, 15:07:18) Meeting ended at 15:14:42 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * imcsk8 (18) * chandankumar (13) * rbowen (12) * zodbot (6) * openstack (3) * jruzicka (2) * remix_tj (2) * coolsvap (1) * weshay (0) * dmsimard (0) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot From cems at ebi.ac.uk Wed Nov 9 16:18:01 2016 From: cems at ebi.ac.uk (Charles Short) Date: Wed, 9 Nov 2016 16:18:01 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> Message-ID: Hi, Just some feedback on this thread. I have redeployed several times and have begun to suspect DNS as being the cause for delays (just a guess as the deployment always competes with no obvious errors) I had a look at the local hosts files on the nodes during deployment and can see that lots of them (not all) are incorrectly formatted as they contain '\n'. For example a small part of one hosts file - << \n10.0.7.30 overcloud-novacompute-32.localdomain overcloud-novacompute-32 192.168.0.39 overcloud-novacompute-32.external.localdomain overcloud-novacompute-32.external 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain overcloud-novacompute-32.internalapi 10.35.5.67 overcloud-novacompute-32.storage.localdomain overcloud-novacompute-32.storage 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain overcloud-novacompute-32.storagemgmt 10.0.8.39 overcloud-novacompute-32.tenant.localdomain overcloud-novacompute-32.tenant 192.168.0.39 overcloud-novacompute-32.management.localdomain overcloud-novacompute-32.management 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain overcloud-novacompute-32.ctlplane \n10.0.7.21 overcloud-novacompute-33.localdomain overcloud-novacompute-33 >> I wondered if maybe the image I was using was the issue so I tried the RH OSP9 official image - Same hosts file formatting issues in deployment. Maybe a workaround would be to change nsswitch.conf in the image to look up from DNS first - my Undercloud dnsmasq server - and have this populated with the correct entries from a node (once all nodes are pingable). Charles On 06/11/2016 23:25, Graeme Gillies wrote: > Hi Charles, > > This definitely looks a bit strange to me, as we do deploys around 42 > nodes and it takes around 2 hours to do so, similar to your setup (1G > link for provisoning, bonded 10G for everything else). > > Would it be possible for you to run an sosreport on your undercloud and > provide it somewhere (if you are comfortable doing so). Also, can you > show us the output of > > openstack stack list --nested > > And most importantly, if we can get a fully copy of the output of the > overcloud deploy command, that has timestamps against when ever stack is > created/finished, so we can try and narrow down where all the time is > being spent. > > You note that you have quite a powerful undercloud (294GB of Memory and > 64 cpus), and we have had issues in the past with very powerful > underclouds, because the Openstack components try and tune themselves > around the hardware they are running on and get it wrong for bigger servers. > > Are we able to get an output from "sar" or some other tool you are using > to track cpu and memory usage during the deployment? I'd like to check > those values look sane. > > Thanks in advance, > > Graeme > > On 05/11/16 01:31, Charles Short wrote: >> Hi, >> >> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. >> The 1Gb deployment NIC is not really causing the delay. It is very busy >> for the time the overcloud image is rolled out (the first 30 to 45 mins >> of deployment), but after that (once all the nodes are up and active >> with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on >> average for the rest of the deployment. For info the NICS in the nodes >> for the Overcloud networks are dual bonded 10Gbit. >> >> The deployment I mentioned before (50 nodes) actually completed in 8 >> hours (which is double the time it took for 35 nodes!) >> >> I am in the process of a new 3 controller 59 compute node deployment >> pinning all the nodes as you suggested. The initial overcloud image roll >> out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5 >> hours in and all is running (slowly). It is currently on Step2 (of 5 >> Steps). I would expect this deployment to take 10 hours on current speed. >> >> Regards >> >> Charles >> >> On 04/11/2016 15:17, Justin Kilpatrick wrote: >>> Hey Charles, >>> >>> What sort of issues are you seeing now? How did node pinning work out >>> and did a slow scale up present any more problems? >>> >>> Deployments tend to be disk and network limited, you don't mention >>> what sort of disks your machines have but you do note 1g nics, which >>> are doable but might require some timeout adjustments or other >>> considerations to give everything time to complete. >>> >>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short >> > wrote: >>> >>> Hi, >>> >>> So you are implying that tripleO is not really currently able to >>> roll out large deployments easily as it is is prone to scaling >>> delays/errors? >>> Is the same true for RH OSP9 (out of the box) as this also uses >>> tripleO? I would expect exactly the same scaling issues. But >>> surely OSP9 is designed for large enterprise Openstack installations? >>> So if OSP9 does work well with large deployments, what are the >>> tripleO tweaks that make this work (if any)? >>> >>> Many Thanks >>> >>> Charles >>> >>> On 03/11/2016 13:30, Justin Kilpatrick wrote: >>>> Hey Charles, >>>> >>>> If you want to deploy a large number of machines, I suggest you >>>> deploy a small configuration (maybe 3 controllers 1 compute) and >>>> then run the overcloud deploy command again with 2 computes, so >>>> on and so forth until you reach your full allocation >>>> >>>> Realistically you can probably do a stride of 5 computes each >>>> time, experiment with it a bit, as you get up to the full >>>> allocation of nodes you might run into a race condition bug with >>>> assigning computes to nodes and need to pin nodes (pinning is >>>> adding as an ironic property that overcloud-novacompute-0 goes >>>> here, 1 here, so on and so forth). >>>> >>>> As for actually solving the deployment issues at scale (instead >>>> of this horrible hack) I'm looking into adding some robustness at >>>> the ironic or tripleo level to these operations. It sounds like >>>> you're running more into node assignment issues rather than pxe >>>> issues though. >>>> >>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >>>> >: >>>> >>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short >>> > wrote: >>>> > Some more testing of different amounts of nodes vs time >>>> taken for successful >>>> > deployments - >>>> > >>>> > 3 controller 3 compute = 1 hour >>>> > 3 controller 15 compute = 1 hour >>>> > 3 controller 25 compute = 1 hour 45 mins >>>> > 3 controller 35 compute = 4 hours >>>> >>>> Hello, >>>> >>>> i'm now preparing my deployment of 3+2 nodes. I'll check what you >>>> reported and give you some feedback. >>>> >>>> Luca >>>> >>>> >>>> -- >>>> "E' assurdo impiegare gli uomini di intelligenza eccellente >>>> per fare >>>> calcoli che potrebbero essere affidati a chiunque se si >>>> usassero delle >>>> macchine" >>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) >>>> >>>> "Internet ? la pi? grande biblioteca del mondo. >>>> Ma il problema ? che i libri sono tutti sparsi sul pavimento" >>>> John Allen Paulos, Matematico (1945-vivente) >>>> >>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >>>> > >>>> >>>> _______________________________________________ >>>> rdo-list mailing list >>>> rdo-list at redhat.com >>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>> >>>> >>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>> >>>> >>>> >>> -- >>> Charles Short >>> Cloud Engineer >>> Virtualization and Cloud Team >>> European Bioinformatics Institute (EMBL-EBI) >>> Tel: +44 (0)1223 494205 >>> >>> >> -- >> Charles Short >> Cloud Engineer >> Virtualization and Cloud Team >> European Bioinformatics Institute (EMBL-EBI) >> Tel: +44 (0)1223 494205 >> >> >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list at redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com >> > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From ggillies at redhat.com Thu Nov 10 03:13:44 2016 From: ggillies at redhat.com (Graeme Gillies) Date: Thu, 10 Nov 2016 13:13:44 +1000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> Message-ID: <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> On 10/11/16 02:18, Charles Short wrote: > Hi, > > Just some feedback on this thread. > > I have redeployed several times and have begun to suspect DNS as being > the cause for delays (just a guess as the deployment always competes > with no obvious errors) > I had a look at the local hosts files on the nodes during deployment and > can see that lots of them (not all) are incorrectly formatted as they > contain '\n'. > > For example a small part of one hosts file - > << > \n10.0.7.30 overcloud-novacompute-32.localdomain overcloud-novacompute-32 > 192.168.0.39 overcloud-novacompute-32.external.localdomain > overcloud-novacompute-32.external > 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain > overcloud-novacompute-32.internalapi > 10.35.5.67 overcloud-novacompute-32.storage.localdomain > overcloud-novacompute-32.storage > 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain > overcloud-novacompute-32.storagemgmt > 10.0.8.39 overcloud-novacompute-32.tenant.localdomain > overcloud-novacompute-32.tenant > 192.168.0.39 overcloud-novacompute-32.management.localdomain > overcloud-novacompute-32.management > 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain > overcloud-novacompute-32.ctlplane > \n10.0.7.21 overcloud-novacompute-33.localdomain overcloud-novacompute-33 >>> > > I wondered if maybe the image I was using was the issue so I tried the > RH OSP9 official image - Same hosts file formatting issues in deployment. > Maybe a workaround would be to change nsswitch.conf in the image to look > up from DNS first - my Undercloud dnsmasq server - and have this > populated with the correct entries from a node (once all nodes are > pingable). > > Charles Hi Charles, If you are getting formatting issues in /etc/hosts, it's possible that the templates directory you are using might have problems, especially if it's been edited on windows machines. Are you using unmodified templates from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9 images will not match RDO Newton templates, as RHOS 9 is mitaka, and overcloud images contain puppet modules which must sync with the templates used on the undercloud. If you are using the templates in /usr/share/openstack-tripleo-heat-templates, can you give the output (if any) from rpm -V openstack-tripleo-heat-templates Also perhaps getting a copy of your full overcloud deploy command will help shed some light as well. Thanks in advance, Graeme > > On 06/11/2016 23:25, Graeme Gillies wrote: >> Hi Charles, >> >> This definitely looks a bit strange to me, as we do deploys around 42 >> nodes and it takes around 2 hours to do so, similar to your setup (1G >> link for provisoning, bonded 10G for everything else). >> >> Would it be possible for you to run an sosreport on your undercloud and >> provide it somewhere (if you are comfortable doing so). Also, can you >> show us the output of >> >> openstack stack list --nested >> >> And most importantly, if we can get a fully copy of the output of the >> overcloud deploy command, that has timestamps against when ever stack is >> created/finished, so we can try and narrow down where all the time is >> being spent. >> >> You note that you have quite a powerful undercloud (294GB of Memory and >> 64 cpus), and we have had issues in the past with very powerful >> underclouds, because the Openstack components try and tune themselves >> around the hardware they are running on and get it wrong for bigger >> servers. >> >> Are we able to get an output from "sar" or some other tool you are using >> to track cpu and memory usage during the deployment? I'd like to check >> those values look sane. >> >> Thanks in advance, >> >> Graeme >> >> On 05/11/16 01:31, Charles Short wrote: >>> Hi, >>> >>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. >>> The 1Gb deployment NIC is not really causing the delay. It is very busy >>> for the time the overcloud image is rolled out (the first 30 to 45 mins >>> of deployment), but after that (once all the nodes are up and active >>> with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on >>> average for the rest of the deployment. For info the NICS in the nodes >>> for the Overcloud networks are dual bonded 10Gbit. >>> >>> The deployment I mentioned before (50 nodes) actually completed in 8 >>> hours (which is double the time it took for 35 nodes!) >>> >>> I am in the process of a new 3 controller 59 compute node deployment >>> pinning all the nodes as you suggested. The initial overcloud image roll >>> out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5 >>> hours in and all is running (slowly). It is currently on Step2 (of 5 >>> Steps). I would expect this deployment to take 10 hours on current >>> speed. >>> >>> Regards >>> >>> Charles >>> >>> On 04/11/2016 15:17, Justin Kilpatrick wrote: >>>> Hey Charles, >>>> >>>> What sort of issues are you seeing now? How did node pinning work out >>>> and did a slow scale up present any more problems? >>>> >>>> Deployments tend to be disk and network limited, you don't mention >>>> what sort of disks your machines have but you do note 1g nics, which >>>> are doable but might require some timeout adjustments or other >>>> considerations to give everything time to complete. >>>> >>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short >>> > wrote: >>>> >>>> Hi, >>>> >>>> So you are implying that tripleO is not really currently able to >>>> roll out large deployments easily as it is is prone to scaling >>>> delays/errors? >>>> Is the same true for RH OSP9 (out of the box) as this also uses >>>> tripleO? I would expect exactly the same scaling issues. But >>>> surely OSP9 is designed for large enterprise Openstack >>>> installations? >>>> So if OSP9 does work well with large deployments, what are the >>>> tripleO tweaks that make this work (if any)? >>>> >>>> Many Thanks >>>> >>>> Charles >>>> >>>> On 03/11/2016 13:30, Justin Kilpatrick wrote: >>>>> Hey Charles, >>>>> >>>>> If you want to deploy a large number of machines, I suggest you >>>>> deploy a small configuration (maybe 3 controllers 1 compute) and >>>>> then run the overcloud deploy command again with 2 computes, so >>>>> on and so forth until you reach your full allocation >>>>> >>>>> Realistically you can probably do a stride of 5 computes each >>>>> time, experiment with it a bit, as you get up to the full >>>>> allocation of nodes you might run into a race condition bug with >>>>> assigning computes to nodes and need to pin nodes (pinning is >>>>> adding as an ironic property that overcloud-novacompute-0 goes >>>>> here, 1 here, so on and so forth). >>>>> >>>>> As for actually solving the deployment issues at scale (instead >>>>> of this horrible hack) I'm looking into adding some robustness at >>>>> the ironic or tripleo level to these operations. It sounds like >>>>> you're running more into node assignment issues rather than pxe >>>>> issues though. >>>>> >>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >>>>> >: >>>>> >>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short >>>> > wrote: >>>>> > Some more testing of different amounts of nodes vs time >>>>> taken for successful >>>>> > deployments - >>>>> > >>>>> > 3 controller 3 compute = 1 hour >>>>> > 3 controller 15 compute = 1 hour >>>>> > 3 controller 25 compute = 1 hour 45 mins >>>>> > 3 controller 35 compute = 4 hours >>>>> >>>>> Hello, >>>>> >>>>> i'm now preparing my deployment of 3+2 nodes. I'll check >>>>> what you >>>>> reported and give you some feedback. >>>>> >>>>> Luca >>>>> >>>>> >>>>> -- >>>>> "E' assurdo impiegare gli uomini di intelligenza eccellente >>>>> per fare >>>>> calcoli che potrebbero essere affidati a chiunque se si >>>>> usassero delle >>>>> macchine" >>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico >>>>> (1646-1716) >>>>> >>>>> "Internet ? la pi? grande biblioteca del mondo. >>>>> Ma il problema ? che i libri sono tutti sparsi sul pavimento" >>>>> John Allen Paulos, Matematico (1945-vivente) >>>>> >>>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >>>>> >>>> > >>>>> >>>>> _______________________________________________ >>>>> rdo-list mailing list >>>>> rdo-list at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>> >>>>> >>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>> >>>>> >>>>> >>>> -- >>>> Charles Short >>>> Cloud Engineer >>>> Virtualization and Cloud Team >>>> European Bioinformatics Institute (EMBL-EBI) >>>> Tel: +44 (0)1223 494205 >>>> >>>> >>> -- >>> Charles Short >>> Cloud Engineer >>> Virtualization and Cloud Team >>> European Bioinformatics Institute (EMBL-EBI) >>> Tel: +44 (0)1223 494205 >>> >>> >>> >>> _______________________________________________ >>> rdo-list mailing list >>> rdo-list at redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>> >> > -- Graeme Gillies Principal Systems Administrator Openstack Infrastructure Red Hat Australia From cems at ebi.ac.uk Thu Nov 10 08:20:27 2016 From: cems at ebi.ac.uk (Charles Short) Date: Thu, 10 Nov 2016 08:20:27 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> Message-ID: <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> Hi, Deploy command here http://pastebin.com/xNZXTWPE no output from rpm command. Yes re OSP9 images I was just interested how they behaved early on in the deployment before any puppet errors (cloud init etc). Not a good test, just morbid fascination out of desperation. No Windows involved, and I have not altered the main puppet template directory at all. I am going to try and update the Undercloud to the latest stable, use the provided images and see how that goes. If all else fails I will install OSP9 and consider myself exhausted from all the swimming upstream ;) Charles On 10/11/2016 03:13, Graeme Gillies wrote: > On 10/11/16 02:18, Charles Short wrote: >> Hi, >> >> Just some feedback on this thread. >> >> I have redeployed several times and have begun to suspect DNS as being >> the cause for delays (just a guess as the deployment always competes >> with no obvious errors) >> I had a look at the local hosts files on the nodes during deployment and >> can see that lots of them (not all) are incorrectly formatted as they >> contain '\n'. >> >> For example a small part of one hosts file - >> << >> \n10.0.7.30 overcloud-novacompute-32.localdomain overcloud-novacompute-32 >> 192.168.0.39 overcloud-novacompute-32.external.localdomain >> overcloud-novacompute-32.external >> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain >> overcloud-novacompute-32.internalapi >> 10.35.5.67 overcloud-novacompute-32.storage.localdomain >> overcloud-novacompute-32.storage >> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain >> overcloud-novacompute-32.storagemgmt >> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain >> overcloud-novacompute-32.tenant >> 192.168.0.39 overcloud-novacompute-32.management.localdomain >> overcloud-novacompute-32.management >> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain >> overcloud-novacompute-32.ctlplane >> \n10.0.7.21 overcloud-novacompute-33.localdomain overcloud-novacompute-33 >> I wondered if maybe the image I was using was the issue so I tried the >> RH OSP9 official image - Same hosts file formatting issues in deployment. >> Maybe a workaround would be to change nsswitch.conf in the image to look >> up from DNS first - my Undercloud dnsmasq server - and have this >> populated with the correct entries from a node (once all nodes are >> pingable). >> >> Charles > Hi Charles, > > If you are getting formatting issues in /etc/hosts, it's possible that > the templates directory you are using might have problems, especially if > it's been edited on windows machines. Are you using unmodified templates > from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9 > images will not match RDO Newton templates, as RHOS 9 is mitaka, and > overcloud images contain puppet modules which must sync with the > templates used on the undercloud. > > If you are using the templates in > /usr/share/openstack-tripleo-heat-templates, can you give the output (if > any) from > > rpm -V openstack-tripleo-heat-templates > > Also perhaps getting a copy of your full overcloud deploy command will > help shed some light as well. > > Thanks in advance, > > Graeme > >> On 06/11/2016 23:25, Graeme Gillies wrote: >>> Hi Charles, >>> >>> This definitely looks a bit strange to me, as we do deploys around 42 >>> nodes and it takes around 2 hours to do so, similar to your setup (1G >>> link for provisoning, bonded 10G for everything else). >>> >>> Would it be possible for you to run an sosreport on your undercloud and >>> provide it somewhere (if you are comfortable doing so). Also, can you >>> show us the output of >>> >>> openstack stack list --nested >>> >>> And most importantly, if we can get a fully copy of the output of the >>> overcloud deploy command, that has timestamps against when ever stack is >>> created/finished, so we can try and narrow down where all the time is >>> being spent. >>> >>> You note that you have quite a powerful undercloud (294GB of Memory and >>> 64 cpus), and we have had issues in the past with very powerful >>> underclouds, because the Openstack components try and tune themselves >>> around the hardware they are running on and get it wrong for bigger >>> servers. >>> >>> Are we able to get an output from "sar" or some other tool you are using >>> to track cpu and memory usage during the deployment? I'd like to check >>> those values look sane. >>> >>> Thanks in advance, >>> >>> Graeme >>> >>> On 05/11/16 01:31, Charles Short wrote: >>>> Hi, >>>> >>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. >>>> The 1Gb deployment NIC is not really causing the delay. It is very busy >>>> for the time the overcloud image is rolled out (the first 30 to 45 mins >>>> of deployment), but after that (once all the nodes are up and active >>>> with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on >>>> average for the rest of the deployment. For info the NICS in the nodes >>>> for the Overcloud networks are dual bonded 10Gbit. >>>> >>>> The deployment I mentioned before (50 nodes) actually completed in 8 >>>> hours (which is double the time it took for 35 nodes!) >>>> >>>> I am in the process of a new 3 controller 59 compute node deployment >>>> pinning all the nodes as you suggested. The initial overcloud image roll >>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5 >>>> hours in and all is running (slowly). It is currently on Step2 (of 5 >>>> Steps). I would expect this deployment to take 10 hours on current >>>> speed. >>>> >>>> Regards >>>> >>>> Charles >>>> >>>> On 04/11/2016 15:17, Justin Kilpatrick wrote: >>>>> Hey Charles, >>>>> >>>>> What sort of issues are you seeing now? How did node pinning work out >>>>> and did a slow scale up present any more problems? >>>>> >>>>> Deployments tend to be disk and network limited, you don't mention >>>>> what sort of disks your machines have but you do note 1g nics, which >>>>> are doable but might require some timeout adjustments or other >>>>> considerations to give everything time to complete. >>>>> >>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short >>>> > wrote: >>>>> >>>>> Hi, >>>>> >>>>> So you are implying that tripleO is not really currently able to >>>>> roll out large deployments easily as it is is prone to scaling >>>>> delays/errors? >>>>> Is the same true for RH OSP9 (out of the box) as this also uses >>>>> tripleO? I would expect exactly the same scaling issues. But >>>>> surely OSP9 is designed for large enterprise Openstack >>>>> installations? >>>>> So if OSP9 does work well with large deployments, what are the >>>>> tripleO tweaks that make this work (if any)? >>>>> >>>>> Many Thanks >>>>> >>>>> Charles >>>>> >>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote: >>>>>> Hey Charles, >>>>>> >>>>>> If you want to deploy a large number of machines, I suggest you >>>>>> deploy a small configuration (maybe 3 controllers 1 compute) and >>>>>> then run the overcloud deploy command again with 2 computes, so >>>>>> on and so forth until you reach your full allocation >>>>>> >>>>>> Realistically you can probably do a stride of 5 computes each >>>>>> time, experiment with it a bit, as you get up to the full >>>>>> allocation of nodes you might run into a race condition bug with >>>>>> assigning computes to nodes and need to pin nodes (pinning is >>>>>> adding as an ironic property that overcloud-novacompute-0 goes >>>>>> here, 1 here, so on and so forth). >>>>>> >>>>>> As for actually solving the deployment issues at scale (instead >>>>>> of this horrible hack) I'm looking into adding some robustness at >>>>>> the ironic or tripleo level to these operations. It sounds like >>>>>> you're running more into node assignment issues rather than pxe >>>>>> issues though. >>>>>> >>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >>>>>> >: >>>>>> >>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short >>>>> > wrote: >>>>>> > Some more testing of different amounts of nodes vs time >>>>>> taken for successful >>>>>> > deployments - >>>>>> > >>>>>> > 3 controller 3 compute = 1 hour >>>>>> > 3 controller 15 compute = 1 hour >>>>>> > 3 controller 25 compute = 1 hour 45 mins >>>>>> > 3 controller 35 compute = 4 hours >>>>>> >>>>>> Hello, >>>>>> >>>>>> i'm now preparing my deployment of 3+2 nodes. I'll check >>>>>> what you >>>>>> reported and give you some feedback. >>>>>> >>>>>> Luca >>>>>> >>>>>> >>>>>> -- >>>>>> "E' assurdo impiegare gli uomini di intelligenza eccellente >>>>>> per fare >>>>>> calcoli che potrebbero essere affidati a chiunque se si >>>>>> usassero delle >>>>>> macchine" >>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico >>>>>> (1646-1716) >>>>>> >>>>>> "Internet ? la pi? grande biblioteca del mondo. >>>>>> Ma il problema ? che i libri sono tutti sparsi sul pavimento" >>>>>> John Allen Paulos, Matematico (1945-vivente) >>>>>> >>>>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >>>>>> >>>>> > >>>>>> >>>>>> _______________________________________________ >>>>>> rdo-list mailing list >>>>>> rdo-list at redhat.com >>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>> >>>>>> >>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Charles Short >>>>> Cloud Engineer >>>>> Virtualization and Cloud Team >>>>> European Bioinformatics Institute (EMBL-EBI) >>>>> Tel: +44 (0)1223 494205 >>>>> >>>>> >>>> -- >>>> Charles Short >>>> Cloud Engineer >>>> Virtualization and Cloud Team >>>> European Bioinformatics Institute (EMBL-EBI) >>>> Tel: +44 (0)1223 494205 >>>> >>>> >>>> >>>> _______________________________________________ >>>> rdo-list mailing list >>>> rdo-list at redhat.com >>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>> >>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>> > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From javier.pena at redhat.com Thu Nov 10 09:03:22 2016 From: javier.pena at redhat.com (Javier Pena) Date: Thu, 10 Nov 2016 04:03:22 -0500 (EST) Subject: [rdo-list] [infra][DLRN] Maintenance of master DLRN worker on November 10 In-Reply-To: <468477697.12458963.1478168516004.JavaMail.zimbra@redhat.com> References: <468477697.12458963.1478168516004.JavaMail.zimbra@redhat.com> Message-ID: <961466581.14255967.1478768602781.JavaMail.zimbra@redhat.com> > Hi RDO, > > We are planning to run a maintenance on the centos-master DLRN worker > starting November 10, 9:00 AM UTC time. > Dear all, Maintenance is starting now. We will inform you as soon as it is completed. Javier > This maintenance is required to improve future version transitions between > master and stable branches, and the expected downtime for > https://trunk.rdoproject.org/centos7-master and > https://trunk.rdoproject.org/centos7-ocata is about 1 hour. All other repos > will remain active during the maintenance. > > You can find further details in > https://trello.com/c/ws1EPmwp/390-improve-master-and-stable-branch-handling-in-dlrn-repos > . Please note that we will also update several promotion jobs to match the > new internal user name. > > If you have any questions, please do not hesitate to contact me. > > Regards, > Javier > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From javier.pena at redhat.com Thu Nov 10 10:45:04 2016 From: javier.pena at redhat.com (Javier Pena) Date: Thu, 10 Nov 2016 05:45:04 -0500 (EST) Subject: [rdo-list] [infra][DLRN] Maintenance of master DLRN worker on November 10 In-Reply-To: <961466581.14255967.1478768602781.JavaMail.zimbra@redhat.com> References: <468477697.12458963.1478168516004.JavaMail.zimbra@redhat.com> <961466581.14255967.1478768602781.JavaMail.zimbra@redhat.com> Message-ID: <2060745734.14269319.1478774704496.JavaMail.zimbra@redhat.com> ----- Original Message ----- > > Hi RDO, > > > > We are planning to run a maintenance on the centos-master DLRN worker > > starting November 10, 9:00 AM UTC time. > > > > Dear all, > > Maintenance is starting now. We will inform you as soon as it is completed. > Hello, The maintenance has been completed. If you had any repo errors, please recheck now, and let us know if they still persist. Regards, Javier > Javier > > > This maintenance is required to improve future version transitions between > > master and stable branches, and the expected downtime for > > https://trunk.rdoproject.org/centos7-master and > > https://trunk.rdoproject.org/centos7-ocata is about 1 hour. All other repos > > will remain active during the maintenance. > > > > You can find further details in > > https://trello.com/c/ws1EPmwp/390-improve-master-and-stable-branch-handling-in-dlrn-repos > > . Please note that we will also update several promotion jobs to match the > > new internal user name. > > > > If you have any questions, please do not hesitate to contact me. > > > > Regards, > > Javier > > > > _______________________________________________ > > rdo-list mailing list > > rdo-list at redhat.com > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com > > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From VlOdintsov at croc.ru Thu Nov 10 12:09:32 2016 From: VlOdintsov at croc.ru (Odintsov Vladislav) Date: Thu, 10 Nov 2016 12:09:32 +0000 Subject: [rdo-list] [package-review] addition of networking-vsphere package Message-ID: <1478779773055.81093@croc.ru> Hi RDO, I've done spec for openstack/networking-vsphere and trying to add a new package to rdoinfo. Could someone take a look at review request and propose any thoughts, or what else should I do for this request to be merged? https://review.rdoproject.org/r/#/c/3421/ Also, when RR is merged, who creates a project distgit repo? Thanks. ________________________________ Regards, Vladislav Odintsov -------------- next part -------------- An HTML attachment was scrubbed... URL: From cems at ebi.ac.uk Fri Nov 11 16:48:20 2016 From: cems at ebi.ac.uk (Charles Short) Date: Fri, 11 Nov 2016 16:48:20 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> Message-ID: Update - I updated Undercloud to latest stable Newton release, and used the provided CentOS Overcloud images. I first completed a small test deployment no problem (3 controller 3 compute) . I then deployed again with a larger environment (40 compute, 3 controllers). When the nodes were up and ACTIVE/pingable early in deployment I checked the hosts files. This time no formatting errors. However during deployment there were lots of long pauses and I noticed plenty of these sorts of messages in the nova logs during the pauses - /var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] raise exceptions.ConnectTimeout(msg) /var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] ConnectTimeout: Request to http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f028412977f&device_id=f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed timed out While this was happening I could not use nova from the Undercloud at all- source stackrc nova list ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5) After 2 hours of deployment and only on Step 1 of 5 the deployment fails with - ERROR: Timed out waiting for a reply to message ID 971157a211e549998bb7a6f6e494688b note - I have a timeout value way over 2 hours in the deployment commands (2000) Post failed deployment I still cannot use nova. Looks like the Undercloud is very unhappy (same error as above) The only way I can get the Undercloud working again is to restart all services (restarting nova alone does not work) sudo systemctl restart neutron* sudo systemctl restart openstack* I think I may try OSP9 as I am running out of ideas. Either that or giving Openstack-Ansible a try..... Charles On 10/11/2016 08:20, Charles Short wrote: > Hi, > > Deploy command here > > http://pastebin.com/xNZXTWPE > > no output from rpm command. > > Yes re OSP9 images I was just interested how they behaved early on in > the deployment before any puppet errors (cloud init etc). > Not a good test, just morbid fascination out of desperation. > > No Windows involved, and I have not altered the main puppet template > directory at all. > > I am going to try and update the Undercloud to the latest stable, use > the provided images and see how that goes. > > If all else fails I will install OSP9 and consider myself exhausted > from all the swimming upstream ;) > > Charles > > On 10/11/2016 03:13, Graeme Gillies wrote: >> On 10/11/16 02:18, Charles Short wrote: >>> Hi, >>> >>> Just some feedback on this thread. >>> >>> I have redeployed several times and have begun to suspect DNS as being >>> the cause for delays (just a guess as the deployment always competes >>> with no obvious errors) >>> I had a look at the local hosts files on the nodes during deployment >>> and >>> can see that lots of them (not all) are incorrectly formatted as they >>> contain '\n'. >>> >>> For example a small part of one hosts file - >>> << >>> \n10.0.7.30 overcloud-novacompute-32.localdomain >>> overcloud-novacompute-32 >>> 192.168.0.39 overcloud-novacompute-32.external.localdomain >>> overcloud-novacompute-32.external >>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain >>> overcloud-novacompute-32.internalapi >>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain >>> overcloud-novacompute-32.storage >>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain >>> overcloud-novacompute-32.storagemgmt >>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain >>> overcloud-novacompute-32.tenant >>> 192.168.0.39 overcloud-novacompute-32.management.localdomain >>> overcloud-novacompute-32.management >>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain >>> overcloud-novacompute-32.ctlplane >>> \n10.0.7.21 overcloud-novacompute-33.localdomain >>> overcloud-novacompute-33 >>> I wondered if maybe the image I was using was the issue so I tried the >>> RH OSP9 official image - Same hosts file formatting issues in >>> deployment. >>> Maybe a workaround would be to change nsswitch.conf in the image to >>> look >>> up from DNS first - my Undercloud dnsmasq server - and have this >>> populated with the correct entries from a node (once all nodes are >>> pingable). >>> >>> Charles >> Hi Charles, >> >> If you are getting formatting issues in /etc/hosts, it's possible that >> the templates directory you are using might have problems, especially if >> it's been edited on windows machines. Are you using unmodified templates >> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9 >> images will not match RDO Newton templates, as RHOS 9 is mitaka, and >> overcloud images contain puppet modules which must sync with the >> templates used on the undercloud. >> >> If you are using the templates in >> /usr/share/openstack-tripleo-heat-templates, can you give the output (if >> any) from >> >> rpm -V openstack-tripleo-heat-templates >> >> Also perhaps getting a copy of your full overcloud deploy command will >> help shed some light as well. >> >> Thanks in advance, >> >> Graeme >> >>> On 06/11/2016 23:25, Graeme Gillies wrote: >>>> Hi Charles, >>>> >>>> This definitely looks a bit strange to me, as we do deploys around 42 >>>> nodes and it takes around 2 hours to do so, similar to your setup (1G >>>> link for provisoning, bonded 10G for everything else). >>>> >>>> Would it be possible for you to run an sosreport on your undercloud >>>> and >>>> provide it somewhere (if you are comfortable doing so). Also, can you >>>> show us the output of >>>> >>>> openstack stack list --nested >>>> >>>> And most importantly, if we can get a fully copy of the output of the >>>> overcloud deploy command, that has timestamps against when ever >>>> stack is >>>> created/finished, so we can try and narrow down where all the time is >>>> being spent. >>>> >>>> You note that you have quite a powerful undercloud (294GB of Memory >>>> and >>>> 64 cpus), and we have had issues in the past with very powerful >>>> underclouds, because the Openstack components try and tune themselves >>>> around the hardware they are running on and get it wrong for bigger >>>> servers. >>>> >>>> Are we able to get an output from "sar" or some other tool you are >>>> using >>>> to track cpu and memory usage during the deployment? I'd like to check >>>> those values look sane. >>>> >>>> Thanks in advance, >>>> >>>> Graeme >>>> >>>> On 05/11/16 01:31, Charles Short wrote: >>>>> Hi, >>>>> >>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. >>>>> The 1Gb deployment NIC is not really causing the delay. It is very >>>>> busy >>>>> for the time the overcloud image is rolled out (the first 30 to 45 >>>>> mins >>>>> of deployment), but after that (once all the nodes are up and active >>>>> with an ip address (pingable)) ,the bandwidth is a fraction of >>>>> 1Gbps on >>>>> average for the rest of the deployment. For info the NICS in the >>>>> nodes >>>>> for the Overcloud networks are dual bonded 10Gbit. >>>>> >>>>> The deployment I mentioned before (50 nodes) actually completed in 8 >>>>> hours (which is double the time it took for 35 nodes!) >>>>> >>>>> I am in the process of a new 3 controller 59 compute node deployment >>>>> pinning all the nodes as you suggested. The initial overcloud >>>>> image roll >>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am >>>>> now 45 >>>>> hours in and all is running (slowly). It is currently on Step2 (of 5 >>>>> Steps). I would expect this deployment to take 10 hours on current >>>>> speed. >>>>> >>>>> Regards >>>>> >>>>> Charles >>>>> >>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote: >>>>>> Hey Charles, >>>>>> >>>>>> What sort of issues are you seeing now? How did node pinning work >>>>>> out >>>>>> and did a slow scale up present any more problems? >>>>>> >>>>>> Deployments tend to be disk and network limited, you don't mention >>>>>> what sort of disks your machines have but you do note 1g nics, which >>>>>> are doable but might require some timeout adjustments or other >>>>>> considerations to give everything time to complete. >>>>>> >>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short >>>>> > wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> So you are implying that tripleO is not really currently >>>>>> able to >>>>>> roll out large deployments easily as it is is prone to scaling >>>>>> delays/errors? >>>>>> Is the same true for RH OSP9 (out of the box) as this also >>>>>> uses >>>>>> tripleO? I would expect exactly the same scaling issues. But >>>>>> surely OSP9 is designed for large enterprise Openstack >>>>>> installations? >>>>>> So if OSP9 does work well with large deployments, what are the >>>>>> tripleO tweaks that make this work (if any)? >>>>>> >>>>>> Many Thanks >>>>>> >>>>>> Charles >>>>>> >>>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote: >>>>>>> Hey Charles, >>>>>>> >>>>>>> If you want to deploy a large number of machines, I >>>>>>> suggest you >>>>>>> deploy a small configuration (maybe 3 controllers 1 >>>>>>> compute) and >>>>>>> then run the overcloud deploy command again with 2 >>>>>>> computes, so >>>>>>> on and so forth until you reach your full allocation >>>>>>> >>>>>>> Realistically you can probably do a stride of 5 computes each >>>>>>> time, experiment with it a bit, as you get up to the full >>>>>>> allocation of nodes you might run into a race condition >>>>>>> bug with >>>>>>> assigning computes to nodes and need to pin nodes (pinning is >>>>>>> adding as an ironic property that overcloud-novacompute-0 >>>>>>> goes >>>>>>> here, 1 here, so on and so forth). >>>>>>> >>>>>>> As for actually solving the deployment issues at scale >>>>>>> (instead >>>>>>> of this horrible hack) I'm looking into adding some >>>>>>> robustness at >>>>>>> the ironic or tripleo level to these operations. It sounds >>>>>>> like >>>>>>> you're running more into node assignment issues rather >>>>>>> than pxe >>>>>>> issues though. >>>>>>> >>>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >>>>>>> >>>>>> >: >>>>>>> >>>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short >>>>>>> >>>>>> > wrote: >>>>>>> > Some more testing of different amounts of nodes vs time >>>>>>> taken for successful >>>>>>> > deployments - >>>>>>> > >>>>>>> > 3 controller 3 compute = 1 hour >>>>>>> > 3 controller 15 compute = 1 hour >>>>>>> > 3 controller 25 compute = 1 hour 45 mins >>>>>>> > 3 controller 35 compute = 4 hours >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> i'm now preparing my deployment of 3+2 nodes. I'll check >>>>>>> what you >>>>>>> reported and give you some feedback. >>>>>>> >>>>>>> Luca >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> "E' assurdo impiegare gli uomini di intelligenza >>>>>>> eccellente >>>>>>> per fare >>>>>>> calcoli che potrebbero essere affidati a chiunque se si >>>>>>> usassero delle >>>>>>> macchine" >>>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico >>>>>>> (1646-1716) >>>>>>> >>>>>>> "Internet ? la pi? grande biblioteca del mondo. >>>>>>> Ma il problema ? che i libri sono tutti sparsi sul >>>>>>> pavimento" >>>>>>> John Allen Paulos, Matematico (1945-vivente) >>>>>>> >>>>>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >>>>>>> >>>>>> > >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rdo-list mailing list >>>>>>> rdo-list at redhat.com >>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>> >>>>>>> >>>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Charles Short >>>>>> Cloud Engineer >>>>>> Virtualization and Cloud Team >>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>> Tel: +44 (0)1223 494205 >>>>>> >>>>>> >>>>> -- >>>>> Charles Short >>>>> Cloud Engineer >>>>> Virtualization and Cloud Team >>>>> European Bioinformatics Institute (EMBL-EBI) >>>>> Tel: +44 (0)1223 494205 >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> rdo-list mailing list >>>>> rdo-list at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>> >>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>> >> > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From bderzhavets at hotmail.com Fri Nov 11 18:44:19 2016 From: bderzhavets at hotmail.com (Boris Derzhavets) Date: Fri, 11 Nov 2016 18:44:19 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk>, Message-ID: ________________________________ From: rdo-list-bounces at redhat.com on behalf of Charles Short Sent: Friday, November 11, 2016 7:48 PM To: rdo-list at redhat.com Subject: Re: [rdo-list] [TripleO] Newton large baremetal deployment issues Update - I updated Undercloud to latest stable Newton release, and used the provided CentOS Overcloud images. I first completed a small test deployment no problem (3 controller 3 compute) . I then deployed again with a larger environment (40 compute, 3 controllers). When the nodes were up and ACTIVE/pingable early in deployment I checked the hosts files. This time no formatting errors. However during deployment there were lots of long pauses and I noticed plenty of these sorts of messages in the nova logs during the pauses - /var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] raise exceptions.ConnectTimeout(msg) /var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] ConnectTimeout: Request to http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f028412977f&device_id=f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed timed out While this was happening I could not use nova from the Undercloud at all- source stackrc nova list ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5) After 2 hours of deployment and only on Step 1 of 5 the deployment fails with - ERROR: Timed out waiting for a reply to message ID 971157a211e549998bb7a6f6e494688b note - I have a timeout value way over 2 hours in the deployment commands (2000) Post failed deployment I still cannot use nova. Looks like the Undercloud is very unhappy (same error as above) The only way I can get the Undercloud working again is to restart all services (restarting nova alone does not work) sudo systemctl restart neutron* sudo systemctl restart openstack* What would happen ? # openstack stack delete overcloud Stack deleted cleanly # shutdown -r now Boris. I think I may try OSP9 as I am running out of ideas. Either that or giving Openstack-Ansible a try..... Charles On 10/11/2016 08:20, Charles Short wrote: > Hi, > > Deploy command here > > http://pastebin.com/xNZXTWPE [http://pastebin.com/i/facebook.png] stack1 - Pastebin.com pastebin.com > > no output from rpm command. > > Yes re OSP9 images I was just interested how they behaved early on in > the deployment before any puppet errors (cloud init etc). > Not a good test, just morbid fascination out of desperation. > > No Windows involved, and I have not altered the main puppet template > directory at all. > > I am going to try and update the Undercloud to the latest stable, use > the provided images and see how that goes. > > If all else fails I will install OSP9 and consider myself exhausted > from all the swimming upstream ;) > > Charles > > On 10/11/2016 03:13, Graeme Gillies wrote: >> On 10/11/16 02:18, Charles Short wrote: >>> Hi, >>> >>> Just some feedback on this thread. >>> >>> I have redeployed several times and have begun to suspect DNS as being >>> the cause for delays (just a guess as the deployment always competes >>> with no obvious errors) >>> I had a look at the local hosts files on the nodes during deployment >>> and >>> can see that lots of them (not all) are incorrectly formatted as they >>> contain '\n'. >>> >>> For example a small part of one hosts file - >>> << >>> \n10.0.7.30 overcloud-novacompute-32.localdomain >>> overcloud-novacompute-32 >>> 192.168.0.39 overcloud-novacompute-32.external.localdomain >>> overcloud-novacompute-32.external >>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain >>> overcloud-novacompute-32.internalapi >>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain >>> overcloud-novacompute-32.storage >>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain >>> overcloud-novacompute-32.storagemgmt >>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain >>> overcloud-novacompute-32.tenant >>> 192.168.0.39 overcloud-novacompute-32.management.localdomain >>> overcloud-novacompute-32.management >>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain >>> overcloud-novacompute-32.ctlplane >>> \n10.0.7.21 overcloud-novacompute-33.localdomain >>> overcloud-novacompute-33 >>> I wondered if maybe the image I was using was the issue so I tried the >>> RH OSP9 official image - Same hosts file formatting issues in >>> deployment. >>> Maybe a workaround would be to change nsswitch.conf in the image to >>> look >>> up from DNS first - my Undercloud dnsmasq server - and have this >>> populated with the correct entries from a node (once all nodes are >>> pingable). >>> >>> Charles >> Hi Charles, >> >> If you are getting formatting issues in /etc/hosts, it's possible that >> the templates directory you are using might have problems, especially if >> it's been edited on windows machines. Are you using unmodified templates >> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9 >> images will not match RDO Newton templates, as RHOS 9 is mitaka, and >> overcloud images contain puppet modules which must sync with the >> templates used on the undercloud. >> >> If you are using the templates in >> /usr/share/openstack-tripleo-heat-templates, can you give the output (if >> any) from >> >> rpm -V openstack-tripleo-heat-templates >> >> Also perhaps getting a copy of your full overcloud deploy command will >> help shed some light as well. >> >> Thanks in advance, >> >> Graeme >> >>> On 06/11/2016 23:25, Graeme Gillies wrote: >>>> Hi Charles, >>>> >>>> This definitely looks a bit strange to me, as we do deploys around 42 >>>> nodes and it takes around 2 hours to do so, similar to your setup (1G >>>> link for provisoning, bonded 10G for everything else). >>>> >>>> Would it be possible for you to run an sosreport on your undercloud >>>> and >>>> provide it somewhere (if you are comfortable doing so). Also, can you >>>> show us the output of >>>> >>>> openstack stack list --nested >>>> >>>> And most importantly, if we can get a fully copy of the output of the >>>> overcloud deploy command, that has timestamps against when ever >>>> stack is >>>> created/finished, so we can try and narrow down where all the time is >>>> being spent. >>>> >>>> You note that you have quite a powerful undercloud (294GB of Memory >>>> and >>>> 64 cpus), and we have had issues in the past with very powerful >>>> underclouds, because the Openstack components try and tune themselves >>>> around the hardware they are running on and get it wrong for bigger >>>> servers. >>>> >>>> Are we able to get an output from "sar" or some other tool you are >>>> using >>>> to track cpu and memory usage during the deployment? I'd like to check >>>> those values look sane. >>>> >>>> Thanks in advance, >>>> >>>> Graeme >>>> >>>> On 05/11/16 01:31, Charles Short wrote: >>>>> Hi, >>>>> >>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. >>>>> The 1Gb deployment NIC is not really causing the delay. It is very >>>>> busy >>>>> for the time the overcloud image is rolled out (the first 30 to 45 >>>>> mins >>>>> of deployment), but after that (once all the nodes are up and active >>>>> with an ip address (pingable)) ,the bandwidth is a fraction of >>>>> 1Gbps on >>>>> average for the rest of the deployment. For info the NICS in the >>>>> nodes >>>>> for the Overcloud networks are dual bonded 10Gbit. >>>>> >>>>> The deployment I mentioned before (50 nodes) actually completed in 8 >>>>> hours (which is double the time it took for 35 nodes!) >>>>> >>>>> I am in the process of a new 3 controller 59 compute node deployment >>>>> pinning all the nodes as you suggested. The initial overcloud >>>>> image roll >>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am >>>>> now 45 >>>>> hours in and all is running (slowly). It is currently on Step2 (of 5 >>>>> Steps). I would expect this deployment to take 10 hours on current >>>>> speed. >>>>> >>>>> Regards >>>>> >>>>> Charles >>>>> >>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote: >>>>>> Hey Charles, >>>>>> >>>>>> What sort of issues are you seeing now? How did node pinning work >>>>>> out >>>>>> and did a slow scale up present any more problems? >>>>>> >>>>>> Deployments tend to be disk and network limited, you don't mention >>>>>> what sort of disks your machines have but you do note 1g nics, which >>>>>> are doable but might require some timeout adjustments or other >>>>>> considerations to give everything time to complete. >>>>>> >>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short >>>>> > wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> So you are implying that tripleO is not really currently >>>>>> able to >>>>>> roll out large deployments easily as it is is prone to scaling >>>>>> delays/errors? >>>>>> Is the same true for RH OSP9 (out of the box) as this also >>>>>> uses >>>>>> tripleO? I would expect exactly the same scaling issues. But >>>>>> surely OSP9 is designed for large enterprise Openstack >>>>>> installations? >>>>>> So if OSP9 does work well with large deployments, what are the >>>>>> tripleO tweaks that make this work (if any)? >>>>>> >>>>>> Many Thanks >>>>>> >>>>>> Charles >>>>>> >>>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote: >>>>>>> Hey Charles, >>>>>>> >>>>>>> If you want to deploy a large number of machines, I >>>>>>> suggest you >>>>>>> deploy a small configuration (maybe 3 controllers 1 >>>>>>> compute) and >>>>>>> then run the overcloud deploy command again with 2 >>>>>>> computes, so >>>>>>> on and so forth until you reach your full allocation >>>>>>> >>>>>>> Realistically you can probably do a stride of 5 computes each >>>>>>> time, experiment with it a bit, as you get up to the full >>>>>>> allocation of nodes you might run into a race condition >>>>>>> bug with >>>>>>> assigning computes to nodes and need to pin nodes (pinning is >>>>>>> adding as an ironic property that overcloud-novacompute-0 >>>>>>> goes >>>>>>> here, 1 here, so on and so forth). >>>>>>> >>>>>>> As for actually solving the deployment issues at scale >>>>>>> (instead >>>>>>> of this horrible hack) I'm looking into adding some >>>>>>> robustness at >>>>>>> the ironic or tripleo level to these operations. It sounds >>>>>>> like >>>>>>> you're running more into node assignment issues rather >>>>>>> than pxe >>>>>>> issues though. >>>>>>> >>>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >>>>>>> >>>>>> >: >>>>>>> >>>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short >>>>>>> >>>>>> > wrote: >>>>>>> > Some more testing of different amounts of nodes vs time >>>>>>> taken for successful >>>>>>> > deployments - >>>>>>> > >>>>>>> > 3 controller 3 compute = 1 hour >>>>>>> > 3 controller 15 compute = 1 hour >>>>>>> > 3 controller 25 compute = 1 hour 45 mins >>>>>>> > 3 controller 35 compute = 4 hours >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> i'm now preparing my deployment of 3+2 nodes. I'll check >>>>>>> what you >>>>>>> reported and give you some feedback. >>>>>>> >>>>>>> Luca >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> "E' assurdo impiegare gli uomini di intelligenza >>>>>>> eccellente >>>>>>> per fare >>>>>>> calcoli che potrebbero essere affidati a chiunque se si >>>>>>> usassero delle >>>>>>> macchine" >>>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico >>>>>>> (1646-1716) >>>>>>> >>>>>>> "Internet ? la pi? grande biblioteca del mondo. >>>>>>> Ma il problema ? che i libri sono tutti sparsi sul >>>>>>> pavimento" >>>>>>> John Allen Paulos, Matematico (1945-vivente) >>>>>>> >>>>>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >>>>>>> >>>>>> > >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rdo-list mailing list >>>>>>> rdo-list at redhat.com >>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>> >>>>>>> >>>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Charles Short >>>>>> Cloud Engineer >>>>>> Virtualization and Cloud Team >>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>> Tel: +44 (0)1223 494205 >>>>>> >>>>>> >>>>> -- >>>>> Charles Short >>>>> Cloud Engineer >>>>> Virtualization and Cloud Team >>>>> European Bioinformatics Institute (EMBL-EBI) >>>>> Tel: +44 (0)1223 494205 >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> rdo-list mailing list >>>>> rdo-list at redhat.com >>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>> >>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>> >> > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 _______________________________________________ rdo-list mailing list rdo-list at redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cems at ebi.ac.uk Sat Nov 12 08:06:01 2016 From: cems at ebi.ac.uk (Charles Short) Date: Sat, 12 Nov 2016 08:06:01 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> Message-ID: On 11/11/2016 18:44, Boris Derzhavets wrote: > What would happen ? > # openstack stack delete overcloud This only works after you restart all services. Then if I delete the stack restart and redeploy I get exactly the same deployment issues on larger deployments (small deployments fine) Charles -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ggillies at redhat.com Mon Nov 14 00:20:30 2016 From: ggillies at redhat.com (Graeme Gillies) Date: Mon, 14 Nov 2016 10:20:30 +1000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> Message-ID: On 12/11/16 02:48, Charles Short wrote: > Update - > > I updated Undercloud to latest stable Newton release, and used the > provided CentOS Overcloud images. I first completed a small test > deployment no problem (3 controller 3 compute) . > I then deployed again with a larger environment (40 compute, 3 > controllers). > When the nodes were up and ACTIVE/pingable early in deployment I checked > the hosts files. This time no formatting errors. > > However during deployment there were lots of long pauses and I noticed > plenty of these sorts of messages in the nova logs during the pauses - > > /var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR > nova.compute.manager [instance: > f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] raise > exceptions.ConnectTimeout(msg) > /var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR > nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] > ConnectTimeout: Request to > http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f028412977f&device_id=f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed > timed out > > While this was happening I could not use nova from the Undercloud at all- > > source stackrc > nova list > ERROR (ClientException): The server has either erred or is incapable of > performing the requested operation. (HTTP 500) (Request-ID: > req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5) > > After 2 hours of deployment and only on Step 1 of 5 the deployment fails > with - > > ERROR: Timed out waiting for a reply to message ID > 971157a211e549998bb7a6f6e494688b > > note - I have a timeout value way over 2 hours in the deployment > commands (2000) > > Post failed deployment I still cannot use nova. Looks like the > Undercloud is very unhappy (same error as above) > > The only way I can get the Undercloud working again is to restart all > services (restarting nova alone does not work) > sudo systemctl restart neutron* > sudo systemctl restart openstack* > > I think I may try OSP9 as I am running out of ideas. Either that or > giving Openstack-Ansible a try..... > > > > Charles So the symptoms you are showing me above almost definitely leads me to believe that neutron-server failed on the undercloud, which would explain why the deploy and nova failed to work. It could have failed before or during the deploy. We regularly see instances where neutron-server times out upon system boot (takes slightly longer to start than systemd expects), so we need to start it manually. To be clear, The undercloud has been installed using this repo http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-newton-tested/ Which overcloud images are you using? I'm not seeing any provided in that repo, and I just want to make sure the undercloud and overcloud packages match (as the tripleo-heat-templates package on the undercloud has to align with the openstack-puppet-modules package on the overcloud iamges). Also, is it possible to get a copy of all the neutron-server log from the undercloud? If we can understand why neutron-server failed, that is the first step towards getting a working deployment. It would be great if we could get a full sosreport with all the system logs, to check for other errors. I'm assuming there were no problems with the 'openstack undercloud install' process? Regards, Graeme > > > On 10/11/2016 08:20, Charles Short wrote: >> Hi, >> >> Deploy command here >> >> http://pastebin.com/xNZXTWPE >> >> no output from rpm command. >> >> Yes re OSP9 images I was just interested how they behaved early on in >> the deployment before any puppet errors (cloud init etc). >> Not a good test, just morbid fascination out of desperation. >> >> No Windows involved, and I have not altered the main puppet template >> directory at all. >> >> I am going to try and update the Undercloud to the latest stable, use >> the provided images and see how that goes. >> >> If all else fails I will install OSP9 and consider myself exhausted >> from all the swimming upstream ;) >> >> Charles >> >> On 10/11/2016 03:13, Graeme Gillies wrote: >>> On 10/11/16 02:18, Charles Short wrote: >>>> Hi, >>>> >>>> Just some feedback on this thread. >>>> >>>> I have redeployed several times and have begun to suspect DNS as being >>>> the cause for delays (just a guess as the deployment always competes >>>> with no obvious errors) >>>> I had a look at the local hosts files on the nodes during deployment >>>> and >>>> can see that lots of them (not all) are incorrectly formatted as they >>>> contain '\n'. >>>> >>>> For example a small part of one hosts file - >>>> << >>>> \n10.0.7.30 overcloud-novacompute-32.localdomain >>>> overcloud-novacompute-32 >>>> 192.168.0.39 overcloud-novacompute-32.external.localdomain >>>> overcloud-novacompute-32.external >>>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain >>>> overcloud-novacompute-32.internalapi >>>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain >>>> overcloud-novacompute-32.storage >>>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain >>>> overcloud-novacompute-32.storagemgmt >>>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain >>>> overcloud-novacompute-32.tenant >>>> 192.168.0.39 overcloud-novacompute-32.management.localdomain >>>> overcloud-novacompute-32.management >>>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain >>>> overcloud-novacompute-32.ctlplane >>>> \n10.0.7.21 overcloud-novacompute-33.localdomain >>>> overcloud-novacompute-33 >>>> I wondered if maybe the image I was using was the issue so I tried the >>>> RH OSP9 official image - Same hosts file formatting issues in >>>> deployment. >>>> Maybe a workaround would be to change nsswitch.conf in the image to >>>> look >>>> up from DNS first - my Undercloud dnsmasq server - and have this >>>> populated with the correct entries from a node (once all nodes are >>>> pingable). >>>> >>>> Charles >>> Hi Charles, >>> >>> If you are getting formatting issues in /etc/hosts, it's possible that >>> the templates directory you are using might have problems, especially if >>> it's been edited on windows machines. Are you using unmodified templates >>> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9 >>> images will not match RDO Newton templates, as RHOS 9 is mitaka, and >>> overcloud images contain puppet modules which must sync with the >>> templates used on the undercloud. >>> >>> If you are using the templates in >>> /usr/share/openstack-tripleo-heat-templates, can you give the output (if >>> any) from >>> >>> rpm -V openstack-tripleo-heat-templates >>> >>> Also perhaps getting a copy of your full overcloud deploy command will >>> help shed some light as well. >>> >>> Thanks in advance, >>> >>> Graeme >>> >>>> On 06/11/2016 23:25, Graeme Gillies wrote: >>>>> Hi Charles, >>>>> >>>>> This definitely looks a bit strange to me, as we do deploys around 42 >>>>> nodes and it takes around 2 hours to do so, similar to your setup (1G >>>>> link for provisoning, bonded 10G for everything else). >>>>> >>>>> Would it be possible for you to run an sosreport on your undercloud >>>>> and >>>>> provide it somewhere (if you are comfortable doing so). Also, can you >>>>> show us the output of >>>>> >>>>> openstack stack list --nested >>>>> >>>>> And most importantly, if we can get a fully copy of the output of the >>>>> overcloud deploy command, that has timestamps against when ever >>>>> stack is >>>>> created/finished, so we can try and narrow down where all the time is >>>>> being spent. >>>>> >>>>> You note that you have quite a powerful undercloud (294GB of Memory >>>>> and >>>>> 64 cpus), and we have had issues in the past with very powerful >>>>> underclouds, because the Openstack components try and tune themselves >>>>> around the hardware they are running on and get it wrong for bigger >>>>> servers. >>>>> >>>>> Are we able to get an output from "sar" or some other tool you are >>>>> using >>>>> to track cpu and memory usage during the deployment? I'd like to check >>>>> those values look sane. >>>>> >>>>> Thanks in advance, >>>>> >>>>> Graeme >>>>> >>>>> On 05/11/16 01:31, Charles Short wrote: >>>>>> Hi, >>>>>> >>>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD. >>>>>> The 1Gb deployment NIC is not really causing the delay. It is very >>>>>> busy >>>>>> for the time the overcloud image is rolled out (the first 30 to 45 >>>>>> mins >>>>>> of deployment), but after that (once all the nodes are up and active >>>>>> with an ip address (pingable)) ,the bandwidth is a fraction of >>>>>> 1Gbps on >>>>>> average for the rest of the deployment. For info the NICS in the >>>>>> nodes >>>>>> for the Overcloud networks are dual bonded 10Gbit. >>>>>> >>>>>> The deployment I mentioned before (50 nodes) actually completed in 8 >>>>>> hours (which is double the time it took for 35 nodes!) >>>>>> >>>>>> I am in the process of a new 3 controller 59 compute node deployment >>>>>> pinning all the nodes as you suggested. The initial overcloud >>>>>> image roll >>>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am >>>>>> now 45 >>>>>> hours in and all is running (slowly). It is currently on Step2 (of 5 >>>>>> Steps). I would expect this deployment to take 10 hours on current >>>>>> speed. >>>>>> >>>>>> Regards >>>>>> >>>>>> Charles >>>>>> >>>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote: >>>>>>> Hey Charles, >>>>>>> >>>>>>> What sort of issues are you seeing now? How did node pinning work >>>>>>> out >>>>>>> and did a slow scale up present any more problems? >>>>>>> >>>>>>> Deployments tend to be disk and network limited, you don't mention >>>>>>> what sort of disks your machines have but you do note 1g nics, which >>>>>>> are doable but might require some timeout adjustments or other >>>>>>> considerations to give everything time to complete. >>>>>>> >>>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short >>>>>> > wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> So you are implying that tripleO is not really currently >>>>>>> able to >>>>>>> roll out large deployments easily as it is is prone to scaling >>>>>>> delays/errors? >>>>>>> Is the same true for RH OSP9 (out of the box) as this also >>>>>>> uses >>>>>>> tripleO? I would expect exactly the same scaling issues. But >>>>>>> surely OSP9 is designed for large enterprise Openstack >>>>>>> installations? >>>>>>> So if OSP9 does work well with large deployments, what are the >>>>>>> tripleO tweaks that make this work (if any)? >>>>>>> >>>>>>> Many Thanks >>>>>>> >>>>>>> Charles >>>>>>> >>>>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote: >>>>>>>> Hey Charles, >>>>>>>> >>>>>>>> If you want to deploy a large number of machines, I >>>>>>>> suggest you >>>>>>>> deploy a small configuration (maybe 3 controllers 1 >>>>>>>> compute) and >>>>>>>> then run the overcloud deploy command again with 2 >>>>>>>> computes, so >>>>>>>> on and so forth until you reach your full allocation >>>>>>>> >>>>>>>> Realistically you can probably do a stride of 5 computes each >>>>>>>> time, experiment with it a bit, as you get up to the full >>>>>>>> allocation of nodes you might run into a race condition >>>>>>>> bug with >>>>>>>> assigning computes to nodes and need to pin nodes (pinning is >>>>>>>> adding as an ironic property that overcloud-novacompute-0 >>>>>>>> goes >>>>>>>> here, 1 here, so on and so forth). >>>>>>>> >>>>>>>> As for actually solving the deployment issues at scale >>>>>>>> (instead >>>>>>>> of this horrible hack) I'm looking into adding some >>>>>>>> robustness at >>>>>>>> the ironic or tripleo level to these operations. It sounds >>>>>>>> like >>>>>>>> you're running more into node assignment issues rather >>>>>>>> than pxe >>>>>>>> issues though. >>>>>>>> >>>>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto >>>>>>>> >>>>>>> >: >>>>>>>> >>>>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short >>>>>>>> >>>>>>> > wrote: >>>>>>>> > Some more testing of different amounts of nodes vs time >>>>>>>> taken for successful >>>>>>>> > deployments - >>>>>>>> > >>>>>>>> > 3 controller 3 compute = 1 hour >>>>>>>> > 3 controller 15 compute = 1 hour >>>>>>>> > 3 controller 25 compute = 1 hour 45 mins >>>>>>>> > 3 controller 35 compute = 4 hours >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> i'm now preparing my deployment of 3+2 nodes. I'll check >>>>>>>> what you >>>>>>>> reported and give you some feedback. >>>>>>>> >>>>>>>> Luca >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> "E' assurdo impiegare gli uomini di intelligenza >>>>>>>> eccellente >>>>>>>> per fare >>>>>>>> calcoli che potrebbero essere affidati a chiunque se si >>>>>>>> usassero delle >>>>>>>> macchine" >>>>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico >>>>>>>> (1646-1716) >>>>>>>> >>>>>>>> "Internet ? la pi? grande biblioteca del mondo. >>>>>>>> Ma il problema ? che i libri sono tutti sparsi sul >>>>>>>> pavimento" >>>>>>>> John Allen Paulos, Matematico (1945-vivente) >>>>>>>> >>>>>>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rdo-list mailing list >>>>>>>> rdo-list at redhat.com >>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>>> >>>>>>>> >>>>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Charles Short >>>>>>> Cloud Engineer >>>>>>> Virtualization and Cloud Team >>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>> Tel: +44 (0)1223 494205 >>>>>>> >>>>>>> >>>>>> -- >>>>>> Charles Short >>>>>> Cloud Engineer >>>>>> Virtualization and Cloud Team >>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>> Tel: +44 (0)1223 494205 >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rdo-list mailing list >>>>>> rdo-list at redhat.com >>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>> >>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com >>>>>> >>> >> > -- Graeme Gillies Principal Systems Administrator Openstack Infrastructure Red Hat Australia From cems at ebi.ac.uk Mon Nov 14 14:08:02 2016 From: cems at ebi.ac.uk (Charles Short) Date: Mon, 14 Nov 2016 14:08:02 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> References: <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> Message-ID: <960f51b9-8aaf-2c10-93bb-08a1d50576d4@ebi.ac.uk> Hi Graeme, Thanks for the reply. I used these images - http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/delorean/ I installed the stable repo following the documentation here - http://docs.openstack.org/developer/tripleo-docs/installation/installation.html for example - sudo curl -L -o /etc/yum.repos.d/delorean-newton.repo https://trunk.rdoproject.org/centos7-newton/current/delorean.repo sudo curl -L -o /etc/yum.repos.d/delorean-deps-newton.repo http://trunk.rdoproject.org/centos7-newton/delorean-deps.repo The difficulty I am having is that when I test with a small deployment all works fine. So you would assume just adding more compute nodes would not be an issue. Testing this is painful due to the time it takes for a large deployment to fail. It seems to be only scale that is the issue. I will try and get you some logs Regards Charles > So the symptoms you are showing me above almost definitely leads me to > believe that neutron-server failed on the undercloud, which would > explain why the deploy and nova failed to work. It could have failed > before or during the deploy. We regularly see instances where > neutron-server times out upon system boot (takes slightly longer to > start than systemd expects), so we need to start it manually. > > To be clear, The undercloud has been installed using this repo > > http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-newton-tested/ > > Which overcloud images are you using? I'm not seeing any provided in > that repo, and I just want to make sure the undercloud and overcloud > packages match (as the tripleo-heat-templates package on the undercloud > has to align with the openstack-puppet-modules package on the overcloud > iamges). > > Also, is it possible to get a copy of all the neutron-server log from > the undercloud? If we can understand why neutron-server failed, that is > the first step towards getting a working deployment. > > It would be great if we could get a full sosreport with all the system > logs, to check for other errors. I'm assuming there were no problems > with the 'openstack undercloud install' process? > > Regards, > > Graeme > From hguemar at fedoraproject.org Mon Nov 14 15:00:03 2016 From: hguemar at fedoraproject.org (hguemar at fedoraproject.org) Date: Mon, 14 Nov 2016 15:00:03 +0000 (UTC) Subject: [rdo-list] [Fedocal] Reminder meeting : RDO meeting Message-ID: <20161114150003.46CB460A3FDC@fedocal02.phx2.fedoraproject.org> Dear all, You are kindly invited to the meeting: RDO meeting on 2016-11-16 from 15:00:00 to 16:00:00 UTC At rdo at irc.freenode.net The meeting will be about: RDO IRC meeting [Agenda at https://etherpad.openstack.org/p/RDO-Meeting ](https://etherpad.openstack.org/p/RDO-Meeting) Every Wednesday on #rdo on Freenode IRC Source: https://apps.fedoraproject.org/calendar/meeting/2017/ From hrybacki at redhat.com Tue Nov 15 18:46:05 2016 From: hrybacki at redhat.com (Harry Rybacki) Date: Tue, 15 Nov 2016 13:46:05 -0500 Subject: [rdo-list] RDO-Infra Weekly Scrum Recap: Message-ID: Greetings All, Links to the RDO-Infra team's scrum etherpad[1] and video recording[2] of the meeting are below. Highlights: - Move to combine tripleo-quickstart-extra roles into single repo - Goal to perform merge early next week after Attila returns from PTO - Opting for flat hierarchy (single role per repo breakdown) - Need baremetal and ovb specific cores - Moving virtual worikloads from bm to cloud on cico is delayed - dmsimard is working on a infra/tooling roadmap for ocata[3] - Liberty hits EOL on Thursday (17Nov2016) [1] - https://review.rdoproject.org/etherpad/p/rdo-infra-scrum [2] - https://bluejeans.com/s/UftmN/ [3] - https://review.rdoproject.org/etherpad/p/infra-ocata /R Harry Rybacki From mbultel at redhat.com Tue Nov 15 22:22:05 2016 From: mbultel at redhat.com (mathieu bultel) Date: Tue, 15 Nov 2016 23:22:05 +0100 Subject: [rdo-list] RDO-Infra Weekly Scrum Recap: In-Reply-To: References: Message-ID: <92d020dc-d630-8621-18ea-779e9d454c96@redhat.com> Hi Harry, Thank you for the recap. I have just few questions/concerns about the following: * review role merging code: https://review.openstack.org/396694 o ---> It missing the tripleo-backup role (https://github.com/redhat-openstack/ansible-role-tripleo-backup) * plan is to run this merge script on Monday Nov. 21, and push the history to the upstream repo o ---> does that means that it will keep the commit history and the committer ids ? I think both a really important * Who has permissions to merge in the new review.openstack.org repo? o ---> I'm afraid about this part, and it's mostly my cons about one big repo. Currently I'm mostly the only one who commit on the overcloud-upgrade role, which needs sometimes some tricky fixes or immediate workarounds. The current situation is ideal to me so I'm afraid to have to loose this "cool" flexibility by waiting for 2 cores to vote on probably something that are really "upgrade" specific and so hard-to-review (s/hard/impossible/ ?). And it's probably the case for all the special roles, which are really features oriented. So did you guys plan to add more cores, and if yes, who ? Afaik today, there is only 4 oooq-cores, which is really weak for all the work that is done on all of those roles... On 11/15/2016 07:46 PM, Harry Rybacki wrote: > Greetings All, > > Links to the RDO-Infra team's scrum etherpad[1] and video recording[2] > of the meeting are below. > > Highlights: > - Move to combine tripleo-quickstart-extra roles into single repo > - Goal to perform merge early next week after Attila returns from PTO > - Opting for flat hierarchy (single role per repo breakdown) > - Need baremetal and ovb specific cores > - Moving virtual worikloads from bm to cloud on cico is delayed > - dmsimard is working on a infra/tooling roadmap for ocata[3] > - Liberty hits EOL on Thursday (17Nov2016) > > [1] - https://review.rdoproject.org/etherpad/p/rdo-infra-scrum > [2] - https://bluejeans.com/s/UftmN/ > [3] - https://review.rdoproject.org/etherpad/p/infra-ocata > > /R > > Harry Rybacki > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cems at ebi.ac.uk Tue Nov 15 22:39:19 2016 From: cems at ebi.ac.uk (Charles Short) Date: Tue, 15 Nov 2016 22:39:19 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> Message-ID: Hi, So I have finally tried OSP9 and here are the results - 3 Controllers 40 compute - 1 hours 20 mins to deploy. This is much more the sort of deployment time I was expecting :) I then tried TripleO Newton Stable again with 3 Controllers 40 Compute - 4 hours and counting..... The two deployment scripts (for OSP9 and TripleO Newton) were pretty much identical (allowing for any changes between releases) During the OSP9 deployment I could use nova list to list the nodes. The Undercloud API access was in general very responsive. During the TripleO Newton deployment 'nova list' hangs - ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) Undercloud API access was very sluggish. I noticed Keystone was stuck at 140% for most of the deployment (albeit multi threaded) which is not the case for OSP9. I know it is hard to compare two releases, but the difference is enormous. I will stick with OSP9 for now as this for me works properly out of the box for large deployments. Charles On 14/11/2016 09:01, Charles Short wrote: > Hi Graeme, > > Thanks for the reply. > > I used these images - > > http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/delorean/ > > > I installed the stable repo following the documentation here - > > http://docs.openstack.org/developer/tripleo-docs/installation/installation.html > > > for example - > > sudo curl -L -o /etc/yum.repos.d/delorean-newton.repo > https://trunk.rdoproject.org/centos7-newton/current/delorean.repo > > sudo curl -L -o /etc/yum.repos.d/delorean-deps-newton.repo > http://trunk.rdoproject.org/centos7-newton/delorean-deps.repo > > > The difficulty I am having is that when I test with a small deployment > all works fine. So you would assume just adding more compute nodes > would not be an issue. > Testing this is painful due to the time it takes for a large > deployment to fail. It seems to be only scale that is the issue. > > I will try and get you some logs > > Regards > > Charles > > > >> So the symptoms you are showing me above almost definitely leads me to >> believe that neutron-server failed on the undercloud, which would >> explain why the deploy and nova failed to work. It could have failed >> before or during the deploy. We regularly see instances where >> neutron-server times out upon system boot (takes slightly longer to >> start than systemd expects), so we need to start it manually. >> >> To be clear, The undercloud has been installed using this repo >> >> http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-newton-tested/ >> >> >> Which overcloud images are you using? I'm not seeing any provided in >> that repo, and I just want to make sure the undercloud and overcloud >> packages match (as the tripleo-heat-templates package on the undercloud >> has to align with the openstack-puppet-modules package on the overcloud >> iamges). >> >> Also, is it possible to get a copy of all the neutron-server log from >> the undercloud? If we can understand why neutron-server failed, that is >> the first step towards getting a working deployment. >> >> It would be great if we could get a full sosreport with all the system >> logs, to check for other errors. I'm assuming there were no problems >> with the 'openstack undercloud install' process? >> >> Regards, >> >> Graeme >> > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From ggillies at redhat.com Tue Nov 15 23:40:30 2016 From: ggillies at redhat.com (Graeme Gillies) Date: Wed, 16 Nov 2016 09:40:30 +1000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0339b3ca-7da8-d601-6df7-42f9b16b7b40@ebi.ac.uk> <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> Message-ID: <7194aed5-e1bc-f434-5f74-a626214125d4@redhat.com> On 16/11/16 08:39, Charles Short wrote: > Hi, > > So I have finally tried OSP9 and here are the results - > > 3 Controllers 40 compute - 1 hours 20 mins to deploy. > > This is much more the sort of deployment time I was expecting :) > > I then tried TripleO Newton Stable again with 3 Controllers 40 Compute - > > 4 hours and counting..... > > The two deployment scripts (for OSP9 and TripleO Newton) were pretty > much identical (allowing for any changes between releases) > > During the OSP9 deployment I could use nova list to list the nodes. The > Undercloud API access was in general very responsive. > > During the TripleO Newton deployment 'nova list' hangs - > ERROR (ClientException): The server has either erred or is incapable of > performing the requested operation. (HTTP 500) > Undercloud API access was very sluggish. > I noticed Keystone was stuck at 140% for most of the deployment (albeit > multi threaded) which is not the case for OSP9. > > I know it is hard to compare two releases, but the difference is enormous. > I will stick with OSP9 for now as this for me works properly out of the > box for large deployments. > > Charles Hi Charles, Thanks for letting us know about your results, this obviously narrows it down to an issue introduced in newton timeframe (OSP 9 is mitaka). If you are still able to provide any logs from the undercloud that failed/took ages for the update that would be good, as any issue like this I would like to understand before it hits other users. However, if you haven't got the time/inclination at this stage, I understand. I'll keep an eye out for it affecting other users. Regards, Graeme > > On 14/11/2016 09:01, Charles Short wrote: >> Hi Graeme, >> >> Thanks for the reply. >> >> I used these images - >> >> http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/delorean/ >> >> >> I installed the stable repo following the documentation here - >> >> http://docs.openstack.org/developer/tripleo-docs/installation/installation.html >> >> >> for example - >> >> sudo curl -L -o /etc/yum.repos.d/delorean-newton.repo >> https://trunk.rdoproject.org/centos7-newton/current/delorean.repo >> >> sudo curl -L -o /etc/yum.repos.d/delorean-deps-newton.repo >> http://trunk.rdoproject.org/centos7-newton/delorean-deps.repo >> >> >> The difficulty I am having is that when I test with a small deployment >> all works fine. So you would assume just adding more compute nodes >> would not be an issue. >> Testing this is painful due to the time it takes for a large >> deployment to fail. It seems to be only scale that is the issue. >> >> I will try and get you some logs >> >> Regards >> >> Charles >> >> >> >>> So the symptoms you are showing me above almost definitely leads me to >>> believe that neutron-server failed on the undercloud, which would >>> explain why the deploy and nova failed to work. It could have failed >>> before or during the deploy. We regularly see instances where >>> neutron-server times out upon system boot (takes slightly longer to >>> start than systemd expects), so we need to start it manually. >>> >>> To be clear, The undercloud has been installed using this repo >>> >>> http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-newton-tested/ >>> >>> >>> Which overcloud images are you using? I'm not seeing any provided in >>> that repo, and I just want to make sure the undercloud and overcloud >>> packages match (as the tripleo-heat-templates package on the undercloud >>> has to align with the openstack-puppet-modules package on the overcloud >>> iamges). >>> >>> Also, is it possible to get a copy of all the neutron-server log from >>> the undercloud? If we can understand why neutron-server failed, that is >>> the first step towards getting a working deployment. >>> >>> It would be great if we could get a full sosreport with all the system >>> logs, to check for other errors. I'm assuming there were no problems >>> with the 'openstack undercloud install' process? >>> >>> Regards, >>> >>> Graeme >>> >> > -- Graeme Gillies Principal Systems Administrator Openstack Infrastructure Red Hat Australia From rafi.fellert at hpe.com Wed Nov 16 10:52:05 2016 From: rafi.fellert at hpe.com (Fellert, Rafi) Date: Wed, 16 Nov 2016 10:52:05 +0000 Subject: [rdo-list] RDO: opendaylight (ODL) packstack support Message-ID: Hi, 1. I'm a new subscriber to RDO-list 2. I want to run a demo in which ODL is installed on centos-7. 3. Is there a way/version in which the packstack supports ODL installation? Thanks, Rafi. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hguemar at fedoraproject.org Wed Nov 16 11:33:42 2016 From: hguemar at fedoraproject.org (=?UTF-8?Q?Ha=C3=AFkel?=) Date: Wed, 16 Nov 2016 12:33:42 +0100 Subject: [rdo-list] RDO: opendaylight (ODL) packstack support In-Reply-To: References: Message-ID: 2016-11-16 11:52 GMT+01:00 Fellert, Rafi : > Hi, > > 1. I?m a new subscriber to RDO-list > Welcome Rafi! > 2. I want to run a demo in which ODL is installed on centos-7. > Great to hear! > 3. Is there a way/version in which the packstack supports ODL > installation? > No, packstack is in maintenance mode, we do accept patches that implements new features though. I suggest that you look at OPNFV installer that is based on TripleO https://www.opnfv.org/opnfv-colorado-tripleo-users-20 Regards, H. > > > Thanks, Rafi. > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From trown at redhat.com Wed Nov 16 14:41:56 2016 From: trown at redhat.com (John Trowbridge) Date: Wed, 16 Nov 2016 09:41:56 -0500 Subject: [rdo-list] RDO-Infra Weekly Scrum Recap: In-Reply-To: <92d020dc-d630-8621-18ea-779e9d454c96@redhat.com> References: <92d020dc-d630-8621-18ea-779e9d454c96@redhat.com> Message-ID: <077bde06-34c5-fd63-349e-9746671dd86d@redhat.com> On 11/15/2016 05:22 PM, mathieu bultel wrote: > Hi Harry, > > Thank you for the recap. > I have just few questions/concerns about the following: > > * review role merging code: https://review.openstack.org/396694 > o ---> It missing the tripleo-backup role > (https://github.com/redhat-openstack/ansible-role-tripleo-backup) > The initial role merging is about getting feature parity with tripleo-ci into the new repo. We can add more roles after we get CI switched over though. > * plan is to run this merge script on Monday Nov. 21, and push the > history to the upstream repo > o ---> does that means that it will keep the commit history and > the committer ids ? I think both a really important > Yes, Attila's merge script is exactly designed for the purpose of keeping attribution when moving the code. > * Who has permissions to merge in the new review.openstack.org repo? > o ---> I'm afraid about this part, and it's mostly my cons about > one big repo. Currently I'm mostly the only one who commit on > the overcloud-upgrade role, which needs sometimes some tricky > fixes or immediate workarounds. The current situation is ideal > to me so I'm afraid to have to loose this "cool" flexibility by > waiting for 2 cores to vote on probably something that are > really "upgrade" specific and so hard-to-review > (s/hard/impossible/ ?). And it's probably the case for all the > special roles, which are really features oriented. So did you > guys plan to add more cores, and if yes, who ? Afaik today, > there is only 4 oooq-cores, which is really weak for all the > work that is done on all of those roles... > This is the price of upstream development to some degree. That said, we will need to add some more cores to tripleo-quickstart group, and I think it could make sense to have a policy similar to tripleo where cores are given +2 everywhere, but trusted not to +2 things they don't understand. This would allow us to give core to "domain experts" such as yourself and a few others. Note, the 2x +2 policy would still be in place, so the loss of "flexibility" will still be there. It is also worth remembering that every tripleo core is also core on tripleo-quickstart, and many of those folks do tripleo-ci reviews. So as we replace tripleo-ci with tripleo-quickstart, we will also get more reviewers for "free". > > On 11/15/2016 07:46 PM, Harry Rybacki wrote: >> Greetings All, >> >> Links to the RDO-Infra team's scrum etherpad[1] and video recording[2] >> of the meeting are below. >> >> Highlights: >> - Move to combine tripleo-quickstart-extra roles into single repo >> - Goal to perform merge early next week after Attila returns from PTO >> - Opting for flat hierarchy (single role per repo breakdown) >> - Need baremetal and ovb specific cores >> - Moving virtual worikloads from bm to cloud on cico is delayed >> - dmsimard is working on a infra/tooling roadmap for ocata[3] >> - Liberty hits EOL on Thursday (17Nov2016) >> >> [1] - https://review.rdoproject.org/etherpad/p/rdo-infra-scrum >> [2] - https://bluejeans.com/s/UftmN/ >> [3] - https://review.rdoproject.org/etherpad/p/infra-ocata >> >> /R >> >> Harry Rybacki >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list at redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com > > > > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From chkumar246 at gmail.com Wed Nov 16 15:48:32 2016 From: chkumar246 at gmail.com (Chandan kumar) Date: Wed, 16 Nov 2016 21:18:32 +0530 Subject: [rdo-list] [Meeting] RDO meeting- (2016-11-16) Minutes Message-ID: ============================== #rdo: RDO meeting - 2016-11-16 ============================== Meeting started by chandankumar at 15:00:59 UTC. The full logs are available at https://meetbot.fedoraproject.org/rdo/2016-11-16/rdo_meeting_-_2016-11-16.2016-11-16-15.00.log.html . Meeting summary --------------- * roll call (chandankumar, 15:01:07) * Liberty EOL : 2016-11-17 (chandankumar, 15:04:10) * LINK: https://etherpad.openstack.org/p/RDO-Meeting (chandankumar, 15:05:13) * LINK: https://trello.com/c/fV69VODx/165-rdo-release-eol (chandankumar, 15:06:25) * LINK: https://etherpad.openstack.org/p/DLRN-new-RDO-release (chandankumar, 15:14:33) * ACTION: chandankumar to start sending BZ reports to rdo-list (chandankumar, 15:18:04) * ACTION: jpena to remote Liberty worker from DLRN servers (jpena, 15:19:06) * ACTION: apevec practice EOL vault.centos.org move on Kilo (not EOLed properly yet!) (apevec, 15:19:06) * ACTION: apevec write announcement and work with rbowen to publicize it (apevec, 15:20:24) * rdopkg reviews (chandankumar, 15:22:06) * altarch status (chandankumar, 15:24:52) * ppc64le architecture will be enabled soon (number80, 15:27:07) * help testing https://bugzilla.redhat.com/show_bug.cgi?id=1391444 (number80, 15:27:48) * chair for next meeting (chandankumar, 15:29:10) * ACTION: jpena to chair for next meeting. (chandankumar, 15:29:39) * open floor (chandankumar, 15:29:46) * LINK: https://github.com/openstack-packages/rdopkg/issues/66 (jruzicka, 15:39:45) Meeting ended at 15:46:06 UTC. Action Items ------------ * chandankumar to start sending BZ reports to rdo-list * jpena to remote Liberty worker from DLRN servers * apevec practice EOL vault.centos.org move on Kilo (not EOLed properly yet!) * apevec write announcement and work with rbowen to publicize it * jpena to chair for next meeting. Action Items, by person ----------------------- * apevec * apevec practice EOL vault.centos.org move on Kilo (not EOLed properly yet!) * apevec write announcement and work with rbowen to publicize it * chandankumar * chandankumar to start sending BZ reports to rdo-list * jpena * jpena to remote Liberty worker from DLRN servers * jpena to chair for next meeting. * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * chandankumar (72) * apevec (26) * number80 (25) * zodbot (15) * Duck (15) * dmsimard (9) * jruzicka (8) * jpena (6) * amoralej (5) * trown (4) * weshay (4) * rdogerrit (4) * dmellado (3) * hrybacki (3) * adarazs (1) * coolsvap (1) * jschluet (1) * jpena|lunch (1) * mengxd (1) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot From michele at acksyn.org Wed Nov 16 16:42:51 2016 From: michele at acksyn.org (Michele Baldessari) Date: Wed, 16 Nov 2016 17:42:51 +0100 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> Message-ID: <20161116164251.GB4901@palahniuk.int.rhx> Hi Charles, On Tue, Nov 15, 2016 at 10:39:19PM +0000, Charles Short wrote: > So I have finally tried OSP9 and here are the results - > > 3 Controllers 40 compute - 1 hours 20 mins to deploy. > > This is much more the sort of deployment time I was expecting :) > > I then tried TripleO Newton Stable again with 3 Controllers 40 Compute - > > 4 hours and counting..... > > The two deployment scripts (for OSP9 and TripleO Newton) were pretty much > identical (allowing for any changes between releases) > > During the OSP9 deployment I could use nova list to list the nodes. The > Undercloud API access was in general very responsive. > > During the TripleO Newton deployment 'nova list' hangs - > ERROR (ClientException): The server has either erred or is incapable of > performing the requested operation. (HTTP 500) > Undercloud API access was very sluggish. > I noticed Keystone was stuck at 140% for most of the deployment (albeit > multi threaded) which is not the case for OSP9. > > I know it is hard to compare two releases, but the difference is enormous. > I will stick with OSP9 for now as this for me works properly out of the box > for large deployments. May I ask what version of python-oslo-messaging you have on Newton? The reason I ask, is that Eck fixed a pretty serious epoll() issue which caused deploys to fail with newton on weaker hardware (which would work in mitaka). The change I mention is this one: https://review.openstack.org/#/c/394963/ It would be great to confirm or not if the python-oslo-messaging package you are using has this change included. Thanks, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D From cems at ebi.ac.uk Wed Nov 16 17:21:39 2016 From: cems at ebi.ac.uk (Charles Short) Date: Wed, 16 Nov 2016 17:21:39 +0000 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <20161116164251.GB4901@palahniuk.int.rhx> References: <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> <20161116164251.GB4901@palahniuk.int.rhx> Message-ID: <83f81c47-8062-2bb6-3c88-c335335ff243@ebi.ac.uk> Hi, Unfortunately I am under some pressure to get my stack into production, so now have OSP9 running. I have little opportunity for more testing of Newton. If I get a window I will send you the info Regards Charles On 16/11/2016 16:42, Michele Baldessari wrote: > Hi Charles, > > On Tue, Nov 15, 2016 at 10:39:19PM +0000, Charles Short wrote: >> So I have finally tried OSP9 and here are the results - >> >> 3 Controllers 40 compute - 1 hours 20 mins to deploy. >> >> This is much more the sort of deployment time I was expecting :) >> >> I then tried TripleO Newton Stable again with 3 Controllers 40 Compute - >> >> 4 hours and counting..... >> >> The two deployment scripts (for OSP9 and TripleO Newton) were pretty much >> identical (allowing for any changes between releases) >> >> During the OSP9 deployment I could use nova list to list the nodes. The >> Undercloud API access was in general very responsive. >> >> During the TripleO Newton deployment 'nova list' hangs - >> ERROR (ClientException): The server has either erred or is incapable of >> performing the requested operation. (HTTP 500) >> Undercloud API access was very sluggish. >> I noticed Keystone was stuck at 140% for most of the deployment (albeit >> multi threaded) which is not the case for OSP9. >> >> I know it is hard to compare two releases, but the difference is enormous. >> I will stick with OSP9 for now as this for me works properly out of the box >> for large deployments. > May I ask what version of python-oslo-messaging you have on Newton? > > The reason I ask, is that Eck fixed a pretty serious epoll() issue which > caused deploys to fail with newton on weaker hardware (which would work > in mitaka). The change I mention is this one: > https://review.openstack.org/#/c/394963/ > > It would be great to confirm or not if the python-oslo-messaging package > you are using has this change included. > > Thanks, > Michele -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205 From apevec at redhat.com Thu Nov 17 00:08:39 2016 From: apevec at redhat.com (Alan Pevec) Date: Thu, 17 Nov 2016 01:08:39 +0100 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: <20161116164251.GB4901@palahniuk.int.rhx> References: <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> <20161116164251.GB4901@palahniuk.int.rhx> Message-ID: On Wed, Nov 16, 2016 at 5:42 PM, Michele Baldessari wrote: > The reason I ask, is that Eck fixed a pretty serious epoll() issue which > caused deploys to fail with newton on weaker hardware (which would work > in mitaka). The change I mention is this one: > https://review.openstack.org/#/c/394963/ This was merged last week and not released upstream yet AFAICT. In order to get this into RDO Newton you need to: * request oslo.messaging minor release on stable/newton * bump it in upstream stable/newton upper-constraints.txt * sync that bump in rdoinfo (should be automatic now) Cheers, Alan From apevec at redhat.com Thu Nov 17 10:56:53 2016 From: apevec at redhat.com (Alan Pevec) Date: Thu, 17 Nov 2016 11:56:53 +0100 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> <20161116164251.GB4901@palahniuk.int.rhx> Message-ID: >> The reason I ask, is that Eck fixed a pretty serious epoll() issue which >> caused deploys to fail with newton on weaker hardware (which would work >> in mitaka). The change I mention is this one: >> https://review.openstack.org/#/c/394963/ This is not released even on master. Mehdi, I see you did last oslo.messaging release 5.12.0 - do you have a plan to have a new release today for Ocata-1 ? If not, I can send release requests for both Ocata and Newton to get this rabbit fix into RDO. Cheers, Alan > This was merged last week and not released upstream yet AFAICT. > In order to get this into RDO Newton you need to: > * request oslo.messaging minor release on stable/newton > * bump it in upstream stable/newton upper-constraints.txt > * sync that bump in rdoinfo (should be automatic now) From apevec at redhat.com Thu Nov 17 11:47:37 2016 From: apevec at redhat.com (Alan Pevec) Date: Thu, 17 Nov 2016 12:47:37 +0100 Subject: [rdo-list] [TripleO] Newton large baremetal deployment issues In-Reply-To: References: <0a336703-1590-d2b8-6cf9-9547057f6b13@ebi.ac.uk> <11f316cb-6c73-8c7c-3569-9ce82a6110c6@redhat.com> <1d3af988-a378-cdfe-8115-b43b684dc37b@ebi.ac.uk> <81643e72-d6f0-012d-f5c5-7ee57414ea38@ebi.ac.uk> <20161116164251.GB4901@palahniuk.int.rhx> Message-ID: > Mehdi, I see you did last oslo.messaging release 5.12.0 - do you have > a plan to have a new release today for Ocata-1 ? Update from IRC: Mehdi has created release requests * newton https://review.openstack.org/398947 * master/ocata https://review.openstack.org/398946 Cheers, Alan From war at undrground.org Sat Nov 19 14:43:49 2016 From: war at undrground.org (Nate) Date: Sat, 19 Nov 2016 09:43:49 -0500 (EST) Subject: [rdo-list] [TripleO] quickstart undercloud problems. Message-ID: <1197516940.158848.1479566629857.JavaMail.zimbra@undrground.org> Good Morning rdo-list, im attempting to get a test openstack install up and running using RDO, and ooo quickstart on Centos 7. I'm following the brief instructions found here: https://www.rdoproject.org/tripleo/ I am running quickstart from the same system i'm targeting. So i'm setting $VIRTHOST to localhost, or the local IP of the box (192.168.3.10), I've tried it both ways, which doesn't seem to make a difference. I'm running quickstart as my local use, not the stack user. Initially things appear to run along nicely, the script downloads images and whatnot, it even starts up the undercloud vm as the stack user. [stack at fog ~]$ virsh list Id Name State ---------------------------------------------------- 2 undercloud running When quickstart gets to the point where it's attempting to determine the IP of the undercloud VM, it fails. I did a little digging on how exactly that works, and found that the script which it runs is not getting any results. >From the script: mac=$(virsh dumpxml $VMNAME | awk -F "'" '/mac address/ { print $2; exit }') # Look up the MAC address in the ARP table. ip=$(ip neigh | grep $mac | awk '{print $1;}') I've tried these manually, and sure enough the mac address is not in ip neigh. stack at fog .quickstart]$ virsh dumpxml undercloud | awk -F "'" '/mac address/ { print $2; exit }' 00:e4:b2:54:b5:fb [stack at fog .quickstart]$ ip neigh | grep '00:e4:b2:54:b5:fb' | awk '{print $1;}' [stack at fog .quickstart]$ ip neigh | grep -v FAILED 192.168.3.1 dev enp13s0 lladdr 52:54:00:ab:d2:15 REACHABLE 192.168.3.251 dev enp13s0 lladdr 52:54:00:f2:90:b7 REACHABLE [stack at fog .quickstart]$ I did a virsh console on the undercloud vm, and reset it, so I could watch it boot, and I see the following in the boot messages: [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. I cant login to the undercloud vm, as I don't have the local username/pass. or I'd troubleshoot that further. As far as I can tell the undercloud system uses standard linux bridges to setup network communication, They appear present, but as they're not configured in the same manner i've used in the past, I can't be certain if they're working properly. [stack at fog ~]$ brctl show bridge name bridge id STP enabled interfaces brext 8000.525400af4796 no brext-nic tap0 brovc 8000.525400180812 no brovc-nic tap1 virbr0 8000.525400a3f6b5 yes virbr0-nic [stack at fog ~]$ ifconfig brext-nic brext-nic: flags=4098 mtu 1500 ether 52:54:00:af:47:96 txqueuelen 500 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [stack at fog ~]$ ifconfig brovc-nic brovc-nic: flags=4098 mtu 1500 ether 52:54:00:18:08:12 txqueuelen 500 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 I've tried some other basics, like setting selinux to permissinv and re-trying (which was a long shot) and it didn't help. Any suggestions on what to tr next would be helpful. Thanks! ----------- RHCSA, RHCE, RHCVA (#110-011-426) gpg public key: https://keybase.io/gangrif - or - MIT's public key server, pgp.mit.edu From war at undrground.org Sat Nov 19 14:52:53 2016 From: war at undrground.org (Nate) Date: Sat, 19 Nov 2016 09:52:53 -0500 (EST) Subject: [rdo-list] [TripleO] quickstart undercloud problems. In-Reply-To: <1582387384.158878.1479567147588.JavaMail.zimbra@undrground.org> References: <1197516940.158848.1479566629857.JavaMail.zimbra@undrground.org> Message-ID: <1598155372.158879.1479567173098.JavaMail.zimbra@undrground.org> Did you ever type up a lengthy email to a list, looking for help, and then the moment you hit send, a lightbulb goes on, and you fix your ow problem? Apparently when I built my test system, my build environment turned on iptables (like I've told it to) and it was too restrictive. I flushed iptables, and disabled puppet (so it wouldnt turn it back on again) rebooted the undercloud vm, and guess what? It's on the network now. Quickstart is now able to communicate with the undercloud vm. To anyone reading this, having a similar problem.... Check iptables. :headdesk: ----------- RHCSA, RHCE, RHCVA (#110-011-426) gpg public key: https://keybase.io/gangrif - or - MIT's public key server, pgp.mit.edu ----- Original Message ----- From: "war" To: rdo-list at redhat.com Sent: Saturday, November 19, 2016 9:43:49 AM Subject: [rdo-list] [TripleO] quickstart undercloud problems. Good Morning rdo-list, im attempting to get a test openstack install up and running using RDO, and ooo quickstart on Centos 7. I'm following the brief instructions found here: https://www.rdoproject.org/tripleo/ I am running quickstart from the same system i'm targeting. So i'm setting $VIRTHOST to localhost, or the local IP of the box (192.168.3.10), I've tried it both ways, which doesn't seem to make a difference. I'm running quickstart as my local use, not the stack user. Initially things appear to run along nicely, the script downloads images and whatnot, it even starts up the undercloud vm as the stack user. [stack at fog ~]$ virsh list Id Name State ---------------------------------------------------- 2 undercloud running When quickstart gets to the point where it's attempting to determine the IP of the undercloud VM, it fails. I did a little digging on how exactly that works, and found that the script which it runs is not getting any results. >From the script: mac=$(virsh dumpxml $VMNAME | awk -F "'" '/mac address/ { print $2; exit }') # Look up the MAC address in the ARP table. ip=$(ip neigh | grep $mac | awk '{print $1;}') I've tried these manually, and sure enough the mac address is not in ip neigh. stack at fog .quickstart]$ virsh dumpxml undercloud | awk -F "'" '/mac address/ { print $2; exit }' 00:e4:b2:54:b5:fb [stack at fog .quickstart]$ ip neigh | grep '00:e4:b2:54:b5:fb' | awk '{print $1;}' [stack at fog .quickstart]$ ip neigh | grep -v FAILED 192.168.3.1 dev enp13s0 lladdr 52:54:00:ab:d2:15 REACHABLE 192.168.3.251 dev enp13s0 lladdr 52:54:00:f2:90:b7 REACHABLE [stack at fog .quickstart]$ I did a virsh console on the undercloud vm, and reset it, so I could watch it boot, and I see the following in the boot messages: [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. I cant login to the undercloud vm, as I don't have the local username/pass. or I'd troubleshoot that further. As far as I can tell the undercloud system uses standard linux bridges to setup network communication, They appear present, but as they're not configured in the same manner i've used in the past, I can't be certain if they're working properly. [stack at fog ~]$ brctl show bridge name bridge id STP enabled interfaces brext 8000.525400af4796 no brext-nic tap0 brovc 8000.525400180812 no brovc-nic tap1 virbr0 8000.525400a3f6b5 yes virbr0-nic [stack at fog ~]$ ifconfig brext-nic brext-nic: flags=4098 mtu 1500 ether 52:54:00:af:47:96 txqueuelen 500 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [stack at fog ~]$ ifconfig brovc-nic brovc-nic: flags=4098 mtu 1500 ether 52:54:00:18:08:12 txqueuelen 500 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 I've tried some other basics, like setting selinux to permissinv and re-trying (which was a long shot) and it didn't help. Any suggestions on what to tr next would be helpful. Thanks! ----------- RHCSA, RHCE, RHCVA (#110-011-426) gpg public key: https://keybase.io/gangrif - or - MIT's public key server, pgp.mit.edu _______________________________________________ rdo-list mailing list rdo-list at redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe at redhat.com From whayutin at redhat.com Mon Nov 21 01:22:18 2016 From: whayutin at redhat.com (Wesley Hayutin) Date: Sun, 20 Nov 2016 20:22:18 -0500 Subject: [rdo-list] [TripleO] quickstart undercloud problems. In-Reply-To: <1598155372.158879.1479567173098.JavaMail.zimbra@undrground.org> References: <1197516940.158848.1479566629857.JavaMail.zimbra@undrground.org> <1582387384.158878.1479567147588.JavaMail.zimbra@undrground.org> <1598155372.158879.1479567173098.JavaMail.zimbra@undrground.org> Message-ID: Sent from my mobile On Nov 19, 2016 09:53, "Nate" wrote: > > Did you ever type up a lengthy email to a list, looking for help, and then the moment you hit send, a lightbulb goes on, and you fix your ow problem? > > Ha, I know I have. Not a problem, we're on #rdo if anything else comes up. Take care > Apparently when I built my test system, my build environment turned on iptables (like I've told it to) and it was too restrictive. I flushed iptables, and disabled puppet (so it wouldnt turn it back on again) rebooted the undercloud vm, and guess what? It's on the network now. > > Quickstart is now able to communicate with the undercloud vm. > > To anyone reading this, having a similar problem.... Check iptables. > > :headdesk: > > ----------- > RHCSA, RHCE, RHCVA (#110-011-426) > gpg public key: > https://keybase.io/gangrif > - or - > MIT's public key server, pgp.mit.edu > > ----- Original Message ----- > From: "war" > To: rdo-list at redhat.com > Sent: Saturday, November 19, 2016 9:43:49 AM > Subject: [rdo-list] [TripleO] quickstart undercloud problems. > > Good Morning rdo-list, > im attempting to get a test openstack install up and running using RDO, and ooo quickstart on Centos 7. > > I'm following the brief instructions found here: https://www.rdoproject.org/tripleo/ > > I am running quickstart from the same system i'm targeting. So i'm setting $VIRTHOST to localhost, or the local IP of the box (192.168.3.10), I've tried it both ways, which doesn't seem to make a difference. I'm running quickstart as my local use, not the stack user. > > Initially things appear to run along nicely, the script downloads images and whatnot, it even starts up the undercloud vm as the stack user. > > [stack at fog ~]$ virsh list > Id Name State > ---------------------------------------------------- > 2 undercloud running > > > When quickstart gets to the point where it's attempting to determine the IP of the undercloud VM, it fails. I did a little digging on how exactly that works, and found that the script which it runs is not getting any results. > > >From the script: > mac=$(virsh dumpxml $VMNAME | awk -F "'" '/mac address/ { print $2; exit }') > > # Look up the MAC address in the ARP table. > ip=$(ip neigh | grep $mac | awk '{print $1;}') > > I've tried these manually, and sure enough the mac address is not in ip neigh. > > stack at fog .quickstart]$ virsh dumpxml undercloud | awk -F "'" '/mac address/ { print $2; exit }' > 00:e4:b2:54:b5:fb > [stack at fog .quickstart]$ ip neigh | grep '00:e4:b2:54:b5:fb' | awk '{print $1;}' > [stack at fog .quickstart]$ ip neigh | grep -v FAILED > 192.168.3.1 dev enp13s0 lladdr 52:54:00:ab:d2:15 REACHABLE > 192.168.3.251 dev enp13s0 lladdr 52:54:00:f2:90:b7 REACHABLE > [stack at fog .quickstart]$ > > I did a virsh console on the undercloud vm, and reset it, so I could watch it boot, and I see the following in the boot messages: > > [FAILED] Failed to start LSB: Bring up/down networking. > See 'systemctl status network.service' for details. > > I cant login to the undercloud vm, as I don't have the local username/pass. or I'd troubleshoot that further. > > > As far as I can tell the undercloud system uses standard linux bridges to setup network communication, They appear present, but as they're not configured in the same manner i've used in the past, I can't be certain if they're working properly. > > [stack at fog ~]$ brctl show > bridge name bridge id STP enabled interfaces > brext 8000.525400af4796 no brext-nic > tap0 > brovc 8000.525400180812 no brovc-nic > tap1 > virbr0 8000.525400a3f6b5 yes virbr0-nic > [stack at fog ~]$ ifconfig brext-nic > brext-nic: flags=4098 mtu 1500 > ether 52:54:00:af:47:96 txqueuelen 500 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > [stack at fog ~]$ ifconfig brovc-nic > brovc-nic: flags=4098 mtu 1500 > ether 52:54:00:18:08:12 txqueuelen 500 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > I've tried some other basics, like setting selinux to permissinv and re-trying (which was a long shot) and it didn't help. > > Any suggestions on what to tr next would be helpful. > > Thanks! > > ----------- > RHCSA, RHCE, RHCVA (#110-011-426) > gpg public key: > https://keybase.io/gangrif > - or - > MIT's public key server, pgp.mit.edu > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From snecklifter at gmail.com Mon Nov 21 13:02:40 2016 From: snecklifter at gmail.com (Christopher Brown) Date: Mon, 21 Nov 2016 13:02:40 +0000 Subject: [rdo-list] Strange API latency issue Message-ID: Hello, Wondering if anyone can shed any light on the following: I'm seeing a ~2 second delay in some API responses using Mitaka trunk. E.g. $ openstack endpoint list --timing +----------------------------------------------------------------------------+-----------+ | URL | Seconds | +----------------------------------------------------------------------------+-----------+ | GET http://PUBLIC_VIP:35357/v3 | 0.009938 | | POST http://PUBLIC_VIP:35357/v3/auth/tokens | 0.257309 | | POST http://PUBLIC_VIP:35357/v3/auth/tokens | 0.171096 | | GET http://PUBLIC_VIP:35357/ | 1.741362 | | GET http://PUBLIC_VIP:35357/v3/endpoints | 0.026326 | | GET http://PUBLIC_VIP:35357/v3/services/b28a9d7986684c52905ea2f45fd6dffe | 0.023991 | | GET http://PUBLIC_VIP:35357/v3/services/1fb856a569714d44b787868151f995b6 | 0.025029 | | GET http://PUBLIC_VIP:35357/v3/services/b28a9d7986684c52905ea2f45fd6dffe | 0.028141 | | GET http://PUBLIC_VIP:35357/v3/services/a857a42350e74106881534bcb19ebaf7 | 1.995946 | | GET http://PUBLIC_VIP:35357/v3/services/442c73aabbb141e6839002665c35579f | 0.022442 | | GET http://PUBLIC_VIP:35357/v3/services/d1a85a3b47de41aeb10a4ae087048a20 | 0.037483 | | GET http://PUBLIC_VIP:35357/v3/services/25c587016a16407fb682e36fe3f62532 | 1.939549 | | GET http://PUBLIC_VIP:35357/v3/services/1fb856a569714d44b787868151f995b6 | 0.031643 | | GET http://PUBLIC_VIP:35357/v3/services/8d072c313df44f639f9d30cdd6165621 | 0.032537 | | GET http://PUBLIC_VIP:35357/v3/services/442c73aabbb141e6839002665c35579f | 0.022227 | | GET http://PUBLIC_VIP:35357/v3/services/25c587016a16407fb682e36fe3f62532 | 1.974827 | | GET http://PUBLIC_VIP:35357/v3/services/734da843f77c4e4689b0772dbd6960f3 | 2.15859 | | GET http://PUBLIC_VIP:35357/v3/services/8d072c313df44f639f9d30cdd6165621 | 0.023067 | | Total | 11.086058 | +----------------------------------------------------------------------------+-----------+ 11 seconds is pretty good, can be anything up to 90 seconds. I have another installation which works fine, all response in milliseconds. I have deployed with network isolation and predictable IPs. The only situation where this does not seem to exhibit is if I disable the external network entirely using controll-no-external.yaml. -- Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From dms at redhat.com Mon Nov 21 14:13:52 2016 From: dms at redhat.com (David Moreau Simard) Date: Mon, 21 Nov 2016 09:13:52 -0500 Subject: [rdo-list] CloudKitty failure to build in Liberty Message-ID: Hi, With the EOL of Liberty, we'd like to do one last release of liberty before archiving it. However, CloudKitty has been failing to build for a good while now [1]. The listed maintainer [2] should be receiving an e-mail whenever there is a failure to build, it is important to react on those as fast as possible. Could you please address the issue ASAP ? We're available in #rdo on freenode if you need assistance. Thanks, [1]: https://trunk.rdoproject.org/centos7-liberty/report.html [2]: https://github.com/redhat-openstack/rdoinfo/blob/54bd0d750311a57a7d693bfdae96df66161029e3/rdo.yml#L1040-L1059 David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] From hguemar at fedoraproject.org Mon Nov 21 15:00:03 2016 From: hguemar at fedoraproject.org (hguemar at fedoraproject.org) Date: Mon, 21 Nov 2016 15:00:03 +0000 (UTC) Subject: [rdo-list] [Fedocal] Reminder meeting : RDO meeting Message-ID: <20161121150003.BBD7B60A3FDC@fedocal02.phx2.fedoraproject.org> Dear all, You are kindly invited to the meeting: RDO meeting on 2016-11-23 from 15:00:00 to 16:00:00 UTC At rdo at irc.freenode.net The meeting will be about: RDO IRC meeting [Agenda at https://etherpad.openstack.org/p/RDO-Meeting ](https://etherpad.openstack.org/p/RDO-Meeting) Every Wednesday on #rdo on Freenode IRC Source: https://apps.fedoraproject.org/calendar/meeting/2017/ From lorenzetto.luca at gmail.com Mon Nov 21 16:59:05 2016 From: lorenzetto.luca at gmail.com (Luca 'remix_tj' Lorenzetto) Date: Mon, 21 Nov 2016 17:59:05 +0100 Subject: [rdo-list] os-collect-config, cnf endpoint and InternalAPI configuration Message-ID: Hello rdo-list, i'm playing now with some stacks using OS::Heat::SoftwareDeployment (first time). I'm running Mitaka on this openstack setup. My stack is using the default signal_transport (CFN as far as i can understand). I've seen os-collect-config not working because after the startup configured in this way: /etc/os-collect-config.conf [DEFAULT] command = os-refresh-config collectors = ec2 collectors = cfn collectors = local [cfn] metadata_url = http://172.25.122.11:8000/v1/ stack_name = test_lab secret_access_key = 13c40f89a12142ca826e059788ad2258 access_key_id = 88111670b6fb41ef816696c871986d89 path = rundeck.Metadata I see that the metadata url is pointing to a right IP from the internal API pool. First question: should i find that url listed as endpoint with openstack endpoint list ? Second (and more important question): It's the right behavior connecting to the InternalAPI url for a VM? I set up my internal API network as a private network between openstack nodes, and it is not routed and not accessible with the rest of the world. Should i set this network as routed (and add default gateways to the hosts?) I tried to understand this from docs (i did the setup through TripleO) both from official, community and RH ones, but had no luck on clarifying this question. Thank you in advance for the explanations, Luca -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet ? la pi? grande biblioteca del mondo. Ma il problema ? che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , From snecklifter at gmail.com Mon Nov 21 17:14:10 2016 From: snecklifter at gmail.com (Christopher Brown) Date: Mon, 21 Nov 2016 17:14:10 +0000 Subject: [rdo-list] os-collect-config, cnf endpoint and InternalAPI configuration In-Reply-To: References: Message-ID: Hi Luca, Most heat templates (E.g. openshift on openstack) expect to see the url on the external endpoint. So I usually run: sudo openstack-config --set /etc/heat/heat.conf DEFAULT heat_metadata_server_url http://EXTERNAL_VIP:8000 sudo openstack-config --set /etc/heat/heat.conf DEFAULT heat_waitcondition_server_url http://EXTERNAL_VIP:8000/v1/waitcondition (on all controllers then pcs resource restart if using HA) I think this can be adjusted in better ways than the above (Endpoint map perhaps or in Newton?) but this gives me what I need. Regards On 21 November 2016 at 16:59, Luca 'remix_tj' Lorenzetto < lorenzetto.luca at gmail.com> wrote: > Hello rdo-list, > > i'm playing now with some stacks using OS::Heat::SoftwareDeployment > (first time). > I'm running Mitaka on this openstack setup. > My stack is using the default signal_transport (CFN as far as i can > understand). > > I've seen os-collect-config not working because after the startup > configured in this way: > > /etc/os-collect-config.conf > > [DEFAULT] > command = os-refresh-config > collectors = ec2 > collectors = cfn > collectors = local > > [cfn] > metadata_url = http://172.25.122.11:8000/v1/ > stack_name = test_lab > secret_access_key = 13c40f89a12142ca826e059788ad2258 > access_key_id = 88111670b6fb41ef816696c871986d89 > path = rundeck.Metadata > > > I see that the metadata url is pointing to a right IP from the > internal API pool. > > First question: should i find that url listed as endpoint with > openstack endpoint list ? > > Second (and more important question): > It's the right behavior connecting to the InternalAPI url for a VM? > > I set up my internal API network as a private network between > openstack nodes, and it is not routed and not accessible with the rest > of the world. > Should i set this network as routed (and add default gateways to the > hosts?) > I tried to understand this from docs (i did the setup through TripleO) > both from official, community and RH ones, but had no luck on > clarifying this question. > > Thank you in advance for the explanations, > > Luca > > > > > > > -- > "E' assurdo impiegare gli uomini di intelligenza eccellente per fare > calcoli che potrebbero essere affidati a chiunque se si usassero delle > macchine" > Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) > > "Internet ? la pi? grande biblioteca del mondo. > Ma il problema ? che i libri sono tutti sparsi sul pavimento" > John Allen Paulos, Matematico (1945-vivente) > > Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , < > lorenzetto.luca at gmail.com> > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com -- Christopher Brown -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.johnston at icloud.com Mon Nov 21 19:15:20 2016 From: dave.johnston at icloud.com (Dave Johnston) Date: Mon, 21 Nov 2016 19:15:20 +0000 (GMT) Subject: [rdo-list] ODL Integration with RDO Newton Message-ID: <9cd37ee0-4048-4414-aeae-ca2ed355fd18@me.com> Hi, I'm attempting to integrate ODL with Newton. I'm hitting an issue when I start the neutron-server, it reports that it can't load the opendaylight driver. ERROR stevedore.extension [-] Could not load 'opendaylight': No module named opendaylight.driver I've installed the networking_odl package that contains the driver, is anyone aware of other changes that might be needed? -- Dave J -------------- next part -------------- An HTML attachment was scrubbed... URL: From alon.dotan at hpe.com Mon Nov 21 20:27:27 2016 From: alon.dotan at hpe.com (Dotan, Alon) Date: Mon, 21 Nov 2016 20:27:27 +0000 Subject: [rdo-list] ODL Integration with RDO Newton In-Reply-To: <9cd37ee0-4048-4414-aeae-ca2ed355fd18@me.com> References: <9cd37ee0-4048-4414-aeae-ca2ed355fd18@me.com> Message-ID: Can you attach the ml2_conf.ini ? ________________________________ From: rdo-list-bounces at redhat.com on behalf of Dave Johnston Sent: Monday, November 21, 2016 9:15:20 PM To: rdo-list at redhat.com Subject: [rdo-list] ODL Integration with RDO Newton Hi, I'm attempting to integrate ODL with Newton. I'm hitting an issue when I start the neutron-server, it reports that it can't load the opendaylight driver. ERROR stevedore.extension [-] Could not load 'opendaylight': No module named opendaylight.driver I've installed the networking_odl package that contains the driver, is anyone aware of other changes that might be needed? -- Dave J -------------- next part -------------- An HTML attachment was scrubbed... URL: From javier.pena at redhat.com Tue Nov 22 09:33:34 2016 From: javier.pena at redhat.com (Javier Pena) Date: Tue, 22 Nov 2016 04:33:34 -0500 (EST) Subject: [rdo-list] CloudKitty failure to build in Liberty In-Reply-To: References: Message-ID: <205072906.1319540.1479807214885.JavaMail.zimbra@redhat.com> ----- Original Message ----- > Hi, > > With the EOL of Liberty, we'd like to do one last release of liberty > before archiving it. > > However, CloudKitty has been failing to build for a good while now [1]. > The listed maintainer [2] should be receiving an e-mail whenever there > is a failure to build, it is important to react on those as fast as > possible. > Looking at the CloudKitty repo [3], it only has stable/mitaka and stable/newton branches, so the Liberty worker is probably trying to build from master. I think we should remove CloudKitty from Liberty, but please correct me if I'm wrong. Regards, Javier [3] https://github.com/openstack/cloudkitty > Could you please address the issue ASAP ? > We're available in #rdo on freenode if you need assistance. > > Thanks, > > [1]: https://trunk.rdoproject.org/centos7-liberty/report.html > [2]: > https://github.com/redhat-openstack/rdoinfo/blob/54bd0d750311a57a7d693bfdae96df66161029e3/rdo.yml#L1040-L1059 > > David Moreau Simard > Senior Software Engineer | Openstack RDO > > dmsimard = [irc, github, twitter] > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From apevec at redhat.com Tue Nov 22 09:55:07 2016 From: apevec at redhat.com (Alan Pevec) Date: Tue, 22 Nov 2016 10:55:07 +0100 Subject: [rdo-list] CloudKitty failure to build in Liberty In-Reply-To: <205072906.1319540.1479807214885.JavaMail.zimbra@redhat.com> References: <205072906.1319540.1479807214885.JavaMail.zimbra@redhat.com> Message-ID: > I think we should remove CloudKitty from Liberty, but please correct me if I'm wrong. You are right, not sure what happened with stable/liberty, there is eol-kilo tag... Anyway, first release we had in RDO was 0.5.0 in Mitaka, it was simply added with defaults 1 year ago and liberty tag was added by mistake when merging rdoinfo-Kilo fork. So removing it from Liberty is correct fix, I've proposed in http://review.rdoproject.org/r/3862 Alan From lorenzetto.luca at gmail.com Tue Nov 22 10:26:42 2016 From: lorenzetto.luca at gmail.com (Luca 'remix_tj' Lorenzetto) Date: Tue, 22 Nov 2016 11:26:42 +0100 Subject: [rdo-list] os-collect-config, cnf endpoint and InternalAPI configuration In-Reply-To: References: Message-ID: On Mon, Nov 21, 2016 at 6:14 PM, Christopher Brown wrote: > Hi Luca, > > Most heat templates (E.g. openshift on openstack) expect to see the url on > the external endpoint. So I usually run: > > sudo openstack-config --set /etc/heat/heat.conf DEFAULT > heat_metadata_server_url http://EXTERNAL_VIP:8000 > sudo openstack-config --set /etc/heat/heat.conf DEFAULT > heat_waitcondition_server_url http://EXTERNAL_VIP:8000/v1/waitcondition > > (on all controllers then pcs resource restart if using HA) > > I think this can be adjusted in better ways than the above (Endpoint map > perhaps or in Newton?) but this gives me what I need. Thanks Christopher for the workaround. I'd like to understand better if these Internal Url endpoints should be accessible from VMs or not. I see several documentations where publicUrl and internalUrl are the same. I found out also that the main difference between this two endpoints is that internalUrl traffic should be unmetered and excluded from billing. But still no documentation explaining what i'd like to understand. If someone can clarify better, it would be nice for me. -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet ? la pi? grande biblioteca del mondo. Ma il problema ? che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , From whayutin at redhat.com Tue Nov 22 13:49:05 2016 From: whayutin at redhat.com (Wesley Hayutin) Date: Tue, 22 Nov 2016 08:49:05 -0500 Subject: [rdo-list] [CI] tripleo-quickstart-extra's roles migrated from gerrithub to openstack Message-ID: Greetings, TLDR version, tripleo-quickstart-roles have moved to the openstack upstream source control sytem. Details: All the additional git and gerrit repos used with tripleo-quickstart that have formally been housed in [1] gerrithub and github/redhat-openstack have been migrated to the upstream openstack git and gerrit [2]. Additional details regarding the move can be found here [3]. There are a few roles that were still being incubated in gerrithub that will be added one at a time via the review process. Additional details regarding the transition of tripleo-quickstart and tripleo-quickstart-extras to upstream can be found here [4] Thank you! [1] https://review.gerrithub.io https://github.com/redhat-openstack/?utf8=%E2%9C%93&q=ansible-role-tripleo&type=&language= [2] http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/ https://github.com/openstack/tripleo-quickstart-extras [3] https://paste.fedoraproject.org/488434/79820118/ [4] https://trello.com/b/HhXlqdiu/rdo?menu=filter&filter=[Q -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbowen at redhat.com Tue Nov 22 15:45:52 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 22 Nov 2016 10:45:52 -0500 Subject: [rdo-list] RDO at FOSDEM? Message-ID: <71a51f04-9292-baac-810b-4f9a797201d2@redhat.com> FOSDEM always sneaks up on us ... it's time to figure out if we're doing something there, and, if so, what. The date would be Friday, February 3rd. The opportunity exists to do something again with CentOS out at the IBM facility, like last year. There was a lot of feedback last time that it was inconvenient and hard to get to. However, it is free, so there's that. I'm also investigating a few venues closer to Grand Place, where many of us tend to stay during FOSDEM. This makes it a lot easier to get to for all of us, but there's a cost involved. And there's also a chance that if we do that, we'd be doing it on our own, without CentOS. So, this is a straw poll. Would people rather do something with CentOS at an inconvenient place, or by ourselves at a place we can walk to? -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From rbowen at redhat.com Tue Nov 22 15:54:14 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 22 Nov 2016 10:54:14 -0500 Subject: [rdo-list] ask.openstack.org unanswered RDO questions Message-ID: <52da0c2a-a9db-331c-c9be-0794315e6a56@redhat.com> 30 unanswered questions: RDO newton release has support for RHEL 7.3 ? https://ask.openstack.org/en/question/99201/rdo-newton-release-has-support-for-rhel-73/ Tags: rdo, rhel, openstack CPU Load on instance and Controller is very high and unresponsive https://ask.openstack.org/en/question/98927/cpu-load-on-instance-and-controller-is-very-high-and-unresponsive/ Tags: mitaka-openstack [Error: No valid host was found.((Network br-ex error)) https://ask.openstack.org/en/question/98818/error-no-valid-host-was-foundnetwork-br-ex-error/ Tags: br-ex Compute nodes are not visible after controller node rebooted https://ask.openstack.org/en/question/98654/compute-nodes-are-not-visible-after-controller-node-rebooted/ Tags: rdo Live migration failing with qemu error "does not support drive-mirror" https://ask.openstack.org/en/question/98144/live-migration-failing-with-qemu-error-does-not-support-drive-mirror/ Tags: newton, nova, qemu, kvm, block-live-migration error while launching instance in openstack 'newton' using rdo packstack https://ask.openstack.org/en/question/98113/error-while-launching-instance-in-openstack-newton-using-rdo-packstack/ Tags: rdo, newton, packstack How can I add a new region to an installed All-In-One RDO openstack? https://ask.openstack.org/en/question/98111/how-can-i-add-a-new-region-to-an-installed-all-in-one-rdo-openstack/ Tags: rdo, all-in-one, multi-region, region, add-region How to setup solr search engine for openstack swift https://ask.openstack.org/en/question/98093/how-to-setup-solr-search-engine-for-openstack-swift/ Tags: openstack, openstack-swift, newton, rdo Heat fails with 504 error. https://ask.openstack.org/en/question/98092/heat-fails-with-504-error/ Tags: rdo, tripleo, mitaka, heat Devstack and OpenvSwitch v2.3+ https://ask.openstack.org/en/question/98004/devstack-and-openvswitch-v23/ Tags: devstack, openvswitch How how does icmp packet travel across two compute nodes without br-int and br-tun running? https://ask.openstack.org/en/question/97905/how-how-does-icmp-packet-travel-across-two-compute-nodes-without-br-int-and-br-tun-running/ Tags: neutron-ovs-pktflow "Parameter outiface failed on Firewall" during installation of openstack rdo on centos 7 https://ask.openstack.org/en/question/95657/parameter-outiface-failed-on-firewall-during-installation-of-openstack-rdo-on-centos-7/ Tags: rdo, devstack#mitaka multi nodes provider network ovs config https://ask.openstack.org/en/question/95423/multi-nodes-provider-network-ovs-config/ Tags: rdo, liberty-neutron Adding additional packages to an RDO installation https://ask.openstack.org/en/question/95380/adding-additional-packages-to-an-rdo-installation/ Tags: rdo, mistral RDO - is there any fedora package newer than puppet-4.2.1-3.fc24.noarch.rpm https://ask.openstack.org/en/question/94969/rdo-is-there-any-fedora-package-newer-than-puppet-421-3fc24noarchrpm/ Tags: rdo, puppet, install-openstack how to deploy haskell-distributed in RDO? https://ask.openstack.org/en/question/94785/how-to-deploy-haskell-distributed-in-rdo/ Tags: rdo rdo tripleO liberty undercloud install failing https://ask.openstack.org/en/question/94023/rdo-tripleo-liberty-undercloud-install-failing/ Tags: rdo, rdo-manager, liberty, undercloud, instack Liberty RDO: stack resource topology icons are pink https://ask.openstack.org/en/question/91347/liberty-rdo-stack-resource-topology-icons-are-pink/ Tags: resource, topology, dashboard, horizon, pink No handlers could be found for logger "oslo_config.cfg" while syncing the glance database https://ask.openstack.org/en/question/91169/no-handlers-could-be-found-for-logger-oslo_configcfg-while-syncing-the-glance-database/ Tags: liberty, glance, install-openstack CentOS OpenStack - compute node can't talk https://ask.openstack.org/en/question/88989/centos-openstack-compute-node-cant-talk/ Tags: rdo How to setup SWIFT_PROXY_NODE and SWIFT_STORAGE_NODEs separately on RDO Liberty ? https://ask.openstack.org/en/question/88897/how-to-setup-swift_proxy_node-and-swift_storage_nodes-separately-on-rdo-liberty/ Tags: rdo, liberty, swift, ha VM and container can't download anything from internet https://ask.openstack.org/en/question/88338/vm-and-container-cant-download-anything-from-internet/ Tags: rdo, neutron, network, connectivity Fedora22, Liberty, horizon VNC console and keymap=sv with ; and/ https://ask.openstack.org/en/question/87451/fedora22-liberty-horizon-vnc-console-and-keymapsv-with-and/ Tags: keyboard, map, keymap, vncproxy, novnc OpenStack-Docker driver failed https://ask.openstack.org/en/question/87243/openstack-docker-driver-failed/ Tags: docker, openstack, liberty Routing between two tenants https://ask.openstack.org/en/question/84645/routing-between-two-tenants/ Tags: kilo, fuel, rdo, routing openstack baremetal introspection internal server error https://ask.openstack.org/en/question/82790/openstack-baremetal-introspection-internal-server-error/ Tags: rdo, ironic-inspector, tripleo Installing openstack using packstack (rdo) failed https://ask.openstack.org/en/question/82473/installing-openstack-using-packstack-rdo-failed/ Tags: rdo, packstack, installation-error, keystone VMware Host Backend causes No valid host was found. Bug ??? https://ask.openstack.org/en/question/79738/vmware-host-backend-causes-no-valid-host-was-found-bug/ Tags: vmware, rdo Mutlinode Devstack with two interfaces https://ask.openstack.org/en/question/78615/mutlinode-devstack-with-two-interfaces/ Tags: devstack, vlan, openstack Overcloud deployment on VM fails as IP address from DHCP is not assigned https://ask.openstack.org/en/question/66272/overcloud-deployment-on-vm-fails-as-ip-address-from-dhcp-is-not-assigned/ Tags: overcloud_in_vm -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From apevec at redhat.com Tue Nov 22 22:47:40 2016 From: apevec at redhat.com (Alan Pevec) Date: Tue, 22 Nov 2016 23:47:40 +0100 Subject: [rdo-list] RDO at FOSDEM? In-Reply-To: <71a51f04-9292-baac-810b-4f9a797201d2@redhat.com> References: <71a51f04-9292-baac-810b-4f9a797201d2@redhat.com> Message-ID: > So, this is a straw poll. Would people rather do something with CentOS > at an inconvenient place, or by ourselves at a place we can walk to? I don't think it was such inconvenient place, at least not when coming directly from the airport, it is on the way to the city. I'd keep RDO meetup collocated with CentOS Dojo, but I'd prefer if we could have it combined like in Barcelona with Ceph (just more serious setup w/ less drinks, so people would actually listen and participate in the sessions :) and not having RDO isolated in it's own room. For that to work, we need to find right topics which would be interesting for the wider CentOS Dojo audience, so let's start with that: what topics are folks proposing? I've added this to the agenda tomorrow. Cheers, Alan From hrybacki at redhat.com Tue Nov 22 22:52:10 2016 From: hrybacki at redhat.com (Harry Rybacki) Date: Tue, 22 Nov 2016 17:52:10 -0500 Subject: [rdo-list] RDO-Infra Weekly Scrum Recap: Message-ID: Greetings All, Links to the RDO-Infra team's scrum etherpad[1] and video recording[2] of the meeting are below. Highlights: - TripleO-Quickstart-extras has moved upstream[3] - Feature freeze on all roles on gerrithub, they have been -2'd, and work must be ported to reviews upstream - rlandy will break down the heat provisioning role and submit reviews against the new repo - A need still exists for baremetal/ovb core(s) - jpena discussed need for a gate for DLRN changes - Reviewed team Trello board[4] [1] - https://review.rdoproject.org/etherpad/p/rdo-infra-scrum [2] - https://bluejeans.com/s/cvDpM/ [3] - https://github.com/openstack/tripleo-quickstart-extras/ [4] - https://trello.com/b/HhXlqdiu/rdo /R Harry Rybacki From mrunge at redhat.com Wed Nov 23 10:49:23 2016 From: mrunge at redhat.com (Matthias Runge) Date: Wed, 23 Nov 2016 11:49:23 +0100 Subject: [rdo-list] RDO at FOSDEM? In-Reply-To: References: <71a51f04-9292-baac-810b-4f9a797201d2@redhat.com> Message-ID: <20161123104923.krlqfsevs4aoriz6@sofja.berg.ol.fritz.box> On Tue, Nov 22, 2016 at 11:47:40PM +0100, Alan Pevec wrote: > > So, this is a straw poll. Would people rather do something with CentOS > > at an inconvenient place, or by ourselves at a place we can walk to? > > I don't think it was such inconvenient place, at least not when coming > directly from the airport, it is on the way to the city. > I'd keep RDO meetup collocated with CentOS Dojo, but I'd prefer if we > could have it combined like in Barcelona with Ceph (just more serious > setup w/ less drinks, so people would actually listen and participate > in the sessions :) and not having RDO isolated in it's own room. For > that to work, we need to find right topics which would be interesting > for the wider CentOS Dojo audience, so let's start with that: what > topics are folks proposing? > I've added this to the agenda tomorrow. > I wouldn't even say, the drinks were the issue in Barcelona. It was more the direct vicinity of the bar and talks, as people tended to get a drink and to connect to each other there. Last year in Brussels there was a coffee area to catch up with each other, where the talks were away from that. I don't see the risk of people disturbing presentations in this set up. Best, Matthias -- Matthias Runge From mscherer at redhat.com Wed Nov 23 13:32:59 2016 From: mscherer at redhat.com (Michael Scherer) Date: Wed, 23 Nov 2016 14:32:59 +0100 Subject: [rdo-list] RDO at FOSDEM? In-Reply-To: References: <71a51f04-9292-baac-810b-4f9a797201d2@redhat.com> Message-ID: <1479907979.4674.338.camel@redhat.com> Le mardi 22 novembre 2016 ? 23:47 +0100, Alan Pevec a ?crit : > > So, this is a straw poll. Would people rather do something with CentOS > > at an inconvenient place, or by ourselves at a place we can walk to? > > I don't think it was such inconvenient place, at least not when coming > directly from the airport, it is on the way to the city. Yeah, but for those of us who come from train, cars or already in Bruxelles, it was inconvenient. And the lack of coffee/restaurant nearby mean that it was not practical to just go out and continue discussion. (but I am also getting grumpy everytime we say "let's move" and then we wait 1h and transform a group of 5 into a blob of 60 persons that can't decide where to go or can't move because we are too polite to kick 55 people out). I suspect next time, I will just announce that if a group is more than 5, I leave the group and do it my way. > I'd keep RDO meetup collocated with CentOS Dojo, but I'd prefer if we > could have it combined like in Barcelona with Ceph (just more serious > setup w/ less drinks, so people would actually listen and participate > in the sessions :) and not having RDO isolated in it's own room. For > that to work, we need to find right topics which would be interesting > for the wider CentOS Dojo audience, so let's start with that: what > topics are folks proposing? > I've added this to the agenda tomorrow. Based on what I did see around Openstack days yesterday, ansible is everywhere (ie, the only talk out of 5 I did see who didn't mention "ansible" was one on pnda.io, who use salt), and openstack orchestration using ansible did interest people. Not sure however how much the target audience did overlap with centos dojo (besides the few people I see during meetup that I did see there too) -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From rbowen at redhat.com Wed Nov 23 16:05:13 2016 From: rbowen at redhat.com (Rich Bowen) Date: Wed, 23 Nov 2016 11:05:13 -0500 Subject: [rdo-list] CentOS dojo at FOSDEM In-Reply-To: References: Message-ID: To followup from my earlier email about an RDO event at FOSDEM, here's some additional information. In short, the story is that we (RDO) would be participating as the Cloud SIG, as part of the main "track", rather than a dedicated RDO room as last year. Does this work for everyone here? --Rich -------- Forwarded Message -------- Subject: Re: [CentOS-promo] Daniel Paulus taking lead on org for bxl Dojo Date: Wed, 23 Nov 2016 15:58:53 +0000 From: Karanbir Singh Reply-To: Events, gatherings and meetings To: centos-promo at centos.org On 23/11/16 15:02, Rich Bowen wrote: > > > On 11/22/2016 10:09 AM, Karanbir Singh wrote: >> hi >> >> Daniel Paulus has offered to help with the organization and be a point >> of contact for the Brussels Dojo in Feb 3rd 2017. The rest of the team >> will still be supporting his efforts through the process. >> >> >> >> Please join me in welcoming the expanding CentOS Dojo organization >> group, and I look forward to seeing everyone in Feb. > > Thanks, Daniel, et al. Is it your desire that the RDO community be > included in this event again, as in the last two years? If so, we're > going to need a room - like last year - and I'll coordinate our > speaker/presentation schedule on our side. Ok? I think it would be nice to have the SIG's all integrate into the consolidated program rather than have dedicated tracks. Would something like that work ? It would certainly generate more interest for content. From javier.pena at redhat.com Wed Nov 23 16:41:10 2016 From: javier.pena at redhat.com (Javier Pena) Date: Wed, 23 Nov 2016 11:41:10 -0500 (EST) Subject: [rdo-list] [Meeting] RDO meeting (2016-11-23) Minutes Message-ID: <2033475194.2130438.1479919270070.JavaMail.zimbra@redhat.com> ============================== #rdo: RDO meeting - 2016-11-23 ============================== Meeting started by jpena at 15:00:26 UTC. The full logs are available at https://meetbot.fedoraproject.org/rdo/2016-11-23/rdo_meeting_-_2016-11-23.2016-11-23-15.00.log.html . Meeting summary --------------- * roll call (jpena, 15:00:58) * LINK: https://etherpad.openstack.org/p/RDO-Meeting (rbowen, 15:03:11) * Proposal: scheduled maintenance for review.rdoproject.org on Nov 30 (jpena, 15:04:52) * LINK: https://github.com/redhat-cip/software-factory/blob/master/CHANGELOG.md (jpena, 15:07:05) * AGREED: review.rdoproject.org maintenance will happen on Nov 30 (jpena, 15:10:30) * RDO at FOSDEM? (jpena, 15:10:48) * Puppet4 packaging (jpena, 15:18:18) * LINK: https://review.openstack.org/#/c/371209/ (EmilienM, 15:19:43) * LINK: https://copr-be.cloud.fedoraproject.org/results/hguemar/puppet4-el7/epel-7-x86_64/00453488-puppet/ (EmilienM, 15:20:10) * LINK: http://logs.openstack.org/09/371209/39/check/gate-tripleo-ci-centos-7-undercloud/12b0783/logs/rpm-qa.txt.gz (EmilienM, 15:21:17) * ACTION: number80 tag puppet4 in ocata (number80, 15:22:23) * : Announcement: rdopkg-0.42 released with %{?milestone} bug correction and more (jpena, 15:23:36) * LINK: https://github.com/openstack-packages/rdopkg/commit/31dc354666466083a1463391089868effc1ebc6e (jruzicka, 15:27:37) * Test day Dec 1-2 (next week) (jpena, 15:31:08) * check EOW if we are still on target for the ocata1 testday (apevec, 15:50:16) * Chair for next meeting (jpena, 15:54:40) * ACTION: amoralej to chair next meeting (jpena, 15:55:44) * open floor (jpena, 15:55:51) Meeting ended at 15:56:58 UTC. Action Items ------------ * number80 tag puppet4 in ocata * amoralej to chair next meeting Action Items, by person ----------------------- * amoralej * amoralej to chair next meeting * number80 * number80 tag puppet4 in ocata * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * amoralej (50) * jpena (42) * number80 (32) * apevec (31) * EmilienM (29) * jruzicka (18) * rbowen (16) * dmellado (6) * zodbot (6) * openstack (3) * misc (1) * rdobot (1) * chandankumar (1) * pabelanger (1) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot From javier.pena at redhat.com Wed Nov 23 16:44:55 2016 From: javier.pena at redhat.com (Javier Pena) Date: Wed, 23 Nov 2016 11:44:55 -0500 (EST) Subject: [rdo-list] CentOS dojo at FOSDEM In-Reply-To: References: Message-ID: <1842023596.2131472.1479919495391.JavaMail.zimbra@redhat.com> ----- Original Message ----- > To followup from my earlier email about an RDO event at FOSDEM, here's > some additional information. > > In short, the story is that we (RDO) would be participating as the Cloud > SIG, as part of the main "track", rather than a dedicated RDO room as > last year. > > Does this work for everyone here? > That would give us a broader audience, not just the usual RDO people, so it works for me. Javier > --Rich > > > > -------- Forwarded Message -------- > Subject: Re: [CentOS-promo] Daniel Paulus taking lead on org for bxl Dojo > Date: Wed, 23 Nov 2016 15:58:53 +0000 > From: Karanbir Singh > Reply-To: Events, gatherings and meetings > To: centos-promo at centos.org > > On 23/11/16 15:02, Rich Bowen wrote: > > > > > > On 11/22/2016 10:09 AM, Karanbir Singh wrote: > >> hi > >> > >> Daniel Paulus has offered to help with the organization and be a point > >> of contact for the Brussels Dojo in Feb 3rd 2017. The rest of the team > >> will still be supporting his efforts through the process. > >> > >> > > >> > >> Please join me in welcoming the expanding CentOS Dojo organization > >> group, and I look forward to seeing everyone in Feb. > > > > Thanks, Daniel, et al. Is it your desire that the RDO community be > > included in this event again, as in the last two years? If so, we're > > going to need a room - like last year - and I'll coordinate our > > speaker/presentation schedule on our side. Ok? > > I think it would be nice to have the SIG's all integrate into the > consolidated program rather than have dedicated tracks. Would something > like that work ? It would certainly generate more interest for content. > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From dave.johnston at icloud.com Wed Nov 23 17:49:40 2016 From: dave.johnston at icloud.com (Dave Johnston) Date: Wed, 23 Nov 2016 17:49:40 +0000 (GMT) Subject: [rdo-list] ODL Integration with RDO Newton Message-ID: Thanks for the response.? Can you attach the ml2_conf.ini ? Actually I seemed to have got past this. ?I did a fresh install. ?Previously I installed networking_odl from pip, and then from rpms. ?This must have caused some sort of issue/conflict. I eventually got round it by merging the contents of entry_points.txt for neutron and networking_odl. ?However I also tried a fresh install. ?This time I just installed networking_odl?directly from the rpms in the RDO repo, and Openstack successfully loads the driver. I'm just trying to figure out how to setup external network access now with ODL, but that's different problem. ? ? Hi, I'm attempting to integrate ODL with Newton. I'm hitting an issue when I start the neutron-server, it reports that it can't load the opendaylight driver. ERROR stevedore.extension [-] Could not load 'opendaylight': No module named opendaylight.driver I've installed the networking_odl package that contains the driver, is anyone aware of other changes that might be needed? -------------- next part -------------- An HTML attachment was scrubbed... URL: From apevec at redhat.com Wed Nov 23 18:33:54 2016 From: apevec at redhat.com (Alan Pevec) Date: Wed, 23 Nov 2016 19:33:54 +0100 Subject: [rdo-list] CentOS dojo at FOSDEM In-Reply-To: References: Message-ID: > In short, the story is that we (RDO) would be participating as the Cloud > SIG, as part of the main "track", rather than a dedicated RDO room as > last year. > > Does this work for everyone here? Makes sense, +1 from me! Cheers, Alan From adarazs at redhat.com Thu Nov 24 16:04:16 2016 From: adarazs at redhat.com (Attila Darazs) Date: Thu, 24 Nov 2016 17:04:16 +0100 Subject: [rdo-list] [CI] tripleo-quickstart-extra's roles migrated from gerrithub to openstack In-Reply-To: References: Message-ID: On 11/22/2016 02:49 PM, Wesley Hayutin wrote: > Greetings, > > TLDR version, tripleo-quickstart-roles have moved to the openstack > upstream source control sytem. > > Details: > > All the additional git and gerrit repos used with tripleo-quickstart > that have formally been housed in [1] gerrithub and > github/redhat-openstack have been migrated to the upstream openstack git > and gerrit [2]. Additional details regarding the move can be found here > [3]. There are a few roles that were still being incubated in > gerrithub that will be added one at a time via the review process. > > Additional details regarding the transition of tripleo-quickstart and > tripleo-quickstart-extras to upstream can be found here [4] Follow-up: The new -extras repo is now referenced in quickstart's quickstart-extras-requirements.txt file, so it will be automatically used in new full job runs. There were a couple of role renamings that might cause failures in jobs. If you find any such error, let me know. There are also working gating jobs on this new repo, covered by the CentOS CI infrastructure. Best regards, Attila > Thank you! > > > [1] > https://review.gerrithub.io > https://github.com/redhat-openstack/?utf8=%E2%9C%93&q=ansible-role-tripleo&type=&language= > > [2] > http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/ > https://github.com/openstack/tripleo-quickstart-extras > > [3] https://paste.fedoraproject.org/488434/79820118/ > > [4] https://trello.com/b/HhXlqdiu/rdo?menu=filter&filter=[Q > > > > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com > From pabelanger at redhat.com Thu Nov 24 16:43:41 2016 From: pabelanger at redhat.com (Paul Belanger) Date: Thu, 24 Nov 2016 11:43:41 -0500 Subject: [rdo-list] [CI] tripleo-quickstart-extra's roles migrated from gerrithub to openstack In-Reply-To: References: Message-ID: <20161124164341.GB3789@localhost.localdomain> On Thu, Nov 24, 2016 at 05:04:16PM +0100, Attila Darazs wrote: > On 11/22/2016 02:49 PM, Wesley Hayutin wrote: > > Greetings, > > > > TLDR version, tripleo-quickstart-roles have moved to the openstack > > upstream source control sytem. > > Well done, I know this was something that was discussed in Barcelona. I'm happy to see us pushing more things upstream to OpenStack CI. In return, I've created a patch[5] to address some linting issues with your playbooks / roles. [5] https://review.openstack.org/402142 > > Details: > > > > All the additional git and gerrit repos used with tripleo-quickstart > > that have formally been housed in [1] gerrithub and > > github/redhat-openstack have been migrated to the upstream openstack git > > and gerrit [2]. Additional details regarding the move can be found here > > [3]. There are a few roles that were still being incubated in > > gerrithub that will be added one at a time via the review process. > > > > Additional details regarding the transition of tripleo-quickstart and > > tripleo-quickstart-extras to upstream can be found here [4] > > Follow-up: The new -extras repo is now referenced in quickstart's > quickstart-extras-requirements.txt file, so it will be automatically used in > new full job runs. > > There were a couple of role renamings that might cause failures in jobs. If > you find any such error, let me know. > > There are also working gating jobs on this new repo, covered by the CentOS > CI infrastructure. > > Best regards, > Attila > > > Thank you! > > > > > > [1] > > https://review.gerrithub.io > > https://github.com/redhat-openstack/?utf8=%E2%9C%93&q=ansible-role-tripleo&type=&language= > > > > [2] > > http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/ > > https://github.com/openstack/tripleo-quickstart-extras > > > > [3] https://paste.fedoraproject.org/488434/79820118/ > > > > [4] https://trello.com/b/HhXlqdiu/rdo?menu=filter&filter=[Q > > > > > > > > > > > > _______________________________________________ > > rdo-list mailing list > > rdo-list at redhat.com > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com > > > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From emilien at redhat.com Thu Nov 24 23:45:10 2016 From: emilien at redhat.com (Emilien Macchi) Date: Thu, 24 Nov 2016 18:45:10 -0500 Subject: [rdo-list] Puppet OpenStack CI has been promoted! Message-ID: After 11 days of blockers and bugs everywhere, we now have a promotion. Special kudos to Alfredo and Ihar, we couldn't promote without your help. Thanks again guys! -- Emilien Macchi From nhicher at redhat.com Fri Nov 25 13:35:53 2016 From: nhicher at redhat.com (Nicolas Hicher) Date: Fri, 25 Nov 2016 08:35:53 -0500 Subject: [rdo-list] Scheduled maintenance of review.rdoproject.org: 2016-11-30 12:00 pm UTC Message-ID: Hello folks, We plan to upgrade review.rdoproject.org on 2016-11-30 12:00 pm UTC. The downtime will be about 1 hour approximately. This is a maintenance upgrade to the stable version of Software Factory 2.2.6. Please have a look to the changelog [1] for improvements. Regards, Software Factory Team, on behalf of rdo-infra [1] https://github.com/redhat-cip/software-factory/blob/master/CHANGELOG.md From rasca at redhat.com Fri Nov 25 15:02:37 2016 From: rasca at redhat.com (Raoul Scarazzini) Date: Fri, 25 Nov 2016 16:02:37 +0100 Subject: [rdo-list] [TripleO] Backport a master patch to newton Message-ID: <5a4a2a27-1d6f-266b-d6d2-074bbe678ef0@redhat.com> Hi everybody, I need some advice to complete the process of making a patch hit a package in the actual newton/stable. Everything starts from this bug [1] which was solved by this review [2] made by Mike and already merged in master [3]. I discovered the bug in newton, so I need to port the patch there and I already cherry-picked a review for this [4]. Now, even the modification is already merged in master it is not yet part of the package python2-oslo-db, at least not in the last repo produced [5] (I merely downloaded the package and looked for the modification, maybe there's a smart way). So I'm trying to understand what else needs to be done here. Following what Alan wrote in the thread "[TripleO] Newton large baremetal deployment issues" talking about a very similar issue, I need to: * request oslo.db minor release on stable/newton * bump it in upstream stable/newton upper-constraints.txt * sync that bump in rdoinfo (should be automatic now) But (if all of the above is sufficient to obtain what I need) I don't know how to request that minor release and neither where to find upper-constraints.txt. Can you please help me to share some light around all this stuff? Many thanks! [1] https://bugs.launchpad.net/tripleo/+bug/1642944 [2] https://review.openstack.org/#/c/400269/ [3] https://github.com/openstack/oslo.db/commit/34f9a3ac7a56883f8a2cd2a9a93bc42e5194bc1e [4] https://review.openstack.org/#/c/402669/ [5] https://trunk.rdoproject.org/centos7-master/73/a9/73a959163274d13a406609ee02d6e9e60534bf01_65436912/ -- Raoul Scarazzini rasca at redhat.com From dms at redhat.com Fri Nov 25 16:23:54 2016 From: dms at redhat.com (David Moreau Simard) Date: Fri, 25 Nov 2016 11:23:54 -0500 Subject: [rdo-list] Puppet OpenStack CI has been promoted! In-Reply-To: References: Message-ID: RDO CI was just promoted manually and it will be visible on the CDN shortly. We decided to promote because the only failure was actually a timeout that occurred during the log collection, the actual installation and tests passed successfully. David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Thu, Nov 24, 2016 at 6:45 PM, Emilien Macchi wrote: > After 11 days of blockers and bugs everywhere, we now have a promotion. > > Special kudos to Alfredo and Ihar, we couldn't promote without your > help. Thanks again guys! > -- > Emilien Macchi > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From apevec at redhat.com Fri Nov 25 16:48:49 2016 From: apevec at redhat.com (Alan Pevec) Date: Fri, 25 Nov 2016 17:48:49 +0100 Subject: [rdo-list] [TripleO] Backport a master patch to newton In-Reply-To: <5a4a2a27-1d6f-266b-d6d2-074bbe678ef0@redhat.com> References: <5a4a2a27-1d6f-266b-d6d2-074bbe678ef0@redhat.com> Message-ID: > Now, even the modification is already merged in master it is not yet > part of the package python2-oslo-db, at least not in the last repo > produced [5] (I merely downloaded the package and looked for the > modification, maybe there's a smart way). You can look at package NVR: https://trunk.rdoproject.org/centos7-master/73/a9/73a959163274d13a406609ee02d6e9e60534bf01_65436912/python-oslo-db-4.14.0-0.20161023004034.21a5c42.el7.centos.src.rpm Release part is 0.RPMBUILDTIMESTAMP.UPSTREAM_SHORT_GIT_HASH So in this case it is built from upstream commit https://github.com/openstack/oslo.db/commit/21a5c42 which is 4.14.0 tag. Fix you are looking for has not been released yet even on master. > how to request that minor release and neither where to find > upper-constraints.txt. Bumping upper-constraints.txt is done as a part of upstream release process: https://review.openstack.org/#/q/topic:new-release Requesting minor release is normally job of PTL or Release Management Liaison, for Oslo harlowja https://wiki.openstack.org/wiki/CrossProjectLiaisons#Release_management but you can make it easier on them by preparing the release request yourself and ask them to review it see README in openstack/releases repo and example https://review.openstack.org/398947 Cheers, Alan From amoralej at redhat.com Fri Nov 25 17:04:56 2016 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Fri, 25 Nov 2016 18:04:56 +0100 Subject: [rdo-list] Getting ready for RDO on Centos 7.3 Message-ID: Hi, In the last days we've been running our RDO-CI pipelines using a pre-release of the incoming Centos 7.3 release. We'd like to share with the RDO community some issues we've being facing and some recommendations to be prepared for this new release: 1. package mariadb-libs-5.5.52 included in CentOS 7.3 conflicts wiht mariadb packages included in RDO repositories. Typically, this package may be in the OS base installation before starting OpenStack deployment, what can lead to errors while installing mariadb-server from RDO. To avoid this, update your system right after enabling RDO repositories in your servers and before starting OpenStack services deployment. 2. The versions of qemu-kvm-ev and openstack-selinux provided by VirtSIG and CloudSIG at 7.3 release will require other packages in CentOS 7.3. If you try to install them without updated repos enabled, you'll get some errors. The recommendation from RDO team is to move to Centos 7.3 based installations to deploy RDO as soon as possible after it's officially released. This will ensure you are using a configuration already tested in RDO CI. If this is not possible for any reason, make sure you have the official Centos 7 repos enabled and available in your server. This repo will provide requirements needed. By following these guidelines we expect a smooth transition to Centos 7.3 for RDO users. Don't hesitate to contact #rdo at freenode or this mail list for any question you may have. Best regards, Alfredo From dms at redhat.com Fri Nov 25 17:17:07 2016 From: dms at redhat.com (David Moreau Simard) Date: Fri, 25 Nov 2016 12:17:07 -0500 Subject: [rdo-list] Getting ready for RDO on Centos 7.3 In-Reply-To: References: Message-ID: I think it'd be great to specify if there are certain releases affected more than others (i.e, mitaka vs newton vs ocata). David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Fri, Nov 25, 2016 at 12:04 PM, Alfredo Moralejo Alonso wrote: > Hi, > > In the last days we've been running our RDO-CI pipelines using a > pre-release of the incoming Centos 7.3 release. We'd like to share > with the RDO community some issues we've being facing and some > recommendations to be prepared for this new release: > > 1. package mariadb-libs-5.5.52 included in CentOS 7.3 conflicts wiht > mariadb packages included in RDO repositories. Typically, this package > may be in the OS base installation before starting OpenStack > deployment, what can lead to errors while installing mariadb-server > from RDO. To avoid this, update your system right after enabling RDO > repositories in your servers and before starting OpenStack services > deployment. > > 2. The versions of qemu-kvm-ev and openstack-selinux provided by > VirtSIG and CloudSIG at 7.3 release will require other packages in > CentOS 7.3. If you try to install them without updated repos enabled, > you'll get some errors. The recommendation from RDO team is to move to > Centos 7.3 based installations to deploy RDO as soon as possible after > it's officially released. This will ensure you are using a > configuration already tested in RDO CI. If this is not possible for > any reason, make sure you have the official Centos 7 repos enabled and > available in your server. This repo will provide requirements needed. > > By following these guidelines we expect a smooth transition to Centos > 7.3 for RDO users. Don't hesitate to contact #rdo at freenode or this > mail list for any question you may have. > > Best regards, > > Alfredo > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From amoralej at redhat.com Fri Nov 25 18:07:20 2016 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Fri, 25 Nov 2016 19:07:20 +0100 Subject: [rdo-list] Getting ready for RDO on Centos 7.3 In-Reply-To: References: Message-ID: On Fri, Nov 25, 2016 at 6:17 PM, David Moreau Simard wrote: > I think it'd be great to specify if there are certain releases > affected more than others (i.e, mitaka vs newton vs ocata). > Expected impact is the same for the three releases as the packages affected, mariadb, openstack-selinux and qemu-kvm-ev, are shared for all of them. Alfredo > David Moreau Simard > Senior Software Engineer | Openstack RDO > > dmsimard = [irc, github, twitter] > > > On Fri, Nov 25, 2016 at 12:04 PM, Alfredo Moralejo Alonso > wrote: >> Hi, >> >> In the last days we've been running our RDO-CI pipelines using a >> pre-release of the incoming Centos 7.3 release. We'd like to share >> with the RDO community some issues we've being facing and some >> recommendations to be prepared for this new release: >> >> 1. package mariadb-libs-5.5.52 included in CentOS 7.3 conflicts wiht >> mariadb packages included in RDO repositories. Typically, this package >> may be in the OS base installation before starting OpenStack >> deployment, what can lead to errors while installing mariadb-server >> from RDO. To avoid this, update your system right after enabling RDO >> repositories in your servers and before starting OpenStack services >> deployment. >> >> 2. The versions of qemu-kvm-ev and openstack-selinux provided by >> VirtSIG and CloudSIG at 7.3 release will require other packages in >> CentOS 7.3. If you try to install them without updated repos enabled, >> you'll get some errors. The recommendation from RDO team is to move to >> Centos 7.3 based installations to deploy RDO as soon as possible after >> it's officially released. This will ensure you are using a >> configuration already tested in RDO CI. If this is not possible for >> any reason, make sure you have the official Centos 7 repos enabled and >> available in your server. This repo will provide requirements needed. >> >> By following these guidelines we expect a smooth transition to Centos >> 7.3 for RDO users. Don't hesitate to contact #rdo at freenode or this >> mail list for any question you may have. >> >> Best regards, >> >> Alfredo >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list at redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com From hguemar at fedoraproject.org Mon Nov 28 15:00:04 2016 From: hguemar at fedoraproject.org (hguemar at fedoraproject.org) Date: Mon, 28 Nov 2016 15:00:04 +0000 (UTC) Subject: [rdo-list] [Fedocal] Reminder meeting : RDO meeting Message-ID: <20161128150004.4D5BE60A3FDC@fedocal02.phx2.fedoraproject.org> Dear all, You are kindly invited to the meeting: RDO meeting on 2016-11-30 from 15:00:00 to 16:00:00 UTC At rdo at irc.freenode.net The meeting will be about: RDO IRC meeting [Agenda at https://etherpad.openstack.org/p/RDO-Meeting ](https://etherpad.openstack.org/p/RDO-Meeting) Every Wednesday on #rdo on Freenode IRC Source: https://apps.fedoraproject.org/calendar/meeting/2017/ From rbowen at redhat.com Mon Nov 28 17:38:03 2016 From: rbowen at redhat.com (Rich Bowen) Date: Mon, 28 Nov 2016 12:38:03 -0500 Subject: [rdo-list] Upcoming meetups Message-ID: <1e60805a-e3f3-90aa-9d3c-6d5edd63e303@redhat.com> The following are the meetups I'm aware of in the coming week where OpenStack and/or RDO enthusiasts are likely to be present. If you know of others, please let me know, and/or add them to http://rdoproject.org/events If there's a meetup in your area, please consider attending. If you attend, please consider taking a few photos, and possibly even writing up a brief summary of what was covered. --Rich * Monday November 28 in Guadalajara, MX: Porqu? los desarrolladores de software est?n adoptando OpenStack - https://www.meetup.com/OpenStack-GDL/events/235809625/ * Monday November 28 in Dresden, DE: OpenStack-Meetup - https://www.meetup.com/OpenStack-Dresden/events/235817240/ * Tuesday November 29 in Canberra, AU: Canberra OpenStack meetup - https://www.meetup.com/Australian-OpenStack-User-Group/events/235562852/ * Tuesday November 29 in Tokyo, JP: ??OpenStack??????31???? - https://www.meetup.com/Japan-OpenStack-User-Group/events/235593184/ * Wednesday November 30 in Dublin, IE: Calling all you early risers - join us for a OpenStack breakfast morning - https://www.meetup.com/OpenStack-Ireland/events/232288557/ * Wednesday November 30 in Z?rich, CH: 14th Swiss OpenStack User Group Meetup - https://www.meetup.com/openstack-ch/events/235124491/ * Wednesday November 30 in San Diego, CA, US: Software Defined Network on OpenStack with PLUMgrid - https://www.meetup.com/OpenStack-SD/events/233829752/ * Wednesday November 30 in Sunnyvale, CA, US: Hands-on Workshop: OpenStack and Containers - https://www.meetup.com/openstack/events/235472541/ * Thursday December 01 in Berlin, DE: CodingBerlin 1.0 - OpenStack + Container Schedulers - see comments 4 Details! - https://www.meetup.com/CODING-BERLIN/events/234917996/ * Thursday December 01 in London, 17, GB: The December Meetup - High Performance Computing - https://www.meetup.com/Openstack-London/events/234178394/ * Saturday December 03 in Nanjing, CN: ??OpenStack?????Meetup 2016/12/03 - https://www.meetup.com/Cloud-Nanjing/events/235729698/ * Monday December 05 in Saint Paul, MN, US: OpenStack Developer Coffee and Code - https://www.meetup.com/Minnesota-OpenStack-Meetup/events/235416638/ * Monday December 05 in Islamabad, PK: Building a case for OpenStack in Pakistan - https://www.meetup.com/Openstack-Pakistan/events/235569537/ -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From rbowen at redhat.com Mon Nov 28 17:57:06 2016 From: rbowen at redhat.com (Rich Bowen) Date: Mon, 28 Nov 2016 12:57:06 -0500 Subject: [rdo-list] RDO Survey Results Message-ID: Thank you all for participating in our first "How are you using RDO?" survey. I've written up the results here: https://www.rdoproject.org/blog/2016/11/how-are-you-using-rdo/ Meanwhile, the longer form questions ("How do you participate in the community?" and "What's missing from RDO?") are going to take a little longer to summarize and write up. --Rich -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From snecklifter at gmail.com Mon Nov 28 18:28:22 2016 From: snecklifter at gmail.com (Christopher Brown) Date: Mon, 28 Nov 2016 18:28:22 +0000 Subject: [rdo-list] redhat-lsb-core missing Message-ID: Hello, Using latest Newton stable and building my own images, they appear to be missing redhat-lsb-core: Nov 28 18:21:33 localhost os-collect-config: ++ lsb_release -is Nov 28 18:21:33 localhost os-collect-config: /usr/libexec/os-refresh-config/configure.d/51-hosts: line 36: lsb_release: command not found Nov 28 18:21:33 localhost os-collect-config: ++ tr -s '[A-Z]' '[a-z]' Nov 28 18:21:33 localhost os-collect-config: + DIST= Nov 28 18:21:33 localhost os-collect-config: [2016-11-28 18:21:33,955] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] Nov 28 18:21:33 localhost os-collect-config: [2016-11-28 18:21:33,955] (os-refresh-config) [ERROR] Aborting... Nov 28 18:21:33 localhost os-collect-config: Command failed, will not cache new data. Command 'os-refresh-config --timeout 14400' returned non-zero exit status 1 Nov 28 18:21:33 localhost os-collect-config: Sleeping 1.00 seconds before re-exec. Installing this package allows it to progress past this stage. There is no Newton version in Red Hat bugzilla. Should I still file a bug? There or somewhere else? -- Christopher Brown -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbowen at redhat.com Mon Nov 28 19:17:20 2016 From: rbowen at redhat.com (Rich Bowen) Date: Mon, 28 Nov 2016 14:17:20 -0500 Subject: [rdo-list] ask.openstack.org: Unanswered RDO questions Message-ID: <8813cd4c-b550-6733-d1c3-9854bcb8473a@redhat.com> 31 unanswered questions: Why openstack packstack is not using ansible? https://ask.openstack.org/en/question/99648/why-openstack-packstack-is-not-using-ansible/ Tags: openstack-packstack, packstack RDO newton release has support for RHEL 7.3 ? https://ask.openstack.org/en/question/99201/rdo-newton-release-has-support-for-rhel-73/ Tags: rdo, rhel, openstack CPU Load on instance and Controller is very high and unresponsive https://ask.openstack.org/en/question/98927/cpu-load-on-instance-and-controller-is-very-high-and-unresponsive/ Tags: mitaka-openstack [Error: No valid host was found.((Network br-ex error)) https://ask.openstack.org/en/question/98818/error-no-valid-host-was-foundnetwork-br-ex-error/ Tags: br-ex Compute nodes are not visible after controller node rebooted https://ask.openstack.org/en/question/98654/compute-nodes-are-not-visible-after-controller-node-rebooted/ Tags: rdo Live migration failing with qemu error "does not support drive-mirror" https://ask.openstack.org/en/question/98144/live-migration-failing-with-qemu-error-does-not-support-drive-mirror/ Tags: newton, nova, qemu, kvm, block-live-migration error while launching instance in openstack 'newton' using rdo packstack https://ask.openstack.org/en/question/98113/error-while-launching-instance-in-openstack-newton-using-rdo-packstack/ Tags: rdo, newton, packstack How can I add a new region to an installed All-In-One RDO openstack? https://ask.openstack.org/en/question/98111/how-can-i-add-a-new-region-to-an-installed-all-in-one-rdo-openstack/ Tags: rdo, all-in-one, multi-region, region, add-region How to setup solr search engine for openstack swift https://ask.openstack.org/en/question/98093/how-to-setup-solr-search-engine-for-openstack-swift/ Tags: openstack, openstack-swift, newton, rdo Heat fails with 504 error. https://ask.openstack.org/en/question/98092/heat-fails-with-504-error/ Tags: rdo, tripleo, mitaka, heat Devstack and OpenvSwitch v2.3+ https://ask.openstack.org/en/question/98004/devstack-and-openvswitch-v23/ Tags: devstack, openvswitch How how does icmp packet travel across two compute nodes without br-int and br-tun running? https://ask.openstack.org/en/question/97905/how-how-does-icmp-packet-travel-across-two-compute-nodes-without-br-int-and-br-tun-running/ Tags: neutron-ovs-pktflow "Parameter outiface failed on Firewall" during installation of openstack rdo on centos 7 https://ask.openstack.org/en/question/95657/parameter-outiface-failed-on-firewall-during-installation-of-openstack-rdo-on-centos-7/ Tags: rdo, devstack#mitaka multi nodes provider network ovs config https://ask.openstack.org/en/question/95423/multi-nodes-provider-network-ovs-config/ Tags: rdo, liberty-neutron Adding additional packages to an RDO installation https://ask.openstack.org/en/question/95380/adding-additional-packages-to-an-rdo-installation/ Tags: rdo, mistral RDO - is there any fedora package newer than puppet-4.2.1-3.fc24.noarch.rpm https://ask.openstack.org/en/question/94969/rdo-is-there-any-fedora-package-newer-than-puppet-421-3fc24noarchrpm/ Tags: rdo, puppet, install-openstack how to deploy haskell-distributed in RDO? https://ask.openstack.org/en/question/94785/how-to-deploy-haskell-distributed-in-rdo/ Tags: rdo rdo tripleO liberty undercloud install failing https://ask.openstack.org/en/question/94023/rdo-tripleo-liberty-undercloud-install-failing/ Tags: rdo, rdo-manager, liberty, undercloud, instack Liberty RDO: stack resource topology icons are pink https://ask.openstack.org/en/question/91347/liberty-rdo-stack-resource-topology-icons-are-pink/ Tags: resource, topology, dashboard, horizon, pink No handlers could be found for logger "oslo_config.cfg" while syncing the glance database https://ask.openstack.org/en/question/91169/no-handlers-could-be-found-for-logger-oslo_configcfg-while-syncing-the-glance-database/ Tags: liberty, glance, install-openstack CentOS OpenStack - compute node can't talk https://ask.openstack.org/en/question/88989/centos-openstack-compute-node-cant-talk/ Tags: rdo How to setup SWIFT_PROXY_NODE and SWIFT_STORAGE_NODEs separately on RDO Liberty ? https://ask.openstack.org/en/question/88897/how-to-setup-swift_proxy_node-and-swift_storage_nodes-separately-on-rdo-liberty/ Tags: rdo, liberty, swift, ha VM and container can't download anything from internet https://ask.openstack.org/en/question/88338/vm-and-container-cant-download-anything-from-internet/ Tags: rdo, neutron, network, connectivity Fedora22, Liberty, horizon VNC console and keymap=sv with ; and/ https://ask.openstack.org/en/question/87451/fedora22-liberty-horizon-vnc-console-and-keymapsv-with-and/ Tags: keyboard, map, keymap, vncproxy, novnc OpenStack-Docker driver failed https://ask.openstack.org/en/question/87243/openstack-docker-driver-failed/ Tags: docker, openstack, liberty Routing between two tenants https://ask.openstack.org/en/question/84645/routing-between-two-tenants/ Tags: kilo, fuel, rdo, routing openstack baremetal introspection internal server error https://ask.openstack.org/en/question/82790/openstack-baremetal-introspection-internal-server-error/ Tags: rdo, ironic-inspector, tripleo Installing openstack using packstack (rdo) failed https://ask.openstack.org/en/question/82473/installing-openstack-using-packstack-rdo-failed/ Tags: rdo, packstack, installation-error, keystone VMware Host Backend causes No valid host was found. Bug ??? https://ask.openstack.org/en/question/79738/vmware-host-backend-causes-no-valid-host-was-found-bug/ Tags: vmware, rdo Mutlinode Devstack with two interfaces https://ask.openstack.org/en/question/78615/mutlinode-devstack-with-two-interfaces/ Tags: devstack, vlan, openstack -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From bderzhavets at hotmail.com Mon Nov 28 21:03:34 2016 From: bderzhavets at hotmail.com (Boris Derzhavets) Date: Mon, 28 Nov 2016 21:03:34 +0000 Subject: [rdo-list] RDO Survey Results In-Reply-To: References: Message-ID: I do realize that Core RDO deployment tool is TripleO ( TripleO QS ) been pushed and stable on bare metal. However, numbers in report bellow:- 38% packstack deployments vs 20% TripleO deployments fairly prove that just for Newton && Ocata releases makes sense to have storage.pp role for packstack. TripleO is exciting , but maintain prod environment via update of heat stack "overcloud" targeting for instance :- 1. Adding one more compute node 2. Replacing Controller node in PCS Cluster would scare too much admins. Regression in packstack been done ahead of right time might have negative drawback and it would. Thanks. Boris. ________________________________ From: rdo-list-bounces at redhat.com on behalf of Rich Bowen Sent: Monday, November 28, 2016 8:57 PM To: rdo-list at redhat.com Subject: [rdo-list] RDO Survey Results Thank you all for participating in our first "How are you using RDO?" survey. I've written up the results here: https://www.rdoproject.org/blog/2016/11/how-are-you-using-rdo/ How are you using RDO? (Survey results) www.rdoproject.org Over the last few weeks, we've been conducting a survey of RDO users, to get an idea of who is using RDO, how, and for what. While the sample size here is small, it's a start at understanding the makeup... Meanwhile, the longer form questions ("How do you participate in the community?" and "What's missing from RDO?") are going to take a little longer to summarize and write up. --Rich -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org RDO rdoproject.org RDO is a community of people using and deploying OpenStack on CentOS, Fedora, and Red Hat Enterprise Linux. Try RDO Now ? @RDOCommunity _______________________________________________ rdo-list mailing list rdo-list at redhat.com https://www.redhat.com/mailman/listinfo/rdo-list rdo-list Info Page - Red Hat www.redhat.com The rdo-list mailing list provides a forum for discussions about installing, running, and using OpenStack on Red Hat based distributions. To see the collection of ... To unsubscribe: rdo-list-unsubscribe at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Tue Nov 29 08:55:10 2016 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Tue, 29 Nov 2016 09:55:10 +0100 Subject: [rdo-list] redhat-lsb-core missing In-Reply-To: References: Message-ID: The bug is reported in https://bugs.launchpad.net/diskimage-builder/+bug/1641106 . This issue was fixed in master branch in https://review.openstack.org/#/c/396759/ and i've just requested to backport it to newton but it may take some time to land in stable repo. Best regards, Alfredo On Mon, Nov 28, 2016 at 7:28 PM, Christopher Brown wrote: > Hello, > > Using latest Newton stable and building my own images, they appear to be > missing redhat-lsb-core: > > Nov 28 18:21:33 localhost os-collect-config: ++ lsb_release -is > Nov 28 18:21:33 localhost os-collect-config: > /usr/libexec/os-refresh-config/configure.d/51-hosts: line 36: lsb_release: > command not found > Nov 28 18:21:33 localhost os-collect-config: ++ tr -s '[A-Z]' '[a-z]' > Nov 28 18:21:33 localhost os-collect-config: + DIST= > Nov 28 18:21:33 localhost os-collect-config: [2016-11-28 18:21:33,955] > (os-refresh-config) [ERROR] during configure phase. [Command > '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned > non-zero exit status 1] > Nov 28 18:21:33 localhost os-collect-config: [2016-11-28 18:21:33,955] > (os-refresh-config) [ERROR] Aborting... > Nov 28 18:21:33 localhost os-collect-config: Command failed, will not cache > new data. Command 'os-refresh-config --timeout 14400' returned non-zero exit > status 1 > Nov 28 18:21:33 localhost os-collect-config: Sleeping 1.00 seconds before > re-exec. > > Installing this package allows it to progress past this stage. > > There is no Newton version in Red Hat bugzilla. Should I still file a bug? > There or somewhere else? > > -- > Christopher Brown > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From snecklifter at gmail.com Tue Nov 29 09:26:05 2016 From: snecklifter at gmail.com (Christopher Brown) Date: Tue, 29 Nov 2016 09:26:05 +0000 Subject: [rdo-list] redhat-lsb-core missing In-Reply-To: References: Message-ID: Thanks. I can use virt-customize as a workaround. On 29 November 2016 at 08:55, Alfredo Moralejo Alonso wrote: > The bug is reported in > https://bugs.launchpad.net/diskimage-builder/+bug/1641106 . This issue > was fixed in master branch in https://review.openstack.org/#/c/396759/ > and i've just requested to backport it to newton but it may take some > time to land in stable repo. > > Best regards, > > Alfredo > > > On Mon, Nov 28, 2016 at 7:28 PM, Christopher Brown > wrote: > > Hello, > > > > Using latest Newton stable and building my own images, they appear to be > > missing redhat-lsb-core: > > > > Nov 28 18:21:33 localhost os-collect-config: ++ lsb_release -is > > Nov 28 18:21:33 localhost os-collect-config: > > /usr/libexec/os-refresh-config/configure.d/51-hosts: line 36: > lsb_release: > > command not found > > Nov 28 18:21:33 localhost os-collect-config: ++ tr -s '[A-Z]' '[a-z]' > > Nov 28 18:21:33 localhost os-collect-config: + DIST= > > Nov 28 18:21:33 localhost os-collect-config: [2016-11-28 18:21:33,955] > > (os-refresh-config) [ERROR] during configure phase. [Command > > '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' > returned > > non-zero exit status 1] > > Nov 28 18:21:33 localhost os-collect-config: [2016-11-28 18:21:33,955] > > (os-refresh-config) [ERROR] Aborting... > > Nov 28 18:21:33 localhost os-collect-config: Command failed, will not > cache > > new data. Command 'os-refresh-config --timeout 14400' returned non-zero > exit > > status 1 > > Nov 28 18:21:33 localhost os-collect-config: Sleeping 1.00 seconds before > > re-exec. > > > > Installing this package allows it to progress past this stage. > > > > There is no Newton version in Red Hat bugzilla. Should I still file a > bug? > > There or somewhere else? > > > > -- > > Christopher Brown > > > > _______________________________________________ > > rdo-list mailing list > > rdo-list at redhat.com > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com > -- Christopher Brown -------------- next part -------------- An HTML attachment was scrubbed... URL: From alon.dotan at hpe.com Tue Nov 29 13:53:14 2016 From: alon.dotan at hpe.com (Dotan, Alon) Date: Tue, 29 Nov 2016 13:53:14 +0000 Subject: [rdo-list] tripleo dpdk deployment Message-ID: Hey All, I'm trying to deploy ovs-dpdk with tripleo, I need to add intel_iommu=on kernel param to grub configuration, Need to build special image for that or there is some way to do so via the deployment configuration? Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbowen at redhat.com Tue Nov 29 17:26:45 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 29 Nov 2016 12:26:45 -0500 Subject: [rdo-list] Call for talks - FOSDEM CentOS Dojo Message-ID: <17563974-8775-1fe9-1268-569029244e42@redhat.com> TL;DR: Call for RDO talks at https://etherpad.openstack.org/p/fosdem-dojo-2017 It appears that the CentOS dojo in Brussels, on Friday, Feb 3rd, prior to FOSDEM, won't have any kind of a formal call for papers. However, I've been asked to propose a list of RDO talks that can be interleaved into the talks coming from other SIGs. Talks should focus on RDO's interface with CentOS - which of course leaves it pretty wide open. KB says: >>>> eg. the contributor process via cbs.centos.org / ci.centos.org etc will be common, so we need not think that through as a RDO specific talk. We are trying to drive to a tutorial / demo / user learning setup as much as possible - with the aim that folks leaving should go away empowered to execute something right away ( as opposed to general talks / intro sessions / talking about features etc ). <<<< To this end, if you expect to be at FOSDEM, and can come in a day early, and are willing to speak, please add your name and topic to https://etherpad.openstack.org/p/fosdem-dojo-2017 by the end of this week, so that we can present a list of topics to them next week. Thanks! -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From hguemar at fedoraproject.org Tue Nov 29 17:48:16 2016 From: hguemar at fedoraproject.org (=?UTF-8?Q?Ha=C3=AFkel?=) Date: Tue, 29 Nov 2016 18:48:16 +0100 Subject: [rdo-list] Call for talks - FOSDEM CentOS Dojo In-Reply-To: <17563974-8775-1fe9-1268-569029244e42@redhat.com> References: <17563974-8775-1fe9-1268-569029244e42@redhat.com> Message-ID: Le 29 nov. 2016 6:26 PM, "Rich Bowen" a ?crit : > > TL;DR: Call for RDO talks at > https://etherpad.openstack.org/p/fosdem-dojo-2017 > > It appears that the CentOS dojo in Brussels, on Friday, Feb 3rd, prior > to FOSDEM, won't have any kind of a formal call for papers. However, > I've been asked to propose a list of RDO talks that can be interleaved > into the talks coming from other SIGs. > > Talks should focus on RDO's interface with CentOS - which of course > leaves it pretty wide open. > > KB says: > > >>>> > eg. the contributor process via > cbs.centos.org / ci.centos.org etc will be common, so we need not > think that through as a RDO specific talk. > > We are trying to drive to a tutorial / demo / user learning setup as > much as possible - with the aim that folks leaving should go away > empowered to execute something right away ( as opposed to general > talks / intro sessions / talking about features etc ). > <<<< > > To this end, if you expect to be at FOSDEM, and can come in a day early, > and are willing to speak, please add your name and topic to > https://etherpad.openstack.org/p/fosdem-dojo-2017 by the end of this > week, so that we can present a list of topics to them next week. > > Thanks! > > -- > Rich Bowen - rbowen at redhat.com > RDO Community Liaison > http://rdoproject.org > @RDOCommunity > > I'll make sure to submit a talk and I will be around the whole day. Regards, H. _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbowen at redhat.com Tue Nov 29 18:34:50 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 29 Nov 2016 13:34:50 -0500 Subject: [rdo-list] Speaker list Message-ID: <6e1991f8-d8b8-5403-b92a-e8d4b6b57ec6@redhat.com> I'm occasionally approached by Meetup organizers looking for local(ish) speakers for their meetups. At one point we had a Subject Matter Experts list somewhere, but it has fallen far out of date. And I can no longer keep in my mind where you are all geographically. I'd like to take another stab at a simple "speaker bureau" page, where people willing to speak on RDO topics can indicate what topics/talks they're willing to do, and what their single day or overnight travel radius might be. I know that the OpenStack Foundation has a speaker bureau already - https://www.openstack.org/community/speakers/ - and I encourage you to add your info there. (I see just 9 of you listed at https://www.openstack.org/community/speakers/results?search_query=RDO right now.) But for RDO-specific speakers, that I can recommend to meetup organizers, I can also sometimes help out with minimal travel/lodging funding, as well as sending swag. For this purpose, if you'd like to be on this list, please submit a pull request for https://www.rdoproject.org/community/speakers/ to add yourself. (This page isn't linked from anywhere yet, and won't be until there's a few more names on it.) Thanks. --Rich -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From dms at redhat.com Tue Nov 29 19:48:28 2016 From: dms at redhat.com (David Moreau Simard) Date: Tue, 29 Nov 2016 14:48:28 -0500 Subject: [rdo-list] Speaker list In-Reply-To: <6e1991f8-d8b8-5403-b92a-e8d4b6b57ec6@redhat.com> References: <6e1991f8-d8b8-5403-b92a-e8d4b6b57ec6@redhat.com> Message-ID: Hmm, I went ahead and added "RDO" in my "areas of expertise" so that your query can find me. Shame there's only five fields for "areas of expertise" :) David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Tue, Nov 29, 2016 at 1:34 PM, Rich Bowen wrote: > I'm occasionally approached by Meetup organizers looking for local(ish) > speakers for their meetups. At one point we had a Subject Matter Experts > list somewhere, but it has fallen far out of date. And I can no longer > keep in my mind where you are all geographically. > > I'd like to take another stab at a simple "speaker bureau" page, where > people willing to speak on RDO topics can indicate what topics/talks > they're willing to do, and what their single day or overnight travel > radius might be. > > I know that the OpenStack Foundation has a speaker bureau already - > https://www.openstack.org/community/speakers/ - and I encourage you to > add your info there. (I see just 9 of you listed at > https://www.openstack.org/community/speakers/results?search_query=RDO > right now.) > > But for RDO-specific speakers, that I can recommend to meetup > organizers, I can also sometimes help out with minimal travel/lodging > funding, as well as sending swag. For this purpose, if you'd like to be > on this list, please submit a pull request for > https://www.rdoproject.org/community/speakers/ to add yourself. > > (This page isn't linked from anywhere yet, and won't be until there's a > few more names on it.) > > Thanks. > > --Rich > > > -- > Rich Bowen - rbowen at redhat.com > RDO Community Liaison > http://rdoproject.org > @RDOCommunity > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com From rbowen at redhat.com Tue Nov 29 21:24:38 2016 From: rbowen at redhat.com (Rich Bowen) Date: Tue, 29 Nov 2016 16:24:38 -0500 Subject: [rdo-list] RDO Survey - what we need to improve Message-ID: <0fde6466-c296-d492-aa46-184acae96d1e@redhat.com> One of the free-form questions on the RDO survey was "What application or service are you missing in RDO?" The answers are all across the board, with installer being the most commonly-repeated complaint, and containers falling close behind. As there's no simple way to summarize the answers, I've just put them all here: https://etherpad.openstack.org/p/rdo-survey-answers -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From apevec at redhat.com Wed Nov 30 09:19:04 2016 From: apevec at redhat.com (Alan Pevec) Date: Wed, 30 Nov 2016 10:19:04 +0100 Subject: [rdo-list] RDO Survey Results In-Reply-To: References: Message-ID: Hi Boris, thanks for the feedback, I'll try to address them inline: On Mon, Nov 28, 2016 at 10:03 PM, Boris Derzhavets wrote: > I do realize that Core RDO deployment tool is TripleO ( TripleO QS ) been > pushed and stable on bare metal. TripleO is one of the deployment tools which work with RDO packages, it never was and never will be the only one, so you have a choice. > However, numbers in report bellow:- 38% packstack deployments vs 20% TripleO > deployments > fairly prove that just for Newton && Ocata releases makes sense to have > storage.pp role for packstack. Sample was rather small so I would not jump to the conclusion, but just to clarify that Packstack is a community supported project and you can report the request details in https://bugs.launchpad.net/packstack and/or propose a patch in gerrit then we can discuss how it can fit. Maybe feature does not fit in Packstack directly e.g. cloud be integrated with some other tool, let's keep open mind and not assume implementation details. As an example, recent Magnum support in Packstack was contributed from outside Red Hat. > TripleO is exciting , but maintain prod environment via update of heat stack > "overcloud" targeting for instance :- > 1. Adding one more compute node > 2. Replacing Controller node in PCS Cluster > would scare too much admins. Can you provide more details, I'm sure Emilien as TripleO PTL would like to hear them :) > Regression in packstack been done ahead of right time might have negative > drawback and it would. I guess you mean focusing Packstack to be excellent all-in-one proof-of-concept installation tool? Repeating myself again: if we don't have it covered in CI, it will be broken by design so focusing is all about _avoiding_ regressions. E.g. we have found out today that Nagios support in Packstack is broken only after being reported on #rdo ... because it is not covered in CI. To fix that, I would propose to drop Nagios support in Packstack and instead look at integrating with CentOS Opstools SIG. Cheers, Alan From bderzhavets at hotmail.com Wed Nov 30 13:23:09 2016 From: bderzhavets at hotmail.com (Boris Derzhavets) Date: Wed, 30 Nov 2016 13:23:09 +0000 Subject: [rdo-list] RDO Survey Results In-Reply-To: References: , Message-ID: ________________________________ From: Alan Pevec Sent: Wednesday, November 30, 2016 12:19 PM To: Boris Derzhavets Cc: rdo-list at redhat.com; Emilien Macchi Subject: Re: [rdo-list] RDO Survey Results Hi Boris, thanks for the feedback, I'll try to address them inline: On Mon, Nov 28, 2016 at 10:03 PM, Boris Derzhavets wrote: > I do realize that Core RDO deployment tool is TripleO ( TripleO QS ) been > pushed and stable on bare metal. TripleO is one of the deployment tools which work with RDO packages, it never was and never will be the only one, so you have a choice. > However, numbers in report bellow:- 38% packstack deployments vs 20% TripleO > deployments > fairly prove that just for Newton && Ocata releases makes sense to have > storage.pp role for packstack. Sample was rather small so I would not jump to the conclusion, but just to clarify that Packstack is a community supported project and you can report the request details in https://bugs.launchpad.net/packstack and/or propose a patch in gerrit Bugs : Packstack bugs.launchpad.net #1453269 wrong endpoint for swift in a multi node setup when using CONFIG_STORAGE_HOST then we can discuss how it can fit. Maybe feature does not fit in Packstack directly e.g. cloud be integrated with some other tool, let's keep open mind and not assume implementation details. As an example, recent Magnum support in Packstack was contributed from outside Red Hat. > TripleO is exciting , but maintain prod environment via update of heat stack > "overcloud" targeting for instance :- > 1. Adding one more compute node > 2. Replacing Controller node in PCS Cluster > would scare too much admins. Can you provide more details, I'm sure Emilien as TripleO PTL would like to hear them :) > Regression in packstack been done ahead of right time might have negative > drawback and it would. I guess you mean focusing Packstack to be excellent all-in-one proof-of-concept installation tool? Repeating myself again: if we don't have it covered in CI, it will be > broken by design so focusing is all about _avoiding_ regressions. Hi Alan, Thanks a lot for feedback. Finally I understood the nature of issue. I don't have in depth knowledge of packstack's internals to write a patch on my own. So, only controller.pp, network.pp and compute.pp are CI compatible. Would I like to have storage.pp it also has to be compatible with CI and this effort should be attempted by community, rather then by RH's staff. What in fact means that TripleO is core deployment tool. Without ability split Storage Server from Controller packstack as quick and efficient deployment utility is almost useless in prod environment via my experience. Looking at https://etherpad.openstack.org/p/rdo-survey-answers ( down from line 25 ) I am really happy to see that I was heard. Thank you once again. Boris. > E.g. we have found out today that Nagios support in Packstack is broken only after being reported on #rdo ... because it is not covered in CI. To fix that, I would propose to drop Nagios support in Packstack and instead look at integrating with CentOS Opstools SIG. Cheers, Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: From fboucher at redhat.com Wed Nov 30 13:29:03 2016 From: fboucher at redhat.com (Fabien Boucher) Date: Wed, 30 Nov 2016 14:29:03 +0100 Subject: [rdo-list] Scheduled maintenance of review.rdoproject.org: 2016-11-30 12:00 pm UTC In-Reply-To: References: Message-ID: Hi folks, The maintenance is now done. The platform has been upgraded successfully to 2.2.6. Regards, Software Factory Team, on behalf of rdo-infra Le 25/11/2016 ? 14:35, Nicolas Hicher a ?crit : > Hello folks, > > We plan to upgrade review.rdoproject.org on 2016-11-30 12:00 pm UTC. The downtime will be about 1 hour approximately. > > This is a maintenance upgrade to the stable version of Software Factory 2.2.6. Please have a look to the changelog [1] for improvements. > > > Regards, > Software Factory Team, on behalf of rdo-infra > > > [1] https://github.com/redhat-cip/software-factory/blob/master/CHANGELOG.md > > _______________________________________________ > rdo-list mailing list > rdo-list at redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe at redhat.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From rbowen at redhat.com Wed Nov 30 15:44:42 2016 From: rbowen at redhat.com (Rich Bowen) Date: Wed, 30 Nov 2016 10:44:42 -0500 Subject: [rdo-list] Speaker list In-Reply-To: <6e1991f8-d8b8-5403-b92a-e8d4b6b57ec6@redhat.com> References: <6e1991f8-d8b8-5403-b92a-e8d4b6b57ec6@redhat.com> Message-ID: The consensus on the IRC meeting today is that it's better to do this entirely in the upstream speaker tool. If you are willing to speak at local events on RDO topics (or any OpenStack topics) please update your listing on https://www.openstack.org/community/speakers/ accordingly. Thanks. --Rich On 11/29/2016 01:34 PM, Rich Bowen wrote: > I'm occasionally approached by Meetup organizers looking for local(ish) > speakers for their meetups. At one point we had a Subject Matter Experts > list somewhere, but it has fallen far out of date. And I can no longer > keep in my mind where you are all geographically. > > I'd like to take another stab at a simple "speaker bureau" page, where > people willing to speak on RDO topics can indicate what topics/talks > they're willing to do, and what their single day or overnight travel > radius might be. > > I know that the OpenStack Foundation has a speaker bureau already - > https://www.openstack.org/community/speakers/ - and I encourage you to > add your info there. (I see just 9 of you listed at > https://www.openstack.org/community/speakers/results?search_query=RDO > right now.) > > But for RDO-specific speakers, that I can recommend to meetup > organizers, I can also sometimes help out with minimal travel/lodging > funding, as well as sending swag. For this purpose, if you'd like to be > on this list, please submit a pull request for > https://www.rdoproject.org/community/speakers/ to add yourself. > > (This page isn't linked from anywhere yet, and won't be until there's a > few more names on it.) > > Thanks. > > --Rich > > -- Rich Bowen - rbowen at redhat.com RDO Community Liaison http://rdoproject.org @RDOCommunity From amoralej at redhat.com Wed Nov 30 16:06:40 2016 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Wed, 30 Nov 2016 17:06:40 +0100 Subject: [rdo-list] [Meeting] RDO meeting (2016-11-30) Minutes Message-ID: ============================== #rdo: RDO meeting - 2016-11-30 ============================== Meeting started by amoralej at 15:02:03 UTC. The full logs are available at https://meetbot.fedoraproject.org/rdo/2016-11-30/rdo_meeting_-_2016-11-30.2016-11-30-15.02.log.html . Meeting summary --------------- * roll call (amoralej, 15:02:40) * Ocata1 testday tomorrow reality check (amoralej, 15:05:28) * puppet4 was added since the CI promotion, tripleo and packstack got fixes but now we need new promotion today! (amoralej, 15:05:37) * LINK: http://cbs.centos.org/koji/buildinfo?buildID=14924 (amoralej, 15:10:49) * RDO at FOSDEM (amoralej, 15:15:01) * Will be part of CentOS dojo, at IBM facility out by the UN. Details still to be determined, on centos-promo mailing list (amoralej, 15:15:12) * Call for RDO content is https://etherpad.openstack.org/p/fosdem-dojo-2017 (amoralej, 15:15:20) * Looking for people to identify themselves as willing to speak on RDO topics - https://www.rdoproject.org/community/speakers/ (amoralej, 15:23:17) * open floor (amoralej, 15:31:24) * chair for next meeting (amoralej, 15:45:01) * trown will chair next week (amoralej, 15:45:46) Meeting ended at 15:46:47 UTC. Action Items ------------ Action Items, by person ----------------------- * **UNASSIGNED** * (none) People Present (lines said) --------------------------- * amoralej (64) * rbowen (48) * dmsimard (25) * mengxd (13) * apevec (10) * zodbot (9) * number80 (9) * dmellado (8) * rdogerrit (4) * misc (3) * chandankumar (3) * jpena (3) * trown (3) * openstack (3) * bkero (2) * jschlueter (2) * rdobot (1) * jruzicka (1) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot From hrybacki at redhat.com Wed Nov 30 16:22:09 2016 From: hrybacki at redhat.com (Harry Rybacki) Date: Wed, 30 Nov 2016 11:22:09 -0500 Subject: [rdo-list] RDO-Infra Weekly Scrum Recap: Message-ID: Greetings All, Links to the RDO-Infra team's scrum etherpad[1] and video recording[2] of the meeting are below. Highlights: - Nominations for Atlanta PTG - Baremetal fallout - rlandy handling final issues getting baremetal gates up - DLRN API PoC[3] review - action: trown and myoung to test, provide feedback for jpena's api work - Liberty EOL - action: Begin clean up of JJB, PoC jobs, workspaces[4] - Reminder of review.rdoproject.org downtime for upgrade on 30-Nov-16 at 1200 UTC - Discuss WeiRDO job stability improvements - Containers and Kolla update - Gate coverage review - action: create gate job that executes tempest[5] - Discuss/demonstrate sova[6] (ci status page[7]) - Trello board review: blockers and in progress [1] - https://review.rdoproject.org/etherpad/p/rdo-infra-scrum [2] - https://redhat.bluejeans.com/m/s8ujc/ [3] - https://review.rdoproject.org/r/#/c/3838/ [4] - https://trello.com/c/9OZvonW6/489-review-and-remove-liberty-specific-items-from-jjb-poc-jobs-and-relevant-repositories [5] - https://trello.com/c/GQ6xstww/491-create-a-gate-job-that-exercises-tempest-via-quickstart [6] - https://github.com/sshnaidm/sova [7] - http://status-tripleoci.rhcloud.com/