<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body> Nine tainted pods. Running apps seem to be okay, where they happened to be running at time the taint flood occurred. This will block IATT efforts, since we can not deploy our apps once we have remediated the vulnerabilities and to confirm remediation with TL and Anchore (there is not local scanning capability). V/R, Russell C Kendall <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr">From: Jonathan Rickard <jrickard@redhat.com> Sent: Friday, December 6, 2019 12:13:29 PM To: ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP Cc: Kendall, Russell C; platformONE@redhat.com; Miller, Timothy J.; Keegan Reap; Bubb, Mike; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP; Jonathan Rickard Subject: Re: [Platformone] [EXT] Re: Unified Platform Pod Deploy Errors <div> </div> </div> <div> <div dir="ltr">Also, is every application having problems or a specific? <div> <div> <div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"> <div dir="ltr"> <div> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> Jonathan Rickard, RHCE, RHCA Consulting Architect <a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat Public Sector</a> <div style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:medium;margin-bottom:4px"> </div> <a href="mailto:jonny@redhat.com" style="color:rgb(0,0,0);margin:0px" target="_blank">jonny@redhat.com</a> M: <a href="tel:210.862.9739" style="color:rgb(0,0,0);margin:0px" target="_blank">210.862.9739</a> <div style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:12px;margin-bottom:8px"> <div><a href="https://twitter.com/redhatjobs" title="twitter" style="background:url("https://static.redhat.com/libs/redhat/brand-assets/2/social/social-media--twitter--53x53.png") 0px 50%/14px no-repeat transparent;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:14px" target="_blank">@redhatjobs</a> <a href="https://www.facebook.com/redhatjobs" title="Facebook" style="background:url("https://static.redhat.com/libs/redhat/brand-assets/2/social/social-media--facebook--53x13.png") 0px 50%/9px no-repeat transparent;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:12px" target="_blank">redhatjobs</a> <a href="https://instagram.com/redhatjobs" title="instagram" style="background:url("https://static.redhat.com/libs/redhat/brand-assets/2/social/social-media--instagram--53x50.png") 0px 50%/13px no-repeat transparent;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:12px;margin-left:8px;padding-bottom:1px" target="_blank">@redhatjobs</a> </div> </div> <div style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:medium;margin-top:12px"> <table border="0"> <tbody> <tr> <td width="100px"><a href="https://www.redhat.com/" target="_blank"><img width="90" height="auto" src="https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png"></a></td> </tr> </tbody> </table> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr">On Fri, Dec 6, 2019 at 12:06 PM Jonathan Rickard <<a href="mailto:jrickard@redhat.com">jrickard@redhat.com</a>> wrote: </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr">Ade, <div> </div> <div>What does that mean? You can't login, you can't deploy? </div> <div> <div> <div dir="ltr"> <div dir="ltr"> <div> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> <div dir="ltr"> Jonathan Rickard, RHCE, RHCA Consulting Architect <a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat Public Sector</a> <div style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:medium;margin-bottom:4px"> </div> <a href="mailto:jonny@redhat.com" style="color:rgb(0,0,0);margin:0px" target="_blank">jonny@redhat.com</a> M: <a href="tel:210.862.9739" style="color:rgb(0,0,0);margin:0px" target="_blank">210.862.9739</a> <div style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:12px;margin-bottom:8px"> <div><a href="https://twitter.com/redhatjobs" title="twitter" style="background:url("https://static.redhat.com/libs/redhat/brand-assets/2/social/social-media--twitter--53x53.png") 0px 50%/14px no-repeat transparent;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:14px" target="_blank">@redhatjobs</a> <a href="https://www.facebook.com/redhatjobs" title="Facebook" style="background:url("https://static.redhat.com/libs/redhat/brand-assets/2/social/social-media--facebook--53x13.png") 0px 50%/9px no-repeat transparent;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:12px" target="_blank">redhatjobs</a> <a href="https://instagram.com/redhatjobs" title="instagram" style="background:url("https://static.redhat.com/libs/redhat/brand-assets/2/social/social-media--instagram--53x50.png") 0px 50%/13px no-repeat transparent;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:12px;margin-left:8px;padding-bottom:1px" target="_blank">@redhatjobs</a> </div> </div> <div style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:medium;margin-top:12px"> <table border="0"> <tbody> <tr> <td width="100px"><a href="https://www.redhat.com/" target="_blank"><img width="90" height="auto" src="https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png"></a></td> </tr> </tbody> </table> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <div class="gmail_quote"> <div dir="ltr" class="gmail_attr">On Fri, Dec 6, 2019 at 12:02 PM ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP <<a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a>> wrote: </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="white" lang="EN-US"> <div> ALCON, The cluster is down again. Please assist. <div> Most Sincerely, Ade Abodunrin, GG-12, USAF Product Owner (Cybertron & Ginyu Force), Unified Platform <img width="52" height="42" style="width: 0.5416in; height: 0.4375in;" id="gmail-m_8017697073979776159gmail-m_3362986349575164388Picture_x0020_3" alt="cid:image001.png@01D4F814.4AA552D0" src="cid:16edc6387444cff311"> LevelUP Code Works Commercial: (210) 890-2113 NIPR email: <a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a> </div> <div> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in"> From: Kendall, Russell C <Russell.Kendall@ManTech.com> Sent: Thursday, December 5, 2019 9:55 AM To: Jonathan Rickard <<a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> Cc: Miller, Timothy J. <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>>; Keegan Reap <<a href="mailto:kreap@redhat.com" target="_blank">kreap@redhat.com</a>>; Bubb, Mike <<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>>; <a href="mailto:platformONE@redhat.com" target="_blank">platformONE@redhat.com</a>; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <<a href="mailto:jose.ramirez.50.ctr@us.af.mil" target="_blank">jose.ramirez.50.ctr@us.af.mil</a>>; ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP <<a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a>>; Jonathan Rickard <<a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a>> Subject: [Non-DoD Source] Re: [Platformone] [EXT] Re: Unified Platform Pod Deploy Errors </div> </div> Jonny, I'll see you Friday at 500 Nav. Travel safe. V/R, Russell C Kendall <div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="2" width="98%" align="center"> </div> <div id="gmail-m_8017697073979776159gmail-m_3362986349575164388divRplyFwdMsg"> From: Jonathan Rickard <<a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> Sent: Wednesday, December 4, 2019 5:29 PM To: Kendall, Russell C Cc: Miller, Timothy J.; Keegan Reap; Bubb, Mike; <a href="mailto:platformONE@redhat.com" target="_blank"> platformONE@redhat.com</a>; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP; ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP; Jonathan Rickard Subject: Re: [Platformone] [EXT] Re: Unified Platform Pod Deploy Errors <div> </div> </div> <div> <div> Russell, <div> </div> <div> I have definitely been terrible with email lately and I apologize for the slow response times. I get back to San Antonio tomorrow but I have a pretty full afternoon. I can stop by Friday if you'd like. </div> <div> </div> <div> Thanks, </div> <div> jonny <div> </div> <div> <div> <div> <div> <div> <div> <div> <div> <div> Jonathan Rickard, RHCA Principal Consultant, NAPS <a href="https://www.redhat.com/" target="_blank">Red Hat Remote - Texas</a> <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> M: <a href="tel:210-862-9739" target="_blank">210-862-9739</a> <div style="margin-top:9pt"> <table border="0" cellpadding="0"> <tbody> <tr> <td style="padding:0.75pt"> <a href="https://www.redhat.com/" target="_blank"><img border="0" width="90" style="width: 0.9375in;" id="gmail-m_8017697073979776159gmail-m_3362986349575164388_x0000_i1026" src="https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9ddafd5b2f1836854d7416a/Logo-RedHat-Email.png"></a> </td> </tr> </tbody> </table> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> <div> <div> On Wed, Dec 4, 2019 at 10:16 AM Kendall, Russell C <<a href="mailto:Russell.Kendall@mantech.com" target="_blank">Russell.Kendall@mantech.com</a>> wrote: </div> <blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"> Jonny, I'd like to suggest you come to 500 to wrap this up, since it seems there are significant delays in communication that are contributing to downtime. V/R, Russell C Kendall ________________________________________ From: Miller, Timothy J. <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>> Sent: Wednesday, December 4, 2019 7:02 AM To: Jonathan Rickard; Keegan Reap Cc: Bubb, Mike; <a href="mailto:platformONE@redhat.com" target="_blank">platformONE@redhat.com</a>; Kendall, Russell C; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP; ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP; Jonathan Rickard Subject: Re: [Platformone] [EXT] Re: Unified Platform Pod Deploy Errors Johnny-- Update the issue, if you would be so kind. -- T On 12/3/19, 18:00, "<a href="mailto:platformone-bounces@redhat.com" target="_blank">platformone-bounces@redhat.com</a> on behalf of Jonathan Rickard" <<a href="mailto:platformone-bounces@redhat.com" target="_blank">platformone-bounces@redhat.com</a> on behalf of <a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> wrote: Hey Guys - Sorry for taking so long - this has been completed. Please run your builds and let us know if you're having any problems. jonny Jonathan Rickard, RHCA Principal Consultant, NAPS Red Hat Remote - Texas <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> M: 210-862-9739 <<a href="tel:210-862-9739" target="_blank">tel:210-862-9739</a>> <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> On Tue, Dec 3, 2019 at 3:47 PM Jonathan Rickard <<a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> wrote: Russell / Team, We believe we've identified the issue with your application deploying. In order to rectify the issue I need to evacuate pods so you will probably see some hiccups while deploying. I will update when this is resolved. Thanks, jonny Jonathan Rickard, RHCA Principal Consultant, NAPS Red Hat Remote - Texas <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> M: 210-862-9739 <<a href="tel:210-862-9739" target="_blank">tel:210-862-9739</a>> <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> On Tue, Dec 3, 2019 at 12:53 PM Keegan Reap <<a href="mailto:kreap@redhat.com" target="_blank">kreap@redhat.com</a>> wrote: Hey all, we have opened an issue below, that we believe to be the cause, we are currently investigating: <a href="https://dccscr.dsop.io/ginyu-force/up-iatt/issues/32" target="_blank"> https://dccscr.dsop.io/ginyu-force/up-iatt/issues/32</a> On Tue, Dec 3, 2019 at 11:27 AM Jonathan Rickard <<a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> wrote: Russell, Getting more eyes on this @<a href="mailto:platformONE@redhat.com" target="_blank">platformONE@redhat.com</a> <mailto:<a href="mailto:platformONE@redhat.com" target="_blank">platformONE@redhat.com</a>> We'll keep you posted. jonny Jonathan Rickard, RHCA Principal Consultant, NAPS Red Hat Remote - Texas <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> M: 210-862-9739 <<a href="tel:210-862-9739" target="_blank">tel:210-862-9739</a>> <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> On Tue, Dec 3, 2019 at 11:59 AM Kendall, Russell C <<a href="mailto:Russell.Kendall@mantech.com" target="_blank">Russell.Kendall@mantech.com</a>> wrote: Kevin, Unfortunately we are receiving deployment errors again. This is the event: 0/9 nodes are available: 1 node(s) had disk pressure, 2 node(s) had taints that the pod didn't tolerate, 6 node(s) didn't match node selector. This is the deployment: <a href="https://cluster.unified-platform.io/console/project/ccat-dev/browse/dc/driveup" target="_blank"> https://cluster.unified-platform.io/console/project/ccat-dev/browse/dc/driveup</a> V/R, Russell C Kendall ________________________________________ From: Miller, Timothy J. <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>> Sent: Monday, December 2, 2019 2:44:21 PM To: Kevin O'Donnell Cc: Kendall, Russell C; Mark Nissley; ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP; Jonathan Rickard; Chris Kuperstein; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP; Brenna Gordon; Bubb, Mike; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP Subject: Re: [EXT] Re: Unified Platform Pod Deploy Errors Tagged you on it. -- T On 12/2/19, 14:03, "Kevin O'Donnell" <<a href="mailto:kodonnel@redhat.com" target="_blank">kodonnel@redhat.com</a>> wrote: Hello, Autoscaling is on our future IAC roadmap. Tim, the additional ticket would be appreciated. We have swapped out the app/worker instances with m5a.8xlarge 32 cores, 128gb of ram. Please let us know if you have any other issues. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a> <mailto:<a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a>%20M:240-605-4654> M: 240-605-4654 <<a href="https://red.ht/sig" target="_blank">https://red.ht/sig</a>> On Mon, Dec 2, 2019 at 2:29 PM Miller, Timothy J. <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>> wrote: I'll open an issue. IaC needs to have instance size as a host_var to facilitate scaling. -- T On 12/2/19, 13:15, "Kevin O'Donnell" <<a href="mailto:kodonnel@redhat.com" target="_blank">kodonnel@redhat.com</a>> wrote: Tim, Thanks for the information. We are undersized on the app/worker nodes, the cluster has 3 and they are m5.xlarge with only 16gb of ram. From what I have read each Labs engagement operated on a 3 node worker cluster with each node having 6core's and 28gb of ram. We will need to swap out the existing instances with larger spec's. We are going to try to flush the existing workload out on one of the workers to see if we can swap them out one at a time. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a> <mailto:<a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a>%20M:240-605-4654> M: 240-605-4654 <<a href="https://red.ht/sig" target="_blank">https://red.ht/sig</a>> On Mon, Dec 2, 2019 at 2:07 PM Miller, Timothy J. <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>> wrote: Here's what I can see, given the perm limits I seem to be under: - NS:develop-misp-app and NS:lp-develop-misp-app both have several sync jobs in CrashLoopBackoff b/c of an HTTP 401 error returned while trying to fetch something from somewhere (URL isn't recorded in the stack trace). - NS:minishift-misp-app has most of its pods/jobs stuck in ImagePullBackoff. No detail there in the event stream so I'll see if I can dig deeper. - NS:aam-ci-cd has Jenkins trying to spin up three workers, those are coming back as unschedulable. I can't see into NS:aam-bases or NS:dsop-images b/c of perm limits. I see no DAS-related project(s). The MISP stuff needs debugging before calling "blocked" since it looks like an internal error from this perspective. In re: AAM Jenkins: If this deployment is coming out of the OCP storefront, then maybe it should be ephemeral rather than persistent. If it's a custom deployment, then it probably needs a rethink. I'm also not sure why there are two MISP dev projects. -- T On 12/2/19, 12:46, "Kevin O'Donnell" <<a href="mailto:kodonnel@redhat.com" target="_blank">kodonnel@redhat.com</a>> wrote: Russell, Thank you for the information. We can switch out the instance type for the worker nodes. How much memory is required by the apps? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a> <mailto:<a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a>%20M:240-605-4654> M: 240-605-4654 <<a href="https://red.ht/sig" target="_blank">https://red.ht/sig</a>> On Mon, Dec 2, 2019 at 1:32 PM Kendall, Russell C <<a href="mailto:Russell.Kendall@mantech.com" target="_blank">Russell.Kendall@mantech.com</a>> wrote: Kevin, The lack of resources on <a href="http://u-p.io" target="_blank">u-p.io</a> <<a href="http://u-p.io" target="_blank">http://u-p.io</a>> <<a href="http://u-p.io" target="_blank">http://u-p.io</a>> <<a href="http://u-p.io" target="_blank">http://u-p.io</a>> <<a href="http://u-p.io" target="_blank">http://u-p.io</a>> cluster is hindering development, testing, and integration of the apps from CCAT AAM DAS, which is putting one of our PI goals at risk. We are blocked by the fact that we (CCAT and AAM) cannot deploy additional pods to the <a href="http://unified-platform.io" target="_blank">unified-platform.io</a> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> cluster. We have a subset of containers deployed, but rolling deployments and new deployments fail. This means that we are not able to execute integration testing or peer reviews. We are temporarily working around by NOT testing/reviewing our code changes live, something that no one likes. Also, we are now running weeks-old instances of our containers, so we are very likely producing some technical debt. We currently have developers approaching idle or doing non-priority work until the resource issue is resolved. Here is the particular error from the OSP cluster I received while attempting a redeploy of one of our apps. 0/9 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 Insufficient pods, 6 node(s) didn't match node selector.11 times in the last minute Since we do not have any cluster permissions, I cannot verify which resource is running out, but from experience, I assess it is a memory issue. It appears the cluster has been provisioned with a silly allocation of node types. Without knowing exactly what was deployed, it appears only 3 of the 9 hosts are suitable worker nodes. We would expect the cluster to respond to resource limitations and scale, but if a scheduled downtime is required, please work with us so we can anticipate. As it stands, the cluster does not support resources required by CCAT and the other dev teams (AAM, DAS, etc.). We would accept any downtime if it will improve the situation, as we are blocked from progressing under the current constraints. My hope was we could get the cluster redeployed over the TG holiday to eliminate developer impact, but as Mark pointed out, there were limited support folks available. Now I am just trying to minimize the losses. V/R, Russell C Kendall ________________________________________ From: Kevin O'Donnell <<a href="mailto:kodonnel@redhat.com" target="_blank">kodonnel@redhat.com</a>> Sent: Monday, December 2, 2019 11:52 AM To: Kendall, Russell C Cc: Mark Nissley; ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP; Jonathan Rickard; Chris Kuperstein; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP; Brenna Gordon; Bubb, Mike (<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>); DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP; Miller, Timothy J.; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP Subject: Re: Unified Platform Pod Deploy Errors Hello Russell, Can you elaborate on the term Blocked? What specific issues are the blockers? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a> <mailto:<a href="mailto:kodonnell@redhat.com" target="_blank">kodonnell@redhat.com</a>%20M:240-605-4654> M: 240-605-4654 <<a href="https://red.ht/sig" target="_blank">https://red.ht/sig</a>> On Mon, Dec 2, 2019 at 11:11 AM Kendall, Russell C <<a href="mailto:Russell.Kendall@mantech.com" target="_blank">Russell.Kendall@mantech.com</a>> wrote: Mark, Thank for acknowledging, please be aware the San Antonio dev teams working in <a href="http://unified-platform.io" target="_blank">unified-platform.io</a> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> are currently blocked. V/R, Russell C Kendall ________________________________________ From: Mark Nissley <<a href="mailto:mnissley@redhat.com" target="_blank">mnissley@redhat.com</a>> Sent: Monday, December 2, 2019 9:36 AM To: ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP; Jonathan Rickard; Chris Kuperstein Cc: BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP; Kevin O'Donnell; Brenna Gordon; Kendall, Russell C; Bubb, Mike (<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>); DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP; Miller, Timothy J.; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP Subject: Re: Unified Platform Pod Deploy Errors As noted, I don't suspect much got done on this over the holiday weekend. I did see the ticket, as dropped some details into it. I also assigned it to @Jonathan Rickard <mailto:<a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a>> and @Chris Kuperstein <mailto:<a href="mailto:ckuperst@redhat.com" target="_blank">ckuperst@redhat.com</a>> . It looks like short term solutions have been easy but the issue is recurring. Mark NISSLEY, PMP, CSM, LEAN PROGRAM MaNAGER & SR technical Project Manager North American Consulting, Public Sector <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> M: 850-530-3234 <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> Scheduled Training: October 14-18 On Mon, Dec 2, 2019 at 10:21 AM ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP <<a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a>> wrote: Mark/Kevin, I just heard at the team stand up that we are still blocked. This is also affecting the AAM team from my investigations. Please let me know if there is something we need to do to move this forward. Most Sincerely, Ade Abodunrin, GG-12, USAF Product Owner (Cybertron & Ginyu Force), Unified Platform LevelUP Code Works Commercial: (210) 890-2113 NIPR email: <a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a> ________________________________________ From: ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP Sent: Wednesday, November 27, 2019 12:58 PM To: BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <<a href="mailto:austen.bryan.1@us.af.mil" target="_blank">austen.bryan.1@us.af.mil</a>>; Mark Nissley <<a href="mailto:mnissley@redhat.com" target="_blank">mnissley@redhat.com</a>>; Kevin O'Donnell <<a href="mailto:kodonnel@redhat.com" target="_blank">kodonnel@redhat.com</a>>; Brenna Gordon <<a href="mailto:bgordon@redhat.com" target="_blank">bgordon@redhat.com</a>> Cc: Kendall, Russell C <<a href="mailto:Russell.Kendall@ManTech.com" target="_blank">Russell.Kendall@ManTech.com</a>>; Bubb, Mike (<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>) <<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>>; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP <<a href="mailto:roger.dirocco.4@us.af.mil" target="_blank">roger.dirocco.4@us.af.mil</a>>; Miller, Timothy J. <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>>; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <<a href="mailto:jose.ramirez.50.ctr@us.af.mil" target="_blank">jose.ramirez.50.ctr@us.af.mil</a>> Subject: Re: Unified Platform Pod Deploy Errors Thanks a lot Capt Bryan! Russell created the ticket on GitLab UP Node Project. Most Sincerely, Ade Abodunrin, GG-12, USAF Product Owner (Cybertron & Ginyu Force), Unified Platform LevelUP Code Works Commercial: (210) 890-2113 NIPR email: <a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a> ________________________________________ From: BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <<a href="mailto:austen.bryan.1@us.af.mil" target="_blank">austen.bryan.1@us.af.mil</a>> Sent: Wednesday, November 27, 2019 12:56 PM To: ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP <<a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a>>; Mark Nissley <<a href="mailto:mnissley@redhat.com" target="_blank">mnissley@redhat.com</a>>; Kevin O'Donnell <<a href="mailto:kodonnel@redhat.com" target="_blank">kodonnel@redhat.com</a>>; Brenna Gordon <<a href="mailto:bgordon@redhat.com" target="_blank">bgordon@redhat.com</a>> Cc: Kendall, Russell C <<a href="mailto:Russell.Kendall@ManTech.com" target="_blank">Russell.Kendall@ManTech.com</a>>; Bubb, Mike (<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>) <<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>>; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP <<a href="mailto:roger.dirocco.4@us.af.mil" target="_blank">roger.dirocco.4@us.af.mil</a>>; Miller, Timothy J. <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>>; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <<a href="mailto:jose.ramirez.50.ctr@us.af.mil" target="_blank">jose.ramirez.50.ctr@us.af.mil</a>> Subject: RE: Unified Platform Pod Deploy Errors Thanks Ade. The team is thin until next week due to the holidays but I will make sure it is addressed. Were there any issues submitted to Gitlab’s UP Node Project on DCCSCR? @Mark/Kevin – can we address? -Austen From: ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP <<a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a>> Sent: Wednesday, November 27, 2019 9:51 AM To: BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <<a href="mailto:austen.bryan.1@us.af.mil" target="_blank">austen.bryan.1@us.af.mil</a>> Cc: Kendall, Russell C <<a href="mailto:Russell.Kendall@ManTech.com" target="_blank">Russell.Kendall@ManTech.com</a>>; Bubb, Mike (<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>) <<a href="mailto:mbubb@mitre.org" target="_blank">mbubb@mitre.org</a>> Subject: Fw: Unified Platform Pod Deploy Errors Capt Bryan, Please see the explanation on the issue that Ginyu Force is currently experiencing below. Most Sincerely, Ade Abodunrin, GG-12, USAF Product Owner (Cybertron & Ginyu Force), Unified Platform LevelUP Code Works Commercial: (210) 890-2113 NIPR email: <a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a> ________________________________________ From: Kendall, Russell C <<a href="mailto:Russell.Kendall@ManTech.com" target="_blank">Russell.Kendall@ManTech.com</a>> Sent: Wednesday, November 27, 2019 9:46 AM To: ABODUNRIN, ADEMOLA A GG-12 USAF AFMC AFLCMC/HNCP <<a href="mailto:ademola.abodunrin@us.af.mil" target="_blank">ademola.abodunrin@us.af.mil</a>>; Buffaloe, Christopher <<a href="mailto:Christopher.Buffaloe@ManTech.com" target="_blank">Christopher.Buffaloe@ManTech.com</a>>; Molina, Toby <<a href="mailto:Toby.Molina@ManTech.com" target="_blank">Toby.Molina@ManTech.com</a>>; Crace, Jared E <<a href="mailto:Jared.Crace@ManTech.com" target="_blank">Jared.Crace@ManTech.com</a>>; SANCHEZ, MARK GG-13 USAF AFMC AFLCMC/HNCP <<a href="mailto:mark.sanchez.8@us.af.mil" target="_blank">mark.sanchez.8@us.af.mil</a>> Cc: <a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a> <mailto:<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>> <<a href="mailto:tmiller@mitre.org" target="_blank">tmiller@mitre.org</a>> Subject: [Non-DoD Source] Fw: Unified Platform Pod Deploy Errors Gentlemen, The application development teams working in the new GovCloud OCP environment (<a href="http://unified-platform.io" target="_blank">unified-platform.io</a> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>> <<a href="http://unified-platform.io" target="_blank">http://unified-platform.io</a>>) are currently blocked in efforts to deploy new pods for testing, development, and UAT. Red Hat and RogueOne SMEs have been notified and have attempted some fixes starting on Monday 11/25, but at this point have not been able to provision resources sufficient to host CCAT and AAM. We have taken steps to minimize our footprint (eliminating demonstration environment, deleting developer namespaces), but this is not a sustainable approach, and has only resulted in moderate improvements in cluster performance. Our hope is the U-P.io cluster compute resources can be increased very soon, so that we may resume normal development activities. Our understanding is that such a scaling requires a complete redeployment of the cluster, which is unusual, but an acceptable loss to productivity. If the cluster can be scaled up over the Thanksgiving holiday, the impact will be minimal to developers and cluster administrators, alike. We are currently collaborating on solutions on the following MatterMost channel behind the space camp VPN (link below), and via the email thread forwarded (further below). <a href="https://chat.spacecamp.ninja/levelup/channels/unified-platform-node" target="_blank"> https://chat.spacecamp.ninja/levelup/channels/unified-platform-node</a> <<a href="https://chat.spacecamp.ninja/levelup/channels/unified-platform-node" target="_blank">https://chat.spacecamp.ninja/levelup/channels/unified-platform-node</a>> <<a href="https://chat.spacecamp.ninja/levelup/channels/unified-platform-node" target="_blank">https://chat.spacecamp.ninja/levelup/channels/unified-platform-node</a>> <<a href="https://chat.spacecamp.ninja/levelup/channels/unified-platform-node" target="_blank">https://chat.spacecamp.ninja/levelup/channels/unified-platform-node</a>> Please keep me posted on developments and I will coordinate developer activities with any scheduled platform outages. V/R, Russell C Kendall ________________________________________ From: Curran, Daniel M Sent: Monday, November 25, 2019 2:47 PM To: Jonathan Rickard Cc: Jonathan Rickard; Chris Kuperstein; Mark Nissley; <a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a> <mailto:<a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a>>; Sison, Mark Anthony; Cepeda, Rolando; Kendall, Russell C; Andrichak IV, John J; Phil Soliz; Buffaloe, Christopher; Torres, Alexander; Crace, Jared E; Middleton, Joseph J Subject: Re: Unified Platform Pod Deploy Errors Sounds great. Appreciate it. I'll watch email and Mattermost in case you need more from us. -Daniel ________________________________________ From: Jonathan Rickard <<a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> Sent: Monday, November 25, 2019 2:44 PM To: Curran, Daniel M Cc: Jonathan Rickard; Chris Kuperstein; Mark Nissley; <a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a> <mailto:<a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a>>; Sison, Mark Anthony; Cepeda, Rolando; Kendall, Russell C; Andrichak IV, John J; Phil Soliz; Buffaloe, Christopher; Torres, Alexander; Crace, Jared E; Middleton, Joseph J Subject: Re: Unified Platform Pod Deploy Errors Thanks Daniel - I'll continue to look into the resource issue that you're seeing - I'd like to identify the root cause and then work with the team to come up with a solution. Jonathan Rickard, RHCA Principal Consultant, NAPS Red Hat Remote - Texas <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> M: 210-862-9739 <<a href="tel:210-862-9739" target="_blank">tel:210-862-9739</a>> <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> On Mon, Nov 25, 2019 at 2:17 PM Curran, Daniel M <<a href="mailto:Daniel.Curran@mantech.com" target="_blank">Daniel.Curran@mantech.com</a>> wrote: Yeah we hit the limit then had AAM kill some of their projects and then our pods got scheduled. We've hit the limit again though. Here's an example pod that cannot be scheduled <a href="https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth" target="_blank"> https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth</a> <<a href="https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth" target="_blank">https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth</a>> <<a href="https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth" target="_blank">https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth</a>> <<a href="https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth" target="_blank">https://cluster.unified-platform.io/console/project/ccat-dev/browse/pods/sso-8-2cjth</a>> They're seeing it when their jenkins slaves can't deploy but it's basically any pod after we hit some limit. -Daniel ________________________________________ From: Jonathan Rickard <<a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> Sent: Monday, November 25, 2019 1:26 PM To: Curran, Daniel M Cc: Jonathan Rickard; Chris Kuperstein; Mark Nissley; <a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a> <mailto:<a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a>>; Sison, Mark Anthony; Cepeda, Rolando; Kendall, Russell C; Andrichak IV, John J; Phil Soliz; Buffaloe, Christopher; Torres, Alexander; Crace, Jared E; Middleton, Joseph J Subject: Re: Unified Platform Pod Deploy Errors Daniel, I can see that you have 3 mongo pods, 1 chatup and 1 upbot pod running ... is your app good to go? Looks like there was an issue with memory on 1 pod, then some node selector being mismatched - just what i could see in the events... Jonathan Rickard, RHCA Principal Consultant, NAPS Red Hat Remote - Texas <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> M: 210-862-9739 <<a href="tel:210-862-9739" target="_blank">tel:210-862-9739</a>> <<a href="https://www.redhat.com/" target="_blank">https://www.redhat.com/</a>> On Mon, Nov 25, 2019 at 12:50 PM Curran, Daniel M <<a href="mailto:Daniel.Curran@mantech.com" target="_blank">Daniel.Curran@mantech.com</a>> wrote: Also, AAM was having similar issues. Looks like they had a lot of namespaces and scaling down the pods on their deployments didn't help but actually deleting the namespaces did. We have pods scheduling now but I'm adding them and we'd still like to work through what resource limit we were hitting to avoid this in the future. -Daniel ________________________________________ From: Curran, Daniel M Sent: Monday, November 25, 2019 12:25 PM To: Jonathan Rickard Cc: Jonathan Rickard; Chris Kuperstein; Mark Nissley; <a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a> <mailto:<a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a>>; Sison, Mark Anthony; Cepeda, Rolando; Kendall, Russell C; Andrichak IV, John J; Phil Soliz; Buffaloe, Christopher; Torres, Alexander Subject: Re: Unified Platform Pod Deploy Errors Thanks, sir. Most important for us to get working is "ccat-demo" but it's also happening in "ccat-dev" and "ccat-ci-cd". -Daniel ________________________________________ From: Jonathan Rickard <<a href="mailto:jrickard@redhat.com" target="_blank">jrickard@redhat.com</a>> Sent: Monday, November 25, 2019 12:22 PM To: Curran, Daniel M Cc: Jonathan Rickard; Chris Kuperstein; Mark Nissley; <a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a> <mailto:<a href="mailto:dlystra@redhat.com" target="_blank">dlystra@redhat.com</a>>; Sison, Mark Anthony; Cepeda, Rolando; Kendall, Russell C; Andrichak IV, John J; Phil Soliz; Buffaloe, Christopher; Torres, Alexander Subject: Re: Unified Platform Pod Deploy Errors What's the name of the project you're working in? I'm going to be back at my laptop in about 30 and will take a look when I get there. Is it just the Jenkins pods failing? On Mon, Nov 25, 2019, 12:20 PM Curran, Daniel M <<a href="mailto:Daniel.Curran@mantech.com" target="_blank">Daniel.Curran@mantech.com</a>> wrote: Adding Dean and Alex. Also, sitting in mattermost if anyone needs to get online and chat for more information. -Daniel ________________________________________ From: Curran, Daniel M Sent: Monday, November 25, 2019 12:07 PM To: <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> <mailto:<a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a>>; <a href="mailto:ckuperst@redhat.com" target="_blank">ckuperst@redhat.com</a> <mailto:<a href="mailto:ckuperst@redhat.com" target="_blank">ckuperst@redhat.com</a>>; Mark Nissley Cc: Sison, Mark Anthony; Cepeda, Rolando; Kendall, Russell C; Andrichak IV, John J; Phil Soliz; Buffaloe, Christopher Subject: Re: Unified Platform Pod Deploy Errors Adding Kupe and Mark. -Daniel ________________________________________ From: Curran, Daniel M Sent: Monday, November 25, 2019 11:43 AM To: <a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a> <mailto:<a href="mailto:jonny@redhat.com" target="_blank">jonny@redhat.com</a>> Cc: Sison, Mark Anthony; Cepeda, Rolando; Kendall, Russell C; Andrichak IV, John J; Phil Soliz; Buffaloe, Christopher Subject: Unified Platform Pod Deploy Errors Hey Jonny, We met briefly at SpaceCAMP a couple weeks ago when <a href="http://cluster.unified-platform.io" target="_blank">cluster.unified-platform.io</a> <<a href="http://cluster.unified-platform.io" target="_blank">http://cluster.unified-platform.io</a>> <<a href="http://cluster.unified-platform.io" target="_blank">http://cluster.unified-platform.io</a>> <<a href="http://cluster.unified-platform.io" target="_blank">http://cluster.unified-platform.io</a>> <<a href="http://cluster.unified-platform.io" target="_blank">http://cluster.unified-platform.io</a>> was stood up. We've been trying to deploy some apps today and so far today we're getting errors on most (if not all) of our pods. 0/9 nodes are available: 3 Insufficient pods, 6 node(s) didn't match node selector. Is what we're seeing. We were thinking it was some volume types weren't correct but some of our pods don't even have volumes attached and still give us this error (i.e. Jenkins slaves or web frontends without persistent storage). Any idea what this could be? We're not running out of space on the nodes themselves are we? We have a demo scheduled for tomorrow at 9:30 AM CST and are hoping to get a demo env up for them today but this error came up unexpectedly. Also, we're here at 500 Navarro St. in San Antonio working through this in person is better/easier. Thanks, Daniel Curran ________________________________________ This e-mail and any attachments are intended only for the use of the addressee(s) named herein and may contain proprietary information. If you are not the intended recipient of this e-mail or believe that you received this email in error, please take immediate action to notify the sender of the apparent error by reply e-mail; permanently delete the e-mail and any attachments from your computer; and do not disseminate, distribute, use, or copy this message and any attachments. _______________________________________________ platformONE mailing list <a href="mailto:platformONE@redhat.com" target="_blank">platformONE@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/platformone" target="_blank"> https://www.redhat.com/mailman/listinfo/platformone</a> </blockquote> </div> </div> </div> </div> </div> </blockquote> </div> </blockquote> </div> </div> </body> </html>