From adam.wise at g2-inc.com Fri Jan 3 16:52:45 2020 From: adam.wise at g2-inc.com (Adam Wise) Date: Fri, 3 Jan 2020 11:52:45 -0500 Subject: [Platformone] Using the DCAR postgres container Message-ID: Hi, we're hoping to find a way to make use of the postgres container located here: https://dccscr.dsop.io/dsop/opensource/postgres/postgres I opened a couple tickets that detail the issue. Apparently we can't find a way to get this launch, deploying to an OpenShift environment. There is an issue where Postgres does not have permission to access its data directory, even when we attempt to mount that folder. Here are the two tickets with more details: https://dccscr.dsop.io/dsop/dccscr/issues/257 https://dccscr.dsop.io/dsop/opensource/postgres/postgres/issues/2 At the moment, we seem to be blocked on this issue. Thanks for any assistance you can provide. -- Adam Wise Software Developer HII Mission Driven Innovative Solutions (HII-MDIS) ? formerly G2, Inc. Technical Solutions Division 302 Sentinel Drive | Annapolis Junction, MD 20701 email: adam.wise at g2-inc.com Work (301) 575-5143 | Mobile (301) 755-3250 -------------- next part -------------- An HTML attachment was scrubbed... URL: From darachch at redhat.com Fri Jan 3 18:40:16 2020 From: darachch at redhat.com (Dino Arachchi) Date: Fri, 3 Jan 2020 12:40:16 -0600 Subject: [Platformone] Deprovisioning UP-Prod Environment Message-ID: All, I hope everyone had a safe and happy new year! As requested, I will be cleaning up the up-prod environment by removing the dev accounts and deleting namespaces. Please let me know ASAP if there are any issues or concerns, as I will being the process at approximately 2 PM CT. This task is being tracked with the issue below: https://dccscr.dsop.io/levelup-automation/consumers/up-node-infrastructure/issues/12 Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 -------------- next part -------------- An HTML attachment was scrubbed... URL: From roger.dirocco.4 at us.af.mil Fri Jan 3 19:16:02 2020 From: roger.dirocco.4 at us.af.mil (DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP) Date: Fri, 3 Jan 2020 19:16:02 +0000 Subject: [Platformone] [Non-DoD Source] Deprovisioning UP-Prod Environment In-Reply-To: References: Message-ID: A non-text attachment was scrubbed... Name: smime.p7m Type: application/pkcs7-mime Size: 15277 bytes Desc: not available URL: From darachch at redhat.com Fri Jan 3 20:05:22 2020 From: darachch at redhat.com (Dino Arachchi) Date: Fri, 3 Jan 2020 14:05:22 -0600 Subject: [Platformone] [Non-DoD Source] Deprovisioning UP-Prod Environment In-Reply-To: References: Message-ID: Roc, Understood, and we'll hold off until Monday. Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 On Fri, Jan 3, 2020 at 1:19 PM DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP wrote: > Dino, in discussion with Austen, unless there is an absolute need to do it > today (like due to security concerns or cost), can we push this off to > Monday? Not everyone is back yet from the holidays and so not sure you > will > get any/all responses by 2pm CT if there are any issues. Unless you get > confirmation from Jet and Tim that it?s okay to press forward today, please > hold off until Monday. > > > > --v/r, Roc > > > > From: platformone-bounces at redhat.com On > Behalf Of Dino Arachchi > Sent: Friday, January 3, 2020 12:40 PM > To: platformONE at redhat.com > Subject: [Non-DoD Source] [Platformone] Deprovisioning UP-Prod Environment > > > > All, > > > > I hope everyone had a safe and happy new year! > > > > As requested, I will be cleaning up the up-prod environment by removing the > dev accounts and deleting namespaces. Please let me know ASAP if there are > any issues or concerns, as I will being the process at approximately 2 PM > CT. > > > > This task is being tracked with the issue below: > > https://dccscr.dsop.io/levelup-automation/consumers/up-node-infrastructure/ > issues/12 > > > > > Best Regards, > > > > DINO ARACHCHI > > SENIOR CONSULTANT > > darachch at redhat.com M: > > 848-203-1809 > > > < > https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9d > dafd5b2f1836854d7416a/Logo-RedHat-Email.png > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmiller at mitre.org Sun Jan 5 19:30:07 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Sun, 5 Jan 2020 19:30:07 +0000 Subject: [Platformone] [EXT] Re: Account for Dirty Dev In-Reply-To: <2033_1577134627_5E012A22_2033_997_1_CAHrof-fBuLdO+38+WvvTg88KZfZ2giK-OJHo1ZnSk__EAJ=OQg@mail.gmail.com> References: <1114349A-1B60-4C16-B453-A44057965097@braingu.com> <2033_1577134627_5E012A22_2033_997_1_CAHrof-fBuLdO+38+WvvTg88KZfZ2giK-OJHo1ZnSk__EAJ=OQg@mail.gmail.com> Message-ID: <026EE33B-A855-4C40-A552-80FB8D9BDDEA@mitre.org> Color me confused--y'all have spent time writing and validating that the ansible roles are idempotent, so why do incremental changes to the plays require a build from scratch? And since 3.5h of the 5h build time is sync'ing Satellite repos, at what point will y'all checkpoint that data into an AMI or a resuable bucket? -- T ?On 12/23/19, 14:57, "platformone-bounces at redhat.com on behalf of Dino Arachchi" wrote: Tim, There were some security features that didn't get merged into the branch used to build the stack. This will require the same 4-5 hours as the last build due to the fully disconnected nature of the build process. We do have an open issue for connected environments that will cut down build time, but that feature is still in progress. Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 On Mon, Dec 23, 2019 at 2:49 PM Tim Gast wrote: Thanks for the update. What was the issue you found that would cause the entire stack to need a rebuild? Are we looking at another 5 hour build? Thanks, -Tim On Dec 23, 2019, at 2:47 PM, Dino Arachchi wrote: All, While provisionin the dev accounts, an issue was identified in the newly created up-appdev-a cluster. Code changes are being made now, after which the stack will be rebuilt. Once this is completed and verified, I will proceed with the creation of the new accounts. These tasks are estimated to be completed today, and an email will be sent out to confirm. Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 On Mon, Dec 23, 2019 at 1:46 PM Kevin O'Donnell wrote: Thanks Dino -Kevin On Mon, Dec 23, 2019 at 2:38 PM Tim Gast wrote: Thanks Dino. -Tim On Dec 23, 2019, at 1:37 PM, Dino Arachchi wrote: Hi Tim, I'm on PTO but am working on creating the dev accounts at the moment. The team should expect to receive credentials within the next couple hours. Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 On Mon, Dec 23, 2019 at 12:53 PM Tim Gast wrote: Hi Kevin, Any word on when to be looking for those dev accounts? Thanks, -Tim On Dec 21, 2019, at 10:22 AM, Kevin O'Donnell wrote: Build completed in 4 hours and 40 min. On Monday we will get the accounts created and also add some code into our build to automate the account creation process. Everyone enjoy your weekend. KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Fri, Dec 20, 2019 at 9:19 PM Tim Gast wrote: Thanks for the update Kevin. -Tim On Dec 20, 2019, at 7:26 PM, Kevin O'Donnell wrote: Were about 3 hours in. KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Fri, Dec 20, 2019 at 8:04 PM Kevin O'Donnell wrote: Hello Tim, The aws rbac changes impacted our deployment and it didn?t get the deployment running till 5pm est. I keep an eye on the job. On Fri, Dec 20, 2019 at 5:19 PM Tim Gast wrote: Hi. Checking in since it?s been a few hours in flight. Do we have a UP-DEV-A cluster and user access now? Thanks, -Tim On Dec 20, 2019, at 11:49 AM, Tim Gast wrote: Kevin, Mark, & Roc, Below is the request for dev accounts for UP-APPDEV-A. Can we please mirror what these folks have in the current UP-Prod-A cluster. Also, is it possible to provide an ?elevated read? role for certain users (dcurrian) for discovery/troubleshooting? Please let me know when the environment is live and ready for the team to begin working with. Thanks, -Tim Begin forwarded message: From: "Curran, Daniel M" Date: December 20, 2019 at 11:15:39 AM CST To: "tg at braingu.com" , "nino at braingu.com" Cc: "Buffaloe, Christopher" Subject: Account for Dirty Dev Hey Tim and Nino, Here's the account list for the Ginyu Force (CCAT) team dcurran rcepeda msison psoliz atorres dpalmer rho kmacias jandrichak Obviously we'll take all the privileges you're willing to give us but if we can keep our dev privileges from the unified-platform cluster and put us in a ginyu/ccat group so we can see each others projects that would be enough. It might also be advantageous to have one admin account (dcurran-adm or ccat-adm) or at least one with elevated read privileges for discovery purposes but I wouldn't call that a hard requirement. Thanks, Daniel -- KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 -- KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 From tmiller at mitre.org Sun Jan 5 19:55:44 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Sun, 5 Jan 2020 19:55:44 +0000 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> Message-ID: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T ?On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen From kodonnel at redhat.com Sun Jan 5 20:33:32 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Sun, 5 Jan 2020 15:33:32 -0500 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> Message-ID: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: > Some more points of interest-- > > - The ec2-user account can't currently be avoided given the current design > of the IaC. > > - This SSH key provisioned for this account is shared across up-prod-a, > up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with Maintainer or > higher access to the GitLab repo that holds the customer's environment > definitions or the platform-infrastructure repo. > > -- T > > ?On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: > > I just want to clarify on the topic of the EC2-user. The key is > attribution. We need to know who or what is configuring the systems. > > We are repeatedly seeing the ec2-user account used to manually > configure systems. Ec2-user should only be used to perform programmatic > tasks from committed and > reviewed code as part of initial provisioning. If the code to > provision and configure systems is unavailable, then you must create > attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user will be flagged as > a violation and the source IP or user account logged as the violating > source. Any manual/command > line activities MUST be performed using an attributable user account. > > If that is not clear, please let me know and I will attempt to > elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) ; > BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; platformONE at redhat.com; Tim Gast < > tg at braingu.com>; > Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC > AFLCMC/HNC ; tj.zimmerman at braingu.com; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP < > richard.lopezdeuralde at us.af.mil>; Blade, Eric D [US] (MS) < > Eric.Blade at ngc.com>; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < > jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; > REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP < > melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; Miller, > Timothy J. ; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; > BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; > Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We will get GIT issues > crated for your findings and will prioritize the mitigation and implement > them as code in our future IAC deployments. Many of the findings in the > current VPC have been > mitigated in up-prod-b with our current code release. > > > > Please let us know when you have finished and we can power down the > host that you have been using for scanning. > > > > Note for everyone: Once we power down the ec2 instance ssh or port 22 > will not be externally accessible. Thus, mitigating many of the risks > associated with the ec2-user and the keys. > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet > wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If you want to take more > time, please use tomorrow to do your testing. > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF > AFMC AFLCMC/HNCP ; DIROCCO, ROGER E > GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin > O'Donnell ; > platformONE at redhat.com ; Tim Gast < > tg at braingu.com>; Bubb, Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC < > elijah.tramble.1 at us.af.mil>; > tj.zimmerman at braingu.com ; LOPEZDEURALDE, > RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; > Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR > USAF AFMC AFLCMC/HNCP ; Leonard, Michael > C. ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs < > taylor at redhat.com>; > Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF > AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security testing will be sent > through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and feel like I have > not done as thorough of a job as required. I was unable to log into the Web > UIs, as no one from the Platform1 team gave me the account information. I > had issues with Nessus, so the CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please read through it, > because there is some new information. I had to test as ec2-user again, > which is another big issue that needs to be resolved ASAP. The more I use > it and find out > how it is being used, the more extremely concerned I am. It has > multiple keys throughout the platform located in the .ssh directory, one of > which is world readable. On some hosts, a real user is using the ec2-user > account to create accounts, groups, and pull > docker files. The account is non-attributable, so we have no way of > knowing who is doing this. Someone could do serious damage with no > consequence. I understand that the ec2-user is needed for standing up an > ec2-image, but > this account should only be used for implementing IAC, so that the > changes implemented by ec2-user are codified. If manual admin is required, > that IAC should provision the appropriate attributable accounts, and those > accounts should > be used from then on. In my opinion, this is a critical finding and > needs to be addressed ASAP. > > > I will be available during the day tomorrow for any questions. > > Thanks > Colleen > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kodonnel at redhat.com Sun Jan 5 20:40:32 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Sun, 5 Jan 2020 15:40:32 -0500 Subject: [Platformone] [EXT] Re: Account for Dirty Dev In-Reply-To: <026EE33B-A855-4C40-A552-80FB8D9BDDEA@mitre.org> References: <1114349A-1B60-4C16-B453-A44057965097@braingu.com> <2033_1577134627_5E012A22_2033_997_1_CAHrof-fBuLdO+38+WvvTg88KZfZ2giK-OJHo1ZnSk__EAJ=OQg@mail.gmail.com> <026EE33B-A855-4C40-A552-80FB8D9BDDEA@mitre.org> Message-ID: Hello Tim, This is by design. We have freedom in AWS Gov. But we do not have the same freedom with the other target providers; CloudOne. We are setting up a blue / green deployment for the dev teams. up-prod-a / b, up-appdev-a / b. As enhancements are developed tested, staged and tested, feedback is received, with a green light we will fully redeploy to either a or b. The teams can test the apps in the new environment to provide feedback and make the decision to roll over to it or not. This also falls in line with the last email thread, on keys. "Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. " I am sure we can come up with some creative ways to reduce the total deployment time however it has not been top on our priority list. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:30 PM Miller, Timothy J. wrote: > Color me confused--y'all have spent time writing and validating that the > ansible roles are idempotent, so why do incremental changes to the plays > require a build from scratch? > > And since 3.5h of the 5h build time is sync'ing Satellite repos, at what > point will y'all checkpoint that data into an AMI or a resuable bucket? > > -- T > > ?On 12/23/19, 14:57, "platformone-bounces at redhat.com on behalf of Dino > Arachchi" > wrote: > > Tim, > > > There were some security features that didn't get merged into the > branch used to build the stack. This will require the same 4-5 hours as the > last build due to the fully disconnected nature of the build process. We do > have an open issue for connected environments > that will cut down build time, but that feature is still in progress. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: 848-203-1809 > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 2:49 PM Tim Gast wrote: > > > Thanks for the update. > What was the issue you found that would cause the entire stack to need > a rebuild? > Are we looking at another 5 hour build? > > > > Thanks, > -Tim > > > > On Dec 23, 2019, at 2:47 PM, Dino Arachchi > wrote: > > All, > > > While provisionin the dev accounts, an issue was identified in the > newly created up-appdev-a cluster. Code changes are being made now, after > which the stack will be rebuilt. Once this is completed and verified, I > will proceed with the creation of the new > accounts. These tasks are estimated to be completed today, and an > email will be sent out to confirm. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: 848-203-1809 > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 1:46 PM Kevin O'Donnell > wrote: > > > Thanks Dino > > > > -Kevin > > On Mon, Dec 23, 2019 at 2:38 PM Tim Gast wrote: > > > Thanks Dino. > > -Tim > > > > On Dec 23, 2019, at 1:37 PM, Dino Arachchi > wrote: > > > > > > > > > > Hi Tim, > > > I'm on PTO but am working on creating the dev accounts at the moment. > The team should expect to receive credentials within the next couple hours. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: 848-203-1809 > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 12:53 PM Tim Gast wrote: > > > Hi Kevin, > > > Any word on when to be looking for those dev accounts? > > Thanks, > -Tim > > > > On Dec 21, 2019, at 10:22 AM, Kevin O'Donnell > wrote: > > Build completed in 4 hours and 40 min. On Monday we will get the > accounts created and also add some code into our build to automate the > account creation process. > > > Everyone enjoy your weekend. > > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 9:19 PM Tim Gast wrote: > > > Thanks for the update Kevin. > > -Tim > > > > On Dec 20, 2019, at 7:26 PM, Kevin O'Donnell > wrote: > > > > > > Were about 3 hours in. > > > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 8:04 PM Kevin O'Donnell > wrote: > > > Hello Tim, > > > > The aws rbac changes impacted our deployment and it didn?t get the > deployment running till 5pm est. I keep an eye on the job. > > On Fri, Dec 20, 2019 at 5:19 PM Tim Gast wrote: > > > Hi. > Checking in since it?s been a few hours in flight. > Do we have a UP-DEV-A cluster and user access now? > > > Thanks, > -Tim > > > > On Dec 20, 2019, at 11:49 AM, Tim Gast wrote: > > > > > > Kevin, Mark, & Roc, > > > > > > > > > Below is the request for dev accounts for UP-APPDEV-A. > Can we please mirror what these folks have in the current UP-Prod-A > cluster. > > > Also, is it possible to provide an ?elevated read? role for certain > users (dcurrian) for discovery/troubleshooting? > > > Please let me know when the environment is live and ready for the team > to begin working with. > > > Thanks, > -Tim > > > Begin forwarded message: > > > > From: "Curran, Daniel M" > Date: December 20, 2019 at 11:15:39 AM CST > To: "tg at braingu.com" , "nino at braingu.com" < > nino at braingu.com> > Cc: "Buffaloe, Christopher" > Subject: Account for Dirty Dev > > > > > > Hey Tim and Nino, > > > Here's the account list for the Ginyu Force (CCAT) team > dcurran > rcepeda > msison > psoliz > atorres > dpalmer > rho > kmacias > jandrichak > > > Obviously we'll take all the privileges you're willing to give us but > if we can keep our dev privileges from the unified-platform cluster and put > us in a ginyu/ccat group so we can see each others projects that > would be enough. > It might also be advantageous to have one admin account (dcurran-adm > or ccat-adm) or at least one with elevated read privileges for discovery > purposes but I wouldn't call that a hard requirement. > > > Thanks, > Daniel > > > > > > > > > > > > > > > > -- > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Colleen.Feiglstok at ngc.com Sun Jan 5 20:42:05 2020 From: Colleen.Feiglstok at ngc.com (Feiglstok, Colleen M [US] (MS)) Date: Sun, 5 Jan 2020 20:42:05 +0000 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> Message-ID: Kevin, You are incorrect - the issue is not the name of the account, it is the fact that it is a non-attributable account. ANY shared/non-attributable accounts, no matter the name or privileges, will be in violation. Does that make more sense? We can?t have non-attributable group accounts. Please refer to Eric?s email sent on 12/20/2019 (and below) in regards to attribution. Let us know if you need more clarification. Colleen From: Kevin O'Donnell Sent: Sunday, January 5, 2020 3:34 PM To: Miller, Timothy J. Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: Re: EXT :Re: [EXT] Platform1 SAR Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 [https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png] On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T ?On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell > Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP >; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP >; platformONE at redhat.com; Tim Gast >; Bubb, Mike >; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP >; Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP >; Leonard, Michael C. >; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP >; Taylor Biggs >; Miller, Timothy J. >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP >; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com %20M:240-605-4654> M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet > wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) > Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP >; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP >; Kevin O'Donnell >; platformONE at redhat.com >; Tim Gast >; Bubb, Mike >; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; tj.zimmerman at braingu.com >; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP >; Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP >; Leonard, Michael C. >; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP >; Taylor Biggs >; Miller, Timothy J. >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP >; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen -------------- next part -------------- An HTML attachment was scrubbed... URL: From kodonnel at redhat.com Sun Jan 5 20:55:33 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Sun, 5 Jan 2020 15:55:33 -0500 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> Message-ID: Hello Colleen, As an example, in the windows world, we had that great user Administrator. This was a user that you could not delete. So in the past the teams that I worked with renamed it to LAdministrator. Given the password or keypair was reset by some official and it was stored in a safe. Same goes for the root account in linux / unix. Once we build the environment with user X (ec2-user) with the keys that we generated. The keys can be swapped by anyone on the team and they can be deleted. That's up to the larger team. As I stated we do not use the ec2-user. In an email that I sent, I asked Jet who if anyone should hold the keys once day two ops steps in? If it is determined that nobody should have an ec2-user/root/Administrator keypair or password that's fine. We will need to switch into a pure immutable build. In that case if we drop any communication with IDM LDAP and are unable to resolve the issue we will destroy the vpc and build a new clean one and the app teams can re run the code to deploy the apps. Just trying to make sure were all on the same page. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 3:42 PM Feiglstok, Colleen M [US] (MS) < Colleen.Feiglstok at ngc.com> wrote: > Kevin, > > > > You are incorrect - the issue is not the *name* of the account, it is the > fact that it is a *non-attributable account*. ANY shared/non-attributable > accounts, *no matter the name or privileges*, will be in violation. > > > > Does that make more sense? We can?t have non-attributable group accounts. > > > > Please refer to Eric?s email sent on 12/20/2019 (and below) in regards to > attribution. Let us know if you need more clarification. > > > > Colleen > > > > *From:* Kevin O'Donnell > *Sent:* Sunday, January 5, 2020 3:34 PM > *To:* Miller, Timothy J. > *Cc:* Blade, Eric D [US] (MS) ; Lastrilla, Jet < > jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP < > austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC > ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim > Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q > Capt USAF AFMC AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC > AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF > AFMC AFLCMC/HNCP ; Leonard, Michael C. < > leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP < > melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; CRISP, > JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, > STEVEN E CTR USAF AFMC AFLCMC/HNCP ; > Wilcox, John R. (San Antonio, TX) [US] (MS) > *Subject:* Re: EXT :Re: [EXT] Platform1 SAR > > > > Hello Tim, > > > > Welcome back. Happy New Year. > > > > We have a plan in place to swap out the ec2-user for another user seeing > as the major issue is the actual name of the user not its privs in prod. > > > > Incorrect up-prod-a keys were switched out before Xmas. This process will > need to be included in the Day Two OPS onboarding process. Jet, who should > be the holder of this key on day two? We are also ensuring that anyone that > authenticates to the nodes in the vpc use dedicated IDM accounts with per > user dedicated keys and sudo privs. > > > > Correct. Re-enforcing the Day Two requirements. And also supporting that > the pipelines and CI process can support CI/CD to only dev/staging. For > prod or any environment were the keys have been swapped and are not > included as private Git variables we can not support CD into prod/appdev if > so deemed. > > > > We are fast approaching policy with this topic. We should have a further > discussion this week. I will be in San Antonio on Wed and Thur. > > > > Do you have any additional mitigation techniques Tim? > > > > Thanks, > > > *KEVIN O'DONNELL * > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com M: > 240-605-4654 > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: > > Some more points of interest-- > > - The ec2-user account can't currently be avoided given the current design > of the IaC. > > - This SSH key provisioned for this account is shared across up-prod-a, > up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with Maintainer or > higher access to the GitLab repo that holds the customer's environment > definitions or the platform-infrastructure repo. > > -- T > > ?On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: > > I just want to clarify on the topic of the EC2-user. The key is > attribution. We need to know who or what is configuring the systems. > > We are repeatedly seeing the ec2-user account used to manually > configure systems. Ec2-user should only be used to perform programmatic > tasks from committed and > reviewed code as part of initial provisioning. If the code to > provision and configure systems is unavailable, then you must create > attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user will be flagged as > a violation and the source IP or user account logged as the violating > source. Any manual/command > line activities MUST be performed using an attributable user account. > > If that is not clear, please let me know and I will attempt to > elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) ; > BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; platformONE at redhat.com; Tim Gast < > tg at braingu.com>; > Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC > AFLCMC/HNC ; tj.zimmerman at braingu.com; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP < > richard.lopezdeuralde at us.af.mil>; Blade, Eric D [US] (MS) < > Eric.Blade at ngc.com>; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < > jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; > REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP < > melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; Miller, > Timothy J. ; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; > BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; > Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We will get GIT issues > crated for your findings and will prioritize the mitigation and implement > them as code in our future IAC deployments. Many of the findings in the > current VPC have been > mitigated in up-prod-b with our current code release. > > > > Please let us know when you have finished and we can power down the > host that you have been using for scanning. > > > > Note for everyone: Once we power down the ec2 instance ssh or port 22 > will not be externally accessible. Thus, mitigating many of the risks > associated with the ec2-user and the keys. > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet > wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If you want to take more > time, please use tomorrow to do your testing. > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF > AFMC AFLCMC/HNCP ; DIROCCO, ROGER E > GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin > O'Donnell ; > platformONE at redhat.com ; Tim Gast < > tg at braingu.com>; Bubb, Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC < > elijah.tramble.1 at us.af.mil>; > tj.zimmerman at braingu.com ; LOPEZDEURALDE, > RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; > Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR > USAF AFMC AFLCMC/HNCP ; Leonard, Michael > C. ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs < > taylor at redhat.com>; > Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF > AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security testing will be sent > through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and feel like I have > not done as thorough of a job as required. I was unable to log into the Web > UIs, as no one from the Platform1 team gave me the account information. I > had issues with Nessus, so the CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please read through it, > because there is some new information. I had to test as ec2-user again, > which is another big issue that needs to be resolved ASAP. The more I use > it and find out > how it is being used, the more extremely concerned I am. It has > multiple keys throughout the platform located in the .ssh directory, one of > which is world readable. On some hosts, a real user is using the ec2-user > account to create accounts, groups, and pull > docker files. The account is non-attributable, so we have no way of > knowing who is doing this. Someone could do serious damage with no > consequence. I understand that the ec2-user is needed for standing up an > ec2-image, but > this account should only be used for implementing IAC, so that the > changes implemented by ec2-user are codified. If manual admin is required, > that IAC should provision the appropriate attributable accounts, and those > accounts should > be used from then on. In my opinion, this is a critical finding and > needs to be addressed ASAP. > > > I will be available during the day tomorrow for any questions. > > Thanks > Colleen > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jayissi at redhat.com Mon Jan 6 19:22:51 2020 From: jayissi at redhat.com (Claude Ayissi) Date: Mon, 6 Jan 2020 14:22:51 -0500 Subject: [Platformone] [Non-DoD Source] Update on IATT Scans and POA&M In-Reply-To: References: Message-ID: +Mark Nissley As of now, status remains the same. - Two completed/false positives - One not applicable - Two outstanding: AV and FIPS in GRUB2 Thanks in advance, JOHNNY AYISSI SENIOR CONSULTANT Red Hat 1600 International Dr. Suite 800 McLean, VA 22102 Johnny.Ayissi at redhat.com M: (832) 390-0091 <832-390-0091> TRIED. TESTED. TRUSTED. @redhatnews Red Hat Red Hat On Thu, Dec 19, 2019 at 5:05 PM BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < steven.bogue.1.ctr at us.af.mil> wrote: > This is the POA&M that I have. I just want to make sure I have the latest > version. > > > > Steve > > > > *From:* Brenna Gordon > *Sent:* Thursday, December 19, 2019 2:53 PM > *To:* BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil> > *Cc:* Mark Nissley ; platformONE at redhat.com; > Lastrilla, Jet (jlastrilla at mitre.org) > *Subject:* Re: [Platformone] [Non-DoD Source] Update on IATT Scans and > POA&M > > > > Steven, > > > > Have there been updated scans run today? If so, can we get a copy? Most > recent I have is from yesterday and the POA&Ms for those are already in > draft. Highs: > > - Two completed/false positives > - One not applicable > - Two outstanding: AV and FIPS in GRUB2 > > Thanks, > > > > Brenna > > > > On Thu, Dec 19, 2019 at 3:32 PM BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil> wrote: > > Cory was working it, but he?s traveling. I think Brenna may be working it > as well. > > > > *From:* Mark Nissley > *Sent:* Thursday, December 19, 2019 2:24 PM > *To:* BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil> > *Cc:* platformONE at redhat.com; Lastrilla, Jet (jlastrilla at mitre.org) < > jlastrilla at mitre.org> > *Subject:* Re: [Non-DoD Source] Update on IATT Scans and POA&M > > > > Who is responsible for compiling this latest POA&M? > > > > *Mark NISSLEY*, PMP, CSM, LEAN > > PROGRAM MaNAGER & SR technical Project Manager > > North American Consulting, Public Sector > > > M: 850-530-3234 > > [image: Image removed by sender.] > > *Scheduled PTO: Dec 23 - Jan 03* > > > > > > On Thu, Dec 19, 2019 at 2:23 PM BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil> wrote: > > I?m still waiting on the latest POA&M from the scans. > > > > *From:* Mark Nissley > *Sent:* Thursday, December 19, 2019 2:17 PM > *To:* platformONE at redhat.com; Lastrilla, Jet (jlastrilla at mitre.org) < > jlastrilla at mitre.org>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil> > *Subject:* [Non-DoD Source] Update on IATT Scans and POA&M > > > > Team - > > > > Could someone provide a quick update on the readiness for this afternoon's > meeting with Nic? are we good to go? > > > > *Mark NISSLEY*, PMP, CSM, LEAN > > PROGRAM MaNAGER & SR technical Project Manager > > North American Consulting, Public Sector > > > M: 850-530-3234 > > [image: Image removed by sender.] > > *Scheduled PTO: Dec 23 - Jan 03* > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > > > > > -- > > *Brenna Gordon* > > Client Manager, NAPS > > Red Hat > > bgordon at redhat.com > M: 703-650-8755 > > [image: Image removed by sender.] > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 440 bytes Desc: not available URL: From mnissley at redhat.com Mon Jan 6 19:25:10 2020 From: mnissley at redhat.com (Mark Nissley) Date: Mon, 6 Jan 2020 13:25:10 -0600 Subject: [Platformone] [Non-DoD Source] Update on IATT Scans and POA&M In-Reply-To: References: Message-ID: No changes to the POA&M from our perspective at this time. We'll keep tracking this between today and Thursday AM. Mark NISSLEY, PMP, CSM, LEAN PROGRAM MaNAGER & SR technical Project Manager North American Consulting, Public Sector M: 850-530-3234 *Scheduled PTO: Dec 23 - Jan 03* On Mon, Jan 6, 2020 at 1:23 PM Claude Ayissi wrote: > +Mark Nissley > > As of now, status remains the same. > > - Two completed/false positives > - One not applicable > - Two outstanding: AV and FIPS in GRUB2 > > > Thanks in advance, > > > JOHNNY AYISSI > > SENIOR CONSULTANT > > Red Hat > > 1600 International Dr. Suite 800 > > McLean, VA 22102 > > Johnny.Ayissi at redhat.com M: (832) 390-0091 > <832-390-0091> > > TRIED. TESTED. TRUSTED. > @redhatnews Red Hat > Red Hat > > > > On Thu, Dec 19, 2019 at 5:05 PM BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil> wrote: > >> This is the POA&M that I have. I just want to make sure I have the >> latest version. >> >> >> >> Steve >> >> >> >> *From:* Brenna Gordon >> *Sent:* Thursday, December 19, 2019 2:53 PM >> *To:* BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < >> steven.bogue.1.ctr at us.af.mil> >> *Cc:* Mark Nissley ; platformONE at redhat.com; >> Lastrilla, Jet (jlastrilla at mitre.org) >> *Subject:* Re: [Platformone] [Non-DoD Source] Update on IATT Scans and >> POA&M >> >> >> >> Steven, >> >> >> >> Have there been updated scans run today? If so, can we get a copy? Most >> recent I have is from yesterday and the POA&Ms for those are already in >> draft. Highs: >> >> - Two completed/false positives >> - One not applicable >> - Two outstanding: AV and FIPS in GRUB2 >> >> Thanks, >> >> >> >> Brenna >> >> >> >> On Thu, Dec 19, 2019 at 3:32 PM BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >> wrote: >> >> Cory was working it, but he?s traveling. I think Brenna may be working >> it as well. >> >> >> >> *From:* Mark Nissley >> *Sent:* Thursday, December 19, 2019 2:24 PM >> *To:* BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < >> steven.bogue.1.ctr at us.af.mil> >> *Cc:* platformONE at redhat.com; Lastrilla, Jet (jlastrilla at mitre.org) < >> jlastrilla at mitre.org> >> *Subject:* Re: [Non-DoD Source] Update on IATT Scans and POA&M >> >> >> >> Who is responsible for compiling this latest POA&M? >> >> >> >> *Mark NISSLEY*, PMP, CSM, LEAN >> >> PROGRAM MaNAGER & SR technical Project Manager >> >> North American Consulting, Public Sector >> >> >> M: 850-530-3234 >> >> [image: Image removed by sender.] >> >> *Scheduled PTO: Dec 23 - Jan 03* >> >> >> >> >> >> On Thu, Dec 19, 2019 at 2:23 PM BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >> wrote: >> >> I?m still waiting on the latest POA&M from the scans. >> >> >> >> *From:* Mark Nissley >> *Sent:* Thursday, December 19, 2019 2:17 PM >> *To:* platformONE at redhat.com; Lastrilla, Jet (jlastrilla at mitre.org) < >> jlastrilla at mitre.org>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < >> steven.bogue.1.ctr at us.af.mil> >> *Subject:* [Non-DoD Source] Update on IATT Scans and POA&M >> >> >> >> Team - >> >> >> >> Could someone provide a quick update on the readiness for this >> afternoon's meeting with Nic? are we good to go? >> >> >> >> *Mark NISSLEY*, PMP, CSM, LEAN >> >> PROGRAM MaNAGER & SR technical Project Manager >> >> North American Consulting, Public Sector >> >> >> M: 850-530-3234 >> >> [image: Image removed by sender.] >> >> *Scheduled PTO: Dec 23 - Jan 03* >> >> _______________________________________________ >> platformONE mailing list >> platformONE at redhat.com >> https://www.redhat.com/mailman/listinfo/platformone >> >> >> >> >> -- >> >> *Brenna Gordon* >> >> Client Manager, NAPS >> >> Red Hat >> >> bgordon at redhat.com >> M: 703-650-8755 >> >> [image: Image removed by sender.] >> >> >> _______________________________________________ >> platformONE mailing list >> platformONE at redhat.com >> https://www.redhat.com/mailman/listinfo/platformone >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 440 bytes Desc: not available URL: From tmiller at mitre.org Tue Jan 7 04:53:05 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Tue, 7 Jan 2020 04:53:05 +0000 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> Message-ID: <6D5DCA96-1940-4CA3-BCBD-E744B63A7B7D@mitre.org> Just before I sent that email, I copied out the SSH privkeys from the CI/CD configuration for each of platform-infrastructure, up-appdev-a, and up-prod, dumped the moduli, and diff'd them. All three are the same. So that was Sunday. What did I miss? The rest of this email didn't parse, but it's been a 16h day for me, so maybe it's just me. The problem isn't necessarily the ec2-user named account by itself, it's the attribution. It's possible to establish attribution when processing logs since sshd will log the pubkey. However, that requires post-facto log analysis to track, and if there are multiple users active it can get confusing. The real problem is the current ec2-user privkey is being shared, intentionally and unintentionally, so there's absolutely no way to establish attribution. This is Yet Another Reason why you want to deploy as immutable infrastructure. This problem simply goes away. -- T ?On 1/5/20, 13:35, "Kevin O'Donnell" wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen From tmiller at mitre.org Tue Jan 7 05:01:18 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Tue, 7 Jan 2020 05:01:18 +0000 Subject: [Platformone] [EXT] Re: Account for Dirty Dev In-Reply-To: References: <1114349A-1B60-4C16-B453-A44057965097@braingu.com> <2033_1577134627_5E012A22_2033_997_1_CAHrof-fBuLdO+38+WvvTg88KZfZ2giK-OJHo1ZnSk__EAJ=OQg@mail.gmail.com> <026EE33B-A855-4C40-A552-80FB8D9BDDEA@mitre.org> Message-ID: <0B155967-5DBA-4B84-97CE-9274A058A71E@mitre.org> Unless you're preserving application data across redeploys, that's not an acceptable operating model. Project history is constantly accrued in the individual elements of the toolchain, and that history is important for analysis of application risk, projection of milestones, and team performance. If that's lost each time a feature is deployed, it will make it much harder to manage development projects. I'm also not sure I see any utility of a/b testing of the platform itself. The major workloads in the dev environment are builds and automated tests. What specific platform features require a/b test and evaluation? Is there a design record for this decision? -- T ?On 1/5/20, 13:41, "Kevin O'Donnell" wrote: Hello Tim, This is by design. We have freedom in AWS Gov. But we do not have the same freedom with the other target providers; CloudOne. We are setting up a blue / green deployment for the dev teams. up-prod-a / b, up-appdev-a / b. As enhancements are developed tested, staged and tested, feedback is received, with a green light we will fully redeploy to either a or b. The teams can test the apps in the new environment to provide feedback and make the decision to roll over to it or not. This also falls in line with the last email thread, on keys. "Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. " I am sure we can come up with some creative ways to reduce the total deployment time however it has not been top on our priority list. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:30 PM Miller, Timothy J. wrote: Color me confused--y'all have spent time writing and validating that the ansible roles are idempotent, so why do incremental changes to the plays require a build from scratch? And since 3.5h of the 5h build time is sync'ing Satellite repos, at what point will y'all checkpoint that data into an AMI or a resuable bucket? -- T On 12/23/19, 14:57, "platformone-bounces at redhat.com on behalf of Dino Arachchi" wrote: Tim, There were some security features that didn't get merged into the branch used to build the stack. This will require the same 4-5 hours as the last build due to the fully disconnected nature of the build process. We do have an open issue for connected environments that will cut down build time, but that feature is still in progress. Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 On Mon, Dec 23, 2019 at 2:49 PM Tim Gast wrote: Thanks for the update. What was the issue you found that would cause the entire stack to need a rebuild? Are we looking at another 5 hour build? Thanks, -Tim On Dec 23, 2019, at 2:47 PM, Dino Arachchi wrote: All, While provisionin the dev accounts, an issue was identified in the newly created up-appdev-a cluster. Code changes are being made now, after which the stack will be rebuilt. Once this is completed and verified, I will proceed with the creation of the new accounts. These tasks are estimated to be completed today, and an email will be sent out to confirm. Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 On Mon, Dec 23, 2019 at 1:46 PM Kevin O'Donnell wrote: Thanks Dino -Kevin On Mon, Dec 23, 2019 at 2:38 PM Tim Gast wrote: Thanks Dino. -Tim On Dec 23, 2019, at 1:37 PM, Dino Arachchi wrote: Hi Tim, I'm on PTO but am working on creating the dev accounts at the moment. The team should expect to receive credentials within the next couple hours. Best Regards, DINO ARACHCHI SENIOR CONSULTANT darachch at redhat.com M: 848-203-1809 On Mon, Dec 23, 2019 at 12:53 PM Tim Gast wrote: Hi Kevin, Any word on when to be looking for those dev accounts? Thanks, -Tim On Dec 21, 2019, at 10:22 AM, Kevin O'Donnell wrote: Build completed in 4 hours and 40 min. On Monday we will get the accounts created and also add some code into our build to automate the account creation process. Everyone enjoy your weekend. KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Fri, Dec 20, 2019 at 9:19 PM Tim Gast wrote: Thanks for the update Kevin. -Tim On Dec 20, 2019, at 7:26 PM, Kevin O'Donnell wrote: Were about 3 hours in. KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Fri, Dec 20, 2019 at 8:04 PM Kevin O'Donnell wrote: Hello Tim, The aws rbac changes impacted our deployment and it didn?t get the deployment running till 5pm est. I keep an eye on the job. On Fri, Dec 20, 2019 at 5:19 PM Tim Gast wrote: Hi. Checking in since it?s been a few hours in flight. Do we have a UP-DEV-A cluster and user access now? Thanks, -Tim On Dec 20, 2019, at 11:49 AM, Tim Gast wrote: Kevin, Mark, & Roc, Below is the request for dev accounts for UP-APPDEV-A. Can we please mirror what these folks have in the current UP-Prod-A cluster. Also, is it possible to provide an ?elevated read? role for certain users (dcurrian) for discovery/troubleshooting? Please let me know when the environment is live and ready for the team to begin working with. Thanks, -Tim Begin forwarded message: From: "Curran, Daniel M" Date: December 20, 2019 at 11:15:39 AM CST To: "tg at braingu.com" , "nino at braingu.com" Cc: "Buffaloe, Christopher" Subject: Account for Dirty Dev Hey Tim and Nino, Here's the account list for the Ginyu Force (CCAT) team dcurran rcepeda msison psoliz atorres dpalmer rho kmacias jandrichak Obviously we'll take all the privileges you're willing to give us but if we can keep our dev privileges from the unified-platform cluster and put us in a ginyu/ccat group so we can see each others projects that would be enough. It might also be advantageous to have one admin account (dcurran-adm or ccat-adm) or at least one with elevated read privileges for discovery purposes but I wouldn't call that a hard requirement. Thanks, Daniel -- KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 -- KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 _______________________________________________ platformONE mailing list platformONE at redhat.com https://www.redhat.com/mailman/listinfo/platformone From kodonnel at redhat.com Tue Jan 7 14:25:32 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Tue, 7 Jan 2020 09:25:32 -0500 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: <6D5DCA96-1940-4CA3-BCBD-E744B63A7B7D@mitre.org> References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> <6D5DCA96-1940-4CA3-BCBD-E744B63A7B7D@mitre.org> Message-ID: Hello Tim, Attached is the email that I sent out before the holiday. Please note that these keys are not stored in GIT for up-prod, the ec2-users authorized_keys file was removed and replaced with the new public key. Jet, who should hold the private key? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Mon, Jan 6, 2020 at 11:53 PM Miller, Timothy J. wrote: > Just before I sent that email, I copied out the SSH privkeys from the > CI/CD configuration for each of platform-infrastructure, up-appdev-a, and > up-prod, dumped the moduli, and diff'd them. All three are the same. So > that was Sunday. What did I miss? > > The rest of this email didn't parse, but it's been a 16h day for me, so > maybe it's just me. > > The problem isn't necessarily the ec2-user named account by itself, it's > the attribution. It's possible to establish attribution when processing > logs since sshd will log the pubkey. However, that requires post-facto log > analysis to track, and if there are multiple users active it can get > confusing. > > The real problem is the current ec2-user privkey is being shared, > intentionally and unintentionally, so there's absolutely no way to > establish attribution. > > This is Yet Another Reason why you want to deploy as immutable > infrastructure. This problem simply goes away. > > -- T > > ?On 1/5/20, 13:35, "Kevin O'Donnell" wrote: > > Hello Tim, > > > Welcome back. Happy New Year. > > > We have a plan in place to swap out the ec2-user for another user > seeing as the major issue is the actual name of the user not its privs in > prod. > > > Incorrect up-prod-a keys were switched out before Xmas. This process > will need to be included in the Day Two OPS onboarding process. Jet, who > should be the holder of this key on day two? We are also ensuring that > anyone that authenticates to the nodes > in the vpc use dedicated IDM accounts with per user dedicated keys > and sudo privs. > > > Correct. Re-enforcing the Day Two requirements. And also supporting > that the pipelines and CI process can support CI/CD to only dev/staging. > For prod or any environment were the keys have been swapped and are not > included as private Git variables we can > not support CD into prod/appdev if so deemed. > > > We are fast approaching policy with this topic. We should have a > further discussion this week. I will be in San Antonio on Wed and Thur. > > > Do you have any additional mitigation techniques Tim? > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: > > > Some more points of interest-- > > - The ec2-user account can't currently be avoided given the current > design of the IaC. > > - This SSH key provisioned for this account is shared across > up-prod-a, up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with Maintainer or > higher access to the GitLab repo that holds the customer's environment > definitions or the platform-infrastructure repo. > > -- T > > On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: > > I just want to clarify on the topic of the EC2-user. The key is > attribution. We need to know who or what is configuring the systems. > > We are repeatedly seeing the ec2-user account used to manually > configure systems. Ec2-user should only be used to perform programmatic > tasks from committed and > reviewed code as part of initial provisioning. If the code to > provision and configure systems is unavailable, then you must create > attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user will be flagged > as a violation and the source IP or user account logged as the violating > source. Any manual/command > line activities MUST be performed using an attributable user > account. > > If that is not clear, please let me know and I will attempt to > elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) ; > BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; > platformONE at redhat.com; Tim Gast ; > Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC > AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC > AFLCMC/HNCP ; Blade, Eric > D [US] (MS) ; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < > jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; > REINHARDT, MELISSA > A GG-13 USAF AFMC AFLCMC/HNCP ; > Taylor Biggs ; Miller, Timothy J. ; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < > joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil>; > Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We will get GIT > issues crated for your findings and will prioritize the mitigation and > implement them as code in our future IAC deployments. Many of the findings > in the current VPC have been > mitigated in up-prod-b with our current code release. > > > > Please let us know when you have finished and we can power down > the host that you have been using for scanning. > > > > > Note for everyone: Once we power down the ec2 instance ssh or port > 22 will not be externally accessible. Thus, mitigating many of the risks > associated with the ec2-user and the keys. > > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet < > jlastrilla at mitre.org> wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If you want to take > more time, please use tomorrow to do your testing. > > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt > USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E > GG-13 USAF AFMC ESC/AFLCMC/HNCP ; > Kevin O'Donnell ; > platformONE at redhat.com ; Tim Gast < > tg at braingu.com>; Bubb, > Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC < > elijah.tramble.1 at us.af.mil>; > tj.zimmerman at braingu.com ; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP < > richard.lopezdeuralde at us.af.mil>; > Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A > CTR USAF AFMC AFLCMC/HNCP ; Leonard, > Michael > C. ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs < > taylor at redhat.com>; > Miller, Timothy J. ; CRISP, JOSHUA M GS-09 > USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF > AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security testing will be sent > through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and feel like I > have not done as thorough of a job as required. I was unable to log into > the Web UIs, as no one from the Platform1 team gave me the account > information. I had issues with Nessus, so the CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please read through it, > because there is some new information. I had to test as ec2-user again, > which is another big issue that needs to be resolved ASAP. The more I use > it and find out > how it is being used, the more extremely concerned I am. It has > multiple keys throughout the platform located in the .ssh directory, one of > which is world readable. On some hosts, a real user is using the ec2-user > account to create accounts, groups, and > pull > docker files. The account is non-attributable, so we have no way > of knowing who is doing this. Someone could do serious damage with no > consequence. I understand that the ec2-user is needed for standing up an > ec2-image, but > this account should only be used for implementing IAC, so that the > changes implemented by ec2-user are codified. If manual admin is required, > that IAC should provision the appropriate attributable accounts, and those > accounts should > be used from then on. In my opinion, this is a critical finding > and needs to be addressed ASAP. > > > I will be available during the day tomorrow for any questions. > > Thanks > Colleen > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded message was scrubbed... From: "Kevin O'Donnell" Subject: Re: SSH Keys for up-prod Date: Sat, 21 Dec 2019 16:28:58 -0500 Size: 18374 URL: From kodonnel at redhat.com Tue Jan 7 14:30:06 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Tue, 7 Jan 2020 09:30:06 -0500 Subject: [Platformone] [EXT] Re: Account for Dirty Dev In-Reply-To: <0B155967-5DBA-4B84-97CE-9274A058A71E@mitre.org> References: <1114349A-1B60-4C16-B453-A44057965097@braingu.com> <2033_1577134627_5E012A22_2033_997_1_CAHrof-fBuLdO+38+WvvTg88KZfZ2giK-OJHo1ZnSk__EAJ=OQg@mail.gmail.com> <026EE33B-A855-4C40-A552-80FB8D9BDDEA@mitre.org> <0B155967-5DBA-4B84-97CE-9274A058A71E@mitre.org> Message-ID: Tim, The application should be preserving application data, project history, individual toolchain elements, and history. As an example, when the application moves into Cloud1 how will the application preserve this information. The move into Cloud1 is no different than the move into an updated product (say OCP 3.11.5, or OCP 4.x) and or a new cloud provider or even on-prem. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Tue, Jan 7, 2020 at 12:01 AM Miller, Timothy J. wrote: > Unless you're preserving application data across redeploys, that's not an > acceptable operating model. Project history is constantly accrued in the > individual elements of the toolchain, and that history is important for > analysis of application risk, projection of milestones, and team > performance. If that's lost each time a feature is deployed, it will make > it much harder to manage development projects. > > I'm also not sure I see any utility of a/b testing of the platform > itself. The major workloads in the dev environment are builds and > automated tests. What specific platform features require a/b test and > evaluation? Is there a design record for this decision? > > -- T > > ?On 1/5/20, 13:41, "Kevin O'Donnell" wrote: > > Hello Tim, > > > This is by design. We have freedom in AWS Gov. But we do not have the > same freedom with the other target providers; CloudOne. We are setting up a > blue / green deployment for the dev teams. up-prod-a / b, up-appdev-a / b. > As enhancements are developed tested, > staged and tested, feedback is received, with a green light we will > fully redeploy to either a or b. The teams can test the apps in the new > environment to provide feedback and make the decision to roll over to it or > not. > > > This also falls in line with the last email thread, on keys. > > > "Correct. Re-enforcing the Day Two requirements. And also supporting > that the pipelines and CI process can support CI/CD to only dev/staging. > For prod or any environment were the keys have been swapped and are not > included as private Git variables we can > not support CD into prod/appdev if so deemed. " > > > I am sure we can come up with some creative ways to reduce the total > deployment time however it has not been top on our priority list. > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:30 PM Miller, Timothy J. > wrote: > > > Color me confused--y'all have spent time writing and validating that > the ansible roles are idempotent, so why do incremental changes to the > plays require a build from scratch? > > And since 3.5h of the 5h build time is sync'ing Satellite repos, at > what point will y'all checkpoint that data into an AMI or a resuable bucket? > > -- T > > On 12/23/19, 14:57, "platformone-bounces at redhat.com on behalf of Dino > Arachchi" of darachch at redhat.com> wrote: > > Tim, > > > There were some security features that didn't get merged into the > branch used to build the stack. This will require the same 4-5 hours as the > last build due to the fully disconnected nature of the build process. We do > have an open issue for connected environments > that will cut down build time, but that feature is still in > progress. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: 848-203-1809 > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 2:49 PM Tim Gast wrote: > > > Thanks for the update. > What was the issue you found that would cause the entire stack to > need a rebuild? > > Are we looking at another 5 hour build? > > > > Thanks, > -Tim > > > > On Dec 23, 2019, at 2:47 PM, Dino Arachchi > wrote: > > All, > > > While provisionin the dev accounts, an issue was identified in the > newly created up-appdev-a cluster. Code changes are being made now, after > which the stack will be rebuilt. Once this is completed and verified, I > will proceed with the creation of the new > accounts. These tasks are estimated to be completed today, and an > email will be sent out to confirm. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: 848-203-1809 > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 1:46 PM Kevin O'Donnell < > kodonnel at redhat.com> wrote: > > > Thanks Dino > > > > -Kevin > > On Mon, Dec 23, 2019 at 2:38 PM Tim Gast wrote: > > > Thanks Dino. > > -Tim > > > > On Dec 23, 2019, at 1:37 PM, Dino Arachchi > wrote: > > > > > > > > > > Hi Tim, > > > I'm on PTO but am working on creating the dev accounts at the > moment. The team should expect to receive credentials within the next > couple hours. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: 848-203-1809 > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 12:53 PM Tim Gast wrote: > > > Hi Kevin, > > > Any word on when to be looking for those dev accounts? > > Thanks, > -Tim > > > > On Dec 21, 2019, at 10:22 AM, Kevin O'Donnell > wrote: > > Build completed in 4 hours and 40 min. On Monday we will get the > accounts created and also add some code into our build to automate the > account creation process. > > > Everyone enjoy your weekend. > > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 9:19 PM Tim Gast wrote: > > > Thanks for the update Kevin. > > -Tim > > > > On Dec 20, 2019, at 7:26 PM, Kevin O'Donnell > wrote: > > > > > > Were about 3 hours in. > > > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 8:04 PM Kevin O'Donnell < > kodonnel at redhat.com> wrote: > > > Hello Tim, > > > > The aws rbac changes impacted our deployment and it didn?t get the > deployment running till 5pm est. I keep an eye on the job. > > On Fri, Dec 20, 2019 at 5:19 PM Tim Gast wrote: > > > Hi. > Checking in since it?s been a few hours in flight. > Do we have a UP-DEV-A cluster and user access now? > > > Thanks, > -Tim > > > > On Dec 20, 2019, at 11:49 AM, Tim Gast wrote: > > > > > > Kevin, Mark, & Roc, > > > > > > > > > Below is the request for dev accounts for UP-APPDEV-A. > Can we please mirror what these folks have in the current > UP-Prod-A cluster. > > > Also, is it possible to provide an ?elevated read? role for > certain users (dcurrian) for discovery/troubleshooting? > > > Please let me know when the environment is live and ready for the > team to begin working with. > > > Thanks, > -Tim > > > Begin forwarded message: > > > > From: "Curran, Daniel M" > Date: December 20, 2019 at 11:15:39 AM CST > To: "tg at braingu.com" , "nino at braingu.com" < > nino at braingu.com> > Cc: "Buffaloe, Christopher" > Subject: Account for Dirty Dev > > > > > > Hey Tim and Nino, > > > Here's the account list for the Ginyu Force (CCAT) team > dcurran > rcepeda > msison > psoliz > atorres > dpalmer > rho > kmacias > jandrichak > > > Obviously we'll take all the privileges you're willing to give us > but if we can keep our dev privileges from the unified-platform cluster and > put us in a ginyu/ccat group so we can see each others projects that > would be enough. > It might also be advantageous to have one admin account > (dcurran-adm or ccat-adm) or at least one with elevated read privileges for > discovery purposes but I wouldn't call that a hard requirement. > > > Thanks, > Daniel > > > > > > > > > > > > > > > > -- > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlastrilla at mitre.org Tue Jan 7 14:48:09 2020 From: jlastrilla at mitre.org (Lastrilla, Jet) Date: Tue, 7 Jan 2020 14:48:09 +0000 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> <6D5DCA96-1940-4CA3-BCBD-E744B63A7B7D@mitre.org> Message-ID: Kevin, Essentially, we shouldn?t need the keys for ec2-user. We will need to rename and disable. We will need to create new accounts (specific to each function) with the correct privilege. I can see each application having a unique service account that will deploy/configure then be disabled. All other service accounts need to be specific and attributable. This is the goal to move toward? no expectation of big bang to get there, just need to plan on when we are going to get there, and what the design is. For prod admin, we will need to define what processes need privileges and create specific accounts for each. We shouldn?t need a traditional break glass account. We should be rebuilding from known good if we cannot recover. The keys for the production service accounts should be held by the day 2 team. Jet Lastrilla, CISSP MITRE | Systems Security Engineer San Antonio, TX 210-208-4867 jlastrilla at mitre.org Jethro.lastrilla.ctr at us.af.mil (NIPR) Jethro.s.lastrilla.ctr at mail.smil.mil (SIPR) Jethro.lastrilla_ctr at af.ic.gov (JWICS) From: Kevin O'Donnell Sent: Tuesday, January 7, 2020 8:26 AM To: Miller, Timothy J. Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: Re: EXT :Re: [EXT] Platform1 SAR Hello Tim, Attached is the email that I sent out before the holiday. Please note that these keys are not stored in GIT for up-prod, the ec2-users authorized_keys file was removed and replaced with the new public key. Jet, who should hold the private key? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 [https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png] On Mon, Jan 6, 2020 at 11:53 PM Miller, Timothy J. > wrote: Just before I sent that email, I copied out the SSH privkeys from the CI/CD configuration for each of platform-infrastructure, up-appdev-a, and up-prod, dumped the moduli, and diff'd them. All three are the same. So that was Sunday. What did I miss? The rest of this email didn't parse, but it's been a 16h day for me, so maybe it's just me. The problem isn't necessarily the ec2-user named account by itself, it's the attribution. It's possible to establish attribution when processing logs since sshd will log the pubkey. However, that requires post-facto log analysis to track, and if there are multiple users active it can get confusing. The real problem is the current ec2-user privkey is being shared, intentionally and unintentionally, so there's absolutely no way to establish attribution. This is Yet Another Reason why you want to deploy as immutable infrastructure. This problem simply goes away. -- T ?On 1/5/20, 13:35, "Kevin O'Donnell" > wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com %20M:240-605-4654> M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell > Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP >; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP >; platformONE at redhat.com; Tim Gast >; Bubb, Mike >; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP >; Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP >; Leonard, Michael C. >; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP >; Taylor Biggs >; Miller, Timothy J. >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP >; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com %20M:240-605-4654> M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet > wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) > Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP >; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP >; Kevin O'Donnell >; platformONE at redhat.com >; Tim Gast >; Bubb, Mike >; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; tj.zimmerman at braingu.com >; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP >; Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP >; Leonard, Michael C. >; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP >; Taylor Biggs >; Miller, Timothy J. >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP >; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmiller at mitre.org Tue Jan 7 15:09:41 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Tue, 7 Jan 2020 15:09:41 +0000 Subject: [Platformone] [EXT] Re: Account for Dirty Dev In-Reply-To: References: <1114349A-1B60-4C16-B453-A44057965097@braingu.com> <2033_1577134627_5E012A22_2033_997_1_CAHrof-fBuLdO+38+WvvTg88KZfZ2giK-OJHo1ZnSk__EAJ=OQg@mail.gmail.com> <026EE33B-A855-4C40-A552-80FB8D9BDDEA@mitre.org> <0B155967-5DBA-4B84-97CE-9274A058A71E@mitre.org> Message-ID: Not what I'm saying; this has nothing to do with mission app data persistence. A *dev toolchain* accrues project history. If you dump this data because the dev cluster was redeployed, we lose valuable history and can complicate future iterations. For example, Sonarqube tracks code quality issues over time for a given project key. With each iteration, developers and security leads are evaluating findings, suppressing false positives, accepting some true positives, planning POA&Ms for others, and updating sonarqube's profile appropriately. If you dump this data when re-dploying the dev environment, each dev team will have to do all of this work again. -- T > -----Original Message----- > From: Kevin O'Donnell > Sent: Tuesday, January 07, 2020 8:30 AM > To: Miller, Timothy J. > Cc: Dino Arachchi ; Tim Gast ; > platformONE at redhat.com; Lastrilla, Jet > Subject: Re: [Platformone] [EXT] Re: Account for Dirty Dev > > Tim, > > The application should be preserving application data, project history, > individual toolchain elements, and history. > > As an example, when the application moves into Cloud1 how will the > application preserve this information. The move into Cloud1 is no different > than the move into an updated product (say OCP 3.11.5, or OCP 4.x) and or a > new cloud provider or even on-prem. > > Thanks, > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com 4654> M: 240-605-4654 > > > > > > On Tue, Jan 7, 2020 at 12:01 AM Miller, Timothy J. > wrote: > > > Unless you're preserving application data across redeploys, that's not > an acceptable operating model. Project history is constantly accrued in the > individual elements of the toolchain, and that history is important for analysis > of application risk, projection of milestones, and team performance. If that's > lost each time a feature is deployed, it will make it much harder to manage > development projects. > > I'm also not sure I see any utility of a/b testing of the platform itself. > The major workloads in the dev environment are builds and automated > tests. What specific platform features require a/b test and evaluation? Is > there a design record for this decision? > > -- T > > ?On 1/5/20, 13:41, "Kevin O'Donnell" > wrote: > > Hello Tim, > > > This is by design. We have freedom in AWS Gov. But we do not > have the same freedom with the other target providers; CloudOne. We are > setting up a blue / green deployment for the dev teams. up-prod-a / b, up- > appdev-a / b. As enhancements are developed tested, > staged and tested, feedback is received, with a green light we will > fully redeploy to either a or b. The teams can test the apps in the new > environment to provide feedback and make the decision to roll over to it or > not. > > > This also falls in line with the last email thread, on keys. > > > "Correct. Re-enforcing the Day Two requirements. And also > supporting that the pipelines and CI process can support CI/CD to only > dev/staging. For prod or any environment were the keys have been > swapped and are not included as private Git variables we can > not support CD into prod/appdev if so deemed. " > > > I am sure we can come up with some creative ways to reduce the > total deployment time however it has not been top on our priority list. > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:30 PM Miller, Timothy J. > wrote: > > > Color me confused--y'all have spent time writing and validating that > the ansible roles are idempotent, so why do incremental changes to the > plays require a build from scratch? > > And since 3.5h of the 5h build time is sync'ing Satellite repos, at > what point will y'all checkpoint that data into an AMI or a resuable bucket? > > -- T > > On 12/23/19, 14:57, "platformone-bounces at redhat.com > on behalf of Dino Arachchi" > bounces at redhat.com> on behalf > of darachch at redhat.com > > wrote: > > Tim, > > > There were some security features that didn't get merged into > the branch used to build the stack. This will require the same 4-5 hours as the > last build due to the fully disconnected nature of the build process. We do > have an open issue for connected environments > that will cut down build time, but that feature is still in progress. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: > 848-203-1809 > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 2:49 PM Tim Gast > wrote: > > > Thanks for the update. > What was the issue you found that would cause the entire stack > to need a rebuild? > > Are we looking at another 5 hour build? > > > > Thanks, > -Tim > > > > On Dec 23, 2019, at 2:47 PM, Dino Arachchi > > wrote: > > All, > > > While provisionin the dev accounts, an issue was identified in the > newly created up-appdev-a cluster. Code changes are being made now, after > which the stack will be rebuilt. Once this is completed and verified, I will > proceed with the creation of the new > accounts. These tasks are estimated to be completed today, and > an email will be sent out to confirm. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: > 848-203-1809 > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 1:46 PM Kevin O'Donnell > > wrote: > > > Thanks Dino > > > > -Kevin > > On Mon, Dec 23, 2019 at 2:38 PM Tim Gast > wrote: > > > Thanks Dino. > > -Tim > > > > On Dec 23, 2019, at 1:37 PM, Dino Arachchi > > wrote: > > > > > > > > > > Hi Tim, > > > I'm on PTO but am working on creating the dev accounts at the > moment. The team should expect to receive credentials within the next > couple hours. > > > Best Regards, > > > DINO ARACHCHI > SENIOR CONSULTANT > darachch at redhat.com M: > 848-203-1809 > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 12:53 PM Tim Gast > wrote: > > > Hi Kevin, > > > Any word on when to be looking for those dev accounts? > > Thanks, > -Tim > > > > On Dec 21, 2019, at 10:22 AM, Kevin O'Donnell > > wrote: > > Build completed in 4 hours and 40 min. On Monday we will get > the accounts created and also add some code into our build to automate the > account creation process. > > > Everyone enjoy your weekend. > > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 9:19 PM Tim Gast > wrote: > > > Thanks for the update Kevin. > > -Tim > > > > On Dec 20, 2019, at 7:26 PM, Kevin O'Donnell > > wrote: > > > > > > Were about 3 hours in. > > > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 8:04 PM Kevin O'Donnell > > wrote: > > > Hello Tim, > > > > The aws rbac changes impacted our deployment and it didn?t get > the deployment running till 5pm est. I keep an eye on the job. > > On Fri, Dec 20, 2019 at 5:19 PM Tim Gast > wrote: > > > Hi. > Checking in since it?s been a few hours in flight. > Do we have a UP-DEV-A cluster and user access now? > > > Thanks, > -Tim > > > > On Dec 20, 2019, at 11:49 AM, Tim Gast > wrote: > > > > > > Kevin, Mark, & Roc, > > > > > > > > > Below is the request for dev accounts for UP-APPDEV-A. > Can we please mirror what these folks have in the current UP- > Prod-A cluster. > > > Also, is it possible to provide an ?elevated read? role for certain > users (dcurrian) for discovery/troubleshooting? > > > Please let me know when the environment is live and ready for > the team to begin working with. > > > Thanks, > -Tim > > > Begin forwarded message: > > > > From: "Curran, Daniel M" > Date: December 20, 2019 at 11:15:39 AM CST > To: "tg at braingu.com " > >, "nino at braingu.com > " > > Cc: "Buffaloe, Christopher" > > Subject: Account for Dirty Dev > > > > > > Hey Tim and Nino, > > > Here's the account list for the Ginyu Force (CCAT) team > dcurran > rcepeda > msison > psoliz > atorres > dpalmer > rho > kmacias > jandrichak > > > Obviously we'll take all the privileges you're willing to give us but > if we can keep our dev privileges from the unified-platform cluster and put > us in a ginyu/ccat group so we can see each others projects that > would be enough. > It might also be advantageous to have one admin account > (dcurran-adm or ccat-adm) or at least one with elevated read privileges for > discovery purposes but I wouldn't call that a hard requirement. > > > Thanks, > Daniel > > > > > > > > > > > > > > > > -- > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > > > > > > From tmiller at mitre.org Tue Jan 7 15:31:48 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Tue, 7 Jan 2020 15:31:48 +0000 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> <6D5DCA96-1940-4CA3-BCBD-E744B63A7B7D@mitre.org> Message-ID: Cloud-ready images all have a default account (in AWS it's ec2-user, root, or ubuntu depending on distro). This account has password disabled in /etc/shadow, and IIRC all cloud-ready images have sshd configured to allow *only* pubkey authN. cloud-init data is used to seed the account's authorized_keys file at first boot. You can't get rid of the cloud-ready account without creating an exactly equivalent account, unless you decide to deploy immutable (no ssh access at all). This would potentially complicate playbooks since the `ansible_user` fact may have to change between tasks. It's trivial to deploy additional accounts w/ appropriate authorized_keys files for each but deleting or disabling the cloud-ready default account is an unnecessary complication. I suggest that the ec2-user be retained but only the appropriate gitlab-runner hold the key. The runner can be whitelisted and any other session using that key can be easily flagged. -- T > -----Original Message----- > From: Lastrilla, Jet > Sent: Tuesday, January 07, 2020 8:48 AM > To: Kevin O'Donnell ; Miller, Timothy J. > > Cc: Blade, Eric D [US] (MS) ; Feiglstok, Colleen M [US] > (MS) ; BRYAN, AUSTEN R Capt USAF AFMC > AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF > AFMC ESC/AFLCMC/HNCP ; > platformONE at redhat.com; Tim Gast ; Bubb, Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > ; tj.zimmerman at braingu.com; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP > ; RAMIREZ, JOSE A CTR USAF AFMC > AFLCMC/HNCP ; Leonard, Michael C. > ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs > ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > ; BOGUE, STEVEN E CTR USAF AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: RE: EXT :Re: [EXT] Platform1 SAR > > Kevin, > > > > Essentially, we shouldn?t need the keys for ec2-user. We will need to > rename and disable. We will need to create new accounts (specific to each > function) with the correct privilege. I can see each application having a > unique service account that will deploy/configure then be disabled. All other > service accounts need to be specific and attributable. This is the goal to > move toward? no expectation of big bang to get there, just need to plan on > when we are going to get there, and what the design is. > > > > For prod admin, we will need to define what processes need privileges and > create specific accounts for each. We shouldn?t need a traditional break glass > account. We should be rebuilding from known good if we cannot recover. > The keys for the production service accounts should be held by the day 2 > team. > > > > Jet Lastrilla, CISSP > > MITRE | Systems Security Engineer > > San Antonio, TX > > 210-208-4867 > > jlastrilla at mitre.org > > Jethro.lastrilla.ctr at us.af.mil (NIPR) > > Jethro.s.lastrilla.ctr at mail.smil.mil > (SIPR) > > Jethro.lastrilla_ctr at af.ic.gov (JWICS) > > > > From: Kevin O'Donnell > Sent: Tuesday, January 7, 2020 8:26 AM > To: Miller, Timothy J. > Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet > ; Feiglstok, Colleen M [US] (MS) > ; BRYAN, AUSTEN R Capt USAF AFMC > AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF > AFMC ESC/AFLCMC/HNCP ; > platformONE at redhat.com; Tim Gast ; Bubb, Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > ; tj.zimmerman at braingu.com; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP > ; RAMIREZ, JOSE A CTR USAF AFMC > AFLCMC/HNCP ; Leonard, Michael C. > ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs > ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > ; BOGUE, STEVEN E CTR USAF AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: Re: EXT :Re: [EXT] Platform1 SAR > > > > Hello Tim, > > > > Attached is the email that I sent out before the holiday. Please note that > these keys are not stored in GIT for up-prod, the ec2-users authorized_keys > file was removed and replaced with the new public key. > > > > Jet, who should hold the private key? > > > > Thanks, > > > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com 4654> M: 240-605-4654 > > > > > > > > > > On Mon, Jan 6, 2020 at 11:53 PM Miller, Timothy J. > wrote: > > Just before I sent that email, I copied out the SSH privkeys from the > CI/CD configuration for each of platform-infrastructure, up-appdev-a, and > up-prod, dumped the moduli, and diff'd them. All three are the same. So > that was Sunday. What did I miss? > > The rest of this email didn't parse, but it's been a 16h day for me, so > maybe it's just me. > > The problem isn't necessarily the ec2-user named account by itself, > it's the attribution. It's possible to establish attribution when processing logs > since sshd will log the pubkey. However, that requires post-facto log analysis > to track, and if there are multiple users active it can get confusing. > > The real problem is the current ec2-user privkey is being shared, > intentionally and unintentionally, so there's absolutely no way to establish > attribution. > > This is Yet Another Reason why you want to deploy as immutable > infrastructure. This problem simply goes away. > > -- T > > ?On 1/5/20, 13:35, "Kevin O'Donnell" > wrote: > > Hello Tim, > > > Welcome back. Happy New Year. > > > We have a plan in place to swap out the ec2-user for another user > seeing as the major issue is the actual name of the user not its privs in prod. > > > Incorrect up-prod-a keys were switched out before Xmas. This > process will need to be included in the Day Two OPS onboarding process. Jet, > who should be the holder of this key on day two? We are also ensuring that > anyone that authenticates to the nodes > in the vpc use dedicated IDM accounts with per user dedicated > keys and sudo privs. > > > Correct. Re-enforcing the Day Two requirements. And also > supporting that the pipelines and CI process can support CI/CD to only > dev/staging. For prod or any environment were the keys have been > swapped and are not included as private Git variables we can > not support CD into prod/appdev if so deemed. > > > We are fast approaching policy with this topic. We should have a > further discussion this week. I will be in San Antonio on Wed and Thur. > > > Do you have any additional mitigation techniques Tim? > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: > > > Some more points of interest-- > > - The ec2-user account can't currently be avoided given the current > design of the IaC. > > - This SSH key provisioned for this account is shared across up- > prod-a, up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with Maintainer or > higher access to the GitLab repo that holds the customer's environment > definitions or the platform-infrastructure repo. > > -- T > > On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: > > I just want to clarify on the topic of the EC2-user. The key is > attribution. We need to know who or what is configuring the systems. > > We are repeatedly seeing the ec2-user account used to manually > configure systems. Ec2-user should only be used to perform programmatic > tasks from committed and > reviewed code as part of initial provisioning. If the code to > provision and configure systems is unavailable, then you must create > attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user will be flagged > as a violation and the source IP or user account logged as the violating source. > Any manual/command > line activities MUST be performed using an attributable user > account. > > If that is not clear, please let me know and I will attempt to > elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > > > Cc: Feiglstok, Colleen M [US] (MS) >; BRYAN, AUSTEN R Capt USAF AFMC > AFLCMC/HNCP >; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP > >; > platformONE at redhat.com ; > Tim Gast >; > Bubb, Mike >; > TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > >; > tj.zimmerman at braingu.com > ; LOPEZDEURALDE, RICHARD A Lt Col > USAF AFMC AFLCMC/HNCP >; Blade, Eric > D [US] (MS) >; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP > >; > Leonard, Michael C. >; > REINHARDT, MELISSA > A GG-13 USAF AFMC AFLCMC/HNCP > >; > Taylor Biggs >; Miller, > Timothy J. >; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > >; BOGUE, > STEVEN E CTR USAF AFMC AFLCMC/HNCP >; > Wilcox, John R. (San Antonio, TX) [US] (MS) > > > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We will get GIT > issues crated for your findings and will prioritize the mitigation and > implement them as code in our future IAC deployments. Many of the > findings in the current VPC have been > mitigated in up-prod-b with our current code release. > > > > Please let us know when you have finished and we can power > down the host that you have been using for scanning. > > > > > Note for everyone: Once we power down the ec2 instance ssh or > port 22 will not be externally accessible. Thus, mitigating many of the risks > associated with the ec2-user and the keys. > > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet > wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If you want to take > more time, please use tomorrow to do your testing. > > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) > > > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet > >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP > >; DIROCCO, > ROGER E > GG-13 USAF AFMC ESC/AFLCMC/HNCP > >; Kevin > O'Donnell >; > platformONE at redhat.com > >; Tim Gast > >; Bubb, > Mike > >; TRAMBLE, > ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; > tj.zimmerman at braingu.com > >; LOPEZDEURALDE, RICHARD A Lt Col > USAF AFMC AFLCMC/HNCP >; > Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC > AFLCMC/HNCP >; Leonard, > Michael > C. >; > REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP > >; > Taylor Biggs >; > Miller, Timothy J. > >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > >; BOGUE, > STEVEN E CTR USAF > AFMC > AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) > [US] (MS) > > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security testing will be > sent through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and feel like I have > not done as thorough of a job as required. I was unable to log into the Web > UIs, as no one from the Platform1 team gave me the account information. I > had issues with Nessus, so the CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please read through it, > because there is some new information. I had to test as ec2-user again, > which is another big issue that needs to be resolved ASAP. The more I use it > and find out > how it is being used, the more extremely concerned I am. It has > multiple keys throughout the platform located in the .ssh directory, one of > which is world readable. On some hosts, a real user is using the ec2-user > account to create accounts, groups, and > pull > docker files. The account is non-attributable, so we have no way > of knowing who is doing this. Someone could do serious damage with no > consequence. I understand that the ec2-user is needed for standing up an > ec2-image, but > this account should only be used for implementing IAC, so that > the changes implemented by ec2-user are codified. If manual admin is > required, that IAC should provision the appropriate attributable accounts, > and those accounts should > be used from then on. In my opinion, this is a critical finding and > needs to be addressed ASAP. > > > I will be available during the day tomorrow for any questions. > > Thanks > Colleen > > > > > > > > > > > > > > > From kodonnel at redhat.com Tue Jan 7 15:38:30 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Tue, 7 Jan 2020 10:38:30 -0500 Subject: [Platformone] [EXT] Re: Account for Dirty Dev In-Reply-To: References: <1114349A-1B60-4C16-B453-A44057965097@braingu.com> <2033_1577134627_5E012A22_2033_997_1_CAHrof-fBuLdO+38+WvvTg88KZfZ2giK-OJHo1ZnSk__EAJ=OQg@mail.gmail.com> <026EE33B-A855-4C40-A552-80FB8D9BDDEA@mitre.org> <0B155967-5DBA-4B84-97CE-9274A058A71E@mitre.org> Message-ID: Tim, This should all be automated. I will be on site tomorrow, let's discuss. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Tue, Jan 7, 2020 at 10:11 AM Miller, Timothy J. wrote: > Not what I'm saying; this has nothing to do with mission app data > persistence. > > A *dev toolchain* accrues project history. If you dump this data because > the dev cluster was redeployed, we lose valuable history and can complicate > future iterations. > > For example, Sonarqube tracks code quality issues over time for a given > project key. With each iteration, developers and security leads are > evaluating findings, suppressing false positives, accepting some true > positives, planning POA&Ms for others, and updating sonarqube's profile > appropriately. > > If you dump this data when re-dploying the dev environment, each dev team > will have to do all of this work again. > > -- T > > > -----Original Message----- > > From: Kevin O'Donnell > > Sent: Tuesday, January 07, 2020 8:30 AM > > To: Miller, Timothy J. > > Cc: Dino Arachchi ; Tim Gast ; > > platformONE at redhat.com; Lastrilla, Jet > > Subject: Re: [Platformone] [EXT] Re: Account for Dirty Dev > > > > Tim, > > > > The application should be preserving application data, project history, > > individual toolchain elements, and history. > > > > As an example, when the application moves into Cloud1 how will the > > application preserve this information. The move into Cloud1 is no > different > > than the move into an updated product (say OCP 3.11.5, or OCP 4.x) and > or a > > new cloud provider or even on-prem. > > > > Thanks, > > > > > > KEVIN O'DONNELL > > > > ARCHITECT MANAGER > > > > Red Hat Red Hat NA Public Sector Consulting > > > > kodonnell at redhat.com > 4654> M: 240-605-4654 > > > > > > > > > > > > On Tue, Jan 7, 2020 at 12:01 AM Miller, Timothy J. > > wrote: > > > > > > Unless you're preserving application data across redeploys, that's > not > > an acceptable operating model. Project history is constantly accrued in > the > > individual elements of the toolchain, and that history is important for > analysis > > of application risk, projection of milestones, and team performance. If > that's > > lost each time a feature is deployed, it will make it much harder to > manage > > development projects. > > > > I'm also not sure I see any utility of a/b testing of the platform > itself. > > The major workloads in the dev environment are builds and automated > > tests. What specific platform features require a/b test and > evaluation? Is > > there a design record for this decision? > > > > -- T > > > > ?On 1/5/20, 13:41, "Kevin O'Donnell" > > wrote: > > > > Hello Tim, > > > > > > This is by design. We have freedom in AWS Gov. But we do not > > have the same freedom with the other target providers; CloudOne. We are > > setting up a blue / green deployment for the dev teams. up-prod-a / b, > up- > > appdev-a / b. As enhancements are developed tested, > > staged and tested, feedback is received, with a green light > we will > > fully redeploy to either a or b. The teams can test the apps in the new > > environment to provide feedback and make the decision to roll over to it > or > > not. > > > > > > This also falls in line with the last email thread, on keys. > > > > > > "Correct. Re-enforcing the Day Two requirements. And also > > supporting that the pipelines and CI process can support CI/CD to only > > dev/staging. For prod or any environment were the keys have been > > swapped and are not included as private Git variables we can > > not support CD into prod/appdev if so deemed. " > > > > > > I am sure we can come up with some creative ways to reduce the > > total deployment time however it has not been top on our priority list. > > > > > > Thanks, > > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > > > > > kodonnell at redhat.com > > > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:30 PM Miller, Timothy J. < > tmiller at mitre.org > > > wrote: > > > > > > Color me confused--y'all have spent time writing and > validating that > > the ansible roles are idempotent, so why do incremental changes to the > > plays require a build from scratch? > > > > And since 3.5h of the 5h build time is sync'ing Satellite > repos, at > > what point will y'all checkpoint that data into an AMI or a resuable > bucket? > > > > -- T > > > > On 12/23/19, 14:57, "platformone-bounces at redhat.com > > on behalf of Dino Arachchi" > > > bounces at redhat.com> on behalf > > of darachch at redhat.com > > > wrote: > > > > Tim, > > > > > > There were some security features that didn't get merged > into > > the branch used to build the stack. This will require the same 4-5 hours > as the > > last build due to the fully disconnected nature of the build process. We > do > > have an open issue for connected environments > > that will cut down build time, but that feature is still > in progress. > > > > > > Best Regards, > > > > > > DINO ARACHCHI > > SENIOR CONSULTANT > > darachch at redhat.com M: > > 848-203-1809 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 2:49 PM Tim Gast > > wrote: > > > > > > Thanks for the update. > > What was the issue you found that would cause the entire > stack > > to need a rebuild? > > > > Are we looking at another 5 hour build? > > > > > > > > Thanks, > > -Tim > > > > > > > > On Dec 23, 2019, at 2:47 PM, Dino Arachchi > > > wrote: > > > > All, > > > > > > While provisionin the dev accounts, an issue was > identified in the > > newly created up-appdev-a cluster. Code changes are being made now, after > > which the stack will be rebuilt. Once this is completed and verified, I > will > > proceed with the creation of the new > > accounts. These tasks are estimated to be completed > today, and > > an email will be sent out to confirm. > > > > > > Best Regards, > > > > > > DINO ARACHCHI > > SENIOR CONSULTANT > > darachch at redhat.com M: > > 848-203-1809 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 1:46 PM Kevin O'Donnell > > > wrote: > > > > > > Thanks Dino > > > > > > > > -Kevin > > > > On Mon, Dec 23, 2019 at 2:38 PM Tim Gast > > wrote: > > > > > > Thanks Dino. > > > > -Tim > > > > > > > > On Dec 23, 2019, at 1:37 PM, Dino Arachchi > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi Tim, > > > > > > I'm on PTO but am working on creating the dev accounts at > the > > moment. The team should expect to receive credentials within the next > > couple hours. > > > > > > Best Regards, > > > > > > DINO ARACHCHI > > SENIOR CONSULTANT > > darachch at redhat.com M: > > 848-203-1809 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 23, 2019 at 12:53 PM Tim Gast > > wrote: > > > > > > Hi Kevin, > > > > > > Any word on when to be looking for those dev accounts? > > > > Thanks, > > -Tim > > > > > > > > On Dec 21, 2019, at 10:22 AM, Kevin O'Donnell > > > wrote: > > > > Build completed in 4 hours and 40 min. On Monday we will > get > > the accounts created and also add some code into our build to automate > the > > account creation process. > > > > > > Everyone enjoy your weekend. > > > > > > > > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > > > > > kodonnell at redhat.com > > > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 9:19 PM Tim Gast > > wrote: > > > > > > Thanks for the update Kevin. > > > > -Tim > > > > > > > > On Dec 20, 2019, at 7:26 PM, Kevin O'Donnell > > > wrote: > > > > > > > > > > > > Were about 3 hours in. > > > > > > > > > > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > > > > > kodonnell at redhat.com > > > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Dec 20, 2019 at 8:04 PM Kevin O'Donnell > > > wrote: > > > > > > Hello Tim, > > > > > > > > The aws rbac changes impacted our deployment and it didn?t > get > > the deployment running till 5pm est. I keep an eye on the job. > > > > On Fri, Dec 20, 2019 at 5:19 PM Tim Gast > > wrote: > > > > > > Hi. > > Checking in since it?s been a few hours in flight. > > Do we have a UP-DEV-A cluster and user access now? > > > > > > Thanks, > > -Tim > > > > > > > > On Dec 20, 2019, at 11:49 AM, Tim Gast > > wrote: > > > > > > > > > > > > Kevin, Mark, & Roc, > > > > > > > > > > > > > > > > > > Below is the request for dev accounts for UP-APPDEV-A. > > Can we please mirror what these folks have in the current > UP- > > Prod-A cluster. > > > > > > Also, is it possible to provide an ?elevated read? role > for certain > > users (dcurrian) for discovery/troubleshooting? > > > > > > Please let me know when the environment is live and ready > for > > the team to begin working with. > > > > > > Thanks, > > -Tim > > > > > > Begin forwarded message: > > > > > > > > From: "Curran, Daniel M" > > Date: December 20, 2019 at 11:15:39 AM CST > > To: "tg at braingu.com " > > >, "nino at braingu.com > > " > > > > Cc: "Buffaloe, Christopher" > > > > Subject: Account for Dirty Dev > > > > > > > > > > > > Hey Tim and Nino, > > > > > > Here's the account list for the Ginyu Force (CCAT) team > > dcurran > > rcepeda > > msison > > psoliz > > atorres > > dpalmer > > rho > > kmacias > > jandrichak > > > > > > Obviously we'll take all the privileges you're willing to > give us but > > if we can keep our dev privileges from the unified-platform cluster and > put > > us in a ginyu/ccat group so we can see each others projects that > > would be enough. > > It might also be advantageous to have one admin account > > (dcurran-adm or ccat-adm) or at least one with elevated read privileges > for > > discovery purposes but I wouldn't call that a hard requirement. > > > > > > Thanks, > > Daniel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > > > > > kodonnell at redhat.com > > > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > > > > > kodonnell at redhat.com > > > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > platformONE mailing list > > platformONE at redhat.com > > https://www.redhat.com/mailman/listinfo/platformone > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlastrilla at mitre.org Tue Jan 7 15:38:53 2020 From: jlastrilla at mitre.org (Lastrilla, Jet) Date: Tue, 7 Jan 2020 15:38:53 +0000 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> <6D5DCA96-1940-4CA3-BCBD-E744B63A7B7D@mitre.org> , Message-ID: Tim, While I agree that this adds complication, this is a compliance thing, not security. It we want to fight this compliance requirement, we need to show how protected that account and keys are. We also need to clearly articulate how our mitigation?s will prevent and notify if this account is used. This account can also not be used for day to day ops. You don?t use root accounts for that. Each machine to machine exchange should have a unique account with minimum privileges. Outlook for iOS ________________________________ From: Miller, Timothy J. Sent: Tuesday, January 7, 2020 9:31:48 AM To: Lastrilla, Jet ; Kevin O'Donnell Cc: Blade, Eric D [US] (MS) ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: RE: EXT :Re: [EXT] Platform1 SAR Cloud-ready images all have a default account (in AWS it's ec2-user, root, or ubuntu depending on distro). This account has password disabled in /etc/shadow, and IIRC all cloud-ready images have sshd configured to allow *only* pubkey authN. cloud-init data is used to seed the account's authorized_keys file at first boot. You can't get rid of the cloud-ready account without creating an exactly equivalent account, unless you decide to deploy immutable (no ssh access at all). This would potentially complicate playbooks since the `ansible_user` fact may have to change between tasks. It's trivial to deploy additional accounts w/ appropriate authorized_keys files for each but deleting or disabling the cloud-ready default account is an unnecessary complication. I suggest that the ec2-user be retained but only the appropriate gitlab-runner hold the key. The runner can be whitelisted and any other session using that key can be easily flagged. -- T > -----Original Message----- > From: Lastrilla, Jet > Sent: Tuesday, January 07, 2020 8:48 AM > To: Kevin O'Donnell ; Miller, Timothy J. > > Cc: Blade, Eric D [US] (MS) ; Feiglstok, Colleen M [US] > (MS) ; BRYAN, AUSTEN R Capt USAF AFMC > AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF > AFMC ESC/AFLCMC/HNCP ; > platformONE at redhat.com; Tim Gast ; Bubb, Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > ; tj.zimmerman at braingu.com; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP > ; RAMIREZ, JOSE A CTR USAF AFMC > AFLCMC/HNCP ; Leonard, Michael C. > ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs > ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > ; BOGUE, STEVEN E CTR USAF AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: RE: EXT :Re: [EXT] Platform1 SAR > > Kevin, > > > > Essentially, we shouldn?t need the keys for ec2-user. We will need to > rename and disable. We will need to create new accounts (specific to each > function) with the correct privilege. I can see each application having a > unique service account that will deploy/configure then be disabled. All other > service accounts need to be specific and attributable. This is the goal to > move toward? no expectation of big bang to get there, just need to plan on > when we are going to get there, and what the design is. > > > > For prod admin, we will need to define what processes need privileges and > create specific accounts for each. We shouldn?t need a traditional break glass > account. We should be rebuilding from known good if we cannot recover. > The keys for the production service accounts should be held by the day 2 > team. > > > > Jet Lastrilla, CISSP > > MITRE | Systems Security Engineer > > San Antonio, TX > > 210-208-4867 > > jlastrilla at mitre.org > > Jethro.lastrilla.ctr at us.af.mil (NIPR) > > Jethro.s.lastrilla.ctr at mail.smil.mil > (SIPR) > > Jethro.lastrilla_ctr at af.ic.gov (JWICS) > > > > From: Kevin O'Donnell > Sent: Tuesday, January 7, 2020 8:26 AM > To: Miller, Timothy J. > Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet > ; Feiglstok, Colleen M [US] (MS) > ; BRYAN, AUSTEN R Capt USAF AFMC > AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF > AFMC ESC/AFLCMC/HNCP ; > platformONE at redhat.com; Tim Gast ; Bubb, Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > ; tj.zimmerman at braingu.com; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP > ; RAMIREZ, JOSE A CTR USAF AFMC > AFLCMC/HNCP ; Leonard, Michael C. > ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs > ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > ; BOGUE, STEVEN E CTR USAF AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: Re: EXT :Re: [EXT] Platform1 SAR > > > > Hello Tim, > > > > Attached is the email that I sent out before the holiday. Please note that > these keys are not stored in GIT for up-prod, the ec2-users authorized_keys > file was removed and replaced with the new public key. > > > > Jet, who should hold the private key? > > > > Thanks, > > > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com 4654> M: 240-605-4654 > > > > > > > > > > On Mon, Jan 6, 2020 at 11:53 PM Miller, Timothy J. > wrote: > > Just before I sent that email, I copied out the SSH privkeys from the > CI/CD configuration for each of platform-infrastructure, up-appdev-a, and > up-prod, dumped the moduli, and diff'd them. All three are the same. So > that was Sunday. What did I miss? > > The rest of this email didn't parse, but it's been a 16h day for me, so > maybe it's just me. > > The problem isn't necessarily the ec2-user named account by itself, > it's the attribution. It's possible to establish attribution when processing logs > since sshd will log the pubkey. However, that requires post-facto log analysis > to track, and if there are multiple users active it can get confusing. > > The real problem is the current ec2-user privkey is being shared, > intentionally and unintentionally, so there's absolutely no way to establish > attribution. > > This is Yet Another Reason why you want to deploy as immutable > infrastructure. This problem simply goes away. > > -- T > > ?On 1/5/20, 13:35, "Kevin O'Donnell" > wrote: > > Hello Tim, > > > Welcome back. Happy New Year. > > > We have a plan in place to swap out the ec2-user for another user > seeing as the major issue is the actual name of the user not its privs in prod. > > > Incorrect up-prod-a keys were switched out before Xmas. This > process will need to be included in the Day Two OPS onboarding process. Jet, > who should be the holder of this key on day two? We are also ensuring that > anyone that authenticates to the nodes > in the vpc use dedicated IDM accounts with per user dedicated > keys and sudo privs. > > > Correct. Re-enforcing the Day Two requirements. And also > supporting that the pipelines and CI process can support CI/CD to only > dev/staging. For prod or any environment were the keys have been > swapped and are not included as private Git variables we can > not support CD into prod/appdev if so deemed. > > > We are fast approaching policy with this topic. We should have a > further discussion this week. I will be in San Antonio on Wed and Thur. > > > Do you have any additional mitigation techniques Tim? > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: > > > Some more points of interest-- > > - The ec2-user account can't currently be avoided given the current > design of the IaC. > > - This SSH key provisioned for this account is shared across up- > prod-a, up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with Maintainer or > higher access to the GitLab repo that holds the customer's environment > definitions or the platform-infrastructure repo. > > -- T > > On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: > > I just want to clarify on the topic of the EC2-user. The key is > attribution. We need to know who or what is configuring the systems. > > We are repeatedly seeing the ec2-user account used to manually > configure systems. Ec2-user should only be used to perform programmatic > tasks from committed and > reviewed code as part of initial provisioning. If the code to > provision and configure systems is unavailable, then you must create > attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user will be flagged > as a violation and the source IP or user account logged as the violating source. > Any manual/command > line activities MUST be performed using an attributable user > account. > > If that is not clear, please let me know and I will attempt to > elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > > > Cc: Feiglstok, Colleen M [US] (MS) >; BRYAN, AUSTEN R Capt USAF AFMC > AFLCMC/HNCP >; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP > >; > platformONE at redhat.com ; > Tim Gast >; > Bubb, Mike >; > TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > >; > tj.zimmerman at braingu.com > ; LOPEZDEURALDE, RICHARD A Lt Col > USAF AFMC AFLCMC/HNCP >; Blade, Eric > D [US] (MS) >; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP > >; > Leonard, Michael C. >; > REINHARDT, MELISSA > A GG-13 USAF AFMC AFLCMC/HNCP > >; > Taylor Biggs >; Miller, > Timothy J. >; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > >; BOGUE, > STEVEN E CTR USAF AFMC AFLCMC/HNCP >; > Wilcox, John R. (San Antonio, TX) [US] (MS) > > > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We will get GIT > issues crated for your findings and will prioritize the mitigation and > implement them as code in our future IAC deployments. Many of the > findings in the current VPC have been > mitigated in up-prod-b with our current code release. > > > > Please let us know when you have finished and we can power > down the host that you have been using for scanning. > > > > > Note for everyone: Once we power down the ec2 instance ssh or > port 22 will not be externally accessible. Thus, mitigating many of the risks > associated with the ec2-user and the keys. > > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet > wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If you want to take > more time, please use tomorrow to do your testing. > > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) > > > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet > >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP > >; DIROCCO, > ROGER E > GG-13 USAF AFMC ESC/AFLCMC/HNCP > >; Kevin > O'Donnell >; > platformONE at redhat.com > >; Tim Gast > >; Bubb, > Mike > >; TRAMBLE, > ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; > tj.zimmerman at braingu.com > >; LOPEZDEURALDE, RICHARD A Lt Col > USAF AFMC AFLCMC/HNCP >; > Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC > AFLCMC/HNCP >; Leonard, > Michael > C. >; > REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP > >; > Taylor Biggs >; > Miller, Timothy J. > >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > >; BOGUE, > STEVEN E CTR USAF > AFMC > AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) > [US] (MS) > > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security testing will be > sent through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and feel like I have > not done as thorough of a job as required. I was unable to log into the Web > UIs, as no one from the Platform1 team gave me the account information. I > had issues with Nessus, so the CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please read through it, > because there is some new information. I had to test as ec2-user again, > which is another big issue that needs to be resolved ASAP. The more I use it > and find out > how it is being used, the more extremely concerned I am. It has > multiple keys throughout the platform located in the .ssh directory, one of > which is world readable. On some hosts, a real user is using the ec2-user > account to create accounts, groups, and > pull > docker files. The account is non-attributable, so we have no way > of knowing who is doing this. Someone could do serious damage with no > consequence. I understand that the ec2-user is needed for standing up an > ec2-image, but > this account should only be used for implementing IAC, so that > the changes implemented by ec2-user are codified. If manual admin is > required, that IAC should provision the appropriate attributable accounts, > and those accounts should > be used from then on. In my opinion, this is a critical finding and > needs to be addressed ASAP. > > > I will be available during the day tomorrow for any questions. > > Thanks > Colleen > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kodonnel at redhat.com Tue Jan 7 15:42:09 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Tue, 7 Jan 2020 10:42:09 -0500 Subject: [Platformone] EXT :Re: [EXT] Platform1 SAR In-Reply-To: References: <14445_1576794925_5DFBFB2D_14445_540_1_666ebf0de51644a89e809ab03bd8be27@XCGVAG22.northgrum.com> <05de1c3d47f34cc2811beea332efbfaf@XCGVAG22.northgrum.com> <6D5DCA96-1940-4CA3-BCBD-E744B63A7B7D@mitre.org> Message-ID: Tim/Jet, I have a clear path forward on this. We can discuss this in more detail tomorrow while I am onsite. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Tue, Jan 7, 2020 at 10:40 AM Lastrilla, Jet wrote: > Tim, > > While I agree that this adds complication, this is a compliance thing, not > security. It we want to fight this compliance requirement, we need to show > how protected that account and keys are. We also need to clearly articulate > how our mitigation?s will prevent and notify if this account is used. > > This account can also not be used for day to day ops. You don?t use root > accounts for that. Each machine to machine exchange should have a unique > account with minimum privileges. > > Outlook for iOS > ------------------------------ > *From:* Miller, Timothy J. > *Sent:* Tuesday, January 7, 2020 9:31:48 AM > *To:* Lastrilla, Jet ; Kevin O'Donnell < > kodonnel at redhat.com> > *Cc:* Blade, Eric D [US] (MS) ; Feiglstok, Colleen M > [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC > AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC > ESC/AFLCMC/HNCP ; platformONE at redhat.com < > platformONE at redhat.com>; Tim Gast ; Bubb, Mike < > mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC < > elijah.tramble.1 at us.af.mil>; tj.zimmerman at braingu.com < > tj.zimmerman at braingu.com>; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC > AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF > AFMC AFLCMC/HNCP ; Leonard, Michael C. < > leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP < > melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; CRISP, > JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, > STEVEN E CTR USAF AFMC AFLCMC/HNCP ; > Wilcox, John R. (San Antonio, TX) [US] (MS) > *Subject:* RE: EXT :Re: [EXT] Platform1 SAR > > Cloud-ready images all have a default account (in AWS it's ec2-user, root, > or ubuntu depending on distro). This account has password disabled in > /etc/shadow, and IIRC all cloud-ready images have sshd configured to allow > *only* pubkey authN. cloud-init data is used to seed the account's > authorized_keys file at first boot. You can't get rid of the cloud-ready > account without creating an exactly equivalent account, unless you decide > to deploy immutable (no ssh access at all). > > This would potentially complicate playbooks since the `ansible_user` fact > may have to change between tasks. > > It's trivial to deploy additional accounts w/ appropriate authorized_keys > files for each but deleting or disabling the cloud-ready default account is > an unnecessary complication. I suggest that the ec2-user be retained but > only the appropriate gitlab-runner hold the key. The runner can be > whitelisted and any other session using that key can be easily flagged. > > -- T > > > -----Original Message----- > > From: Lastrilla, Jet > > Sent: Tuesday, January 07, 2020 8:48 AM > > To: Kevin O'Donnell ; Miller, Timothy J. > > > > Cc: Blade, Eric D [US] (MS) ; Feiglstok, Colleen M > [US] > > (MS) ; BRYAN, AUSTEN R Capt USAF AFMC > > AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF > > AFMC ESC/AFLCMC/HNCP ; > > platformONE at redhat.com; Tim Gast ; Bubb, Mike > > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > > ; tj.zimmerman at braingu.com; > > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP > > ; RAMIREZ, JOSE A CTR USAF AFMC > > AFLCMC/HNCP ; Leonard, Michael C. > > ; REINHARDT, MELISSA A GG-13 USAF AFMC > > AFLCMC/HNCP ; Taylor Biggs > > ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > > ; BOGUE, STEVEN E CTR USAF AFMC > > AFLCMC/HNCP ; Wilcox, John R. (San > > Antonio, TX) [US] (MS) > > Subject: RE: EXT :Re: [EXT] Platform1 SAR > > > > Kevin, > > > > > > > > Essentially, we shouldn?t need the keys for ec2-user. We will need to > > rename and disable. We will need to create new accounts (specific to > each > > function) with the correct privilege. I can see each application having > a > > unique service account that will deploy/configure then be disabled. All > other > > service accounts need to be specific and attributable. This is the goal > to > > move toward? no expectation of big bang to get there, just need to plan > on > > when we are going to get there, and what the design is. > > > > > > > > For prod admin, we will need to define what processes need privileges and > > create specific accounts for each. We shouldn?t need a traditional > break glass > > account. We should be rebuilding from known good if we cannot recover. > > The keys for the production service accounts should be held by the day 2 > > team. > > > > > > > > Jet Lastrilla, CISSP > > > > MITRE | Systems Security Engineer > > > > San Antonio, TX > > > > 210-208-4867 > > > > jlastrilla at mitre.org > > > > > > Jethro.lastrilla.ctr at us.af.mil > (NIPR) > > > > Jethro.s.lastrilla.ctr at mail.smil.mil < > mailto:Jethro.s.lastrilla.ctr at mail.smil.mil > > > > (SIPR) > > > > Jethro.lastrilla_ctr at af.ic.gov > (JWICS) > > > > > > > > From: Kevin O'Donnell > > Sent: Tuesday, January 7, 2020 8:26 AM > > To: Miller, Timothy J. > > Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet > > ; Feiglstok, Colleen M [US] (MS) > > ; BRYAN, AUSTEN R Capt USAF AFMC > > AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF > > AFMC ESC/AFLCMC/HNCP ; > > platformONE at redhat.com; Tim Gast ; Bubb, Mike > > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > > ; tj.zimmerman at braingu.com; > > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP > > ; RAMIREZ, JOSE A CTR USAF AFMC > > AFLCMC/HNCP ; Leonard, Michael C. > > ; REINHARDT, MELISSA A GG-13 USAF AFMC > > AFLCMC/HNCP ; Taylor Biggs > > ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > > ; BOGUE, STEVEN E CTR USAF AFMC > > AFLCMC/HNCP ; Wilcox, John R. (San > > Antonio, TX) [US] (MS) > > Subject: Re: EXT :Re: [EXT] Platform1 SAR > > > > > > > > Hello Tim, > > > > > > > > Attached is the email that I sent out before the holiday. Please note > that > > these keys are not stored in GIT for up-prod, the ec2-users > authorized_keys > > file was removed and replaced with the new public key. > > > > > > > > Jet, who should hold the private key? > > > > > > > > Thanks, > > > > > > > > > > KEVIN O'DONNELL > > > > ARCHITECT MANAGER > > > > Red Hat Red Hat NA Public Sector Consulting > > > > kodonnell at redhat.com > 4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > On Mon, Jan 6, 2020 at 11:53 PM Miller, Timothy J. > > > wrote: > > > > Just before I sent that email, I copied out the SSH privkeys from > the > > CI/CD configuration for each of platform-infrastructure, up-appdev-a, and > > up-prod, dumped the moduli, and diff'd them. All three are the same. So > > that was Sunday. What did I miss? > > > > The rest of this email didn't parse, but it's been a 16h day for > me, so > > maybe it's just me. > > > > The problem isn't necessarily the ec2-user named account by > itself, > > it's the attribution. It's possible to establish attribution when > processing logs > > since sshd will log the pubkey. However, that requires post-facto log > analysis > > to track, and if there are multiple users active it can get confusing. > > > > The real problem is the current ec2-user privkey is being shared, > > intentionally and unintentionally, so there's absolutely no way to > establish > > attribution. > > > > This is Yet Another Reason why you want to deploy as immutable > > infrastructure. This problem simply goes away. > > > > -- T > > > > ?On 1/5/20, 13:35, "Kevin O'Donnell" > > > wrote: > > > > Hello Tim, > > > > > > Welcome back. Happy New Year. > > > > > > We have a plan in place to swap out the ec2-user for another > user > > seeing as the major issue is the actual name of the user not its privs > in prod. > > > > > > Incorrect up-prod-a keys were switched out before Xmas. This > > process will need to be included in the Day Two OPS onboarding process. > Jet, > > who should be the holder of this key on day two? We are also ensuring > that > > anyone that authenticates to the nodes > > in the vpc use dedicated IDM accounts with per user dedicated > > keys and sudo privs. > > > > > > Correct. Re-enforcing the Day Two requirements. And also > > supporting that the pipelines and CI process can support CI/CD to only > > dev/staging. For prod or any environment were the keys have been > > swapped and are not included as private Git variables we can > > not support CD into prod/appdev if so deemed. > > > > > > We are fast approaching policy with this topic. We should > have a > > further discussion this week. I will be in San Antonio on Wed and Thur. > > > > > > Do you have any additional mitigation techniques Tim? > > > > > > Thanks, > > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > > > > > kodonnell at redhat.com > > > > > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. < > tmiller at mitre.org > > > > wrote: > > > > > > Some more points of interest-- > > > > - The ec2-user account can't currently be avoided given the > current > > design of the IaC. > > > > - This SSH key provisioned for this account is shared across > up- > > prod-a, up-dev-a, and DSOP. > > > > - The ec2-user SSH privkey is available to anyone with > Maintainer or > > higher access to the GitLab repo that holds the customer's environment > > definitions or the platform-infrastructure repo. > > > > -- T > > > > On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" < > Eric.Blade at ngc.com > > > > wrote: > > > > I just want to clarify on the topic of the EC2-user. The > key is > > attribution. We need to know who or what is configuring the systems. > > > > We are repeatedly seeing the ec2-user account used to > manually > > configure systems. Ec2-user should only be used to perform programmatic > > tasks from committed and > > reviewed code as part of initial provisioning. If the > code to > > provision and configure systems is unavailable, then you must create > > attributable accounts to perform changes to the systems. > > > > Any use of ec2-user remote login or su to ec2-user will > be flagged > > as a violation and the source IP or user account logged as the violating > source. > > Any manual/command > > line activities MUST be performed using an attributable > user > > account. > > > > If that is not clear, please let me know and I will > attempt to > > elaborate further. > > > > Thank you > > > > > > Eric Blade, GXPN, GPEN > > Unified Platform System Coordinator > > Northrop Grumman Mission Systems > > Work: 410.649.0706 > > Mobile: 240.258.8089 > > > > > > > > From: Kevin O'Donnell > > > > > > > Sent: Thursday, December 19, 2019 6:22 PM > > To: Lastrilla, Jet mailto:jlastrilla at mitre.org > > > > > > Cc: Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com > > > >; > BRYAN, AUSTEN R Capt USAF AFMC > > AFLCMC/HNCP > > >; > > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP > > > >; > > platformONE at redhat.com > ; > > Tim Gast > >; > > Bubb, Mike > >; > > TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > > > >; > > tj.zimmerman at braingu.com > > > ; > LOPEZDEURALDE, RICHARD A Lt Col > > USAF AFMC AFLCMC/HNCP > > >; Blade, Eric > > D [US] (MS) > >; > > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP > > > >; > > Leonard, Michael C. > >; > > REINHARDT, MELISSA > > A GG-13 USAF AFMC AFLCMC/HNCP > > > >; > > Taylor Biggs > >; Miller, > > Timothy J. > >; > > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > > > >; BOGUE, > > STEVEN E CTR USAF AFMC AFLCMC/HNCP > > >; > > Wilcox, John R. (San Antonio, TX) [US] (MS) > > > > > > Subject: EXT :Re: [EXT] Platform1 SAR > > > > Colleen, > > > > > > Thank you for the results and recommendations. We will > get GIT > > issues crated for your findings and will prioritize the mitigation and > > implement them as code in our future IAC deployments. Many of the > > findings in the current VPC have been > > mitigated in up-prod-b with our current code release. > > > > > > > > Please let us know when you have finished and we can power > > down the host that you have been using for scanning. > > > > > > > > > > Note for everyone: Once we power down the ec2 instance > ssh or > > port 22 will not be externally accessible. Thus, mitigating many of the > risks > > associated with the ec2-user and the keys. > > > > > > > > > > Thanks, > > > > > > > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > > > kodonnell at redhat.com > > > > > > %20M:240-605-4654> M: 240-605-4654 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet < > jlastrilla at mitre.org > > > > wrote: > > > > > > Thanks Colleen. Sorry for the rushed feeling. If you want > to take > > more time, please use tomorrow to do your testing. > > > > > > > > Thank you for all you do!!!! > > > > Get Outlook for iOS > > > > > > > > ________________________________________ > > > > From: Feiglstok, Colleen M [US] (MS) > > > > > > Sent: Thursday, December 19, 2019 4:35:15 PM > > To: Lastrilla, Jet mailto:jlastrilla at mitre.org > > > >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP > > > >; DIROCCO, > > ROGER E > > GG-13 USAF AFMC ESC/AFLCMC/HNCP > > > >; Kevin > > O'Donnell > >; > > platformONE at redhat.com > > > > >; Tim Gast > > > >; Bubb, > > Mike > > > >; TRAMBLE, > > ELIJAH Q Capt USAF AFMC AFLCMC/HNC > > >; > > tj.zimmerman at braingu.com > > > < > tj.zimmerman at braingu.com > > > >; > LOPEZDEURALDE, RICHARD A Lt Col > > USAF AFMC AFLCMC/HNCP > > >; > > Blade, Eric D [US] (MS) > > >; RAMIREZ, JOSE A CTR > USAF AFMC > > AFLCMC/HNCP > > > >; Leonard, > > Michael > > C. > >; > > REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP > > > >; > > Taylor Biggs > >; > > Miller, Timothy J. mailto:tmiller at mitre.org > > > >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP > > > >; BOGUE, > > STEVEN E CTR USAF > > AFMC > > AFLCMC/HNCP > > >; > Wilcox, John R. (San Antonio, TX) > > [US] (MS) > > > > Subject: [EXT] Platform1 SAR > > > > > > > > All, > > > > The SAR and raw results from the new security testing > will be > > sent through NGSafe in a few moments. > > > > As usual, I felt very rushed with the testing, and feel > like I have > > not done as thorough of a job as required. I was unable to log into the > Web > > UIs, as no one from the Platform1 team gave me the account information. I > > had issues with Nessus, so the CVE?s > > were found through OSCAP this time. > > > > A lot is the same as the last report, but please read > through it, > > because there is some new information. I had to test as ec2-user again, > > which is another big issue that needs to be resolved ASAP. The more I > use it > > and find out > > how it is being used, the more extremely concerned I am. > It has > > multiple keys throughout the platform located in the .ssh directory, one > of > > which is world readable. On some hosts, a real user is using the ec2-user > > account to create accounts, groups, and > > pull > > docker files. The account is non-attributable, so we > have no way > > of knowing who is doing this. Someone could do serious damage with no > > consequence. I understand that the ec2-user is needed for standing up an > > ec2-image, but > > this account should only be used for implementing IAC, so > that > > the changes implemented by ec2-user are codified. If manual admin is > > required, that IAC should provision the appropriate attributable > accounts, > > and those accounts should > > be used from then on. In my opinion, this is a critical > finding and > > needs to be addressed ASAP. > > > > > > I will be available during the day tomorrow for any > questions. > > > > Thanks > > Colleen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wayne.starr.1 at us.af.mil Thu Jan 9 19:55:45 2020 From: wayne.starr.1 at us.af.mil (STARR, WAYNE E 1st Lt USAF AETC 333 TRS/DIU) Date: Thu, 9 Jan 2020 19:55:45 +0000 Subject: [Platformone] Current Unified Platform IaC Message-ID: Good Afternoon, I was asked by Capt Austen Bryan to get myself familiar with the current UP IATT environment as well as the new LevelUP IaC UP environment that will replace it as I move into my new role here with LevelUP/UP. I was able to find the IaC ansible configuration that I believe is for the new IaC ansible environment (https://dccscr.dsop.io/levelup-automation/aws-infrastructure) and Keegan was able to get me access to the Gitlab groups, but wanted me to throw this email together so that I could get pointed in the right direction for what is currently in production awaiting IATT (there seem to be many repos out there that may or may not be up to date). Thanks, Very Respectfully, WAYNE E. STARR, 1st Lt, USAF 90th Cyber Operations Squadron (e) wayne.starr.1 at us.af.mil (c) (970) 366-2530 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kodonnel at redhat.com Fri Jan 10 17:04:39 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Fri, 10 Jan 2020 12:04:39 -0500 Subject: [Platformone] Fwd: ALARM: "awsec2-i-097ad12e6af679146-High-Network-Out" in AWS GovCloud (US-West) In-Reply-To: <0101016f90689e74-51e3e90b-49d9-4c4b-9d81-fe6ee9f3691d-000000@us-west-2.amazonses.com> References: <0101016f90689e74-51e3e90b-49d9-4c4b-9d81-fe6ee9f3691d-000000@us-west-2.amazonses.com> Message-ID: Hello All, Can we disable these KubeCon alerts? I have gotten over a thousand emails in the last couple of days. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 ---------- Forwarded message --------- From: KubeCon Date: Fri, Jan 10, 2020 at 12:02 PM Subject: ALARM: "awsec2-i-097ad12e6af679146-High-Network-Out" in AWS GovCloud (US-West) To: You are receiving this email because your Amazon CloudWatch Alarm "awsec2-i-097ad12e6af679146-High-Network-Out" in the AWS GovCloud (US-West) region has entered the ALARM state, because "Threshold Crossed: 1 datapoint [2.077810916E9 (10/01/20 17:01:00)] was greater than or equal to the threshold (1.0E9)." at "Friday 10 January, 2020 17:02:52 UTC". View this alarm in the AWS Management Console: https://us-gov-west-1.console.amazonaws-us-gov.com/cloudwatch/home?region=us-gov-west-1#s=Alarms&alarm=awsec2-i-097ad12e6af679146-High-Network-Out Alarm Details: - Name: awsec2-i-097ad12e6af679146-High-Network-Out - Description: Created from EC2 Console - State Change: OK -> ALARM - Reason for State Change: Threshold Crossed: 1 datapoint [2.077810916E9 (10/01/20 17:01:00)] was greater than or equal to the threshold (1.0E9). - Timestamp: Friday 10 January, 2020 17:02:52 UTC - AWS Account: 651942969000 Threshold: - The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0E9 for 60 seconds. Monitored Metric: - MetricNamespace: AWS/EC2 - MetricName: NetworkOut - Dimensions: [InstanceId = i-097ad12e6af679146] - Period: 60 seconds - Statistic: Average - Unit: not specified State Change Actions: - OK: - ALARM: [arn:aws-us-gov:sns:us-gov-west-1:651942969000:KubeCon2019] - INSUFFICIENT_DATA: -- If you wish to stop receiving notifications from this topic, please click or visit the link below to unsubscribe: https://sns.us-gov-west-1.amazonaws.com/unsubscribe.html?SubscriptionArn=arn:aws-us-gov:sns:us-gov-west-1:651942969000:KubeCon2019:d207609e-1390-4fbc-b654-3c71d1dcba19&Endpoint=kodonnel at redhat.com Please do not reply directly to this email. If you have any questions or comments regarding this email, please contact us at https://aws.amazon.com/govcloud-us/support -------------- next part -------------- An HTML attachment was scrubbed... URL: From austen.bryan.1 at us.af.mil Fri Jan 10 19:42:52 2020 From: austen.bryan.1 at us.af.mil (BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP) Date: Fri, 10 Jan 2020 19:42:52 +0000 Subject: [Platformone] FW: [Non-DoD Source] Re: [External] Re: 3 merge requests waiting In-Reply-To: References: <040FE598-7C88-4B8C-8EE7-2D4F1B094255@accenturefederal.com> <837D9911-0DF6-4ACB-81A8-36F2F9204502@accenturefederal.com> Message-ID: A non-text attachment was scrubbed... Name: smime.p7m Type: application/pkcs7-mime Size: 86966 bytes Desc: not available URL: From andrew.goss at accenturefederal.com Fri Jan 10 20:27:49 2020 From: andrew.goss at accenturefederal.com (Goss, Andrew [Semper Valens Solutions (SVS)]) Date: Fri, 10 Jan 2020 20:27:49 +0000 Subject: [Platformone] [External] Fwd: ALARM: "awsec2-i-097ad12e6af679146-High-Network-Out" in AWS GovCloud (US-West) In-Reply-To: References: <0101016f90689e74-51e3e90b-49d9-4c4b-9d81-fe6ee9f3691d-000000@us-west-2.amazonses.com> Message-ID: @Kevin O'Donnell, Your email subscription has been removed from the Topic KubeCon2019. From: Kevin O'Donnell Sent: Friday, January 10, 2020 11:05 AM To: platformONE at redhat.com; Goss, Andrew [Semper Valens Solutions (SVS)] Subject: [External] Fwd: ALARM: "awsec2-i-097ad12e6af679146-High-Network-Out" in AWS GovCloud (US-West) This message is from an EXTERNAL SENDER - be CAUTIOUS of links and attachments. THINK BEFORE YOU CLICK. ________________________________ Hello All, Can we disable these KubeCon alerts? I have gotten over a thousand emails in the last couple of days. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 [https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png] ---------- Forwarded message --------- From: KubeCon > Date: Fri, Jan 10, 2020 at 12:02 PM Subject: ALARM: "awsec2-i-097ad12e6af679146-High-Network-Out" in AWS GovCloud (US-West) To: > You are receiving this email because your Amazon CloudWatch Alarm "awsec2-i-097ad12e6af679146-High-Network-Out" in the AWS GovCloud (US-West) region has entered the ALARM state, because "Threshold Crossed: 1 datapoint [2.077810916E9 (10/01/20 17:01:00)] was greater than or equal to the threshold (1.0E9)." at "Friday 10 January, 2020 17:02:52 UTC". View this alarm in the AWS Management Console: https://us-gov-west-1.console.amazonaws-us-gov.com/cloudwatch/home?region=us-gov-west-1#s=Alarms&alarm=awsec2-i-097ad12e6af679146-High-Network-Out Alarm Details: - Name: awsec2-i-097ad12e6af679146-High-Network-Out - Description: Created from EC2 Console - State Change: OK -> ALARM - Reason for State Change: Threshold Crossed: 1 datapoint [2.077810916E9 (10/01/20 17:01:00)] was greater than or equal to the threshold (1.0E9). - Timestamp: Friday 10 January, 2020 17:02:52 UTC - AWS Account: 651942969000 Threshold: - The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0E9 for 60 seconds. Monitored Metric: - MetricNamespace: AWS/EC2 - MetricName: NetworkOut - Dimensions: [InstanceId = i-097ad12e6af679146] - Period: 60 seconds - Statistic: Average - Unit: not specified State Change Actions: - OK: - ALARM: [arn:aws-us-gov:sns:us-gov-west-1:651942969000:KubeCon2019] - INSUFFICIENT_DATA: -- If you wish to stop receiving notifications from this topic, please click or visit the link below to unsubscribe: https://sns.us-gov-west-1.amazonaws.com/unsubscribe.html?SubscriptionArn=arn:aws-us-gov:sns:us-gov-west-1:651942969000:KubeCon2019:d207609e-1390-4fbc-b654-3c71d1dcba19&Endpoint=kodonnel at redhat.com Please do not reply directly to this email. If you have any questions or comments regarding this email, please contact us at https://aws.amazon.com/govcloud-us/support -------------- next part -------------- An HTML attachment was scrubbed... URL: From kodonnel at redhat.com Tue Jan 14 19:04:15 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Tue, 14 Jan 2020 12:04:15 -0700 Subject: [Platformone] VPC to simulate C1DL Sat build Message-ID: Hello Adrian, Can you please provision a dedicated VPC for the master satellite system. We will leverage this VPC and the instance in it to fully prep for the move into C1DL. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmckee at redhat.com Tue Jan 14 22:12:44 2020 From: cmckee at redhat.com (Cory McKee) Date: Tue, 14 Jan 2020 17:12:44 -0500 Subject: [Platformone] VPC to simulate C1DL Sat build In-Reply-To: References: Message-ID: What is the status on this ? On Tue, Jan 14, 2020 at 2:13 PM Kevin O'Donnell wrote: > Hello Adrian, > > Can you please provision a dedicated VPC for the master satellite system. > We will leverage this VPC and the instance in it to fully prep for the move > into C1DL. > > Thanks, > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com M: > 240-605-4654 > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > -- Cory McKee Senior Consultant Red Hat Public Sector 8260 Greensboro Drive McLean, VA 22102 cmckee at redhat.com M: 5714090193 TRIED. TESTED. TRUSTED. @redhatjobs redhatjobs @redhatjobs -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmiller at mitre.org Wed Jan 15 21:19:45 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Wed, 15 Jan 2020 21:19:45 +0000 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) Message-ID: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> I'm resurrecting this thread. I went back and inspected the IaC. Per Kevin in the thread below, changes *were* made to the ec2-instance role. This role now provisions ec2-user with a named key registered in AWS (recall that AWS only manages public keys, but this is all that's needed. The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. I inspected all the current repositories for CI/CD definitions, extracted all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA keys and inspected the fingerprints for all keys registered in the CYBERCOM AWS account. TL;DR Findings: (Data in the raw results, below, and in the attached.) - The ec2-user SSH private key is shared across DSOP dev, staging, and production, *and* is shared with up-prod and up-dev. - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD variables to anyone with Maintainer or higher access. - The GitLab holding these keys is publicly accessible. - There are several additional RSA private keys present in the CI/CD variables, but their use is unclear. Risks: - All keys contained in a GitLab CI/CD variable should be considered compromised. There is no attribution for these keys. - Any user who can escalate privileges in GitLab via a bug, error, or social engineering can exfiltrate any of the private keys contained in CI/CD environment variables. Recommended mitigations: - Rotate all keys in raw results in all current platform1 instances. - Remove all exposed RSA keys from GitLab CI/CD configurations. - Move all deployment git repositories and CI/CD pipelines to a private GitLab instance. - Move all keys to a secrets manager, either under AWS control or deployed as a new platform1 capability. Outstanding issues: - I don't know how the gitlab-runner is being provisioned with the named key. If anyone can walk me through the runner architecture, I'd appreciate it. - I need to track down any use/reuse of the additional identified RSA private keys. The simple expedient is to remove them from GitLab CI/CD variables and see what breaks, but YMMV. Raw results: Repository DEFAULT_KEY_NAME Domains serviced ---------- ---------------- ---------------- up-node-infrastructure up-ss-ansible unified-platform.io up-appdev-a up-ss-ansible up-appdev-a.io platform-up-infrastructure-up-node n/a platform-infrastructure up-ss-ansible levelup-staging.io levelup-dev.io dsop.io platform-apps n/a Key usages ---------- 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 up-ss-ansible platform-apps/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d up-node-infrastructure/SSH_PRIVATE_KEY_FILE platform-apps/SSH_PRIVATE_KEY platform-infrastructure-up-node/CI_RUNNER_KEY platform-infrastructure/CI_RUNNER_KEY up-appdev-a/CI_RUNNER_KEY up-node-infrastructure/CI_RUNNER_KEY 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d platform-apps/GITLAB_PRIVATE_KEY e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 up-prod up-node-infrastructure/BASTION_KEY 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d up-node-infrastructure/SSO_CERT_KEY -- T ?On 1/5/20, 14:35, "Kevin O'Donnell" wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen -------------- next part -------------- A non-text attachment was scrubbed... Name: key_names Type: application/octet-stream Size: 1281 bytes Desc: key_names URL: From elijah.tramble.1 at us.af.mil Wed Jan 15 23:54:50 2020 From: elijah.tramble.1 at us.af.mil (TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC) Date: Wed, 15 Jan 2020 23:54:50 +0000 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> Message-ID: +Wayne -----Original Message----- From: Miller, Timothy J. Sent: Wednesday, January 15, 2020 3:20 PM To: Kevin O'Donnell Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) Importance: High I'm resurrecting this thread. I went back and inspected the IaC. Per Kevin in the thread below, changes *were* made to the ec2-instance role. This role now provisions ec2-user with a named key registered in AWS (recall that AWS only manages public keys, but this is all that's needed. The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. I inspected all the current repositories for CI/CD definitions, extracted all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA keys and inspected the fingerprints for all keys registered in the CYBERCOM AWS account. TL;DR Findings: (Data in the raw results, below, and in the attached.) - The ec2-user SSH private key is shared across DSOP dev, staging, and production, *and* is shared with up-prod and up-dev. - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD variables to anyone with Maintainer or higher access. - The GitLab holding these keys is publicly accessible. - There are several additional RSA private keys present in the CI/CD variables, but their use is unclear. Risks: - All keys contained in a GitLab CI/CD variable should be considered compromised. There is no attribution for these keys. - Any user who can escalate privileges in GitLab via a bug, error, or social engineering can exfiltrate any of the private keys contained in CI/CD environment variables. Recommended mitigations: - Rotate all keys in raw results in all current platform1 instances. - Remove all exposed RSA keys from GitLab CI/CD configurations. - Move all deployment git repositories and CI/CD pipelines to a private GitLab instance. - Move all keys to a secrets manager, either under AWS control or deployed as a new platform1 capability. Outstanding issues: - I don't know how the gitlab-runner is being provisioned with the named key. If anyone can walk me through the runner architecture, I'd appreciate it. - I need to track down any use/reuse of the additional identified RSA private keys. The simple expedient is to remove them from GitLab CI/CD variables and see what breaks, but YMMV. Raw results: Repository DEFAULT_KEY_NAME Domains serviced ---------- ---------------- ---------------- up-node-infrastructure up-ss-ansible unified-platform.io up-appdev-a up-ss-ansible up-appdev-a.io platform-up-infrastructure-up-node n/a platform-infrastructure up-ss-ansible levelup-staging.io levelup-dev.io dsop.io platform-apps n/a Key usages ---------- 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 up-ss-ansible platform-apps/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d up-node-infrastructure/SSH_PRIVATE_KEY_FILE platform-apps/SSH_PRIVATE_KEY platform-infrastructure-up-node/CI_RUNNER_KEY platform-infrastructure/CI_RUNNER_KEY up-appdev-a/CI_RUNNER_KEY up-node-infrastructure/CI_RUNNER_KEY 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d platform-apps/GITLAB_PRIVATE_KEY e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 up-prod up-node-infrastructure/BASTION_KEY 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d up-node-infrastructure/SSO_CERT_KEY -- T ?On 1/5/20, 14:35, "Kevin O'Donnell" wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen From kodonnel at redhat.com Thu Jan 16 04:47:41 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Wed, 15 Jan 2020 21:47:41 -0700 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> Message-ID: Hello All, I know we are putting a call together for tomorrow to discuss. In prep for that call to clarify the current keys that are used for the ec2-user post-deployment, and the key var's that are in Git (Or NOT). "The up-prod-a keys were switched out before Xmas." And are NOT stored in Git. The ec2-user does not exist and is not used in our current pipeline or code and or in our AMI's that are used to build the VPC's. The ec2-user account has been removed and the local "pipeline" named account is used to provision the VPC's and is the only account on the ec2 instances. To recap, the ec2-user no longer exists, admins are using IDM accounts with private keys that are not stored in git, and the keys for the pipeline account and or the old ec2-user account are cycled post-deployment and are not stored in Git. As we move into Cloud1 we will also be deploying a series of tooling such as Certbot to cycle keys automatically. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC wrote: > +Wayne > > -----Original Message----- > From: Miller, Timothy J. > Sent: Wednesday, January 15, 2020 3:20 PM > To: Kevin O'Donnell > Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet < > jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP < > austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC > ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim > Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q > Capt USAF AFMC AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC > AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF > AFMC AFLCMC/HNCP ; Leonard, Michael C. < > leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP < > melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; CRISP, > JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, > STEVEN E CTR USAF AFMC AFLCMC/HNCP ; > Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. > (was: Re: EXT :Re: [EXT] Platform1 SAR) > Importance: High > > I'm resurrecting this thread. > > I went back and inspected the IaC. Per Kevin in the thread below, changes > *were* made to the ec2-instance role. This role now provisions ec2-user > with a named key registered in AWS (recall that AWS only manages public > keys, but this is all that's needed. The named key is stored in the CI/CD > variables as DEFAULT_KEY_NAME. > > I inspected all the current repositories for CI/CD definitions, extracted > all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if > present, for each. Then I fingerprinted all the RSA keys and inspected the > fingerprints for all keys registered in the CYBERCOM AWS account. > > TL;DR Findings: > (Data in the raw results, below, and in the attached.) > > - The ec2-user SSH private key is shared across DSOP dev, staging, and > production, *and* is shared with up-prod and up-dev. > > - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, > and up-dev is still being shared in GitLab CI/CD variables to anyone with > Maintainer or higher access. > > - The GitLab holding these keys is publicly accessible. > > - There are several additional RSA private keys present in the CI/CD > variables, but their use is unclear. > > Risks: > > - All keys contained in a GitLab CI/CD variable should be considered > compromised. There is no attribution for these keys. > > - Any user who can escalate privileges in GitLab via a bug, error, or > social engineering can exfiltrate any of the private keys contained in > CI/CD environment variables. > > Recommended mitigations: > > - Rotate all keys in raw results in all current platform1 instances. > > - Remove all exposed RSA keys from GitLab CI/CD configurations. > > - Move all deployment git repositories and CI/CD pipelines to a private > GitLab instance. > > - Move all keys to a secrets manager, either under AWS control or deployed > as a new platform1 capability. > > Outstanding issues: > > - I don't know how the gitlab-runner is being provisioned with the named > key. If anyone can walk me through the runner architecture, I'd appreciate > it. > > - I need to track down any use/reuse of the additional identified RSA > private keys. The simple expedient is to remove them from GitLab CI/CD > variables and see what breaks, but YMMV. > > Raw results: > > Repository DEFAULT_KEY_NAME Domains > serviced > ---------- ---------------- > ---------------- > up-node-infrastructure up-ss-ansible > unified-platform.io > up-appdev-a up-ss-ansible > up-appdev-a.io > platform-up-infrastructure-up-node n/a > platform-infrastructure up-ss-ansible > levelup-staging.io > > levelup-dev.io > dsop.io > platform-apps n/a > > Key usages > ---------- > 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 > up-ss-ansible > platform-apps/ANSIBLE_PRIVATE_KEY_FILE > platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE > platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE > up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE > up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE > > d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d > up-node-infrastructure/SSH_PRIVATE_KEY_FILE > platform-apps/SSH_PRIVATE_KEY > platform-infrastructure-up-node/CI_RUNNER_KEY > platform-infrastructure/CI_RUNNER_KEY > up-appdev-a/CI_RUNNER_KEY > up-node-infrastructure/CI_RUNNER_KEY > > 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d > platform-apps/GITLAB_PRIVATE_KEY > > e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 > up-prod > up-node-infrastructure/BASTION_KEY > > 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d > up-node-infrastructure/SSO_CERT_KEY > > -- T > > ?On 1/5/20, 14:35, "Kevin O'Donnell" wrote: > > Hello Tim, > > > Welcome back. Happy New Year. > > > We have a plan in place to swap out the ec2-user for another user > seeing as the major issue is the actual name of the user not its privs in > prod. > > > Incorrect up-prod-a keys were switched out before Xmas. This process > will need to be included in the Day Two OPS onboarding process. Jet, who > should be the holder of this key on day two? We are also ensuring that > anyone that authenticates to the nodes > in the vpc use dedicated IDM accounts with per user dedicated keys > and sudo privs. > > > Correct. Re-enforcing the Day Two requirements. And also supporting > that the pipelines and CI process can support CI/CD to only dev/staging. > For prod or any environment were the keys have been swapped and are not > included as private Git variables we can > not support CD into prod/appdev if so deemed. > > > We are fast approaching policy with this topic. We should have a > further discussion this week. I will be in San Antonio on Wed and Thur. > > > Do you have any additional mitigation techniques Tim? > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: > > > Some more points of interest-- > > - The ec2-user account can't currently be avoided given the current > design of the IaC. > > - This SSH key provisioned for this account is shared across > up-prod-a, up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with Maintainer or > higher access to the GitLab repo that holds the customer's environment > definitions or the platform-infrastructure repo. > > -- T > > On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: > > I just want to clarify on the topic of the EC2-user. The key is > attribution. We need to know who or what is configuring the systems. > > We are repeatedly seeing the ec2-user account used to manually > configure systems. Ec2-user should only be used to perform programmatic > tasks from committed and > reviewed code as part of initial provisioning. If the code to > provision and configure systems is unavailable, then you must create > attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user will be flagged > as a violation and the source IP or user account logged as the violating > source. Any manual/command > line activities MUST be performed using an attributable user > account. > > If that is not clear, please let me know and I will attempt to > elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) ; > BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; > platformONE at redhat.com; Tim Gast ; > Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC > AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC > AFLCMC/HNCP ; Blade, Eric > D [US] (MS) ; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < > jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; > REINHARDT, MELISSA > A GG-13 USAF AFMC AFLCMC/HNCP ; > Taylor Biggs ; Miller, Timothy J. ; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < > joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil>; > Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We will get GIT > issues crated for your findings and will prioritize the mitigation and > implement them as code in our future IAC deployments. Many of the findings > in the current VPC have been > mitigated in up-prod-b with our current code release. > > > > Please let us know when you have finished and we can power down > the host that you have been using for scanning. > > > > > Note for everyone: Once we power down the ec2 instance ssh or port > 22 will not be externally accessible. Thus, mitigating many of the risks > associated with the ec2-user and the keys. > > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet < > jlastrilla at mitre.org> wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If you want to take > more time, please use tomorrow to do your testing. > > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt > USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E > GG-13 USAF AFMC ESC/AFLCMC/HNCP ; > Kevin O'Donnell ; > platformONE at redhat.com ; Tim Gast < > tg at braingu.com>; Bubb, > Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC < > elijah.tramble.1 at us.af.mil>; > tj.zimmerman at braingu.com ; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP < > richard.lopezdeuralde at us.af.mil>; > Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A > CTR USAF AFMC AFLCMC/HNCP ; Leonard, > Michael > C. ; REINHARDT, MELISSA A GG-13 USAF AFMC > AFLCMC/HNCP ; Taylor Biggs < > taylor at redhat.com>; > Miller, Timothy J. ; CRISP, JOSHUA M GS-09 > USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF > AFMC > AFLCMC/HNCP ; Wilcox, John R. (San > Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security testing will be sent > through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and feel like I > have not done as thorough of a job as required. I was unable to log into > the Web UIs, as no one from the Platform1 team gave me the account > information. I had issues with Nessus, so the CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please read through it, > because there is some new information. I had to test as ec2-user again, > which is another big issue that needs to be resolved ASAP. The more I use > it and find out > how it is being used, the more extremely concerned I am. It has > multiple keys throughout the platform located in the .ssh directory, one of > which is world readable. On some hosts, a real user is using the ec2-user > account to create accounts, groups, and > pull > docker files. The account is non-attributable, so we have no way > of knowing who is doing this. Someone could do serious damage with no > consequence. I understand that the ec2-user is needed for standing up an > ec2-image, but > this account should only be used for implementing IAC, so that the > changes implemented by ec2-user are codified. If manual admin is required, > that IAC should provision the appropriate attributable accounts, and those > accounts should > be used from then on. In my opinion, this is a critical finding > and needs to be addressed ASAP. > > > I will be available during the day tomorrow for any questions. > > Thanks > Colleen > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kodonnel at redhat.com Thu Jan 16 21:14:52 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Thu, 16 Jan 2020 14:14:52 -0700 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> Message-ID: All, Attached is the output from the authorized_keys files on all nodes in up-prod. Please let me know if you need any additional information. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell wrote: > Hello All, > > I know we are putting a call together for tomorrow to discuss. In prep for > that call to clarify the current keys that are used for the ec2-user > post-deployment, and the key var's that are in Git (Or NOT). > > "The up-prod-a keys were switched out before Xmas." And are NOT stored in > Git. > > The ec2-user does not exist and is not used in our current pipeline or > code and or in our AMI's that are used to build the VPC's. The ec2-user > account has been removed and the local "pipeline" named account is used to > provision the VPC's and is the only account on the ec2 instances. > > To recap, the ec2-user no longer exists, admins are using IDM accounts > with private keys that are not stored in git, and the keys for the pipeline > account and or the old ec2-user account are cycled post-deployment and are > not stored in Git. > > As we move into Cloud1 we will also be deploying a series of tooling such > as Certbot to cycle keys automatically. > > Thanks, > > KEVIN O'DONNELL > > ARCHITECT MANAGER > > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com M: > 240-605-4654 > > > > On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC > AFLCMC/HNC wrote: > >> +Wayne >> >> -----Original Message----- >> From: Miller, Timothy J. >> Sent: Wednesday, January 15, 2020 3:20 PM >> To: Kevin O'Donnell >> Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet < >> jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS) < >> Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP < >> austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC >> ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim >> Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q >> Capt USAF AFMC AFLCMC/HNC ; >> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC >> AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF >> AFMC AFLCMC/HNCP ; Leonard, Michael C. < >> leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP < >> melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; CRISP, >> JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, >> STEVEN E CTR USAF AFMC AFLCMC/HNCP ; >> Wilcox, John R. (San Antonio, TX) [US] (MS) >> Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. >> (was: Re: EXT :Re: [EXT] Platform1 SAR) >> Importance: High >> >> I'm resurrecting this thread. >> >> I went back and inspected the IaC. Per Kevin in the thread below, >> changes *were* made to the ec2-instance role. This role now provisions >> ec2-user with a named key registered in AWS (recall that AWS only manages >> public keys, but this is all that's needed. The named key is stored in the >> CI/CD variables as DEFAULT_KEY_NAME. >> >> I inspected all the current repositories for CI/CD definitions, extracted >> all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if >> present, for each. Then I fingerprinted all the RSA keys and inspected the >> fingerprints for all keys registered in the CYBERCOM AWS account. >> >> TL;DR Findings: >> (Data in the raw results, below, and in the attached.) >> >> - The ec2-user SSH private key is shared across DSOP dev, staging, and >> production, *and* is shared with up-prod and up-dev. >> >> - The ec2-user SSH private key for DSOP production, staging, dev, >> up-prod, and up-dev is still being shared in GitLab CI/CD variables to >> anyone with Maintainer or higher access. >> >> - The GitLab holding these keys is publicly accessible. >> >> - There are several additional RSA private keys present in the CI/CD >> variables, but their use is unclear. >> >> Risks: >> >> - All keys contained in a GitLab CI/CD variable should be considered >> compromised. There is no attribution for these keys. >> >> - Any user who can escalate privileges in GitLab via a bug, error, or >> social engineering can exfiltrate any of the private keys contained in >> CI/CD environment variables. >> >> Recommended mitigations: >> >> - Rotate all keys in raw results in all current platform1 instances. >> >> - Remove all exposed RSA keys from GitLab CI/CD configurations. >> >> - Move all deployment git repositories and CI/CD pipelines to a private >> GitLab instance. >> >> - Move all keys to a secrets manager, either under AWS control or >> deployed as a new platform1 capability. >> >> Outstanding issues: >> >> - I don't know how the gitlab-runner is being provisioned with the named >> key. If anyone can walk me through the runner architecture, I'd appreciate >> it. >> >> - I need to track down any use/reuse of the additional identified RSA >> private keys. The simple expedient is to remove them from GitLab CI/CD >> variables and see what breaks, but YMMV. >> >> Raw results: >> >> Repository DEFAULT_KEY_NAME Domains >> serviced >> ---------- ---------------- >> ---------------- >> up-node-infrastructure up-ss-ansible >> unified-platform.io >> up-appdev-a up-ss-ansible >> up-appdev-a.io >> platform-up-infrastructure-up-node n/a >> platform-infrastructure up-ss-ansible >> levelup-staging.io >> >> levelup-dev.io >> dsop.io >> platform-apps n/a >> >> Key usages >> ---------- >> 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 >> up-ss-ansible >> platform-apps/ANSIBLE_PRIVATE_KEY_FILE >> platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE >> platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE >> up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE >> up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE >> >> d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d >> up-node-infrastructure/SSH_PRIVATE_KEY_FILE >> platform-apps/SSH_PRIVATE_KEY >> platform-infrastructure-up-node/CI_RUNNER_KEY >> platform-infrastructure/CI_RUNNER_KEY >> up-appdev-a/CI_RUNNER_KEY >> up-node-infrastructure/CI_RUNNER_KEY >> >> 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d >> platform-apps/GITLAB_PRIVATE_KEY >> >> e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 >> up-prod >> up-node-infrastructure/BASTION_KEY >> >> 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d >> up-node-infrastructure/SSO_CERT_KEY >> >> -- T >> >> ?On 1/5/20, 14:35, "Kevin O'Donnell" wrote: >> >> Hello Tim, >> >> >> Welcome back. Happy New Year. >> >> >> We have a plan in place to swap out the ec2-user for another user >> seeing as the major issue is the actual name of the user not its privs in >> prod. >> >> >> Incorrect up-prod-a keys were switched out before Xmas. This process >> will need to be included in the Day Two OPS onboarding process. Jet, who >> should be the holder of this key on day two? We are also ensuring that >> anyone that authenticates to the nodes >> in the vpc use dedicated IDM accounts with per user dedicated keys >> and sudo privs. >> >> >> Correct. Re-enforcing the Day Two requirements. And also supporting >> that the pipelines and CI process can support CI/CD to only dev/staging. >> For prod or any environment were the keys have been swapped and are not >> included as private Git variables we can >> not support CD into prod/appdev if so deemed. >> >> >> We are fast approaching policy with this topic. We should have a >> further discussion this week. I will be in San Antonio on Wed and Thur. >> >> >> Do you have any additional mitigation techniques Tim? >> >> >> Thanks, >> >> KEVIN O'DONNELL >> ARCHITECT MANAGER >> Red Hat Red Hat NA Public Sector Consulting >> >> kodonnell at redhat.com >> M: 240-605-4654 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. >> wrote: >> >> >> Some more points of interest-- >> >> - The ec2-user account can't currently be avoided given the current >> design of the IaC. >> >> - This SSH key provisioned for this account is shared across >> up-prod-a, up-dev-a, and DSOP. >> >> - The ec2-user SSH privkey is available to anyone with Maintainer or >> higher access to the GitLab repo that holds the customer's environment >> definitions or the platform-infrastructure repo. >> >> -- T >> >> On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" >> wrote: >> >> I just want to clarify on the topic of the EC2-user. The key is >> attribution. We need to know who or what is configuring the systems. >> >> We are repeatedly seeing the ec2-user account used to manually >> configure systems. Ec2-user should only be used to perform programmatic >> tasks from committed and >> reviewed code as part of initial provisioning. If the code to >> provision and configure systems is unavailable, then you must create >> attributable accounts to perform changes to the systems. >> >> Any use of ec2-user remote login or su to ec2-user will be >> flagged as a violation and the source IP or user account logged as the >> violating source. Any manual/command >> line activities MUST be performed using an attributable user >> account. >> >> If that is not clear, please let me know and I will attempt to >> elaborate further. >> >> Thank you >> >> >> Eric Blade, GXPN, GPEN >> Unified Platform System Coordinator >> Northrop Grumman Mission Systems >> Work: 410.649.0706 >> Mobile: 240.258.8089 >> >> >> >> From: Kevin O'Donnell >> >> Sent: Thursday, December 19, 2019 6:22 PM >> To: Lastrilla, Jet >> Cc: Feiglstok, Colleen M [US] (MS) ; >> BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; >> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < >> roger.dirocco.4 at us.af.mil>; >> platformONE at redhat.com; Tim Gast ; >> Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC >> AFLCMC/HNC ; >> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC >> AFLCMC/HNCP ; Blade, Eric >> D [US] (MS) ; >> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < >> jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; >> REINHARDT, MELISSA >> A GG-13 USAF AFMC AFLCMC/HNCP ; >> Taylor Biggs ; Miller, Timothy J. ; >> CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < >> joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < >> steven.bogue.1.ctr at us.af.mil>; >> Wilcox, John R. (San Antonio, TX) [US] (MS) >> Subject: EXT :Re: [EXT] Platform1 SAR >> >> Colleen, >> >> >> Thank you for the results and recommendations. We will get GIT >> issues crated for your findings and will prioritize the mitigation and >> implement them as code in our future IAC deployments. Many of the findings >> in the current VPC have been >> mitigated in up-prod-b with our current code release. >> >> >> >> Please let us know when you have finished and we can power down >> the host that you have been using for scanning. >> >> >> >> >> Note for everyone: Once we power down the ec2 instance ssh or >> port 22 will not be externally accessible. Thus, mitigating many of the >> risks associated with the ec2-user and the keys. >> >> >> >> >> Thanks, >> >> >> >> KEVIN O'DONNELL >> ARCHITECT MANAGER >> Red Hat Red Hat NA Public Sector Consulting < >> https://www.redhat.com/> >> kodonnell at redhat.com >> M: 240-605-4654 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet < >> jlastrilla at mitre.org> wrote: >> >> >> Thanks Colleen. Sorry for the rushed feeling. If you want to take >> more time, please use tomorrow to do your testing. >> >> >> >> Thank you for all you do!!!! >> >> Get Outlook for iOS >> >> >> >> ________________________________________ >> >> From: Feiglstok, Colleen M [US] (MS) >> Sent: Thursday, December 19, 2019 4:35:15 PM >> To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt >> USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E >> GG-13 USAF AFMC ESC/AFLCMC/HNCP ; >> Kevin O'Donnell ; >> platformONE at redhat.com ; Tim Gast < >> tg at braingu.com>; Bubb, >> Mike >> ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC < >> elijah.tramble.1 at us.af.mil>; >> tj.zimmerman at braingu.com ; >> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP < >> richard.lopezdeuralde at us.af.mil>; >> Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A >> CTR USAF AFMC AFLCMC/HNCP ; Leonard, >> Michael >> C. ; REINHARDT, MELISSA A GG-13 USAF AFMC >> AFLCMC/HNCP ; Taylor Biggs < >> taylor at redhat.com>; >> Miller, Timothy J. ; CRISP, JOSHUA M GS-09 >> USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR >> USAF >> AFMC >> AFLCMC/HNCP ; Wilcox, John R. >> (San Antonio, TX) [US] (MS) >> Subject: [EXT] Platform1 SAR >> >> >> >> All, >> >> The SAR and raw results from the new security testing will be >> sent through NGSafe in a few moments. >> >> As usual, I felt very rushed with the testing, and feel like I >> have not done as thorough of a job as required. I was unable to log into >> the Web UIs, as no one from the Platform1 team gave me the account >> information. I had issues with Nessus, so the CVE?s >> were found through OSCAP this time. >> >> A lot is the same as the last report, but please read through it, >> because there is some new information. I had to test as ec2-user again, >> which is another big issue that needs to be resolved ASAP. The more I use >> it and find out >> how it is being used, the more extremely concerned I am. It has >> multiple keys throughout the platform located in the .ssh directory, one of >> which is world readable. On some hosts, a real user is using the ec2-user >> account to create accounts, groups, and >> pull >> docker files. The account is non-attributable, so we have no way >> of knowing who is doing this. Someone could do serious damage with no >> consequence. I understand that the ec2-user is needed for standing up an >> ec2-image, but >> this account should only be used for implementing IAC, so that >> the changes implemented by ec2-user are codified. If manual admin is >> required, that IAC should provision the appropriate attributable accounts, >> and those accounts should >> be used from then on. In my opinion, this is a critical finding >> and needs to be addressed ASAP. >> >> >> I will be available during the day tomorrow for any questions. >> >> Thanks >> Colleen >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- ok: [10.40.14.58] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.39.225] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDSVNmbVd4dWZMNzFCOTRvSElzTkt3ZVVRZkZrZlJIaStSNVRlcW9OQzdKYlYweVd6NGViUFpzTFVqbk1FeEwvNlVEYzROT1VrZGpCS1F5NVhPNUNJZnh3NERDblVQd2duNzdrTkVCcHNRSk42UDBPWGV2aU9tblBPcXkrcWlHVVo0WTVtenF3ZVBCcGFXN0NMaEt1eEk4Y1lucGk3N09VL0xpb1NOV093aFg0VnRyYmo3QTM3aXV3N2ZDTTNsYStlZlVuMHFXN1VzNUxwZDZpTXlTdlNFWWtyUHBGLzc2SjQ0YTIwMlovYiszZGFxVUJCRm96U2Y2TkIwWjR6QS9HUWFGb1ZZeDNPbnJGdXhKaFBEa0NXUkw2TFN5K1lkeTNpbnA2RlhqNWU1QkJFUVZkMDVRdjYzTlRCck1waHJFNDNjNVZjSmt0K2dNQXhjVzJqWlFiUjEgdXAtc3MtYW5zaWJsZQo=" } ok: [10.40.39.128] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.37.29] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.41.176] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.34.15] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.40.164] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.33.86] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.2.65] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.36.216] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.44.191] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.42.202] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.35.166] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.37.82] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" } ok: [10.40.45.233] => { "msg": "c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDZDNheU1OcmYvckJiVlFiT1lnNUEvY2pHcC8wNVlYcHlTaUNRMTREVjk0SStxVmNlL2tFRkM5RW8rcTNGclZLQUIzeERtRElyL0JDMUNldW1pMFJwcitGKzc0Z3dnZlM1OG9lSERXaWxHZXRxN3l6ZEtQNHpKL3pRYmtMa2kza3hlTGtQQ2lnUHFjNW5YaW5wRGxDbXFWRUVtK3lHaVk4NmxXamU2Tzh2azB2UkJsbVRWdGZqNStJNjFWbkR2S3BRbUxOc3RFbDIwaHZ5SWt5amV4RkUrRFQ4VG84dDhqMzU3aG5rK2JqWkxQVmY1ZSszczlwd001L0Y4SUIybU5NZ25YK0puS3RqQXl4aElWOHdRSnpwejhhb3R0L2xuK2xwVDd0MEFQdWRRVEhkVkZUVDlDdTIzQVQ1UnMyMmVBVEpFZGx6Nm0wc0cxTFRuemRLNWRwNjUgdXAtcHJvZAo=" From tmiller at mitre.org Fri Jan 17 13:23:24 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Fri, 17 Jan 2020 13:23:24 +0000 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> Message-ID: <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> Kevin-- Can you share the ansible play / ad-hoc command used to collect this? Thanks. -- T ?On 1/16/20, 15:16, "Kevin O'Donnell" wrote: All, Attached is the output from the authorized_keys files on all nodes in up-prod. Please let me know if you need any additional information. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell wrote: Hello All, I know we are putting a call together for tomorrow to discuss. In prep for that call to clarify the current keys that are used for the ec2-user post-deployment, and the key var's that are in Git (Or NOT). "The up-prod-a keys were switched out before Xmas." And are NOT stored in Git. The ec2-user does not exist and is not used in our current pipeline or code and or in our AMI's that are used to build the VPC's. The ec2-user account has been removed and the local "pipeline" named account is used to provision the VPC's and is the only account on the ec2 instances. To recap, the ec2-user no longer exists, admins are using IDM accounts with private keys that are not stored in git, and the keys for the pipeline account and or the old ec2-user account are cycled post-deployment and are not stored in Git. As we move into Cloud1 we will also be deploying a series of tooling such as Certbot to cycle keys automatically. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC wrote: +Wayne -----Original Message----- From: Miller, Timothy J. Sent: Wednesday, January 15, 2020 3:20 PM To: Kevin O'Donnell Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) Importance: High I'm resurrecting this thread. I went back and inspected the IaC. Per Kevin in the thread below, changes *were* made to the ec2-instance role. This role now provisions ec2-user with a named key registered in AWS (recall that AWS only manages public keys, but this is all that's needed. The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. I inspected all the current repositories for CI/CD definitions, extracted all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA keys and inspected the fingerprints for all keys registered in the CYBERCOM AWS account. TL;DR Findings: (Data in the raw results, below, and in the attached.) - The ec2-user SSH private key is shared across DSOP dev, staging, and production, *and* is shared with up-prod and up-dev. - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD variables to anyone with Maintainer or higher access. - The GitLab holding these keys is publicly accessible. - There are several additional RSA private keys present in the CI/CD variables, but their use is unclear. Risks: - All keys contained in a GitLab CI/CD variable should be considered compromised. There is no attribution for these keys. - Any user who can escalate privileges in GitLab via a bug, error, or social engineering can exfiltrate any of the private keys contained in CI/CD environment variables. Recommended mitigations: - Rotate all keys in raw results in all current platform1 instances. - Remove all exposed RSA keys from GitLab CI/CD configurations. - Move all deployment git repositories and CI/CD pipelines to a private GitLab instance. - Move all keys to a secrets manager, either under AWS control or deployed as a new platform1 capability. Outstanding issues: - I don't know how the gitlab-runner is being provisioned with the named key. If anyone can walk me through the runner architecture, I'd appreciate it. - I need to track down any use/reuse of the additional identified RSA private keys. The simple expedient is to remove them from GitLab CI/CD variables and see what breaks, but YMMV. Raw results: Repository DEFAULT_KEY_NAME Domains serviced ---------- ---------------- ---------------- up-node-infrastructure up-ss-ansible unified-platform.io up-appdev-a up-ss-ansible up-appdev-a.io platform-up-infrastructure-up-node n/a platform-infrastructure up-ss-ansible levelup-staging.io levelup-dev.io dsop.io platform-apps n/a Key usages ---------- 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 up-ss-ansible platform-apps/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d up-node-infrastructure/SSH_PRIVATE_KEY_FILE platform-apps/SSH_PRIVATE_KEY platform-infrastructure-up-node/CI_RUNNER_KEY platform-infrastructure/CI_RUNNER_KEY up-appdev-a/CI_RUNNER_KEY up-node-infrastructure/CI_RUNNER_KEY 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d platform-apps/GITLAB_PRIVATE_KEY e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 up-prod up-node-infrastructure/BASTION_KEY 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d up-node-infrastructure/SSO_CERT_KEY -- T On 1/5/20, 14:35, "Kevin O'Donnell" wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen From mholmes at redhat.com Fri Jan 17 16:25:23 2020 From: mholmes at redhat.com (Michael Holmes) Date: Fri, 17 Jan 2020 10:25:23 -0600 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> Message-ID: Hi Tim, here is the ansible play I used. --- - hosts: dev-prod become: true tasks: - name: fetch file slurp: src: /home/ec2-user/.ssh/authorized_keys register: keys - debug: msg: "{{ keys['content'] }}" On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. wrote: > Kevin-- > > Can you share the ansible play / ad-hoc command used to collect this? > > Thanks. > > -- T > > ?On 1/16/20, 15:16, "Kevin O'Donnell" wrote: > > All, > > > Attached is the output from the authorized_keys files on all nodes in > up-prod. Please let me know if you need any additional information. > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell > wrote: > > > Hello All, > > > I know we are putting a call together for tomorrow to discuss. In prep > for that call to clarify the current keys that are used for the ec2-user > post-deployment, and the key var's that are in Git (Or NOT). > > > "The up-prod-a keys were switched out before Xmas." And are NOT stored > in Git. > > > The ec2-user does not exist and is not used in our current pipeline or > code and or in our AMI's that are used to build the VPC's. The ec2-user > account has been removed and the local "pipeline" named account is used to > provision the VPC's and is the only > account on the ec2 instances. > > > To recap, the ec2-user no longer exists, admins are using IDM accounts > with private keys that are not stored in git, and the keys for the pipeline > account and or the old ec2-user account are cycled post-deployment and are > not stored in Git. > > > As we move into Cloud1 we will also be deploying a series of tooling > such as Certbot to cycle keys automatically. > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC > AFLCMC/HNC wrote: > > > +Wayne > > -----Original Message----- > From: Miller, Timothy J. > Sent: Wednesday, January 15, 2020 3:20 PM > To: Kevin O'Donnell > Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet < > jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com>; > BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; > platformONE at redhat.com; Tim Gast ; Bubb, Mike < > mbubb at mitre.org>; TRAMBLE, ELIJAH > Q Capt USAF AFMC AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC > AFLCMC/HNCP ; RAMIREZ, JOSE > A CTR USAF AFMC AFLCMC/HNCP ; > Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF > AFMC AFLCMC/HNCP ; Taylor Biggs < > taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < > joshua.crisp.2 at us.af.mil>; > BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San Antonio, TX) [US] > (MS) > Subject: [Non-DoD Source] Key sharing across multiple platform1 > builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) > Importance: High > > I'm resurrecting this thread. > > I went back and inspected the IaC. Per Kevin in the thread below, > changes *were* made to the ec2-instance role. This role now provisions > ec2-user with a named key registered in AWS (recall that AWS only manages > public keys, but this is all that's needed. > The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. > > I inspected all the current repositories for CI/CD definitions, > extracted all the RSA private keys I could find, noted the > DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA > keys and inspected the fingerprints for all keys registered > in the CYBERCOM AWS account. > > TL;DR Findings: > (Data in the raw results, below, and in the attached.) > > - The ec2-user SSH private key is shared across DSOP dev, staging, and > production, *and* is shared with up-prod and up-dev. > > - The ec2-user SSH private key for DSOP production, staging, dev, > up-prod, and up-dev is still being shared in GitLab CI/CD variables to > anyone with Maintainer or higher access. > > - The GitLab holding these keys is publicly accessible. > > - There are several additional RSA private keys present in the CI/CD > variables, but their use is unclear. > > Risks: > > - All keys contained in a GitLab CI/CD variable should be considered > compromised. There is no attribution for these keys. > > - Any user who can escalate privileges in GitLab via a bug, error, or > social engineering can exfiltrate any of the private keys contained in > CI/CD environment variables. > > Recommended mitigations: > > - Rotate all keys in raw results in all current platform1 instances. > > - Remove all exposed RSA keys from GitLab CI/CD configurations. > > - Move all deployment git repositories and CI/CD pipelines to a > private GitLab instance. > > - Move all keys to a secrets manager, either under AWS control or > deployed as a new platform1 capability. > > Outstanding issues: > > - I don't know how the gitlab-runner is being provisioned with the > named key. If anyone can walk me through the runner architecture, I'd > appreciate it. > > - I need to track down any use/reuse of the additional identified RSA > private keys. The simple expedient is to remove them from GitLab CI/CD > variables and see what breaks, but YMMV. > > Raw results: > > Repository DEFAULT_KEY_NAME > Domains serviced > ---------- ---------------- > ---------------- > up-node-infrastructure up-ss-ansible > unified-platform.io > up-appdev-a up-ss-ansible > up-appdev-a.io > platform-up-infrastructure-up-node n/a > platform-infrastructure up-ss-ansible > levelup-staging.io > > levelup-dev.io > > dsop.io > platform-apps n/a > > Key usages > ---------- > 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 > up-ss-ansible > platform-apps/ANSIBLE_PRIVATE_KEY_FILE > platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE > platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE > up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE > up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE > > d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d > up-node-infrastructure/SSH_PRIVATE_KEY_FILE > platform-apps/SSH_PRIVATE_KEY > platform-infrastructure-up-node/CI_RUNNER_KEY > platform-infrastructure/CI_RUNNER_KEY > up-appdev-a/CI_RUNNER_KEY > up-node-infrastructure/CI_RUNNER_KEY > > 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d > platform-apps/GITLAB_PRIVATE_KEY > > e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 > up-prod > up-node-infrastructure/BASTION_KEY > > 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d > up-node-infrastructure/SSO_CERT_KEY > > -- T > > On 1/5/20, 14:35, "Kevin O'Donnell" wrote: > > Hello Tim, > > > Welcome back. Happy New Year. > > > We have a plan in place to swap out the ec2-user for another user > seeing as the major issue is the actual name of the user not its privs in > prod. > > > Incorrect up-prod-a keys were switched out before Xmas. This > process will need to be included in the Day Two OPS onboarding process. > Jet, who should be the holder of this key on day two? We are also ensuring > that anyone that authenticates to the nodes > in the vpc use dedicated IDM accounts with per user dedicated > keys and sudo privs. > > > Correct. Re-enforcing the Day Two requirements. And also > supporting that the pipelines and CI process can support CI/CD to only > dev/staging. For prod or any environment were the keys have been swapped > and are not included as private Git variables we can > not support CD into prod/appdev if so deemed. > > > We are fast approaching policy with this topic. We should have a > further discussion this week. I will be in San Antonio on Wed and Thur. > > > Do you have any additional mitigation techniques Tim? > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. < > tmiller at mitre.org> wrote: > > > Some more points of interest-- > > - The ec2-user account can't currently be avoided given the > current design of the IaC. > > - This SSH key provisioned for this account is shared across > up-prod-a, up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with Maintainer > or higher access to the GitLab repo that holds the customer's environment > definitions or the platform-infrastructure repo. > > -- T > > On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: > > I just want to clarify on the topic of the EC2-user. The key > is attribution. We need to know who or what is configuring the systems. > > We are repeatedly seeing the ec2-user account used to manually > configure systems. Ec2-user should only be used to perform programmatic > tasks from committed and > reviewed code as part of initial provisioning. If the code > to provision and configure systems is unavailable, then you must create > attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user will be > flagged as a violation and the source IP or user account logged as the > violating source. Any manual/command > line activities MUST be performed using an attributable user > account. > > If that is not clear, please let me know and I will attempt to > elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) ; > BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; > platformONE at redhat.com; Tim Gast ; > Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF > AFMC AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF > AFMC AFLCMC/HNCP ; Blade, > Eric > D [US] (MS) ; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < > jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; > REINHARDT, > MELISSA > A GG-13 USAF AFMC AFLCMC/HNCP ; > Taylor Biggs ; Miller, Timothy J. ; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < > joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil>; > Wilcox, John R. (San Antonio, TX) [US] (MS) < > John.R.Wilcox at ngc.com> > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We will get GIT > issues crated for your findings and will prioritize the mitigation and > implement them as code in our future IAC deployments. Many of the findings > in the current VPC have been > mitigated in up-prod-b with our current code release. > > > > Please let us know when you have finished and we can power > down the host that you have been using for scanning. > > > > > Note for everyone: Once we power down the ec2 instance ssh or > port 22 will not be externally accessible. Thus, mitigating many of the > risks associated with the ec2-user and the keys. > > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet < > jlastrilla at mitre.org> wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If you want to > take more time, please use tomorrow to do your testing. > > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com> > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet ; BRYAN, AUSTEN R > Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER > E > GG-13 USAF AFMC ESC/AFLCMC/HNCP ; > Kevin O'Donnell ; > platformONE at redhat.com ; Tim Gast < > tg at braingu.com>; Bubb, > Mike > ; TRAMBLE, ELIJAH Q Capt USAF AFMC > AFLCMC/HNC ; > tj.zimmerman at braingu.com ; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP < > richard.lopezdeuralde at us.af.mil>; > Blade, Eric D [US] (MS) ; RAMIREZ, JOSE > A CTR USAF AFMC AFLCMC/HNCP ; Leonard, > Michael > C. ; REINHARDT, MELISSA A GG-13 USAF > AFMC AFLCMC/HNCP ; Taylor Biggs < > taylor at redhat.com>; > Miller, Timothy J. ; CRISP, JOSHUA M > GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E > CTR > USAF > AFMC > AFLCMC/HNCP ; Wilcox, John R. > (San Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security testing will be > sent through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and feel like I > have not done as thorough of a job as required. I was unable to log into > the Web UIs, as no one from the Platform1 team gave me the account > information. I had issues with Nessus, so the > CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please read through > it, because there is some new information. I had to test as ec2-user again, > which is another big issue that needs to be resolved ASAP. The more I use > it and find out > how it is being used, the more extremely concerned I am. It > has multiple keys throughout the platform located in the .ssh directory, > one of which is world readable. On some hosts, a real user is using the > ec2-user account to create accounts, groups, > and > pull > docker files. The account is non-attributable, so we have no > way of knowing who is doing this. Someone could do serious damage with no > consequence. I understand that the ec2-user is needed for standing up an > ec2-image, but > this account should only be used for implementing IAC, so that > the changes implemented by ec2-user are codified. If manual admin is > required, that IAC should provision the appropriate attributable accounts, > and those accounts should > be used from then on. In my opinion, this is a critical > finding and needs to be addressed ASAP. > > > I will be available during the day tomorrow for any questions. > > Thanks > Colleen > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > -- Michael Holmes, RHCSA Senior Consultant Red Hat Remote - Texas mholmes at redhat.com M: 808-780-4877 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmiller at mitre.org Fri Jan 17 19:45:56 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Fri, 17 Jan 2020 19:45:56 +0000 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> Message-ID: <73A0E99F-E966-4A03-A058-21EC44B8D35A@mitre.org> Can you run this instead, then tarball the resultant `authorized_keys` dir and send me the tarball? --- - hosts: localhost become: true tasks: - name: find all authorized_keys find: paths: - /home recurse: true patterns: 'authorized_keys' register: key_files - name: write match data to a file copy: content: "{{ key_files }}" dest: /tmp/match_data.json - name: fetch all matched files fetch: src: "{{ item.path }}" dest: ./authorized_keys loop: "{{ key_files.files }}" - name: fetch match data fetch: src: /tmp/match_data.json dest: ./authorized_keys -- T ?On 1/17/20, 10:26, "Michael Holmes" wrote: Hi Tim, here is the ansible play I used. --- - hosts: dev-prod become: true tasks: - name: fetch file slurp: src: /home/ec2-user/.ssh/authorized_keys register: keys - debug: msg: "{{ keys['content'] }}" On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. wrote: Kevin-- Can you share the ansible play / ad-hoc command used to collect this? Thanks. -- T On 1/16/20, 15:16, "Kevin O'Donnell" wrote: All, Attached is the output from the authorized_keys files on all nodes in up-prod. Please let me know if you need any additional information. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell wrote: Hello All, I know we are putting a call together for tomorrow to discuss. In prep for that call to clarify the current keys that are used for the ec2-user post-deployment, and the key var's that are in Git (Or NOT). "The up-prod-a keys were switched out before Xmas." And are NOT stored in Git. The ec2-user does not exist and is not used in our current pipeline or code and or in our AMI's that are used to build the VPC's. The ec2-user account has been removed and the local "pipeline" named account is used to provision the VPC's and is the only account on the ec2 instances. To recap, the ec2-user no longer exists, admins are using IDM accounts with private keys that are not stored in git, and the keys for the pipeline account and or the old ec2-user account are cycled post-deployment and are not stored in Git. As we move into Cloud1 we will also be deploying a series of tooling such as Certbot to cycle keys automatically. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC wrote: +Wayne -----Original Message----- From: Miller, Timothy J. Sent: Wednesday, January 15, 2020 3:20 PM To: Kevin O'Donnell Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) Importance: High I'm resurrecting this thread. I went back and inspected the IaC. Per Kevin in the thread below, changes *were* made to the ec2-instance role. This role now provisions ec2-user with a named key registered in AWS (recall that AWS only manages public keys, but this is all that's needed. The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. I inspected all the current repositories for CI/CD definitions, extracted all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA keys and inspected the fingerprints for all keys registered in the CYBERCOM AWS account. TL;DR Findings: (Data in the raw results, below, and in the attached.) - The ec2-user SSH private key is shared across DSOP dev, staging, and production, *and* is shared with up-prod and up-dev. - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD variables to anyone with Maintainer or higher access. - The GitLab holding these keys is publicly accessible. - There are several additional RSA private keys present in the CI/CD variables, but their use is unclear. Risks: - All keys contained in a GitLab CI/CD variable should be considered compromised. There is no attribution for these keys. - Any user who can escalate privileges in GitLab via a bug, error, or social engineering can exfiltrate any of the private keys contained in CI/CD environment variables. Recommended mitigations: - Rotate all keys in raw results in all current platform1 instances. - Remove all exposed RSA keys from GitLab CI/CD configurations. - Move all deployment git repositories and CI/CD pipelines to a private GitLab instance. - Move all keys to a secrets manager, either under AWS control or deployed as a new platform1 capability. Outstanding issues: - I don't know how the gitlab-runner is being provisioned with the named key. If anyone can walk me through the runner architecture, I'd appreciate it. - I need to track down any use/reuse of the additional identified RSA private keys. The simple expedient is to remove them from GitLab CI/CD variables and see what breaks, but YMMV. Raw results: Repository DEFAULT_KEY_NAME Domains serviced ---------- ---------------- ---------------- up-node-infrastructure up-ss-ansible unified-platform.io up-appdev-a up-ss-ansible up-appdev-a.io platform-up-infrastructure-up-node n/a platform-infrastructure up-ss-ansible levelup-staging.io levelup-dev.io dsop.io platform-apps n/a Key usages ---------- 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 up-ss-ansible platform-apps/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d up-node-infrastructure/SSH_PRIVATE_KEY_FILE platform-apps/SSH_PRIVATE_KEY platform-infrastructure-up-node/CI_RUNNER_KEY platform-infrastructure/CI_RUNNER_KEY up-appdev-a/CI_RUNNER_KEY up-node-infrastructure/CI_RUNNER_KEY 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d platform-apps/GITLAB_PRIVATE_KEY e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 up-prod up-node-infrastructure/BASTION_KEY 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d up-node-infrastructure/SSO_CERT_KEY -- T On 1/5/20, 14:35, "Kevin O'Donnell" wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen _______________________________________________ platformONE mailing list platformONE at redhat.com https://www.redhat.com/mailman/listinfo/platformone -- Michael Holmes, RHCSA Senior Consultant Red Hat Remote - Texas mholmes at redhat.com M: 808-780-4877 From tmiller at mitre.org Fri Jan 17 19:49:12 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Fri, 17 Jan 2020 19:49:12 +0000 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: <73A0E99F-E966-4A03-A058-21EC44B8D35A@mitre.org> References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> <73A0E99F-E966-4A03-A058-21EC44B8D35A@mitre.org> Message-ID: <3F6AC4AD-D539-4153-BC15-230C5A870997@mitre.org> Buglet/minor mod: find:paths: should have /root added. I'd prefer to just use `/`, but that might take too long. -- T ?On 1/17/20, 13:45, "Miller, Timothy J." wrote: Can you run this instead, then tarball the resultant `authorized_keys` dir and send me the tarball? --- - hosts: localhost become: true tasks: - name: find all authorized_keys find: paths: - /home recurse: true patterns: 'authorized_keys' register: key_files - name: write match data to a file copy: content: "{{ key_files }}" dest: /tmp/match_data.json - name: fetch all matched files fetch: src: "{{ item.path }}" dest: ./authorized_keys loop: "{{ key_files.files }}" - name: fetch match data fetch: src: /tmp/match_data.json dest: ./authorized_keys -- T On 1/17/20, 10:26, "Michael Holmes" wrote: Hi Tim, here is the ansible play I used. --- - hosts: dev-prod become: true tasks: - name: fetch file slurp: src: /home/ec2-user/.ssh/authorized_keys register: keys - debug: msg: "{{ keys['content'] }}" On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. wrote: Kevin-- Can you share the ansible play / ad-hoc command used to collect this? Thanks. -- T On 1/16/20, 15:16, "Kevin O'Donnell" wrote: All, Attached is the output from the authorized_keys files on all nodes in up-prod. Please let me know if you need any additional information. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell wrote: Hello All, I know we are putting a call together for tomorrow to discuss. In prep for that call to clarify the current keys that are used for the ec2-user post-deployment, and the key var's that are in Git (Or NOT). "The up-prod-a keys were switched out before Xmas." And are NOT stored in Git. The ec2-user does not exist and is not used in our current pipeline or code and or in our AMI's that are used to build the VPC's. The ec2-user account has been removed and the local "pipeline" named account is used to provision the VPC's and is the only account on the ec2 instances. To recap, the ec2-user no longer exists, admins are using IDM accounts with private keys that are not stored in git, and the keys for the pipeline account and or the old ec2-user account are cycled post-deployment and are not stored in Git. As we move into Cloud1 we will also be deploying a series of tooling such as Certbot to cycle keys automatically. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC wrote: +Wayne -----Original Message----- From: Miller, Timothy J. Sent: Wednesday, January 15, 2020 3:20 PM To: Kevin O'Donnell Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) Importance: High I'm resurrecting this thread. I went back and inspected the IaC. Per Kevin in the thread below, changes *were* made to the ec2-instance role. This role now provisions ec2-user with a named key registered in AWS (recall that AWS only manages public keys, but this is all that's needed. The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. I inspected all the current repositories for CI/CD definitions, extracted all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA keys and inspected the fingerprints for all keys registered in the CYBERCOM AWS account. TL;DR Findings: (Data in the raw results, below, and in the attached.) - The ec2-user SSH private key is shared across DSOP dev, staging, and production, *and* is shared with up-prod and up-dev. - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD variables to anyone with Maintainer or higher access. - The GitLab holding these keys is publicly accessible. - There are several additional RSA private keys present in the CI/CD variables, but their use is unclear. Risks: - All keys contained in a GitLab CI/CD variable should be considered compromised. There is no attribution for these keys. - Any user who can escalate privileges in GitLab via a bug, error, or social engineering can exfiltrate any of the private keys contained in CI/CD environment variables. Recommended mitigations: - Rotate all keys in raw results in all current platform1 instances. - Remove all exposed RSA keys from GitLab CI/CD configurations. - Move all deployment git repositories and CI/CD pipelines to a private GitLab instance. - Move all keys to a secrets manager, either under AWS control or deployed as a new platform1 capability. Outstanding issues: - I don't know how the gitlab-runner is being provisioned with the named key. If anyone can walk me through the runner architecture, I'd appreciate it. - I need to track down any use/reuse of the additional identified RSA private keys. The simple expedient is to remove them from GitLab CI/CD variables and see what breaks, but YMMV. Raw results: Repository DEFAULT_KEY_NAME Domains serviced ---------- ---------------- ---------------- up-node-infrastructure up-ss-ansible unified-platform.io up-appdev-a up-ss-ansible up-appdev-a.io platform-up-infrastructure-up-node n/a platform-infrastructure up-ss-ansible levelup-staging.io levelup-dev.io dsop.io platform-apps n/a Key usages ---------- 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 up-ss-ansible platform-apps/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d up-node-infrastructure/SSH_PRIVATE_KEY_FILE platform-apps/SSH_PRIVATE_KEY platform-infrastructure-up-node/CI_RUNNER_KEY platform-infrastructure/CI_RUNNER_KEY up-appdev-a/CI_RUNNER_KEY up-node-infrastructure/CI_RUNNER_KEY 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d platform-apps/GITLAB_PRIVATE_KEY e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 up-prod up-node-infrastructure/BASTION_KEY 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d up-node-infrastructure/SSO_CERT_KEY -- T On 1/5/20, 14:35, "Kevin O'Donnell" wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen _______________________________________________ platformONE mailing list platformONE at redhat.com https://www.redhat.com/mailman/listinfo/platformone -- Michael Holmes, RHCSA Senior Consultant Red Hat Remote - Texas mholmes at redhat.com M: 808-780-4877 From tmiller at mitre.org Fri Jan 17 19:58:38 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Fri, 17 Jan 2020 19:58:38 +0000 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: <3F6AC4AD-D539-4153-BC15-230C5A870997@mitre.org> References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> <73A0E99F-E966-4A03-A058-21EC44B8D35A@mitre.org> <3F6AC4AD-D539-4153-BC15-230C5A870997@mitre.org> Message-ID: <5260BFAD-1FC1-4E6E-9409-73DA5A8D83E5@mitre.org> Aaaand that should obviously be `hosts: all`. Testing locally, forgot to fix before sending. :? -- T ?On 1/17/20, 13:49, "Miller, Timothy J." wrote: Buglet/minor mod: find:paths: should have /root added. I'd prefer to just use `/`, but that might take too long. -- T On 1/17/20, 13:45, "Miller, Timothy J." wrote: Can you run this instead, then tarball the resultant `authorized_keys` dir and send me the tarball? --- - hosts: localhost become: true tasks: - name: find all authorized_keys find: paths: - /home recurse: true patterns: 'authorized_keys' register: key_files - name: write match data to a file copy: content: "{{ key_files }}" dest: /tmp/match_data.json - name: fetch all matched files fetch: src: "{{ item.path }}" dest: ./authorized_keys loop: "{{ key_files.files }}" - name: fetch match data fetch: src: /tmp/match_data.json dest: ./authorized_keys -- T On 1/17/20, 10:26, "Michael Holmes" wrote: Hi Tim, here is the ansible play I used. --- - hosts: dev-prod become: true tasks: - name: fetch file slurp: src: /home/ec2-user/.ssh/authorized_keys register: keys - debug: msg: "{{ keys['content'] }}" On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. wrote: Kevin-- Can you share the ansible play / ad-hoc command used to collect this? Thanks. -- T On 1/16/20, 15:16, "Kevin O'Donnell" wrote: All, Attached is the output from the authorized_keys files on all nodes in up-prod. Please let me know if you need any additional information. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell wrote: Hello All, I know we are putting a call together for tomorrow to discuss. In prep for that call to clarify the current keys that are used for the ec2-user post-deployment, and the key var's that are in Git (Or NOT). "The up-prod-a keys were switched out before Xmas." And are NOT stored in Git. The ec2-user does not exist and is not used in our current pipeline or code and or in our AMI's that are used to build the VPC's. The ec2-user account has been removed and the local "pipeline" named account is used to provision the VPC's and is the only account on the ec2 instances. To recap, the ec2-user no longer exists, admins are using IDM accounts with private keys that are not stored in git, and the keys for the pipeline account and or the old ec2-user account are cycled post-deployment and are not stored in Git. As we move into Cloud1 we will also be deploying a series of tooling such as Certbot to cycle keys automatically. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC wrote: +Wayne -----Original Message----- From: Miller, Timothy J. Sent: Wednesday, January 15, 2020 3:20 PM To: Kevin O'Donnell Cc: Blade, Eric D [US] (MS) ; Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) Importance: High I'm resurrecting this thread. I went back and inspected the IaC. Per Kevin in the thread below, changes *were* made to the ec2-instance role. This role now provisions ec2-user with a named key registered in AWS (recall that AWS only manages public keys, but this is all that's needed. The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. I inspected all the current repositories for CI/CD definitions, extracted all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA keys and inspected the fingerprints for all keys registered in the CYBERCOM AWS account. TL;DR Findings: (Data in the raw results, below, and in the attached.) - The ec2-user SSH private key is shared across DSOP dev, staging, and production, *and* is shared with up-prod and up-dev. - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD variables to anyone with Maintainer or higher access. - The GitLab holding these keys is publicly accessible. - There are several additional RSA private keys present in the CI/CD variables, but their use is unclear. Risks: - All keys contained in a GitLab CI/CD variable should be considered compromised. There is no attribution for these keys. - Any user who can escalate privileges in GitLab via a bug, error, or social engineering can exfiltrate any of the private keys contained in CI/CD environment variables. Recommended mitigations: - Rotate all keys in raw results in all current platform1 instances. - Remove all exposed RSA keys from GitLab CI/CD configurations. - Move all deployment git repositories and CI/CD pipelines to a private GitLab instance. - Move all keys to a secrets manager, either under AWS control or deployed as a new platform1 capability. Outstanding issues: - I don't know how the gitlab-runner is being provisioned with the named key. If anyone can walk me through the runner architecture, I'd appreciate it. - I need to track down any use/reuse of the additional identified RSA private keys. The simple expedient is to remove them from GitLab CI/CD variables and see what breaks, but YMMV. Raw results: Repository DEFAULT_KEY_NAME Domains serviced ---------- ---------------- ---------------- up-node-infrastructure up-ss-ansible unified-platform.io up-appdev-a up-ss-ansible up-appdev-a.io platform-up-infrastructure-up-node n/a platform-infrastructure up-ss-ansible levelup-staging.io levelup-dev.io dsop.io platform-apps n/a Key usages ---------- 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 up-ss-ansible platform-apps/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d up-node-infrastructure/SSH_PRIVATE_KEY_FILE platform-apps/SSH_PRIVATE_KEY platform-infrastructure-up-node/CI_RUNNER_KEY platform-infrastructure/CI_RUNNER_KEY up-appdev-a/CI_RUNNER_KEY up-node-infrastructure/CI_RUNNER_KEY 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d platform-apps/GITLAB_PRIVATE_KEY e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 up-prod up-node-infrastructure/BASTION_KEY 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d up-node-infrastructure/SSO_CERT_KEY -- T On 1/5/20, 14:35, "Kevin O'Donnell" wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet Cc: Feiglstok, Colleen M [US] (MS) ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; platformONE at redhat.com; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet ; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP ; Kevin O'Donnell ; platformONE at redhat.com ; Tim Gast ; Bubb, Mike ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; tj.zimmerman at braingu.com ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs ; Miller, Timothy J. ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Wilcox, John R. (San Antonio, TX) [US] (MS) Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen _______________________________________________ platformONE mailing list platformONE at redhat.com https://www.redhat.com/mailman/listinfo/platformone -- Michael Holmes, RHCSA Senior Consultant Red Hat Remote - Texas mholmes at redhat.com M: 808-780-4877 From ckuperst at redhat.com Fri Jan 17 20:53:02 2020 From: ckuperst at redhat.com (Chris Kuperstein) Date: Fri, 17 Jan 2020 12:53:02 -0800 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: <5260BFAD-1FC1-4E6E-9409-73DA5A8D83E5@mitre.org> References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> <73A0E99F-E966-4A03-A058-21EC44B8D35A@mitre.org> <3F6AC4AD-D539-4153-BC15-230C5A870997@mitre.org> <5260BFAD-1FC1-4E6E-9409-73DA5A8D83E5@mitre.org> Message-ID: Hi Tim, You have maintainer access to the repositories that store the private keys, shouldn't you be able to run these ansible plays yourself? On Fri, Jan 17, 2020 at 11:59 AM Miller, Timothy J. wrote: > Aaaand that should obviously be `hosts: all`. Testing locally, forgot to > fix before sending. :? > > -- T > > ?On 1/17/20, 13:49, "Miller, Timothy J." wrote: > > Buglet/minor mod: find:paths: should have /root added. I'd prefer to > just use `/`, but that might take too long. > > -- T > > On 1/17/20, 13:45, "Miller, Timothy J." wrote: > > Can you run this instead, then tarball the resultant > `authorized_keys` dir and send me the tarball? > > --- > - hosts: localhost > become: true > tasks: > > - name: find all authorized_keys > find: > paths: > - /home > recurse: true > patterns: 'authorized_keys' > register: key_files > > - name: write match data to a file > copy: > content: "{{ key_files }}" > dest: /tmp/match_data.json > > - name: fetch all matched files > fetch: > src: "{{ item.path }}" > dest: ./authorized_keys > loop: "{{ key_files.files }}" > > - name: fetch match data > fetch: > src: /tmp/match_data.json > dest: ./authorized_keys > > > -- T > > On 1/17/20, 10:26, "Michael Holmes" wrote: > > Hi Tim, here is the ansible play I used. > > > --- > - hosts: dev-prod > become: true > tasks: > > - name: fetch file > slurp: > src: /home/ec2-user/.ssh/authorized_keys > register: keys > > - debug: > msg: "{{ keys['content'] }}" > > > > On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. < > tmiller at mitre.org> wrote: > > > Kevin-- > > Can you share the ansible play / ad-hoc command used to > collect this? > > Thanks. > > -- T > > On 1/16/20, 15:16, "Kevin O'Donnell" > wrote: > > All, > > > Attached is the output from the authorized_keys files on > all nodes in up-prod. Please let me know if you need any additional > information. > > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell < > kodonnel at redhat.com> wrote: > > > Hello All, > > > I know we are putting a call together for tomorrow to > discuss. In prep for that call to clarify the current keys that are used > for the ec2-user post-deployment, and the key var's that are in Git (Or > NOT). > > > > "The up-prod-a keys were switched out before Xmas." And > are NOT stored in Git. > > > > The ec2-user does not exist and is not used in our current > pipeline or code and or in our AMI's that are used to build the VPC's. The > ec2-user account has been removed and the local "pipeline" named account is > used to provision the VPC's and is the only > account on the ec2 instances. > > > To recap, the ec2-user no longer exists, admins are using > IDM accounts with private keys that are not stored in git, and the keys for > the pipeline account and or the old ec2-user account are cycled > post-deployment and are not stored in Git. > > > > As we move into Cloud1 we will also be deploying a series > of tooling such as Certbot to cycle keys automatically. > > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt > USAF AFMC AFLCMC/HNC wrote: > > > +Wayne > > -----Original Message----- > From: Miller, Timothy J. > Sent: Wednesday, January 15, 2020 3:20 PM > To: Kevin O'Donnell > Cc: Blade, Eric D [US] (MS) ; > Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com>; > BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP < > austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC > ESC/AFLCMC/HNCP ; > platformONE at redhat.com; Tim Gast ; Bubb, > Mike ; TRAMBLE, > ELIJAH > Q Capt USAF AFMC AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col > USAF AFMC AFLCMC/HNCP ; RAMIREZ, > JOSE > A CTR USAF AFMC AFLCMC/HNCP < > jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; > REINHARDT, MELISSA A GG-13 > USAF > AFMC AFLCMC/HNCP ; Taylor > Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < > joshua.crisp.2 at us.af.mil>; > BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San Antonio, TX) [US] > (MS) > Subject: [Non-DoD Source] Key sharing across multiple > platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) > Importance: High > > I'm resurrecting this thread. > > I went back and inspected the IaC. Per Kevin in the > thread below, changes *were* made to the ec2-instance role. This role now > provisions ec2-user with a named key registered in AWS (recall that AWS > only manages public keys, but this is all that's needed. > > The named key is stored in the CI/CD variables as > DEFAULT_KEY_NAME. > > I inspected all the current repositories for CI/CD > definitions, extracted all the RSA private keys I could find, noted the > DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA > keys and inspected the fingerprints for all keys registered > in the CYBERCOM AWS account. > > TL;DR Findings: > (Data in the raw results, below, and in the attached.) > > - The ec2-user SSH private key is shared across DSOP dev, > staging, and production, *and* is shared with up-prod and up-dev. > > - The ec2-user SSH private key for DSOP production, > staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD > variables to anyone with Maintainer or higher access. > > - The GitLab holding these keys is publicly accessible. > > - There are several additional RSA private keys present in > the CI/CD variables, but their use is unclear. > > Risks: > > - All keys contained in a GitLab CI/CD variable should be > considered compromised. There is no attribution for these keys. > > - Any user who can escalate privileges in GitLab via a > bug, error, or social engineering can exfiltrate any of the private keys > contained in CI/CD environment variables. > > Recommended mitigations: > > - Rotate all keys in raw results in all current platform1 > instances. > > - Remove all exposed RSA keys from GitLab CI/CD > configurations. > > - Move all deployment git repositories and CI/CD pipelines > to a private GitLab instance. > > - Move all keys to a secrets manager, either under AWS > control or deployed as a new platform1 capability. > > Outstanding issues: > > - I don't know how the gitlab-runner is being provisioned > with the named key. If anyone can walk me through the runner architecture, > I'd appreciate it. > > - I need to track down any use/reuse of the additional > identified RSA private keys. The simple expedient is to remove them from > GitLab CI/CD variables and see what breaks, but YMMV. > > Raw results: > > Repository DEFAULT_KEY_NAME > Domains serviced > ---------- ---------------- > ---------------- > up-node-infrastructure up-ss-ansible > unified-platform.io < > http://unified-platform.io> > up-appdev-a up-ss-ansible > up-appdev-a.io > platform-up-infrastructure-up-node n/a > platform-infrastructure up-ss-ansible > levelup-staging.io < > http://levelup-staging.io> > > levelup-dev.io < > http://levelup-dev.io> > > dsop.io > platform-apps n/a > > Key usages > ---------- > 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 > up-ss-ansible > platform-apps/ANSIBLE_PRIVATE_KEY_FILE > > platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE > platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE > up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE > up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE > > d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d > up-node-infrastructure/SSH_PRIVATE_KEY_FILE > platform-apps/SSH_PRIVATE_KEY > platform-infrastructure-up-node/CI_RUNNER_KEY > platform-infrastructure/CI_RUNNER_KEY > up-appdev-a/CI_RUNNER_KEY > up-node-infrastructure/CI_RUNNER_KEY > > 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d > platform-apps/GITLAB_PRIVATE_KEY > > e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 > up-prod > up-node-infrastructure/BASTION_KEY > > 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d > up-node-infrastructure/SSO_CERT_KEY > > -- T > > On 1/5/20, 14:35, "Kevin O'Donnell" > wrote: > > Hello Tim, > > > Welcome back. Happy New Year. > > > We have a plan in place to swap out the ec2-user for > another user seeing as the major issue is the actual name of the user not > its privs in prod. > > > Incorrect up-prod-a keys were switched out before > Xmas. This process will need to be included in the Day Two OPS onboarding > process. Jet, who should be the holder of this key on day two? We are also > ensuring that anyone that authenticates to the nodes > in the vpc use dedicated IDM accounts with per user > dedicated keys and sudo privs. > > > Correct. Re-enforcing the Day Two requirements. And > also supporting that the pipelines and CI process can support CI/CD to only > dev/staging. For prod or any environment were the keys have been swapped > and are not included as private Git variables we > can > not support CD into prod/appdev if so deemed. > > > We are fast approaching policy with this topic. We > should have a further discussion this week. I will be in San Antonio on Wed > and Thur. > > > Do you have any additional mitigation techniques Tim? > > > Thanks, > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. < > tmiller at mitre.org> wrote: > > > Some more points of interest-- > > - The ec2-user account can't currently be avoided > given the current design of the IaC. > > - This SSH key provisioned for this account is shared > across up-prod-a, up-dev-a, and DSOP. > > - The ec2-user SSH privkey is available to anyone with > Maintainer or higher access to the GitLab repo that holds the customer's > environment definitions or the platform-infrastructure repo. > > -- T > > On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" < > Eric.Blade at ngc.com> wrote: > > I just want to clarify on the topic of the > EC2-user. The key is attribution. We need to know who or what is > configuring the systems. > > We are repeatedly seeing the ec2-user account used > to manually configure systems. Ec2-user should only be used to perform > programmatic tasks from committed and > reviewed code as part of initial provisioning. > If the code to provision and configure systems is unavailable, then you > must create attributable accounts to perform changes to the systems. > > Any use of ec2-user remote login or su to ec2-user > will be flagged as a violation and the source IP or user account logged as > the violating source. Any manual/command > line activities MUST be performed using an > attributable user account. > > If that is not clear, please let me know and I > will attempt to elaborate further. > > Thank you > > > Eric Blade, GXPN, GPEN > Unified Platform System Coordinator > Northrop Grumman Mission Systems > Work: 410.649.0706 > Mobile: 240.258.8089 > > > > From: Kevin O'Donnell > > Sent: Thursday, December 19, 2019 6:22 PM > To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP < > austen.bryan.1 at us.af.mil>; > DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; > platformONE at redhat.com; Tim Gast ; > Bubb, Mike ; TRAMBLE, ELIJAH Q > Capt USAF AFMC AFLCMC/HNC ; > tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt > Col USAF AFMC AFLCMC/HNCP ; Blade, > Eric > D [US] (MS) ; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < > jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; > REINHARDT, > MELISSA > A GG-13 USAF AFMC AFLCMC/HNCP < > melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; Miller, > Timothy J. ; > CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < > joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < > steven.bogue.1.ctr at us.af.mil>; > Wilcox, John R. (San Antonio, TX) [US] (MS) < > John.R.Wilcox at ngc.com> > Subject: EXT :Re: [EXT] Platform1 SAR > > Colleen, > > > Thank you for the results and recommendations. We > will get GIT issues crated for your findings and will prioritize the > mitigation and implement them as code in our future IAC deployments. Many > of the findings in the current VPC have been > mitigated in up-prod-b with our current code > release. > > > > Please let us know when you have finished and we > can power down the host that you have been using for scanning. > > > > > Note for everyone: Once we power down the ec2 > instance ssh or port 22 will not be externally accessible. Thus, mitigating > many of the risks associated with the ec2-user and the keys. > > > > > Thanks, > > > > KEVIN O'DONNELL > ARCHITECT MANAGER > Red Hat Red Hat NA Public Sector Consulting < > https://www.redhat.com/> > kodonnell at redhat.com > M: 240-605-4654 > > > > > > > > > > > > > > > > > > On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet < > jlastrilla at mitre.org> wrote: > > > Thanks Colleen. Sorry for the rushed feeling. If > you want to take more time, please use tomorrow to do your testing. > > > > Thank you for all you do!!!! > > Get Outlook for iOS > > > > ________________________________________ > > From: Feiglstok, Colleen M [US] (MS) < > Colleen.Feiglstok at ngc.com> > Sent: Thursday, December 19, 2019 4:35:15 PM > To: Lastrilla, Jet ; BRYAN, > AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; DIROCCO, > ROGER > E > GG-13 USAF AFMC ESC/AFLCMC/HNCP < > roger.dirocco.4 at us.af.mil>; Kevin O'Donnell ; > platformONE at redhat.com ; > Tim Gast ; > Bubb, > Mike > ; TRAMBLE, ELIJAH Q Capt USAF > AFMC AFLCMC/HNC ; > tj.zimmerman at braingu.com ; > LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP < > richard.lopezdeuralde at us.af.mil>; > Blade, Eric D [US] (MS) ; > RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; > Leonard, > Michael > C. ; REINHARDT, MELISSA A > GG-13 USAF AFMC AFLCMC/HNCP ; Taylor Biggs > ; > Miller, Timothy J. ; CRISP, > JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, > STEVEN E > CTR > USAF > AFMC > AFLCMC/HNCP ; > Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR > > > > All, > > The SAR and raw results from the new security > testing will be sent through NGSafe in a few moments. > > As usual, I felt very rushed with the testing, and > feel like I have not done as thorough of a job as required. I was unable to > log into the Web UIs, as no one from the Platform1 team gave me the account > information. I had issues with Nessus, so > the > CVE?s > were found through OSCAP this time. > > A lot is the same as the last report, but please > read through it, because there is some new information. I had to test as > ec2-user again, which is another big issue that needs to be resolved ASAP. > The more I use it and find out > how it is being used, the more extremely > concerned I am. It has multiple keys throughout the platform located in the > .ssh directory, one of which is world readable. On some hosts, a real user > is using the ec2-user account to create accounts, groups, > and > pull > docker files. The account is non-attributable, so > we have no way of knowing who is doing this. Someone could do serious > damage with no consequence. I understand that the ec2-user is needed for > standing up an ec2-image, but > this account should only be used for implementing > IAC, so that the changes implemented by ec2-user are codified. If manual > admin is required, that IAC should provision the appropriate attributable > accounts, and those accounts should > be used from then on. In my opinion, this is a > critical finding and needs to be addressed ASAP. > > > I will be available during the day tomorrow for > any questions. > > Thanks > Colleen > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > > > > > > > -- > Michael Holmes, RHCSA > Senior Consultant > Red Hat Remote - Texas > > mholmes at redhat.com > M: 808-780-4877 > > > > > > > > > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > -- Chris Kuperstein Architect Red Hat North American Public Sector ckuperst at redhat.com M: 503.616.6209 @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From kodonnel at redhat.com Fri Jan 17 20:59:51 2020 From: kodonnel at redhat.com (Kevin O'Donnell) Date: Fri, 17 Jan 2020 15:59:51 -0500 Subject: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> <73A0E99F-E966-4A03-A058-21EC44B8D35A@mitre.org> <3F6AC4AD-D539-4153-BC15-230C5A870997@mitre.org> <5260BFAD-1FC1-4E6E-9409-73DA5A8D83E5@mitre.org> Message-ID: All, Please direct any security related questions and or requests to Wayne. We will then get the requests and or questions entered into git as an issue and will prioritizes them accordingly. Thanks, -kevin On Fri, Jan 17, 2020 at 3:53 PM Chris Kuperstein wrote: > Hi Tim, > > You have maintainer access to the repositories that store the private > keys, shouldn't you be able to run these ansible plays yourself? > > On Fri, Jan 17, 2020 at 11:59 AM Miller, Timothy J. > wrote: > >> Aaaand that should obviously be `hosts: all`. Testing locally, forgot to >> fix before sending. :? >> >> -- T >> >> ?On 1/17/20, 13:49, "Miller, Timothy J." wrote: >> >> Buglet/minor mod: find:paths: should have /root added. I'd prefer >> to just use `/`, but that might take too long. >> >> -- T >> >> On 1/17/20, 13:45, "Miller, Timothy J." wrote: >> >> Can you run this instead, then tarball the resultant >> `authorized_keys` dir and send me the tarball? >> >> --- >> - hosts: localhost >> become: true >> tasks: >> >> - name: find all authorized_keys >> find: >> paths: >> - /home >> recurse: true >> patterns: 'authorized_keys' >> register: key_files >> >> - name: write match data to a file >> copy: >> content: "{{ key_files }}" >> dest: /tmp/match_data.json >> >> - name: fetch all matched files >> fetch: >> src: "{{ item.path }}" >> dest: ./authorized_keys >> loop: "{{ key_files.files }}" >> >> - name: fetch match data >> fetch: >> src: /tmp/match_data.json >> dest: ./authorized_keys >> >> >> -- T >> >> On 1/17/20, 10:26, "Michael Holmes" wrote: >> >> Hi Tim, here is the ansible play I used. >> >> >> --- >> - hosts: dev-prod >> become: true >> tasks: >> >> - name: fetch file >> slurp: >> src: /home/ec2-user/.ssh/authorized_keys >> register: keys >> >> - debug: >> msg: "{{ keys['content'] }}" >> >> >> >> On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. < >> tmiller at mitre.org> wrote: >> >> >> Kevin-- >> >> Can you share the ansible play / ad-hoc command used to >> collect this? >> >> Thanks. >> >> -- T >> >> On 1/16/20, 15:16, "Kevin O'Donnell" >> wrote: >> >> All, >> >> >> Attached is the output from the authorized_keys files on >> all nodes in up-prod. Please let me know if you need any additional >> information. >> >> >> >> Thanks, >> >> KEVIN O'DONNELL >> ARCHITECT MANAGER >> Red Hat Red Hat NA Public Sector Consulting < >> https://www.redhat.com/> >> >> kodonnell at redhat.com >> M: 240-605-4654 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell < >> kodonnel at redhat.com> wrote: >> >> >> Hello All, >> >> >> I know we are putting a call together for tomorrow to >> discuss. In prep for that call to clarify the current keys that are used >> for the ec2-user post-deployment, and the key var's that are in Git (Or >> NOT). >> >> >> >> "The up-prod-a keys were switched out before Xmas." And >> are NOT stored in Git. >> >> >> >> The ec2-user does not exist and is not used in our >> current pipeline or code and or in our AMI's that are used to build the >> VPC's. The ec2-user account has been removed and the local "pipeline" named >> account is used to provision the VPC's and is the only >> account on the ec2 instances. >> >> >> To recap, the ec2-user no longer exists, admins are using >> IDM accounts with private keys that are not stored in git, and the keys for >> the pipeline account and or the old ec2-user account are cycled >> post-deployment and are not stored in Git. >> >> >> >> As we move into Cloud1 we will also be deploying a series >> of tooling such as Certbot to cycle keys automatically. >> >> >> >> Thanks, >> >> KEVIN O'DONNELL >> ARCHITECT MANAGER >> Red Hat Red Hat NA Public Sector Consulting < >> https://www.redhat.com/> >> >> kodonnell at redhat.com >> M: 240-605-4654 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt >> USAF AFMC AFLCMC/HNC wrote: >> >> >> +Wayne >> >> -----Original Message----- >> From: Miller, Timothy J. >> Sent: Wednesday, January 15, 2020 3:20 PM >> To: Kevin O'Donnell >> Cc: Blade, Eric D [US] (MS) ; >> Lastrilla, Jet ; Feiglstok, Colleen M [US] (MS) < >> Colleen.Feiglstok at ngc.com>; >> BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP < >> austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC >> ESC/AFLCMC/HNCP ; >> platformONE at redhat.com; Tim Gast ; Bubb, >> Mike ; TRAMBLE, >> ELIJAH >> Q Capt USAF AFMC AFLCMC/HNC > >; >> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt >> Col USAF AFMC AFLCMC/HNCP ; RAMIREZ, >> JOSE >> A CTR USAF AFMC AFLCMC/HNCP < >> jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; >> REINHARDT, MELISSA A GG-13 >> USAF >> AFMC AFLCMC/HNCP ; >> Taylor Biggs ; CRISP, JOSHUA M GS-09 USAF AFMC >> AFLCMC/HNCP ; >> BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < >> steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San Antonio, TX) [US] >> (MS) >> Subject: [Non-DoD Source] Key sharing across multiple >> platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) >> Importance: High >> >> I'm resurrecting this thread. >> >> I went back and inspected the IaC. Per Kevin in the >> thread below, changes *were* made to the ec2-instance role. This role now >> provisions ec2-user with a named key registered in AWS (recall that AWS >> only manages public keys, but this is all that's needed. >> >> The named key is stored in the CI/CD variables as >> DEFAULT_KEY_NAME. >> >> I inspected all the current repositories for CI/CD >> definitions, extracted all the RSA private keys I could find, noted the >> DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA >> keys and inspected the fingerprints for all keys registered >> in the CYBERCOM AWS account. >> >> TL;DR Findings: >> (Data in the raw results, below, and in the attached.) >> >> - The ec2-user SSH private key is shared across DSOP dev, >> staging, and production, *and* is shared with up-prod and up-dev. >> >> - The ec2-user SSH private key for DSOP production, >> staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD >> variables to anyone with Maintainer or higher access. >> >> - The GitLab holding these keys is publicly accessible. >> >> - There are several additional RSA private keys present >> in the CI/CD variables, but their use is unclear. >> >> Risks: >> >> - All keys contained in a GitLab CI/CD variable should be >> considered compromised. There is no attribution for these keys. >> >> - Any user who can escalate privileges in GitLab via a >> bug, error, or social engineering can exfiltrate any of the private keys >> contained in CI/CD environment variables. >> >> Recommended mitigations: >> >> - Rotate all keys in raw results in all current platform1 >> instances. >> >> - Remove all exposed RSA keys from GitLab CI/CD >> configurations. >> >> - Move all deployment git repositories and CI/CD >> pipelines to a private GitLab instance. >> >> - Move all keys to a secrets manager, either under AWS >> control or deployed as a new platform1 capability. >> >> Outstanding issues: >> >> - I don't know how the gitlab-runner is being provisioned >> with the named key. If anyone can walk me through the runner architecture, >> I'd appreciate it. >> >> - I need to track down any use/reuse of the additional >> identified RSA private keys. The simple expedient is to remove them from >> GitLab CI/CD variables and see what breaks, but YMMV. >> >> Raw results: >> >> Repository DEFAULT_KEY_NAME >> Domains serviced >> ---------- ---------------- >> ---------------- >> up-node-infrastructure up-ss-ansible >> unified-platform.io < >> http://unified-platform.io> >> up-appdev-a up-ss-ansible >> up-appdev-a.io >> platform-up-infrastructure-up-node n/a >> platform-infrastructure up-ss-ansible >> levelup-staging.io < >> http://levelup-staging.io> >> >> levelup-dev.io < >> http://levelup-dev.io> >> >> dsop.io >> platform-apps n/a >> >> Key usages >> ---------- >> >> 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 >> up-ss-ansible >> platform-apps/ANSIBLE_PRIVATE_KEY_FILE >> >> platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE >> platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE >> up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE >> up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE >> >> >> d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d >> up-node-infrastructure/SSH_PRIVATE_KEY_FILE >> platform-apps/SSH_PRIVATE_KEY >> platform-infrastructure-up-node/CI_RUNNER_KEY >> platform-infrastructure/CI_RUNNER_KEY >> up-appdev-a/CI_RUNNER_KEY >> up-node-infrastructure/CI_RUNNER_KEY >> >> >> 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d >> platform-apps/GITLAB_PRIVATE_KEY >> >> >> e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 >> up-prod >> up-node-infrastructure/BASTION_KEY >> >> >> 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d >> up-node-infrastructure/SSO_CERT_KEY >> >> -- T >> >> On 1/5/20, 14:35, "Kevin O'Donnell" >> wrote: >> >> Hello Tim, >> >> >> Welcome back. Happy New Year. >> >> >> We have a plan in place to swap out the ec2-user for >> another user seeing as the major issue is the actual name of the user not >> its privs in prod. >> >> >> Incorrect up-prod-a keys were switched out before >> Xmas. This process will need to be included in the Day Two OPS onboarding >> process. Jet, who should be the holder of this key on day two? We are also >> ensuring that anyone that authenticates to the nodes >> in the vpc use dedicated IDM accounts with per user >> dedicated keys and sudo privs. >> >> >> Correct. Re-enforcing the Day Two requirements. And >> also supporting that the pipelines and CI process can support CI/CD to only >> dev/staging. For prod or any environment were the keys have been swapped >> and are not included as private Git variables we >> can >> not support CD into prod/appdev if so deemed. >> >> >> We are fast approaching policy with this topic. We >> should have a further discussion this week. I will be in San Antonio on Wed >> and Thur. >> >> >> Do you have any additional mitigation techniques Tim? >> >> >> Thanks, >> >> KEVIN O'DONNELL >> ARCHITECT MANAGER >> Red Hat Red Hat NA Public Sector Consulting < >> https://www.redhat.com/> >> >> kodonnell at redhat.com >> M: 240-605-4654 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. < >> tmiller at mitre.org> wrote: >> >> >> Some more points of interest-- >> >> - The ec2-user account can't currently be avoided >> given the current design of the IaC. >> >> - This SSH key provisioned for this account is shared >> across up-prod-a, up-dev-a, and DSOP. >> >> - The ec2-user SSH privkey is available to anyone >> with Maintainer or higher access to the GitLab repo that holds the >> customer's environment definitions or the platform-infrastructure repo. >> >> -- T >> >> On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" < >> Eric.Blade at ngc.com> wrote: >> >> I just want to clarify on the topic of the >> EC2-user. The key is attribution. We need to know who or what is >> configuring the systems. >> >> We are repeatedly seeing the ec2-user account >> used to manually configure systems. Ec2-user should only be used to >> perform programmatic tasks from committed and >> reviewed code as part of initial provisioning. >> If the code to provision and configure systems is unavailable, then you >> must create attributable accounts to perform changes to the systems. >> >> Any use of ec2-user remote login or su to >> ec2-user will be flagged as a violation and the source IP or user account >> logged as the violating source. Any manual/command >> line activities MUST be performed using an >> attributable user account. >> >> If that is not clear, please let me know and I >> will attempt to elaborate further. >> >> Thank you >> >> >> Eric Blade, GXPN, GPEN >> Unified Platform System Coordinator >> Northrop Grumman Mission Systems >> Work: 410.649.0706 >> Mobile: 240.258.8089 >> >> >> >> From: Kevin O'Donnell >> >> Sent: Thursday, December 19, 2019 6:22 PM >> To: Lastrilla, Jet >> Cc: Feiglstok, Colleen M [US] (MS) < >> Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP < >> austen.bryan.1 at us.af.mil>; >> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP < >> roger.dirocco.4 at us.af.mil>; >> platformONE at redhat.com; Tim Gast ; >> Bubb, Mike ; TRAMBLE, ELIJAH Q >> Capt USAF AFMC AFLCMC/HNC ; >> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A >> Lt Col USAF AFMC AFLCMC/HNCP ; Blade, >> Eric >> D [US] (MS) ; >> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP < >> jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. ; >> REINHARDT, >> MELISSA >> A GG-13 USAF AFMC AFLCMC/HNCP < >> melissa.reinhardt.2 at us.af.mil>; Taylor Biggs ; >> Miller, Timothy J. ; >> CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP < >> joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP < >> steven.bogue.1.ctr at us.af.mil>; >> Wilcox, John R. (San Antonio, TX) [US] (MS) < >> John.R.Wilcox at ngc.com> >> Subject: EXT :Re: [EXT] Platform1 SAR >> >> Colleen, >> >> >> Thank you for the results and recommendations. We >> will get GIT issues crated for your findings and will prioritize the >> mitigation and implement them as code in our future IAC deployments. Many >> of the findings in the current VPC have been >> mitigated in up-prod-b with our current code >> release. >> >> >> >> Please let us know when you have finished and we >> can power down the host that you have been using for scanning. >> >> >> >> >> Note for everyone: Once we power down the ec2 >> instance ssh or port 22 will not be externally accessible. Thus, mitigating >> many of the risks associated with the ec2-user and the keys. >> >> >> >> >> Thanks, >> >> >> >> KEVIN O'DONNELL >> ARCHITECT MANAGER >> Red Hat Red Hat NA Public Sector Consulting < >> https://www.redhat.com/> >> kodonnell at redhat.com >> M: 240-605-4654 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet < >> jlastrilla at mitre.org> wrote: >> >> >> Thanks Colleen. Sorry for the rushed feeling. If >> you want to take more time, please use tomorrow to do your testing. >> >> >> >> Thank you for all you do!!!! >> >> Get Outlook for iOS >> >> >> >> ________________________________________ >> >> From: Feiglstok, Colleen M [US] (MS) < >> Colleen.Feiglstok at ngc.com> >> Sent: Thursday, December 19, 2019 4:35:15 PM >> To: Lastrilla, Jet ; >> BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP ; >> DIROCCO, ROGER >> E >> GG-13 USAF AFMC ESC/AFLCMC/HNCP < >> roger.dirocco.4 at us.af.mil>; Kevin O'Donnell ; >> platformONE at redhat.com ; >> Tim Gast ; >> Bubb, >> Mike >> ; TRAMBLE, ELIJAH Q Capt USAF >> AFMC AFLCMC/HNC ; >> tj.zimmerman at braingu.com < >> tj.zimmerman at braingu.com>; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC >> AFLCMC/HNCP ; >> Blade, Eric D [US] (MS) ; >> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP > >; >> Leonard, >> Michael >> C. ; REINHARDT, MELISSA A >> GG-13 USAF AFMC AFLCMC/HNCP ; Taylor >> Biggs ; >> Miller, Timothy J. ; CRISP, >> JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; BOGUE, >> STEVEN E >> CTR >> USAF >> AFMC >> AFLCMC/HNCP ; >> Wilcox, John R. (San Antonio, TX) [US] (MS) >> Subject: [EXT] Platform1 SAR >> >> >> >> All, >> >> The SAR and raw results from the new security >> testing will be sent through NGSafe in a few moments. >> >> As usual, I felt very rushed with the testing, >> and feel like I have not done as thorough of a job as required. I was >> unable to log into the Web UIs, as no one from the Platform1 team gave me >> the account information. I had issues with Nessus, so >> the >> CVE?s >> were found through OSCAP this time. >> >> A lot is the same as the last report, but please >> read through it, because there is some new information. I had to test as >> ec2-user again, which is another big issue that needs to be resolved ASAP. >> The more I use it and find out >> how it is being used, the more extremely >> concerned I am. It has multiple keys throughout the platform located in the >> .ssh directory, one of which is world readable. On some hosts, a real user >> is using the ec2-user account to create accounts, groups, >> and >> pull >> docker files. The account is non-attributable, >> so we have no way of knowing who is doing this. Someone could do serious >> damage with no consequence. I understand that the ec2-user is needed for >> standing up an ec2-image, but >> this account should only be used for implementing >> IAC, so that the changes implemented by ec2-user are codified. If manual >> admin is required, that IAC should provision the appropriate attributable >> accounts, and those accounts should >> be used from then on. In my opinion, this is a >> critical finding and needs to be addressed ASAP. >> >> >> I will be available during the day tomorrow for >> any questions. >> >> Thanks >> Colleen >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> platformONE mailing list >> platformONE at redhat.com >> https://www.redhat.com/mailman/listinfo/platformone >> >> >> >> >> >> >> -- >> Michael Holmes, RHCSA >> Senior Consultant >> Red Hat Remote - Texas >> >> mholmes at redhat.com >> M: 808-780-4877 >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> platformONE mailing list >> platformONE at redhat.com >> https://www.redhat.com/mailman/listinfo/platformone >> > > > -- > > Chris Kuperstein > > Architect > > Red Hat North American Public Sector > > ckuperst at redhat.com > M: 503.616.6209 > @RedHat Red Hat > Red Hat > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > -- KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 -------------- next part -------------- An HTML attachment was scrubbed... URL: From wayne.starr.1 at us.af.mil Fri Jan 17 22:39:41 2020 From: wayne.starr.1 at us.af.mil (STARR, WAYNE E 1st Lt USAF AETC 333 TRS/DIU) Date: Fri, 17 Jan 2020 22:39:41 +0000 Subject: [Platformone] [Non-DoD Source] Re: Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) In-Reply-To: References: <0237A769-0227-4005-BA51-D4B9E6EBB7F1@mitre.org> <773C717A-F41E-426F-AAD2-2A32E7A4D339@mitre.org> <73A0E99F-E966-4A03-A058-21EC44B8D35A@mitre.org> <3F6AC4AD-D539-4153-BC15-230C5A870997@mitre.org> <5260BFAD-1FC1-4E6E-9409-73DA5A8D83E5@mitre.org> Message-ID: Yes, please direct things through me and we can find the right place for it. I went ahead and entered the notes into the UP IATT project directly, so they can be viewed there (prepended them with [Security]). https://dccscr.dsop.io/ginyu-force/up-iatt/issues I was not sure how priority was currently being done within Gitlab by the P1 team, so they were created in the order of what should be easiest to hardest. Also, not sure if Mark or Kevin knows, but do you know why issues are tracked here as well as on the actual code project for up-prod itself? Both are private projects you need to get added to in order to see them, and it just seemed odd that they would be split apart from each other (and in separate Gitlab groups). Very Respectfully, WAYNE E. STARR, 1st Lt, USAF 90th Cyber Operations Squadron (e) wayne.starr.1 at us.af.mil (c) (970) 366-2530 From: Kevin O'Donnell Sent: Friday, January 17, 2020 3:00 PM To: Chris Kuperstein Cc: BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP ; Blade, Eric D [US] (MS) ; Bubb, Mike ; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP ; Feiglstok, Colleen M [US] (MS) ; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP ; Leonard, Michael C. ; Miller, Timothy J. ; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP ; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP ; STARR, WAYNE E 1st Lt USAF AETC 333 TRS/DIU ; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC ; Taylor Biggs ; Tim Gast ; Wilcox, John R. (San Antonio, TX) [US] (MS) ; platformONE at redhat.com; tj.zimmerman at braingu.com Subject: [Non-DoD Source] Re: [Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) All, Please direct any security related questions and or requests to Wayne. We will then get the requests and or questions entered into git as an issue and will prioritizes them accordingly. Thanks, -kevin On Fri, Jan 17, 2020 at 3:53 PM Chris Kuperstein > wrote: Hi Tim, You have maintainer access to the repositories that store the private keys, shouldn't you be able to run these ansible plays yourself? On Fri, Jan 17, 2020 at 11:59 AM Miller, Timothy J. > wrote: Aaaand that should obviously be `hosts: all`. Testing locally, forgot to fix before sending. :? -- T ?On 1/17/20, 13:49, "Miller, Timothy J." > wrote: Buglet/minor mod: find:paths: should have /root added. I'd prefer to just use `/`, but that might take too long. -- T On 1/17/20, 13:45, "Miller, Timothy J." > wrote: Can you run this instead, then tarball the resultant `authorized_keys` dir and send me the tarball? --- - hosts: localhost become: true tasks: - name: find all authorized_keys find: paths: - /home recurse: true patterns: 'authorized_keys' register: key_files - name: write match data to a file copy: content: "{{ key_files }}" dest: /tmp/match_data.json - name: fetch all matched files fetch: src: "{{ item.path }}" dest: ./authorized_keys loop: "{{ key_files.files }}" - name: fetch match data fetch: src: /tmp/match_data.json dest: ./authorized_keys -- T On 1/17/20, 10:26, "Michael Holmes" > wrote: Hi Tim, here is the ansible play I used. --- - hosts: dev-prod become: true tasks: - name: fetch file slurp: src: /home/ec2-user/.ssh/authorized_keys register: keys - debug: msg: "{{ keys['content'] }}" On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. > wrote: Kevin-- Can you share the ansible play / ad-hoc command used to collect this? Thanks. -- T On 1/16/20, 15:16, "Kevin O'Donnell" > wrote: All, Attached is the output from the authorized_keys files on all nodes in up-prod. Please let me know if you need any additional information. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com %20M:240-605-4654> M: 240-605-4654 On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell > wrote: Hello All, I know we are putting a call together for tomorrow to discuss. In prep for that call to clarify the current keys that are used for the ec2-user post-deployment, and the key var's that are in Git (Or NOT). "The up-prod-a keys were switched out before Xmas." And are NOT stored in Git. The ec2-user does not exist and is not used in our current pipeline or code and or in our AMI's that are used to build the VPC's. The ec2-user account has been removed and the local "pipeline" named account is used to provision the VPC's and is the only account on the ec2 instances. To recap, the ec2-user no longer exists, admins are using IDM accounts with private keys that are not stored in git, and the keys for the pipeline account and or the old ec2-user account are cycled post-deployment and are not stored in Git. As we move into Cloud1 we will also be deploying a series of tooling such as Certbot to cycle keys automatically. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com %20M:240-605-4654> M: 240-605-4654 On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC > wrote: +Wayne -----Original Message----- From: Miller, Timothy J. > Sent: Wednesday, January 15, 2020 3:20 PM To: Kevin O'Donnell > Cc: Blade, Eric D [US] (MS) >; Lastrilla, Jet >; Feiglstok, Colleen M [US] (MS) >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP >; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP >; platformONE at redhat.com; Tim Gast >; Bubb, Mike >; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP >; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP >; Leonard, Michael C. >; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP >; Taylor Biggs >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP >; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: [Non-DoD Source] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR) Importance: High I'm resurrecting this thread. I went back and inspected the IaC. Per Kevin in the thread below, changes *were* made to the ec2-instance role. This role now provisions ec2-user with a named key registered in AWS (recall that AWS only manages public keys, but this is all that's needed. The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME. I inspected all the current repositories for CI/CD definitions, extracted all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA keys and inspected the fingerprints for all keys registered in the CYBERCOM AWS account. TL;DR Findings: (Data in the raw results, below, and in the attached.) - The ec2-user SSH private key is shared across DSOP dev, staging, and production, *and* is shared with up-prod and up-dev. - The ec2-user SSH private key for DSOP production, staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD variables to anyone with Maintainer or higher access. - The GitLab holding these keys is publicly accessible. - There are several additional RSA private keys present in the CI/CD variables, but their use is unclear. Risks: - All keys contained in a GitLab CI/CD variable should be considered compromised. There is no attribution for these keys. - Any user who can escalate privileges in GitLab via a bug, error, or social engineering can exfiltrate any of the private keys contained in CI/CD environment variables. Recommended mitigations: - Rotate all keys in raw results in all current platform1 instances. - Remove all exposed RSA keys from GitLab CI/CD configurations. - Move all deployment git repositories and CI/CD pipelines to a private GitLab instance. - Move all keys to a secrets manager, either under AWS control or deployed as a new platform1 capability. Outstanding issues: - I don't know how the gitlab-runner is being provisioned with the named key. If anyone can walk me through the runner architecture, I'd appreciate it. - I need to track down any use/reuse of the additional identified RSA private keys. The simple expedient is to remove them from GitLab CI/CD variables and see what breaks, but YMMV. Raw results: Repository DEFAULT_KEY_NAME Domains serviced ---------- ---------------- ---------------- up-node-infrastructure up-ss-ansible unified-platform.io up-appdev-a up-ss-ansible up-appdev-a.io platform-up-infrastructure-up-node n/a platform-infrastructure up-ss-ansible levelup-staging.io levelup-dev.io dsop.io platform-apps n/a Key usages ---------- 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1 up-ss-ansible platform-apps/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d up-node-infrastructure/SSH_PRIVATE_KEY_FILE platform-apps/SSH_PRIVATE_KEY platform-infrastructure-up-node/CI_RUNNER_KEY platform-infrastructure/CI_RUNNER_KEY up-appdev-a/CI_RUNNER_KEY up-node-infrastructure/CI_RUNNER_KEY 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d platform-apps/GITLAB_PRIVATE_KEY e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65 up-prod up-node-infrastructure/BASTION_KEY 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d up-node-infrastructure/SSO_CERT_KEY -- T On 1/5/20, 14:35, "Kevin O'Donnell" > wrote: Hello Tim, Welcome back. Happy New Year. We have a plan in place to swap out the ec2-user for another user seeing as the major issue is the actual name of the user not its privs in prod. Incorrect up-prod-a keys were switched out before Xmas. This process will need to be included in the Day Two OPS onboarding process. Jet, who should be the holder of this key on day two? We are also ensuring that anyone that authenticates to the nodes in the vpc use dedicated IDM accounts with per user dedicated keys and sudo privs. Correct. Re-enforcing the Day Two requirements. And also supporting that the pipelines and CI process can support CI/CD to only dev/staging. For prod or any environment were the keys have been swapped and are not included as private Git variables we can not support CD into prod/appdev if so deemed. We are fast approaching policy with this topic. We should have a further discussion this week. I will be in San Antonio on Wed and Thur. Do you have any additional mitigation techniques Tim? Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com %20M:240-605-4654> M: 240-605-4654 On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. > wrote: Some more points of interest-- - The ec2-user account can't currently be avoided given the current design of the IaC. - This SSH key provisioned for this account is shared across up-prod-a, up-dev-a, and DSOP. - The ec2-user SSH privkey is available to anyone with Maintainer or higher access to the GitLab repo that holds the customer's environment definitions or the platform-infrastructure repo. -- T On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" > wrote: I just want to clarify on the topic of the EC2-user. The key is attribution. We need to know who or what is configuring the systems. We are repeatedly seeing the ec2-user account used to manually configure systems. Ec2-user should only be used to perform programmatic tasks from committed and reviewed code as part of initial provisioning. If the code to provision and configure systems is unavailable, then you must create attributable accounts to perform changes to the systems. Any use of ec2-user remote login or su to ec2-user will be flagged as a violation and the source IP or user account logged as the violating source. Any manual/command line activities MUST be performed using an attributable user account. If that is not clear, please let me know and I will attempt to elaborate further. Thank you Eric Blade, GXPN, GPEN Unified Platform System Coordinator Northrop Grumman Mission Systems Work: 410.649.0706 Mobile: 240.258.8089 From: Kevin O'Donnell > Sent: Thursday, December 19, 2019 6:22 PM To: Lastrilla, Jet > Cc: Feiglstok, Colleen M [US] (MS) >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP >; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP >; platformONE at redhat.com; Tim Gast >; Bubb, Mike >; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP >; Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP >; Leonard, Michael C. >; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP >; Taylor Biggs >; Miller, Timothy J. >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP >; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: EXT :Re: [EXT] Platform1 SAR Colleen, Thank you for the results and recommendations. We will get GIT issues crated for your findings and will prioritize the mitigation and implement them as code in our future IAC deployments. Many of the findings in the current VPC have been mitigated in up-prod-b with our current code release. Please let us know when you have finished and we can power down the host that you have been using for scanning. Note for everyone: Once we power down the ec2 instance ssh or port 22 will not be externally accessible. Thus, mitigating many of the risks associated with the ec2-user and the keys. Thanks, KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com %20M:240-605-4654> M: 240-605-4654 On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet > wrote: Thanks Colleen. Sorry for the rushed feeling. If you want to take more time, please use tomorrow to do your testing. Thank you for all you do!!!! Get Outlook for iOS ________________________________________ From: Feiglstok, Colleen M [US] (MS) > Sent: Thursday, December 19, 2019 4:35:15 PM To: Lastrilla, Jet >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP >; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP >; Kevin O'Donnell >; platformONE at redhat.com >; Tim Gast >; Bubb, Mike >; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC >; tj.zimmerman at braingu.com >; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP >; Blade, Eric D [US] (MS) >; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP >; Leonard, Michael C. >; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP >; Taylor Biggs >; Miller, Timothy J. >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP >; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP >; Wilcox, John R. (San Antonio, TX) [US] (MS) > Subject: [EXT] Platform1 SAR All, The SAR and raw results from the new security testing will be sent through NGSafe in a few moments. As usual, I felt very rushed with the testing, and feel like I have not done as thorough of a job as required. I was unable to log into the Web UIs, as no one from the Platform1 team gave me the account information. I had issues with Nessus, so the CVE?s were found through OSCAP this time. A lot is the same as the last report, but please read through it, because there is some new information. I had to test as ec2-user again, which is another big issue that needs to be resolved ASAP. The more I use it and find out how it is being used, the more extremely concerned I am. It has multiple keys throughout the platform located in the .ssh directory, one of which is world readable. On some hosts, a real user is using the ec2-user account to create accounts, groups, and pull docker files. The account is non-attributable, so we have no way of knowing who is doing this. Someone could do serious damage with no consequence. I understand that the ec2-user is needed for standing up an ec2-image, but this account should only be used for implementing IAC, so that the changes implemented by ec2-user are codified. If manual admin is required, that IAC should provision the appropriate attributable accounts, and those accounts should be used from then on. In my opinion, this is a critical finding and needs to be addressed ASAP. I will be available during the day tomorrow for any questions. Thanks Colleen _______________________________________________ platformONE mailing list platformONE at redhat.com https://www.redhat.com/mailman/listinfo/platformone -- Michael Holmes, RHCSA Senior Consultant Red Hat Remote - Texas mholmes at redhat.com M: 808-780-4877 _______________________________________________ platformONE mailing list platformONE at redhat.com https://www.redhat.com/mailman/listinfo/platformone -- Chris Kuperstein Architect Red Hat North American Public Sector ckuperst at redhat.com M: 503.616.6209 @RedHat Red Hat Red Hat [https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9ddafd5b2f1836854d7416a/Logo-RedHat-Email.png] _______________________________________________ platformONE mailing list platformONE at redhat.com https://www.redhat.com/mailman/listinfo/platformone -- KEVIN O'DONNELL ARCHITECT MANAGER Red Hat Red Hat NA Public Sector Consulting kodonnell at redhat.com M: 240-605-4654 [https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmiller at mitre.org Tue Jan 21 16:02:18 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Tue, 21 Jan 2020 16:02:18 +0000 Subject: [Platformone] Is there still a stand up, or has the team refactoring mooted this particular one? [EOM] Message-ID: From tmiller at mitre.org Thu Jan 23 20:14:52 2020 From: tmiller at mitre.org (Miller, Timothy J.) Date: Thu, 23 Jan 2020 20:14:52 +0000 Subject: [Platformone] Alternative build methods for DCAR images? Message-ID: <54E45A9D-9918-4196-B6EE-186EB1F67265@mitre.org> Probably a Q: for Taylor alone, but are there plans to support alternative image builders? Specifically, bazel? -- T? From taylor at redhat.com Thu Jan 23 20:29:39 2020 From: taylor at redhat.com (Taylor Biggs) Date: Thu, 23 Jan 2020 15:29:39 -0500 Subject: [Platformone] Alternative build methods for DCAR images? In-Reply-To: <54E45A9D-9918-4196-B6EE-186EB1F67265@mitre.org> References: <54E45A9D-9918-4196-B6EE-186EB1F67265@mitre.org> Message-ID: Currently, no plans to. Thanks, Taylor ---- Taylor Biggs taylor at redhat.com 850-449-2220 On Thu, Jan 23, 2020 at 3:15 PM Miller, Timothy J. wrote: > Probably a Q: for Taylor alone, but are there plans to support alternative > image builders? > > Specifically, bazel? > > -- T? > > > _______________________________________________ > platformONE mailing list > platformONE at redhat.com > https://www.redhat.com/mailman/listinfo/platformone > -------------- next part -------------- An HTML attachment was scrubbed... URL: From taylor at redhat.com Thu Jan 23 21:22:25 2020 From: taylor at redhat.com (Taylor Biggs) Date: Thu, 23 Jan 2020 16:22:25 -0500 Subject: [Platformone] Request for DCCSCR Upgrade Outage Message-ID: Hi Nic, Request a momentary outage (<5 minutes) in order to let the DCCSCR GitLab instance restart after monthly GitLab upgrade and OS patches at 7pm EST tonight, or tomorrow night if this isn't early enough notice. Thank you, Taylor ---- Taylor Biggs taylor at redhat.com 850-449-2220 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.m.chaillan.civ at mail.mil Fri Jan 24 00:45:44 2020 From: nicolas.m.chaillan.civ at mail.mil (Chaillan, Nicolas M HQE USAF SAF-AQ (USA)) Date: Fri, 24 Jan 2020 00:45:44 +0000 Subject: [Platformone] [Non-DoD Source] Request for DCCSCR Upgrade Outage In-Reply-To: References: Message-ID: That?s fine, sorry I missed the email so you can do it either now or tomorrow after 5PM Best regards, Nicolas M. Chaillan, HQE Chief Software Officer, Air Force Co-Lead DoD Enterprise DevSecOps Initiative Pentagon 5E741 Office: (703) 693 4740 Cell: 202-421-9845 Office of the Chief Software Officer: https://software.af.mil - usaf.cso at mail.mil From: Taylor Biggs Sent: Thursday, January 23, 2020 4:22 PM To: Chaillan, Nicolas M HQE USAF SAF-AQ (USA) Cc: platformONE at redhat.com Subject: [Non-DoD Source] Request for DCCSCR Upgrade Outage All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _____ Hi Nic, Request a momentary outage (<5 minutes) in order to let the DCCSCR GitLab instance restart after monthly GitLab upgrade and OS patches at 7pm EST tonight, or tomorrow night if this isn't early enough notice. Thank you, Taylor ---- Taylor Biggs taylor at redhat.com < Caution-mailto:taylor at redhat.com > 850-449-2220 < tel:850-449-2220 > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6710 bytes Desc: not available URL: From nicolas.m.chaillan.civ at mail.mil Fri Jan 24 00:46:11 2020 From: nicolas.m.chaillan.civ at mail.mil (Chaillan, Nicolas M HQE USAF SAF-AQ (USA)) Date: Fri, 24 Jan 2020 00:46:11 +0000 Subject: [Platformone] [Non-DoD Source] Request for DCCSCR Upgrade Outage In-Reply-To: References: Message-ID: Also, once GitLab gives us the final version of the containers, I?d like us to move to containers so we don?t have that kind of issue in the future! Best regards, Nicolas M. Chaillan, HQE Chief Software Officer, Air Force Co-Lead DoD Enterprise DevSecOps Initiative Pentagon 5E741 Office: (703) 693 4740 Cell: 202-421-9845 Office of the Chief Software Officer: https://software.af.mil - usaf.cso at mail.mil From: Taylor Biggs Sent: Thursday, January 23, 2020 4:22 PM To: Chaillan, Nicolas M HQE USAF SAF-AQ (USA) Cc: platformONE at redhat.com Subject: [Non-DoD Source] Request for DCCSCR Upgrade Outage All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. _____ Hi Nic, Request a momentary outage (<5 minutes) in order to let the DCCSCR GitLab instance restart after monthly GitLab upgrade and OS patches at 7pm EST tonight, or tomorrow night if this isn't early enough notice. Thank you, Taylor ---- Taylor Biggs taylor at redhat.com < Caution-mailto:taylor at redhat.com > 850-449-2220 < tel:850-449-2220 > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6710 bytes Desc: not available URL: From taylor at redhat.com Fri Jan 24 00:56:45 2020 From: taylor at redhat.com (Taylor Biggs) Date: Thu, 23 Jan 2020 19:56:45 -0500 Subject: [Platformone] [Non-DoD Source] Request for DCCSCR Upgrade Outage In-Reply-To: References: Message-ID: Yes sir, the upgrade is complete. Without any replies to the negative I went ahead and got it done at 7:11pET. Copy re: GitLab containers, that is absolutely the plan. Thanks, Taylor ---- Taylor Biggs taylor at redhat.com 850-449-2220 On Thu, Jan 23, 2020 at 7:46 PM Chaillan, Nicolas M HQE USAF SAF-AQ (USA) < nicolas.m.chaillan.civ at mail.mil> wrote: > Also, once GitLab gives us the final version of the containers, I?d like > us to move to containers so we don?t have that kind of issue in the future! > > > > Best regards, > > > > Nicolas M. Chaillan, HQE > > Chief Software Officer, Air Force > > Co-Lead DoD Enterprise DevSecOps Initiative > > Pentagon 5E741 > > Office: (703) 693 4740 > > Cell: 202-421-9845 > > > > Office of the Chief Software Officer: https://software.af.mil - > usaf.cso at mail.mil > > > > *From:* Taylor Biggs > *Sent:* Thursday, January 23, 2020 4:22 PM > *To:* Chaillan, Nicolas M HQE USAF SAF-AQ (USA) < > nicolas.m.chaillan.civ at mail.mil> > *Cc:* platformONE at redhat.com > *Subject:* [Non-DoD Source] Request for DCCSCR Upgrade Outage > > > > All active links contained in this email were disabled. Please verify the > identity of the sender, and confirm the authenticity of all links contained > within the message prior to copying and pasting the address to a Web > browser. > ------------------------------ > > > > Hi Nic, > > > > Request a momentary outage (<5 minutes) in order to let the DCCSCR GitLab > instance restart after monthly GitLab upgrade and OS patches at 7pm EST > tonight, or tomorrow night if this isn't early enough notice. > > > > Thank you, > Taylor > > > ---- > Taylor Biggs > taylor at redhat.com < Caution-mailto:taylor at redhat.com > > 850-449-2220 < tel:850-449-2220 <850-449-2220> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: