[Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR)
Michael Holmes
mholmes at redhat.com
Fri Jan 17 16:25:23 UTC 2020
Hi Tim, here is the ansible play I used.
---
- hosts: dev-prod
become: true
tasks:
- name: fetch file
slurp:
src: /home/ec2-user/.ssh/authorized_keys
register: keys
- debug:
msg: "{{ keys['content'] }}"
On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. <tmiller at mitre.org>
wrote:
> Kevin--
>
> Can you share the ansible play / ad-hoc command used to collect this?
>
> Thanks.
>
> -- T
>
> On 1/16/20, 15:16, "Kevin O'Donnell" <kodonnel at redhat.com> wrote:
>
> All,
>
>
> Attached is the output from the authorized_keys files on all nodes in
> up-prod. Please let me know if you need any additional information.
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell <kodonnel at redhat.com>
> wrote:
>
>
> Hello All,
>
>
> I know we are putting a call together for tomorrow to discuss. In prep
> for that call to clarify the current keys that are used for the ec2-user
> post-deployment, and the key var's that are in Git (Or NOT).
>
>
> "The up-prod-a keys were switched out before Xmas." And are NOT stored
> in Git.
>
>
> The ec2-user does not exist and is not used in our current pipeline or
> code and or in our AMI's that are used to build the VPC's. The ec2-user
> account has been removed and the local "pipeline" named account is used to
> provision the VPC's and is the only
> account on the ec2 instances.
>
>
> To recap, the ec2-user no longer exists, admins are using IDM accounts
> with private keys that are not stored in git, and the keys for the pipeline
> account and or the old ec2-user account are cycled post-deployment and are
> not stored in Git.
>
>
> As we move into Cloud1 we will also be deploying a series of tooling
> such as Certbot to cycle keys automatically.
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC
> AFLCMC/HNC <elijah.tramble.1 at us.af.mil> wrote:
>
>
> +Wayne
>
> -----Original Message-----
> From: Miller, Timothy J. <tmiller at mitre.org>
> Sent: Wednesday, January 15, 2020 3:20 PM
> To: Kevin O'Donnell <kodonnel at redhat.com>
> Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; Lastrilla, Jet <
> jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS) <
> Colleen.Feiglstok at ngc.com>;
> BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <austen.bryan.1 at us.af.mil>;
> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP <
> roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>; Bubb, Mike <
> mbubb at mitre.org>; TRAMBLE, ELIJAH
> Q Capt USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC
> AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil>; RAMIREZ, JOSE
> A CTR USAF AFMC AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>;
> Leonard, Michael C. <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF
> AFMC AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs <
> taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <
> joshua.crisp.2 at us.af.mil>;
> BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP <
> steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San Antonio, TX) [US]
> (MS) <John.R.Wilcox at ngc.com>
> Subject: [Non-DoD Source] Key sharing across multiple platform1
> builds. (was: Re: EXT :Re: [EXT] Platform1 SAR)
> Importance: High
>
> I'm resurrecting this thread.
>
> I went back and inspected the IaC. Per Kevin in the thread below,
> changes *were* made to the ec2-instance role. This role now provisions
> ec2-user with a named key registered in AWS (recall that AWS only manages
> public keys, but this is all that's needed.
> The named key is stored in the CI/CD variables as DEFAULT_KEY_NAME.
>
> I inspected all the current repositories for CI/CD definitions,
> extracted all the RSA private keys I could find, noted the
> DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA
> keys and inspected the fingerprints for all keys registered
> in the CYBERCOM AWS account.
>
> TL;DR Findings:
> (Data in the raw results, below, and in the attached.)
>
> - The ec2-user SSH private key is shared across DSOP dev, staging, and
> production, *and* is shared with up-prod and up-dev.
>
> - The ec2-user SSH private key for DSOP production, staging, dev,
> up-prod, and up-dev is still being shared in GitLab CI/CD variables to
> anyone with Maintainer or higher access.
>
> - The GitLab holding these keys is publicly accessible.
>
> - There are several additional RSA private keys present in the CI/CD
> variables, but their use is unclear.
>
> Risks:
>
> - All keys contained in a GitLab CI/CD variable should be considered
> compromised. There is no attribution for these keys.
>
> - Any user who can escalate privileges in GitLab via a bug, error, or
> social engineering can exfiltrate any of the private keys contained in
> CI/CD environment variables.
>
> Recommended mitigations:
>
> - Rotate all keys in raw results in all current platform1 instances.
>
> - Remove all exposed RSA keys from GitLab CI/CD configurations.
>
> - Move all deployment git repositories and CI/CD pipelines to a
> private GitLab instance.
>
> - Move all keys to a secrets manager, either under AWS control or
> deployed as a new platform1 capability.
>
> Outstanding issues:
>
> - I don't know how the gitlab-runner is being provisioned with the
> named key. If anyone can walk me through the runner architecture, I'd
> appreciate it.
>
> - I need to track down any use/reuse of the additional identified RSA
> private keys. The simple expedient is to remove them from GitLab CI/CD
> variables and see what breaks, but YMMV.
>
> Raw results:
>
> Repository DEFAULT_KEY_NAME
> Domains serviced
> ---------- ----------------
> ----------------
> up-node-infrastructure up-ss-ansible
> unified-platform.io <http://unified-platform.io>
> up-appdev-a up-ss-ansible
> up-appdev-a.io <http://up-appdev-a.io>
> platform-up-infrastructure-up-node n/a
> platform-infrastructure up-ss-ansible
> levelup-staging.io <http://levelup-staging.io>
>
> levelup-dev.io <http://levelup-dev.io>
>
> dsop.io <http://dsop.io>
> platform-apps n/a
>
> Key usages
> ----------
> 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1
> up-ss-ansible
> platform-apps/ANSIBLE_PRIVATE_KEY_FILE
> platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE
> platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE
> up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE
> up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE
>
> d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d
> up-node-infrastructure/SSH_PRIVATE_KEY_FILE
> platform-apps/SSH_PRIVATE_KEY
> platform-infrastructure-up-node/CI_RUNNER_KEY
> platform-infrastructure/CI_RUNNER_KEY
> up-appdev-a/CI_RUNNER_KEY
> up-node-infrastructure/CI_RUNNER_KEY
>
> 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d
> platform-apps/GITLAB_PRIVATE_KEY
>
> e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65
> up-prod
> up-node-infrastructure/BASTION_KEY
>
> 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d
> up-node-infrastructure/SSO_CERT_KEY
>
> -- T
>
> On 1/5/20, 14:35, "Kevin O'Donnell" <kodonnel at redhat.com> wrote:
>
> Hello Tim,
>
>
> Welcome back. Happy New Year.
>
>
> We have a plan in place to swap out the ec2-user for another user
> seeing as the major issue is the actual name of the user not its privs in
> prod.
>
>
> Incorrect up-prod-a keys were switched out before Xmas. This
> process will need to be included in the Day Two OPS onboarding process.
> Jet, who should be the holder of this key on day two? We are also ensuring
> that anyone that authenticates to the nodes
> in the vpc use dedicated IDM accounts with per user dedicated
> keys and sudo privs.
>
>
> Correct. Re-enforcing the Day Two requirements. And also
> supporting that the pipelines and CI process can support CI/CD to only
> dev/staging. For prod or any environment were the keys have been swapped
> and are not included as private Git variables we can
> not support CD into prod/appdev if so deemed.
>
>
> We are fast approaching policy with this topic. We should have a
> further discussion this week. I will be in San Antonio on Wed and Thur.
>
>
> Do you have any additional mitigation techniques Tim?
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <
> https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. <
> tmiller at mitre.org> wrote:
>
>
> Some more points of interest--
>
> - The ec2-user account can't currently be avoided given the
> current design of the IaC.
>
> - This SSH key provisioned for this account is shared across
> up-prod-a, up-dev-a, and DSOP.
>
> - The ec2-user SSH privkey is available to anyone with Maintainer
> or higher access to the GitLab repo that holds the customer's environment
> definitions or the platform-infrastructure repo.
>
> -- T
>
> On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" <Eric.Blade at ngc.com>
> wrote:
>
> I just want to clarify on the topic of the EC2-user. The key
> is attribution. We need to know who or what is configuring the systems.
>
> We are repeatedly seeing the ec2-user account used to manually
> configure systems. Ec2-user should only be used to perform programmatic
> tasks from committed and
> reviewed code as part of initial provisioning. If the code
> to provision and configure systems is unavailable, then you must create
> attributable accounts to perform changes to the systems.
>
> Any use of ec2-user remote login or su to ec2-user will be
> flagged as a violation and the source IP or user account logged as the
> violating source. Any manual/command
> line activities MUST be performed using an attributable user
> account.
>
> If that is not clear, please let me know and I will attempt to
> elaborate further.
>
> Thank you
>
>
> Eric Blade, GXPN, GPEN
> Unified Platform System Coordinator
> Northrop Grumman Mission Systems
> Work: 410.649.0706
> Mobile: 240.258.8089
>
>
>
> From: Kevin O'Donnell <kodonnel at redhat.com>
>
> Sent: Thursday, December 19, 2019 6:22 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org>
> Cc: Feiglstok, Colleen M [US] (MS) <Colleen.Feiglstok at ngc.com>;
> BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <austen.bryan.1 at us.af.mil>;
> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP <
> roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>;
> Bubb, Mike <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF
> AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF
> AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil>; Blade,
> Eric
> D [US] (MS) <Eric.Blade at ngc.com>;
> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <
> jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. <leonardm at mitre.org>;
> REINHARDT,
> MELISSA
> A GG-13 USAF AFMC AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>;
> Taylor Biggs <taylor at redhat.com>; Miller, Timothy J. <tmiller at mitre.org>;
> CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <
> joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP <
> steven.bogue.1.ctr at us.af.mil>;
> Wilcox, John R. (San Antonio, TX) [US] (MS) <
> John.R.Wilcox at ngc.com>
> Subject: EXT :Re: [EXT] Platform1 SAR
>
> Colleen,
>
>
> Thank you for the results and recommendations. We will get GIT
> issues crated for your findings and will prioritize the mitigation and
> implement them as code in our future IAC deployments. Many of the findings
> in the current VPC have been
> mitigated in up-prod-b with our current code release.
>
>
>
> Please let us know when you have finished and we can power
> down the host that you have been using for scanning.
>
>
>
>
> Note for everyone: Once we power down the ec2 instance ssh or
> port 22 will not be externally accessible. Thus, mitigating many of the
> risks associated with the ec2-user and the keys.
>
>
>
>
> Thanks,
>
>
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <
> https://www.redhat.com/>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet <
> jlastrilla at mitre.org> wrote:
>
>
> Thanks Colleen. Sorry for the rushed feeling. If you want to
> take more time, please use tomorrow to do your testing.
>
>
>
> Thank you for all you do!!!!
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
> ________________________________________
>
> From: Feiglstok, Colleen M [US] (MS) <
> Colleen.Feiglstok at ngc.com>
> Sent: Thursday, December 19, 2019 4:35:15 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org>; BRYAN, AUSTEN R
> Capt USAF AFMC AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO, ROGER
> E
> GG-13 USAF AFMC ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>;
> Kevin O'Donnell <kodonnel at redhat.com>;
> platformONE at redhat.com <platformONE at redhat.com>; Tim Gast <
> tg at braingu.com>; Bubb,
> Mike
> <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC
> AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com <tj.zimmerman at braingu.com>;
> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP <
> richard.lopezdeuralde at us.af.mil>;
> Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; RAMIREZ, JOSE
> A CTR USAF AFMC AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard,
> Michael
> C. <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF
> AFMC AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs <
> taylor at redhat.com>;
> Miller, Timothy J. <tmiller at mitre.org>; CRISP, JOSHUA M
> GS-09 USAF AFMC AFLCMC/HNCP <joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E
> CTR
> USAF
> AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>; Wilcox, John R.
> (San Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: [EXT] Platform1 SAR
>
>
>
> All,
>
> The SAR and raw results from the new security testing will be
> sent through NGSafe in a few moments.
>
> As usual, I felt very rushed with the testing, and feel like I
> have not done as thorough of a job as required. I was unable to log into
> the Web UIs, as no one from the Platform1 team gave me the account
> information. I had issues with Nessus, so the
> CVE’s
> were found through OSCAP this time.
>
> A lot is the same as the last report, but please read through
> it, because there is some new information. I had to test as ec2-user again,
> which is another big issue that needs to be resolved ASAP. The more I use
> it and find out
> how it is being used, the more extremely concerned I am. It
> has multiple keys throughout the platform located in the .ssh directory,
> one of which is world readable. On some hosts, a real user is using the
> ec2-user account to create accounts, groups,
> and
> pull
> docker files. The account is non-attributable, so we have no
> way of knowing who is doing this. Someone could do serious damage with no
> consequence. I understand that the ec2-user is needed for standing up an
> ec2-image, but
> this account should only be used for implementing IAC, so that
> the changes implemented by ec2-user are codified. If manual admin is
> required, that IAC should provision the appropriate attributable accounts,
> and those accounts should
> be used from then on. In my opinion, this is a critical
> finding and needs to be addressed ASAP.
>
>
> I will be available during the day tomorrow for any questions.
>
> Thanks
> Colleen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> platformONE mailing list
> platformONE at redhat.com
> https://www.redhat.com/mailman/listinfo/platformone
>
--
Michael Holmes, RHCSA
Senior Consultant
Red Hat Remote - Texas <https://www.redhat.com/>
mholmes at redhat.com
M: 808-780-4877
<https://www.redhat.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/platformone/attachments/20200117/6ac83b5b/attachment.htm>
More information about the platformONE
mailing list