[Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR)
Kevin O'Donnell
kodonnel at redhat.com
Thu Jan 16 04:47:41 UTC 2020
Hello All,
I know we are putting a call together for tomorrow to discuss. In prep for
that call to clarify the current keys that are used for the ec2-user
post-deployment, and the key var's that are in Git (Or NOT).
"The up-prod-a keys were switched out before Xmas." And are NOT stored in
Git.
The ec2-user does not exist and is not used in our current pipeline or code
and or in our AMI's that are used to build the VPC's. The ec2-user account
has been removed and the local "pipeline" named account is used to
provision the VPC's and is the only account on the ec2 instances.
To recap, the ec2-user no longer exists, admins are using IDM accounts with
private keys that are not stored in git, and the keys for the pipeline
account and or the old ec2-user account are cycled post-deployment and are
not stored in Git.
As we move into Cloud1 we will also be deploying a series of tooling such
as Certbot to cycle keys automatically.
Thanks,
KEVIN O'DONNELL
ARCHITECT MANAGER
Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
kodonnell at redhat.com <kodonnell at redhat.com%20M:240-605-4654> M: 240-605-4654
<https://red.ht/sig>
On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC
<elijah.tramble.1 at us.af.mil> wrote:
> +Wayne
>
> -----Original Message-----
> From: Miller, Timothy J. <tmiller at mitre.org>
> Sent: Wednesday, January 15, 2020 3:20 PM
> To: Kevin O'Donnell <kodonnel at redhat.com>
> Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; Lastrilla, Jet <
> jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS) <
> Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <
> austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC
> ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>; platformONE at redhat.com; Tim
> Gast <tg at braingu.com>; Bubb, Mike <mbubb at mitre.org>; TRAMBLE, ELIJAH Q
> Capt USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC
> AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil>; RAMIREZ, JOSE A CTR USAF
> AFMC AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. <
> leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP <
> melissa.reinhardt.2 at us.af.mil>; Taylor Biggs <taylor at redhat.com>; CRISP,
> JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <joshua.crisp.2 at us.af.mil>; BOGUE,
> STEVEN E CTR USAF AFMC AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>;
> Wilcox, John R. (San Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: [Non-DoD Source] Key sharing across multiple platform1 builds.
> (was: Re: EXT :Re: [EXT] Platform1 SAR)
> Importance: High
>
> I'm resurrecting this thread.
>
> I went back and inspected the IaC. Per Kevin in the thread below, changes
> *were* made to the ec2-instance role. This role now provisions ec2-user
> with a named key registered in AWS (recall that AWS only manages public
> keys, but this is all that's needed. The named key is stored in the CI/CD
> variables as DEFAULT_KEY_NAME.
>
> I inspected all the current repositories for CI/CD definitions, extracted
> all the RSA private keys I could find, noted the DEFAULT_KEY_NAME, if
> present, for each. Then I fingerprinted all the RSA keys and inspected the
> fingerprints for all keys registered in the CYBERCOM AWS account.
>
> TL;DR Findings:
> (Data in the raw results, below, and in the attached.)
>
> - The ec2-user SSH private key is shared across DSOP dev, staging, and
> production, *and* is shared with up-prod and up-dev.
>
> - The ec2-user SSH private key for DSOP production, staging, dev, up-prod,
> and up-dev is still being shared in GitLab CI/CD variables to anyone with
> Maintainer or higher access.
>
> - The GitLab holding these keys is publicly accessible.
>
> - There are several additional RSA private keys present in the CI/CD
> variables, but their use is unclear.
>
> Risks:
>
> - All keys contained in a GitLab CI/CD variable should be considered
> compromised. There is no attribution for these keys.
>
> - Any user who can escalate privileges in GitLab via a bug, error, or
> social engineering can exfiltrate any of the private keys contained in
> CI/CD environment variables.
>
> Recommended mitigations:
>
> - Rotate all keys in raw results in all current platform1 instances.
>
> - Remove all exposed RSA keys from GitLab CI/CD configurations.
>
> - Move all deployment git repositories and CI/CD pipelines to a private
> GitLab instance.
>
> - Move all keys to a secrets manager, either under AWS control or deployed
> as a new platform1 capability.
>
> Outstanding issues:
>
> - I don't know how the gitlab-runner is being provisioned with the named
> key. If anyone can walk me through the runner architecture, I'd appreciate
> it.
>
> - I need to track down any use/reuse of the additional identified RSA
> private keys. The simple expedient is to remove them from GitLab CI/CD
> variables and see what breaks, but YMMV.
>
> Raw results:
>
> Repository DEFAULT_KEY_NAME Domains
> serviced
> ---------- ----------------
> ----------------
> up-node-infrastructure up-ss-ansible
> unified-platform.io
> up-appdev-a up-ss-ansible
> up-appdev-a.io
> platform-up-infrastructure-up-node n/a
> platform-infrastructure up-ss-ansible
> levelup-staging.io
>
> levelup-dev.io
> dsop.io
> platform-apps n/a
>
> Key usages
> ----------
> 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1
> up-ss-ansible
> platform-apps/ANSIBLE_PRIVATE_KEY_FILE
> platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE
> platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE
> up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE
> up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE
>
> d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d
> up-node-infrastructure/SSH_PRIVATE_KEY_FILE
> platform-apps/SSH_PRIVATE_KEY
> platform-infrastructure-up-node/CI_RUNNER_KEY
> platform-infrastructure/CI_RUNNER_KEY
> up-appdev-a/CI_RUNNER_KEY
> up-node-infrastructure/CI_RUNNER_KEY
>
> 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d
> platform-apps/GITLAB_PRIVATE_KEY
>
> e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65
> up-prod
> up-node-infrastructure/BASTION_KEY
>
> 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d
> up-node-infrastructure/SSO_CERT_KEY
>
> -- T
>
> On 1/5/20, 14:35, "Kevin O'Donnell" <kodonnel at redhat.com> wrote:
>
> Hello Tim,
>
>
> Welcome back. Happy New Year.
>
>
> We have a plan in place to swap out the ec2-user for another user
> seeing as the major issue is the actual name of the user not its privs in
> prod.
>
>
> Incorrect up-prod-a keys were switched out before Xmas. This process
> will need to be included in the Day Two OPS onboarding process. Jet, who
> should be the holder of this key on day two? We are also ensuring that
> anyone that authenticates to the nodes
> in the vpc use dedicated IDM accounts with per user dedicated keys
> and sudo privs.
>
>
> Correct. Re-enforcing the Day Two requirements. And also supporting
> that the pipelines and CI process can support CI/CD to only dev/staging.
> For prod or any environment were the keys have been swapped and are not
> included as private Git variables we can
> not support CD into prod/appdev if so deemed.
>
>
> We are fast approaching policy with this topic. We should have a
> further discussion this week. I will be in San Antonio on Wed and Thur.
>
>
> Do you have any additional mitigation techniques Tim?
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. <tmiller at mitre.org>
> wrote:
>
>
> Some more points of interest--
>
> - The ec2-user account can't currently be avoided given the current
> design of the IaC.
>
> - This SSH key provisioned for this account is shared across
> up-prod-a, up-dev-a, and DSOP.
>
> - The ec2-user SSH privkey is available to anyone with Maintainer or
> higher access to the GitLab repo that holds the customer's environment
> definitions or the platform-infrastructure repo.
>
> -- T
>
> On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" <Eric.Blade at ngc.com>
> wrote:
>
> I just want to clarify on the topic of the EC2-user. The key is
> attribution. We need to know who or what is configuring the systems.
>
> We are repeatedly seeing the ec2-user account used to manually
> configure systems. Ec2-user should only be used to perform programmatic
> tasks from committed and
> reviewed code as part of initial provisioning. If the code to
> provision and configure systems is unavailable, then you must create
> attributable accounts to perform changes to the systems.
>
> Any use of ec2-user remote login or su to ec2-user will be flagged
> as a violation and the source IP or user account logged as the violating
> source. Any manual/command
> line activities MUST be performed using an attributable user
> account.
>
> If that is not clear, please let me know and I will attempt to
> elaborate further.
>
> Thank you
>
>
> Eric Blade, GXPN, GPEN
> Unified Platform System Coordinator
> Northrop Grumman Mission Systems
> Work: 410.649.0706
> Mobile: 240.258.8089
>
>
>
> From: Kevin O'Donnell <kodonnel at redhat.com>
>
> Sent: Thursday, December 19, 2019 6:22 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org>
> Cc: Feiglstok, Colleen M [US] (MS) <Colleen.Feiglstok at ngc.com>;
> BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <austen.bryan.1 at us.af.mil>;
> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP <
> roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>;
> Bubb, Mike <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC
> AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC
> AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil>; Blade, Eric
> D [US] (MS) <Eric.Blade at ngc.com>;
> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <
> jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. <leonardm at mitre.org>;
> REINHARDT, MELISSA
> A GG-13 USAF AFMC AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>;
> Taylor Biggs <taylor at redhat.com>; Miller, Timothy J. <tmiller at mitre.org>;
> CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <
> joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP <
> steven.bogue.1.ctr at us.af.mil>;
> Wilcox, John R. (San Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: EXT :Re: [EXT] Platform1 SAR
>
> Colleen,
>
>
> Thank you for the results and recommendations. We will get GIT
> issues crated for your findings and will prioritize the mitigation and
> implement them as code in our future IAC deployments. Many of the findings
> in the current VPC have been
> mitigated in up-prod-b with our current code release.
>
>
>
> Please let us know when you have finished and we can power down
> the host that you have been using for scanning.
>
>
>
>
> Note for everyone: Once we power down the ec2 instance ssh or port
> 22 will not be externally accessible. Thus, mitigating many of the risks
> associated with the ec2-user and the keys.
>
>
>
>
> Thanks,
>
>
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <
> https://www.redhat.com/>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet <
> jlastrilla at mitre.org> wrote:
>
>
> Thanks Colleen. Sorry for the rushed feeling. If you want to take
> more time, please use tomorrow to do your testing.
>
>
>
> Thank you for all you do!!!!
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
> ________________________________________
>
> From: Feiglstok, Colleen M [US] (MS) <Colleen.Feiglstok at ngc.com>
> Sent: Thursday, December 19, 2019 4:35:15 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org>; BRYAN, AUSTEN R Capt
> USAF AFMC AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E
> GG-13 USAF AFMC ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>;
> Kevin O'Donnell <kodonnel at redhat.com>;
> platformONE at redhat.com <platformONE at redhat.com>; Tim Gast <
> tg at braingu.com>; Bubb,
> Mike
> <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC <
> elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com <tj.zimmerman at braingu.com>;
> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP <
> richard.lopezdeuralde at us.af.mil>;
> Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; RAMIREZ, JOSE A
> CTR USAF AFMC AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard,
> Michael
> C. <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC
> AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs <
> taylor at redhat.com>;
> Miller, Timothy J. <tmiller at mitre.org>; CRISP, JOSHUA M GS-09
> USAF AFMC AFLCMC/HNCP <joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF
> AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San
> Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: [EXT] Platform1 SAR
>
>
>
> All,
>
> The SAR and raw results from the new security testing will be sent
> through NGSafe in a few moments.
>
> As usual, I felt very rushed with the testing, and feel like I
> have not done as thorough of a job as required. I was unable to log into
> the Web UIs, as no one from the Platform1 team gave me the account
> information. I had issues with Nessus, so the CVE’s
> were found through OSCAP this time.
>
> A lot is the same as the last report, but please read through it,
> because there is some new information. I had to test as ec2-user again,
> which is another big issue that needs to be resolved ASAP. The more I use
> it and find out
> how it is being used, the more extremely concerned I am. It has
> multiple keys throughout the platform located in the .ssh directory, one of
> which is world readable. On some hosts, a real user is using the ec2-user
> account to create accounts, groups, and
> pull
> docker files. The account is non-attributable, so we have no way
> of knowing who is doing this. Someone could do serious damage with no
> consequence. I understand that the ec2-user is needed for standing up an
> ec2-image, but
> this account should only be used for implementing IAC, so that the
> changes implemented by ec2-user are codified. If manual admin is required,
> that IAC should provision the appropriate attributable accounts, and those
> accounts should
> be used from then on. In my opinion, this is a critical finding
> and needs to be addressed ASAP.
>
>
> I will be available during the day tomorrow for any questions.
>
> Thanks
> Colleen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/platformone/attachments/20200115/fb27bd9b/attachment.htm>
More information about the platformONE
mailing list