[Platformone] Key sharing across multiple platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR)
Chris Kuperstein
ckuperst at redhat.com
Fri Jan 17 20:53:02 UTC 2020
Hi Tim,
You have maintainer access to the repositories that store the private keys,
shouldn't you be able to run these ansible plays yourself?
On Fri, Jan 17, 2020 at 11:59 AM Miller, Timothy J. <tmiller at mitre.org>
wrote:
> Aaaand that should obviously be `hosts: all`. Testing locally, forgot to
> fix before sending. :?
>
> -- T
>
> On 1/17/20, 13:49, "Miller, Timothy J." <tmiller at mitre.org> wrote:
>
> Buglet/minor mod: find:paths: should have /root added. I'd prefer to
> just use `/`, but that might take too long.
>
> -- T
>
> On 1/17/20, 13:45, "Miller, Timothy J." <tmiller at mitre.org> wrote:
>
> Can you run this instead, then tarball the resultant
> `authorized_keys` dir and send me the tarball?
>
> ---
> - hosts: localhost
> become: true
> tasks:
>
> - name: find all authorized_keys
> find:
> paths:
> - /home
> recurse: true
> patterns: 'authorized_keys'
> register: key_files
>
> - name: write match data to a file
> copy:
> content: "{{ key_files }}"
> dest: /tmp/match_data.json
>
> - name: fetch all matched files
> fetch:
> src: "{{ item.path }}"
> dest: ./authorized_keys
> loop: "{{ key_files.files }}"
>
> - name: fetch match data
> fetch:
> src: /tmp/match_data.json
> dest: ./authorized_keys
>
>
> -- T
>
> On 1/17/20, 10:26, "Michael Holmes" <mholmes at redhat.com> wrote:
>
> Hi Tim, here is the ansible play I used.
>
>
> ---
> - hosts: dev-prod
> become: true
> tasks:
>
> - name: fetch file
> slurp:
> src: /home/ec2-user/.ssh/authorized_keys
> register: keys
>
> - debug:
> msg: "{{ keys['content'] }}"
>
>
>
> On Fri, Jan 17, 2020 at 7:24 AM Miller, Timothy J. <
> tmiller at mitre.org> wrote:
>
>
> Kevin--
>
> Can you share the ansible play / ad-hoc command used to
> collect this?
>
> Thanks.
>
> -- T
>
> On 1/16/20, 15:16, "Kevin O'Donnell" <kodonnel at redhat.com>
> wrote:
>
> All,
>
>
> Attached is the output from the authorized_keys files on
> all nodes in up-prod. Please let me know if you need any additional
> information.
>
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <
> https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jan 15, 2020 at 9:47 PM Kevin O'Donnell <
> kodonnel at redhat.com> wrote:
>
>
> Hello All,
>
>
> I know we are putting a call together for tomorrow to
> discuss. In prep for that call to clarify the current keys that are used
> for the ec2-user post-deployment, and the key var's that are in Git (Or
> NOT).
>
>
>
> "The up-prod-a keys were switched out before Xmas." And
> are NOT stored in Git.
>
>
>
> The ec2-user does not exist and is not used in our current
> pipeline or code and or in our AMI's that are used to build the VPC's. The
> ec2-user account has been removed and the local "pipeline" named account is
> used to provision the VPC's and is the only
> account on the ec2 instances.
>
>
> To recap, the ec2-user no longer exists, admins are using
> IDM accounts with private keys that are not stored in git, and the keys for
> the pipeline account and or the old ec2-user account are cycled
> post-deployment and are not stored in Git.
>
>
>
> As we move into Cloud1 we will also be deploying a series
> of tooling such as Certbot to cycle keys automatically.
>
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <
> https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jan 15, 2020 at 4:55 PM TRAMBLE, ELIJAH Q Capt
> USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil> wrote:
>
>
> +Wayne
>
> -----Original Message-----
> From: Miller, Timothy J. <tmiller at mitre.org>
> Sent: Wednesday, January 15, 2020 3:20 PM
> To: Kevin O'Donnell <kodonnel at redhat.com>
> Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>;
> Lastrilla, Jet <jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS) <
> Colleen.Feiglstok at ngc.com>;
> BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <
> austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC
> ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>; Bubb,
> Mike <mbubb at mitre.org>; TRAMBLE,
> ELIJAH
> Q Capt USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt Col
> USAF AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil>; RAMIREZ,
> JOSE
> A CTR USAF AFMC AFLCMC/HNCP <
> jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. <leonardm at mitre.org>;
> REINHARDT, MELISSA A GG-13
> USAF
> AFMC AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor
> Biggs <taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <
> joshua.crisp.2 at us.af.mil>;
> BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP <
> steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San Antonio, TX) [US]
> (MS) <John.R.Wilcox at ngc.com>
> Subject: [Non-DoD Source] Key sharing across multiple
> platform1 builds. (was: Re: EXT :Re: [EXT] Platform1 SAR)
> Importance: High
>
> I'm resurrecting this thread.
>
> I went back and inspected the IaC. Per Kevin in the
> thread below, changes *were* made to the ec2-instance role. This role now
> provisions ec2-user with a named key registered in AWS (recall that AWS
> only manages public keys, but this is all that's needed.
>
> The named key is stored in the CI/CD variables as
> DEFAULT_KEY_NAME.
>
> I inspected all the current repositories for CI/CD
> definitions, extracted all the RSA private keys I could find, noted the
> DEFAULT_KEY_NAME, if present, for each. Then I fingerprinted all the RSA
> keys and inspected the fingerprints for all keys registered
> in the CYBERCOM AWS account.
>
> TL;DR Findings:
> (Data in the raw results, below, and in the attached.)
>
> - The ec2-user SSH private key is shared across DSOP dev,
> staging, and production, *and* is shared with up-prod and up-dev.
>
> - The ec2-user SSH private key for DSOP production,
> staging, dev, up-prod, and up-dev is still being shared in GitLab CI/CD
> variables to anyone with Maintainer or higher access.
>
> - The GitLab holding these keys is publicly accessible.
>
> - There are several additional RSA private keys present in
> the CI/CD variables, but their use is unclear.
>
> Risks:
>
> - All keys contained in a GitLab CI/CD variable should be
> considered compromised. There is no attribution for these keys.
>
> - Any user who can escalate privileges in GitLab via a
> bug, error, or social engineering can exfiltrate any of the private keys
> contained in CI/CD environment variables.
>
> Recommended mitigations:
>
> - Rotate all keys in raw results in all current platform1
> instances.
>
> - Remove all exposed RSA keys from GitLab CI/CD
> configurations.
>
> - Move all deployment git repositories and CI/CD pipelines
> to a private GitLab instance.
>
> - Move all keys to a secrets manager, either under AWS
> control or deployed as a new platform1 capability.
>
> Outstanding issues:
>
> - I don't know how the gitlab-runner is being provisioned
> with the named key. If anyone can walk me through the runner architecture,
> I'd appreciate it.
>
> - I need to track down any use/reuse of the additional
> identified RSA private keys. The simple expedient is to remove them from
> GitLab CI/CD variables and see what breaks, but YMMV.
>
> Raw results:
>
> Repository DEFAULT_KEY_NAME
> Domains serviced
> ---------- ----------------
> ----------------
> up-node-infrastructure up-ss-ansible
> unified-platform.io <http://unified-platform.io> <
> http://unified-platform.io>
> up-appdev-a up-ss-ansible
> up-appdev-a.io <http://up-appdev-a.io> <http://up-appdev-a.io>
> platform-up-infrastructure-up-node n/a
> platform-infrastructure up-ss-ansible
> levelup-staging.io <http://levelup-staging.io> <
> http://levelup-staging.io>
>
> levelup-dev.io <http://levelup-dev.io> <
> http://levelup-dev.io>
>
> dsop.io <http://dsop.io> <http://dsop.io>
> platform-apps n/a
>
> Key usages
> ----------
> 45:c6:a1:91:43:2a:17:17:9d:a3:d4:b9:0d:84:11:c1:43:54:1d:e1
> up-ss-ansible
> platform-apps/ANSIBLE_PRIVATE_KEY_FILE
>
> platform-infrastructure-up-node/ANSIBLE_PRIVATE_KEY_FILE
> platform-infrastructure/ANSIBLE_PRIVATE_KEY_FILE
> up-appdev-a/ANSIBLE_PRIVATE_KEY_FILE
> up-node-infrastructure/ANSIBLE_PRIVATE_KEY_FILE
>
> d5:c2:6a:b7:5c:a8:65:ed:94:eb:1d:01:48:90:7d:5b:11:3d:e3:1d
> up-node-infrastructure/SSH_PRIVATE_KEY_FILE
> platform-apps/SSH_PRIVATE_KEY
> platform-infrastructure-up-node/CI_RUNNER_KEY
> platform-infrastructure/CI_RUNNER_KEY
> up-appdev-a/CI_RUNNER_KEY
> up-node-infrastructure/CI_RUNNER_KEY
>
> 58:47:84:db:40:4b:0a:2d:82:d2:ff:c3:69:ed:ec:0e:5f:02:82:1d
> platform-apps/GITLAB_PRIVATE_KEY
>
> e9:df:54:86:2f:60:35:19:1e:09:63:80:13:73:aa:fb:1f:54:50:65
> up-prod
> up-node-infrastructure/BASTION_KEY
>
> 69:05:7d:9e:51:f6:f7:be:fb:b6:14:bd:d0:6e:70:ee:7a:ad:de:5d
> up-node-infrastructure/SSO_CERT_KEY
>
> -- T
>
> On 1/5/20, 14:35, "Kevin O'Donnell" <kodonnel at redhat.com>
> wrote:
>
> Hello Tim,
>
>
> Welcome back. Happy New Year.
>
>
> We have a plan in place to swap out the ec2-user for
> another user seeing as the major issue is the actual name of the user not
> its privs in prod.
>
>
> Incorrect up-prod-a keys were switched out before
> Xmas. This process will need to be included in the Day Two OPS onboarding
> process. Jet, who should be the holder of this key on day two? We are also
> ensuring that anyone that authenticates to the nodes
> in the vpc use dedicated IDM accounts with per user
> dedicated keys and sudo privs.
>
>
> Correct. Re-enforcing the Day Two requirements. And
> also supporting that the pipelines and CI process can support CI/CD to only
> dev/staging. For prod or any environment were the keys have been swapped
> and are not included as private Git variables we
> can
> not support CD into prod/appdev if so deemed.
>
>
> We are fast approaching policy with this topic. We
> should have a further discussion this week. I will be in San Antonio on Wed
> and Thur.
>
>
> Do you have any additional mitigation techniques Tim?
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <
> https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. <
> tmiller at mitre.org> wrote:
>
>
> Some more points of interest--
>
> - The ec2-user account can't currently be avoided
> given the current design of the IaC.
>
> - This SSH key provisioned for this account is shared
> across up-prod-a, up-dev-a, and DSOP.
>
> - The ec2-user SSH privkey is available to anyone with
> Maintainer or higher access to the GitLab repo that holds the customer's
> environment definitions or the platform-infrastructure repo.
>
> -- T
>
> On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" <
> Eric.Blade at ngc.com> wrote:
>
> I just want to clarify on the topic of the
> EC2-user. The key is attribution. We need to know who or what is
> configuring the systems.
>
> We are repeatedly seeing the ec2-user account used
> to manually configure systems. Ec2-user should only be used to perform
> programmatic tasks from committed and
> reviewed code as part of initial provisioning.
> If the code to provision and configure systems is unavailable, then you
> must create attributable accounts to perform changes to the systems.
>
> Any use of ec2-user remote login or su to ec2-user
> will be flagged as a violation and the source IP or user account logged as
> the violating source. Any manual/command
> line activities MUST be performed using an
> attributable user account.
>
> If that is not clear, please let me know and I
> will attempt to elaborate further.
>
> Thank you
>
>
> Eric Blade, GXPN, GPEN
> Unified Platform System Coordinator
> Northrop Grumman Mission Systems
> Work: 410.649.0706
> Mobile: 240.258.8089
>
>
>
> From: Kevin O'Donnell <kodonnel at redhat.com>
>
> Sent: Thursday, December 19, 2019 6:22 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org>
> Cc: Feiglstok, Colleen M [US] (MS) <
> Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <
> austen.bryan.1 at us.af.mil>;
> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP <
> roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>;
> Bubb, Mike <mbubb at mitre.org>; TRAMBLE, ELIJAH Q
> Capt USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com; LOPEZDEURALDE, RICHARD A Lt
> Col USAF AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil>; Blade,
> Eric
> D [US] (MS) <Eric.Blade at ngc.com>;
> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <
> jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. <leonardm at mitre.org>;
> REINHARDT,
> MELISSA
> A GG-13 USAF AFMC AFLCMC/HNCP <
> melissa.reinhardt.2 at us.af.mil>; Taylor Biggs <taylor at redhat.com>; Miller,
> Timothy J. <tmiller at mitre.org>;
> CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <
> joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP <
> steven.bogue.1.ctr at us.af.mil>;
> Wilcox, John R. (San Antonio, TX) [US] (MS) <
> John.R.Wilcox at ngc.com>
> Subject: EXT :Re: [EXT] Platform1 SAR
>
> Colleen,
>
>
> Thank you for the results and recommendations. We
> will get GIT issues crated for your findings and will prioritize the
> mitigation and implement them as code in our future IAC deployments. Many
> of the findings in the current VPC have been
> mitigated in up-prod-b with our current code
> release.
>
>
>
> Please let us know when you have finished and we
> can power down the host that you have been using for scanning.
>
>
>
>
> Note for everyone: Once we power down the ec2
> instance ssh or port 22 will not be externally accessible. Thus, mitigating
> many of the risks associated with the ec2-user and the keys.
>
>
>
>
> Thanks,
>
>
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting <
> https://www.redhat.com/>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-4654>
> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet <
> jlastrilla at mitre.org> wrote:
>
>
> Thanks Colleen. Sorry for the rushed feeling. If
> you want to take more time, please use tomorrow to do your testing.
>
>
>
> Thank you for all you do!!!!
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
> ________________________________________
>
> From: Feiglstok, Colleen M [US] (MS) <
> Colleen.Feiglstok at ngc.com>
> Sent: Thursday, December 19, 2019 4:35:15 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org>; BRYAN,
> AUSTEN R Capt USAF AFMC AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO,
> ROGER
> E
> GG-13 USAF AFMC ESC/AFLCMC/HNCP <
> roger.dirocco.4 at us.af.mil>; Kevin O'Donnell <kodonnel at redhat.com>;
> platformONE at redhat.com <platformONE at redhat.com>;
> Tim Gast <tg at braingu.com>;
> Bubb,
> Mike
> <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF
> AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil>;
> tj.zimmerman at braingu.com <tj.zimmerman at braingu.com>;
> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP <
> richard.lopezdeuralde at us.af.mil>;
> Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>;
> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>;
> Leonard,
> Michael
> C. <leonardm at mitre.org>; REINHARDT, MELISSA A
> GG-13 USAF AFMC AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs
> <taylor at redhat.com>;
> Miller, Timothy J. <tmiller at mitre.org>; CRISP,
> JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <joshua.crisp.2 at us.af.mil>; BOGUE,
> STEVEN E
> CTR
> USAF
> AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>;
> Wilcox, John R. (San Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: [EXT] Platform1 SAR
>
>
>
> All,
>
> The SAR and raw results from the new security
> testing will be sent through NGSafe in a few moments.
>
> As usual, I felt very rushed with the testing, and
> feel like I have not done as thorough of a job as required. I was unable to
> log into the Web UIs, as no one from the Platform1 team gave me the account
> information. I had issues with Nessus, so
> the
> CVE’s
> were found through OSCAP this time.
>
> A lot is the same as the last report, but please
> read through it, because there is some new information. I had to test as
> ec2-user again, which is another big issue that needs to be resolved ASAP.
> The more I use it and find out
> how it is being used, the more extremely
> concerned I am. It has multiple keys throughout the platform located in the
> .ssh directory, one of which is world readable. On some hosts, a real user
> is using the ec2-user account to create accounts, groups,
> and
> pull
> docker files. The account is non-attributable, so
> we have no way of knowing who is doing this. Someone could do serious
> damage with no consequence. I understand that the ec2-user is needed for
> standing up an ec2-image, but
> this account should only be used for implementing
> IAC, so that the changes implemented by ec2-user are codified. If manual
> admin is required, that IAC should provision the appropriate attributable
> accounts, and those accounts should
> be used from then on. In my opinion, this is a
> critical finding and needs to be addressed ASAP.
>
>
> I will be available during the day tomorrow for
> any questions.
>
> Thanks
> Colleen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> platformONE mailing list
> platformONE at redhat.com
> https://www.redhat.com/mailman/listinfo/platformone
>
>
>
>
>
>
> --
> Michael Holmes, RHCSA
> Senior Consultant
> Red Hat Remote - Texas <https://www.redhat.com/>
>
> mholmes at redhat.com
> M: 808-780-4877 <tel:808-780-4877>
> <https://www.redhat.com/>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> platformONE mailing list
> platformONE at redhat.com
> https://www.redhat.com/mailman/listinfo/platformone
>
--
Chris Kuperstein
Architect
Red Hat North American Public Sector <https://www.redhat.com>
ckuperst at redhat.com
M: 503.616.6209
@RedHat <https://twitter.com/redhat> Red Hat
<https://www.linkedin.com/company/red-hat> Red Hat
<https://www.facebook.com/RedHatInc>
<https://www.redhat.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/platformone/attachments/20200117/2ed098a5/attachment.htm>
More information about the platformONE
mailing list