[Platformone] EXT :Re: [EXT] Platform1 SAR
Miller, Timothy J.
tmiller at mitre.org
Tue Jan 7 15:31:48 UTC 2020
Cloud-ready images all have a default account (in AWS it's ec2-user, root, or ubuntu depending on distro). This account has password disabled in /etc/shadow, and IIRC all cloud-ready images have sshd configured to allow *only* pubkey authN. cloud-init data is used to seed the account's authorized_keys file at first boot. You can't get rid of the cloud-ready account without creating an exactly equivalent account, unless you decide to deploy immutable (no ssh access at all).
This would potentially complicate playbooks since the `ansible_user` fact may have to change between tasks.
It's trivial to deploy additional accounts w/ appropriate authorized_keys files for each but deleting or disabling the cloud-ready default account is an unnecessary complication. I suggest that the ec2-user be retained but only the appropriate gitlab-runner hold the key. The runner can be whitelisted and any other session using that key can be easily flagged.
-- T
> -----Original Message-----
> From: Lastrilla, Jet <jlastrilla at mitre.org>
> Sent: Tuesday, January 07, 2020 8:48 AM
> To: Kevin O'Donnell <kodonnel at redhat.com>; Miller, Timothy J.
> <tmiller at mitre.org>
> Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; Feiglstok, Colleen M [US]
> (MS) <Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC
> AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF
> AFMC ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>; Bubb, Mike
> <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC
> <elijah.tramble.1 at us.af.mil>; tj.zimmerman at braingu.com;
> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP
> <richard.lopezdeuralde at us.af.mil>; RAMIREZ, JOSE A CTR USAF AFMC
> AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C.
> <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC
> AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs
> <taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San
> Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: RE: EXT :Re: [EXT] Platform1 SAR
>
> Kevin,
>
>
>
> Essentially, we shouldn’t need the keys for ec2-user. We will need to
> rename and disable. We will need to create new accounts (specific to each
> function) with the correct privilege. I can see each application having a
> unique service account that will deploy/configure then be disabled. All other
> service accounts need to be specific and attributable. This is the goal to
> move toward… no expectation of big bang to get there, just need to plan on
> when we are going to get there, and what the design is.
>
>
>
> For prod admin, we will need to define what processes need privileges and
> create specific accounts for each. We shouldn’t need a traditional break glass
> account. We should be rebuilding from known good if we cannot recover.
> The keys for the production service accounts should be held by the day 2
> team.
>
>
>
> Jet Lastrilla, CISSP
>
> MITRE | Systems Security Engineer
>
> San Antonio, TX
>
> 210-208-4867
>
> jlastrilla at mitre.org <mailto:jlastrilla at mitre.org>
>
> Jethro.lastrilla.ctr at us.af.mil <mailto:Jethro.lastrilla.ctr at us.af.mil> (NIPR)
>
> Jethro.s.lastrilla.ctr at mail.smil.mil <mailto:Jethro.s.lastrilla.ctr at mail.smil.mil>
> (SIPR)
>
> Jethro.lastrilla_ctr at af.ic.gov <mailto:Jethro.lastrilla_ctr at af.ic.gov> (JWICS)
>
>
>
> From: Kevin O'Donnell <kodonnel at redhat.com>
> Sent: Tuesday, January 7, 2020 8:26 AM
> To: Miller, Timothy J. <tmiller at mitre.org>
> Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; Lastrilla, Jet
> <jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS)
> <Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC
> AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF
> AFMC ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>; Bubb, Mike
> <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC
> <elijah.tramble.1 at us.af.mil>; tj.zimmerman at braingu.com;
> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP
> <richard.lopezdeuralde at us.af.mil>; RAMIREZ, JOSE A CTR USAF AFMC
> AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C.
> <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC
> AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs
> <taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San
> Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: Re: EXT :Re: [EXT] Platform1 SAR
>
>
>
> Hello Tim,
>
>
>
> Attached is the email that I sent out before the holiday. Please note that
> these keys are not stored in GIT for up-prod, the ec2-users authorized_keys
> file was removed and replaced with the new public key.
>
>
>
> Jet, who should hold the private key?
>
>
>
> Thanks,
>
>
>
>
> KEVIN O'DONNELL
>
> ARCHITECT MANAGER
>
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-
> 4654> M: 240-605-4654
>
> <https://red.ht/sig>
>
>
>
>
>
>
>
> On Mon, Jan 6, 2020 at 11:53 PM Miller, Timothy J. <tmiller at mitre.org
> <mailto:tmiller at mitre.org> > wrote:
>
> Just before I sent that email, I copied out the SSH privkeys from the
> CI/CD configuration for each of platform-infrastructure, up-appdev-a, and
> up-prod, dumped the moduli, and diff'd them. All three are the same. So
> that was Sunday. What did I miss?
>
> The rest of this email didn't parse, but it's been a 16h day for me, so
> maybe it's just me.
>
> The problem isn't necessarily the ec2-user named account by itself,
> it's the attribution. It's possible to establish attribution when processing logs
> since sshd will log the pubkey. However, that requires post-facto log analysis
> to track, and if there are multiple users active it can get confusing.
>
> The real problem is the current ec2-user privkey is being shared,
> intentionally and unintentionally, so there's absolutely no way to establish
> attribution.
>
> This is Yet Another Reason why you want to deploy as immutable
> infrastructure. This problem simply goes away.
>
> -- T
>
> On 1/5/20, 13:35, "Kevin O'Donnell" <kodonnel at redhat.com
> <mailto:kodonnel at redhat.com> > wrote:
>
> Hello Tim,
>
>
> Welcome back. Happy New Year.
>
>
> We have a plan in place to swap out the ec2-user for another user
> seeing as the major issue is the actual name of the user not its privs in prod.
>
>
> Incorrect up-prod-a keys were switched out before Xmas. This
> process will need to be included in the Day Two OPS onboarding process. Jet,
> who should be the holder of this key on day two? We are also ensuring that
> anyone that authenticates to the nodes
> in the vpc use dedicated IDM accounts with per user dedicated
> keys and sudo privs.
>
>
> Correct. Re-enforcing the Day Two requirements. And also
> supporting that the pipelines and CI process can support CI/CD to only
> dev/staging. For prod or any environment were the keys have been
> swapped and are not included as private Git variables we can
> not support CD into prod/appdev if so deemed.
>
>
> We are fast approaching policy with this topic. We should have a
> further discussion this week. I will be in San Antonio on Wed and Thur.
>
>
> Do you have any additional mitigation techniques Tim?
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting
> <https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> <mailto:kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> %20M:240-605-4654> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. <tmiller at mitre.org
> <mailto:tmiller at mitre.org> > wrote:
>
>
> Some more points of interest--
>
> - The ec2-user account can't currently be avoided given the current
> design of the IaC.
>
> - This SSH key provisioned for this account is shared across up-
> prod-a, up-dev-a, and DSOP.
>
> - The ec2-user SSH privkey is available to anyone with Maintainer or
> higher access to the GitLab repo that holds the customer's environment
> definitions or the platform-infrastructure repo.
>
> -- T
>
> On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" <Eric.Blade at ngc.com
> <mailto:Eric.Blade at ngc.com> > wrote:
>
> I just want to clarify on the topic of the EC2-user. The key is
> attribution. We need to know who or what is configuring the systems.
>
> We are repeatedly seeing the ec2-user account used to manually
> configure systems. Ec2-user should only be used to perform programmatic
> tasks from committed and
> reviewed code as part of initial provisioning. If the code to
> provision and configure systems is unavailable, then you must create
> attributable accounts to perform changes to the systems.
>
> Any use of ec2-user remote login or su to ec2-user will be flagged
> as a violation and the source IP or user account logged as the violating source.
> Any manual/command
> line activities MUST be performed using an attributable user
> account.
>
> If that is not clear, please let me know and I will attempt to
> elaborate further.
>
> Thank you
>
>
> Eric Blade, GXPN, GPEN
> Unified Platform System Coordinator
> Northrop Grumman Mission Systems
> Work: 410.649.0706
> Mobile: 240.258.8089
>
>
>
> From: Kevin O'Donnell <kodonnel at redhat.com
> <mailto:kodonnel at redhat.com> >
>
> Sent: Thursday, December 19, 2019 6:22 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org <mailto:jlastrilla at mitre.org>
> >
> Cc: Feiglstok, Colleen M [US] (MS) <Colleen.Feiglstok at ngc.com
> <mailto:Colleen.Feiglstok at ngc.com> >; BRYAN, AUSTEN R Capt USAF AFMC
> AFLCMC/HNCP <austen.bryan.1 at us.af.mil
> <mailto:austen.bryan.1 at us.af.mil> >;
> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP
> <roger.dirocco.4 at us.af.mil <mailto:roger.dirocco.4 at us.af.mil> >;
> platformONE at redhat.com <mailto:platformONE at redhat.com> ;
> Tim Gast <tg at braingu.com <mailto:tg at braingu.com> >;
> Bubb, Mike <mbubb at mitre.org <mailto:mbubb at mitre.org> >;
> TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC
> <elijah.tramble.1 at us.af.mil <mailto:elijah.tramble.1 at us.af.mil> >;
> tj.zimmerman at braingu.com
> <mailto:tj.zimmerman at braingu.com> ; LOPEZDEURALDE, RICHARD A Lt Col
> USAF AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil
> <mailto:richard.lopezdeuralde at us.af.mil> >; Blade, Eric
> D [US] (MS) <Eric.Blade at ngc.com <mailto:Eric.Blade at ngc.com> >;
> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP
> <jose.ramirez.50.ctr at us.af.mil <mailto:jose.ramirez.50.ctr at us.af.mil> >;
> Leonard, Michael C. <leonardm at mitre.org <mailto:leonardm at mitre.org> >;
> REINHARDT, MELISSA
> A GG-13 USAF AFMC AFLCMC/HNCP
> <melissa.reinhardt.2 at us.af.mil <mailto:melissa.reinhardt.2 at us.af.mil> >;
> Taylor Biggs <taylor at redhat.com <mailto:taylor at redhat.com> >; Miller,
> Timothy J. <tmiller at mitre.org <mailto:tmiller at mitre.org> >;
> CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil <mailto:joshua.crisp.2 at us.af.mil> >; BOGUE,
> STEVEN E CTR USAF AFMC AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil
> <mailto:steven.bogue.1.ctr at us.af.mil> >;
> Wilcox, John R. (San Antonio, TX) [US] (MS)
> <John.R.Wilcox at ngc.com <mailto:John.R.Wilcox at ngc.com> >
> Subject: EXT :Re: [EXT] Platform1 SAR
>
> Colleen,
>
>
> Thank you for the results and recommendations. We will get GIT
> issues crated for your findings and will prioritize the mitigation and
> implement them as code in our future IAC deployments. Many of the
> findings in the current VPC have been
> mitigated in up-prod-b with our current code release.
>
>
>
> Please let us know when you have finished and we can power
> down the host that you have been using for scanning.
>
>
>
>
> Note for everyone: Once we power down the ec2 instance ssh or
> port 22 will not be externally accessible. Thus, mitigating many of the risks
> associated with the ec2-user and the keys.
>
>
>
>
> Thanks,
>
>
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting
> <https://www.redhat.com/>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> <mailto:kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> %20M:240-605-4654> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet <jlastrilla at mitre.org
> <mailto:jlastrilla at mitre.org> > wrote:
>
>
> Thanks Colleen. Sorry for the rushed feeling. If you want to take
> more time, please use tomorrow to do your testing.
>
>
>
> Thank you for all you do!!!!
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
> ________________________________________
>
> From: Feiglstok, Colleen M [US] (MS)
> <Colleen.Feiglstok at ngc.com <mailto:Colleen.Feiglstok at ngc.com> >
> Sent: Thursday, December 19, 2019 4:35:15 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org <mailto:jlastrilla at mitre.org>
> >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP
> <austen.bryan.1 at us.af.mil <mailto:austen.bryan.1 at us.af.mil> >; DIROCCO,
> ROGER E
> GG-13 USAF AFMC ESC/AFLCMC/HNCP
> <roger.dirocco.4 at us.af.mil <mailto:roger.dirocco.4 at us.af.mil> >; Kevin
> O'Donnell <kodonnel at redhat.com <mailto:kodonnel at redhat.com> >;
> platformONE at redhat.com <mailto:platformONE at redhat.com>
> <platformONE at redhat.com <mailto:platformONE at redhat.com> >; Tim Gast
> <tg at braingu.com <mailto:tg at braingu.com> >; Bubb,
> Mike
> <mbubb at mitre.org <mailto:mbubb at mitre.org> >; TRAMBLE,
> ELIJAH Q Capt USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil
> <mailto:elijah.tramble.1 at us.af.mil> >;
> tj.zimmerman at braingu.com
> <mailto:tj.zimmerman at braingu.com> <tj.zimmerman at braingu.com
> <mailto:tj.zimmerman at braingu.com> >; LOPEZDEURALDE, RICHARD A Lt Col
> USAF AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil
> <mailto:richard.lopezdeuralde at us.af.mil> >;
> Blade, Eric D [US] (MS) <Eric.Blade at ngc.com
> <mailto:Eric.Blade at ngc.com> >; RAMIREZ, JOSE A CTR USAF AFMC
> AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil
> <mailto:jose.ramirez.50.ctr at us.af.mil> >; Leonard,
> Michael
> C. <leonardm at mitre.org <mailto:leonardm at mitre.org> >;
> REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP
> <melissa.reinhardt.2 at us.af.mil <mailto:melissa.reinhardt.2 at us.af.mil> >;
> Taylor Biggs <taylor at redhat.com <mailto:taylor at redhat.com> >;
> Miller, Timothy J. <tmiller at mitre.org <mailto:tmiller at mitre.org>
> >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil <mailto:joshua.crisp.2 at us.af.mil> >; BOGUE,
> STEVEN E CTR USAF
> AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil
> <mailto:steven.bogue.1.ctr at us.af.mil> >; Wilcox, John R. (San Antonio, TX)
> [US] (MS) <John.R.Wilcox at ngc.com <mailto:John.R.Wilcox at ngc.com> >
> Subject: [EXT] Platform1 SAR
>
>
>
> All,
>
> The SAR and raw results from the new security testing will be
> sent through NGSafe in a few moments.
>
> As usual, I felt very rushed with the testing, and feel like I have
> not done as thorough of a job as required. I was unable to log into the Web
> UIs, as no one from the Platform1 team gave me the account information. I
> had issues with Nessus, so the CVE’s
> were found through OSCAP this time.
>
> A lot is the same as the last report, but please read through it,
> because there is some new information. I had to test as ec2-user again,
> which is another big issue that needs to be resolved ASAP. The more I use it
> and find out
> how it is being used, the more extremely concerned I am. It has
> multiple keys throughout the platform located in the .ssh directory, one of
> which is world readable. On some hosts, a real user is using the ec2-user
> account to create accounts, groups, and
> pull
> docker files. The account is non-attributable, so we have no way
> of knowing who is doing this. Someone could do serious damage with no
> consequence. I understand that the ec2-user is needed for standing up an
> ec2-image, but
> this account should only be used for implementing IAC, so that
> the changes implemented by ec2-user are codified. If manual admin is
> required, that IAC should provision the appropriate attributable accounts,
> and those accounts should
> be used from then on. In my opinion, this is a critical finding and
> needs to be addressed ASAP.
>
>
> I will be available during the day tomorrow for any questions.
>
> Thanks
> Colleen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
More information about the platformONE
mailing list