[Platformone] EXT :Re: [EXT] Platform1 SAR
Lastrilla, Jet
jlastrilla at mitre.org
Tue Jan 7 15:38:53 UTC 2020
Tim,
While I agree that this adds complication, this is a compliance thing, not security. It we want to fight this compliance requirement, we need to show how protected that account and keys are. We also need to clearly articulate how our mitigation’s will prevent and notify if this account is used.
This account can also not be used for day to day ops. You don’t use root accounts for that. Each machine to machine exchange should have a unique account with minimum privileges.
Outlook<https://aka.ms/qtex0l> for iOS
________________________________
From: Miller, Timothy J. <tmiller at mitre.org>
Sent: Tuesday, January 7, 2020 9:31:48 AM
To: Lastrilla, Jet <jlastrilla at mitre.org>; Kevin O'Donnell <kodonnel at redhat.com>
Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; Feiglstok, Colleen M [US] (MS) <Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>; platformONE at redhat.com <platformONE at redhat.com>; Tim Gast <tg at braingu.com>; Bubb, Mike <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil>; tj.zimmerman at braingu.com <tj.zimmerman at braingu.com>; LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil>; RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C. <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs <taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP <joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
Subject: RE: EXT :Re: [EXT] Platform1 SAR
Cloud-ready images all have a default account (in AWS it's ec2-user, root, or ubuntu depending on distro). This account has password disabled in /etc/shadow, and IIRC all cloud-ready images have sshd configured to allow *only* pubkey authN. cloud-init data is used to seed the account's authorized_keys file at first boot. You can't get rid of the cloud-ready account without creating an exactly equivalent account, unless you decide to deploy immutable (no ssh access at all).
This would potentially complicate playbooks since the `ansible_user` fact may have to change between tasks.
It's trivial to deploy additional accounts w/ appropriate authorized_keys files for each but deleting or disabling the cloud-ready default account is an unnecessary complication. I suggest that the ec2-user be retained but only the appropriate gitlab-runner hold the key. The runner can be whitelisted and any other session using that key can be easily flagged.
-- T
> -----Original Message-----
> From: Lastrilla, Jet <jlastrilla at mitre.org>
> Sent: Tuesday, January 07, 2020 8:48 AM
> To: Kevin O'Donnell <kodonnel at redhat.com>; Miller, Timothy J.
> <tmiller at mitre.org>
> Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; Feiglstok, Colleen M [US]
> (MS) <Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC
> AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF
> AFMC ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>; Bubb, Mike
> <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC
> <elijah.tramble.1 at us.af.mil>; tj.zimmerman at braingu.com;
> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP
> <richard.lopezdeuralde at us.af.mil>; RAMIREZ, JOSE A CTR USAF AFMC
> AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C.
> <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC
> AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs
> <taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San
> Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: RE: EXT :Re: [EXT] Platform1 SAR
>
> Kevin,
>
>
>
> Essentially, we shouldn’t need the keys for ec2-user. We will need to
> rename and disable. We will need to create new accounts (specific to each
> function) with the correct privilege. I can see each application having a
> unique service account that will deploy/configure then be disabled. All other
> service accounts need to be specific and attributable. This is the goal to
> move toward… no expectation of big bang to get there, just need to plan on
> when we are going to get there, and what the design is.
>
>
>
> For prod admin, we will need to define what processes need privileges and
> create specific accounts for each. We shouldn’t need a traditional break glass
> account. We should be rebuilding from known good if we cannot recover.
> The keys for the production service accounts should be held by the day 2
> team.
>
>
>
> Jet Lastrilla, CISSP
>
> MITRE | Systems Security Engineer
>
> San Antonio, TX
>
> 210-208-4867
>
> jlastrilla at mitre.org <mailto:jlastrilla at mitre.org>
>
> Jethro.lastrilla.ctr at us.af.mil <mailto:Jethro.lastrilla.ctr at us.af.mil> (NIPR)
>
> Jethro.s.lastrilla.ctr at mail.smil.mil <mailto:Jethro.s.lastrilla.ctr at mail.smil.mil>
> (SIPR)
>
> Jethro.lastrilla_ctr at af.ic.gov <mailto:Jethro.lastrilla_ctr at af.ic.gov> (JWICS)
>
>
>
> From: Kevin O'Donnell <kodonnel at redhat.com>
> Sent: Tuesday, January 7, 2020 8:26 AM
> To: Miller, Timothy J. <tmiller at mitre.org>
> Cc: Blade, Eric D [US] (MS) <Eric.Blade at ngc.com>; Lastrilla, Jet
> <jlastrilla at mitre.org>; Feiglstok, Colleen M [US] (MS)
> <Colleen.Feiglstok at ngc.com>; BRYAN, AUSTEN R Capt USAF AFMC
> AFLCMC/HNCP <austen.bryan.1 at us.af.mil>; DIROCCO, ROGER E GG-13 USAF
> AFMC ESC/AFLCMC/HNCP <roger.dirocco.4 at us.af.mil>;
> platformONE at redhat.com; Tim Gast <tg at braingu.com>; Bubb, Mike
> <mbubb at mitre.org>; TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC
> <elijah.tramble.1 at us.af.mil>; tj.zimmerman at braingu.com;
> LOPEZDEURALDE, RICHARD A Lt Col USAF AFMC AFLCMC/HNCP
> <richard.lopezdeuralde at us.af.mil>; RAMIREZ, JOSE A CTR USAF AFMC
> AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil>; Leonard, Michael C.
> <leonardm at mitre.org>; REINHARDT, MELISSA A GG-13 USAF AFMC
> AFLCMC/HNCP <melissa.reinhardt.2 at us.af.mil>; Taylor Biggs
> <taylor at redhat.com>; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil>; BOGUE, STEVEN E CTR USAF AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil>; Wilcox, John R. (San
> Antonio, TX) [US] (MS) <John.R.Wilcox at ngc.com>
> Subject: Re: EXT :Re: [EXT] Platform1 SAR
>
>
>
> Hello Tim,
>
>
>
> Attached is the email that I sent out before the holiday. Please note that
> these keys are not stored in GIT for up-prod, the ec2-users authorized_keys
> file was removed and replaced with the new public key.
>
>
>
> Jet, who should hold the private key?
>
>
>
> Thanks,
>
>
>
>
> KEVIN O'DONNELL
>
> ARCHITECT MANAGER
>
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com%20M:240-605-
> 4654> M: 240-605-4654
>
> <https://red.ht/sig>
>
>
>
>
>
>
>
> On Mon, Jan 6, 2020 at 11:53 PM Miller, Timothy J. <tmiller at mitre.org
> <mailto:tmiller at mitre.org> > wrote:
>
> Just before I sent that email, I copied out the SSH privkeys from the
> CI/CD configuration for each of platform-infrastructure, up-appdev-a, and
> up-prod, dumped the moduli, and diff'd them. All three are the same. So
> that was Sunday. What did I miss?
>
> The rest of this email didn't parse, but it's been a 16h day for me, so
> maybe it's just me.
>
> The problem isn't necessarily the ec2-user named account by itself,
> it's the attribution. It's possible to establish attribution when processing logs
> since sshd will log the pubkey. However, that requires post-facto log analysis
> to track, and if there are multiple users active it can get confusing.
>
> The real problem is the current ec2-user privkey is being shared,
> intentionally and unintentionally, so there's absolutely no way to establish
> attribution.
>
> This is Yet Another Reason why you want to deploy as immutable
> infrastructure. This problem simply goes away.
>
> -- T
>
> On 1/5/20, 13:35, "Kevin O'Donnell" <kodonnel at redhat.com
> <mailto:kodonnel at redhat.com> > wrote:
>
> Hello Tim,
>
>
> Welcome back. Happy New Year.
>
>
> We have a plan in place to swap out the ec2-user for another user
> seeing as the major issue is the actual name of the user not its privs in prod.
>
>
> Incorrect up-prod-a keys were switched out before Xmas. This
> process will need to be included in the Day Two OPS onboarding process. Jet,
> who should be the holder of this key on day two? We are also ensuring that
> anyone that authenticates to the nodes
> in the vpc use dedicated IDM accounts with per user dedicated
> keys and sudo privs.
>
>
> Correct. Re-enforcing the Day Two requirements. And also
> supporting that the pipelines and CI process can support CI/CD to only
> dev/staging. For prod or any environment were the keys have been
> swapped and are not included as private Git variables we can
> not support CD into prod/appdev if so deemed.
>
>
> We are fast approaching policy with this topic. We should have a
> further discussion this week. I will be in San Antonio on Wed and Thur.
>
>
> Do you have any additional mitigation techniques Tim?
>
>
> Thanks,
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting
> <https://www.redhat.com/>
>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> <mailto:kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> %20M:240-605-4654> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jan 5, 2020 at 2:58 PM Miller, Timothy J. <tmiller at mitre.org
> <mailto:tmiller at mitre.org> > wrote:
>
>
> Some more points of interest--
>
> - The ec2-user account can't currently be avoided given the current
> design of the IaC.
>
> - This SSH key provisioned for this account is shared across up-
> prod-a, up-dev-a, and DSOP.
>
> - The ec2-user SSH privkey is available to anyone with Maintainer or
> higher access to the GitLab repo that holds the customer's environment
> definitions or the platform-infrastructure repo.
>
> -- T
>
> On 12/20/19, 08:13, "Blade, Eric D [US] (MS)" <Eric.Blade at ngc.com
> <mailto:Eric.Blade at ngc.com> > wrote:
>
> I just want to clarify on the topic of the EC2-user. The key is
> attribution. We need to know who or what is configuring the systems.
>
> We are repeatedly seeing the ec2-user account used to manually
> configure systems. Ec2-user should only be used to perform programmatic
> tasks from committed and
> reviewed code as part of initial provisioning. If the code to
> provision and configure systems is unavailable, then you must create
> attributable accounts to perform changes to the systems.
>
> Any use of ec2-user remote login or su to ec2-user will be flagged
> as a violation and the source IP or user account logged as the violating source.
> Any manual/command
> line activities MUST be performed using an attributable user
> account.
>
> If that is not clear, please let me know and I will attempt to
> elaborate further.
>
> Thank you
>
>
> Eric Blade, GXPN, GPEN
> Unified Platform System Coordinator
> Northrop Grumman Mission Systems
> Work: 410.649.0706
> Mobile: 240.258.8089
>
>
>
> From: Kevin O'Donnell <kodonnel at redhat.com
> <mailto:kodonnel at redhat.com> >
>
> Sent: Thursday, December 19, 2019 6:22 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org <mailto:jlastrilla at mitre.org>
> >
> Cc: Feiglstok, Colleen M [US] (MS) <Colleen.Feiglstok at ngc.com
> <mailto:Colleen.Feiglstok at ngc.com> >; BRYAN, AUSTEN R Capt USAF AFMC
> AFLCMC/HNCP <austen.bryan.1 at us.af.mil
> <mailto:austen.bryan.1 at us.af.mil> >;
> DIROCCO, ROGER E GG-13 USAF AFMC ESC/AFLCMC/HNCP
> <roger.dirocco.4 at us.af.mil <mailto:roger.dirocco.4 at us.af.mil> >;
> platformONE at redhat.com <mailto:platformONE at redhat.com> ;
> Tim Gast <tg at braingu.com <mailto:tg at braingu.com> >;
> Bubb, Mike <mbubb at mitre.org <mailto:mbubb at mitre.org> >;
> TRAMBLE, ELIJAH Q Capt USAF AFMC AFLCMC/HNC
> <elijah.tramble.1 at us.af.mil <mailto:elijah.tramble.1 at us.af.mil> >;
> tj.zimmerman at braingu.com
> <mailto:tj.zimmerman at braingu.com> ; LOPEZDEURALDE, RICHARD A Lt Col
> USAF AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil
> <mailto:richard.lopezdeuralde at us.af.mil> >; Blade, Eric
> D [US] (MS) <Eric.Blade at ngc.com <mailto:Eric.Blade at ngc.com> >;
> RAMIREZ, JOSE A CTR USAF AFMC AFLCMC/HNCP
> <jose.ramirez.50.ctr at us.af.mil <mailto:jose.ramirez.50.ctr at us.af.mil> >;
> Leonard, Michael C. <leonardm at mitre.org <mailto:leonardm at mitre.org> >;
> REINHARDT, MELISSA
> A GG-13 USAF AFMC AFLCMC/HNCP
> <melissa.reinhardt.2 at us.af.mil <mailto:melissa.reinhardt.2 at us.af.mil> >;
> Taylor Biggs <taylor at redhat.com <mailto:taylor at redhat.com> >; Miller,
> Timothy J. <tmiller at mitre.org <mailto:tmiller at mitre.org> >;
> CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil <mailto:joshua.crisp.2 at us.af.mil> >; BOGUE,
> STEVEN E CTR USAF AFMC AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil
> <mailto:steven.bogue.1.ctr at us.af.mil> >;
> Wilcox, John R. (San Antonio, TX) [US] (MS)
> <John.R.Wilcox at ngc.com <mailto:John.R.Wilcox at ngc.com> >
> Subject: EXT :Re: [EXT] Platform1 SAR
>
> Colleen,
>
>
> Thank you for the results and recommendations. We will get GIT
> issues crated for your findings and will prioritize the mitigation and
> implement them as code in our future IAC deployments. Many of the
> findings in the current VPC have been
> mitigated in up-prod-b with our current code release.
>
>
>
> Please let us know when you have finished and we can power
> down the host that you have been using for scanning.
>
>
>
>
> Note for everyone: Once we power down the ec2 instance ssh or
> port 22 will not be externally accessible. Thus, mitigating many of the risks
> associated with the ec2-user and the keys.
>
>
>
>
> Thanks,
>
>
>
> KEVIN O'DONNELL
> ARCHITECT MANAGER
> Red Hat Red Hat NA Public Sector Consulting
> <https://www.redhat.com/>
> kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> <mailto:kodonnell at redhat.com <mailto:kodonnell at redhat.com>
> %20M:240-605-4654> M: 240-605-4654
> <https://red.ht/sig>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Dec 19, 2019 at 4:52 PM Lastrilla, Jet <jlastrilla at mitre.org
> <mailto:jlastrilla at mitre.org> > wrote:
>
>
> Thanks Colleen. Sorry for the rushed feeling. If you want to take
> more time, please use tomorrow to do your testing.
>
>
>
> Thank you for all you do!!!!
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
> ________________________________________
>
> From: Feiglstok, Colleen M [US] (MS)
> <Colleen.Feiglstok at ngc.com <mailto:Colleen.Feiglstok at ngc.com> >
> Sent: Thursday, December 19, 2019 4:35:15 PM
> To: Lastrilla, Jet <jlastrilla at mitre.org <mailto:jlastrilla at mitre.org>
> >; BRYAN, AUSTEN R Capt USAF AFMC AFLCMC/HNCP
> <austen.bryan.1 at us.af.mil <mailto:austen.bryan.1 at us.af.mil> >; DIROCCO,
> ROGER E
> GG-13 USAF AFMC ESC/AFLCMC/HNCP
> <roger.dirocco.4 at us.af.mil <mailto:roger.dirocco.4 at us.af.mil> >; Kevin
> O'Donnell <kodonnel at redhat.com <mailto:kodonnel at redhat.com> >;
> platformONE at redhat.com <mailto:platformONE at redhat.com>
> <platformONE at redhat.com <mailto:platformONE at redhat.com> >; Tim Gast
> <tg at braingu.com <mailto:tg at braingu.com> >; Bubb,
> Mike
> <mbubb at mitre.org <mailto:mbubb at mitre.org> >; TRAMBLE,
> ELIJAH Q Capt USAF AFMC AFLCMC/HNC <elijah.tramble.1 at us.af.mil
> <mailto:elijah.tramble.1 at us.af.mil> >;
> tj.zimmerman at braingu.com
> <mailto:tj.zimmerman at braingu.com> <tj.zimmerman at braingu.com
> <mailto:tj.zimmerman at braingu.com> >; LOPEZDEURALDE, RICHARD A Lt Col
> USAF AFMC AFLCMC/HNCP <richard.lopezdeuralde at us.af.mil
> <mailto:richard.lopezdeuralde at us.af.mil> >;
> Blade, Eric D [US] (MS) <Eric.Blade at ngc.com
> <mailto:Eric.Blade at ngc.com> >; RAMIREZ, JOSE A CTR USAF AFMC
> AFLCMC/HNCP <jose.ramirez.50.ctr at us.af.mil
> <mailto:jose.ramirez.50.ctr at us.af.mil> >; Leonard,
> Michael
> C. <leonardm at mitre.org <mailto:leonardm at mitre.org> >;
> REINHARDT, MELISSA A GG-13 USAF AFMC AFLCMC/HNCP
> <melissa.reinhardt.2 at us.af.mil <mailto:melissa.reinhardt.2 at us.af.mil> >;
> Taylor Biggs <taylor at redhat.com <mailto:taylor at redhat.com> >;
> Miller, Timothy J. <tmiller at mitre.org <mailto:tmiller at mitre.org>
> >; CRISP, JOSHUA M GS-09 USAF AFMC AFLCMC/HNCP
> <joshua.crisp.2 at us.af.mil <mailto:joshua.crisp.2 at us.af.mil> >; BOGUE,
> STEVEN E CTR USAF
> AFMC
> AFLCMC/HNCP <steven.bogue.1.ctr at us.af.mil
> <mailto:steven.bogue.1.ctr at us.af.mil> >; Wilcox, John R. (San Antonio, TX)
> [US] (MS) <John.R.Wilcox at ngc.com <mailto:John.R.Wilcox at ngc.com> >
> Subject: [EXT] Platform1 SAR
>
>
>
> All,
>
> The SAR and raw results from the new security testing will be
> sent through NGSafe in a few moments.
>
> As usual, I felt very rushed with the testing, and feel like I have
> not done as thorough of a job as required. I was unable to log into the Web
> UIs, as no one from the Platform1 team gave me the account information. I
> had issues with Nessus, so the CVE’s
> were found through OSCAP this time.
>
> A lot is the same as the last report, but please read through it,
> because there is some new information. I had to test as ec2-user again,
> which is another big issue that needs to be resolved ASAP. The more I use it
> and find out
> how it is being used, the more extremely concerned I am. It has
> multiple keys throughout the platform located in the .ssh directory, one of
> which is world readable. On some hosts, a real user is using the ec2-user
> account to create accounts, groups, and
> pull
> docker files. The account is non-attributable, so we have no way
> of knowing who is doing this. Someone could do serious damage with no
> consequence. I understand that the ec2-user is needed for standing up an
> ec2-image, but
> this account should only be used for implementing IAC, so that
> the changes implemented by ec2-user are codified. If manual admin is
> required, that IAC should provision the appropriate attributable accounts,
> and those accounts should
> be used from then on. In my opinion, this is a critical finding and
> needs to be addressed ASAP.
>
>
> I will be available during the day tomorrow for any questions.
>
> Thanks
> Colleen
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/platformone/attachments/20200107/619169d6/attachment.htm>
More information about the platformONE
mailing list