[Platformone] Support Assistance

Fri Dec 13 01:30:17 UTC 2019

Chris,

Thank you for your response and follow up. Your detail is greatly
appreciated. Our deployments after the host issues have been resolved and
we have not faced the same issues. I appreciate your team's dedication to
addressing and resolving these issues. This is the second time that our
team has had host-related issues. How can we as a combined team supporting
our DoD customers resolve and or remediate potential issues? Should we
follow a dedicated path and or a specific coms route? I just want to make
sure we are leveraging all the paths forward.

Once again, Thank you for your time and support.

Thanks,

KEVIN O'DONNELL

ARCHITECT MANAGER

Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>

kodonnell at redhat.com <kodonnell at redhat.com%20M:240-605-4654> M: 240-605-4654
<https://red.ht/sig>

On Thu, Dec 12, 2019 at 3:26 PM Kozlowski, Chris <kozlowck at amazon.com>
wrote:

> Kevin,
>
>
>
> I’m sorry to hear that you and your team felt the case took too long to
> resolve.
>
>
>
> I’ve reviewed the case with an eye towards seeing what we could have done
> better. According to the case notes, I’ve put together a general timeline
> of what took place.
>
>
>
> ·         The case was opened the evening of the 9th.
>
> o   “In our "staging-up-vpc", the SSH from the EC2 "i-0fb3eb5a3efa04e18" (bastion-dino) is very slow to another EC2 "i-038792be35142d7f6".”
>
> ·         AWS Support replied an hour later, examining the connection
> between the two servers. Support did not see any immediate issues at the
> infrastructure level. Ed, the support technician, noted that he had limited
> visibility due to the system being on GovCloud. But he put together a list
> of troubleshooting steps for your engineer to look at on the instance. AWS
> engineers have no visibility into your buckets, volumes, or instances by
> design, for security.
>
> ·         Three hours after the above message, it’s clarified in an email
> that there’s actually no issue with SSH between instances, but just to the
> bastion host itself. A call was requested at this time, and your engineer
> states that they’d like to use a screen share.
>
> ·         Minutes later, AWS reached out to get an ITAR-compliant
> technician to work further (US citizen), as per GovCloud requirements, only
> ITAR-compliant engineers can participate in such.
>
> ·         An day later (Dec 10th), Wayne (US citizen) takes ownership of
> the case, and starts a series of remote sessions with your engineer, going
> over the SSH configuration.
>
> ·         Later on the 10th, Wayne worked with your engineer to look
> deeper at the instance, and at some point it was discovered that some of
> the instance checks were failing even though the host looked healthy. After
> reviewing the other configuration items, Wayne flagged the host for review
> by the service team. The bastion seemed to be working fine at this point as
> it launched on a different host.
>
> ·         On the 11th, your engineer reported a similar issue on a
> different instance. A non-ITAR engineer fielded the original call, then
> handed off to Wayne who took on the case again from there. He identified
> three other instances already running on the host flagged for examination
> later. The instances were relaunched on another host and the original host
> prevented from new launches.
>
>
>
> My takeaways from the case are as follows:
>
>
>
> 1.       Support originally regarded the case as a networking issue as it
> originally seemed to be an intra-EC2 communication issue. Once this was
> clarified to not be the case, it was examined as a single instance issue.
>
> 2.       It should be understood that cases for GovCloud may be initially
> responded to by non-US persons, but will be routed to US Persons where ITAR
> compliance is needed. In this case, the case was routed to a US person the
> moment your engineers asked for a screen-share. As there was no visibility
> into your data prior to that point (viewing a screenshare is viewed as
> potentially exposing us to your data), that’s where the handover occurred.
> You can read more about how GovCloud Support works here:
> https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/customer-supp.html
>
> 3.       It is strongly recommended you not place sensitive data into a
> ticket. That did not occur in this case.
>
>
>
> I thought given the sequence of events in the given context lead to a
> resolution in a feasible time, but we’re always striving to do better. I’ll
> talk to the technicians on the case about whether this could have been
> spotted sooner, but issues can arise anywhere, and it’s important we look
> at all avenues.
>
>
>
> Let me know what you think, and feel free to reach out to me further with
> any questions or concerns.
>
>
>
> Thanks!
>
>
>
>
>
>
>
> *[image: cid:image001.jpg at 01D52B59.2C1253E0]*
>
> *Chris **Kozlowski*  | Sr. Technical Account Manager
>
> AWS Enterprise Support, National Security Programs
>
> kozlowck at amazon.com | m: 703.831.5110
>
>
>
>                 Thoughts on our interaction? Provide feedback here
> <https://feedback.aws.amazon.com/?ea=kozlowck&fn=Chris&ln=Kozlowski>.
>
>
>
> *From:* Kevin O'Donnell <kodonnel at redhat.com>
> *Sent:* Wednesday, December 11, 2019 4:53 PM
> *To:* Kozlowski, Chris <kozlowck at amazon.com>
> *Cc:* Settle, Rob <settler at amazon.com>; Carta, Mike <miccarta at amazon.com>;
> platformONE at redhat.com
> *Subject:* Re: Support Assistance
>
>
>
> After speaking with my engineer I have some concerns. about how long this
> took to get a resolution from aws.
>
>
>
> "
>
> We were able to resolve the case this took approximately 3 days of
> troubleshooting before they decided to pull the physical host. From working
> with your engineer I understood that he was only able to look at govcloud
> metrics and not actually able to do any investigative work without using my
> screen. I also have a concern that when we submitted the ticket from the
> support center it is picked up by the next available tech from anywhere in
> the world and not a US-based one specifically for DoD. That technician was
> able to read any information I put in there and is a bit concerning
> considering what systems we could possibly be calling in issues about.
>
> "
>
>
>
> Please let me know how we can improve this process.
>
>
>
> *KEVIN O'DONNELL *
>
> ARCHITECT MANAGER
>
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <kodonnell at redhat.com%20M:240-605-4654> M:
> 240-605-4654
>
> <https://red.ht/sig>
>
>
>
>
>
> On Wed, Dec 11, 2019 at 4:05 PM Kevin O'Donnell <kodonnel at redhat.com>
> wrote:
>
> Hello Chris,
>
>
>
> Thank you.. As it turns out we are deploying to a new VPC now and we have
> encountered the same issue again. This new ec2 landed on the same problem
> host that we had issues with yesterday. We are working with support to move
> the ec2 to another host, how can we pull this problem host out of the mix?
>
>
>
> The support engineer asked if you would reach out to him.
>
>
>
> https://console.amazonaws-us-gov.com/support/cases?#/6655545661/en
>
>
>
>
>
> Thanks,
>
>
> *KEVIN O'DONNELL *
>
> ARCHITECT MANAGER
>
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <kodonnell at redhat.com%20M:240-605-4654> M:
> 240-605-4654
>
> <https://red.ht/sig>
>
>
>
>
>
> On Wed, Dec 11, 2019 at 11:14 AM Kozlowski, Chris <kozlowck at amazon.com>
> wrote:
>
> Kevin,
>
>
>
> No problem. I actually didn’t need to get involved this time myself;
> support is generally really responsive. Glad they were able to help you
> quickly!
>
>
>
> Getting timely support first starts with classification of the ticket.
> I’ve included a link below with the breakdown of targeted response times
> for Enterprise Support. These are the response times the support team will
> target when you submit a case. For Urgent and Critical cases, the times are
> 1 hour and 15 minutes, especially. I recommend reserving these for times
> when you have production or mission-critical systems down. In addition, if
> you submit a case at this severity level, I will be immediately paged. I
> recommend using these sparingly, but if you have one of the scenarios
> above, don’t hesitate to do so.
>
>
>
> For any other levels, if you need to escalate, feel free to reach out to
> me directly; that’s one of the roles your TAM is for. Thanks!
>
>
>
> *[image: cid:image001.jpg at 01D52B59.2C1253E0]*
>
> *Chris **Kozlowski*  | Sr. Technical Account Manager
>
> AWS Enterprise Support, National Security Programs
>
> kozlowck at amazon.com | m: 703.831.5110
>
>
>
>                 Thoughts on our interaction? Provide feedback here
> <https://feedback.aws.amazon.com/?ea=kozlowck&fn=Chris&ln=Kozlowski>.
>
>
>
> *From:* Kevin O'Donnell <kodonnel at redhat.com>
> *Sent:* Tuesday, December 10, 2019 10:51 PM
> *To:* Kozlowski, Chris <kozlowck at amazon.com>
> *Cc:* Settle, Rob <settler at amazon.com>; Carta, Mike <miccarta at amazon.com>;
> platformONE at redhat.com
> *Subject:* Re: Support Assistance
>
>
>
> Hello Chris,
>
>
>
> I am not sure if you worked anything on your side or not. But we with AWS
> support were able to identify that that issue was due to a aws node and we
> were able to resolve the issue. Thank you for your support. Who should we
> contact moving forward for escalations on support tickets? I would like to
> ensure that we are following the correct path.
>
>
>
> Thanks,
>
>
> *KEVIN O'DONNELL *
>
> ARCHITECT MANAGER
>
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <kodonnell at redhat.com%20M:240-605-4654> M:
> 240-605-4654
>
> <https://red.ht/sig>
>
>
>
>
>
> On Tue, Dec 10, 2019 at 11:32 AM Kevin O'Donnell <kodonnel at redhat.com>
> wrote:
>
> levelup-factory. Yes we did get a general response back from an individual
> from Sydney. But nothing past that.
>
>
>
> Thanks,
>
>
> *KEVIN O'DONNELL *
>
> ARCHITECT MANAGER
>
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <kodonnell at redhat.com%20M:240-605-4654> M:
> 240-605-4654
>
> <https://red.ht/sig>
>
>
>
>
>
> On Tue, Dec 10, 2019 at 11:06 AM Kozlowski, Chris <kozlowck at amazon.com>
> wrote:
>
> Kevin,
>
>
>
> Can you give me the account # that you submitted the case from?
>
>
>
> When was the case submitted? Has support reached out to you?
>
>
>
> Thanks!
>
>
>
> *[image: cid:image001.jpg at 01D52B59.2C1253E0]*
>
> *Chris **Kozlowski*  | Sr. Technical Account Manager
>
> AWS Enterprise Support, National Security Programs
>
> kozlowck at amazon.com | m: 703.831.5110
>
>
>
>                 Thoughts on our interaction? Provide feedback here
> <https://feedback.aws.amazon.com/?ea=kozlowck&fn=Chris&ln=Kozlowski>.
>
>
>
> *From:* Kevin O'Donnell <kodonnel at redhat.com>
> *Sent:* Tuesday, December 10, 2019 10:53 AM
> *To:* Settle, Rob <settler at amazon.com>; Carta, Mike <miccarta at amazon.com>;
> Kozlowski, Chris <kozlowck at amazon.com>; platformONE at redhat.com
> *Subject:* Support Assistance
>
>
>
> Hello Team.
>
>
>
> We are having some performance issues on one of our EC2 instances. We
> created a case and would like some assistance in case resolution. Please
> let us know who can assist us.
>
>
>
> https://console.amazonaws-us-gov.com/support/cases?#/6655545661/en
>
>
>
> Thanks,
>
>
> *KEVIN O'DONNELL *
>
> ARCHITECT MANAGER
>
> Red Hat Red Hat NA Public Sector Consulting <https://www.redhat.com/>
>
> kodonnell at redhat.com <kodonnell at redhat.com%20M:240-605-4654> M:
> 240-605-4654
>
> <https://red.ht/sig>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/platformone/attachments/20191212/f4a7696a/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2163 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/platformone/attachments/20191212/f4a7696a/attachment.jpg>