AWS: How to regain access to an EC2 instance after losing your SSH private key
In this article, I am going to explain to you about how you can recover lost SSH Key pair for an instance if it is EBS backed and for Instance Store( Ephemeral storage) backed EC2 by using AWS System Manager (SSM).
So let’s talk about the AWS systems manager. It is an AWS service that helps you manage your EC2 and On-premise systems or VMs, at scale. You will get operational insights about the state of your infrastructure. You can do patching automation for enhanced compliance by using systems manager. It works for both Windows and Linux OS. But we need to install SSM agent onto the systems we control. Installed by default on Amazon Linux AMI & some Ubuntu AMI. Make sure the EC2 instances have a proper IAM role to allow SSM actions.
Lets Start
IAM User Policy:
To start working with Systems Manager, you will need user permissions to do so. You will either need to be an administrator or you will need to have the AmazonSSMFullAccess policy attached to your user. You also need to create a new IAM role called AmazonEC2RoleForSSM and attach it to the EC2 instance.
So there are two methods you can recover your lost SSH key pair for EBS Backed EC2 Instance as mentioned below:
- Old method if the instance is EBS backed:
First, take the snapshot of the instance then Stop the instance, detach the root volume and attach it to another instance as a data volume and mount it. Now modify the ~/.ssh/authorized_keys file with your new key. Move the volume back to the stopped instance and start the instance and you can SSH into it.
2. New method if the instance is EBS backed:
We will be using the Systems Manager AWS service interface found under Management & Governance.
Now go to Managed Instances under the Instance & nodes section on the left column here you can see all the instances that have both SSM agent and an instance profile that allows SSM access. If you just replaced or updated the Instance Profile.
In the left column, go to Actions & Change > Automation. From here we are looking for AWSSupport-ResetAccess. For that click Execute Automation button.
Now we need to give Input Parameters like Instance ID and EC2RescueinstanceType (Recommended size is T2.small). Remember that this will NOT work on instance store volumes, as it will stop and then start the instance. If you do not have an EIP, your IP address will also change.
Do not worry if a few tasks will go failed or pending as the process includes both Windows and Linux steps verification. So if you are running for Windows instance then Linux tasks will go failed because those are not needed to verify and vice versa.
It’s done. Now we can go to Parameter Store and it will be listed as /ec2rl/openssh/{instanceid}/key. Now you can use this new key to connect to your instance.
Now, this new method is using SSM. And if the instance again is only EBS backed, not instance store-backed, so only EBS backed, then you can run this automation called AWS support reset access, and that will basically create something called EC2 rescue and run some tasks on your Windows or Linux machines to basically restore your admin and SSH key pair.
Recovering SSH key for Instance stored backed EC2.
And then finally if your instance, EC2 instance is Instance stored backed, you cannot stop it because if you stop it you lose all data on it so in that case, AWS says you cannot recover your SSH key and you cannot do anything and AWS recommends you to terminate your instance and create a new one with a new SSH key that you have access to.
So here’s a tip. You can use Session Manager to access and get a shell through SSM and then from that shell you can edit that file ~/.ssh/authorized_keys but you need to create a new SSH key I hope you know how to create that. And you can take login back to that instance store. It also works if your instance is EBS backed.
Step 1: Create a new EC2 instance with a new SSH key pair and login to a new EC2 instance with the new key. Now go to same file location ~/.ssh/authorized_keys and copy the key text in the notepad as shown below:
Step 2: Go to Session Manager in AWS systems manager. Now here you can see your all EC2 Instances which have IAM role AmazonEC2RoleForSSM attached to it.
Note: Remember if the IAM role is not attached to the instance then that instance will not be listed in the Session Manager.
Step 3: Now select your Lost EC2 instance ssh key and take direct login in it. And open ~/.ssh/authorized_keys file and form notepad copy and paste the new SSH key pair text in ~/.ssh/authorized_keys file.
Done. Now you can log in using putty or Linux terminal with the new SSH key pair.
Conclusion:
I hope this guide will help you to Recreate a lost SSH key and to use the new SSH key to log in your old running instance for both EBS backed EC2 and Instance store backed EC2 instance using AWS Systems Manager. Look out for our next post as we share some of our learnings.
