Access Recovery to AWS Instances if .Pem Key/File is lost
Introduction
In AWS, the Linux instances are accessed with .pem key/file, .pem is a private key for which it’s corresponding public key is stored on the server/instance. When we access the instance via SSH, authentication uses the private and public key and if these two match we receive the access to the instance.
Background
Sometimes System Admin or Developers launch instances in AWS. They used to login to Instance via .pem key but sometimes the system got corrupted or people left the organization. In such a situation we lost our .pem file too and ultimately lost access to Instance.This article will help you to understand the different ways to connect your Instance.
Points covered
- Prerequisites
- Solutions
- Summary
- References
1) Prerequisites
- Access to AWS Platform
- Write access to EC2, EBS,IAM service
2) Problem Statement
In a hypothetical scenario if you misplaced/lost the .pem key for an AWS linux instance and you would like to access your server for any purpose then can we get it ?
3) Solution the .Pem Key/File Lost Problem
We can gain access to the instance in case of missing .pem key using either of the below solutions.
A. AMI
B. Volume Exchange
C. SSM Agent
D. User-data
E. EC2 connect (Compatible with Amazon Linux 2.2 and above or ubuntu 16.04 and above)
3-A. AMI:
- Which stands for Amazon Machine Image. Through AMI, create a new AWS instance and launch it.
- Choose/select your current instance.
- Right click on it and choose option create AMI.
- Give a name to the AMI and it will create a copy of your server.
- After creation of AMI , select AMI and right click on it to select the launch option.
- Launch process will get started.
- Now we have to choose appropriate details for launching AMI as it will create a new server for us.
- Details like VPC,Subnet need to select properly from the Configure Instance option (see below screenshot for reference) and after that Add Storage.
- Choose to create a new key pair, provide a name for the key pair and launch the instance”.
- Download the key pair before we try to use that with our ssh connection.
3-B. Volume Exchange
- Suppose you have one server called primary (let’s say it is production) server to be recovered via volume exchange.
- Create a Recovery server.
- Stop the prod server and remove/detach the root volume of the prod server. (make a note of root device attachment e.g. here it is dev/xvda)
- Attach that primary server volume(root volume) as secondary to your recovery server
- Login to Recovery server and go to the volume of prod server which is attached as secondary here, find out available volumes using below command.
lsblk -f
Create new directory for the mount purpose
sudo mkdir /tmp/tmpvolume
To mount on old Amazon Linux:
sudo mount /dev/xvdf1 /tmp/tmpvolume
To mount on new Amazon Linux:
sudo mount -o nouuid /dev/xvdf1 /tmp/tmpvolume
- Copy the Recovery instance’s private key to the mounted volume of Primary Instance authorized_keys and you can see the same via cat command
cp .ssh/authorized_keys /tmp/tmpvolume/home/ec2-user/.ssh/authorized_keys
- Unmount and detach the volume that we mounted as secondary from the Recovery server.
- Command for unmount:
sudo umount /tmp/tmpvolume
For Detaching the Volume:
- Select the volume , Right click and select Detach volume.
- Click on the Yes, Detach button.
- Attach the volume back to the prod server as root volume. (e.g. dev/xvda)
- Start the prod server again.
- Try to ssh via the newly copied private key, In this case login to the Primary server with the Recovery server’s pem file.
3-C. SSM
- Create an IAM Role for the EC2 instance
- Attach the AmazonSSMFullAccess policy to the IAM Role
- Attach the role to the EC2 instance
- Wait up to 30 minutes before the SSM agent on the instance tries to reconnect Or reboot it to trigger the re-try.
3-D. User data
- To create a new key pair using AWS CLI.
aws ec2 create-key-pair — key-name my-key-pair — query “KeyMaterial” — output text > my-key-pair.pem
- If you create the private key in the Amazon EC2 console, then retrieve the public key for the key pair.
ssh-keygen -y -f /pathofpemfile/pemfile
- Open the Amazon EC2 console.
- Stop your instance.
- Choose Actions, Instance settings, Edit user data.
- Copy the following script into the Edit user data dialog box:
Below is userData File
Content-Type: multipart/mixed; boundary=”//”
MIME-Version: 1.0
— //
Content-Type: text/cloud-config; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”cloud-config.txt”
#cloud-config
cloud_final_modules:
- [users-groups, once]
users:
- name: username
ssh-authorized_keys:
- Your Public Key pair
- Replace username with your user name, such as ec2-user. You can either enter the default user name, or enter a custom user name.
- Replace PublicKeypair with the public key retrieved from the above step. Be sure to enter the entire public key, starting with ssh-rsa.
- Choose Save.
- Start your instance.
- It’s important to remove the script from the User data field since it contains a key pair.
- Stop your instance.
- Choose Actions, Instance Settings, Edit user data.
- Delete all the text in the Edit user data dialog box, and then choose Save.
- Start your instance.
3-E. Ec2- connect
Limitations
- Supported Linux distributions are supported:
- Amazon Linux 2 (any version)
- Ubuntu 16.04 or later
- You need to check permissions for your user from which you have login to system(AWS Console) — Your user should have EC2 Connect permissions.If not then you will get an error message while connecting to Ec2 via ec2 connect
- You can add permissions to your user
- Go to IAM
- Select your user
- Click on Add permissions
- Select proper permissions e.g.here- EC2InstacnceConnect
- Click on add permissions.
Once added It will look like below..
- To connect using the Amazon EC2 console (browser-based client), the instance must have a public IPv4 address.
- If the instance does not have a public IP address, you can connect to the instance using a SSH client or the EC2 Instance Connect CLI, Over a private network, Such as from a machine within the same VPC or from a machine connected to the VPC over VPN or AWS Direct Connect.
- EC2 Instance Connect does not support connection using an IPv6 address.
- Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
- In the navigation pane, choose Instances.
- Select the instance and choose Connect.
- Choose EC2 Instance Connect.
- Verify the user name and choose Connect to open a terminal window.
- Now once you get connected to the instance via the browser you can copy your system’s pub key or create a new key via ssh-keygen and save it to authorized_keys under the .ssh folder.
- Now you can connect with instance via ssh
This way you can get connected to your instance without restarting it.
4) Conclusion
Based on the article following can be the priorities for solutions to choose from first to last.
- Ec2-Connect:Will be the highest priority as in this we can avoid downtime.
- SSM:Can be chosen as the next priority as we need to install an SSM agent in this method also we can avoid downtime.
- User-Data:In this instance needs to be rebooted to apply changes to user data.
- Volume Exchange:This method will take time to complete and will require some downtime too.
- AMI: Will be the last method to follow as it will change on IP address and Instance ID.
