AWS Instance-Id is different than the original one

Omer Sen
Omer Sen
Jun 19 · 3 min read

While I was installing AWS CloudWatch Perl scripts (This is not agent, only bunch of perl scripts that puts Metrics to AWS CloudWatch) I have realised that it was not reporting right Instance-Id and I was getting INSUFFICIENT_DATA for alarm that I have created.

What was more interesting that when I search for that instance-id simply there was no instance-id named like that.

While trying to debug problem I have seen some CloudWatch Metrics under Linux System like

I know that CloudWatch Perl Script ( aws-scripts-mon/mon-put-instance-data.pl) was putting Metrics under Linux System NameSpace so I have found out that instead of submitting right instace-id, mon-put-instance-data.pl was simply sending i-058245505bc02e75d as Instance Id

So it was a very strange moment that how come I get a report for an Instance whose Instace-Id simply non-existent in my AWS Account. What I have done to verify this issue. I debugged this perl script using following command

This verifies that cloudwatch perl script clearly sends a non-existent instance-id.

So what I have done to debug this problem was to run perl with strace and redirected stderr to /tmp/test file

And on this /tmp/test file created I have searched for this instance-id

and i have found out what is the cause for that wrong Instance-Id. If you look the bold line above (which also shown below)

We simply read the Instance-Id from /var/tmp/aws-mon/instance-id instead of getting it via meta-data which is

After I delete this directory

/var/tmp/aws-mon/ is re-created and now I get the right Instance-Id. So if you experience something like that remember to delete /var/tmp/aws-mon/ :)

For more info about this Perl Script (provided by AWS)

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html

Also there maybe other cases which AWS Instance-Id was reported mistakenly which this directory causes that.

PS: The root cause for this wrong Instance-Id was this EBS volume was a snapshot and clone of another Instance (which was destroyed a long time ago).

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts

Omer Sen

Written by

Omer Sen

Devops, Secops, CyberSecurity are all over me

Faun

Faun

The Must-Read Publication for Aspiring Developers & DevOps Enthusiasts