I ran into a weird issue recently where a single AWS EC2 instance was failing to work properly with it’s IAM role for EC2. Whilst the role allowed access to DescribeInstances action, awscli would repeatedly claim it didn’t have permission to perform that action.
For a moment I was thinking there was some bug/issue with my box and was readying the terminate instance button, but decided to check out the –debug output to see if it was actually loading the IAM role for EC2 or not.
$ aws ec2 describe-instances --instance-ids i-hiandre--region 'ap-southeast-2' --debug 2015-11-16 11:53:20,576 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env 2015-11-16 11:53:20,576 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role 2015-11-16 11:53:20,576 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file 2015-11-16 11:53:20,576 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file 2015-11-16 11:53:20,576 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file 2015-11-16 11:53:20,576 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config 2015-11-16 11:53:20,576 - MainThread - botocore.credentials - INFO - Found credentials in boto config file: /etc/boto.cfg
Turns out in my case, there was a /etc/boto.cfg file with a different set of credentials – and since Boto takes preference of on disk configuration over IAM Roles for EC2, it resulted in the wrong access credentials being used.
The –debug param is a handy one to remember if you’re having weird permissions issue and want to check exactly what credentials from where are being used.
