AWS::CloudFormation::Init fails to create files from S3 Bucket although the retrieval via aws "s3 cp s3" is successful - amazon-s3

Question

I was trying to retreive a file from S3 Bucket during initialization of EC2 instance.
There are no failures (Syntax). But there is something missing.
The initial_setup.sh file is not created in the root directory
There are so many articles all state the same (or at least as per my humble understanding as newbie)
Parameters:
MyVPC: { Type: String, Default: vpc-000xxx }
myNstdKeyName: { Type: String, Default: xxx-key-test }
myNstdBucket: { Type: String, Default: myBucket }
myNstdEC2HostSubnet: { Type: String, Default: subnet-0xxxxx }
myNstdImageId: { Type: 'AWS::EC2::Image::Id', Default: 'ami-0xxxx' }
Resources:
#Allow incoming SSH and all types of outgoing traffic
SSHSecGrp4Pub:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group to allow SSH connection in public subnet
VpcId: !Ref MyVPC
SecurityGroupIngress: [ { IpProtocol: tcp, FromPort: 22, ToPort: 22, CidrIp: 0.0.0.0/0 } ]
SecurityGroupEgress: [ { IpProtocol: -1, CidrIp: 0.0.0.0/0, FromPort: 1, ToPort: 65535 } ]
#Assume Role
SAPEC2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
RoleName: EC2AWSAccess
AssumeRolePolicyDocument:
Statement: [ { Effect: Allow, Principal: { Service: ec2.amazonaws.com } , Action: [ 'sts:AssumeRole' ] } ]
#Policy for the above role
S3RolePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: "S3DownloadPolicy"
Roles: [ !Ref SAPEC2Role ]
PolicyDocument:
Statement:
- Effect: Allow
Action: [ 's3:GetObject' ]
Resource: !Sub "arn:aws:s3:::${myNstdBucket}/*"
#Profile for EC2 Instance
SAPEC2Profile:
Type: AWS::IAM::InstanceProfile
Properties: { InstanceProfileName: SAPEC2Profile, Roles: [ !Ref SAPEC2Role ] }
#My EC2 Instance
EC2:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Authentication:
S3Access:
type: "S3"
roleName: { Ref: "SAPEC2Role" }
buckets: [ !Ref myNstdBucket ]
AWS::CloudFormation::Init:
config:
files:
/root/initial_setup.sh: {
source: !Sub "https://${myNstdBucket}.s3.eu-central-1.amazonaws.com/initial_setup.sh",
mode: "000777",
owner: root,
group: root,
authentication: "S3Access"
}
commands:
myStarter:
command: "/bin/bash /root/initial_setup.sh"
Properties:
SubnetId: !Ref myNstdEC2HostSubnet
ImageId: !Ref myNstdImageId
InstanceType: t2.micro
KeyName: !Ref myNstdKeyName
IamInstanceProfile: !Ref SAPEC2Profile
SecurityGroupIds: [ !Ref SSHSecGrp4Pub ]
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo "my test file" > /root/testfile.txt
After the instance is initialized and i try it out with aws cp s3://mybucket/initial_setup.sh
It works but i have to go over with dos2unix.
The alternative would be to put it in UserData. But this should also work with commands. (^)
Someone also here had almost the same situation but it was mentioned that :
"For any commands to work we need to provide a shell environment in Userdata without which it cannot create any files"
And i added it too as a last line after the security group.
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo "my test file" > /root/testfile.txt
So the /root/testfile.txt gets created with the specified text in it.
But the required file from the bucket did not show up.

Answer 1

i had a misconception that just by stating Metadata Stuff, it will get executed as well. But now i'm half way through.
This was the missing part under the UserData
UserData:
Fn::Base64: !Sub |
cd /tmp
wget https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
gzip -df aws-cfn-bootstrap-latest.tar.gz
tar -xvf aws-cfn-bootstrap-latest.tar
chmod -R 755 /tmp/aws-cfn-bootstrap-1.4
pip install --upgrade pip
pip install --upgrade setuptools
pip install awscli --ignore-installed six &> /dev/null
export PYTHONPATH=/tmp/aws-cfn-bootstrap-1.4
/tmp/aws-cfn-bootstrap-1.4/bin/cfn-init -v --stack ${AWS::StackName} --resource EC2 --region ${AWS::Region}
It is actually the last line which makes the difference. The lines before it were used to setup the configuration as the image used is a non-aws image. So it does not bring the capability out of the box.