Unable to deploy application on EC2 instance using AWS CloudFormation template through cfn-init and UserData script - amazon-s3

I am trying to deploy sample.war application on EC2 instance at the time of launch. That is when an instance is launched the application should be deployed automatically on it using cfn-init and Metadata. I added a user with policy and authentication with no luck. If I wget with the S3 path, the file is being downloaded. Below is my script. What am I missing in this, or is there any other way to do this?
---
AWSTemplateFormatVersion: 2010-09-09
Description: Test QA Template
Resources:
MyInstance:
Type: AWS::EC2::Instance
Properties:
ImageId: !Ref AMIIdParam
InstanceType: !Ref InstanceType
Metadata:
AWS::CloudFormation::Init:
config:
packages:
yum:
java-1.8.0-openjdk.x86_64: []
tomcat: []
httpd.x86_64: []
services:
sysvinit:
httpd:
enabled: true
ensureRunning: true
files:
/usr/share/tomcat/webapps/sample.zip:
source: https://s3.amazonaws.com/mybucket/sample.zip
mode: '000500'
owner: tomcat
group: tomcat
authentication: S3AccessCreds
AWS::CloudFormation::Authentication:
S3AccessCreds:
type: 'S3'
accessKeyId: !Ref HostKeys
secretKey: Fn::GetAtt:
- HostKeys
- SecretAccessKey
buckets: !Ref BucketName
CfnUser:
Type: AWS::IAM::User
Properties:
Path: '/'
Policies:
- PolicyName: 'S3Access'
PolicyDocument:
Statement:
- Effect: 'Allow'
Action: s3:*
Resource: '*'
HostKeys:
Type: AWS::IAM::AccessKey
Properties:
UserName: !Ref CfnUser

I was unable to reproduce this using the following template:
---
AWSTemplateFormatVersion: 2010-09-09
Description: Test QA Template
Resources:
MyInstance:
Type: AWS::EC2::Instance
Properties:
ImageId: ami-08589eca6dcc9b39c
InstanceType: t2.micro
KeyName: default
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
/opt/aws/bin/cfn-init -s ${AWS::StackId} --resource MyInstance --region ${AWS::Region}
Metadata:
AWS::CloudFormation::Init:
config:
packages:
yum:
java-1.8.0-openjdk.x86_64: []
tomcat: []
httpd.x86_64: []
services:
sysvinit:
httpd:
enabled: true
ensureRunning: true
files:
/usr/share/tomcat/webapps/sample.zip:
source: https://s3.amazonaws.com/mybucket/sample.zip
mode: '000500'
owner: tomcat
group: tomcat
(In other words, use of the above template allowed me to install a sample.zip file using cfn-init.)
Thus there is something permissions-related in the way you're accessing the S3 bucket.
Suffice to say it is a bad practice to use Access Keys. Have a look at this document on best practices of assigning an IAM Role to an EC2 instance and then adding a Bucket Policy that grants appropriate access to that Role.

Related

"X-Cache: Miss from cloudfront" as a result of a call to AWS API Gateway

When I send a GET request to AWS APIGateway's URL "https://blablabla.execute-api.us-east-1.amazonaws.com/dev/crs/blablabla.png" or Custom Domain's URL "devblablabla.bla.com" via browser or POSTMAN I receive a 200 response with the "X-Cache: Miss from cloudfront" header:
GET request to AWS APIGateway
Do you have any idea how I can rewrite the serverless.yml file for receiving 200 response with the "X-cache:HIT" header?
This is the configuration that I deploy:
# serverless.yml
service: s3-blablabla-service
provider:
name: aws
stage: dev
region: us-east-1
environment:
SERVICE_NAME: ${self:service}
apiGateway:
binaryMediaTypes: "*/*"
plugins:
- serverless-apigateway-service-proxy
- serverless-domain-manager
- serverless-finch
custom:
c3launchBucketName: "blabla-pl-${self:provider.stage}"
c3scormBucketName: "blabla-crs-${self:provider.stage}"
domainName: "${self:provider.stage}blablabla.bla.com" # Change this to your domain.
basePath: "" # This will be prefixed to all routes
apiGatewayServiceProxies:
- s3:
path: /pl/{myKey+} # use path param
method: get
action: GetObject
bucket:
# ${self:custom.c3launchBucketName}
Ref: S3Bucket
key:
pathParam: myKey
requestParameters:
"integration.request.header.cache-control": "'public, max-age=31536000, immutable'"
- s3:
path: /crs/{myKey+} # use path param
method: get
action: GetObject
bucket:
# ${self:custom.c3scormBucketName}
Ref: S3ScormBucket
key:
pathParam: myKey
requestParameters:
"integration.request.header.cache-control": "'public, max-age=31536000, immutable'"
customDomain:
domainName: ${self:custom.domainName}
basePath: ${self:custom.basePath}
stage: ${self:provider.stage}
createRoute53Record: true
autoDomain: true
client:
bucketName: ${self:custom.c3launchBucketName}
resources:
Resources:
S3Bucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: ${self:custom.c3launchBucketName}
S3ScormBucket:
Type: 'AWS::S3::Bucket'
Properties:
BucketName: ${self:custom.c3scormBucketName}
After the deployment I receive this result:
endpoints:
GET - https://blablabla.execute-api.us-east-1.amazonaws.com/dev/pl/{myKey+}
GET - https://blablabla.execute-api.us-east-1.amazonaws.com/dev/crs/{myKey+}
Service deployed to stack s3-blablabla-service-dev
Serverless Domain Manager:
Domain Name: devblablabla.bla.com
Target Domain: abrakadabra.cloudfront.net
Hosted Zone Id: BARBARBAR

How to refer the environment variable value for s3 bucket name in sam template?

I am trying to build s3 event application by referring the environment variable defined.
Below the template i am referring
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Sub "${S3}"
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: ""How can refer the s3 bucket form the environment here"
Events: s3:ObjectCreated:*
i actually need to refer the 'BUCKET NAME' from the 'Environment variable' and also need to add event trigger from the same s3.
can anyone help on this?
Thanks
There are two options,
Either you have defined the S3 bucket in the same template, and then you can reference it using the logical CFN ID:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-name
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Ref S3Bucket
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref S3Bucket
Events: s3:ObjectCreated:*
Or you do not have it defined in the same template, in which case you have to pass it to the stack as a parameter during deployment like this:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
S3Bucket:
Description: Name of the S3 Bucket
Type: String
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Ref S3Bucket
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref S3Bucket
Events: s3:ObjectCreated:*

AWS::CloudFormation::Init fails to create files from S3 Bucket although the retrieval via aws "s3 cp s3" is successful

I was trying to retreive a file from S3 Bucket during initialization of EC2 instance.
There are no failures (Syntax). But there is something missing.
The initial_setup.sh file is not created in the root directory
There are so many articles all state the same (or at least as per my humble understanding as newbie)
Parameters:
MyVPC: { Type: String, Default: vpc-000xxx }
myNstdKeyName: { Type: String, Default: xxx-key-test }
myNstdBucket: { Type: String, Default: myBucket }
myNstdEC2HostSubnet: { Type: String, Default: subnet-0xxxxx }
myNstdImageId: { Type: 'AWS::EC2::Image::Id', Default: 'ami-0xxxx' }
Resources:
#Allow incoming SSH and all types of outgoing traffic
SSHSecGrp4Pub:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group to allow SSH connection in public subnet
VpcId: !Ref MyVPC
SecurityGroupIngress: [ { IpProtocol: tcp, FromPort: 22, ToPort: 22, CidrIp: 0.0.0.0/0 } ]
SecurityGroupEgress: [ { IpProtocol: -1, CidrIp: 0.0.0.0/0, FromPort: 1, ToPort: 65535 } ]
#Assume Role
SAPEC2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
RoleName: EC2AWSAccess
AssumeRolePolicyDocument:
Statement: [ { Effect: Allow, Principal: { Service: ec2.amazonaws.com } , Action: [ 'sts:AssumeRole' ] } ]
#Policy for the above role
S3RolePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: "S3DownloadPolicy"
Roles: [ !Ref SAPEC2Role ]
PolicyDocument:
Statement:
- Effect: Allow
Action: [ 's3:GetObject' ]
Resource: !Sub "arn:aws:s3:::${myNstdBucket}/*"
#Profile for EC2 Instance
SAPEC2Profile:
Type: AWS::IAM::InstanceProfile
Properties: { InstanceProfileName: SAPEC2Profile, Roles: [ !Ref SAPEC2Role ] }
#My EC2 Instance
EC2:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Authentication:
S3Access:
type: "S3"
roleName: { Ref: "SAPEC2Role" }
buckets: [ !Ref myNstdBucket ]
AWS::CloudFormation::Init:
config:
files:
/root/initial_setup.sh: {
source: !Sub "https://${myNstdBucket}.s3.eu-central-1.amazonaws.com/initial_setup.sh",
mode: "000777",
owner: root,
group: root,
authentication: "S3Access"
}
commands:
myStarter:
command: "/bin/bash /root/initial_setup.sh"
Properties:
SubnetId: !Ref myNstdEC2HostSubnet
ImageId: !Ref myNstdImageId
InstanceType: t2.micro
KeyName: !Ref myNstdKeyName
IamInstanceProfile: !Ref SAPEC2Profile
SecurityGroupIds: [ !Ref SSHSecGrp4Pub ]
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo "my test file" > /root/testfile.txt
After the instance is initialized and i try it out with aws cp s3://mybucket/initial_setup.sh
It works but i have to go over with dos2unix.
The alternative would be to put it in UserData. But this should also work with commands. (^)
Someone also here had almost the same situation but it was mentioned that :
"For any commands to work we need to provide a shell environment in Userdata without which it cannot create any files"
And i added it too as a last line after the security group.
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo "my test file" > /root/testfile.txt
So the /root/testfile.txt gets created with the specified text in it.
But the required file from the bucket did not show up.
i had a misconception that just by stating Metadata Stuff, it will get executed as well. But now i'm half way through.
This was the missing part under the UserData
UserData:
Fn::Base64: !Sub |
cd /tmp
wget https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz
gzip -df aws-cfn-bootstrap-latest.tar.gz
tar -xvf aws-cfn-bootstrap-latest.tar
chmod -R 755 /tmp/aws-cfn-bootstrap-1.4
pip install --upgrade pip
pip install --upgrade setuptools
pip install awscli --ignore-installed six &> /dev/null
export PYTHONPATH=/tmp/aws-cfn-bootstrap-1.4
/tmp/aws-cfn-bootstrap-1.4/bin/cfn-init -v --stack ${AWS::StackName} --resource EC2 --region ${AWS::Region}
It is actually the last line which makes the difference. The lines before it were used to setup the configuration as the image used is a non-aws image. So it does not bring the capability out of the box.

Serverless Enterprise deployment

I recently updated to v1.44.0 and used the #serverless/enterprise-plugin and am now unable to deploy. I’m simply trying to create a User Pool, but keep getting an error.
An error occurred: EnterpriseLogAccessIamRole - Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: dc158686-378c-4d01-97fb-1414d55a735d)
serverless.yml
tenant: [omitted]
app: [omitted]
service: auth
frameworkVersion: ">=1.44.0"
plugins:
- '#serverless/enterprise-plugin'
provider:
name: aws
runtime: nodejs8.10
region: us-east-1
custom:
stage: ${opt:stage, self:provider.stage}
cognito:
app:
userPool: ${self:service}-app-user-pool-${self:custom.stage}
identityPool: AppIdentityPoolDev
resources:
Resources:
AppUserPool:
Type: AWS::Cognito::UserPool
Properties:
UserPoolName: ${self:custom.cognito.app.userPool}
UsernameAttributes:
- email
AutoVerifiedAttributes:
- email
MobileAppClient:
Type: AWS::Cognito::UserPoolClient
Properties:
ClientName: ${self:service}-mobile-app-client-${self:custom.stage}
UserPoolId:
Ref: AppUserPool
GenerateSecret: true
Outputs:
AppUserPool:
Value:
Ref: AppUserPool
MobileAppClient:
Value:
Ref: MobileAppClient

CloudFormation BucketPolicy stuck at CREATE. Never finished CREATE

Cloud forming an S3 bucket with user, user access keys, and policy. It should create the stack and output the user access keys needed to work with the created S3 bucket via the SDK. Bucket Policy gets stuck in the CREATING phase forever when trying to reference the BucketUser ARN in the BucketPolicy Principal.
CloudFormation is successful with
BucketPolicy: ... Principal: "*"
But BucketPolicy resource is stuck in CREATE forever with
BucketPolicy: ... Principal: !GetAtt BucketUser.Arn
This successfully returns the BucketUser.Arn when BucketPolicy: ... Principal: "*"
Outputs:
BucketUserArn:
Value: !GetAtt BucketUser.Arn
Desired Template:
AWSTemplateFormatVersion: "2010-09-09"
Description: "Creates bucket with bucket policy"
#Metadata:
Parameters:
app:
Type: String
Description: (required) Application name (Also used for bucket name. Follow S3 bucket name conventions)
Default: ymessage-bucket-test
Resources:
BucketUser:
Type: "AWS::IAM::User"
Properties:
UserName: !Ref app
UserAccessKey:
Type: "AWS::IAM::AccessKey"
Properties:
Status: Active
UserName: !Ref app
DependsOn: BucketUser
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref app
BucketPolicy:
Type: "AWS::S3::BucketPolicy"
Properties:
Bucket: !Ref app
PolicyDocument:
Statement:
-
Action:
- "s3:*"
Effect: "Allow"
Resource:
Fn::Join:
- ""
-
- "arn:aws:s3:::"
- !Ref app
- "/*"
Principal: !GetAtt BucketUser.Arn
DependsOn: BucketUser
Outputs:
AccessKeyId:
Value: !Ref UserAccessKey
AccessKeySecret:
Value: !GetAtt UserAccessKey.SecretAccessKey
BucketURL:
Value: !GetAtt Bucket.WebsiteURL
BucketUserArn:
Value: !GetAtt BucketUser.Arn
Working Template:
AWSTemplateFormatVersion: "2010-09-09"
Description: "Creates bucket with bucket policy"
#Metadata:
Parameters:
app:
Type: String
Description: (required) Application name (Also used for bucket name. Follow S3 bucket name conventions)
Default: ymessage-bucket-test
Resources:
BucketUser:
Type: "AWS::IAM::User"
Properties:
UserName: !Ref app
UserAccessKey:
Type: "AWS::IAM::AccessKey"
Properties:
Status: Active
UserName: !Ref app
DependsOn: BucketUser
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref app
BucketPolicy:
Type: "AWS::S3::BucketPolicy"
Properties:
Bucket: !Ref app
PolicyDocument:
Statement:
-
Action:
- "s3:*"
Effect: "Allow"
Resource:
Fn::Join:
- ""
-
- "arn:aws:s3:::"
- !Ref app
- "/*"
Principal: "*"
DependsOn: BucketUser
Outputs:
AccessKeyId:
Value: !Ref UserAccessKey
AccessKeySecret:
Value: !GetAtt UserAccessKey.SecretAccessKey
BucketURL:
Value: !GetAtt Bucket.WebsiteURL
BucketUserArn:
Value: !GetAtt BucketUser.Arn
Found the issue: In the BucketPolicy it can directly accept Principal: "*" but if you want to use an arn do this:
Principal:
AWS:
- !GetAtt BucketUser.Arn