I've been using Apache Whirr(0.8.0) to create Hadoop+HBase clusters on EC2 using the CDH ami, turns out that Whirr is not using my whirr.location-id parameter. I specifically pass it as an argument --location-id=us-west-2 when I create the cluster and I also added it to the whirr recipe as whirr.location-id=us-west-2 but when I fire up the clusters they always get created in the us-east-1 region. My guess is that the Whirr recipe for CDH is overriding the region and I wonder if there is any way to disable this or force an specific EC2 region?
Related
Is it possible to change the original creator of an EKS cluster to another role. I still have access to the cluster, with both the original creator role and the new one I want to transfer the cluster to.
The new role is now encoded in de aws_auth config map, but we locked ourselves out by deleting the config map (in a terraform update). We were able to restore it using the creator role, but we'd rather not use that one anymore.
Is it possible to update the creator user, or do I need to create a new cluster with the proper role, and then transfer everything over?
From the Amazon Docs:
You don't need to add cluster_creator to the aws-auth ConfigMap to get admin access to the Amazon EKS cluster. By default, the cluster_creator has admin access to the Amazon EKS cluster that it created.
I can't seem to make a neptune notebook, everytime I try I get the following error:
Notebook Instance Lifecycle Config 'arn:aws:sagemaker:us-west-2:XXXXXXXX:notebook-instance-lifecycle-config/aws-neptune-tutorial-lc'
for Notebook Instance 'arn:aws:sagemaker:us-west-2:XXXXXXXXX:notebook-instance/aws-neptune-tutorial'
took longer than 5 minutes.
Please check your CloudWatch logs for more details if your Notebook Instance has Internet access.
Note that the cloudwatch logs that it suggests to look at don't exist.
The neptune database was created using this cloudformation template: https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/services/NeptuneDB/Neptune.yaml
Which created the neptune cluster in the default VPC.
The notebook instance was created using this cloudformation template: https://s3.amazonaws.com/aws-neptune-customer-samples/neptune-sagemaker/cloudformation-templates/neptune-sagemaker/neptune-sagemaker-nested-stack.json
passing in the relevant values from in for the created neptune stack.
Has anyone seen this type of error and knows how to get over it?
I had to go in and modify the predefined install script used by neptune and add and nohup command to the final section of the install as described here https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-lifecycle-script-timeout/
Probably what is happening is that your notebook instance does not have access to the internet. Check your NAT configuration for your VPC and their security groups have allowed outbound rules to all
Creating AMI's from EBS backed instances is exceedingly easy, but doing the same from an instance-store based instance seems like it can only be done manually using the CLI.
So far I've been able to bootstrap the creation of an 'instance-store' based server off of an HVM Amazon Linux AMI with Ansible, but I'm getting lost on the steps that follow... I'm trying to follow this: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-instance-store-ami.html#amazon_linux_instructions
Apparently I need to store my x.509 cert and key on the instance, but which key is that? Is that...
one I have to generate on the instance with openssl,
one that I generate/convert from AWS,
one I generate with Putty, or
one that already exists in my AWS account?
After that, I can't find any reference to ec2-bundle-vol in Ansible. So I'm left wondering if the only way to do this is with Ansible's command module.
Basically what I'm hoping to find out is: Is there a way to easily create instance-store based AMI's using Ansible, and if not, if anyone can reference the steps necessary to automate this? Thanks!
Generally speaking, Ansible AWS modules are meant to manage AWS resources by interacting with AWS HTTP API (ie. actions you could otherwise do in the AWS Management Console).
They are not intended to run AWS specific system tools on EC2 instances.
ec2-bundle-vol and ec2-upload-bundle must be run on the EC2 instance itself. It is not callable via the HTTP API.
I'm afraid you need to write a custom playbook / role to automate the process.
On the other hand, aws ec2 register-image is an AWS API call and correspond to the ec2_ami Ansible module.
Unfortunately, this module doesn't seem to support image registering from an S3 bucket.
when i execute the command ec2-describe-availability-zones it shows
AVAILABILITYZONE us-east-1a available us-east-1
AVAILABILITYZONE us-east-1c available us-east-1
AVAILABILITYZONE us-east-1d available us-east-1
so.. i can only run instance in us-east-1
how can i use other region if i want to run instance in other refion like ue-west-1?
i had copy ami form us-east-1 to us-west-1
and i execute the command
ec2-run-instances ami-526a0662 -n 1 -k USweastOregon -g launch-wizard-2 --monitor
it shows
Client.InvalidAMIID.NotFound: The image id '[ami-526a0662]' does not exist
AMIs are region-specific. If you want to use an AMI in a different region, you must first copy it:
To use AMI Copy, simply select the AMI to be copied from within the
AWS Management Console, choose the destination region, and start the
copy. AMI Copy can also be accessed via the EC2 Command Line
Interface or EC2 API as described in the EC2 User’s Guide. Once the
copy is complete, the new AMI can be used to launch new EC2 instances
in the destination region.
The AMI in the new region will have a different AMI ID.
You should use the --region and specify eu-west-1 or us-west-1.
--region REGION
Specify REGION as the web service region to use.
This option will override the URL specified by the "-U URL" option
and EC2_URL environment variable.
I'm querying the AWS API and trying to launch an instance with the AMI ID that is located in us-west-1a. This is failing with an error of InvalidAMIID. Using the same API keys I'm able to launch an AMI in us-east-1b. Has anyone have experience with this? I'm positive I'm doing something wrong. Not sure if this is the right place to ask.
The AMI's are different from region to region -- you can't call an AMI in us-east-* in us-west-*. If this is a custom AMI, you'll need to move it over to the new region, or just find the corresponding AMI if it's a public AMI.
I've been able to narrow this down to one of two things:
The AWS REST API requires both region and availability zone to be specified. It isn't possible to just specify the availability zone.
OR
The above problem lies actually in the 'aws' ruby gem