I am trying to set up AWS DMS to copy data from S3 to Redshift.
For configuration of the source endpoint (S3), after setting the parameters, when I run the test, get the following error:
AWSDatabaseMigrationService: Cannot change the engine for endpoint
with no authentication credentials
What does this error mean?
Make sure you have followed the instructions to the letter
http://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.S3.html
specifically:
Prerequisites When Using S3 as a Source for AWS DMS
When you use S3 as a source for AWS DMS, the source S3 bucket that you
use must be in the same AWS Region as the AWS DMS replication instance
that you use to migrate your data. In addition, the AWS account you
use for the migration must have read access to the source bucket.
The AWS Identity and Access Management (IAM) role assigned to the user
account used to create the migration task must have the following set
of permissions.
Related
I am totally new to AWS. So we have this s3 endpoint already created by sysadmin and another S3 bucket created (which I need to access files from). We are using amazon sdk.(We have the composer aws/aws-sdk-php")
If two apache environment variables(AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY) are set for S3 access keys, how can we easily test it without doing a code? any frontend tool to check the connection?
I am trying to see the files in the s3 bucket has particular name and planning to code using PHP.
I would like to create an ECS task with Fargate, and have that upload a file to S3 using the AWS CLI (among other things). I know that it's possible to create task roles, which can provide the task with permissions on AWS services/resources. Similarly, in OpsWorks, the AWS SDK is able to query instance metadata to obtain temporary credentials for its instance profile. I also found these docs suggesting that something similar is possible with the AWS CLI on EC2 instances.
Is there an equivalent for Fargate—i.e., can the AWS CLI, running in a Fargate container, query the metadata service for temporary credentials? If not, what's a good way to authenticate so that I can upload a file to S3? Should I just create a user for this task and pass in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables?
(I know it's possible to have an ECS task backed by EC2, but this task is short-lived and run maybe monthly; it seemed a good fit for Fargate.)
"I know that it's possible to create task roles, which can provide the
task with permissions on AWS services/resources."
"Is there an equivalent for Fargate"
You already know the answer. The ECS task role isn't specific to EC2 deployments, it works with Fargate deployments as well.
You can get the task metadata, including IAM access keys, through the ECS metadata service. But you don't need to worry about that, because the AWS CLI, and any AWS SDK, will automatically pull that information when it is running inside an ECS task.
so I'm using a MLFlow tracking server where I define a S3 bucket to be the artifact stores. Right now, MLFlow by default is getting the credentials to write/read the bucket via my default profile in .aws/credentials but I do have a staging and dev profile as well. So my question is is there a way to explicitly tells MLFlow to use the staging or dev profile credentials instead of default? I can't seem to find this info anywhere. Thanks!
To allow the server and clients to access the artifact location, you should configure your cloud provider credentials as normal. For example, for S3, you can set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, use an IAM role, or configure a default profile in ~/.aws/credentials. See Set up AWS Credentials and Region for Development for more info.
Apparently there is no option to set another profile. I use aws-vault so it is easy to change profiles
Assuming I'm on an EC2 instance which is configured with the destination bucket, is there a way to use keys for the source S3 bucket and do a copy something like this?
aws s3 cp s3://<Access key>:<secret key>#<source bucket folder> <destination bucket folder>
The AWS CLI does not support specifying two different accounts to access buckets.
You do have options:
Use the credentials for the destination bucket. In the account for the source bucket add a bucket policy granting your destination account read access to the bucket. Details.
If you cannot grant read access to the source account, create your own client using your favorite language and the AWS SDK. Initialize two client handles, one for each account. Then do a read/write copy operation. This is very easy to do in Python with boto3.
I want to publish a tutorial where a data from sample tsv file S3 is used by Redshift. Ideally I want it to be simple copy paste operation required to follow the exercises step by step, similar to what's in Load Sample Data from Amazon S3. The problem is with the first data import task using COPY command as it only supports S3, or EMR based load.
This seems like a simple requirement but no hassle-free way to really do it with Redshift COPY (I can make the file available for browser download without any problem but COPY requires CREDENTIALS parameter…)
Variety of options for Redshift COPY Authorization parameters is quite rich:
Should I ask user to Create an IAM Role for Amazon Redshift
himself?
Should I create it myself and publish the IAM role ARN? Sounds most hassle
free (copy paste) but security wise doesn't sound well…? Do I need to restrict S3 permissions to limit the access to only that particular file for that role?
Should I try temporary access instead?
You are correct:
Data can be imported into Amazon Redshift from Amazon S3 via the COPY command
The COPY command requires permission to access the data stored in Amazon S3. This can be granted either via:
Credentials (Access Key + Secret Key) associated with an IAM User, or
An IAM Role
You cannot create a Role for people and let them use it, because their Amazon Redshift cluster will be running in a different AWS Account than your IAM Role. You could possibly grant trust access so that other accounts can use the role, but this is not necessarily a wise thing to do.
As for credentials, they could either use their own or ones the you supply. They can access their own Access Key + Secret Key in the IAM console.
If you wish to supply credentials for them to use, you could create an IAM User that has permission only to access the Amazon S3 files they need. It is normally unwise to publish your AWS credentials because they might expose a security hole, so you should think carefully before doing this.
At the end of the day, it's probably best to show them the correct process so they understand how to obtain their own credentials. Security is very important in the cloud, so you would also be teaching them good security practice, in additional to Amazon Redshift itself.