My Airflow application is running in AWS EC2 instance which has IAM role as well. Currently I am creating Airflow S3 connection using hardcoded access and secret key. But I want my application to pickup this AWS credentials from this instance itself.
How to achieve this?
We have a similar setup, our Airflow instance run inside containers deployed inside an EC2 machine. We set up the policies to access S3 on the EC2 machine instance profile. You don't need to pick up the credentials in the EC2 machine, because the machine has an instance profile that should have all the permissions that you need. From the Airflow side, we only use aws_default connection, in the extra parameter we only setup the default region, but there aren't any credentials.
Here a details article about Intance Profiles: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
The question is answered but for future reference, it is possible to do it without relying on aws_default and just doing it via Environment Variables. Here is an example to write logs to s3 using an AWS connection to benefit form IAM:
AIRFLOW_CONN_AWS_LOG="aws://"
AIRFLOW__CORE__REMOTE_LOG_CONN_ID=aws_log
AIRFLOW__CORE__REMOTE_LOGGING=true
AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER="s3://path to bucket"
Related
Hello: I do have Django based application deployed in AWS EC2 instance. The application accesses a hospital database hosted on the premises for query and update through as set of APIs written in Django application. However, we cant have production IP, database UserID, database Pwd in our EC2 application due to security reasons. What are the options available for this problem. One option that is thought is set-up a reverse proxy server on a different machine at on-premise environment and access the prod database through it. I wanted to know,
If there are other better practiced solutions available
A pointer to an example configuration of ReverseProxy maintaining database UserID, databasePwd etc in the proxy and running the API through it.
We are still trying setting-up an UBUNTU based Ngnx server. However, still not sure where to maintain the DBUserID, DBPwd etc.
Any help or guiding pointer will be greatly appreciated. Thanks
I haven't tried this in django, here is what I consider as a neat solution:
Assign an IAM role to the EC2 instance. Assign permission to the role so that it can read from the Systems manager parameter store
Store the credentials (Username, Password, Host, etc) in the aws System manager (SSM) parameter store
Add a boot script to the ec2 and query the systems manager for the parameter. The script will run once when the ec2 instance boots, read the values from SSM and set the values as environment variables
In the django setting file where the db credentials are set, i would read the credential from the environment variables for e.g os.environ['DATABASE_PASSWORD']
References:
EC2 Boot Script
Systems manager parameter store
How to set environment variables in python
I'm writing an aspnet core app which can be deployed to either azure or aws. The Microsoft libraries for accessing azure logging / configuration sources are well-behaved and fail silently if they're not in an appropriate environment. However the AWS SDK blows up with and exception Unable to get IAM security credentials from EC2 Instance Metadata Service. if the providers are configured outside of AWS. Are there some environment variables I can look at to determine if my application is running in AWS so I can skip those?
All EC2 instances (and therefore all AWS hosting methods) have access to an instance meta-data http service running on a local-link address at http://169.254.169.254/latest/meta-data/. The best process I can come up with is to make a call to this service. If the call fails -- the process is not hosted on an EC2 instance.
I have installed an App with Apache tomcat on AWS ec2 instance. I am able to access the tomcat url (which is server_name:8080/BOE/BI) from the AWS instance on Win2016. Also I installed IIS on the server.
Now what are the configurations I need to do to on AWS ec2 instance to access the URL from outside the AWS instance like from my personal PC.
I also tried disabling the firewalls, it did not helped.
You need to look at security groups. This will allow you to open up ports on your EC2 instance from the outside world.
I have deployed the individual spinnaker components to kubernetes and when I am trying to access spinnaker through http://localhost:9000 I get an empty response from the server. I verified the configuration for clouddriver-local.yml, spinnaker-local.yml and everything seems good. Am i missing anything here? when I am trying to curl localhost:9000, I get an empty response from the server
here is the kubernetes setup info
Hi Spinnaker has evolved by this time and it should be easier to set up by now. If you want to do PoC only or deploy to small enterprise projects then i suggest you use Armory's Minnaker
Now if you want to deploy large projects to a robust and fully enhanced kubernetes cluster then that is a different story and the steps are as it follows:
Minimum 4 CPUs and 12 GB of memory
Access to an existing object storage bucket
Access to an IAM role or user with access to the bucket. (AWS IAM for AWS S3)
An existing Kubernetes Ingress controller or the permissions to install the NGINX Ingress Controller (ForDeck UI access)
Installation
Create a Kubernetes namespace for Spinnaker and Halyard
Grant the default ServiceAccount in the namespace access to the cluster-admin ClusterRole in the namespace.
Run Halyard (Spinnaker installer) as a Pod in the created namespace (with a StatefulSet).
Create a storage bucket for Spinnaker to store persistent configuration in.
Create an user (AWS IAM in case of AWS deployment) that Spinnaker will use to access the bucket (or alternately, granting access to the bucket via roles).
Rung hal client interactively in the Kubernetes Pod:
Build out the hal config YAML file (.hal/config)
Configure Spinnaker with the IAM credentials and bucket information
Turn on other recommended settings (artifacts and http artifact providers: github, bitbucket, etc)
Install Spinnaker hal deploy
Expose Spinnaker (Deck through ingress)
For more details refer to
Armory's doc
Spinnaker Distributed installation in Kubernetes
Hope the guideline helps
I have a cron job in a docker image that I deploy to elastic beanstalk. In that job I wish to include read and write operations on S3 and have included the AWS CLI tools for that purpose.
But AWS CLI isn't very useful without credentials. How can I securely include the AWS credentials in the Docker image, such that that AWS CLI will work? Or should I take some other approach?
Always try to avoid setting credentials on machines if they run within AWS.
Do the following:
Go into the IAM console and create an IAM role, then edit the policy of that role to have appropriate S3 read/write permissions.
Then go the Elastic Beanstalk console, find your environment and go to the the configuration/instances section. Set the "instance profile" to use the role you created (a profile is associated with a role, you can see it in the IAM console when you're viewing the role).
This will mean that each beanstalk EC2 instance will have the permissions you set in the IAM role (the AWS CLI will automatically use the instance profile of the current machine if available).
More info:
http://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html#use-roles
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html