I've a connection to AWS S3 on Airflow that is made with Extra config:
aws_access_key_id
aws_secret_access_key
However, since this credentials are stored on Extra field, it's not encrypted and everyone that has access to Airflow UI can see those value:
Is there any way to encrypt those connection/values on Extra fields?
Thanks?
Related
I am trying to code an AWS ConfigRule in CloudFormation. A rule to make sure S3 buckets are encrypted, ALB access logs are logging, also that EBS volumes are encrypted.
Do I need to code a new scope and source for each one of these mentioned above, or can I code them all together? meaning, is this below ok
Alluxio on kubernetes(EKS) supports s3 connection without aws accessKey and secretKey? Is the s3 connection configurable with role arn alone?
We are installing Alluxio on EKS using s3 as a underlaying storage layer. Alluxio cluster is up and running with s3 storage when configurations are done like below (Using aws aceess key and secret)
ALLUXIO_JAVA_OPTS: |--
Dalluxio.master.hostname=alluxio-master-0 -
Dalluxio.master.journal.type=UFS -
Dalluxio.master.journal.folder=/journal -
Dalluxio.security.stale.channel.purge.interval=365d -
Dalluxio.master.mount.table.root.ufs=s3://cubixalluxiodata/ -
Dalluxio.master.mount.table.root.option.aws.accessKeyId=AxxxxxxxxxxxxO -
Dalluxio.master.mount.table.root.option.aws.secretKey=DxxxxxxxxxxxxD*
However we are looking for approach to configure s3 storage for alluxio without accessKey/secretKey. But with a role arn based authentication alone. Please suggest on possibility of the approach.
looks to me you need to use AWS credential profile file (https://docs.alluxio.io/os/user/stable/en/ufs/S3.html#advanced-credentials-setup) to connect to S3. Possibly to setup your AWS instance profile file and share that file to your running image
We have an s3 bucket that Airflow uses as the source of all of our DAG data pipelines. We have a bucket for dev, test and production. Let's say the bucket in dev is called dev-data-bucket, in test it's called test-data-bucket etc.
I don't want to manually specify the bucket name in our DAG code because this code gets migrated between environments. If I manually specify the dev-data-bucket in our dev environment and this DAG code goes to our test environment, the bucket name would need to change to test-data-bucket and prod-data-bucket for prod.
I understand that the usual way to do this would be to create an Airflow connection in each environment which has the same name like data-bucket. However, I don't know where to specify the bucket name in the connection screen for airflow like I would for a database connection?
How do I create an Airflow s3 connection with the same name in each environment but which specifies a different bucket name for each environment?
The way to do this is to specify the bucket name in the Schema field in the Airflow connection screen. This is what the connection screen would look like in dev:
Then, when you use one of the provided s3 operators in airflow you don't need to specify a bucket name because the s3 hook in Airflow is setup to fetch the bucket name from the connection if you haven't specified one.
so I'm using a MLFlow tracking server where I define a S3 bucket to be the artifact stores. Right now, MLFlow by default is getting the credentials to write/read the bucket via my default profile in .aws/credentials but I do have a staging and dev profile as well. So my question is is there a way to explicitly tells MLFlow to use the staging or dev profile credentials instead of default? I can't seem to find this info anywhere. Thanks!
To allow the server and clients to access the artifact location, you should configure your cloud provider credentials as normal. For example, for S3, you can set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, use an IAM role, or configure a default profile in ~/.aws/credentials. See Set up AWS Credentials and Region for Development for more info.
Apparently there is no option to set another profile. I use aws-vault so it is easy to change profiles
I want to add password to Redis.
I interested if there is a way to save encrypted password in redis.conf and not as plain text?
Or a way not to store the password in redis.conf at all?
By default redis.conf atleast until today with its most recent version - 6.0.1 still doesnt support encrypting a password.
While this is a situation is not fully avoidable, at the best, you can automate this by writing a wrapper startup script that would accept password as an argument and bring up the service. And then, once the service is up, ALTHOUGH THIS IS TO BE AVOIDED AND IS NOT RECOMMENDED you can delete the conf file or change the password in that file. and, before the startup of REDIS, you would require to run the startup script again/ re-enter the original password. BUT THIS CAN ADDITIONALY CAUSE PROBLEMS.
Please note -> redis.conf can be secured by linux/OS permissions and thats the best way to do so
No Redis doesn't support encrypted password for auth. You may check the details in official documentation
The password is set by the system administrator in clear text inside the redis.conf file. It should be long enough to prevent brute force attacks.
Additionally;
The AUTH command, like every other Redis command, is sent unencrypted, so it does not protect against an attacker that has enough access to the network to perform eavesdropping.
You may use config set requirepass yourpassword to set password and this will not require a server restart but set it on-fly, but when the server is restarted your previous password(written in conf file)/no password(if it is not set) will be used to authenticate requests.
Well while encryption is till now not an option, Redis 6 introduced ACL (Access Control List) where you can store your SHA256-hashed passwords in the redis.conf file.
Please note that this not an Encryption though!
From redis-cli:
acl setuser yourUser on #951249c8e32817cb0727ba2b1440f008c49c582e5daca4a0bd6d64eed1291a37
From redis.conf
user yourUser on #951249c8e32817cb0727ba2b1440f008c49c582e5daca4a0bd6d64eed1291a37
Additional note:
You may need to disable the default user which does not have a password:
From redis-cli:
acl setuser default off
From redis.conf
user default off