How can I tell airflow to authenticate on an external server? - amazon-s3

Question

I have the following task in airflow, which works like a charm:
t = SparkSubmitOperator(
task_id = 'some_id',
application = '/path/to/app.py',
name = 'airflow-spark',
conf = {
'spark.hadoop.fs.s3a.endpoint': 'https://some.url.com/',
'spark.hadoop.fs.s3a.access.key': 'myuser',
'spark.hadoop.fs.s3a.secret.key': 'my_super_secret_password',
},
dag = dag,
)
As you can guess, my spark job needs to authenticate on an S3 server instance to retrieve data. While this works, I don't want to put my password as cleartext in the dag. How can I authenticate with the S3 server, without using my password in cleartext? I tried setting up connections in airflow, which seems to be exactly for this use case, but when I use conn_id = 'my_connection' inside the task, it tries to run the spark job on the server instead.

Answer 1

If you are running airflow in AWS infra, you can use the IAM-granted permissions of the container/VM as the accessor.
If you can update the config regularly, you could issue session tokens on your desktop and update the spec. You'll need a hadoop version which supports session credentials (2.8+).
You can also use JCEKs files to store the credentials -you'd then get that file onto all VMs for work and set the hadoop/spark config to load it. See https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#Storing_secrets_with_Hadoop_Credential_Providers. This does need to be on a shared store, e.g, HDFS, mounted EBS, etc, as each spark worker will resolve the jceks path locally to load the secrets.
Simplest is use IAM permissions. If you are running on a shared cluster, JCEKS files is better
Finally, Hadoop 3.3+ allows for the S3A to dynamically generate session/role tokens from a user with full credentials, and pass these with the spark job. If you play with that you can have credentials on your desktop/JCEKs file only airflow can read, and have session/role credentials generated from those. Useful, but trickier to set up