Is it possible to crawl S3 file encrypted using CSE-KMS in AWS Glue? I know that Athena can do that, but haven't found similar functionality in Glue crawler
I do not think AWS Glue supports reading from client-side encryption. They have just added server-side encryption support, which is much simpler to support compared client-side encryption.
Glue Does not support Client Side Encrypted data. It only supports AWS KMS-managed keys (SSE-KMS) or Amazon S3-managed encryption keys (SSE-S3). these are the only two currently available in encryption models in Security Configuration in Glue [1].
[1] https://docs.aws.amazon.com/glue/latest/dg/console-security-configurations.html
Related
Due to some compliance requirements, whatever we put into the S3 bucket we have to enforce bucket encryption and object encryption. How we can attain this after we do "sls deploy" it automatically does the whole thing. For the bucket-level encryption, we are using "serverless-deployment-bucket" but what to do for object-level encryption?
Trying to send records to Amazon S3 with Flink: however these records need to be sent with an AES256 SSE header to request server side encryption
see aws documentation:
If you need server-side encryption for all of the objects that are stored in a bucket, use a bucket policy. For example, the following bucket policy denies permissions to upload an object unless the request includes the x-amz-server-side-encryption header to request server-side encryption:
Is this something that can be set for specific file sinks? have not found any documentation on the matter and beginning to think a forwarding lambda will be needed to transform the data.
I'm setting up a Kinesis Firehose delivery stream to S3, and I noticed you can set a custom KMS key to be used for encrypting the files on S3.
However, if the S3 bucket already has KMS encryption enabled, files would be encrypted anyway. The difference is of course that the default AWS manager S3 KMS key will be used instead of the customer managed custom KMS key supplied to Firehose.
What reason is there typically to use a custom KMS key for the encryption of Firehose data on S3, as opposed to relying on the default S3 KMS key? Is there any point at all in doing so if you're also the owner of the S3 bucket and in control of its settings, or is the primary use to enable using encryption also when you're not in control of the settings of the target bucket?
Or is the Firehose associated KMS key also used for encrypting data in transit, as opposed to the S3 provided KMS key used to encrypt data at rest?
Kinesis Firehose will use the KMS key you specify to encrypt the objects when landing in S3. You may not have control over the S3 bucket's encryption settings, and you may want to use a different KMS key (with different permissions) than the S3 default KMS encryption key for whatever reason. There can be many different objects in that S3 bucket at different hierarchies, requiring different KMS encryption, or not.
S3 should not "double encrypt" your data. The KMS encryption from Kinesis Firehose will be specified in the S3 put header, so S3 will know which encryption settings to use when it does the actual write. If there are default KMS settings on the S3 bucket, and it does not find an encryption setting in the put header (whether SSE or KMS), then S3 should apply the default encryption specified in the bucket settings.
We have a number of google cloud storage transfer job that sync from aws s3 buckets to google buckets. I am assuming that they are using https to transfer the data but where can I get a confirmation that they do. Where can I get information about minimum TLS version used in these transfer jobs.
Regarding the Cloud Storage TLS, in this document you could find the TLS information for the gsutil commands which requests are done via the JSON API. These requests are via HTTPS only, and are used within the cloud console too.
if I'm using the AmazonS3Client to put and fetch files, is my connection encrypted? This seems basic, but my googling seems to return things about encrypting the S3 storage and not whether the transmission from this client is secure. If it's not secure is there a setting to make it secure?
Amazon S3 endpoints support both HTTP and HTTPS. It is recommended that you communicate via HTTPS to ensure your data is encrypted in transit.
You can also create a Bucket Policy that enforces communication via HTTPS. See:
Stackoverflow: Force SSL on Amazon S3
Sample policy: s3BucketPolicyEncryptionSSL.json