I've looked everywhere on the Interweb but couldn't find a satisfying answer...
Does anybody know what "protocol" the AWS S3 speaks?
Our idea is to write a Function for a PLC (no chance to use the provided API) to communicate directly with AWS S3.
For Example PLC to "AWS IoT" works in MQTT/HTTP - how can I skip "AWS IoT"?
I know there is the possibility to put an IoT device inbetween - but we are evaluating our possibilities right now.
Thank you in advance
All of the AWS services have a documented REST API - the S3 one is here. In addition, all of their libraries are open source so you could likely get some ideas from them too.
Related
long time reader but I've usually been able to find the answers I've been looking for in existing posts - but this time I've not been able to.
I am essentially teaching myself AWS CDK from scratch, I've only really just started with it so not finding anything which helps me on my mission may be a result of not knowing enough yet to be asking the right questions... so please bare with me.
Thus far I've used the AWS CDK with Python to create a stack which creates an S3 bucket, and also fires up an EC2 instance with an AWS file storage gateway AMI loaded on it (so running Amazon Linux). This deploys and runs fine - however now I'd like to programmatically set up the S3 bucket to be accessed via an NFS share on the EC2 instance. From what I've seen I'd assumed it is or should be fairly trivial however I keep getting a bit lost in documentation and internet hunts and not quite sure I'm looking in the right places or asking search engines the right questions to unlock the path to achieve this.
It looks like I should be able to script something up to make it happen when the instance is start using user-data but I'm a bit lost. Is anyone able to throw me some crumbs to follow to find a good way of achieving this, or a better way of achieving what I want to happen (which is basically accessing the S3 bucket contents as though they are files on an EC2 instance) - if not tell me how to do it if it's trivial enough?
Much appreciated :)
Dan
You are on good track. user_data can be used for that.
I don't have full code to give you as its use case specific (e.g. which OS are you using?), but the user_data would have to download and install s3fs:
s3fs allows Linux and macOS to mount an S3 bucket via FUSE. s3fs preserves the native object format for files, allowing use of other tools like AWS CLI.
However, S3 is an object storage system, and it can't be really mounted on an instance like you would do with NFS or EBS storage solutions. But with s3fs-fuse you can mimic such a behavior. And for some use-cases it will be sufficient.
So what you can do, is to setup the user_data script through console, verify that it works, and then basically just copy and paste to CDK. Its more of a trial-and-see approach, but this is the best way to learn.
I'm looking for some benchmark or article explaining what is faster.
Inside a lambda function, is it faster to....:
A) Download an S3 file through cloudfront with a regular request module (i.e. hit the cloudfront URL with request or axios and download it)
B) Use the AWS SDK to get the file through the getObject methods
I've been googling this for a while now and I don't quite get to the answer, and I'm hoping I can skip benchmark it if someone else did already.
I'm talking about pretty small files, like fonts or images.
And the root of the question is, I believe AWS uses some sort of backbone communication for some cases. Given that lambda is inside their system, as S3 is, maybe requesting the image through the internet (HTTP) is not that fast.
Thanks!
In the same region it should be faster to use the SDK to download it. If it's not in the same region you might want to replicated it so that it is.
I would like to start using serverless-framework to manage lambda deploys at my company, but we handle PHI so security’s tight. Our compliance director and CTO had concerns about passing our AWS key and secret to another company.
When doing a serverless deploy, do AWS credentials ever actually pass through to Serverless, Inc?
If not, can someone point me to where in the code I can prove that?
Thanks!
Running serverless deploy isn't just one call, it's many.
AWS example (oversimplification):
Check if deployment s3 bucket already exists
Create an S3 bucket
Upload packages to s3 bucket
Call CloudFormation
Check CloudFormation stack status
Get info of created recourses (e.g. endpoint urls of created APIs)
And those calls can change dependent on what you are doing and what you have done before.
The point I'm trying to make is is that these calls which contain your credentials are not all located in one place and if you want to do a full code review of Serverless Framework and all it's dependencies, have fun with that.
But under the hood, we know that it's actually using the JavaScript aws-sdk (go check out the package.json), and we know what endpoints that uses {service}.{region}.amazonaws.com.
So to prove to your employers that nothing with your credentials is going anywhere except AWS you can just run a serverless deploy with wireshark running (other network packet analyzers are available). That way you can see anything that's not going to amazonaws.com
But wait, why are calls being made to serverless.com and serverlessteam.com when I run a deploy?
Well that's just tracking some stats and you can see what they track here. But if you are uber paranoid, this can be turned off with serverless slstats --disable.
Given the popularity of hosting static sites from AWS S3 buckets it would be great to be able to do that from Cloud9 too.
Is there any way I can set up an FTP-based workspace that uses an S3 bucket as the source?
Transmit and other FTP apps have the ability to work directly with an S3 bucket. I did try setting up an FTP workspace in Cloud9 using the following:
Host: s3.amazonaws.com
Username: My-Access-Key
Password: My-Secret-Key
I know it was a long-shot and I have since read confirmation that Amazon doesn't allow simple FTP access to buckets like that.
Any ideas if this is possible?
FTP workspaces on Cloud9 are actually being phased out, so I'd recommend using the mounting feature described in this blog post to mount an FTP source: https://c9.io/site/blog/2014/12/ftp-sftp-mounting-beta
Unfortunately, S3 doesn't support the FTP protocol, so this would have to be a new feature. Luckily we're opening up our SDK to be able to implement features like this. If you're interested in contributing please email us via https://support.c9.io
Codeanywhere (https://codeanywhere.com) does this now. However, you'll have to shell out $7 to $10/m for that capability.
But then again, like Cloud9 (which I'm a big fan of), you get a bunch of features on the Codeanywhere IDE.
I was disappointed when Cloud9 discontinued its efforts on S/FTP. Codeanywhere seems to be taking on the cloud/storage issue head on by handling cloud access to S3, FTP, SFTP, Google Drive and others.
I have a JAR file - jets3t-0.7.4.jar, by which I can access Amazon's S3 storage. I need to modify its source code so that it accesses Ceph object storage instead. I know it can done by modfying the S3 API, but do not know how. Does anyone know how to do this? I googled for information, but didn't really find anything informative. Any help is appreciated. Thanks!
Just let the S3 endpoint resolve to your ceph radosgw (ceph's S3 API interface.), via /etc/resolv.conf, dnsmasq, jets3t's config....many ways available.
Many object storage claim that they are S3 compatible, but in fact they are not. I think ceph is one of them. If what you want is fully compatible, google cloudian.