I am using the brotli plugin for compressing my bundles in gatsby: https://github.com/ovhemert/gatsby-plugin-brotli
the plugin is configured as follows:
{
resolve: "gatsby-plugin-brotli",
options: {
extensions: ["css", "html", "js", "svg", "ttf"],
},
},
}
For some reason, it seems to work only when I "gatsby serve" it on my local machine (localhost:9000), but uploading it to a bucket on S3, shows no compressions what so ever:
local deployment:
s3 deployment:
Nothing works, no matter what I've tried.
for uploading to S3, I've been using the gatsby-plugin-s3 package:
https://github.com/jariz/gatsby-plugin-s3
Any idea?
Thanks!
EDIT:
I checked the bucket to make sure that it contains the compressed files and it does, but also the decompressed files:
So I guess I need to refine my questions, but not sure how... not sure what caused this issue
So after spending quite some time on this issue, I finally found the answer in AWS docs.
I'm using CloudFront to serve the site, but since this site is a demo site for testing purposes, I didn't purchase an HTTPS Cert for it.
According to AWS Docs, CF doesn't serve compressed content over HTTP, ONLY on HTTPS:
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html
Purchasing and installing a certificate SOLVED this problem for me.
Related
So to explain a problem I have a static website hosted on s3 and CloudFront as the CDN. I have used create-react-app(CRA) to create the react package for my website. CRA by default does versioning of webpack build files and the versions are visible in the s3 bucket as well.
Still when I do a deployment, the latest changes don't come up(I have waited even a day hoping it would come). I am not sure what is causing this issue. Can anyone please help.
I have added the screenshots of my cloudfront behaviour tab and the s3 bucket files having build versions.
Ps, If it is the case of browser cache how can I disable it so that my clients always see the most latest version of my website.
Hi you have to invalidate cache in the distribution settings tab. I ideally invalidate all cache by passing /* or you can also specify folder or file to clear cacheing.
example: /index.html
Even in your CI/CD pools line you can ask deploy agent to invalidate cache by passing distribution ID and path
I deployed a pretty standard Rails 5 app with AWS EBS.
My /robots.txt is not reacheable and requests to it's URL return a 404 error.
I put it in the /public folder along with 404.html, 422.html and 500.html pages, which are correctly served by nginx.
Any clue about what might be wrong? What shall I check?
EB CLI 3.14.6 (Python 2.7.1)
Ruby 2.4.3 / Rails 5.1.4 / Puma (gem) 3.7
Looks like a very similar question have been asked 4 years ago on the official AWS forum: https://forums.aws.amazon.com/thread.jspa?threadID=150904
Only 4 years later a brave guy from AWS stepped in with a reply! Here below the quoted reply:
Hello hello! I'm Chris, the new Ruby platforms person at Elastic
Beanstalk. Visiting this thread today, it looks like there's been a
lot of pain (and also confusion!) from Beanstalk's Ruby+Puma's
handling of static files.
Quick summary: When this thread was created (in 2014), Beanstalk was
essentially using the default Nginx that comes with Amazon Linux, with
only some logging modifications to support the health monitoring. That
spawned this thread, as static files are generally expected to be
served the the web server when one is present.
So, the folks here went and fixed the /assets folder. Great!
Unfortunately, there was a misunderstanding with the request to fix
serving the /public folder - Beanstalk's Puma platform instead serves
things in '/public' from '/pubilc', not from '/'. This is definitely
an issue, so here's some workarounds:
Workaround 1: Turning on serve static assets. Yes, this wastes some
application threads here or there, but if your use case is only
robots.txt and favicon.ico, you're only robbing a couple of appserver
threads. I'd pick this one unless I was running my application servers
hot.
Workaround 2: Write an .ebextension to modify the Nginx configuration
to serve /public at /. I'm in the process of writing one, so I'll tack
it as a reply to this when I've given it the thought it deserves. Some
of the current ones may serve your app's code, so double check the
configuration if you've already done this workaround.
I've created a tracking issue for the team with this level of detail,
so we'll work to get this corrected. Thank you all for your feedback -
we'd love to serve you and your apps better.
Since then, no further replies; if anybody knows the "aws-approved-way" to edit nginx config with .ebextensions let's post it here please! :)
In AWS EB with PUMA, static files under the public folder are served under the /public/ url. Webcrawlers expect the file available at /robots.txt
I've struggled to try and implement routing to these files and settled instead on a more 'Rails' way of implementing this.
1) config/routes.rb
get "/robots.txt", to: "robots#show"
2) app/controllers/robots_controller.rb
class RobotsController < ApplicationController
def show
render "show", layout: false, content_type: "text/plain"
end
end
3) app/views/robots_txts/show.erb
User-agent: *
Disallow: /
The above link to AWS forums is erroring with a 400 right now, so here's how I fixed this issue. Ruby 2.7 running on AWS2 platform:
Static Files in sub-directory of /public:
Create a file under the .ebextensions folder called static-files.conf. Content should look similar to:
option_settings:
aws:elasticbeanstalk:environment:proxy:staticfiles:
/w3c: public/w3c
/images: public/images
This will ensure that all requests to domain.com/images and domain.com/w3c are served from the appropriate /public sub-directory.
Static Files in top level of /public directory:
For top-level files like robots.txt or sitemap.xml add appropriate entry to routes.rb to serve the static content directly:
get '/robots.txt', to: proc {|env| [200, {}, [File.open(Rails.root.join('public', 'robots.txt')).read]] }
get '/sitemap.xml', to: proc {|env| [200, {}, [File.open(Rails.root.join('public', 'sitemap.xml')).read]] }
Ensure production.rb has static files config set properly:
config.serve_static_files = false
This last part is most-important.
Recently i have added the gulp-gzip task to my gulp script pipeline, that also has a livereload to refresh the browser when any file changes.
gulp.task('webserver', function() {
connect.server({ port: 8080, root: 'public/', livereload: true });
});
The reload works fine but the server doesnt serve the gziped files fine.
In the browser network it appears the files content compressed:
If i launch a simple-http-server in the path the gz files are served correctly. It is possible to tune the gulp-connect server to solve de gz issue?
From my understanding, instead of gzipping your files and then delivering them from the server, you should, as you said, tune the gulp-connect server to deliver the files compressing them beforehand. For this, you can use the middleware configuration attribute in the gulp-connect server to provide the gzip extension as it is used in this gist and, of course, you will need to add the connect-gzipdependency to your project. Hope it helps!
Current Situation
I have a project on GitHub that builds after every commit on Travis-CI. After each successful build Travis uploads the artifacts to an S3 bucket. Is there some way for me to easily let anyone access the files in the bucket? I know I could generate a read-only access key, but it'd be easier for the user to access the files through their web browser.
I have website hosting enabled with the root document of "." set.
However, I still get an 403 Forbidden when trying to go to the bucket's endpoint.
The Question
How can I let users easily browse and download artifacts stored on Amazon S3 from their web browser? Preferably without a third-party client.
I found this related question: Directory Listing in S3 Static Website
As it turns out, if you enable public read for the whole bucket, S3 can serve directory listings. Problem is they are in XML instead of HTML, so not very user-friendly.
There are three ways you could go for generating listings:
Generate index.html files for each directory on your own computer, upload them to s3, and update them whenever you add new files to a directory. Very low-tech. Since you're saying you're uploading build files straight from Travis, this may not be that practical since it would require doing extra work there.
Use a client-side S3 browser tool.
s3-bucket-listing by Rufus Pollock
s3-file-list-page by Adam Pritchard
Use a server-side browser tool.
s3browser (PHP)
s3index Scala. Going by the existence of a Procfile, it may be readily deployable to Heroku. Not sure since I don't have any experience with Scala.
Filestash is the perfect tool for that:
login to your bucket from https://www.filestash.app/s3-browser.html:
create a shared link:
Share it with the world
Also Filestash is open source. (Disclaimer: I am the author)
I had the same problem and I fixed it by using the
new context menu "Make Public".
Go to https://console.aws.amazon.com/s3/home,
select the bucket and then for each Folder or File (or multiple selects) right click and
"make public"
You can use a bucket policy to give anonymous users full read access to your objects. Depending on whether you need them to LIST or just perform a GET, you'll want to tweak this. (I.e. permissions for listing the contents of a bucket have the action set to "s3:ListBucket").
http://docs.aws.amazon.com/AmazonS3/latest/dev/AccessPolicyLanguage_UseCases_s3_a.html
Your policy will look something like the following. You can use the S3 console at http://aws.amazon.com/console to upload it.
{
"Version":"2008-10-17",
"Statement":[{
"Sid":"AddPerm",
"Effect":"Allow",
"Principal": {
"AWS": "*"
},
"Action":["s3:GetObject"],
"Resource":["arn:aws:s3:::bucket/*"
]
}
]
}
If you're truly opening up your objects to the world, you'll want to look into setting up CloudWatch rules on your billing so you can shut off permissions to your objects if they become too popular.
https://github.com/jupierce/aws-s3-web-browser-file-listing is a solution I developed for this use case. It leverages AWS CloudFront and Lambda#Edge functions to dynamically render and deliver file listings to a client's browser.
To use it, a simple CloudFormation template will create an S3 bucket and have your file server interface up and running in just a few minutes.
There are many viable alternatives, as already suggested by other posters, but I believe this approach has a unique range of benefits:
Completely serverless and built for web-scale.
Open source and free to use (though, of course, you must pay AWS for resource utilization -- such S3 storage costs).
Simple / static client browser content:
No Ajax or third party libraries to worry about.
No browser compatibility worries.
All backing systems are native AWS components.
You never share account credentials or rely on 3rd party services.
The S3 bucket remains private - allowing you to only expose parts of the bucket.
A custom hostname / SSL certificate can be established for your file server interface.
Some or all of the host files can be protected behind Basic Auth username/password.
An AWS WebACL can be configured to prevent abusive access to the service.
How do I configure Plupload properly so that it will upload files directly to Amazon S3?
In addition to condictions for bucket, key, and acl, the policy document must contain rules for name, Filename, and success_action_status. For instance:
["starts-with", "$name", ""],
["starts-with", "$Filename", ""],
["starts-with", "$success_action_status", ""],
Filename is a field that the Flash backend sends, but the HTML5 backend does not.
The multipart setting must be True, but that is the default these days.
The multipart_params setting must be a dictionary with the following fields:
key
AWSAccessKeyId
acl = 'private'
policy
signature
success_action_status = '201'
Setting success_action_status to 201 causes S3 to return an XML document with HTTP status code 201. This is necessary to make the flash backend work. (The flash upload stalls when the response is empty and the code is 200 or 204. It results in an I/O error if the response is a redirect.)
S3 does not understand chunks, so remove the chunk_size config option.
unique_names can be either True or False, both work.
Latest Plupload release has illustrative example included, that shows nicely how one might use Plupload to upload files to Amazon S3, using Flash and SilverLight runtimes.
Here is the fresh write-up: Upload to Amazon S3
The official Plupload tutorial, much more detailed than the answers here: https://github.com/moxiecode/plupload/wiki/Upload-to-Amazon-S3
If you are using Rails 3, please check out my sample projects:
Sample project using Rails 3, Flash and MooTools-based FancyUploader to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-FancyUploader
Sample project using Rails 3, Flash/Silverlight/GoogleGears/BrowserPlus and jQuery-based Plupload to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload
I want to notice, that don't forget to upload crossdomain.xml to your s3 host, and also if you have success_action_redirect url, you need to have crossdomain.xml file on that domain too. I spent 1 day fighting with that problem, and finally found what's wrong. So next time think how flash work inside.
Hope I save time for someone.