AWS EBS - Rails5 / nginx - robots.txt not found error (404) - ruby-on-rails-5

Question

I deployed a pretty standard Rails 5 app with AWS EBS.
My /robots.txt is not reacheable and requests to it's URL return a 404 error.
I put it in the /public folder along with 404.html, 422.html and 500.html pages, which are correctly served by nginx.
Any clue about what might be wrong? What shall I check?
EB CLI 3.14.6 (Python 2.7.1)
Ruby 2.4.3 / Rails 5.1.4 / Puma (gem) 3.7

Answer 1

Looks like a very similar question have been asked 4 years ago on the official AWS forum: https://forums.aws.amazon.com/thread.jspa?threadID=150904
Only 4 years later a brave guy from AWS stepped in with a reply! Here below the quoted reply:
Hello hello! I'm Chris, the new Ruby platforms person at Elastic
Beanstalk. Visiting this thread today, it looks like there's been a
lot of pain (and also confusion!) from Beanstalk's Ruby+Puma's
handling of static files.
Quick summary: When this thread was created (in 2014), Beanstalk was
essentially using the default Nginx that comes with Amazon Linux, with
only some logging modifications to support the health monitoring. That
spawned this thread, as static files are generally expected to be
served the the web server when one is present.
So, the folks here went and fixed the /assets folder. Great!
Unfortunately, there was a misunderstanding with the request to fix
serving the /public folder - Beanstalk's Puma platform instead serves
things in '/public' from '/pubilc', not from '/'. This is definitely
an issue, so here's some workarounds:
Workaround 1: Turning on serve static assets. Yes, this wastes some
application threads here or there, but if your use case is only
robots.txt and favicon.ico, you're only robbing a couple of appserver
threads. I'd pick this one unless I was running my application servers
hot.
Workaround 2: Write an .ebextension to modify the Nginx configuration
to serve /public at /. I'm in the process of writing one, so I'll tack
it as a reply to this when I've given it the thought it deserves. Some
of the current ones may serve your app's code, so double check the
configuration if you've already done this workaround.
I've created a tracking issue for the team with this level of detail,
so we'll work to get this corrected. Thank you all for your feedback -
we'd love to serve you and your apps better.
Since then, no further replies; if anybody knows the "aws-approved-way" to edit nginx config with .ebextensions let's post it here please! :)

Answer 2

In AWS EB with PUMA, static files under the public folder are served under the /public/ url. Webcrawlers expect the file available at /robots.txt
I've struggled to try and implement routing to these files and settled instead on a more 'Rails' way of implementing this.
1) config/routes.rb
get "/robots.txt", to: "robots#show"
2) app/controllers/robots_controller.rb
class RobotsController < ApplicationController
def show
render "show", layout: false, content_type: "text/plain"
end
end
3) app/views/robots_txts/show.erb
User-agent: *
Disallow: /

Answer 3

The above link to AWS forums is erroring with a 400 right now, so here's how I fixed this issue. Ruby 2.7 running on AWS2 platform:
Static Files in sub-directory of /public:
Create a file under the .ebextensions folder called static-files.conf. Content should look similar to:
option_settings:
aws:elasticbeanstalk:environment:proxy:staticfiles:
/w3c: public/w3c
/images: public/images
This will ensure that all requests to domain.com/images and domain.com/w3c are served from the appropriate /public sub-directory.
Static Files in top level of /public directory:
For top-level files like robots.txt or sitemap.xml add appropriate entry to routes.rb to serve the static content directly:
get '/robots.txt', to: proc {|env| [200, {}, [File.open(Rails.root.join('public', 'robots.txt')).read]] }
get '/sitemap.xml', to: proc {|env| [200, {}, [File.open(Rails.root.join('public', 'sitemap.xml')).read]] }
Ensure production.rb has static files config set properly:
config.serve_static_files = false
This last part is most-important.