Apache Reverse Proxy not Working with Grafana - apache

Question

I'm pulling my hair out because it must be something simple.
I've setup Grafana to run through a subdomain via proxy which works fine. I'm doing basic authentication as well to login to Grafana, this is my apache config:
<VirtualHost *:80>
ServerAdmin webmaster#example.co
ServerName example.co
ServerAlias www.example.co
DocumentRoot /var/www/example.co/public_html/
ErrorLog /var/www/example.co/logs/error.log
CustomLog /var/www/example.co/logs/access.log combined
<Location "/application">
AuthType Basic
AuthName "Graphs Login"
AuthUserFile /var/www/example.co/members/.htpasswd
Require valid-user
ProxyPass http://localhost:3000/
</Location>
ProxyPassReverse /application http://example.co:3000/
</VirtualHost>
And my config in grafana.ini
# The public facing domain name used to access grafana from a browser
domain = example.co
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false
# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = %(protocol)s://%(domain)s:%(http_port)s/application/
I've also tried to do this with a 301 redirect:
Redirect 301 /application http://example.co/application/
<Location "/application/">
AuthType Basic
AuthName "Graphs Login"
AuthUserFile /var/www/example.co/members/.htpasswd
Require valid-user
ProxyPass http://localhost:3000/
ProxyPassReverse http://localhost:3000/
</Location>
The ProxyPass to the domain works fine, however if I use the IP address it doesn't go to the domain, but rather it tries to load it as IP:3000/application which then gives a Grafana error (Blank page with {{alert.title}})
Any idea what I am doing wrong and how I can redirect the IP:3000 to DOMAIN:3000 such as the ProxyPass is doing with Reverse Proxy?
I'm using Ubuntu 12.04
I've done all sorts of things such as adding trailing slashes, removing them from both apache and from grafana configs, i've been using incognito everytime to ensure there is no caching and I just can't get it to work in Reverse Proxy where I'm trying to redirect the IP to the domain that I've set in Grafana and ProxyPass.
Please help!

Answer 1

I am not 100% sure if this may be the issue. However let's give it a try: Try removing the slashes after the ProxyPass directives:
ProxyPass /application http://localhost:3000
ProxyPassReverse /application http://localhost:3000
Grafana seems to be very picky with regard to those slashes. I had several issues with those little buggers :) I will explain the details further below. A very simple working setup (without specific authentication though) looks like this:
Apache
<VirtualHost *:443>
ServerName example.co
ServerAdmin webmaster#localhost
DocumentRoot /var/www/html
# grafana
ProxyPass /grafana http://thorin:3000
ProxyPassReverse /grafana http://thorin:3000
</VirtualHost>
Grafana
domain = example.co
root_url = %(protocol)s://%(domain)s/grafana
Explanation: Slashes
Next I explain the reason for the unexpected behavior as far as I understood it. For completeness sake I am also adding an issue I had with certain rewrite rules in addition to the proxy setup.
Proxy
So what happens if you keep the slashes?
# WRONG!
ProxyPass /application http://localhost:3000/
ProxyPassReverse /application http://localhost:3000/
In this case, the response you get from Grafana is a ugly page showing {{alert.title} and a bunch of other unformated HTML. The reason for this is that Grafana can not load certain resources:
http://example.co/application/public/build/grafana.dark.css?v5.2.4
http://example.co/application/public/build/vendor.4f5454f867a0cc2fe8dd.js
However, you proxy settings work correctly, right? Well, partially. They have that tiny slash / resulting in the following lookups on your Grafana installation:
http://localhost:3000//public/build/grafana.dark.css?v5.2.4
http://localhost:3000//application/public/build/vendor.4f5454f867a0cc2fe8dd.js
Notice the extra slash / after http://localhost:3000. Try calling those URLs. They do not work. This is Grafana being very picky about URLs :) Thus, removing the extra slashes from your Apache config will do the trick.
At least that is what I came up with so far:)
Rewrite
No, rewrite rules. In our setup we have Jekyll setup in another subpath, say http://example.co/jekyll where we use relative URLs to access resources. This requires a slash at the end of each URL. We solved this by adding following rewrite rule in Apache (there may be better solutions for this; if you have suggestions please let me know):
# add trailing slashes to support relative URLs
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !(.*)\.[a-zA-Z0-9]+$
RewriteCond %{REQUEST_URI} !(.*)/$
RewriteRule ^(.*)$ $1/ [R=301,L]
This resulted in some funny effects on Grafana:
we could not log in
the admin/admin account worked however it got stuck at changing the default password
all data sources were gone
Turns out that Grafana did not like the redirects and slash policy we introduced here. The solution was to only enable this rewrite rule for applications that needed it:
# add trailing slashes to support relative URLs
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !(.*)\.[a-zA-Z0-9]+$
RewriteCond %{REQUEST_URI} !(.*)/$
RewriteCond %{REQUEST_URI} ^jekyll.*$
RewriteRule ^(.*)$ $1/ [R=301,L]
Let me know if this helps :)