503 Service unavailable Apache, Magento 2, Varnish, Redis - apache

Question

i have a digitalocean droplet 1 click Magento 2 install that has Redis & Varnish, and because varnish uses port 80 so installing SSL is a little bit tricky, that i had to include http --http-01-port 8080 so it works, today was supposed to be the certificate renewal but it didn't auto renew and i received site not secured error, so i logged in to the server and i renewed the certificate using
certbot --force-renewal --http-01-port 8080 -d domainname.com,www.domainname.com
and the renewal is done, but after i did that i'm now receiving 503 Service unavailable with varnish disabled and with varnish enabled i receive Error 503 backend fetch failed
/etc/varnish/default.vcl
backend default {
.host = "localhost";
.port = "8080";
.first_byte_timeout = 600s;
.probe = {
.url = "/health_check.php";
.timeout = 2s;
.interval = 5s;
.window = 10;
.threshold = 5;
}
/etc/apache2/sites-enabled/000-default.conf
<VirtualHost 127.0.0.1:8080>
ServerAdmin webmaster#localhost
ServerName domainname.com
ServerAlias www.domainname.com
DocumentRoot /var/www/html/pub
<Directory /var/www/html/pub>
Options FollowSymLinks
AllowOverride All
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
SetEnvIf X-Forwarded-Proto https HTTPS=on
RewriteEngine on
RewriteCond %{SERVER_NAME} =domainname.com [OR]
RewriteCond %{SERVER_NAME} =www.domainname.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
my apache error log shows
[Sun Sep 25 12:08:54.528941 2022] [proxy_http:error] [pid 51038] [client IP:19124] AH01114: HTTP: failed to make connection to backend: 127.0.0.1
the 503 error showing even in Phpmyadmin, so i guess this error has nothing to do with Magento
what could be the issue?
Edit:
sudo varnishlog -g request -q "RespStatus == 503"
* << Request >> 2
- Begin req 1 rxreq
- Timestamp Start: 1664111575.119591 0.000000 0.000000
- Timestamp Req: 1664111575.119591 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 7840 a0
- ReqMethod GET
- ReqURL /en/stores/store/redirect/___store/fr/___from_store/en/uenc/aHR0cHM6Ly83eWFhay5jb20vYXIvY3VzdG9tZXIvYWNjb3VudC9sb2dpblBvc3QvcmVmZXJlci9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZaVzR2WTIxekwyNXZjbTkxZEdVdmFXNWtaWGd2ZFdWdVl5OWhTRkl3WTBoTk5reDVPRE5sVjBab1lYazFhbUl5TU
- ReqProtocol HTTP/1.1
- ReqHeader Host: domain.com
- ReqHeader Accept: text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml
- ReqHeader Accept-Encoding: gzip,deflate
- ReqHeader User-Agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: 185.191.171.40
- ReqHeader X-Forwarded-Host: domain.com
- ReqHeader X-Forwarded-Server: domain.com
- ReqHeader Connection: Keep-Alive
- ReqUnset X-Forwarded-For: 185.191.171.40
- ReqHeader X-Forwarded-For: 185.191.171.40, 127.0.0.1
- VCL_call RECV
- ReqHeader grace: none
- ReqURL /en/stores/store/redirect/___store/fr/___from_store/en/uenc/aHR0cHM6Ly83eWFhay5jb20vYXIvY3VzdG9tZXIvYWNjb3VudC9sb2dpblBvc3QvcmVmZXJlci9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZaVzR2WTIxekwyNXZjbTkxZEdVdmFXNWtaWGd2ZFdWdVl5OWhTRkl3WTBoTk5reDVPRE5sVjBab1lYazFhbUl5TU
- ReqUnset Accept-Encoding: gzip,deflate
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 3 fetch
- Timestamp Fetch: 1664111575.120014 0.000422 0.000422
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Sun, 25 Sep 2022 13:12:55 GMT
- RespHeader Server: Varnish
- RespHeader content-type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 2
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.4)
- VCL_call DELIVER
- RespUnset Age: 0
- RespHeader Pragma: no-cache
- RespHeader Expires: -1
- RespHeader Cache-Control: no-store, no-cache, must-revalidate, max-age=0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 2
- RespUnset Via: 1.1 varnish (Varnish/6.4)
- VCL_return deliver
- Timestamp Process: 1664111575.120075 0.000484 0.000061
- Filters
- RespHeader Content-Length: 278
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664111575.120135 0.000544 0.000060
- ReqAcct 791 0 791 269 278 547
- End
** << BeReq >> 3
-- Begin bereq 2 fetch
-- VCL_use boot
-- Timestamp Start: 1664111575.119704 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /en/stores/store/redirect/___store/fr/___from_store/en/uenc/aHR0cHM6Ly83eWFhay5jb20vYXIvY3VzdG9tZXIvYWNjb3VudC9sb2dpblBvc3QvcmVmZXJlci9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZaVzR2WTIxekwyNXZjbTkxZEdVdmFXNWtaWGd2ZFdWdVl5OWhTRkl3WTBoTk5reDVPRE5sVjBab1lYazFhbUl5TU
-- BereqProtocol HTTP/1.1
-- BereqHeader Host: domain.com
-- BereqHeader Accept: text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml
-- BereqHeader User-Agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Forwarded-Host: domain.com
-- BereqHeader X-Forwarded-Server: domain.com
-- BereqHeader X-Forwarded-For: 185.191.171.40, 127.0.0.1
-- BereqHeader grace: none
-- BereqHeader Accept-Encoding: gzip
-- BereqHeader X-Varnish: 3
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- FetchError backend default: unhealthy
-- Timestamp Beresp: 1664111575.119731 0.000027 0.000027
-- Timestamp Error: 1664111575.119732 0.000028 0.000000
-- BerespProtocol HTTP/1.1
-- BerespStatus 503
-- BerespReason Backend fetch failed
-- BerespHeader Date: Sun, 25 Sep 2022 13:12:55 GMT
-- BerespHeader Server: Varnish
-- VCL_call BACKEND_ERROR
-- BerespHeader content-type: text/html; charset=utf-8
-- BerespHeader Retry-After: 5
-- VCL_return deliver
-- Storage malloc Transient
-- Length 278
-- BereqAcct 0 0 0 0 0 0
-- End
* << Request >> 5
- Begin req 4 rxreq
- Timestamp Start: 1664111597.397651 0.000000 0.000000
- Timestamp Req: 1664111597.397651 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 47420 a0
- ReqMethod GET
- ReqURL /robots.txt
- ReqProtocol HTTP/1.1
- ReqHeader Host: domain.com
- ReqHeader Accept: text/plain,text/html,*/*
- ReqHeader User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: 66.249.66.47
- ReqHeader X-Forwarded-Host: domain.com
- ReqHeader X-Forwarded-Server: domain.com
- ReqHeader Connection: Keep-Alive
- ReqUnset X-Forwarded-For: 66.249.66.47
- ReqHeader X-Forwarded-For: 66.249.66.47, 127.0.0.1
- VCL_call RECV
- ReqHeader grace: none
- ReqURL /robots.txt
- ReqUnset Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 6 fetch
- Timestamp Fetch: 1664111597.397807 0.000156 0.000156
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Sun, 25 Sep 2022 13:13:17 GMT
- RespHeader Server: Varnish
- RespHeader content-type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 5
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.4)
- VCL_call DELIVER
- RespUnset Age: 0
- RespHeader Pragma: no-cache
- RespHeader Expires: -1
- RespHeader Cache-Control: no-store, no-cache, must-revalidate, max-age=0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 5
- RespUnset Via: 1.1 varnish (Varnish/6.4)
- VCL_return deliver
- Timestamp Process: 1664111597.397819 0.000167 0.000011
- Filters
- RespHeader Content-Length: 278
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664111597.397856 0.000205 0.000037
- ReqAcct 342 0 342 269 278 547
- End
** << BeReq >> 6
-- Begin bereq 5 fetch
-- VCL_use boot
-- Timestamp Start: 1664111597.397716 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /robots.txt
-- BereqProtocol HTTP/1.1
-- BereqHeader Host: domain.com
-- BereqHeader Accept: text/plain,text/html,*/*
-- BereqHeader User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Forwarded-Host: domain.com
-- BereqHeader X-Forwarded-Server: domain.com
-- BereqHeader X-Forwarded-For: 66.249.66.47, 127.0.0.1
-- BereqHeader grace: none
-- BereqHeader Accept-Encoding: gzip
-- BereqHeader X-Varnish: 6
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- FetchError backend default: unhealthy
-- Timestamp Beresp: 1664111597.397732 0.000015 0.000015
-- Timestamp Error: 1664111597.397733 0.000016 0.000000
-- BerespProtocol HTTP/1.1
-- BerespStatus 503
-- BerespReason Backend fetch failed
-- BerespHeader Date: Sun, 25 Sep 2022 13:13:17 GMT
-- BerespHeader Server: Varnish
-- VCL_call BACKEND_ERROR
-- BerespHeader content-type: text/html; charset=utf-8
-- BerespHeader Retry-After: 5
-- VCL_return deliver
-- Storage malloc Transient
-- Length 278
-- BereqAcct 0 0 0 0 0 0
-- End
* << Request >> 32770
- Begin req 32769 rxreq
- Timestamp Start: 1664111601.616549 0.000000 0.000000
- Timestamp Req: 1664111601.616549 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 47432 a0
- ReqMethod GET
- ReqURL /fr/stores/store/redirect/___store/en/___from_store/fr/uenc/aHR0cHM6Ly83eWFhay5jb20vZW4vY2hlY2tvdXQvY2FydC9hZGQvdWVuYy9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZZWEl2YUc5MWMyVm9iMnhrTG1oMGJXd19jRDB6L3Byb2R1Y3QvNDEzLw,,/
- ReqProtocol HTTP/1.1
- ReqHeader Host: domain.com
- ReqHeader Accept: text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml
- ReqHeader Accept-Encoding: gzip,deflate
- ReqHeader User-Agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: 185.191.171.22
- ReqHeader X-Forwarded-Host: domain.com
- ReqHeader X-Forwarded-Server: domain.com
- ReqHeader Connection: Keep-Alive
- ReqUnset X-Forwarded-For: 185.191.171.22
- ReqHeader X-Forwarded-For: 185.191.171.22, 127.0.0.1
- VCL_call RECV
- ReqHeader grace: none
- ReqURL /fr/stores/store/redirect/___store/en/___from_store/fr/uenc/aHR0cHM6Ly83eWFhay5jb20vZW4vY2hlY2tvdXQvY2FydC9hZGQvdWVuYy9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZZWEl2YUc5MWMyVm9iMnhrTG1oMGJXd19jRDB6L3Byb2R1Y3QvNDEzLw,,/
- ReqUnset Accept-Encoding: gzip,deflate
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 32771 fetch
- Timestamp Fetch: 1664111601.616769 0.000220 0.000220
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Sun, 25 Sep 2022 13:13:21 GMT
- RespHeader Server: Varnish
- RespHeader content-type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 32770
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.4)
- VCL_call DELIVER
- RespUnset Age: 0
- RespHeader Pragma: no-cache
- RespHeader Expires: -1
- RespHeader Cache-Control: no-store, no-cache, must-revalidate, max-age=0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 32770
- RespUnset Via: 1.1 varnish (Varnish/6.4)
- VCL_return deliver
- Timestamp Process: 1664111601.616782 0.000233 0.000013
- Filters
- RespHeader Content-Length: 282
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664111601.616807 0.000258 0.000024
- ReqAcct 615 0 615 269 282 551
- End
** << BeReq >> 32771
-- Begin bereq 32770 fetch
-- VCL_use boot
-- Timestamp Start: 1664111601.616650 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /fr/stores/store/redirect/___store/en/___from_store/fr/uenc/aHR0cHM6Ly83eWFhay5jb20vZW4vY2hlY2tvdXQvY2FydC9hZGQvdWVuYy9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZZWEl2YUc5MWMyVm9iMnhrTG1oMGJXd19jRDB6L3Byb2R1Y3QvNDEzLw,,/
-- BereqProtocol HTTP/1.1
-- BereqHeader Host: domain.com
-- BereqHeader Accept: text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml
-- BereqHeader User-Agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Forwarded-Host: domain.com
-- BereqHeader X-Forwarded-Server: domain.com
-- BereqHeader X-Forwarded-For: 185.191.171.22, 127.0.0.1
-- BereqHeader grace: none
-- BereqHeader Accept-Encoding: gzip
-- BereqHeader X-Varnish: 32771
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- FetchError backend default: unhealthy
-- Timestamp Beresp: 1664111601.616665 0.000014 0.000014
-- Timestamp Error: 1664111601.616665 0.000015 0.000000
-- BerespProtocol HTTP/1.1
-- BerespStatus 503
-- BerespReason Backend fetch failed
-- BerespHeader Date: Sun, 25 Sep 2022 13:13:21 GMT
-- BerespHeader Server: Varnish
-- VCL_call BACKEND_ERROR
-- BerespHeader content-type: text/html; charset=utf-8
-- BerespHeader Retry-After: 5
-- VCL_return deliver
-- Storage malloc Transient
-- Length 282
-- BereqAcct 0 0 0 0 0 0
-- End
* << Request >> 8
- Begin req 7 rxreq
- Timestamp Start: 1664111607.366603 0.000000 0.000000
- Timestamp Req: 1664111607.366603 0.000000 0.000000
- VCL_use boot
- ReqStart 127.0.0.1 13422 a0
- ReqMethod GET
- ReqURL /en/stores/store/redirect/___store/fr/___from_store/en/uenc/aHR0cHM6Ly83eWFhay5jb20vYXIvY2hlY2tvdXQvY2FydC9hZGQvdWVuYy9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZaVzR2WW5KaGJtUXZhRzU2TG1oMGJXdywvcHJvZHVjdC8xMTMwLw,,/
- ReqProtocol HTTP/1.1
- ReqHeader Host: domain.com
- ReqHeader Accept: text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml
- ReqHeader Accept-Encoding: gzip,deflate
- ReqHeader User-Agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-For: 185.191.171.5
- ReqHeader X-Forwarded-Host: domain.com
- ReqHeader X-Forwarded-Server: domain.com
- ReqHeader Connection: Keep-Alive
- ReqUnset X-Forwarded-For: 185.191.171.5
- ReqHeader X-Forwarded-For: 185.191.171.5, 127.0.0.1
- VCL_call RECV
- ReqHeader grace: none
- ReqURL /en/stores/store/redirect/___store/fr/___from_store/en/uenc/aHR0cHM6Ly83eWFhay5jb20vYXIvY2hlY2tvdXQvY2FydC9hZGQvdWVuYy9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZaVzR2WW5KaGJtUXZhRzU2TG1oMGJXdywvcHJvZHVjdC8xMTMwLw,,/
- ReqUnset Accept-Encoding: gzip,deflate
- ReqHeader Accept-Encoding: gzip
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- Link bereq 9 fetch
- Timestamp Fetch: 1664111607.366837 0.000233 0.000233
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Sun, 25 Sep 2022 13:13:27 GMT
- RespHeader Server: Varnish
- RespHeader content-type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 8
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.4)
- VCL_call DELIVER
- RespUnset Age: 0
- RespHeader Pragma: no-cache
- RespHeader Expires: -1
- RespHeader Cache-Control: no-store, no-cache, must-revalidate, max-age=0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 8
- RespUnset Via: 1.1 varnish (Varnish/6.4)
- VCL_return deliver
- Timestamp Process: 1664111607.366855 0.000252 0.000018
- Filters
- RespHeader Content-Length: 278
- RespHeader Connection: keep-alive
- Timestamp Resp: 1664111607.366893 0.000289 0.000037
- ReqAcct 610 0 610 269 278 547
- End
** << BeReq >> 9
-- Begin bereq 8 fetch
-- VCL_use boot
-- Timestamp Start: 1664111607.366701 0.000000 0.000000
-- BereqMethod GET
-- BereqURL /en/stores/store/redirect/___store/fr/___from_store/en/uenc/aHR0cHM6Ly83eWFhay5jb20vYXIvY2hlY2tvdXQvY2FydC9hZGQvdWVuYy9hSFIwY0hNNkx5ODNlV0ZoYXk1amIyMHZaVzR2WW5KaGJtUXZhRzU2TG1oMGJXdywvcHJvZHVjdC8xMTMwLw,,/
-- BereqProtocol HTTP/1.1
-- BereqHeader Host: domain.com
-- BereqHeader Accept: text/html, application/rss+xml, application/atom+xml, text/xml, text/rss+xml, application/xhtml+xml
-- BereqHeader User-Agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Forwarded-Host: domain.com
-- BereqHeader X-Forwarded-Server: domain.com
-- BereqHeader X-Forwarded-For: 185.191.171.5, 127.0.0.1
-- BereqHeader grace: none
-- BereqHeader Accept-Encoding: gzip
-- BereqHeader X-Varnish: 9
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- FetchError backend default: unhealthy
-- Timestamp Beresp: 1664111607.366723 0.000021 0.000021
-- Timestamp Error: 1664111607.366724 0.000022 0.000001
-- BerespProtocol HTTP/1.1
-- BerespStatus 503
-- BerespReason Backend fetch failed
-- BerespHeader Date: Sun, 25 Sep 2022 13:13:27 GMT
-- BerespHeader Server: Varnish
-- VCL_call BACKEND_ERROR
-- BerespHeader content-type: text/html; charset=utf-8
-- BerespHeader Retry-After: 5
-- VCL_return deliver
-- Storage malloc Transient
-- Length 278
-- BereqAcct 0 0 0 0 0 0
-- End

Answer 1

Backend unhealthy
I noticed the following log line in your log output:
-- FetchError backend default: unhealthy
This means that your backend is administratively set to an unhealthy state because too many health checks failed.
See https://www.varnish-software.com/developers/tutorials/troubleshooting-varnish/#backend-health-monitoring for a handy tutorial about backend health monitoring in Varnish
You could run sudo varnishlog -g raw -i Backend_health to validate this and see how many failures have taken place.
Backend health criteria
The criteria for the backend to be considered healthy in your configuration are the following:
The backend should connect within 3.5 seconds (default value)
The first byte should be received within 600 seconds after sending the backend request
The response should have a 200 status code
5 (the threshold value) out of 10 (the window value) health checks should succeed
Debugging the health check probe
In your VCL code /health_check.php is the endpoint of the health check probe. Run the following command to debug what's going on when Varnish loads it every 5 seconds:
sudo varnishlog -g request -q "ReqUrl eq '/health_check.php'"
You can add the output of this command to your question and I'll help you debug.
However, if the returned status code for these health checks is 404, it means you're using the wrong URL and you need to fix that first.
See https://www.varnish-software.com/developers/tutorials/configuring-varnish-magento/#fixing-the-backend-health-checks-for-magento-24 for a tutorial section about a common mistake in Magento when using health_check.php.