Kubernetes behind proxy misses CSRF token - apache

Question

I have a problem. On my local laptop I can build my docker image and run it. A Ruby-On-Rails application is then running on localhost:3000. Going to that page gives me a login screen and I can successfully login. Now when I publish this code to my docker repository and run it in a kubernetes cluster, I get the following error:
With status code:
Status Code: 422 Unprocessable Entity
After some googling and debugging, I read some stuff about having the correct headers in Nginx-Ingress and Apache to handle the CSRF requests. Using this I tried many ways of allowing headers and manually adding RequestHeaders, but all resulted in the same error.
In both pages I can see in the network tab the cookie in the page. The cookie in local environment contains:
csrftoken=7tzlvuigfaUatDBMTEJUBE53OREXHxOJ1ZzJ8h4cuHvnvUza0xLH2lglIFZ1gS4S; __profilin=p=t; _app_session=BBr2bnKi6qYJwW4yvQUU0J3Q2K8JW2i9DZ5zJS0qQmCc48uRNbPrThJhbzhnGXrxLDwD/Y8ZAW5w/x2VhXeotq3TWjWcjVucUzNPA6u6ePMcN8O7XVfcIwV2HR/aGpJ1oxB1B/TbsAcfB4LaSqckVWMZ70anWDOD+xFz0xkKPiOkd++NwnMuWix2TyaicqXmQ+JsK1yWacS2dnTWvC70xyX38JjuC/VUqu2hUFIQHeNuK7Gfp7lMCh0MEJeob43lFhgFZLOJXTC+rPYIUxMTN3aIwD0=--LsM3EqEzNEglvj6H--RHcx/5CRIYG/CWJMUER1eQ==
In my production environment, I can see that the entire csrftoken part is missing:
__profilin=p=t; _app_session=FWpoSIJ8zzB8nkRsF2A30f552HjyAVlwbMva/HN4vnKaTECnPWzfWoc/+P9k/xO+fpmd2j9ntxLFOnflB5vttwxpLL1RtjZWcPmpCWfdR0nqG3EOLPLK9yR0xrvmfKMbFsYeJ2F/hZrJIs+4iOO92CioXo1s51sZX5E3R6c9XIKn5NjHUKWKuEArVV/4MibTg/CSqPWyH3lg1bjxKoZvixSZUBdtxTq1sW5NKsJnnwPgC0ZSkQzsSquCwcxKEf9L1wPxgZQTLGRREY7JeD/+mUc0WL4=--n+JNO4hoF17EGI5K--vY2DcanQEF7ICpZwDxq9tg==
Why is the CSRF token not comming through in a production environment? I already tried setting RAILS_ENV to development in the production environment, so it uses a different config, but that also gives the Status Code: 422 Unprocessable Entity
IMPORTANT
All trafic goes to my portal proxy server, which runs Apache2 and handles the SSL certificates for my domain. Apache forwards the traffic in my internal network using HTTP. Nginx-Ingress-Controller only handles http traffic because of this setup. I am not really sure if Nginx is the problem then, or Apache2?
How can I fix this CSRF issue ASAP?
UPDATE
I looked into the rails logs and found the following which might cause the problem:
I, [2022-12-12T08:16:40.202890 #1] INFO -- : [f785b7f7f797c67c0ab0f008e8a0eb83] Started POST "/login" for 192.168.1.10 at 2022-12-12 08:16:40 +0000
I, [2022-12-12T08:16:40.204343 #1] INFO -- : [f785b7f7f797c67c0ab0f008e8a0eb83] Processing by AuthController#login as HTML
I, [2022-12-12T08:16:40.204460 #1] INFO -- : [f785b7f7f797c67c0ab0f008e8a0eb83] Parameters: {"authenticity_token"=>"[FILTERED]", "login"=>{"email"=>"myemail#gmail.com", "password"=>"[F
W, [2022-12-12T08:16:40.204795 #1] WARN -- : [f785b7f7f797c67c0ab0f008e8a0eb83] HTTP Origin header (https://my.site.com) didn't match request.base_url (http://my.site.com)
I, [2022-12-12T08:16:40.205536 #1] INFO -- : [f785b7f7f797c67c0ab0f008e8a0eb83] Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Allocations: 192)
F, [2022-12-12T08:16:40.207591 #1] FATAL -- : [f785b7f7f797c67c0ab0f008e8a0eb83]
In my Ingress resource I now have the following:
annotations:
nginx.org/proxy-connect-timeout: 3600s
nginx.org/proxy-read-timeout: 3600s
nginx.org/proxy-send-timeout: 3600s
nginx.org/proxy-pass-headers: "Set-Cookie"
nginx.ingress.kubernetes.io/proxy-body-size: "8g"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Host $http_host";
more_set_headers "X-Real-IP $remote_addr";
more_set_headers "X-Forwarded-Proto $scheme";
more_set_headers "X-Forwarded-For $proxy_add_x_forwarded_for";
And my apache conf looks like this:
<VirtualHost *:80>
ServerName my.site.com
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPreserveHost on
ProxyPass / http://192.168.1.8/
ProxyPassReverse / http://192.168.1.8
RewriteEngine on
RewriteCond %{SERVER_NAME} =my.site.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName my.site.com
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
SSLProxyEngine On
SSLProxyCheckPeerCN on
SSLProxyCheckPeerExpire on
ProxyPreserveHost on
ProxyPass / http://192.168.1.8/
ProxyPassReverse / http://192.168.1.8
Include /etc/letsencrypt/options-ssl-apache.conf
ErrorLog /var/log/apache2/sites/my.site.com/error.log
CustomLog /var/log/apache2/sites/my.site.com/access.log combined
SSLCertificateFile /etc/letsencrypt/live/my.site.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/my.site.com/privkey.pem
</VirtualHost>
</IfModule>
I already tried adding to my Apache conf:
RequestHeader set X-Forwarded-Proto "https"
but that doesn't change anything...
Can someone tell me what I should change here to get this to work?

Answer 1

I would propose adding the origin header in nginx. You probably also have to set it in your apache configuration to the external URL.
annotations:
nginx.org/proxy-connect-timeout: 3600s
nginx.org/proxy-read-timeout: 3600s
nginx.org/proxy-send-timeout: 3600s
nginx.org/proxy-pass-headers: "Set-Cookie"
nginx.ingress.kubernetes.io/proxy-body-size: "8g"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Host $http_host";
more_set_headers "X-Real-IP $remote_addr";
more_set_headers "X-Forwarded-Proto $scheme";
more_set_headers "X-Forwarded-For $proxy_add_x_forwarded_for";
more_set_headers "origin $remote_addr";