We've configured NGINX to use mutual authentication. When a client makes a request to us, we get an info line in the NGINX log saying, "client sent no required SSL certificate while reading client request headers". We believe that the client actually is sending a certificate.
The evidence is that we saw the curl command the client used and it was sending a cert. We correlated their call to this log message. Also, we have an F5 proxy that is configured to use mutual auth and their same curl command is accepted if they change it to hit the F5. This call would be rejected if they were not sending a client cert. In the F5 logs we see that F5 is receiving a client cert. The client is not changing its behavior, we are changing the DNS to point between F5 and NGINX.
When I google for "client sent no required SSL certificate while reading client request headers" I can't find any official NGINX documentation about this message. How can I troubleshoot this better? Right now we're starting nginx with nginx-debug but the debug output doesn't seem to clearly explain why we're seeing this problem. Here are some of these logs:
2016/12/19 23:27:59 [debug] 179#179: epoll: fd:6 ev:0001 d:0000000000C7AEB0
2016/12/19 23:27:59 [debug] 179#179: accept on 0.0.0.0:443, ready: 0
2016/12/19 23:27:59 [debug] 179#179: posix_memalign: 0000000000B8D530:512 #16
2016/12/19 23:27:59 [debug] 179#179: *4539 accept: 172.20.72.125:23211 fd:3
2016/12/19 23:27:59 [debug] 179#179: *4539 event timer add: 3: 60000:1482190139859
2016/12/19 23:27:59 [debug] 179#179: *4539 reusable connection: 1
2016/12/19 23:27:59 [debug] 179#179: *4539 epoll add event: fd:3 op:1 ev:80002001
2016/12/19 23:27:59 [debug] 179#179: timer delta: 873
2016/12/19 23:27:59 [debug] 179#179: worker cycle
2016/12/19 23:27:59 [debug] 179#179: epoll timer: 60000
2016/12/19 23:27:59 [debug] 179#179: epoll: fd:3 ev:0001 d:0000000000C7B360
2016/12/19 23:27:59 [debug] 179#179: *4539 http check ssl handshake
2016/12/19 23:27:59 [debug] 179#179: *4539 http recv(): 1
2016/12/19 23:27:59 [debug] 179#179: *4539 https ssl handshake: 0x16
2016/12/19 23:27:59 [debug] 181#181: accept on 0.0.0.0:443, ready: 0
2016/12/19 23:27:59 [debug] 181#181: accept() not ready (11: Resource temporarily unavailable)
2016/12/19 23:27:59 [debug] 179#179: *4539 ssl get session: DB2C8809:32
2016/12/19 23:27:59 [debug] 179#179: shmtx lock
2016/12/19 23:27:59 [debug] 179#179: shmtx unlock
2016/12/19 23:27:59 [debug] 181#181: timer delta: 873
2016/12/19 23:27:59 [debug] 181#181: worker cycle
2016/12/19 23:27:59 [debug] 181#181: epoll timer: -1
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_do_handshake: -1
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_get_error: 2
2016/12/19 23:27:59 [debug] 179#179: *4539 reusable connection: 0
2016/12/19 23:27:59 [debug] 179#179: timer delta: 0
2016/12/19 23:27:59 [debug] 179#179: worker cycle
2016/12/19 23:27:59 [debug] 179#179: epoll timer: 60000
2016/12/19 23:27:59 [debug] 179#179: epoll: fd:3 ev:0001 d:0000000000C7B360
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL handshake handler: 0
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_do_handshake: -1
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_get_error: 2
2016/12/19 23:27:59 [debug] 179#179: timer delta: 29
2016/12/19 23:27:59 [debug] 179#179: worker cycle
2016/12/19 23:27:59 [debug] 179#179: epoll timer: 59971
2016/12/19 23:27:59 [debug] 179#179: epoll: fd:3 ev:0001 d:0000000000C7B360
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL handshake handler: 0
2016/12/19 23:27:59 [debug] 179#179: shmtx lock
2016/12/19 23:27:59 [debug] 179#179: slab alloc: 136 slot: 5
2016/12/19 23:27:59 [debug] 179#179: slab alloc: 00007FF33D86B000
2016/12/19 23:27:59 [debug] 179#179: slab alloc: 128 slot: 4
2016/12/19 23:27:59 [debug] 179#179: slab alloc: 00007FF33D869080
2016/12/19 23:27:59 [debug] 179#179: *4539 ssl new session: B0945ECD:32:136
2016/12/19 23:27:59 [debug] 179#179: shmtx unlock
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_do_handshake: 1
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL: TLSv1.1, cipher: "ECDHE-RSA-AES256-SHA SSLv3 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA1"
2016/12/19 23:27:59 [debug] 179#179: *4539 reusable connection: 1
2016/12/19 23:27:59 [debug] 179#179: *4539 http wait request handler
2016/12/19 23:27:59 [debug] 179#179: *4539 malloc: 0000000000B89230:1024
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_read: -1
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_get_error: 2
2016/12/19 23:27:59 [debug] 179#179: *4539 free: 0000000000B89230
2016/12/19 23:27:59 [debug] 179#179: timer delta: 3
2016/12/19 23:27:59 [debug] 179#179: worker cycle
2016/12/19 23:27:59 [debug] 179#179: epoll timer: 59968
2016/12/19 23:27:59 [debug] 179#179: epoll: fd:3 ev:0001 d:0000000000C7B360
2016/12/19 23:27:59 [debug] 179#179: *4539 http wait request handler
2016/12/19 23:27:59 [debug] 179#179: *4539 malloc: 0000000000B89230:1024
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_read: 172
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_read: -1
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_get_error: 2
2016/12/19 23:27:59 [debug] 179#179: *4539 reusable connection: 0
2016/12/19 23:27:59 [debug] 179#179: *4539 posix_memalign: 0000000000C71800:4096 #16
2016/12/19 23:27:59 [debug] 179#179: *4539 http process request line
2016/12/19 23:27:59 [debug] 179#179: *4539 http request line: "GET /myapp-myapi/v2/id12345/endpoint HTTP/1.1"
2016/12/19 23:27:59 [debug] 179#179: *4539 http uri: "/myapp-myapi/v2/id12345/endpoint"
2016/12/19 23:27:59 [debug] 179#179: *4539 http args: ""
2016/12/19 23:27:59 [debug] 179#179: *4539 http exten: ""
2016/12/19 23:27:59 [debug] 179#179: *4539 http process request header line
2016/12/19 23:27:59 [debug] 179#179: *4539 http header: "a-request-header: client-qa"
2016/12/19 23:27:59 [debug] 179#179: *4539 posix_memalign: 0000000000B9C640:4096 #16
2016/12/19 23:27:59 [debug] 179#179: *4539 http header: "User-Agent: Jakarta Commons-HttpClient/3.1"
2016/12/19 23:27:59 [debug] 179#179: *4539 http header: "Host: pre.myapp.com"
2016/12/19 23:27:59 [debug] 179#179: *4539 http header done
2016/12/19 23:27:59 [info] 179#179: *4539 client sent no required SSL certificate while reading client request headers, client: 172.20.72.125, server: pre.myapp.com, request: "GET /myapp-myapi/v2/id12345/endpoint HTTP/1.1", host: "pre.myapp.com"
2016/12/19 23:27:59 [debug] 179#179: ssl remove session: B0945ECD:32
2016/12/19 23:27:59 [debug] 179#179: shmtx lock
2016/12/19 23:27:59 [debug] 179#179: slab free: 00007FF33D86B000
2016/12/19 23:27:59 [debug] 179#179: slab free: 00007FF33D869080
2016/12/19 23:27:59 [debug] 179#179: shmtx unlock
2016/12/19 23:27:59 [debug] 179#179: *4539 http finalize request: 496, "/myapp-myapi/v2/id12345/endpoint?" a:1, c:1
2016/12/19 23:27:59 [debug] 179#179: *4539 event timer del: 3: 1482190139859
2016/12/19 23:27:59 [debug] 179#179: *4539 http special response: 496, "/myapp-myapi/v2/id12345/endpoint?"
2016/12/19 23:27:59 [debug] 179#179: *4539 http set discard body
2016/12/19 23:27:59 [debug] 179#179: *4539 HTTP/1.1 400 Bad Request
Server: nginx/1.11.4
Date: Mon, 19 Dec 2016 23:27:59 GMT
Content-Type: text/html
Content-Length: 253
Connection: close
2016/12/19 23:27:59 [debug] 179#179: *4539 write new buf t:1 f:0 0000000000B9C6C0, pos 0000000000B9C6C0, size: 152 file: 0, size: 0
2016/12/19 23:27:59 [debug] 179#179: *4539 http write filter: l:0 f:0 s:152
2016/12/19 23:27:59 [debug] 179#179: *4539 http output filter "/myapp-myapi/v2/id12345/endpoint?"
2016/12/19 23:27:59 [debug] 179#179: *4539 http copy filter: "/myapp-myapi/v2/id12345/endpoint?"
2016/12/19 23:27:59 [debug] 179#179: *4539 http postpone filter "/myapp-myapi/v2/id12345/endpoint?" 0000000000B9C8A0
2016/12/19 23:27:59 [debug] 179#179: *4539 write old buf t:1 f:0 0000000000B9C6C0, pos 0000000000B9C6C0, size: 152 file: 0, size: 0
2016/12/19 23:27:59 [debug] 179#179: *4539 write new buf t:0 f:0 0000000000000000, pos 0000000000711B80, size: 200 file: 0, size: 0
2016/12/19 23:27:59 [debug] 179#179: *4539 write new buf t:0 f:0 0000000000000000, pos 0000000000712DE0, size: 53 file: 0, size: 0
2016/12/19 23:27:59 [debug] 179#179: *4539 http write filter: l:1 f:0 s:405
2016/12/19 23:27:59 [debug] 179#179: *4539 http write filter limit 0
2016/12/19 23:27:59 [debug] 179#179: *4539 posix_memalign: 0000000000BF6100:512 #16
2016/12/19 23:27:59 [debug] 179#179: *4539 malloc: 0000000000C01FE0:16384
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL buf copy: 152
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL buf copy: 200
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL buf copy: 53
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL to write: 405
2016/12/19 23:27:59 [debug] 179#179: *4539 SSL_write: 405
2016/12/19 23:27:59 [debug] 179#179: *4539 http write filter 0000000000000000
2016/12/19 23:27:59 [debug] 179#179: *4539 http copy filter: 0 "/myapp-myapi/v2/id12345/endpoint?"
2016/12/19 23:27:59 [debug] 179#179: *4539 http finalize request: 0, "/myapp-myapi/v2/id12345/endpoint?" a:1, c:1
2016/12/19 23:27:59 [debug] 179#179: *4539 http request count:1 blk:0
2016/12/19 23:27:59 [debug] 179#179: *4539 http close request
2016/12/19 23:27:59 [debug] 179#179: *4539 http log handler
172.20.72.125 - - [19/Dec/2016:23:27:59 +0000] https "GET /myapp-myapi/v2/id12345/endpoint HTTP/1.1" 400 253 "-" "Jakarta Commons-HttpClient/3.1" "-" "-" "NONE" "" "client-qa"
This is our nginx.conf file:
#daemon off;
user nginx;
worker_processes 4;
error_log /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
use epoll;
}
http {
index index.php index.htm index.html;
include /etc/nginx/mime.types;
upstream backend-myapi {
server myapp-myapi:8087 max_fails=0 fail_timeout=0s;
server myapp-myapi:8087 max_fails=0 fail_timeout=0s;
}
map $a_request_header|$ssl_client_verify $ssl_common_name {
default $ssl_client_s_dn;
40011|NONE CN=mycn;
}
ssl_protocols TLSv1 TLSv1.1;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_certificate /etc/secrets/servercert-legacy;
ssl_certificate_key /etc/secrets/serverkey-legacy;
ssl_client_certificate /etc/nginx/ca.crt;
proxy_set_header Host $host;
proxy_set_header SSL-COMMON-NAME $ssl_common_name; # TODO change this header to just DN
proxy_set_header VERIFIED $ssl_client_verify;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_max_temp_file_size 0;
proxy_connect_timeout 30;
proxy_send_timeout 30;
proxy_read_timeout 300;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_next_upstream error http_502;
proxy_temp_file_write_size 64k;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] $scheme "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'"$ssl_client_s_dn" "$ssl_client_verify" '
'"$ssl_common_name" "$a_request_header"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
client_max_body_size 10m;
client_body_buffer_size 128k;
large_client_header_buffers 4 16k;
# gzip on;
# gzip_buffers 16 8k;
# gzip_comp_level 3;
# gzip_disable "msie6";
# gzip_http_version 1.0;
# gzip_min_length 1024;
# gzip_proxied any;
# gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
# gzip_vary on;
include /etc/nginx/conf.d/*.conf;
}
We are using nginx/1.11.4.
I don't see a configuration there that requires ssl client auth. Try adding
ssl_verify_client on
ssl_verify_depth 3;
ssl_client_certificate /path/to/accepted/CAs.pem;
(from http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_verify_client)
Related
I deployed a PWA app with my nginx server and I have no problems connecting with my laptop using chrome or safari. I only get an issue when I re-connect with my iPhone and with a particular file only: sw.js (service worker).
Since it is fundamental for a PWA to get this file downloaded in order to decide whether a new version of the app is available or not, having to clear the cache from safari iOS in order to get it done is really annoying.
ok so, let me explain:
The app is hosted on a nginx server with TLS + Mutual TSL.
Each client I'm connecting from has been configured with both the certs and works fine, so I guess this is not a cert problem (neither from the TLS nor from the M-TLS).
If I connect from my laptop with chrome or safari, I have no problems at all.
2023/01/16 08:54:18 [debug] 19158#19158: *4 http process request line
2023/01/16 08:54:18 [debug] 19158#19158: *4 http request line: "GET /sw.js HTTP/1.1"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http uri: "/sw.js"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http args: ""
2023/01/16 08:54:18 [debug] 19158#19158: *4 http exten: "js"
2023/01/16 08:54:18 [debug] 19158#19158: *4 posix_memalign: 00005566D933EEA0:4096 #16
2023/01/16 08:54:18 [debug] 19158#19158: *4 http process request header line
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Host: xx.xx.net:4765"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Connection: keep-alive"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Cache-Control: max-age=0"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Accept: */*"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Service-Worker: script"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Sec-Fetch-Site: same-origin"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Sec-Fetch-Mode: same-origin"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Sec-Fetch-Dest: serviceworker"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Referer: https://xx.xx.net:4765/sw.js"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Accept-Encoding: gzip, deflate, br"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "Accept-Language: it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "If-None-Match: "63c47f4b-2945""
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header: "If-Modified-Since: Sun, 15 Jan 2023 22:33:47 GMT"
2023/01/16 08:54:18 [debug] 19158#19158: *4 http header done
if I connect from my iPhone with safari mobile:
1) where connecting the first time with cleared cache: no issues
23/01/16 09:31:55 [debug] 19156#19156: *42 http process request line
2023/01/16 09:31:55 [debug] 19156#19156: *42 http request line: "GET /sw.js HTTP/1.1"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http uri: "/sw.js"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http args: ""
2023/01/16 09:31:55 [debug] 19156#19156: *42 http exten: "js"
2023/01/16 09:31:55 [debug] 19156#19156: *42 posix_memalign: 00005566D93FF330:4096 #16
2023/01/16 09:31:55 [debug] 19156#19156: *42 http process request header line
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Host: xx.xx.net:4765"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Cache-Control: max-age=0"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Accept-Encoding: gzip, deflate, br"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Connection: keep-alive"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Accept: */*"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Accept-Language: it-IT,it;q=0.9"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Referer: https://xx.xx.net:4765/index.html"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header: "Service-Worker: script"
2023/01/16 09:31:55 [debug] 19156#19156: *42 http header done
2023/01/16 09:31:55 [debug] 19156#19156: *42 http filename: "/var/www/html/sw.js"
2023/01/16 09:31:55 [debug] 19156#19156: *42 add cleanup: 00005566D93FF728
2023/01/16 09:31:55 [debug] 19156#19156: *42 http static fd: 23
2023/01/16 09:31:55 [debug] 19156#19156: *42 http set discard body
2023/01/16 09:31:55 [debug] 19156#19156: *42 xslt filter header
2023/01/16 09:31:55 [debug] 19156#19156: *42 **HTTP/1.1 200 OK**
2) where closing the app and then resuming it: I get a 496 on a specific file only: the sw.js (service worker) --> client sent no required SSL certificate while reading client request headers
I can't understand why but it definitely prevents my PWA from being updated.
2023/01/16 09:15:39 [debug] 19156#19156: *38 http request line: "GET /sw.js HTTP/1.1"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http uri: "/sw.js"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http args: ""
2023/01/16 09:15:39 [debug] 19156#19156: *38 http exten: "js"
2023/01/16 09:15:39 [debug] 19156#19156: *38 posix_memalign: 00005566D94111B0:4096 #16
2023/01/16 09:15:39 [debug] 19156#19156: *38 http process request header line
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Host: xx.xx.net:4765"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Origin: https://xx.xx.net:4765"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Accept-Encoding: gzip, deflate, br"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Connection: keep-alive"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Accept: */*"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 Safari/604.1"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Accept-Language: it-IT,it;q=0.9"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Referer: "
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header: "Service-Worker: script"
2023/01/16 09:15:39 [debug] 19156#19156: *38 http header done
2023/01/16 09:15:39 [info] 19156#19156: *38 **client sent no required SSL certificate while reading client request headers, client: 76.87.343.434, server: xx.xx.net, request: "GET /sw.js HTTP/1.1", host: "xx.xx.net:4765", referrer: ""**
2023/01/16 09:15:39 [debug] 19156#19156: *38 http finalize request: 496, "/sw.js?" a:1, c:1
2023/01/16 09:15:39 [debug] 19156#19156: *38 event timer del: 20: 202080132
2023/01/16 09:15:39 [debug] 19156#19156: *38 http special response: 496, "/sw.js?"
2023/01/16 09:15:39 [debug] 19156#19156: *38 internal redirect: "/custom_404.html?"
Here are my config files:
nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
types_hash_max_size 2048;
server_tokens off;
client_body_buffer_size 1k;
client_header_buffer_size 1k;
client_max_body_size 1k;
large_client_header_buffers 2 1k;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log debug;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
site-available file:
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server {
listen 80;
return 301 https://xx.xx.net:port$request_uri;
}
server {
listen 4765 ssl;
server_name xx.xx.net;
# SSL
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# TLS
ssl_certificate /etc/ssh/TLS/cert.crt;
ssl_certificate_key /etc/ssh/TLS/key.key;
# M-TLS
ssl_client_certificate /etc/ssh/mutual-tls.crt;
ssl_verify_client on;
ssl_verify_depth 2;
# ERRORS
error_page 400 404 495 496 497 /custom_404.html;
location = /custom_404.html {
root /usr/share/nginx/html;
internal;
}
location / {
# Simple requests
if ($request_method ~* "(GET|POST)") {
add_header "Access-Control-Allow-Origin" '*';
add_header "Access-Control-Allow-Headers" "Authorization, Origin, X-Requested-With, Content-Type, Accept";
}
# Preflighted requests
if ($request_method = OPTIONS ) {
add_header "Access-Control-Allow-Origin" '*';
add_header "Access-Control-Allow-Methods" "GET, POST, OPTIONS, HEAD";
add_header "Access-Control-Allow-Headers" "Authorization, Origin, X-Requested-With, Content-Type, Accept";
}
}
location ~* \.(html|css|js)$ {
expires -1;
}
}
Strange thing: if I request the sw.js from my browser (typing https://xx.xx.net:sslport/sw.js), I get the file with no errors.
Do you have an idea on what's going on here? maybe is a header /response-header issue?
Update:
I've successfully tried to send the request via Postman: here is the result.
Again: it only happens with iOs, I really don't know how to solve it.
I am using the mvn org.sonatype.ossindex.maven:ossindex-maven-plugin from command line to check the dependencies of a maven project for CVEs. Locally, everything works fine, but in my build pipeline in Azure Devops, I get the following error:
[DEBUG] Connecting to ossindex.sonatype.org/18.118.116.156:443
[DEBUG] Connecting socket to ossindex.sonatype.org/18.118.116.156:443 with timeout 0
[DEBUG] Enabled protocols: [TLSv1.3, TLSv1.2]
[DEBUG] Enabled cipher suites:[TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[DEBUG] Starting handshake
[DEBUG] Secure session established
[DEBUG] negotiated protocol: TLSv1.2
[DEBUG] negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[DEBUG] peer principal: CN=ossindex.sonatype.org
[DEBUG] peer alternative names: [ossindex.sonatype.org]
[DEBUG] issuer principal: CN=Amazon, OU=Server CA 1B, O=Amazon, C=US
[DEBUG] Connection established 172.21.1.242:33030<->18.118.116.156:443
[DEBUG] Executing request POST /api/v3/component-report HTTP/1.1
[DEBUG] Proxy auth state: UNCHALLENGED
[DEBUG] http-outgoing-2 >> POST /api/v3/component-report HTTP/1.1
[DEBUG] http-outgoing-2 >> User-Agent: ossindex-client/1.1.1 (Linux; 5.4.0-124-generic; amd64; 11.0.12) Maven/3.8.1 Maven-Plugin/3.1.0
[DEBUG] http-outgoing-2 >> Authorization: Basic {SOME_BASE_64_THAT_TRANSLATES_TO:AzureDevOps:ey...}
[DEBUG] http-outgoing-2 >> Accept: application/vnd.ossindex.component-report.v1+json
[DEBUG] http-outgoing-2 >> Content-Length: 6737
[DEBUG] http-outgoing-2 >> Content-Type: application/vnd.ossindex.component-report-request.v1+json; charset=UTF-8
[DEBUG] http-outgoing-2 >> Host: ossindex.sonatype.org
[DEBUG] http-outgoing-2 >> Connection: Keep-Alive
[DEBUG] http-outgoing-2 >> Accept-Encoding: gzip,deflate
[DEBUG] http-outgoing-2 >> "POST /api/v3/component-report HTTP/1.1[\r][\n]"
[DEBUG] http-outgoing-2 >> "User-Agent: ossindex-client/1.1.1 (Linux; 5.4.0-124-generic; amd64; 11.0.12) Maven/3.8.1 Maven-Plugin/3.1.0[\r][\n]"
[DEBUG] http-outgoing-2 >> "Authorization: Basic {SOME_OTHER_BASE64_NO_IDEA_WHERE_IT_COMES_FROM=[\r][\n]"
[DEBUG] http-outgoing-2 >> "Accept: application/vnd.ossindex.component-report.v1+json[\r][\n]"
[DEBUG] http-outgoing-2 >> "Content-Length: 6737[\r][\n]"
[DEBUG] http-outgoing-2 >> "Content-Type: application/vnd.ossindex.component-report-request.v1+json; charset=UTF-8[\r][\n]"
[DEBUG] http-outgoing-2 >> "Host: ossindex.sonatype.org[\r][\n]"
[DEBUG] http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]"
[DEBUG] http-outgoing-2 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[DEBUG] http-outgoing-2 >> "[\r][\n]"
[DEBUG] http-outgoing-2 >> "{"coordinates":[ A_LIST_OF_128_COORDINATES]
[DEBUG] http-outgoing-2 << "HTTP/1.1 401 Unauthorized[\r][\n]"
[DEBUG] http-outgoing-2 << "Date: Wed, 07 Dec 2022 13:51:42 GMT[\r][\n]"
[DEBUG] http-outgoing-2 << "Content-Length: 0[\r][\n]"
[DEBUG] http-outgoing-2 << "Connection: keep-alive[\r][\n]"
so I tried setting the clientConfiguration parameter described here.
it has a dead link but I believe it references this class which has this class as a member.
mvn org.sonatype.ossindex.maven:ossindex-maven-plugin:3.1.0:audit -f $(Build.SourcesDirectory)/pom.xml -"Dossindex.authId=MY_SERVER" -"Dossindex.clientConfiguration={\"auth\": {\"username\": \"myemail#myorg.com\", \"password\": \"$(OSS_INDEX_API_TOKEN)\"} }"
but it didn't change anything.
Is this how one passes this parameter? Why am I getting a 401 in my build pipeline but everything works locally? I tried reading the docs on what requests are allowed/authorized but found them rather lacking...
I want something like this:
example.com/anything (public site)
example.com/app/ (vue app)
example.com/api/ (backend API)
So, I there is a folder on Ubuntu server "/srv/root" with static index.html(public site) and "app" folder with vue app:
1. For backend API I just need redirect to local port:
location /api/ {
proxy_pass http://localhost:5000/api/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
2. Public site looks simple:
location / {
root /srv/root/;
try_files $uri $uri/ /index.html;
}
3. Root of vue virtual route looks simple too:
location /app {
root /srv/root;
try_files $uri $uri/ /index.html;
}
location /app/ {
root /srv/root;
try_files $uri $uri/ /index.html;
}
4. Then, for all specific sub-routes(like /app/overview/dashboard) I need to return index.html except vue app resources:
location ~* ^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).+$ {
root /srv/root;
rewrite ^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).*$ /app last;
try_files $uri $uri/ /index.html;
}
The problem is that when I make page refresh on virtual "example.com/app/overview/dashboard" the browser returns root page: "example.com/app/". So seams the vue app doesn't know the source requested url, it's always getting root.
How can I fix it?
/etc/nginx/sites-available/default
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name example.com *.example.com;
return 301 https://$server_name$request_uri;
}
server {
#rewrite_log on;
#error_log /var/log/nginx/example.com.error_log debug;
listen 443 ssl;
server_name example.com *.example.com;
ssl_certificate /etc/example.com/bundle.crt;
ssl_certificate_key /etc/example.com/private.key;
include /etc/nginx/proxy_params;
index index.html;
location / {
root /srv/root/;
try_files $uri $uri/ /index.html;
}
location /app {
root /srv/Floom.Web;
try_files $uri $uri/ /index.html;
}
location /app/ {
root /srv/Floom.Web;
try_files $uri $uri/ /index.html;
}
location ~* ^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).+$ {
root /srv/Floom.Web;
rewrite ^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).*$ /app last;
try_files $uri $uri/ /index.html;
}
location /api/ {
proxy_pass http://localhost:5000/api/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
/var/log/nginx/floom.error_log
2019/11/14 09:34:08 [debug] 13096#13096: epoll add event: fd:9 op:1 ev:10000001
2019/11/14 09:34:08 [debug] 13097#13097: epoll add event: fd:9 op:1 ev:10000001
2019/11/14 09:34:08 [debug] 13098#13098: epoll add event: fd:9 op:1 ev:10000001
2019/11/14 09:34:08 [debug] 13099#13099: epoll add event: fd:9 op:1 ev:10000001
2019/11/14 09:34:16 [debug] 13096#13096: accept on 0.0.0.0:443, ready: 0
2019/11/14 09:34:16 [debug] 13096#13096: posix_memalign: 000055D8FA6FCCC0:512 #16
2019/11/14 09:34:16 [debug] 13096#13096: *1 accept: 213.80.249.126:54496 fd:15
2019/11/14 09:34:16 [debug] 13096#13096: *1 event timer add: 15: 60000:266964738
2019/11/14 09:34:16 [debug] 13096#13096: *1 reusable connection: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 epoll add event: fd:15 op:1 ev:80002001
2019/11/14 09:34:16 [debug] 13096#13096: accept on 0.0.0.0:443, ready: 0
2019/11/14 09:34:16 [debug] 13096#13096: posix_memalign: 000055D8FA6E26B0:512 #16
2019/11/14 09:34:16 [debug] 13096#13096: *2 accept: 213.80.249.126:54497 fd:16
2019/11/14 09:34:16 [debug] 13096#13096: *2 event timer add: 16: 60000:266964738
2019/11/14 09:34:16 [debug] 13096#13096: *2 reusable connection: 1
2019/11/14 09:34:16 [debug] 13096#13096: *2 epoll add event: fd:16 op:1 ev:80002001
2019/11/14 09:34:16 [debug] 13096#13096: *1 http check ssl handshake
2019/11/14 09:34:16 [debug] 13096#13096: *1 http recv(): 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 https ssl handshake: 0x16
2019/11/14 09:34:16 [debug] 13096#13096: *1 tcp_nodelay
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL server name: "example.com"
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL ALPN supported by client: h2
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL ALPN supported by client: http/1.1
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL ALPN selected: http/1.1
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_do_handshake: -1
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_get_error: 2
2019/11/14 09:34:16 [debug] 13096#13096: *1 reusable connection: 0
2019/11/14 09:34:16 [debug] 13096#13096: *2 http check ssl handshake
2019/11/14 09:34:16 [debug] 13096#13096: *2 http recv(): 1
2019/11/14 09:34:16 [debug] 13096#13096: *2 https ssl handshake: 0x16
2019/11/14 09:34:16 [debug] 13096#13096: *2 tcp_nodelay
2019/11/14 09:34:16 [debug] 13096#13096: *2 SSL server name: "example.com"
2019/11/14 09:34:16 [debug] 13096#13096: *2 SSL ALPN supported by client: h2
2019/11/14 09:34:16 [debug] 13096#13096: *2 SSL ALPN supported by client: http/1.1
2019/11/14 09:34:16 [debug] 13096#13096: *2 SSL ALPN selected: http/1.1
2019/11/14 09:34:16 [debug] 13096#13096: *2 SSL_do_handshake: -1
2019/11/14 09:34:16 [debug] 13096#13096: *2 SSL_get_error: 2
2019/11/14 09:34:16 [debug] 13096#13096: *2 reusable connection: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL handshake handler: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_do_handshake: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL: TLSv1.3, cipher: "TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD"
2019/11/14 09:34:16 [debug] 13096#13096: *1 reusable connection: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 http wait request handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 malloc: 000055D8FA6E30C0:1024
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_read: 567
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_read: -1
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_get_error: 2
2019/11/14 09:34:16 [debug] 13096#13096: *1 reusable connection: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 posix_memalign: 000055D8FA704290:4096 #16
2019/11/14 09:34:16 [debug] 13096#13096: *1 http process request line
2019/11/14 09:34:16 [debug] 13096#13096: *1 http request line: "GET /app/overview/dashboard HTTP/1.1"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http uri: "/app/overview/dashboard"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http args: ""
2019/11/14 09:34:16 [debug] 13096#13096: *1 http exten: ""
2019/11/14 09:34:16 [debug] 13096#13096: *1 posix_memalign: 000055D8FA7052A0:4096 #16
2019/11/14 09:34:16 [debug] 13096#13096: *1 http process request header line
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Host: example.com"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Connection: keep-alive"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Cache-Control: max-age=0"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Upgrade-Insecure-Requests: 1"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Sec-Fetch-User: ?1"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Sec-Fetch-Site: same-origin"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Sec-Fetch-Mode: navigate"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Accept-Encoding: gzip, deflate, br"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,fr;q=0.6"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header done
2019/11/14 09:34:16 [debug] 13096#13096: *1 event timer del: 15: 266964738
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 rewrite phase: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: ~ "^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).+$"
2019/11/14 09:34:16 [debug] 13096#13096: *1 using configuration "^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).+$"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http cl:-1 max:1048576
2019/11/14 09:34:16 [debug] 13096#13096: *1 rewrite phase: 3
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script regex: "^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).*$"
2019/11/14 09:34:16 [notice] 13096#13096: *1 "^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).*$" matches "/app/overview/dashboard", client: 213.80.249.126, server: example.com, request: "GET /app/overview/dashboard HTTP/1.1", host: "example.com"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script copy: "/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script regex end
2019/11/14 09:34:16 [notice] 13096#13096: *1 rewritten data: "/app", args: "", client: 213.80.249.126, server: example.com, request: "GET /app/overview/dashboard HTTP/1.1", host: "example.com"
2019/11/14 09:34:16 [debug] 13096#13096: *1 post rewrite phase: 4
2019/11/14 09:34:16 [debug] 13096#13096: *1 uri changes: 11
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: ~ "^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).+$"
2019/11/14 09:34:16 [debug] 13096#13096: *1 using configuration "/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http cl:-1 max:1048576
2019/11/14 09:34:16 [debug] 13096#13096: *1 rewrite phase: 3
2019/11/14 09:34:16 [debug] 13096#13096: *1 post rewrite phase: 4
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 5
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 6
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 7
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 8
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 9
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 10
2019/11/14 09:34:16 [debug] 13096#13096: *1 post access phase: 11
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 12
2019/11/14 09:34:16 [debug] 13096#13096: *1 try files handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script var: "/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 trying to use file: "/app" "/srv/root/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script var: "/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 trying to use dir: "/app" "/srv/root/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 try file uri: "/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 13
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 14
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 15
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 16
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 17
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 18
2019/11/14 09:34:16 [debug] 13096#13096: *1 http filename: "/srv/root/app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 add cleanup: 000055D8FA705230
2019/11/14 09:34:16 [debug] 13096#13096: *1 http static fd: -1
2019/11/14 09:34:16 [debug] 13096#13096: *1 http dir
2019/11/14 09:34:16 [debug] 13096#13096: *1 http finalize request: 301, "/app?" a:1, c:1
2019/11/14 09:34:16 [debug] 13096#13096: *1 http special response: 301, "/app?"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http set discard body
2019/11/14 09:34:16 [debug] 13096#13096: *1 xslt filter header
2019/11/14 09:34:16 [debug] 13096#13096: *1 HTTP/1.1 301 Moved Permanently
Server: nginx/1.15.9 (Ubuntu)
Date: Thu, 14 Nov 2019 09:34:16 GMT
Content-Type: text/html
Content-Length: 178
Location: https://example.com/app/
Connection: keep-alive
2019/11/14 09:34:16 [debug] 13096#13096: *1 write new buf t:1 f:0 000055D8FA7056D0, pos 000055D8FA7056D0, size: 206 file: 0, size: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 http write filter: l:0 f:0 s:206
2019/11/14 09:34:16 [debug] 13096#13096: *1 http output filter "/app?"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http copy filter: "/app?"
2019/11/14 09:34:16 [debug] 13096#13096: *1 image filter
2019/11/14 09:34:16 [debug] 13096#13096: *1 xslt filter body
2019/11/14 09:34:16 [debug] 13096#13096: *1 http postpone filter "/app?" 000055D8FA705270
2019/11/14 09:34:16 [debug] 13096#13096: *1 write old buf t:1 f:0 000055D8FA7056D0, pos 000055D8FA7056D0, size: 206 file: 0, size: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 write new buf t:0 f:0 0000000000000000, pos 000055D8F9CB8AE0, size: 116 file: 0, size: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 write new buf t:0 f:0 0000000000000000, pos 000055D8F9CB8DE0, size: 62 file: 0, size: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 http write filter: l:1 f:0 s:384
2019/11/14 09:34:16 [debug] 13096#13096: *1 http write filter limit 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 posix_memalign: 000055D8FA79C6A0:512 #16
2019/11/14 09:34:16 [debug] 13096#13096: *1 malloc: 000055D8FA791C80:16384
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL buf copy: 206
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL buf copy: 116
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL buf copy: 62
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL to write: 384
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_write: 384
2019/11/14 09:34:16 [debug] 13096#13096: *1 http write filter 0000000000000000
2019/11/14 09:34:16 [debug] 13096#13096: *1 http copy filter: 0 "/app?"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http finalize request: 0, "/app?" a:1, c:1
2019/11/14 09:34:16 [debug] 13096#13096: *1 set http keepalive handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 http close request
2019/11/14 09:34:16 [debug] 13096#13096: *1 http log handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA704290, unused: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA7052A0, unused: 2222
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA6E30C0
2019/11/14 09:34:16 [debug] 13096#13096: *1 hc free: 0000000000000000
2019/11/14 09:34:16 [debug] 13096#13096: *1 hc busy: 0000000000000000 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA791C80
2019/11/14 09:34:16 [debug] 13096#13096: *1 reusable connection: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 event timer add: 15: 65000:266969754
2019/11/14 09:34:16 [debug] 13096#13096: *1 http keepalive handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 malloc: 000055D8FA6E30C0:1024
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_read: 632
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_read: -1
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_get_error: 2
2019/11/14 09:34:16 [debug] 13096#13096: *1 reusable connection: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 posix_memalign: 000055D8FA704290:4096 #16
2019/11/14 09:34:16 [debug] 13096#13096: *1 event timer del: 15: 266969754
2019/11/14 09:34:16 [debug] 13096#13096: *1 http process request line
2019/11/14 09:34:16 [debug] 13096#13096: *1 http request line: "GET /app/ HTTP/1.1"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http uri: "/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http args: ""
2019/11/14 09:34:16 [debug] 13096#13096: *1 http exten: ""
2019/11/14 09:34:16 [debug] 13096#13096: *1 posix_memalign: 000055D8FA7052A0:4096 #16
2019/11/14 09:34:16 [debug] 13096#13096: *1 http process request header line
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Host: example.com"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Connection: keep-alive"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Cache-Control: max-age=0"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Upgrade-Insecure-Requests: 1"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Sec-Fetch-User: ?1"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Sec-Fetch-Site: same-origin"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Sec-Fetch-Mode: navigate"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Accept-Encoding: gzip, deflate, br"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,fr;q=0.6"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "If-None-Match: W/"5dcbc7e8-611""
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header: "If-Modified-Since: Wed, 13 Nov 2019 09:07:52 GMT"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http header done
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 rewrite phase: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: ~ "^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).+$"
2019/11/14 09:34:16 [debug] 13096#13096: *1 using configuration "/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http cl:-1 max:1048576
2019/11/14 09:34:16 [debug] 13096#13096: *1 rewrite phase: 3
2019/11/14 09:34:16 [debug] 13096#13096: *1 post rewrite phase: 4
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 5
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 6
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 7
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 8
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 9
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 10
2019/11/14 09:34:16 [debug] 13096#13096: *1 post access phase: 11
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 12
2019/11/14 09:34:16 [debug] 13096#13096: *1 try files handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script var: "/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 trying to use file: "/app/" "/srv/root/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script var: "/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 trying to use dir: "/app/" "/srv/root/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 try file uri: "/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 13
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 14
2019/11/14 09:34:16 [debug] 13096#13096: *1 open index "/srv/root/app/index.html"
2019/11/14 09:34:16 [debug] 13096#13096: *1 internal redirect: "/app/index.html?"
2019/11/14 09:34:16 [debug] 13096#13096: *1 rewrite phase: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "app"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: "/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 test location: ~ "^\/app\/(?!(index\.html|favicon\.ico|css\/|fonts\/|img\/|js\/)).+$"
2019/11/14 09:34:16 [debug] 13096#13096: *1 using configuration "/app/"
2019/11/14 09:34:16 [debug] 13096#13096: *1 http cl:-1 max:1048576
2019/11/14 09:34:16 [debug] 13096#13096: *1 rewrite phase: 3
2019/11/14 09:34:16 [debug] 13096#13096: *1 post rewrite phase: 4
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 5
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 6
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 7
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 8
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 9
2019/11/14 09:34:16 [debug] 13096#13096: *1 access phase: 10
2019/11/14 09:34:16 [debug] 13096#13096: *1 post access phase: 11
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 12
2019/11/14 09:34:16 [debug] 13096#13096: *1 try files handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 http script var: "/app/index.html"
2019/11/14 09:34:16 [debug] 13096#13096: *1 trying to use file: "/app/index.html" "/srv/root/app/index.html"
2019/11/14 09:34:16 [debug] 13096#13096: *1 try file uri: "/app/index.html"
2019/11/14 09:34:16 [debug] 13096#13096: *1 generic phase: 13
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 14
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 15
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 16
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 17
2019/11/14 09:34:16 [debug] 13096#13096: *1 content phase: 18
2019/11/14 09:34:16 [debug] 13096#13096: *1 http filename: "/srv/root/app/index.html"
2019/11/14 09:34:16 [debug] 13096#13096: *1 add cleanup: 000055D8FA7051A8
2019/11/14 09:34:16 [debug] 13096#13096: *1 http static fd: 17
2019/11/14 09:34:16 [debug] 13096#13096: *1 http set discard body
2019/11/14 09:34:16 [debug] 13096#13096: *1 http ims:1573636072 lm:1573636072
2019/11/14 09:34:16 [debug] 13096#13096: *1 http im:"W/"5dcbc7e8-611"" etag:"5dcbc7e8-611"
2019/11/14 09:34:16 [debug] 13096#13096: *1 xslt filter header
2019/11/14 09:34:16 [debug] 13096#13096: *1 HTTP/1.1 304 Not Modified
Server: nginx/1.15.9 (Ubuntu)
Date: Thu, 14 Nov 2019 09:34:16 GMT
Last-Modified: Wed, 13 Nov 2019 09:07:52 GMT
Connection: keep-alive
ETag: "5dcbc7e8-611"
2019/11/14 09:34:16 [debug] 13096#13096: *1 write new buf t:1 f:0 000055D8FA7057B8, pos 000055D8FA7057B8, size: 189 file: 0, size: 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 http write filter: l:1 f:0 s:189
2019/11/14 09:34:16 [debug] 13096#13096: *1 http write filter limit 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 malloc: 000055D8FA791C80:16384
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL buf copy: 189
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL to write: 189
2019/11/14 09:34:16 [debug] 13096#13096: *1 SSL_write: 189
2019/11/14 09:34:16 [debug] 13096#13096: *1 http write filter 0000000000000000
2019/11/14 09:34:16 [debug] 13096#13096: *1 http finalize request: 0, "/app/index.html?" a:1, c:2
2019/11/14 09:34:16 [debug] 13096#13096: *1 http request count:2 blk:0
2019/11/14 09:34:16 [debug] 13096#13096: *1 http finalize request: -4, "/app/index.html?" a:1, c:1
2019/11/14 09:34:16 [debug] 13096#13096: *1 set http keepalive handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 http close request
2019/11/14 09:34:16 [debug] 13096#13096: *1 http log handler
2019/11/14 09:34:16 [debug] 13096#13096: *1 run cleanup: 000055D8FA7051A8
2019/11/14 09:34:16 [debug] 13096#13096: *1 file cleanup: fd:17
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA704290, unused: 40
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA7052A0, unused: 2371
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA6E30C0
2019/11/14 09:34:16 [debug] 13096#13096: *1 hc free: 0000000000000000
2019/11/14 09:34:16 [debug] 13096#13096: *1 hc busy: 0000000000000000 0
2019/11/14 09:34:16 [debug] 13096#13096: *1 free: 000055D8FA791C80
2019/11/14 09:34:16 [debug] 13096#13096: *1 reusable connection: 1
2019/11/14 09:34:16 [debug] 13096#13096: *1 event timer add: 15: 65000:266969762
2019/11/14 09:34:17 [debug] 13096#13096: *2 SSL handshake handler: 0
2019/11/14 09:34:17 [debug] 13096#13096: *2 SSL_do_handshake: 1
2019/11/14 09:34:17 [debug] 13096#13096: *2 SSL: TLSv1.3, cipher: "TLS_AES_256_GCM_SHA384 TLSv1.3 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD"
2019/11/14 09:34:17 [debug] 13096#13096: *2 reusable connection: 1
2019/11/14 09:34:17 [debug] 13096#13096: *2 http wait request handler
2019/11/14 09:34:17 [debug] 13096#13096: *2 malloc: 000055D8FA6E30C0:1024
2019/11/14 09:34:17 [debug] 13096#13096: *2 SSL_read: -1
2019/11/14 09:34:17 [debug] 13096#13096: *2 SSL_get_error: 2
2019/11/14 09:34:17 [debug] 13096#13096: *2 free: 000055D8FA6E30C0
I am trying to set up nginx to map TLS connections to different backends based on the SNI server name. From what I can tell, my client is sending the server name, but the preread module is only reading a hyphen.
Here is my nginx congif:
stream {
map_hash_bucket_size 64;
############################################################
### logging
log_format log_stream '$remote_addr [$time_local] $protocol [$ssl_preread_server_name] [$ssl_preread_alpn_protocols] [$instanceport] '
'$status $bytes_sent $bytes_received $session_time';
error_log /usr/home/glance/Logs/pservernginx.error.log info;
access_log /usr/home/glance/Logs/pservernginx.access.log log_stream;
############################################################
### ssl configuration
ssl_certificate /usr/home/glance/GlanceReleases/star.myglance.org.pem;
ssl_certificate_key /usr/home/glance/GlanceReleases/star.myglance.org.pem;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5:!RC4;
limit_conn_zone $binary_remote_addr zone=ip_addr:10m;
########################################################################
### Raw TLS PServer Connections
### Listen for TLS on 5501 and forward to TCP sock 6500 (socket port)
### https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
map $ssl_preread_server_name $instanceport {
presence.myglance.org 6500;
presence-1.myglance.org 6501;
presence-2.myglance.org 6502;
default glance-no-upstream-instance-configured;
}
server {
listen 5501 ssl;
ssl_preread on;
proxy_connect_timeout 20s; # max time to connect to pserver
proxy_timeout 30s; # max time between successive reads or writes
proxy_pass 127.0.0.1:$instanceport;
}
}
wireshark shows the Server Name header:
The nginx access log shows only hyphens for the preread variables:
108.49.96.66 [12/Apr/2019:11:50:58 +0000] TCP [-] [-] [glance-no-upstream-instance-configured] 500 0 0 0.066
I'm running nginx 1.14.2 on FreeBSD. How can I debug what is happening in the preread module?
================ UPDATE ===============
Turned on debug logging. Maybe "ssl preread: not a handshake" is a clue.
2019/04/12 14:49:50 [info] 61420#0: *9 client 108.49.96.66:54740 connected to 0.0.0.0:5501
2019/04/12 14:49:50 [debug] 61420#0: *9 posix_memalign: 0000000801C35000:256 #16
2019/04/12 14:49:50 [debug] 61419#0: accept on 0.0.0.0:5501, ready: 1
2019/04/12 14:49:50 [debug] 61419#0: accept() not ready (35: Resource temporarily unavailable)
2019/04/12 14:49:50 [debug] 61420#0: *9 posix_memalign: 0000000801C35600:256 #16
2019/04/12 14:49:50 [debug] 61420#0: *9 generic phase: 0
2019/04/12 14:49:50 [debug] 61420#0: *9 generic phase: 1
2019/04/12 14:49:50 [debug] 61420#0: *9 generic phase: 2
2019/04/12 14:49:50 [debug] 61420#0: *9 tcp_nodelay
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_do_handshake: -1
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_get_error: 2
2019/04/12 14:49:50 [debug] 61420#0: *9 kevent set event: 5: ft:-1 fl:0025
2019/04/12 14:49:50 [debug] 61420#0: *9 event timer add: 5: 60000:29203481224
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL handshake handler: 0
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_do_handshake: 1
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2019/04/12 14:49:50 [debug] 61420#0: *9 event timer del: 5: 29203481224
2019/04/12 14:49:50 [debug] 61420#0: *9 generic phase: 2
2019/04/12 14:49:50 [debug] 61420#0: *9 ssl preread handler
2019/04/12 14:49:50 [debug] 61420#0: *9 malloc: 0000000801CFF000:16384
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_read: -1
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_get_error: 2
2019/04/12 14:49:50 [debug] 61420#0: *9 ssl preread handler
2019/04/12 14:49:50 [debug] 61420#0: *9 posix_memalign: 0000000801C35900:256 #16
2019/04/12 14:49:50 [debug] 61420#0: *9 event timer add: 5: 30000:29203451252
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_read: 81
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_read: -1
2019/04/12 14:49:50 [debug] 61420#0: *9 SSL_get_error: 2
2019/04/12 14:49:50 [debug] 61420#0: *9 ssl preread handler
2019/04/12 14:49:50 [debug] 61420#0: *9 ssl preread: not a handshake
2019/04/12 14:49:50 [debug] 61420#0: *9 event timer del: 5: 29203451252
2019/04/12 14:49:50 [debug] 61420#0: *9 proxy connection handler
2019/04/12 14:49:50 [debug] 61420#0: *9 malloc: 0000000801DF7000:400
2019/04/12 14:49:50 [debug] 61420#0: *9 malloc: 0000000801CD9000:16384
2019/04/12 14:49:50 [debug] 61420#0: *9 stream map started
2019/04/12 14:49:50 [debug] 61420#0: *9 stream map: "" "glance-no-upstream-instance-configured"
================= UPDATE 2 ======================
I tested using
openssl s_client -connect ... -servername ...
instead of my client. Now it appears that the preread module is blocked waiting for data for 30 seconds (error code 2 is WANT_READ):
2019/04/23 13:04:30 [debug] 61419#0: *12844 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD"
2019/04/23 13:04:30 [debug] 61419#0: *12844 event timer del: 3: 30147561850
2019/04/23 13:04:30 [debug] 61419#0: *12844 generic phase: 2
2019/04/23 13:04:30 [debug] 61419#0: *12844 ssl preread handler
2019/04/23 13:04:30 [debug] 61419#0: *12844 malloc: 0000000801CA6140:16384
2019/04/23 13:04:30 [debug] 61419#0: *12844 SSL_read: -1
2019/04/23 13:04:30 [debug] 61419#0: *12844 SSL_get_error: 2
2019/04/23 13:04:30 [debug] 61419#0: *12844 ssl preread handler
2019/04/23 13:04:30 [debug] 61419#0: *12844 posix_memalign: 0000000801DB3400:256 #16
2019/04/23 13:04:30 [debug] 61419#0: *12844 event timer add: 3: 30000:30147531898
2019/04/23 13:05:00 [debug] 61419#0: *12844 event timer del: 3: 30147531898
2019/04/23 13:05:00 [debug] 61419#0: *12844 finalize stream session: 200
2019/04/23 13:05:00 [debug] 61419#0: *12844 stream log handler
2019/04/23 13:05:00 [debug] 61419#0: *12844 stream map started
2019/04/23 13:05:00 [debug] 61419#0: *12844 stream script var: ""
I found the problem:
listen 5501 **ssl**;
ssl_preread on;
ssl in the listen directive caused that nginx server to do the ssl handshake. By the time the preread module was notified, the handshake bytes had already been consumed, which is all consistent with the behavior I was seeing. In my case, I still want nginx to offload the encryption. So I created a set of nginx server directives to terminate the ssl connection before passing to my back end.
This is the relevant portion of my nginx config after fixing it. Note that the last server directive (the one that uses ssl_preread) does not terminate the SSL connection.
########################################################################
### TLS Connections
### Listen for TLS on 5501 and forward to TCP sock 6500 (socket port)
### https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html
map $ssl_preread_server_name $instanceport {
presence.myglance.org 5502;
presence-1.myglance.org 5503;
presence-2.myglance.org 5504;
default glance-no-upstream-instance-configured;
}
server {
listen 5502 ssl;
ssl_preread off;
proxy_pass 127.0.0.1:6502;
}
server {
listen 5503 ssl;
ssl_preread off;
proxy_pass 127.0.0.1:6503;
}
server {
listen 5504 ssl;
ssl_preread off;
proxy_pass 127.0.0.1:6504;
}
server {
listen 5501;
ssl_preread on;
proxy_connect_timeout 20s; # max time to connect to pserver
proxy_timeout 30s; # max time between successive reads or writes
proxy_pass 127.0.0.1:$instanceport;
}
In case you need to use ssl in listen directive, you can simply use $ssl_server_name in the map block instead of $ssl_preread_server_name
we have ingress-nginx running for a while and about 10% of requests ending up with some SSL handshake problem.
Here is an example of a failing connection:
2019/02/14 10:15:35 [debug] 237#237: *4612 accept: **.**.**.**:40928 fd:53
2019/02/14 10:15:35 [debug] 237#237: *4612 event timer add: 53: 60000:5527050245
2019/02/14 10:15:35 [debug] 237#237: *4612 reusable connection: 1
2019/02/14 10:15:35 [debug] 237#237: *4612 epoll add event: fd:53 op:1 ev:80002001
2019/02/14 10:15:45 [debug] 237#237: *4612 http check ssl handshake
2019/02/14 10:15:45 [debug] 237#237: *4612 http recv(): 0
2019/02/14 10:15:45 [info] 237#237: *4612 client closed connection while SSL handshaking, client: **.**.**.**, server: 0.0.0.0:443
2019/02/14 10:15:45 [debug] 237#237: *4612 close http connection: 53
2019/02/14 10:15:45 [debug] 237#237: *4612 event timer del: 53: 5527050245
2019/02/14 10:15:45 [debug] 237#237: *4612 reusable connection: 0
2019/02/14 10:15:45 [debug] 237#237: *4612 free: 00007F4CC5858E00, unused: 232
10% of failures seems to be quite a lot to expect.
I really would appreciate any help in this!