I have a bare minimum, default puppet master/slave configuration on a newer version (6) of puppet which I'm attempting to initiate in virtualbox VMs for a prototype. However, the agent doesnt seem to be able to properly connect to the master.
10.0.2.2 - - [14/Apr/2019:18:22:14 +0000] "GET /production/certificate/localhost? HTTP/1.1" 404 36 "-" "Ruby" 3
10.0.2.2 - - [14/Apr/2019:18:22:14 +0000] "GET /production/certificate/ca?fail_on_404=true HTTP/1.1" 200 1939 "-" "Ruby" 3
10.0.2.2 - - [14/Apr/2019:18:22:14 +0000] "GET /production/certificate/localhost? HTTP/1.1" 404 36 "-" "Ruby" 2
10.0.2.2 - - [14/Apr/2019:18:22:14 +0000] "GET /production/certificate/localhost? HTTP/1.1" 404 36 "-" "Ruby" 3
10.0.2.2 - - [14/Apr/2019:18:22:14 +0000] "GET /production/certificate/localhost? HTTP/1.1" 404 36 "-" "Ruby" 2
On my master, I only 2 certs, so clearly the request is getting through:
"localhost" (SHA256) 1C:E7:D0:FF:35:A3:5B:CA:37:02:13:CC:75:20:B5:54:42:BA:AA:C9:61:9D:02:22:B3:28:E3:C3:4D:FE:5F:CC
"slave1" (SHA256) 35:A8:C5:E8:8A:1D:58:F6:DA:EC:8A:4D:9F:30:53:3E:F8:A1:01:27:F4:D7:62:5F:82:1C:E0:6B:37:82:A8:A2
My agent is able to connect just fine to the master, however, it seems to never get back a healthy cert:
Nothing to do
waiting to run puppet....
Info: Creating a new SSL key for localhost
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for localhost
Info: Certificate Request fingerprint (SHA256): 1C:E7:D0:FF:35:A3:5B:CA:37:02:13:CC:75:20:B5:54:42:BA:AA:C9:61:9D:02:22:B3:28:E3:C3:4D:FE:5F:CC
Info: Caching certificate for ca
Notice: Did not receive certificate
Notice: Did not receive certificate
How can I determine why these 404 errors are occuring?
As mentioned you have to sign the certificate..
For Puppet Version 6 would be
puppetserver ca sign --certname slave1
For Puppet Version < 6
puppet cert sign slave1
It turns out that I hadn't signed my requests.
In order to do this, you simply use puppet cert list and then puppet cert sign to stand the outstanding request. At that point, the server will be able to output a certificate, which the agent can then download and use locally.
Related
I am configuring readiness and liveness probes for my kuberenetes deployment.
Here is how i added it:
ports:
- name: http
containerPort: {{ .Values.service.internalPort }}
protocol: TCP
livenessProbe:
tcpSocket:
port: http
readinessProbe:
tcpSocket:
port: http
But this is causing the error logs in the pod:
2021/03/24 03:23:06 http: TLS handshake error from 10.244.0.1:48476: EOF
If i remove the probes and create the deployment, this logs will not appear.
I have an ingress setup such that all the http requests to that container as https. Because my container expects only https requests to it.
I thought this error logs are shown because the tcp probes are not sending https requests here.
Is there some other way to setup probes without these error logs?
if you are looking forward to send to the HTTPS request to the service you have to change the scheme.
livenessProbe:
httpGet:
path: /
port: 443
scheme: HTTPS
readinessProbe:
httpGet:
path: /
port: 443
scheme: HTTPS
you can check more at : https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes
scheme: Scheme to use for connecting to the host (HTTP or HTTPS). Defaults to HTTP.
if HTTPS is set kubelet will send to HTTPS request or else by default it will be HTTP.
if request if failing you will see logs like : 400 bad request
10.165.18.52 - - [24/March/2021:17:06:40 +0000] "GET / HTTP/1.1" 400 271 "-" "kube-probe/1.16"
for the successful request, it will be 200 request
10.165.18.52 - - [24/March/2021:18:10:06 +0000] "GET / HTTP/1.1" 200 "-" "kube-probe/1.16"
I am using nginx 1.17.10.1 Unicorn build from http://nginx-win.ecsds.eu/ and Apache/2.4.43 build from Apachelounge on Windows Server 2012 R2.
Nginx serves static files and proxies Apache responses for PHP scripts. Everything was fine until recently.
Two times in a day without any distinct reason the websites stop responding. Memory/CPU/Network usages are ok. Apache starts logging like
XX.XX.XX.XX 0ms [01/Jul/2020:05:05:20 -0700] "-" 408 - "-" "-"
for each request.
Nginx log shows
2020/07/01 06:04:54 [error] 11800#12192: *5002230 WSARecv() failed (10053: An established connection was aborted by the software in your host machine) while reading response header from upstream, client: YY.YY.YY.YY, server: example.com, request: "GET /the/url/here HTTP/2.0", upstream: "http://XX.XX.XX.XX:8181/the/url/here", host: "example.com"
Server reboot doesn't help. I can connect to the backend directly and it serves the response without any problem.
The only way I could resolve the problem was to switch HTTP/2 off in nginx configuration.
So what can cause this behavior?
hi # all i have a problem with pass through the TCP Communication to a Synology NAS with SSL.
I want to connect with the Synology Drive Client to the NAS, and the Drive Client Software communicate over the TCP Port 6690 with the NAS.
When i try to connect i get an 500 Error.
Without SSL it works fine, but than the Synology encrypt the communication with a own untrusted Cert., that should not be the solution.
The Build:
Internet| --> |Router(Port forwarding 6690)| --> |nginx| -->| NAS(192.168.10.2)|
Nginx:
stream{
log_format log_stream '$remote_addr [$time_local] $protocol [$ssl_preread_server_name] [$ssl_preread_alpn_protocols]'
'$status $bytes_sent $bytes_received $session_time';
access_log /var/log/nginx/access.log log_stream;
ssl_certificate /etc/letsencrypt/live/{mydomain}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{mydomain}/privkey.pem;
ssl_protocols TLSv1.1 TLSv1.2;
server {
listen 6690 ssl;
proxy_pass 192.168.10.2:6690;
}
}
Log:
xx.xx.xxx.xxx [08/Nov/2019:15:09:37 +0100] TCP [-] [-]500 0 0 0.000
xx.xx.xxx.xxx [08/Nov/2019:15:09:37 +0100] TCP [-] [-]500 0 0 0.000
xx.xx.xxx.xxx [08/Nov/2019:15:10:37 +0100] TCP [-] [-]500 0 0 0.000
xx.xx.xxx.xxx [08/Nov/2019:15:10:37 +0100] TCP [-] [-]500 0 0 0.000
xx.xx.xxx.xxx [08/Nov/2019:15:11:37 +0100] TCP [-] [-]500 0 0 0.000
xx.xx.xxx.xxx [08/Nov/2019:15:11:37 +0100] TCP [-] [-]500 0 0 0.000
i also try to check the SSL handshake with:
openssl s_client -host mydomain.net -port 6690
and that works fine.
Does somebody has any idea where is my mistake??? :-(
Similar setup with the goal to provide access to Synology Drive behind "safer" Rev-Proxy.
The problem is not the SSL handshake.
The problem is: no http protocol running no port 6690 for Synology Drive etc. (Reference in German, contains assumptions about non-http protocol on 6690: https://www.synology-forum.de/showthread.html?74773-Cloud-Station-über-Reverse-Proxy/page3)
In addition, I call with the Drive-Client to my NginxRevProxy (which is working in general) and get from the access.log:
172.18.0.1 - - [09/Nov/2019:10:17:26 +0000] "%R\x18\x14F\x0B\x00\x00" 400 157 "-" "-" "-"
So your approach (and mine) are not sufficient.
Possible path to a solution, which is beyond my current knowledge:
"Upgrade" Nginx to a reverse tcp-proxy, if this works with Synology Drive, use "fancyness" of Nginx in conjunction with own auth-script within and for the TCP-Proxy. (i.e. "abuse" + extend the TCP-Loadbalancer shown here: https://www.debinux.de/2014/12/nginx-als-tcp-proxy-beispiel-dovecot/)
Very new to haproxy and loving it, apart from a 504 issue that we're getting. The relevant log output is:
Jun 21 13:52:06 localhost haproxy[1431]: 192.168.0.2:51435 [21/Jun/2017:13:50:26.740] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 13:54:26 localhost haproxy[1431]: 192.168.0.2:51447 [21/Jun/2017:13:52:46.577] www-https~ beFootprints/foorprints 0/0/3/-1/100005 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:15:57 localhost haproxy[1431]: 192.168.0.1:50225 [21/Jun/2017:14:14:17.771] www-https~ beFootprints/foorprints 0/0/2/-1/100004 504 195 - - sH-- 3/3/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Jun 21 14:22:26 localhost haproxy[1431]: 192.168.0.1:50258 [21/Jun/2017:14:20:46.608] www-https~ beFootprints/foorprints 0/0/2/-1/100003 504 195 - - sH-- 2/2/0/0/0 0/0 "POST /MRcgi/MRlogin.pl HTTP/1.1"
Using the following timeout values in the haproxy.cfg
defaults
log global
mode http
option forwardfor
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 100000
Running on Ubuntu 16.04.2 LTS
Any help and comment very much appreciated!
The problem appears to be with the web server. Check the logs, there, and you should find long-running requests.
Here's how I conclude that.
Note sH-- in your logs. This is the session state at disconnection. It's extremely valuable for troubleshooting. The values are positional and case-sensitive.
s: the server-side timeout expired while waiting for the server to send or receive data.
...so, timeout server fired, while...
H: the proxy was waiting for complete, valid response HEADERS from the server (HTTP only).
The server had not finished (perhaps not even started) returing all the response headers to the proxy, but the connection was established and the request had been sent.
HAProxy returns 504 Gateway Timeout, indicating that the backend did not respond in a timely fashion.
If your backend needs longer than 100 seconds (?!) then you need to increase timeout server. Otherwise, your Apache server seems to have a problem being too slow to respond.
I had a similar issue and found the problem was with how I had configured my backend server section.
backend no_match_backend
mode http
balance roundrobin
option forwardfor
option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com
server nginx-example 192.168.0.10 check port 80
My problem is that I did not specify the port for the connection.
When connecting via HTTP it would work but as I have my SSL terminated on my haproxy.
This attempts to connect via 443 to the backends.
As the backends cannot / don't correctly communicate. The setup of the SSL session with haproxy and the backend that causes the gateway to time out.
I need to force unencrypted communications to the backends.
backend no_match_backend
mode http
balance roundrobin
option forwardfor
option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com
server nginx-example 192.168.0.10:80 check port 80
The change might be hard to spot server nginx-example 192.168.0.10 check port 80 now has :80 after the ip 192.168.0.10:80
This problem was made more complicated by my backend servers having SSL redirects configured. So all my requests would arrive as HTTP and be redirected to HTTPS. So it was difficult to identify where the problem was. I
It looked like https requests were being redirected correctly to the backend servers. I need to disable this redirect on the backend servers and move it forward to haproxy config.
I repeated the following using SLES and OS X. I compiled apache 2.4.18, nghttp2-1.8.0 and OpenSSL 1.0.2g and using a recently built version of curl it appears everything is working fine.
curl https://macbookpro.xxxx/ --cacert /usr/local/apache2/conf/ssl/server.crt --verbose
Output from the above shows the connection upgrading to http/2 and the Apache access logs report:
192.168.0.1 - - [20/Mar/2016:15:12:05 +0000] "GET / HTTP/2" 200 45
The certificate chain is all locally generated. With a root, intermediary and server certificates and the following commands report everything is OK
openssl s_client -connect macbookpro:443 -CAfile /usr/local/apache2/conf/ssl/server.crt
However I just cant get any of my browsers Safari/Firefox or Chrome to utilise http/2.
Firefox 45.01 (with spdy indicator extension)
Safari 6.2.8 (8537.85.17.9.1)
Google Chrome 49.0.2623.87 (64-bit)
The page displays fine, but firefox's spdy indicator extension shows nothing. The access logs all report HTTP/1.1 connections. How do I get http2 working?