Openshift on VCenter UPI installation bootstrap not working - api

Trying to install openshift in vcenter. The bootstrap is booting up but not grabbing the bootstrap.ign from my web server.
set semanage fcontext public_content_rw_t on /var/www/openshift directory but I am getting
DEBUG Still waiting for the Kubernetes API: Get "https://api.cl.ops.local:6443/version": http: server gave HTTP response to HTTPS client
my haproxy server looks like this I am only testing with static IP addreses.
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
# utilize system-wide crypto-policies
#ssl-default-bind-ciphers PROFILE=SYSTEM
#ssl-default-server-ciphers PROFILE=SYSTEM
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
listen stats
bind :9000
mode http
stats enable
stats uri /
monitor-uri /healthz
frontend api
bind *:6443
default_backend api
mode http
backend api
balance source
server bootstrap.cl.ops.local 192.168.61.22:6443 check
# server master0 192.168.61.23:6443 check
server master0.cl.ops.local 192.168.61.23:6443 check
server master1 192.168.61.24:6443 check
server master2 192.168.61.25:6443 check
mode http
frontend api-int
bind *:22623
default_backend api-int
mode http
backend api-int
balance source
server bootstrap.cl.ops.local 192.168.61.22:22623 check # Can be removed or commented out after install completes
server master0.cl.ops.local 192.168.61.23:22623 check
# server master0 192.168.61.23:22623 check
server master1 192.168.61.24:22623 check
server master2 192.168.61.25:22623 check
mode http
frontend secure
bind *:443
default_backend secure
mode http
backend secure
balance source
server worker0 192.168.61.65:443 check
server worker1 192.168.61.66:443 check
server worker2 192.168.61.67:443 check
mode http
frontend insecure
bind *:80
default_backend insecure
mode http
backend insecure
balance source
server worker0 192.168.61.65:80 check
server worker1 192.168.61.66:80 check
server worker2 192.168.61.67:80 check
mode h 14,3 11%

Related

Module Apache mod_remoteip cause SSL Handshake failed (5) when load testing

I have Haproxy which load balance the traffic to two apache servers.
When I benchmark with apache benchmark tool I have no errors and everything works as expected.
But when I activate the proxy protocol communication between HAProxy and apaches I start having issues.
There are two scenarios
1- Load testing with less then 500 request/second everything works well.
2- Load testing with concurrent requests > 600. I start having SSL handshake failed (5) issues.
Here is my HAproxy configuration:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local0 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 3000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode tcp
log global
option tcplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
#---------------------------------------------------------------------
# Servers sections
#---------------------------------------------------------------------
frontend XXXX:443
bind XXX:443
mode tcp
use_backend https_front
backend https_front
balance roundrobin
server XXX XXX check send-proxy
My apache vhost conf:
<VirtualHost *:443>
...
ServerName XXXX
RemoteIPProxyProtocol On
My benchmark command:
> ab -c 1020 -n 9000 https://XXXX
...
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL handshake failed (5).
SSL read failed (1) - closing connection
139921183344512:error:140E0197:SSL routines:SSL_shutdown:shutdown while in init:ssl/ssl_lib.c:2094:
...
I can not find any logs neither in haproxy nor in the apache logs. Did someone face the same issue before ?
Thanks in advance

Enable stats uri page in HAProxy tcp mode

Below is my haxproxy configuration. I am trying to enable stats page so that I can see the status of the servers. But after validating the file, getting a warning message as below and stats page is not shown after hitting the configured uri.
'stats' statement ignored for frontend 'http-in' as it requires HTTP mode
I am using haproxy version 1.8.x.
global
log /dev/log local0
log /dev/log local1 notice
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
timeout connect 120s
timeout client 300s
timeout server 180s
frontend http-in
bind *:443
stats enable
stats refresh 10s
stats uri /haproxy?stats
default_backend servers
backend servers
balance roundrobin
server sec-server1 192.68.29.230:8080 check
server sec-server2 192.68.29.229:8080 check
Please assist. can we not see the stats page in tcp mode?
Add the below things and try
global
global
# turn on stats unix socket
stats socket /var/lib/haproxy/stats mode 666 level admin
server-state-file /var/lib/haproxy/haproxy.state
defaults
defaults
default-server init-addr last,libc,none
load-server-state-from-file global
frontend
frontend http-in
bind *:443
option http-server-close
option forwardfor
http-request add-header X-Forwarded-Proto https
http-request add-header X-Forwarded-Port 1443
default_backend servers
stats uri /hap?_stats
stats auth haadm:username#pwd
stats hide-version
stats show-node

Haproxy 503 Service Unavailable No server is available to handle this request

I have balancing host
192.168.1.12
receive input HTTP/HTTPS traffic
and balancing on backends
10.0.1.12
10.0.1.13
Use
HA-Proxy version 1.8.4-1deb90d 2018/02/08
config
global
log 127.0.0.1 local2
chroot /var/opt/rh/rh-haproxy18/lib/haproxy
pidfile /var/run/rh-haproxy18-haproxy.pid
maxconn 20000
daemon
# turn on stats unix socket
stats socket /var/opt/rh/rh-haproxy18/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 15s
timeout server 15s
timeout http-keep-alive 5s
timeout check 3s
maxconn 20001
frontend http_frontend
bind *:80
default_backend http_backend
backend http_backend
mode http
server server1 10.0.1.12:8081 check
server server1 10.0.1.13:8081 check
Start service OK
Check curl
# curl -iv 10.0.1.12:8081
# curl -iv 10.0.1.13:8081
Return OK
Why
curl http://localhost
return 503 Service Unavailable
No server is available to handle this request.
?
here is my haproxy.cfg file hope this will help you to resolve your issue.
# Nur Load Balancer #
frontend tomcat-service
bind *:8081
default_backend tomcat-server
mode http
backend tomcat-server
balance roundrobin
server mfsys-cm-01 192.168.10.31:8080 check
server mfsys-cm-02 192.168.10.30:8080 check
listen stats
bind *:8082
stats enable
stats hide-version
stats show-node
stats uri /stats
stats auth admin:mypassword
stats refresh 5s
i got same error when i access stats, and solution was simple i was not giving proper url as it should be http://192.168.10.1:8082/stats

HAProxy with SSL passthrough to multiple domains with multiple backends

I need to setup a load balancer for all our applications.
At the moment all our applications are clustered (2-node appservers, and 1 apache on each node as well) and we do not have a LB so we just point our DNS alias to the first webserver of each node, making the second node useless (have to manually do a DNS switch in case of a failure of node1, and we don't have load balanced https queries).
Each application uses SSL with a specific domain & SSL certificate. we cannot accept to decrypt SSL and send unencrypted traffic to the backends as the LB might be located in another country etc. so we need to use passthrough.
Before anything, i just wanted to know if this is actually possible in HAProxy or not ?
I am talking about ~50 different applications. Our LB configuration would have to be HA so i guess we'll use something like keepalived with a shared VIP for HAProxy itself.
The setup would look like this i suppose :
domain-a.com-' '-> backend_dom_a -> 1.1.1.1 (app node1 dom a)
| | 1.1.1.2 (app node2 dom a)
domain-b.com-' '-> backend_dom_b -> 2.1.1.1 (app node1 dom b)
| | 2.1.1.2 (app node2 dom b)
domain-c.com-' '-> backend_dom_c -> 3.1.1.1 (app node1 dom c)
| | 3.1.1.2 (app node2 dom c)
domain-N.com-' '-> backend_dom_N -> 4.1.1.1 (app node1 dom N)
| | 4.1.1.2 (app node2 dom N)
+-> haproxy -+
Thanks for your support, best regards
FYI i'm using this configuration that works like a charm.
i have replaced the values in the files to hide our domains & hostnames, and limited the numbers of urls/backends but we have about 50 running now with the load balancer forwarding requests to many apache servers (and each apache forwards requests to tomcat servers behind)
feel free if you have any question
we use balance source to ensure session stickyness
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
daemon
user haproxy
group haproxy
log /dev/log local6 notice
log /dev/log local5 info
maxconn 50000
#chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode tcp
option tcplog
log global
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
#---------------------------------------------------------------------
# dedicated stats page
#---------------------------------------------------------------------
listen stats
mode http
bind :22222
stats enable
stats uri /haproxy?stats
stats realm Haproxy\ Statistics
stats auth <mylogin>:<mypass>
stats refresh 30s
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main_https_listen
bind <ip address>:443
mode tcp
option tcplog
log global
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
#---------------------------------------------------------------------
# Common HAProxy nodes configuration
#---------------------------------------------------------------------
# -------------------------------
# ACLs
# -------------------------------
acl acl_SIT_AT35073 req.ssl_sni -i <app_url1>.my.domain.net # SIT_AT35073 is just an internal code we use, but you can use any alias
acl acl_SIT_AT34305 req.ssl_sni -i <app_url2>.my.domain.net
acl acl_SIT_AT28548 req.ssl_sni -i <app_urlN>.my.domain.net
# -------------------------------
# Conditions
# -------------------------------
use_backend backend_SIT_AT35073 if acl_SIT_AT35073 # same here
use_backend backend_SIT_AT34305 if acl_SIT_AT34305
use_backend backend_SIT_AT28548 if acl_SIT_AT28548
#---------------------------------------------------------------------
# Backends
#---------------------------------------------------------------------
# APP 1
backend backend_SIT_AT35073
description APPNAME1
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT35073_1 <apache_server1>.my.domain.net:443 check
server server_SIT_AT35073_2 <apache_server2>.my.domain.net:443 check
# APP 2
backend backend_SIT_AT34305
description APPNAME2
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT34305_1 <apache_server3>.my.domain.net:443 check
server server_SIT_AT34305_2 <apache_server4>.my.domain.net:443 check
# APP N
backend backend_SIT_AT28548
description APPNAMEN
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT28548_1 <apache_server5>.my.domain.net:443 check
server server_SIT_AT28548_2 <apache_server6>.my.domain.net:443 check
I think you have two options:
pass the traffic through to the backend by using the TCP mode in haproxy frontend and backend. This has the benefit that your backend SSL certificate is passed through. Though you lose the possibility to have one SSL termination in your site. So I present you
Have one (usual) SSL certificate, acting as termination for your site and enable SSL between your backend and haproxy instance. This gives you the advantage that you still have only one entry point but different backends with unique certificates.
The second option might look like this:
frontend f_foo
bind :443 ssl crt /path/to/bundle
mode http
log global
use_backend b2_foo
backend be_foo
mode http
timeout connect 5s
server FOO address:port ssl check crt /path/to/client/bundle force-tlsv10 verify none
The drawback is that you need a client certificate for each backend server but that should be easily automatable.
AS more of an update answer for multi domain configs I use the below for routing different domains.
in the frontend is where you bind the port and add the certs which multiple have to be on the same line afaik.
frontend https_in
bind *:443 ssl crt /link/to/cert+key-file.pem crt /link/to/cert+key-file.pem
The acl host is where you specify the domain name and which backend to use based on that domain name.
acl host_example.com hdr(host) -i example.com
use_backend BACKEND_NAME if host_example.com
The backend where you specify the server that domain is running on.
backend BACKEND_NAME
mode http
option httpclose
option forwardfor
cookie JSESSIONID prefix
server server-name server-ip:443 check ssl verify none

HAProxy using 100% CPU 200 concurrent connections

I've recently started to load test my app and found that HAProxy for some reason is not able to handle a lot of concurrent connections.
Im only using HAProxy to load balance my SSL traffic, for non-ssl (99% of my traffic is ssl) i use nginx.
I have tested my setup on blitz.io and when sending traffic to non-ssl (200 concurrent) i get no timeouts or errors. However when doing the same test over SSL (which HAProxy handles) i immediately get 100% CPU and requests are timing out.
This leads me to believe there is something wrong in my HAProxy config.
Below is my config, any ideas what could be wrong?
Oh and i am running this on a medium EC2 CPU optimized instance
My haproxy.cfg:
global
maxconn 400000
ulimit-n 800019
nbproc 1
debug
daemon
log 127.0.0.1 local0 notice
defaults
mode http
option httplog
log global
stats enable
stats refresh 60s
stats uri /stats
maxconn 32768
frontend secured
timeout client 86400000
mode http
timeout client 120s
option httpclose
#option forceclose
option forwardfor
bind 0.0.0.0:443 ssl crt /etc/nginx/ssl/ssl-bundle.pem
acl is_sockjs path_beg /echo /broadcast /close # SockJS
acl is_express path_beg /probe /loadHistory /activity # Express
use_backend www_express if is_express
use_backend sockjs if is_sockjs
default_backend www_nginx
backend tcp_socket
mode http
server server1 xx.xx.xx.xx:8080 check port 8080
backend www_express
mode http
option forwardfor #this sets X-Forwarded-For
timeout server 30000
timeout connect 4000
server server1 xx.xx.xx.xx:8008 weight 1 maxconn 32768 check
backend sockjs
mode http
option forwardfor #this sets X-Forwarded-For
timeout server 30000
timeout connect 4000
server server1 xx.xx.xx.xx:8081 weight 1 maxconn 32768 check
backend www_nginx
mode http
option forwardfor #this sets X-Forwarded-For
timeout server 30000
timeout connect 4000
server server1 localhost:80 weight 1 maxconn 8024 check
listen stats :8181
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth helloxx:xx