Cannot get TCP port information from Kubernetes host-Openshift Origin-oc - openshift-origin

I was following Openshift's Local Cluster Management documentation.
After I ran oc cluster up
[root#user ~]# oc cluster up
Starting OpenShift using openshift/origin:v3.6.0 ...
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ...
WARNING: Docker version is 1.21, it needs to be >= 1.22
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v3.6.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... FAIL
Error: Cannot get TCP port information from Kubernetes host
Caused By:
Error: cannot create container using image openshift/origin:v3.6.0
Caused By:
Error: Error response from daemon: SHM size must be greater then 0
[root#ip-172-31-0-186 ~]# oc cluster up --loglevel=5
-- Checking OpenShift client ...
-- Checking Docker client ...
I0803 04:30:33.543172 1417 up.go:590] No Docker environment variables found. Will attempt default socket.
I0803 04:30:33.543221 1417 up.go:595] No Docker host (DOCKER_HOST) configured. Will attempt default socket.
-- Checking Docker version ...
I0803 04:30:33.543240 1417 helper.go:114] Retrieving Docker version
I0803 04:30:33.554087 1417 helper.go:120] Docker version results: &types.Version{Version:"1.9.1", APIVersion:"1.21", GitCommit:"78ee77d/1.9.1", GoVersion:"go1.4.2", Os:"linux", Arch:"amd64", KernelVersion:"3.10.0-693.el7.x86_64", Experimental:false, BuildTime:""}
I0803 04:30:33.554126 1417 helper.go:124] APIVersion: 1.21
I0803 04:30:33.554158 1417 up.go:686] Checking that docker API version is at least 1.22
WARNING: Docker version is 1.21, it needs to be >= 1.22
-- Checking for existing OpenShift container ...
I0803 04:30:33.554181 1417 helper.go:171] Inspecting docker container "origin"
I0803 04:30:33.555084 1417 helper.go:175] Container "origin" was not found
-- Checking for openshift/origin:v3.6.0 image ...
I0803 04:30:33.555101 1417 helper.go:143] Inspecting Docker image "openshift/origin:v3.6.0"
I0803 04:30:33.556444 1417 helper.go:146] Image "openshift/origin:v3.6.0" found: &types.ImageInspect{ID:"c6d16974c8a3a5da3ab799533daa2dbd54e56b1f0ebbad59345154fc8e836ff2", RepoTags:[]string{"docker.io/openshift/origin:v3.6.0"}, RepoDigests:[]string{}, Parent:"395d30169bc02cca2e7083926b0fd6f2e6b7034a6de41a811cce0ab7c7473fca", Comment:"", Created:"2017-08-01T18:34:13.736398725Z", Container:"ae53137cc1b98b2f93051589d6aee252e505ac82f8e7a31f5ab49bfc0e9dc91a", ContainerConfig:(*container.Config)(0xc420277b00), DockerVersion:"1.12.6", Author:"", Config:(*container.Config)(0xc4202e2120), Architecture:"amd64", Os:"linux", Size:611206034, VirtualSize:974248741, GraphDriver:types.GraphDriverData{Name:"devicemapper", Data:map[string]string{"DeviceId":"7", "DeviceName":"docker-202:2-25214823-c6d16974c8a3a5da3ab799533daa2dbd54e56b1f0ebbad59345154fc8e836ff2", "DeviceSize":"107374182400"}}, RootFS:types.RootFS{Type:"", Layers:[]string(nil), BaseLayer:""}}
-- Checking Docker daemon configuration ...
I0803 04:30:33.556503 1417 helper.go:65] Retrieving Docker daemon info
I0803 04:30:33.681753 1417 helper.go:71] Docker daemon info: &types.Info{ID:"IITV:S6LY:XNQS:LA63:VAH6:POZR:RGCW:MFWK:OTI7:DEII:AQK5:FDC6", Containers:0, ContainersRunning:0, ContainersPaused:0, ContainersStopped:0, Images:6, Driver:"devicemapper", DriverStatus:[][2]string{[2]string{"Pool Name", "docker-202:2-25214823-pool"}, [2]string{"Pool Blocksize", "65.54 kB"}, [2]string{"Base Device Size", "107.4 GB"}, [2]string{"Backing Filesystem", ""}, [2]string{"Data file", "/dev/loop0"}, [2]string{"Metadata file", "/dev/loop1"}, [2]string{"Data Space Used", "1.091 GB"}, [2]string{"Data Space Total", "107.4 GB"}, [2]string{"Data Space Available", "18.09 GB"}, [2]string{"Metadata Space Used", "1.339 MB"}, [2]string{"Metadata Space Total", "2.147 GB"}, [2]string{"Metadata Space Available", "2.146 GB"}, [2]string{"Udev Sync Supported", "true"}, [2]string{"Deferred Removal Enabled", "false"}, [2]string{"Deferred Deletion Enabled", "false"}, [2]string{"Deferred Deleted Device Count", "0"}, [2]string{"Data loop file", "/var/lib/docker/devicemapper/devicemapper/data"}, [2]string{"Metadata loop file", "/var/lib/docker/devicemapper/devicemapper/metadata"}, [2]string{"Library Version", "1.02.140-RHEL7 (2017-05-03)"}}, SystemStatus:[][2]string(nil), Plugins:types.PluginsInfo{Volume:[]string(nil), Network:[]string(nil), Authorization:[]string(nil)}, MemoryLimit:true, SwapLimit:true, KernelMemory:false, CPUCfsPeriod:true, CPUCfsQuota:true, CPUShares:false, CPUSet:false, IPv4Forwarding:true, BridgeNfIptables:true, BridgeNfIP6tables:true, Debug:false, NFd:15, OomKillDisable:true, NGoroutines:25, SystemTime:"2017-08-03T04:30:33.681150233-04:00", ExecutionDriver:"native-0.2", LoggingDriver:"json-file", CgroupDriver:"", NEventsListener:0, KernelVersion:"3.10.0-693.el7.x86_64", OperatingSystem:"Red Hat Enterprise Linux Server 7.4 (Maipo)", OSType:"", Architecture:"", IndexServerAddress:"https://index.docker.io/v1/", RegistryConfig:(*registry.ServiceConfig)(0xc4210fb700), NCPU:2, MemTotal:3973541888, DockerRootDir:"/var/lib/docker", HTTPProxy:"", HTTPSProxy:"", NoProxy:"", Name:"ip-172-31-0-186.us-west-2.compute.internal", Labels:[]string(nil), ExperimentalBuild:false, ServerVersion:"1.9.1", ClusterStore:"", ClusterAdvertise:"", SecurityOptions:[]string(nil)}
I0803 04:30:33.681847 1417 helper.go:42] Looking for "172.30.0.0/16" in []*registry.NetIPNet{(*registry.NetIPNet)(0xc4210f1a10), (*registry.NetIPNet)(0xc4210f1a70)}
I0803 04:30:33.681859 1417 helper.go:46] Found "172.30.0.0/16"
-- Checking for available ports ...
I0803 04:30:33.681920 1417 run.go:181] Creating container named ""
config:
image: openshift/origin:v3.6.0
entry point:
/bin/bash
command:
-c
cat /proc/net/tcp && ( [ -e /proc/net/tcp6 ] && cat /proc/net/tcp6 || true)
host config:
pid mode: host
user mode:
network mode: host
FAIL
Error: Cannot get TCP port information from Kubernetes host
Caused By:
Error: cannot create container using image openshift/origin:v3.6.0
Caused By:
Error: Error response from daemon: SHM size must be greater then 0
I have placed kubernetes config file in .kube/config. Still getting same error. Kubernetes cluster should be in same machine?
UPDATE-1
Install latest version from docker docs
To resolve dependency, installed container-selinux (sudo yum install ftp://fr2.rpmfind.net/linux/centos/7.3.1611/extras/x86_64/Packages/container-selinux-2.9-4.el7.noarch.rpm)
After I try to bringup the cluster with oc cluster up. This time, it failing at docker configuration.
[root#ip-172-31-0-186 ~]# oc cluster up
Starting OpenShift using openshift/origin:v3.6.0 ...
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v3.6.0 image ... OK
-- Checking Docker daemon configuration ... FAIL
Error: did not detect an --insecure-registry argument on the Docker daemon
Solution:
Ensure that the Docker daemon is running with the following argument:
--insecure-registry 172.30.0.0/16
Docs says, update the --insecure-registry 172.30.0.0/16 in /etc/sysconf/docker. But for new version of docker there is no file in that location. Any way I have created and updated the /etc/sysconf/docker. But still getting the above error.

Ok, the problem is insecure registry configuration. Specify insecure registry in daemon.json in /etc/docker with below config
{
"insecure-registries": [
"172.30.0.0/16"
]
}
This is working latest docker also.

For any particular version of Kubernetes or OpenShift the supported Docker version is little behind.
So I would advise you to install docker not the latest one from docker documentation. But install using your linux distribution's package manager. For Fedora and CentOS just do:
sudo yum install -y docker
Once you have done that all the dependency management will be taken care of and you don't need to manually install anything else.
Now that you have installed docker using the package manager you will find the /etc/sysconfig/docker. And you can add that line --insecure-registry 172.30.0.0/16.
HTH.

Related

haproxy pods keep crashloopbackoff

I'm setting up a redis-ha in my kubernetes cluster. And I used helm to install it. But my haproxy pods keep crashloopbackoff
I'm using helm to install a redis-ha in my kubernetes cluster with command: helm install -f develop-redis-values.yaml stable/redis-ha --namespace=develop -n=develop-redis
In develop-redis-values.yaml, I set haproxy.enabled to true
This is the logs in my crashloopbackoff pod
> [ALERT] 268/104750 (1) : parsing [/usr/local/etc/haproxy/haproxy.cfg:34] : 'tcp-check expect string' expects <string> as an argument.
> [ALERT] 268/104750 (1) : Error(s) found in configuration file : /usr/local/etc/haproxy/haproxy.cfg
> [ALERT] 268/104750 (1) : Fatal errors found in configuration.
I'm expected the haproxy pods is running
CrashLoopBackError can be related to these possible reasons:
the application inside your pod is not starting due to an error;
the image your pod is based on is not present in the registry, or the node where your pod has been scheduled cannot pull from the registry;
some parameters of the pod has not been configured correctly.
In your case, it seems that there are some errors in you haproxy configuration files.
Have you tried to pull the image you're using locally, and start a container to verify it?
You can enter in the container and check the configuration with:
haproxy -c -V -f /usr/local/etc/haproxy/haproxy.cfg
For more information and debugging ways:
https://pillsfromtheweb.blogspot.com/2020/05/troubleshooting-kubernetes.html

minishift start using virtualbox on windows throws timeout error

I am trying to start minishift on my machine. It successfully creates minishift VM but throws time out error.
Configuration:
Minishift version: v1.34.0+f5db7cb
OS: Windows 10
Hypervisor: Virtual Box v6.0.10
PS C:\WINDOWS\system32> minishift start
-- Starting OpenShift cluster .......................................................................Error during 'cluster up' execution: Error starting the cluster. ssh command error:
command : /var/lib/minishift/bin/oc cluster up --image 'openshift/origin-${component}:v3.11.0' --public-hostname 192.168.99.100 --routing-suffix 192.168.99.100.nip.io --base-dir /var/lib/minishift/base
err : exit status 1
output : Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11.0 is available ...
Pulling image openshift/origin-cli:v3.11.0
E0725 17:15:42.919928 5316 helper.go:173] Reading docker config from /home/docker/.docker/config.json failed: open /home/docker/.docker/config.json: no such file or directory, will attempt to pull image docker.io/openshift/origin-cli:v3.11.0 anonymously
Image pull complete
E0725 17:15:44.643860 5316 helper.go:173] Reading docker config from /home/docker/.docker/config.json failed: open /home/docker/.docker/config.json: no such file or directory, will attempt to pull image docker.io/openshift/origin-node:v3.11.0 anonymously
Pulling image openshift/origin-node:v3.11.0
Pulled 5/6 layers, 85% complete
Pulled 6/6 layers, 100% complete
Extracting
Image pull complete
Checking type of volume mount ...
Determining server IP ...
Using public hostname IP 192.168.99.100 as the host IP
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11.0 is available ...
I0725 17:16:20.775520 5316 config.go:40] Running "create-master-config"
Starting OpenShift using openshift/origin-control-plane:v3.11.0 ...
I0725 17:16:31.108342 5316 config.go:46] Running "create-node-config"
I0725 17:16:35.237968 5316 flags.go:30] Running "create-kubelet-flags"
I0725 17:16:36.785234 5316 run_kubelet.go:49] Running "start-kubelet"
I0725 17:16:37.288388 5316 run_self_hosted.go:181] Waiting for the kube-apiserver to be ready ...
E0725 17:21:37.300062 5316 run_self_hosted.go:571] API server error: Get https://192.168.99.100:8443/healthz?timeout=32s: dial tcp 192.168.99.100:8443: connect: connection refused ()
Error: timed out waiting for the condition
Expected result: It should provide me without errors link to open web console
This happens to me sometimes too.
Solutions include:
minishift stop && minishift start (turn it off and on again)
restart Windows (perhaps VBox has corrupted itself again)
More info on my minishift setup is here:
http://divby0.blogspot.com/2019/07/configuring-minishift-for-use-with.html
For what it's worth I use a combination of linux shells in Windows 10 to interact with minishift / docker daemon:
Git Bash (usually the best)
Docker Toolbox (plan B when something won't run in Git Bash shell)
WSL (ubuntu based, plan C in desperation)

How to install schema registry

I am looking options to install confluent schema registry, is it possible to download and install registry alone and make it work with existing kafka setup ?
Thanks
Assuming you have Zookeeper/Kafka running already, you can easily run Confulent Schema Registry using Docker with running the following command:
docker run -p 8081:8081 -e \
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=host.docker.internal:2181 \
-e SCHEMA_REGISTRY_HOST_NAME=localhost \
-e SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081 \
-e SCHEMA_REGISTRY_DEBUG=true confluentinc/cp-schema-registry:5.3.2
parameters:
-p 8081:8081 - will open the port 8081 between the container to your machine
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL - is your Zookeeper host and port, I'm using host.docker.internal to resolve local machine that is hosting Zookeeper (outside of the container)
SCHEMA_REGISTRY_HOST_NAME - The hostname advertised in Zookeeper. This is required if if you are running Schema Registry with multiple nodes. Hostname is required because it defaults to the Java canonical hostname for the container, which may not always be resolvable in a Docker environment.
SCHEMA_REGISTRY_LISTENERS - the Schema Registry host and port number to open
SCHEMA_REGISTRY_DEBUG Run in debug mode
note: the script was using the version 5.3.2, make sure this version is aligned with your Kafka version.
Yes you can use your existing Kafka setup, just match to the compatible version of Confluent Platform. Here are the docs on getting started
https://docs.confluent.io/current/schema-registry/docs/intro.html#installation
tl;dr download the platform to pull out the pieces you need or get the docker image and point it at your Kafka cluster.

Redis installed on FreeBSD via pkg or ports can not run

I had installed redis on FreeBSD via pkg yesterday.And also add redis_enable="YES"in my /etc/rc.conf.
When I run service redis start,It displays Starting redis,but when run service redis status,displays redis is not running.
After run redis-server,It runs,but use redis-cli cannot connect the server.
When run redis-cli or redis-cli ping,it will jammed with no error.
The redis log file also has no error.
I cannot find the solution with google or others。How should I do to use redis?
The same with installed redis via ports
The followings are the base information about my company:
FreeBSD version: 10.3-RELEASE-p18
Pkg version: 1.10.1
Redis version: 3.29
I would suggest to update, reinstall by doing this:
# pkg update
then:
# pkg upgrade -y
To reinstall redis:
# pkg remove -y redis
To install it again:
# pkg install -y redis
At this point you should see this in the terminal:
===> CONFIGURATION NOTE:
To setup "redis" you need to edit the configuration file:
/usr/local/etc/redis.conf
To run redis from startup, add redis_enable="YES"
in your /etc/rc.conf.
To ensure redis is enabled on startup use sysrc
# sysrc redis_enable="YES"
Not this time try to starting calling the script directly:
# /usr/local/etc/rc.d/redis start
You could use sockstat to check if is up and listenin:
# sockstat -4l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
redis redis-serv 9553 4 tcp4 127.0.0.1:6379 *:*
After this, redis-cli, or telnet 0 6379 should be working:
# telnet 0 6379
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
If that still not working, maybe ktrace can help to find more details:
To trace a PID:
$ ktrace -p 9553
To stop tracing:
# ktrace -C
use kdump to display kernel trace data
# kdump -f my-ktrace.out | less
The first thing to do on BSD when a command "hungs" is to press ^T (ie ctrl-t). Can you do that and paste the output?

How check the httpd is enabled and running using InSpec with Kitchen-docker on CentOS?

Running my test with InSpec I am unable to test if the httpd is enabled and running.
InSpec test
describe package 'httpd' do
it { should be_installed }
end
describe service 'httpd' do
it { should be_enabled }
it { should be_running }
end
describe port 80 do
it { should be_listening }
end
The output for kitchen verify is:
System Package
✔ httpd should be installed
Service httpd
✖ should be enabled
expected that `Service httpd` is enabled
✖ should be running
expected that `Service httpd` is running
Port 80
✖ should be listening
expected `Port 80.listening?` to return true, got false
Test Summary: 1 successful, 3 failures, 0 skipped
Recipe for httpd installation:
if node['platform'] == 'centos'
# do centos installation
package 'httpd' do
action :install
end
execute "chkconfig httpd on" do
command "chkconfig httpd on"
end
execute 'apache start' do
command '/usr/sbin/httpd -DFOREGROUND &'
action :run
end
I do not know what I am doing wrong.
More info
CentOS version on docker instance
kitchen exec --command 'cat /etc/centos-release'
-----> Execute command on default-centos-72.
CentOS Linux release 7.2.1511 (Core)
Chef version installed in my host
Chef Development Kit Version: 1.0.3
chef-client version: 12.16.42
delivery version: master (83358fb62c0f711c70ad5a81030a6cae4017f103)
berks version: 5.2.0
kitchen version: 1.13.2
UPDATE 1: Kitchen yml with driver attributes
The platform has the configuration recommended by coderanger :
---
driver:
name: docker
use_sudo: false
provisioner:
name: chef_zero
verifier: inspec
platforms:
- name: centos-7.2
driver:
platform: rhel
run_command: /usr/lib/systemd/systemd
provision_command:
- /bin/yum install -y iniscripts net-tools wget
suites:
- name: default
run_list:
- recipe[apache::default]
verifier:
inspec_tests:
- test/integration
attributes:
And it is the output when run kitchen test:
... some docker steps...
Step 16 : RUN echo ssh-rsa\ AAAAB3NzaC1yc2EAAAADAQABAAABAQDIp1HE9Zbtl3zAH2KKL1mVzb7BU1WxK7mi5xpIxNRBar7EZAAzxi1pVb1JwUXFSCVoAmUyfn/lBsKlgXnUD49pKrqkeLQQW7NoG3uCFiXBUTof8nFVuLYtw4CTiAudplyMvu5J7HQIP1Hve1caY27tFs/kpkQaXHCEuIkqgrM2rreMKK0n8im9b36L2SwWyM/GwqcIS1z9mMttid7ux0\+HOWWHqZ\+7gumOauh6tLRbtjrm3YYoaIAMyv945MIX8BFPXSQixThBVOlXGA9iTwUZWjU6WvZThxVFkKPR9KZtUTuTCT7Y8\+wFtQ/9XCHpPR00YDQvS0Vgdb/LhZUDoNqV\ kitchen_docker_key >> /home/kitchen/.ssh/authorized_keys
---> Using cache
---> c0e6b9e98d6a
Successfully built c0e6b9e98d6a
d486d7ebfe000a3138db06b1424c943a0a1ee7b2a00e8a396cb8c09f9527fb4b
0.0.0.0:32841
Waiting for SSH service on localhost:32841, retrying in 3 seconds
Waiting for SSH service on localhost:32841, retrying in 3 seconds
Waiting for SSH service on localhost:32841, retrying in 3 seconds
Waiting for SSH service on localhost:32841, retrying in 3 seconds
.....
You cannot, at least not out of the box. This is one area where kitchen-docker shows its edges. We try to pretend that a container is like a tiny VM but in reality it isn't, and one notable place where the pretending breaks down is init systems. With CentOS 7, it uses systemd. It is possible to get systemd to run inside the container (see https://github.com/poise/yolover-example/blob/master/.kitchen.yml#L17-L33) but not all features are supported and it can generally be a bit odd :-/ That example should be enough to make your tests work though. For completeness, CentOS 6 uses Upstart which just flat out won't run inside Docker so no love there either.