Selenium 4: Chrome Node does not register correctly to the hub - selenium

I have an Openshift 3 Cluster containing the two following containers: selenium-hub and selenium-node-chrome. Please see below the attached deployment and service yaml files.
Hub Deployment:
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
labels:
app: selenium-hub
selenium-hub: master
name: selenium-hub
spec:
replicas: 1
selector:
type: selenium-hub
template:
metadata:
labels:
type: selenium-hub
name: selenium-hub
spec:
containers:
- image: 'selenium/hub:latest'
imagePullPolicy: IfNotPresent
name: master
ports:
- containerPort: 4444
protocol: TCP
- containerPort: 4442
protocol: TCP
- containerPort: 4443
protocol: TCP
triggers:
- type: ConfigChange
Hub Service:
apiVersion: v1
kind: Service
metadata:
labels:
app: selenium-hub
selenium-hub: master
name: selenium-hub
spec:
ports:
- name: selenium-hub
port: 4444
protocol: TCP
targetPort: 4444
- name: publish
port: 4442
protocol: TCP
targetPort: 4442
- name: subscribe
port: 4443
protocol: TCP
targetPort: 4443
selector:
type: selenium-hub
type: ClusterIP
Node Deployment:
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
labels:
app: selenium-node-chrome
name: selenium-node-chrome
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
browser: chrome
template:
metadata:
labels:
app: node-chrome
browser: chrome
name: selenium-node-chrome-master
spec:
containers:
- env:
- name: SE_EVENT_BUS_HOST
value: selenium-hub
- name: SE_EVENT_BUS_PUBLISH_PORT
value: '4442'
- name: SE_EVENT_BUS_SUBSCRIBE_PORT
value: '4443'
- name: SE_NODE_HOST
value: node-chrome
- name: SE_NODE_PORT
value: '5555'
image: 'selenium/node-chrome:4.0.0-20211102'
imagePullPolicy: IfNotPresent
name: master
ports:
- containerPort: 5555
protocol: TCP
triggers:
- type: ConfigChange
Node Service:
apiVersion: v1
kind: Service
metadata:
labels:
app: selenium-node-chrome
name: selenium-node-chrome
spec:
ports:
- name: node-port
port: 5555
protocol: TCP
targetPort: 5555
- name: node-port-grid
port: 4444
protocol: TCP
targetPort: 4444
selector:
browser: chrome
type: ClusterIP
My Issue:
The hub and the node are starting, but the node just keeps sending the registration event and the hub is logging some infos, which i dont really understand. Please see the logs attached below.
Node Log:
Setting up SE_NODE_GRID_URL...
Selenium Grid Node configuration:
[events]
publish = "tcp://selenium-hub:4442"
subscribe = "tcp://selenium-hub:4443"
[server]
host = "node-chrome"
port = "5555"
[node]
session-timeout = "300"
override-max-sessions = false
detect-drivers = false
max-sessions = 1
[[node.driver-configuration]]
display-name = "chrome"
stereotype = '{"browserName": "chrome", "browserVersion": "95.0", "platformName": "Linux"}'
max-sessions = 1
Starting Selenium Grid Node...
11:34:31.635 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
11:34:31.643 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
11:34:31.774 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://selenium-hub:4442 and tcp://selenium-hub:4443
11:34:31.843 INFO [UnboundZmqEventBus.<init>] - Sockets created
11:34:32.854 INFO [UnboundZmqEventBus.<init>] - Event bus ready
11:34:33.018 INFO [NodeServer.createHandlers] - Reporting self as: http://node-chrome:5555
11:34:33.044 INFO [NodeOptions.getSessionFactories] - Detected 1 available processors
11:34:33.115 INFO [NodeOptions.report] - Adding chrome for {"browserVersion": "95.0","browserName": "chrome","platformName": "Linux","se:vncEnabled": true} 1 times
11:34:33.130 INFO [Node.<init>] - Binding additional locator mechanisms: name, relative, id
11:34:33.471 INFO [NodeServer$1.start] - Starting registration process for node id 2832e819-cf31-4bd9-afcc-cd2b27578d58
11:34:33.473 INFO [NodeServer.execute] - Started Selenium node 4.0.0 (revision 3a21814679): http://node-chrome:5555
11:34:33.476 INFO [NodeServer$1.lambda$start$1] - Sending registration event...
11:34:43.479 INFO [NodeServer$1.lambda$start$1] - Sending registration event...
11:34:53.481 INFO [NodeServer$1.lambda$start$1] - Sending registration event...
Hub Log:
2021-12-07 11:14:22,663 INFO spawned: 'selenium-grid-hub' with pid 11
2021-12-07 11:14:23,664 INFO success: selenium-grid-hub entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
11:14:23.953 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
11:14:23.961 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
11:14:24.136 INFO [BoundZmqEventBus.<init>] - XPUB binding to [binding to tcp://*:4442, advertising as tcp://XXXXXXX:4442], XSUB binding to [binding to tcp://*:4443, advertising as tcp://XXXXXX:4443]
11:14:24.246 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://XXXXXX:4442 and tcp://XXXXXXX:4443
11:14:24.275 INFO [UnboundZmqEventBus.<init>] - Sockets created
11:14:25.278 INFO [UnboundZmqEventBus.<init>] - Event bus ready
11:14:26.232 INFO [Hub.execute] - Started Selenium Hub 4.1.0 (revision 87802e897b): http://XXXXXXX:4444
11:14:46.965 INFO [Node.<init>] - Binding additional locator mechanisms: name, relative, id
11:15:46.916 INFO [Node.<init>] - Binding additional locator mechanisms: relative, name, id
11:17:52.377 INFO [Node.<init>] - Binding additional locator mechanisms: relative, id, name
Can anyone tell me why the hub wont register the node?
If you need any further informations, let me know.
Thanks alot

So, bit late, but still I had this same issue - the docker-compose example gave me selenium-hub as the host, which is correct in that scenario as it points towards the container defined by the selenium-hub service.
However, in Kubernetes, the inter-pod communication needs to go via a Service. There are multiple kinds of Service, but in order to access it from inside the cluster, it's easiest in this case to use a ClusterIP (docs here for more info).
The way I resolved it was to have a Service for both the ports that the event bus uses:
bus-publisher (port 4442)
bus-subscription (port 4443)
In a manifest yaml, this looks like:
apiVersion: v1
kind: Service
metadata:
labels:
app-name: selenium
name: bus-sub
namespace: selenium
spec:
ports:
- port: 4443
protocol: TCP
targetPort: 4443
selector:
app: selenium-hub
type: ClusterIP

you didn't expose the ports 4443 and 4442 from the hub container (see ports section of spec.containers)

You are in same machine so I think you don't need to use the environment variable: SE_NODE_HOST in the node deployment only use these variables:
SE_EVENT_BUS_HOST=selenium-hub
SE_EVENT_BUS_PUBLISH_PORT=4442
SE_EVENT_BUS_SUBSCRIBE_PORT=4443
If you think you aren't in the same VM, you need to config the node deployment correctly by using these environment variables :
SE_EVENT_BUS_HOST=<ip-of-hub-machine>
SE_EVENT_BUS_PUBLISH_PORT=4442
SE_EVENT_BUS_SUBSCRIBE_PORT=4443
SE_NODE_HOST=<ip-of-node-machine>
Please don't add unused environment variables like:'SE_NODE_PORT' because selenium image doesn't support different environment variables besides the environment variables you can read in the documents in Github 'docker-selenium' project: https://github.com/SeleniumHQ/docker-selenium.
If you are so much want to use your variable. So create your own selenium image (I don't recommend that) I succuss with what I say to you.

Related

DEX and Amazonn ALB Load Balancer Controller and Argo Workflows

I'm trying to build ALB -> Kube -> Dex using Load Balancer Controller. As a result, I have ALB with correctly binding instances into the target group, but the instance is Unhealthy.
The load Balancer Controller uses the 31845 as a health check port. A tried the port 5556, but still unhealthy.
So I can assume the setting is correct. But I'm not sure.
Another possibility, the DEX container isn't set up correctly.
And yet another version, I configured everything in the wrong way.
Does anyone have already configured DEX in this way and can prompt me?
Dex service
apiVersion: v1
kind: Service
metadata:
name: dex
...
spec:
ports:
- name: http
protocol: TCP
appProtocol: http
port: 5556
targetPort: http
nodePort: 31845
...
selector:
app.kubernetes.io/instance: dex
app.kubernetes.io/name: dex
clusterIP: 172.20.97.132
clusterIPs:
- 172.20.97.132
type: NodePort
sessionAffinity: None
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
internalTrafficPolicy: Cluster
DEX pod
containerStatuses:
- name: dex
state:
running:
startedAt: '2022-09-19T17:41:43Z'
...
containers:
- name: dex
image: ghcr.io/dexidp/dex:v2.34.0
args:
- dex
- serve
- '--web-http-addr'
- 0.0.0.0:5556
- '--telemetry-addr'
- 0.0.0.0:5558
- /etc/dex/config.yaml
ports:
- name: http
containerPort: 5556
protocol: TCP
- name: telemetry
containerPort: 5558
protocol: TCP
env:
- name: ARGO_WORKFLOWS_SSO_CLIENT_SECRET
load Balancer Controller
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ${name_http_ingress}
namespace: ${namespace}
labels:
app.kubernetes.io/component: server
app.kubernetes.io/instance: argo-cd
app.kubernetes.io/part-of: argocd
app.kubernetes.io/name: argocd-server
annotations:
alb.ingress.kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/backend-protocol-version: HTTP1
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-path: /
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '10'
alb.ingress.kubernetes.io/unhealthy-threshold-count: '3'
alb.ingress.kubernetes.io/success-codes: 200,301,302,307
alb.ingress.kubernetes.io/conditions.argogrpc: >-
[{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["^application/grpc.*$"]}}]
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: >-
{"type":"redirect","redirectConfig":{"port":"443","protocol":"HTTPS","statusCode":"HTTP_301"}}
# external-dns.alpha.kubernetes.io/hostname: ${domain_name_public}
alb.ingress.kubernetes.io/certificate-arn: ${domain_certificate}
# alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/load-balancer-name: ${name_http_ingress}
alb.ingress.kubernetes.io/target-type: instance
# alb.ingress.kubernetes.io/target-type: ip # require to enable sticky sessions ,stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60
alb.ingress.kubernetes.io/target-group-attributes: load_balancing.algorithm.type=least_outstanding_requests
alb.ingress.kubernetes.io/target-node-labels: ${tolerations_key}=${tolerations_value}
alb.ingress.kubernetes.io/tags: Environment=${tags_env},Restricted=false,Customer=customer,Project=ops,Name=${name_http_ingress}
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true,idle_timeout.timeout_seconds=180
spec:
ingressClassName: alb
tls:
- hosts:
- ${domain_name_public}
- ${domain_name_public_dex}
rules:
- host: ${domain_name_public}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ssl-redirect
port:
name: use-annotation
- host: ${domain_name_public_dex}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ssl-redirect
port:
name: use-annotation
- host: ${domain_name_public_dex}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: dex
port:
number: 5556

Zalenium: 504 Gateway Time-out in OpenShift environment

My Zalenium installation on an OpenShift environment is far from stable. The web ui (admin view with vnc, dashboard, selenium console) works about 50% of the time and connecting with a RemoteWebDriver doesn't work at all.
Error:
504 Gateway Time-out
The server didn't respond in time.
WebDriver error:
org.openqa.selenium.WebDriverException: Unable to parse remote response: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:115)
oc version:
oc v3.9.0+191fece
kubernetes v1.9.1+a0ce1bc657
Zalenium template:
apiVersion: v1
kind: Template
metadata:
name: zalenium
annotations:
"openshift.io/display-name": "Zalenium"
"description": "Disposable Selenium Grid for use in OpenShift"
message: |-
A Zalenium grid has been created in your project. Continue to overview to verify that it exists and start the deployment.
parameters:
- name: PROJECTNAME
description: The namespace / project name of this project
displayName: Namespace
required: true
- name: HOSTNAME
description: hostname used for route creation
displayName: route hostname
required: true
- name: "VOLUME_CAPACITY"
displayName: "Volume capacity for the disk that contains the test results."
description: "The volume is used to store all the test results, including logs and video recordings of the tests."
value: "10Gi"
required: true
objects:
- apiVersion: v1
kind: DeploymentConfig
metadata:
generation: 1
labels:
app: zalenium
role: hub
name: zalenium
spec:
replicas: 1
selector:
app: zalenium
role: hub
strategy:
activeDeadlineSeconds: 21600
resources: {}
type: Rolling
template:
metadata:
labels:
app: zalenium
role: hub
spec:
containers:
- args:
- start
- --seleniumImageName
- "elgalu/selenium:latest"
- --sendAnonymousUsageInfo
- "false"
image: dosel/zalenium:latest
imagePullPolicy: Always
name: zalenium
ports:
- containerPort: 4444
protocol: TCP
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /home/seluser/videos
name: zalenium-volume
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext: {}
serviceAccount: deployer
serviceAccountName: deployer
volumes:
- name: zalenium-volume
persistentVolumeClaim:
claimName: zalenium-pvc
test: false
triggers:
- type: ConfigChange
- apiVersion: v1
kind: Route
metadata:
labels:
app: zalenium
annotations:
openshift.io/host.generated: 'true'
haproxy.router.openshift.io/timeout: "60"
name: zalenium
spec:
host: zalenium-4444-${PROJECTNAME}.${HOSTNAME}
to:
kind: Service
name: zalenium
port:
targetPort: selenium-4444
- apiVersion: v1
kind: Route
metadata:
labels:
app: zalenium
annotations:
openshift.io/host.generated: 'true'
haproxy.router.openshift.io/timeout: "60"
name: zalenium-4445
spec:
host: zalenium-4445-${PROJECTNAME}.${HOSTNAME}
to:
kind: Service
name: zalenium
port:
targetPort: selenium-4445
- apiVersion: v1
kind: Service
metadata:
labels:
app: zalenium
name: zalenium
spec:
ports:
- name: selenium-4444
port: 4444
protocol: TCP
targetPort: 4444
- name: selenium-4445
port: 4445
protocol: TCP
targetPort: 4445
selector:
app: zalenium
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: zalenium
name: zalenium-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: ${VOLUME_CAPACITY}
Errors in main pod:
I get about 2-3 errors in 30 minutes.
[OkHttp https://172.17.0.1/ ...] ERROR i.f.k.c.d.i.ExecWebSocketListener - Exec Failure: HTTP:403. Message:pods "zalenium-40000-wvpjb" is forbidden: User "system:serviceaccount:PROJECT:deployer" cannot get pods/exec in the namespace "PROJECT": User "system:serviceaccount:PROJECT:deployer" cannot get pods/exec in project "PROJECT"
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
[OkHttp https://172.17.0.1/ ...] ERROR d.z.e.z.c.k.KubernetesContainerClient - zalenium-40000-wvpjb Failed to execute command [bash, -c, notify 'Zalenium', 'TEST COMPLETED', --icon=/home/seluser/images/completed.png]
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
With own service account:
yml template of the sa:
- apiVersion: v1
kind: Role
metadata:
name: zalenium-role
labels:
app: zalenium
rules:
- apiGroups:
- ""
attributeRestrictions: null
resources:
- pods
verbs:
- create
- delete
- deletecollection
- get
- list
- watch
- apiGroups:
- ""
attributeRestrictions: null
resources:
- pods/exec
verbs:
- create
- delete
- list
- get
- apiGroups:
- ""
attributeRestrictions: null
resources:
- services
verbs:
- create
- delete
- get
- list
- apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: zalenium
name: zalenium-sa
- apiVersion: v1
kind: RoleBinding
metadata:
labels:
app: zalenium
name: zalenium-rolebinding
roleRef:
kind: Role
name: zalenium-role
namespace: ${PROJECTNAME}
subjects:
- kind: ServiceAccount
name: zalenium-sa
namespace: ${PROJECTNAME}
userNames:
- zalenium-sa
Result:
--WARN 10:22:28:182931026 We don't have sudo
Kubernetes service account found.
Copying files for Dashboard...
Starting Nginx reverse proxy...
Starting Selenium Hub...
.....10:22:29.626 [main] INFO o.o.grid.selenium.GridLauncherV3 - Selenium server version: 3.141.59, revision: unknown
.10:22:29.771 [main] INFO o.o.grid.selenium.GridLauncherV3 - Launching Selenium Grid hub on port 4445
..10:22:30.292 [main] INFO d.z.e.z.c.k.KubernetesContainerClient - Initialising Kubernetes support
..10:22:30.700 [main] WARN d.z.e.z.c.k.KubernetesContainerClient - Error initialising Kubernetes support.
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/PROJECT/pods/zalenium-1-j6s4q . Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "zalenium-1-j6s4q" is forbidden: User "system:serviceaccount:PROJECT:zalenium-sa" cannot get pods in the namespace "PROJECT": User "system:serviceaccount:PROJECT:zalenium-sa" cannot get pods in project "PROJECT".
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:476)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:413)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:313)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:296)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:794)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:210)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:177)
at de.zalando.ep.zalenium.container.kubernetes.KubernetesContainerClient.<init>(KubernetesContainerClient.java:91)
at de.zalando.ep.zalenium.container.ContainerFactory.createKubernetesContainerClient(ContainerFactory.java:43)
at de.zalando.ep.zalenium.container.ContainerFactory.getContainerClient(ContainerFactory.java:22)
at de.zalando.ep.zalenium.proxy.DockeredSeleniumStarter.<clinit>(DockeredSeleniumStarter.java:63)
at de.zalando.ep.zalenium.registry.ZaleniumRegistry.<init>(ZaleniumRegistry.java:97)
at de.zalando.ep.zalenium.registry.ZaleniumRegistry.<init>(ZaleniumRegistry.java:83)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.openqa.grid.web.Hub.<init>(Hub.java:94)
at org.openqa.grid.selenium.GridLauncherV3.lambda$buildLaunchers$5(GridLauncherV3.java:264)
at org.openqa.grid.selenium.GridLauncherV3.lambda$launch$0(GridLauncherV3.java:86)
at java.util.Optional.map(Optional.java:215)
at org.openqa.grid.selenium.GridLauncherV3.launch(GridLauncherV3.java:86)
at org.openqa.grid.selenium.GridLauncherV3.main(GridLauncherV3.java:70)
10:22:30.701 [main] INFO d.z.e.z.c.k.KubernetesContainerClient - About to clean up any left over docker-selenium pods created by Zalenium
Exception in thread "main" org.openqa.grid.common.exception.GridConfigurationException: Error creating class with de.zalando.ep.zalenium.registry.ZaleniumRegistry : null
at org.openqa.grid.web.Hub.<init>(Hub.java:99)
at org.openqa.grid.selenium.GridLauncherV3.lambda$buildLaunchers$5(GridLauncherV3.java:264)
at org.openqa.grid.selenium.GridLauncherV3.lambda$launch$0(GridLauncherV3.java:86)
at java.util.Optional.map(Optional.java:215)
at org.openqa.grid.selenium.GridLauncherV3.launch(GridLauncherV3.java:86)
at org.openqa.grid.selenium.GridLauncherV3.main(GridLauncherV3.java:70)
Caused by: java.lang.ExceptionInInitializerError
at de.zalando.ep.zalenium.registry.ZaleniumRegistry.<init>(ZaleniumRegistry.java:97)
at de.zalando.ep.zalenium.registry.ZaleniumRegistry.<init>(ZaleniumRegistry.java:83)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.openqa.grid.web.Hub.<init>(Hub.java:94)
... 5 more
Caused by: java.lang.NullPointerException
at java.util.TreeMap.putAll(TreeMap.java:313)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.withLabels(BaseOperation.java:426)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.withLabels(BaseOperation.java:63)
at de.zalando.ep.zalenium.container.kubernetes.KubernetesContainerClient.deleteSeleniumPods(KubernetesContainerClient.java:402)
at de.zalando.ep.zalenium.container.kubernetes.KubernetesContainerClient.initialiseContainerEnvironment(KubernetesContainerClient.java:348)
at de.zalando.ep.zalenium.container.ContainerFactory.createKubernetesContainerClient(ContainerFactory.java:46)
at de.zalando.ep.zalenium.container.ContainerFactory.getContainerClient(ContainerFactory.java:22)
at de.zalando.ep.zalenium.proxy.DockeredSeleniumStarter.<clinit>(DockeredSeleniumStarter.java:63)
... 13 more
[OkHttp https://172.17.0.1/ ...] ERROR i.f.k.c.d.i.ExecWebSocketListener - Exec Failure: HTTP:403. Message:pods "zalenium-40000-wvpjb" is forbidden: User "system:serviceaccount:PROJECT:deployer" cannot get pods/exec in the namespace "PROJECT": User "system:serviceaccount:PROJECT:deployer" cannot get pods/exec in project "PROJECT"
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
Usually this means that the service account does not have enough rights, perhaps start by checking that.

Jenkinsfile Kubernetes expose container ports

I am trying to export a port on my Kubernetes container in my Jenkins pipeline.
I have looked online, and I just can't work out how to get this to work.
I need to export port 4444 to the selenium-hub container.
def label = "selenium-hub-${UUID.randomUUID().toString()}"
podTemplate(label: label, yaml: """
apiVersion: v1
kind: Pod
spec:
containers:
- name: selenium-hub
image: selenium/hub:3.4.0
- name: selenium-chrome
image: selenium/node-chrome:3.4.0
env:
- name: HUB_PORT_4444_TCP_ADDR
value: localhost
- name: HUB_PORT_4444_TCP_PORT
value: 4444
- name: DISPLAY
value: :99.0
- name: SE_OPTS
value: -port 5556
- name: selenium-firefox
image: selenium/node-firefox:3.4.0
env:
- name: HUB_PORT_4444_TCP_ADDR
value: localhost
- name: HUB_PORT_4444_TCP_PORT
value: 4444
- name: DISPLAY
value: :98.0
- name: SE_OPTS
value: -port 5557
I currently get the ip from the hub, and pass that onto my tests to run via the grid
node(label) {
containerLog('selenium-hub')
sh('hostname -i')
POD_IP = sh(script: "hostname -i", returnStdout: true).toString().replaceAll('\\n', '');
node() {
sh("POD_IP='${POD_IP}' UI_URL=https://EEEEEE.net/ SERVICE_URL=https://WWWWWW.net/ ./node_modules/webdriverio/bin/wdio ./tests/UI/helpers/configs/wdio.GRID.conf.js --nolazy")
}
I would seem that at the point the tests run, it is unable to access the grid, so I believe it is due to port 4444 not being exposed, but I can't work out how to do it in Kubernetes.
I have tried some examples online, but don't seem to be getting any closer.
Think of a Kubernetes pod as an abstraction of a traditional server. If a container in the pod is exposing a service like mysql or in this case selenium on a particular port then all the other containers in the pod can access that service on the selected port by default:
https://kubernetes.io/docs/concepts/workloads/pods/pod/#resource-sharing-and-communication
Rather than using the POD_IP just try using localhost:4444 (or whatever port you are needing) to access the service.
I use the same pattern for accessing mysql in a sidecar container during unit tests in various builds.
Service allows your applications to receive traffic. So you need one of those to expose access to your selenium deployment:
apiVersion: v1
kind: Service
metadata:
name: selenium-hub
labels:
app: selenium-hub
spec:
ports:
- port: 4444
targetPort: 4444
name: port0
selector:
app: selenium-hub
type: NodePort
sessionAffinity: None
For more details https://github.com/kubernetes/examples/tree/master/staging/selenium

Apache Ignite activating cluster takes a long time

I am trying to set up a cluster of Apache Ignite with persistence enabled. I am trying to start the cluster on Azure Kubernetes with 10 nodes. The problem is that the cluster activation seems to get stuck, but I am able to activate a cluster with 3 nodes in less than 5 minutes.
Here is the configuration I am using to start the cluster:
apiVersion: v1
kind: Service
metadata:
name: ignite-main
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
labels:
main: ignite-main
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
ports:
- port: 10800 # JDBC port
targetPort: 10800
name: jdbc
- port: 11211 # Activating the baseline (port)
targetPort: 11211
name: control
- port: 8080 # REST port
targetPort: 8080
name: rest
selector:
main: ignite-main
---
#########################################
# Ignite service configuration
#########################################
# Service for discovery of ignite nodes
apiVersion: v1
kind: Service
metadata:
name: ignite
labels:
app: ignite
spec:
clusterIP: None
# externalTrafficPolicy: Cluster
ports:
# - port: 9042 # custom value.
# name: discovery
- port: 47500
name: discovery
- port: 47100
name: communication
- port: 11211
name: control
selector:
app: ignite
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ignite-cluster
labels:
app: ignite
main: ignite-main
spec:
selector:
matchLabels:
app: ignite
main: ignite-main
replicas: 5
template:
metadata:
labels:
app: ignite
main: ignite-main
spec:
volumes:
- name: ignite-storage
persistentVolumeClaim:
claimName: ignite-volume-claim # Must be equal to the PersistentVolumeClaim created before.
containers:
- name: ignite-node
image: ignite.azurecr.io/apacheignite/ignite:2.7.0-SNAPSHOT
env:
- name: OPTION_LIBS
value: ignite-kubernetes
- name: CONFIG_URI
value: https://file-location
- name: IGNITE_H2_DEBUG_CONSOLE
value: 'true'
- name: IGNITE_QUIET
value: 'false'
- name: java.net.preferIPv4Stack
value: 'true'
- name: JVM_OPTS
value: -server -Xms10g -Xmx10g -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
ports:
- containerPort: 47100 # communication SPI port number.
- containerPort: 47500 # discovery SPI port number.
- containerPort: 49112 # JMX port number.
- containerPort: 8080 # REST port number.
- containerPort: 10800 # SQL port number.
- containerPort: 11211 # Activating the baseline (port)
imagePullSecrets:
- name: docker-cred
I was trying to activate the cluster remotely by providing --host parameter, like:
./control.sh --host x.x.x.x --activate
Instead, I tried activating the cluster by logging into one of the kubernetes nodes and activating from there. The detailed steps are mentioned here

Expose every pod in Redis cluster in Kubernetes

I'm trying to setup Redis cluster in Kubernetes. The major requirement is that all of nodes from Redis cluster have to be available from outside of Kubernetes. So clients can connect every node directly. But I got no idea how to configure service that way.
Basic config of cluster right now. It's ok for services into k8s but no full access from outside.
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster
labels:
app: redis-cluster
data:
redis.conf: |+
cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file /data/nodes.conf
cluster-migration-barrier 1
appendonly no
protected-mode no
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "false"
name: redis-cluster
labels:
app: redis-cluster
spec:
type: NodePort
ports:
- port: 6379
targetPort: 6379
name: client
- port: 16379
targetPort: 16379
name: gossip
selector:
app: redis-cluster
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: redis-cluster
labels:
app: redis-cluster
spec:
serviceName: redis-cluster
replicas: 6
template:
metadata:
labels:
app: redis-cluster
spec:
hostNetwork: true
containers:
- name: redis-cluster
image: redis:4.0.10
ports:
- containerPort: 6379
name: client
- containerPort: 16379
name: gossip
command: ["redis-server"]
args: ["/conf/redis.conf"]
readinessProbe:
exec:
command:
- sh
- -c
- "redis-cli -h $(hostname) ping"
initialDelaySeconds: 15
timeoutSeconds: 5
livenessProbe:
exec:
command:
- sh
- -c
- "redis-cli -h $(hostname) ping"
initialDelaySeconds: 20
periodSeconds: 3
volumeMounts:
- name: conf
mountPath: /conf
readOnly: false
volumes:
- name: conf
configMap:
name: redis-cluster
items:
- key: redis.conf
path: redis.conf
Given:
spec:
hostNetwork: true
containers:
- name: redis-cluster
ports:
- containerPort: 6379
name: client
It appears that your StatefulSet is misconfigured, since if hostNetwork is true, you have to provide hostPort, and that value should match containerPort, according to the PodSpec docs:
hostPort integer - Number of port to expose on the host. If specified, this must be a valid port number, 0 < x < 65536. If HostNetwork is specified, this must match ContainerPort.
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#containerport-v1-core