I have a k8s deployment of Kibana in IBM Cloud. It is exposed through ClusterIP k8s service, a k8s Ingress and it is accessible for a single Cloud Directory user authenticated through IBM Cloud App ID.
Kubernetes correctly re-directs to App ID login screen. The issue is that Kibana deployment is not accessible after successful AppID authentication. I get 301 Moved Permanently in a loop.
The same k8s deployment as above is exposed through k8s NodePort and works fine.
The same setup as above works correctly for a simple hello-world app with authentication.
I followed this tutorial.
In App ID Authentication Settings, the redirect URL is:
https://our-domain/app/kibana/appid_callback
Here are portions of the k8s definitions, which are relevant:
---
kind: Service
apiVersion: v1
metadata:
name: kibana-sec
namespace: default
labels:
app: kibana-sec
spec:
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 8080
targetPort: 5601
selector:
app: kibana-sec
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.bluemix.net/redirect-to-https: "True"
ingress.bluemix.net/appid-auth: "bindSecret=<our-bindSecret> namespace=default requestType=web serviceName=kibana-sec"
...
spec:
rules:
- host: <our-domain>
http:
paths:
...
- backend:
serviceName: kibana-sec
servicePort: 8080
path: /app/kibana/
tls:
- hosts:
- <our-domain>
secretName: <our-secretName>
status:
loadBalancer:
ingress:
- ip: <IPs>
- ip: <IPs>
There is no "ingress.bluemix.net/rewrite-path" annotation for our service.
Related
I have an AKS cluster with Istio install and I'm trying to deploy a containerised web api with TLS.
The api runs and is accessible but is showing as Not secure.
I have followed the directions on istios website to set this so not sure what I've missed.
I have created the secret with the command
kubectl create secret tls mycredential -n istio-system --key mycert.key --cert mycert.crt
and setup a gateway as follows
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: my-gateway
namespace: mynamespace
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: mycredential # must be the same as secret
hosts:
- 'dev.api2.mydomain.com'
The following virtual service
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: myapi
namespace: mynamespace
spec:
hosts:
- "dev.api2.mydomain.com"
gateways:
- my-gateway
http:
- match:
- uri:
prefix: "/myendpoint"
rewrite:
uri: " "
route:
- destination:
port:
number: 8080
host: myapi
and service
apiVersion: v1
kind: Service
metadata:
name: myapi
namespace: mynamespace
labels:
app: myapi
service: myapi
spec:
ports:
- name: http
port: 8080
targetPort: 80
selector:
app: myapi
The container exposes port 80
Can someone please point me in the right direction because I'm not sure what I've done wrong
I managed to resolve the issue by setting up cert manager and pointing it at letsencrypt to generate the certificate, rather than using the pre-purchased one I was trying to add manually.
Although it took some searching to find how to correctly configure this, it is now working and actually saves having to purchase certificates, so win win :)
I'm trying configure for TLS (LetsEncrypt) on a multi-tenant GKE+Istio setup.
I mainly followed this guide -> Full Isolation in Multi-Tenant SAAS with Kubernetes & Istio for setting up the multi-tenancy in GKE+Istio, which I was able to successfully pull-off. I'm able to deploy simple apps on their separate namespaces which are accessible through their respective subdomains.
I then tried to move forward and setup the TLS with LetsEncrypt. For this I mainly followed a different guide which is can be found here-> istio-gke . But unfortunately, following this guide didn't produce the result I wanted. When I was done with it, LetsEncrypt aren't even issuing certificates to my deployment or domain.
Thus I tried to follow a different guide which is as follows -> istio-gateway-tls-setup. Here I managed to get LetsEncrypt issue a certificate for my domain, but when I tried to test it out with openssl or other online ssl checkers, it says that I still aren't communicating securely.
Below are the results when I try describe the configurations of my certificates, issuer & gateway:
Certificate: kubectl -n istio-system describe certificate istio-gateway
Issuer: kubectl -n istio-system describe issuer letsencrypt-prod
Gateway: kubectl -n istio-system describe gateway istio-gateway
While here's the dry-run results for my helm install <tenant>
install.go:173: [debug] Original chart version: ""
install.go:190: [debug] CHART PATH: /home/cjcabero/projects/aqt-ott-msging-dev/gke-setup/helmchart
NAME: tenanta
LAST DEPLOYED: Wed Feb 17 21:15:08 2021
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
{}
COMPUTED VALUES:
frontend:
image:
pullPolicy: IfNotPresent
repository: paulbouwer/hello-kubernetes
tag: "1.8"
ports:
containerPort: 8080
service:
name: http
port: 80
type: ClusterIP
HOOKS:
MANIFEST:
---
# Source: helmchart/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: tenanta
labels:
istio-injection: enabled
---
# Source: helmchart/templates/frontend-service.yaml
apiVersion: v1
kind: Service
metadata:
name: frontend
namespace: tenanta
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 8080
selector:
app: frontend
---
# Source: helmchart/templates/frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
namespace: tenanta
spec:
replicas: 3
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: hello-kubernetes
image: paulbouwer/hello-kubernetes:1.8
ports:
- containerPort: 8080
env:
- name: MESSAGE
value: Hello tenanta
---
# Source: helmchart/templates/virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: tenanta-frontend-ingress
namespace: istio-system
spec:
hosts:
- tenanta.cjcabero.dev
gateways:
- istio-gateway
http:
- route:
- destination:
host: frontend.tenanta.svc.cluster.local
port:
number: 80
I don't understand how come even though LetsEncrypt seem to be able issue the certificate for my domain, it still aren't communicating securely.
Google Domains even managed to find that a certificate was issued in the domain in it's Transparency Report.
Anyway, I'm not sure if this could help, but I also tried the check the domain with an online ssl checker and here are the results -> https://check-your-website.server-daten.de/?q=cjcabero.dev.
By the way I did use Istio on GKE which results with Istio v1.4.10 & Kubernetes v1.18.15-gke.1100.
I kind of new on Kubernetes stuff and I'm trying to improve one current system we have here.
The Application is developed using Spring Boot and until now it was using HTTP (Port 8080) without any encryption. The system requirement is to enable e2e-encryption for all Data In-Transit. So here is the problem.
Currently, we have GCE Ingress with TLS enabled using Let's Encrypt to provide the Certificates on Cluster entrance. This is working fine. Our Ingress has some Path Rules to redirect the traffic to the correct microservice and those microservices are not using TLS on the communication.
I managed to create a Self-Signed certificate and embedded it inside the WAR and this is working on the Local machine just fine (using certificate validation disabled). When I deploy this on GKE, the GCP Health Check and Kubernetes Probes are not working at all (I can't see any communication attempt on the Application logs).
When I try to configure the Backend and Health Check on GCP changing both to HTTPS, they don't show any error, but after some time they quietly switch back to HTTP.
Here are my YAML files:
admin-service.yaml
---
apiVersion: v1
kind: Service
metadata:
name: admin-service
namespace: default
spec:
type: NodePort
selector:
app: admin
ports:
- port: 443
targetPort: 8443
name: https
protocol: TCP
admin-deployment.yaml
---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "admin"
namespace: "default"
labels:
app: "admin"
spec:
replicas: 1
selector:
matchLabels:
app: "admin"
template:
metadata:
labels:
app: "admin"
spec:
containers:
- name: "backend-admin"
image: "gcr.io/my-project/backend-admin:X.Y.Z-SNAPSHOT"
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8443
scheme: HTTPS
initialDelaySeconds: 8
periodSeconds: 30
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8443
scheme: HTTPS
initialDelaySeconds: 8
periodSeconds: 30
env:
- name: "FIREBASE_PROJECT_ID"
valueFrom:
configMapKeyRef:
key: "FIREBASE_PROJECT_ID"
name: "service-config"
---
apiVersion: "autoscaling/v2beta1"
kind: "HorizontalPodAutoscaler"
metadata:
name: "admin-etu-vk1a"
namespace: "default"
labels:
app: "admin"
spec:
scaleTargetRef:
kind: "Deployment"
name: "admin"
apiVersion: "apps/v1"
minReplicas: 1
maxReplicas: 3
metrics:
- type: "Resource"
resource:
name: "cpu"
targetAverageUtilization: 80
ingress.yaml
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: my-ingress-addr
kubernetes.io/ingress.class: "gce"
kubernetes.io/tls-acme: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
acme.cert-manager.io/http01-edit-in-place: "true"
kubernetes.io/ingress.allow-http: "false"
spec:
tls:
- hosts:
- my-domain.com
secretName: mydomain-com-tls
rules:
- host: my-domain.com
http:
paths:
- path: /admin/v1/*
backend:
serviceName: admin-service
servicePort: 443
status:
loadBalancer:
ingress:
- ip: XXX.YYY.WWW.ZZZ
Reading this document from GCP I understood that Loadbalancer it's compatible with Self-signed certificates.
I would appreciate any insight or new directions you guys can provide.
Thanks in advance.
EDIT 1: I've added here the ingress YAML file which may help to a better understanding of the issue.
EDIT 2: I've updated the deployment YAML with the solution I found for liveness and readiness probes (scheme).
EDIT 3: I've found the solution for GCP Health Checks using annotation on Services declaration. I will put all the details on the response to my own question.
Here is what I found on how to fix the issue.
After reading a lot of documentation related to Kubernetes and GCP I found a document on GCP explaining to use annotations on Service declaration. Take a look at lines 7-8.
---
apiVersion: v1
kind: Service
metadata:
name: admin-service
namespace: default
annotations:
cloud.google.com/app-protocols: '{"https":"HTTPS"}'
spec:
type: NodePort
selector:
app: iteam-admin
ports:
- port: 443
targetPort: 8443
name: https
protocol: TCP
This will hint GCP to create the backend-service and health-check using HTTPS and everything will work as expected.
Reference: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#https_tls_between_load_balancer_and_your_application
I had my original infrastructure built around this tutorial. https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes.
Now I am trying to migrate to managing my own cert and terminating SSL at the load balancer.
With my YAML updates, the load balancer in DigitalOcean shows that all nodes are unhealthy and I the URL response with "503 Service Unavailable
No server is available to handle this request." However, the endpoint shows a secure HTTPS connection. What am I doing wrong?
My new non-functional YAML definitions below.
LoadBalancer
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
annotations:
service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
service.beta.kubernetes.io/do-loadbalancer-certificate-id: "**************"
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: http
Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
tls:
- hosts:
- ******.com
- api.*******.com
rules:
- host: **********.com
http:
paths:
- backend:
serviceName: frontend-angular
servicePort: 80
- host: api.********.com
http:
paths:
- backend:
serviceName: backend-server
servicePort: 80
I reached out to DigitalOcean support (which is incredible). My issue was I didn't create the ingress-nginx pod. These are the two steps, as listed in the tutorial, that I missed.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml.
My actual YAML definitions were correct.
The best way is just using Digital Ocean Marketplace (https://marketplace.digitalocean.com/apps/nginx-ingress-controller).
Installing it manually will cause a lot of issues due do outdated yml files.
I created a basic react/express app with IAP authentication and deployed to google app engine and everything works as expected. Now i'm moving from app engine deployment to kubernetes, all things work except the user authentication with IAP on kubernetes. How do I enable IAP user authentication with kubernetes?
Do I have to create a kubernetes secret to get user authentication to work? https://cloud.google.com/iap/docs/enabling-kubernetes-howto
Authentication code in my server.js https://cloud.google.com/nodejs/getting-started/authenticate-users#cloud-identity-aware-proxy
In order for Cloud IAP to be working with Kubernetes, you will need a group of one or more GKE instances, served by an HTTPS load balancer. The load balancer should be created automatically when you create an Ingress object in a GKE cluster.
Also required for enabling Cloud IAP in GKE: a domain name registered to the address of your load balancer and an App code to verify that all requests have an identity.
Once these requirements have been achieved, you can move forward with enabling Cloud IAP on Kubernetes Engine. This includes the steps to set up Cloud IAP access and creating OAuth credentials.
You will need to create a Kubernetes Secret to configure BackendConfig for Cloud IAP. The BackendConfig uses a Kubernetes Secret to wrap the OAuth client that you create when enabling the Cloud IAP.
You need to add an Ingress and enable IAP on the backend service
Create a BackendConfig object:
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: app-bc
spec:
iap:
enabled: true
oauthclientCredentials:
secretName: oauth-client-secret
attached it to the service:
apiVersion: v1
kind: Service
metadata:
name: app-service
annotations:
cloud.google.com/backend-config: '{"default": "app-bc"}'
cloud.google.com/neg: '{"ingress": true}'
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
selector:
app: nginx
and then create the ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
spec:
tls:
- secretName: ingress-tls-secret
rules:
- host: ""
http:
paths:
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: app-service
port:
number: 80
You can find the full tutorial here hodo.dev/posts/post-26-gcp-iap/