I am running Traefik and first I configured to use cloudflare as my certresolver for domain1.com. But I have domain2.net hosted on Route 53.
This is what I have so far:
--entrypoints.websecure.http.tls.certresolver=cloudflare
--entrypoints.websecure.http.tls.domains[0].main=local.domain1.com
--entrypoints.websecure.http.tls.domains[0].sans=*.local.domain1.com
--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare
--certificatesresolvers.cloudflare.acme.email=myemail#gmail.com
--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1
--certificatesresolvers.cloudflare.acme.storage=/certs/acme.json
--entrypoints.websecure.web.tls.domains[1].main=local.domain2.net
--entrypoints.websecure.web.tls.domains[1].sans=*.local.domain2.net
--certificatesresolvers.route53.acme.dnschallenge.provider=route53
--certificatesresolvers.route53.acme.email=myemail#gmail.com
--certificatesresolvers.route53.acme.storage=/certs/acme.json
But when I setup this way, only route53 is configured as a certificate resolver. That's because it's being called last.
Is there a way to make this work with multiple certificate resolvers?
Thanks!
I figure this out and forgot to update.
So just create additional args on traefik deployment:
- --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare
- --certificatesresolvers.cloudflare.acme.email=myemail#gmail.com
- --certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1
- --certificatesresolvers.cloudflare.acme.storage=/certs/cloudflare.json
- --certificatesresolvers.route53.acme.dnschallenge.provider=route53
- --certificatesresolvers.route53.acme.email=myemail#gmail.com
- --certificatesresolvers.route53.acme.storage=/certs/route53.json
And then the entrypoints you add to the annotation of the app deployment with its own domain.
Related
I'm out of ideas on this and appreciate any suggestions. I have a handful of dockerized springboot microservices which include a config server. Here are the characteristics:
Springboot version 2.3.0-RELEASE
Standard Springboot config server with basic auth turned on.
3 Springboot microservices that are also config clients to config server.
-- I use a simple Dockerfile model for microservices and springboot maven plugin with default docker layers capabilities.
SSL is enabled for all including the config server.
-- For dev and testing, I use a self signed cert.
All microservices use a JKS to sign JWTs
Docker image for java is openjdk8 alpine.
Docker compose is used to orchestrate container launch and settings.
The docker container for config server runs perfectly fine. I can even query for config via a browser following the HTTPS URL: https://app-dev.localhost.com:8443/config-server/shopping-svc/dev.
The Problem
I cannot manage to successfully start container 'shopping-svc'. It fails with this error.
2023-01-25T23:44:12.375221300Z
2023-01-25 23:44:12.575 INFO 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : https://app-dev.localhost.com:8443/config-server
2023-01-25 23:44:12.829 INFO 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Connect Timeout Exception on Url - https://app-dev.localhost.com:8443/config-server. Will be trying the next url if available
2023-01-25 23:44:12.836 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
2023-01-25T23:44:12.837487100Z
java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:155) ~[spring-cloud-config-client-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:52) ~[spring-cloud-context-2.2.9.RELEASE.jar:2.2.9.RELEASE]
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locateCollection(ConfigServicePropertySourceLocator.java:170) ~[spring-cloud-config-client-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:98) ~[spring-cloud-context-2.2.9.RELEASE.jar:2.2.9.RELEASE]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at com.shopping.app.ShoppingApplication.main(ShoppingApplication.java:35) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_212]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_212]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [application/:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109) [application/:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [application/:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [application/:na]
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://app-dev.localhost.com:8443/config-server/shopping-app/dev": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:748) ~[spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674) ~[spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:583) ~[spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
Investigations
At first, I thought perhaps the port 8443 is blocked somehow by my OS firewall but that's not it. Clearing the port makes no difference
Then I thought, perhaps it's a cert issue so I tried supplying the cert differently with the JAVA_TOOL_OPTIONS argument with the override populated: -Djavax.net.ssl.trustStore=/path/to/cert, etc... No dice.
I read several posts and articles suggesting services inside docker containers should refer to others via service name. While this poses a bit of confusion for me since my certs are generated against a hostname, I tried swapping the URL of config server in shopping-app YML to something like: https://config-server:8443/config-server/ or the same without https to see if at least successful connection would be made.
Last thing I tried was to change the compose network driver to 'host' instead of 'bridge' so the containers would use the host machine network config. The rationale was that at least, it's obvious it's all on same network.
I am not sure what or where to look anymore
References
=====
Docker compose file:
version: "3"
networks:
default:
driver: bridge
frontend:
driver: bridge
backend:
driver: bridge
services:
config-server:
image: config-server
env_file: .env
hostname: app-dev.localhost.com # Not sure this is necessary I add this because the self signed cert was generated with this domain name
volumes: #I'm developping on windows, hence the backslash "\"
- shoppingapp:/var/opt
- shoppingapp\certs\server.jks:/etc/certs/server.jks
- shoppingapp\certs\ssl/app-dev.localhost.com.p12:/etc/certs/ssl/app-dev.localhost.com.p12
ports:
- "8443:8443"
networks:
- backend
shopping-svc:
image: shopping-svc
env_file: .env
hostname: app-dev.localhost.com # Not sure this is necessary I add this because the self signed cert was generated with this domain name
volumes:
- shoppingapp:/var/opt
- shoppingapp\certs\server.jks:/etc/certs/server.jks
- shoppingapp\certs\ssl\app-dev.localhost.com.p12:/etc/certs/ssl/app-dev.localhost.com.p12
ports:
- "8444:8444"
depends_on:
config-server:
condition: service_started
networks:
- backend
We are converting existing k8s services to use istio & knative. The services receive requests from external users as well as from within the cluster. We are trying to setup Istio AuthorizationPolicy to achieve the below requirements:
Certain paths (like docs/healthchecks) should not require any special header or anything and must be accessible from anywhere
Health & metric collection paths required to be accessed by knative must be accisible only by knative controllers
Any request coming from outside the cluster (through knative-serving/knative-ingress-gateway basically) must contain a key header matching a pre-shared key
Any request coming from any service within the cluster can access all the paths
Below is a sample of what I am trying. I am able to get the first 3 requirements working but not the last one...
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: my-svc
namespace: my-ns
spec:
selector:
matchLabels:
serving.knative.dev/service: my-svc
action: "ALLOW"
rules:
- to:
- operation:
methods:
- "GET"
paths:
- "/docs"
- "/openapi.json"
- "/redoc"
- "/rest/v1/healthz"
- to:
- operation:
methods:
- "GET"
paths:
- "/healthz*"
- "/metrics*"
when:
- key: "request.headers[User-Agent]"
values:
- "Knative-Activator-Probe"
- "Go-http-client/1.1"
- to:
- operation:
paths:
- "/rest/v1/myapp*"
when:
- key: "request.headers[my-key]"
values:
- "asjhfhjgdhjsfgjhdgsfjh"
- from:
- source:
namespaces:
- "*"
We have made no changes to the mTLS configuration provided by default by istio-knative setup, so assume that the mtls mode is currently PERMISSIVE.
Details of tech stack involved
AWS EKS - Version 1.21
Knative Serving - Version 1.1 (with Istio
1.11.5)
I'm not an Istio expert, but you might be able to express the last policy based on either the ingress gateway (have one which is listening only on a ClusterIP address), or based on the SourceIP being within the cluster. For the latter, I'd want to test that Istio is using the actual SourceIP and not substituting in the Forwarded header's IP address (a different reasonable configuration).
I have a server running Docker containers with Traefik. Let's say the machine's hostname is machine1.example.com, and each service runs as a subdomain, e.g. srv1.machine1.example.com, srv2.machine1.example.com, srv3.machine1.example.com....
I want to have LetsEncrypt generate a Wildcard certificate for *.machine1.example.com and use it for all of the services instead of generating a separate certificate for each service.
The annoyance is that I have to put the configuration lines into every single service's labels:
labels:
- traefik.http.routers.srv1.rule=Host(`srv1.machine1.example.com`)
- traefik.http.routers.srv1.tls=true
- traefik.http.routers.srv1.tls.certresolver=myresolver
- traefik.http.routers.srv1.tls.domains[0].main=machine1.example.com
- traefik.http.routers.srv1.tls.domains[0].sans=*.machine1.example.com
labels:
- traefik.http.routers.srv2.rule=Host(`srv2.machine1.example.com`)
- traefik.http.routers.srv2.tls=true
- traefik.http.routers.srv2.tls.certresolver=myresolver
- traefik.http.routers.srv2.tls.domains[0].main=machine1.example.com
- traefik.http.routers.srv2.tls.domains[0].sans=*.machine1.example.com
# etc.
This gets to be a lot of seemingly-needless boilerplate.
I tried work around it (in a way that is still ugly and annoying, but less so) by using the templating feature in the file provider like this:
[http]
[http.routers]
{{ range $i, $e := list "srv1" "srv2 }}
[http.routers."{{ $e }}".tls]
certResolver = "letsencrypt"
[[http.routers."{{ $e }}".tls.domains]]
main = "machine1.example.com"
sans = ["*.machine1.example.com"]
{{ end }}
That did not work because the routers created here are srv1#file, srv2#file instead of srv1#docker, srv2#docker which are created by the docker-compose configuration.
Is there any way to specify this configuration only once and have it apply to multiple services?
I am very new to using helm charts for deploying containers, and I have also never worked with nginx controllers or ingress controllers.
However, I am being asked to look into improving our internal nginx ingress controllers to allow for SSL-passthrough.
Right now we have external (public facing) and internal controllers. Where the public ones allow SSL-passthrough, and the internal ones have SSL-termination.
I have also been told that nginx is a reverse proxy, and that it works based on headers in the URL.
I am hoping someone can help me out on this helm chart that I have for the internal ingress controllers.
Currently I am under the impression that having SSL termination as well as SSL-passthrough on the same ingress controllers would not be possible.
Answered this one myself: https://serversforhackers.com/c/tcp-load-balancing-with-nginx-ssl-pass-thru
Our current (internal) ingress code:
---
rbac:
create: true
controller:
ingressClass: nginx-internal
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu:110:certificate/62-b3
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: !!str 443
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: !!str 3600
targetPorts:
https: 80
replicaCount: 3
defaultBackend:
replicaCount: 3
Can I simply add the following? :
controller:
extraArgs:
enable-ssl-passthrough: ""
Note: The above piece of code is what we use on our external ingress controller.
additionally, I found this:
Ingress and SSL Passthrough
Can I just go and mix the annotations? Or do annotations only care about the 'top domain level' where the annotation comes from?
eg:
service.beta.kubernetes.io
nginx.ingress.kubernetes.io
Both come from the domain kubernetes.io, or does the sub-domain make a difference?
I mean: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
That page doesn't show any of the service.beta annotations on it ..
What's the difference between the extraArg ssl-passthrough configuration and the ssl-passthrough configuration in the annotations?
I'm looking mostly for an answer on how to get the SSL-passthrough working without breaking the SSL-termination on the internal ingress controllers.
However, any extra information to gain more insight and knowledge as far as my other questions go would also be very appreciated :)
So I found the answer to my own question(s):
The annotations appear to be 'configuration items'. I'm using quotes because i can't find a better term.
The extraArgs parameter is where you can pass any parameter to the controller as if it were a commandline parameter.
And I think it is also safe to say that the annotations can be either any of the same top-level domain. I have not found any that weren't from another domain then kubernetes.io
To get my ingress controller to work side-by-side with the SSL-termination controller the helm chart looks as following:
---
rbac:
create: true
controller:
ingressClass: nginx-internal-ssl-passthrough
service:
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "tag3=value3, tag3=value3, tag3=value3, tag3=value3"
targetPorts:
https: 443
replicaCount: 2
extraArgs:
enable-ssl-passthrough: ""
defaultBackend:
replicaCount: 2
Toke me about 2 days of researching/searching the web & 6 deployments to get the whole setup working with AWS nlb, ssl-passthrough enabled, cross-zone loadbalancing, etc. But after having found the following pages it went pretty fast:
https://kubernetes.github.io/ingress-nginx/deploy/
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/
https://kubernetes.io/docs/concepts/services-networking/service/
This last page helped me a lot. If someone else gets to deploy SSL-termination and SSL-passthrough for either public or private connections, I hope this helps too.
From here you can find out how to redirect the HTTPS traffic to the pod without SSL-termination
https://stackoverflow.com/a/66767691/1938507
I have few containers, brought up with docker-compose and I want to perform authentication on of the containers.
Below the piece that I assume should do that, but it doesn't go to the authentication-backend-nginx-private, directly lands on the mds-backend-nginx-private. I'm out of idea, what could be wrong about the config...
it works if authforward configured globally: in toml file under entrypoint section, but I want it to be per particular container..
mds-backend-nginx-private:
<<: *nginx-common
ports:
- 8186:80
networks:
- cloud_private
- mds-backend
restart: on-failure
environment:
- NGINX_SERVER_NAME=mds-backend-nginx-private
- WEBSITE_PROXY_NAME=mds-backend-web-private
- WEBSITE_PROXY_PORT=8000
labels:
- "traefik.http.middlewares.authf.ForwardAuth.Address=http://authentication-backend-nginx-private/api/v1/gateway/account?with_credentials=true"
- "traefik.docker.network=cloud_private"
- "traefik.http.routers.mds-backend.middlewares=authf"
- "traefik.frontend.rule=PathPrefix: /api/v1/mds/"```
Maybe, you are trying to use "middleware feature" with old traefik version.
Works in the toml file because you are using the "forward feature" present in old versions.
Check traefik tag image is equal or greater than 2.0
https://hub.docker.com/_/traefik