Mercure keeps binding to port 80 - mercure

I'm using the Mercure hub 0.13, everything works fine on my development machine, but on my test server the hub keeps on trying to bind on port 80, resulting in a error, as nginx is already running on port 80.
run: loading initial config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: address already in use
I'm starting the hub with the following command:
MERCURE_PUBLISHER_JWT_KEY=$(cat publisher.key.pub) \
MERCURE_PUBLISHER_JWT_ALG=RS256 \
MERCURE_SUBSCRIBER_JWT_KEY=$(cat publisher.key.pub) \
MERCURE_SUBSCRIBER_JWT_ALG=RS256 \
./mercure run -config Caddyfile.dev
Caddyfile.dev is as follows:
# Learn how to configure the Mercure.rocks Hub on https://mercure.rocks/docs/hub/config
{
{$GLOBAL_OPTIONS}
}
{$SERVER_NAME:localhost:3000}
log
route {
redir / /.well-known/mercure/ui/
encode zstd gzip
mercure {
# Transport to use (default to Bolt)
transport_url {$MERCURE_TRANSPORT_URL:bolt://mercure.db}
# Publisher JWT key
publisher_jwt {env.MERCURE_PUBLISHER_JWT_KEY} {env.MERCURE_PUBLISHER_JWT_ALG}
# Subscriber JWT key
subscriber_jwt {env.MERCURE_SUBSCRIBER_JWT_KEY} {env.MERCURE_SUBSCRIBER_JWT_ALG}
# Permissive configuration for the development environment
cors_origins *
publish_origins *
demo
anonymous
subscriptions
# Extra directives
{$MERCURE_EXTRA_DIRECTIVES}
}
respond /healthz 200
respond "Not Found" 404
}
When I provider the SERVER_NAME as an environment variable, without a domain, SERVER_NAME=:3000, the hub actually starts on port 3000, but runs in http mode, which only allows for anonymous subscriptions and is not what I need.
Server:
Operating System: CentOS Stream 8
Kernel: Linux 4.18.0-383.el8.x86_64
Architecture: x86-64
Full output when trying to start the Mercure hub:
2022/05/10 04:50:29.605 INFO using provided configuration {"config_file": "Caddyfile.dev", "config_adapter": ""}
2022/05/10 04:50:29.606 WARN input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile.dev", "line": 3}
2022/05/10 04:50:29.609 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/05/10 04:50:29.610 INFO http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2022/05/10 04:50:29.610 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc0003d6150"}
2022/05/10 04:50:29.627 INFO tls cleaning storage unit {"description": "FileStorage:/root/.local/share/caddy"}
2022/05/10 04:50:29.628 INFO tls finished cleaning storage units
2022/05/10 04:50:29.642 INFO pki.ca.local root certificate is already trusted by system {"path": "storage:pki/authorities/local/root.crt"}
2022/05/10 04:50:29.643 INFO tls.cache.maintenance stopped background certificate maintenance {"cache": "0xc0003d6150"}
run: loading initial config: loading new config: http app module: start: tcp: listening on :80: listen tcp :80: bind: address already in use

I'm a bit late, but I hope that will help someone.
As mentionned here, you can specify the http_port manually in your caddy configuration file.

Related

Why is dockerized config client unable to connect to dockerized config server

I'm out of ideas on this and appreciate any suggestions. I have a handful of dockerized springboot microservices which include a config server. Here are the characteristics:
Springboot version 2.3.0-RELEASE
Standard Springboot config server with basic auth turned on.
3 Springboot microservices that are also config clients to config server.
-- I use a simple Dockerfile model for microservices and springboot maven plugin with default docker layers capabilities.
SSL is enabled for all including the config server.
-- For dev and testing, I use a self signed cert.
All microservices use a JKS to sign JWTs
Docker image for java is openjdk8 alpine.
Docker compose is used to orchestrate container launch and settings.
The docker container for config server runs perfectly fine. I can even query for config via a browser following the HTTPS URL: https://app-dev.localhost.com:8443/config-server/shopping-svc/dev.
The Problem
I cannot manage to successfully start container 'shopping-svc'. It fails with this error.
2023-01-25T23:44:12.375221300Z
2023-01-25 23:44:12.575 INFO 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : https://app-dev.localhost.com:8443/config-server
2023-01-25 23:44:12.829 INFO 1 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Connect Timeout Exception on Url - https://app-dev.localhost.com:8443/config-server. Will be trying the next url if available
2023-01-25 23:44:12.836 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed
2023-01-25T23:44:12.837487100Z
java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:155) ~[spring-cloud-config-client-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:52) ~[spring-cloud-context-2.2.9.RELEASE.jar:2.2.9.RELEASE]
at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locateCollection(ConfigServicePropertySourceLocator.java:170) ~[spring-cloud-config-client-2.2.8.RELEASE.jar:2.2.8.RELEASE]
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:98) ~[spring-cloud-context-2.2.9.RELEASE.jar:2.2.9.RELEASE]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
at com.shopping.app.ShoppingApplication.main(ShoppingApplication.java:35) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_212]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_212]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [application/:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109) [application/:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [application/:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [application/:na]
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://app-dev.localhost.com:8443/config-server/shopping-app/dev": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:748) ~[spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674) ~[spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:583) ~[spring-web-5.2.6.RELEASE.jar:5.2.6.RELEASE]
Investigations
At first, I thought perhaps the port 8443 is blocked somehow by my OS firewall but that's not it. Clearing the port makes no difference
Then I thought, perhaps it's a cert issue so I tried supplying the cert differently with the JAVA_TOOL_OPTIONS argument with the override populated: -Djavax.net.ssl.trustStore=/path/to/cert, etc... No dice.
I read several posts and articles suggesting services inside docker containers should refer to others via service name. While this poses a bit of confusion for me since my certs are generated against a hostname, I tried swapping the URL of config server in shopping-app YML to something like: https://config-server:8443/config-server/ or the same without https to see if at least successful connection would be made.
Last thing I tried was to change the compose network driver to 'host' instead of 'bridge' so the containers would use the host machine network config. The rationale was that at least, it's obvious it's all on same network.
I am not sure what or where to look anymore
References
=====
Docker compose file:
version: "3"
networks:
default:
driver: bridge
frontend:
driver: bridge
backend:
driver: bridge
services:
config-server:
image: config-server
env_file: .env
hostname: app-dev.localhost.com # Not sure this is necessary I add this because the self signed cert was generated with this domain name
volumes: #I'm developping on windows, hence the backslash "\"
- shoppingapp:/var/opt
- shoppingapp\certs\server.jks:/etc/certs/server.jks
- shoppingapp\certs\ssl/app-dev.localhost.com.p12:/etc/certs/ssl/app-dev.localhost.com.p12
ports:
- "8443:8443"
networks:
- backend
shopping-svc:
image: shopping-svc
env_file: .env
hostname: app-dev.localhost.com # Not sure this is necessary I add this because the self signed cert was generated with this domain name
volumes:
- shoppingapp:/var/opt
- shoppingapp\certs\server.jks:/etc/certs/server.jks
- shoppingapp\certs\ssl\app-dev.localhost.com.p12:/etc/certs/ssl/app-dev.localhost.com.p12
ports:
- "8444:8444"
depends_on:
config-server:
condition: service_started
networks:
- backend

I want to configure the Prometheus using kube-prom-stack to federate metrics from https server

I have kube-prom-stack runnning on a node and i have configure it using federate endpoint to scrape metrics from a remote server as below in my values.yaml file:
additionalScrapeConfigs:
- job_name: 'sp-federation'
scrape_interval: 20s
scrape_timeout: 20s
scheme: http
metrics_path: /federate
honor_labels: true
metric_relabel_configs:
- source_labels: [id]
regex: '^static-agent$'
action: drop
params:
match[]:
- '{__name__=~"kube_.*"}'
static_configs:
- targets: ['X.X.X.X:9090']
But the remote server only accepts connection on https.
I want to configure it in such a way that it pull the metrics from https by skipping the certificate validation part as i don't have the remote server certificate now.
Please help, as i am a beginner in the networking and prometheus.
I used below configuration to deploy the kube-prom-stack, but not sure what to do next to skip the server certificate validation.
serviceMonitor:
## Scrape interval. If not set, the Prometheus default scrape interval is used.
##
interval: ""
selfMonitor: true
## proxyUrl: URL of a proxy that should be used for scraping.
##
proxyUrl: ""
## scheme: HTTP scheme to use for scraping. Can be used with `tlsConfig` for example if using istio mTLS.
scheme: ""
## enableHttp2: Whether to enable HTTP2.
## See https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#endpoint
enableHttp2: true
## tlsConfig: TLS configuration to use when scraping the endpoint. For example if using istio mTLS.
## Of type: https://github.com/coreos/prometheus-operator/blob/main/Documentation/api.md#tlsconfig
tlsConfig: {}
bearerTokenFile:

Socket Io and Kubernetes

I am trying to migrate a socket io service from GCP (App Engine) to a kubernetes cluster.
Everything works fine on the GCP side (we have one instance of the server without replicas).
The migration to k8s is going very well, except that when connecting the client socket to the server, it does not receive some information:
In transport 'polling': Of course, as there are two pods, this doesn't work properly anymore and the client socket keeps deconnecting / reconnecting in loop.
In 'websocket' transport: The connection is correctly established, the client can receive data from the server in 'broadcast to all client' mode => socket.emit('getDeviceList', os.hostname()) but, as soon as the server tries to send data only to the concerned client io.of(namespace).to(socket.id).emit('getDeviceList', JSON.stringify(obj)), this one doesn't receive anything...
Moreover, I modified my service to have only one pod for a test, the polling mode works correctly, but, I find myself in the same case as the websocket mode => I can't send an information to a precise client...
Of course, the same code on the App Engine side works correctly and the client receives everything correctly.
I'm working with:
"socket.io": "^3.1.0",
"socket.io-redis": "^5.2.0",
"vue": "^2.5.18",
"vue-socket.io": "3.0.7",
My server side configuration:
var io = require('socket.io')(server, {
pingTimeout: 5000,
pingInterval : 2000,
cors: {
origin: true,
methods: ["GET", "POST"],
transports: ['websocket', 'polling'],
credentials: true
},
allowEIO3: true
});
io.adapter(redis({ host: redis_host, port: redis_port }))
My front side configuration:
Vue.use(new VueSocketIO({
debug: true,
connection: 'path_to_the_socket_io/namespace,
options: {
query: `id=..._timestamp`,
transports: ['polling']
}
}));
My ingress side annotation:
kubernetes.io/ingress.class: nginx kubernetes.io/ingress.global-static-ip-name: ip-loadbalancer
meta.helm.sh/release-name: xxx
meta.helm.sh/release-namespace: xxx -release nginx.ingress.kubernetes.io/affinity: cookie nginx.ingress.kubernetes.io/affinity-mode: persistent nginx.ingress.kubernetes.io/force-ssl-redirect: true nginx.ingress.kubernetes.io/proxy-connect-timeout: 10800
nginx.ingress.kubernetes.io/proxy-read-timeout: 10800
nginx.ingress.kubernetes.io/proxy-send-timeout: 10800
nginx.org/websocket-services: app-sockets-cluster-ip-service
My question is : why i can get broadcast to all user message and not specific message to my socket ?
Can someone try to help me ? :)
Thanks a lot !
I found the solution in the day.and share it.
In fact, the problem is not due to the kubernetes Cluster but due to the socket io and socket io redis adapter version.
I was using socket.io: 3.x.x and using socket.io-redis: 5.x.x
In fact, i need to use the socket.io-redis: 6.x.x with this version of socket io :)
You can find the compatible version of socket io and redis adapter here:
https://github.com/socketio/socket.io-redis-adapter#compatibility-table
Thanks a lot.

Let's encrypt SSL with traefick on ECS Fargate

I've been trying to solve this for days, but without any luck:
Situation:
I have a ECS cluster on AWS using Fargate, this cluster contains an instance of Traefick 2.3.4 and other containers. I'm using Traefick as reverse proxy to forward the requests to the other containers.
Using HTTP everything works fine, so I've decided to add also the secure connection to Traefick. I've tried everything that I could find on the Internet but nothing works, when I try to connect to the specified domain with curl it returns:
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Here there are some test that I've done:
traefick.yml:
log:
level: DEBUG
api:
dashboard: true
entryPoints:
web:
address: :80
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: ":443"
providers:
ecs:
clusters:
- tools-cluster
region: eu-west-2
exposedByDefault: false
certificatesResolvers:
letsencrypt:
acme:
caServer: https://acme-staging-v02.api.letsencrypt.org/directory
email: #########################
storage: acme.json
httpchallenge:
entrypoint: web
Labels:
"dockerLabels": {
"traefik.enable": "true",
"traefik.http.services.traefik.loadbalancer.server.port": "8080",
"traefik.http.routers.traefik.rule": "Host(`${host}`)",
"traefik.http.routers.traefik.entrypoints": "websecure",
"traefik.http.routers.traefik.tls.certresolver": "letsencrypt",
"traefik.http.routers.traefik.service": "api#internal"
}
this version returns this error:
rror: 400 :: urn:ietf:params:acme:error:connection :: Fetching https://traefik.baaluu.com/.well-known/acme-challenge/td8IdOvJ1_GkigY-jPYaA4YsgeiS5FUiuUS-avbpsuY: Error getting validation data, url
It tries to retrieve that data but it can't because it is redirected to the https and it can't retrieve because https doesn't work, I've tried also without the auto redirect, and it returns a similar error, it can't retrieve that data.
But following this guide it should work correctly.
So I've decided to move to the dnsChallenge with this configuration:
Traefick.yml
log:
level: DEBUG
api:
dashboard: true
entryPoints:
web:
address: :80
websecure:
address: ":443"
providers:
ecs:
clusters:
- tools-cluster
region: eu-west-2
exposedByDefault: false
certificatesResolvers:
letsencrypt:
acme:
caServer: https://acme-staging-v02.api.letsencrypt.org/directory
email: ######################
storage: acme.json
dnsChallenge:
provider: route53
delayBeforeCheck: 3
and same labels as before:
"dockerLabels": {
"traefik.enable": "true",
"traefik.http.services.traefik.loadbalancer.server.port": "8080",
"traefik.http.routers.traefik.rule": "Host(`${host}`)",
"traefik.http.routers.traefik.entrypoints": "websecure",
"traefik.http.routers.traefik.tls.certresolver": "letsencrypt",
"traefik.http.routers.traefik.service": "api#internal"
}
Still nothing, and I've this inside the logs:AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/170242259"
That url contains:
{
"type": "urn:ietf:params:acme:error:malformed",
"detail": "Method not allowed",
"status": 405
}
The latest test that I did is to remove the staging ca server:
log:
level: DEBUG
api:
dashboard: true
entryPoints:
web:
address: :80
websecure:
address: :443
providers:
ecs:
clusters:
- tools-cluster
region: eu-west-2
exposedByDefault: false
certificatesResolvers:
letsencrypt:
acme:
email: ###############
storage: acme.json
dnsChallenge:
provider: route53
delayBeforeCheck: 2
The ssl still doesn't work but I don't see any error message inside the logs: this is the last message that I get about a certificate:
Try to challenge certificate for domain [traefik.baaluu.com] found in HostSNI rule" providerName=letsencrypt.acme routerName=traefik#ecs rule="Host(`traefik.baaluu.com`)"
And there is not much more after that:
(I'm sorry for the picture but I don't find a way to extract that logs from ECS)
The other containers are still reachable on the http protocol.
If I try to connect to it using telnet I can reach the service:
telnet traefik.baaluu.com 443
Trying 3.8.30.164...
Connected to traefik-1547500306.eu-west-2.elb.amazonaws.com.
Escape character is '^]'.
Same goes for the 80
Looking better inside the logs I've also find this
retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/chall-v3/9205340157/1Wh0tQ :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: \"0004cbkFTGjCALFGDYOmhruMl6_F_fRSj33cOMvdpx5Xd2M\", url: "
time="2020-12-10T13:08:21Z" level=debug msg="legolog: [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/chall-v3/9205340157/1Wh0tQ :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: \"0004cbkFTGjCALFGDYOmhruMl6_F_fRSj33cOMvdpx5Xd2M\", url: "
that contains this url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/9205340157/1Wh0tQ
{
"type": "dns-01",
"status": "valid",
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/9205340157/1Wh0tQ",
"token": "44R4gD4_ZmemiCn5rtkqJyWOcjoj09sEgobUvZLH6yc",
"validationRecord": [
{
"hostname": "traefik.baaluu.com"
}
]
}
So I suppose that the ssl has been generated correctly but I'm not sure.
Any idea or suggestion?
Thanks in advance.
H2K
Edit:
I've removed the ssl from the dashboard and I've put it on another container, now entering inside the dashboard I can see this:
So I suppose that the ssl is working for that domain, but I still can't connect to it.
Edit 2:
with telnet if I connect to that url on the port 443 and I request the page I can see the content:
telnet xxxxxxxxxxxxxxxxx 443
Trying 3.10.148.201...
Connected to traefik-1547500306.eu-west-2.elb.amazonaws.com.
Escape character is '^]'.
GET /index.html HTTP/1.1
Host: xxxxxxxxxxxxxxxxx
And the content of the page appears, so it is not a load balacer problem or routing problem, it seems that I can reach the container using the 443, simply the ssl is not there. It is like to have 2 http port and both are behaving in the same way. The 443 at the moment is like a port 80.
I've have also spent a number of days trying to work it out so i feel your pain.
The error is misleading, the request doesn't even make it past the ALB let alone traefik.
There are two factors to this issue,
The first being that when you specify a port 443 through docker compose as "443:443" you would assume that this creates a HTTPS listener, it actually creates a listener for 443 on the HTTP protocol. In addition the listener also sent the data to the fargate HTTP port and didn't redirect. I'm not sure if this is a bug, or because because i haven't specified that the protocol should be "x-aws-protocol: https" on the target port.
I also found some AWS documentation that said if you use a HTTPS port on a ALB that you need an SSL certificate in place at a ALB level. This kind of makes sense that you can't terminate the connection at a task level if you consider the swarm nature and security implications (better minds are welcome to explain)
With the above in mind i created a certificate in the ACM that covered all the the domains that i needed, changed the listener to the HTTPS protocol and specified the certificate i created. At this point i was able to configure traefik to accept traefik to the frontend.

I can see live app on secured port 443 red5pro

I prepared server ubuntu like from docs. I created SSL cert to my domin and i have open required ports. I installed red5pro in to /usr/local/red5pro/ and server fine. When i will go to http://example.com:5080/ i can see home page red5pro and is ok. But when i click on broadcast i have a info: No suitable Publisher found. WebRTC & Flash not supported. Ok, maybe because is http not https. I decided create test index page in to /var/www/test/index.html and i have basic configuration like:
var config = {
protocol: 'wss',
host: 'example.com',
port: 443,
app: 'live',
streamName: 'abccaccaa',
rtcConfiguration: {
iceServers: [{urls: 'stun:stun2.l.google.com:19302'}],
iceCandidatePoolSize: 2,
bundlePolicy: 'max-bundle'
} // See https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection#RTCConfiguration_dictionary
};
And now when i try broadcast have an info: WebSocket connection to 'wss://example.com/live/?id=abccaccaa' failed: Error during WebSocket handshake: Unexpected response code: 404
Looks like have no example.com/live and cant figure out what is wrong :( since 2 days. Maybe someone could give me an advice ? Or alternative on other application than red5pro