Balancing export to jaeger in openTelemetry collector - jaeger

I have configuration as documentation says
exporters:
jaeger:
endpoint: "ipv4:firstHost:14250,secondHost:14250"
balancer_name: "round_robin"
Collector produces error.
How I can configure collector to balance exporter for sending requests in different backends?
info exporterhelper/queued_retry.go:276 Exporting failed. Will retry the request after interval. {"component_kind": "exporter", "component_type": "jaeger", "component_name": "jaeger", "error": "failed to push trace data via Jaeger exporter: rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp: address ipv4:firstHost:14250,secondHost:14250: too many colons in address"", "interval": "30.456378855s"}

It doesn't work in golang grpc client. I used openTelemetry load balancing Another option - use kubernetes to balance requests to backends.

Related

Spring Cloud Gateway hides server websocket handshake 401 failures to clients

I'm reverse proxying a websocket backend API with spring-cloud-gateway 2.2.3. When this backend API rejects some websocket handshake request with a 401 Unauthorized status response, then spring-cloud-gateway still returns a 101 handshake status to the client (which gets confused and then misbehaves)
I need spring-cloud-gateway to return the original 401 websocket handshake error to the client so the SCG reverse proxy is transparent to the client (which is conforming to the WebSocket specs handshake)
Here are the full wiretap traces and exception (I have redacted hostnames).
The client-side response in this WSS request is available as a HAR file captured from chrome and which displays in chrome
as this screenpshot.
Here is my spring cloud gateway configuration
spring:
cloud:
gateway:
routes:
- id: route_shield
uri: https://shield-webui-cf-mysql.nd-int-cfapi.was.redacted
predicates:
- Host=**
filters:
- SetRequestHostHeader=shield-webui-cf-mysql.nd-int-cfapi.was.redacted
ssl:
useInsecureTrustManager: true
I'm wondering whether this is a spring-cloud-gateway bug, or a desired behavior which I can override.
To override it, here are alternatives I'm considering:
using circuit breaker filter and fallback to a local handler returning a 401
write a custom post-filter
Override/patch the WebsocketRoutingFilter
However my debugger breakpoint in the handle(WebSocketSession session) method does not trigger, suspecting it is not called
Likely would need to provide a RequestUpgradeStrategy bean as an alternative to the default implementation of org.springframework.web.reactive.socket.server.upgrade.ReactorNettyRequestUpgradeStrategy#getNativeResponse mentionned in the trace
io.netty.handler.codec.http.websocketx.WebSocketHandshakeException: Invalid handshake response getStatus: 401 Unauthorized
at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:274) ~[netty-codec-http-4.1.51.Final.jar:4.1.51.Final]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ http://localhost:8080/v2/events [ReactorNettyRequestUpgradeStrategy]

Proxy Error: Error reading from remote server

I am fetching a report from my apache including a 41000 rows data report and it gives me following error. I checked the logs and there is no exception there:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /reporting/cards/TransactionHistory_cp_trd_trid1.rpt.
Reason: Error reading from remote server
But when I fetched about 20000 rows, it fetched perfectly.

Why SpringBoot RabbitMQ client auto shutdown connection to rabbitmq server

**
geting rabbitmq connection error as follows.
**
2019-07-11 13:14:51.147.AMQP Connection 127.0.0.1:5672> ERROR - TID[] UID[] MID[] CID[] - Channel shutdown: connection error; protocol method: #method(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
2019-07-11 13:14:51.831.bulkNotificationContainer-100> WARN - TID[] UID[] MID[] CID[] - Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:742) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:732) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.handleConnectionClose(AMQConnection.java:671) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.processControlCommand(AMQConnection.java:625) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection$1.processAsync(AMQConnection.java:102) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:143) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:90) ~[amqp-client-3.6.5.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:549) ~[amqp-client-3.6.5.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
My spring-boot service using rabbitmq to send push notification asynchronously. So push payload publish/received via rabbitmq then send on FCM. However, this connection is working for last one year without any problem. But this morning rabbitmq server is restarted as follows
rabbitmq stop
kill process bean ( related erlang )
rabbitmq start
after that we restart spring-boot service successfully, and health-api shows rabbit-up status.
but having error when trying to send push.
application properties configuration.
spring.rabbitmq.host=127.0.0.1 spring.rabbitmq.port=5672
spring.rabbitmq.username=rabbitadmin spring.rabbitmq.password=admin
custom configuration
rabbitmq.listeners.retry-policy = UNIFORM_RANDOM_DELAY
rabbitmq.listeners.max-interval=15000
rabbitmq.listener.push-router.concurrent-consumers=2
rabbitmq.listener.push-router.max-concurrent-consumers=10
rabbitmq.binding.push-notification.queue.name=pushqueue
rabbitmq.binding.push-notification.exchange.name=pushexchange
rabbitmq.binding.push-notification.binding.routing-key=pushroute-binding
I want to publish/receive data to rabbitmq channel.
This issue is resolved after reinstalling the packages of rabbitmq.
Still dont know why this happened..

Kue - Error: Auth error: ERR max number of clients reached

I am getting an redis connection error when depolyed in APP Fog. I don't know if KUE is generating too many connections.
Here is stack
/mnt/var/vcap.local/dea/apps/kue-0-543205581cf47e8551d042ab06df92e1/app/node_mod
ules/redis/index.js:516
throw callback_err;
^
Error: Auth error: ERR max number of clients reached
at Command.RedisClient.do_auth.self.send_anyway as callback
at RedisClient.return_error (/mnt/var/vcap.local/dea/apps/kue-0-543205581cf4
7e8551d042ab06df92e1/app/node_modules/redis/index.js:512:25)
at ReplyParser.RedisClient.init_parser (/mnt/var/vcap.local/dea/apps/kue-0-5
43205581cf47e8551d042ab06df92e1/app/node_modules/redis/index.js:263:14)
at ReplyParser.EventEmitter.emit (events.js:96:17)
at ReplyParser.send_error (/mnt/var/vcap.local/dea/apps/kue-0-543205581cf47e
8551d042ab06df92e1/app/node_modules/redis/lib/parser/javascript.js:296:10)
at ReplyParser.execute (/mnt/var/vcap.local/dea/apps/kue-0-543205581cf47e855
1d042ab06df92e1/app/node_modules/redis/lib/parser/javascript.js:181:22)
at RedisClient.on_data (/mnt/var/vcap.local/dea/apps/kue-0-543205581cf47e855
1d042ab06df92e1/app/node_modules/redis/index.js:488:27)
at Socket. (/mnt/var/vcap.local/dea/apps/kue-0-543205581cf47e8551
d042ab06df92e1/app/node_modules/redis/index.js:82:14)
at Socket.EventEmitter.emit (events.js:96:17)
at TCP.onread (net.js:396:14)
The redis server log shows it made 5 client connections. is there a way to reduce the no of connections
Thanks

How to receive sms via smpp connection - Kannel

I sent messages through smpp connection (using selenium SmppSim) from Kannel and it worked.
But somehow when I try to receive messages or in other words when I try to send messages from SmppSim It doesn't work. The MO messages of the SmppSim queue into the MO-queue.
I tried these things.
Used same port for send and receive (Kannel/SmppSim).
Used different ports for send and receive (Kannel/SmppSim).
Two groups for same smsc-smpp for send and receive. (It may be wrong)
Now I'm using port 2775 for send and port 2776 for receive.
#kannel.conf
group=smsc
smsc=smpp
....
port = 2775
receive-port = 2776
transceiver-mode = true
....
In SmppSim
#smppsim.props
SMPP_PORT=2775
....
SYSTEM_IDS=smppclient
PASSWORDS=password
OUTBIND_ENABLED=true
OUTBIND_ESME_IP_ADDRESS=127.0.0.1
OUTBIND_ESME_PORT=2776
OUTBIND_ESME_SYSTEMID=smppclient
OUTBIND_ESME_PASSWORD=password
....
When I run the bearerbox, it shows like below. (sms send is working)
....
connect failed
System error 111: Connection refused
ERROR: error connecting to server `localhost' at port `2776'
SMPP[SMPPSim]: Couldn't connect to server.
SMPP[SMPPSim]: Couldn't connect to SMS center (retrying in 10 seconds).
....
How do I configure this?
Thank you!
Please read SMPP v3.4 specification, part 2.2.1.
The purpose of the outbind operation is to allow the SMSC signal an ESME to originate a
bind_receiver request to the SMSC.
So it's used for SMSC (SMPPSim) to connect to ESME (Kannel) and request for callback connection.
However you can run few SMPPSim instances listening on different ports. Each instance should use own configuration file this case.