I've run a turn server(from Coturn) on a subnet machine 192.168.1.5, while its Internet IP is 183.xxx.xxx.xxx, and from https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/ .
I can see the subnet ip is gathered.
The TURN startup command is:
turnserver -o -a -f -v --mobility -m 10 --max-bps=100000 --min-port=32355 --max-port=65535 --user=webrtc0:webrtc1234 -r demo
But when I open Chrome to access the WebRTC service, in console, I saw the error log:
But in FireFox, the error log is:
I'm wondering why it says WebRTC: ICE failed, add a STUN server and see about:webrtc for more details.
And if I visit 183.xxx.xxx.xxx:3478, The connection was reset appears :
Some points to notice:
In intranet environment, use 192.168.1.5 or 183.xxx.xxx.xxx to access the WebRTC service is OK!
2.In Extranet environment, which is Internet environment, use 183.xxx.xxx.xxx to access the WebRTC service failed!
The log's here:
1068: : session 128000000000000003: realm <demo> user <>: incoming packet BINDING processed, success
1068: : session 128000000000000003: realm <demo> user <>: incoming packet message processed, error 401: Unauthorized
1068: : IPv4. Local relay addr: 192.168.1.5:43040
1068: : session 128000000000000003: new, realm=<demo>, username=<webrtc0>, lifetime=600
1068: : session 128000000000000003: realm <demo> user <webrtc0>: incoming packet ALLOCATE processed, success
1068: : session 128000000000000003: refreshed, realm=<demo>, username=<webrtc0>, lifetime=0
1068: : session 128000000000000003: realm <demo> user <webrtc0>: incoming packet REFRESH processed, success
1069: : session 128000000000000003: usage: realm=<demo>, username=<webrtc0>, rp=4, rb=232, sp=4, sb=416
1069: : session 128000000000000003: peer usage: realm=<demo>, username=<webrtc0>, rp=0, rb=0, sp=0, sb=0
1069: : session 128000000000000003: closed (2nd stage), user <webrtc0> realm <demo> origin <>, local 192.168.1.5:3478, remote 58.101.35.211:48671, reason: allocation timeout
1069: : session 128000000000000003: delete: realm=<demo>, username=<webrtc0>
1073: : session 004000000000000003: refreshed, realm=<demo>, username=<webrtc0>, lifetime=600
1073: : session 004000000000000003: realm <demo> user <webrtc0>: incoming packet REFRESH processed, success
1073: : IPv4. tcp or tls connected to: 183.xxx.xxx.xxx:64128
1073: : session 001000000000000008: realm <demo> user <>: incoming packet message processed, error 401: Unauthorized
1073: : IPv4. Local relay addr: 192.168.1.5:48887
1073: : session 001000000000000008: new, realm=<demo>, username=<webrtc0>, lifetime=600
1073: : session 001000000000000008: realm <demo> user <webrtc0>: incoming packet ALLOCATE processed, success
1088: : session 008000000000000008: refreshed, realm=<demo>, username=<webrtc0>, lifetime=600
1088: : session 008000000000000008: realm <demo> user <webrtc0>: incoming packet REFRESH processed, success
1088: : IPv4. tcp or tls connected to: 183.xxx.xxx.xxx:64156
1088: : session 002000000000000008: realm <demo> user <>: incoming packet message processed, error 401: Unauthorized
1088: : IPv4. Local relay addr: 192.168.1.5:42791
1088: : session 002000000000000008: new, realm=<demo>, username=<webrtc0>, lifetime=600
1088: : session 002000000000000008: realm <demo> user <webrtc0>: incoming packet ALLOCATE processed, success
1089: : session 003000000000000001: realm <demo> user <webrtc0>: incoming packet message processed, error 438: Stale nonce
1089: : session 003000000000000001: refreshed, realm=<demo>, username=<webrtc0>, lifetime=600
1089: : session 003000000000000001: realm <demo> user <webrtc0>: incoming packet REFRESH processed, success
1104: : session 001000000000000003: refreshed, realm=<demo>, username=<webrtc0>, lifetime=600
1104: : session 001000000000000003: realm <demo> user <webrtc0>: incoming packet REFRESH processed, success
1104: : session 003000000000000002: realm <demo> user <webrtc0>: incoming packet message processed, error 438: Stale nonce
1104: : session 003000000000000002: refreshed, realm=<demo>, username=<webrtc0>, lifetime=600
1104: : session 003000000000000002: realm <demo> user <webrtc0>: incoming packet REFRESH processed, success
Can anyone give me a clue?
Related
I used this tutorial https://github.com/jitsi/jibri and this video https://www.youtube.com/watch?v=OHHoqKCjJ0E 2
to install jibri
But I am not able to start the recording
On the Jitsi side
In the /etc/prosody/conf.d/osboxes.osboxes.demoanuswadh.info.cfg.lua
I add the users for jibri, recorded and using prosodyctl
Added the internal.auth and recorder components
In the /etc/jitsi/meet/osboxes.demoanuswadh.info-config.js
Enabled fileRecordingsEnabled: ture
Added the hidden domain
In the /etc/jitsi/jicofo/sip-communicator.properties
org.jitsi.jicofo.jibri.BREWERY=JibriBrewery#internal.auth.osboxes.demoanuswadh.info
org.jitsi.jicofo.jibri.PENDING_TIMEOUT=90
these are the last hundred lines of the log - log.0.txt
ava:312)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:369)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.jitsi.jibri.MainKt.launchHttpServer(Main.kt:173)
at org.jitsi.jibri.MainKt.main(Main.kt:158)
Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
… 66 more
MultiException stack 2 of 2
java.lang.IllegalArgumentException: Errors were discovered while reifying SystemDescriptor(
implementation=org.glassfish.jersey.message.internal.DataSourceProvider
contracts={javax.ws.rs.ext.MessageBodyReader,javax.ws.rs.ext.MessageBodyWriter}
scope=javax.inject.Singleton
qualifiers={}
descriptorType=CLASS
descriptorVisibility=NORMAL
metadata=
rank=0
loader=null
proxiable=null
proxyForSameScope=null
analysisName=null
id=106
locatorId=1
identityHashCode=1778422985
reified=false)
at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:705)
at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:464)
at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2310)
at org.jvnet.hk2.internal.ServiceLocatorImpl.access$1200(ServiceLocatorImpl.java:128)
at org.jvnet.hk2.internal.ServiceLocatorImpl$9.compute(ServiceLocatorImpl.java:1395)
at org.jvnet.hk2.internal.ServiceLocatorImpl$9.compute(ServiceLocatorImpl.java:1390)
at org.glassfish.hk2.utilities.cache.internal.WeakCARCacheImpl.compute(WeakCARCacheImpl.java:128)
at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetAllServiceHandles(ServiceLocatorImpl.java:1452)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServiceHandles(ServiceLocatorImpl.java:1377)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getAllServiceHandles(ServiceLocatorImpl.java:1366)
at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getAllServiceHolders(AbstractHk2InjectionManager.java:158)
at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getAllServiceHolders(ImmediateHk2InjectionManager.java:54)
at org.glassfish.jersey.internal.inject.Providers.getServiceHolders(Providers.java:329)
at org.glassfish.jersey.internal.inject.Providers.getProviders(Providers.java:157)
at org.glassfish.jersey.message.internal.MessageBodyFactory.initialize(MessageBodyFactory.java:265)
at org.glassfish.jersey.message.internal.MessageBodyFactory$MessageBodyWorkersConfigurator.postInit(MessageBodyFactory.java:136)
at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$2(ApplicationHandler.java:372)
at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4390)
at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:372)
at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:256)
at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:315)
at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:282)
at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:335)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:178)
at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:370)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:671)
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:428)
at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750)
at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:369)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.jitsi.jibri.MainKt.launchHttpServer(Main.kt:173)
at org.jitsi.jibri.MainKt.main(Main.kt:158)
2020-04-18 19:06:12.543 SEVERE: [21] org.jitsi.xmpp.mucclient.MucClientManager.log() Failed to initialize and start a MucClient:
org.jivesoftware.smack.SmackException$ConnectionException: The following addresses failed: ‘osboxes.demoanuswadh.info:5222’ failed because: osboxes.demoanuswadh.info/77.525.75.82 exception: java.net.ConnectException: Connection refused (Connection refused)
at org.jivesoftware.smack.SmackException$ConnectionException.from(SmackException.java:278)
at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectUsingConfiguration(XMPPTCPConnection.java:619)
at org.jivesoftware.smack.tcp.XMPPTCPConnection.connectInternal(XMPPTCPConnection.java:902)
at org.jivesoftware.smack.AbstractXMPPConnection.connect(AbstractXMPPConnection.java:383)
at org.jitsi.xmpp.mucclient.MucClient.initializeConnectAndJoin(MucClient.java:277)
at org.jitsi.xmpp.mucclient.MucClientManager.lambda$addMucClient$0(MucClientManager.java:152)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-04-18 19:06:42.348 WARNING: [30] org.jivesoftware.smackx.ping.PingManager.pingServerIfNecessary() XMPPConnection was not authenticated
2020-04-18 19:22:59.687 INFO: [17] org.jitsi.jibri.api.http.internal.InternalHttpApi.gracefulShutdown() Jibri gracefully shutting down
The /etc/jitsi/jibri/config.json
{
// NOTE: this is a *SAMPLE* config file, it will need to be configured with
// values from your environment
// Where recording files should be temporarily stored
"recording_directory":"/xxxxxxxxxx/xxxxxxxxx/recordings",
// The path to the script which will be run on completed recordings
"finalize_recording_script_path": "/path/to/finalize_recording.sh",
"xmpp_environments": [
{
// A friendly name for this environment which can be used
// for logging, stats, etc.
"name": "prod environment",
// The hosts of the XMPP servers to connect to as part of
// this environment
"xmpp_server_hosts": [
"osboxes.demoanuswadh.info"
],
// The xmpp domain we'll connect to on the XMPP server
"xmpp_domain": "osboxes.demoanuswadh.info",
// Jibri will login to the xmpp server as a privileged user
"control_login": {
// The domain to use for logging in
"domain": "auth.osboxes.demoanuswadh.info",
// The credentials for logging in
"username": "jibri",
"password": "xxxxxxxxxxxxxxxxxx"
},
// Using the control_login information above, Jibri will join
// a control muc as a means of announcing its availability
// to provide services for a given environment
"control_muc": {
"domain": "internal.auth.osboxes.demoanuswadh.info",
"room_name": "JibriBrewery",
"nickname": "jibri-nickname"
},
// All participants in a call join a muc so they can exchange
// information. Jibri can be instructed to join a special muc
// with credentials to give it special abilities (e.g. not being
// displayed to other users like a normal participant)
"call_login": {
"domain": "recorder.osboxes.demoanuswadh.info",
"username": "recorder",
"password": "xxxxxxxxxxxxxxxxxxxxxx"
},
// When jibri gets a request to start a service for a room, the room
// jid will look like:
// roomName#optional.prefixes.subdomain.xmpp_domain
// We'll build the url for the call by transforming that into:
// https://xmpp_domain/subdomain/roomName
// So if there are any prefixes in the jid (like jitsi meet, which
// has its participants join a muc at conference.xmpp_domain) then
// list that prefix here so it can be stripped out to generate
// the call url correctly
"room_jid_domain_string_to_strip_from_start": "conference.",
// The amount of time, in minutes, a service is allowed to continue.
// Once a service has been running for this long, it will be
// stopped (cleanly). A value of 0 means an indefinite amount
// of time is allowed
"usage_timeout": "0"
}
]
}
I will appreciate any help.
Thanks in advance
In your log says connection refused.
2020-04-18 19:06:12.543 SEVERE: [21] org.jitsi.xmpp.mucclient.MucClientManager.log() Failed to initialize and start a MucClient:[...] failed because: osboxes.demoanuswadh.info/77.525.75.82 exception: java.net.ConnectException: Connection refused (Connection refused)
Verify user, domain and password in jitsi.
When ready, delete log.0.txt or rename it and restart jibri service to obtain a clear log.
Before installing Jibri you have to enable the 5222 port in the Jitsi server. Better if you can go with Debian server for only Jibri. Not the Jitsi server. Because FFmpeg and chromium services can work smoothly on Debian server.
2020-04-18 19:06:12.543 SEVERE: [21] org.jitsi.xmpp.mucclient.MucClientManager.log() Failed to initialize and start a MucClient:
org.jivesoftware.smack.SmackException$ConnectionException: The following addresses failed: ‘osboxes.demoanuswadh.info:5222’ failed because: osboxes.demoanuswadh.info/77.525.75.82 exception: java.net.ConnectException: Connection refused (Connection refused)
Your log said jibri connecting to osboxes.demoanuswadh.info (will it resolved to localhost or a public IP ?)
If you installed jitsi-meet and jibri on the same host, make sure osboxes.demoanuswadh.info should be resolved to localhost.
If you installed on different machine within the same subnet,
osboxes.demoanuswadh.info should be resolved to a private IP
If you installed on different machine within the same subnet,
osboxes.demoanuswadh.info should be resolved to a public IP
For the last 2 cases,
make sure your prosody is listening on 5222 port bind to 0.0.0.0, also allow firewall access using ufw (ubuntu firewall)
I'm completely new to RabbitMQ and now I'm looking for a configuration error. The client doesn't receive any messages from RabbitMQ and I debugged it as far as possible.
Frontend messages:
Message 1:
CONNECT
login:frontend_listener
passcode:xxx
accept-version:1.0,1.1,1.2
heart-beat:20000,0
Message 2:
ERROR
message:Bad CONNECT
content-type:text/plain
version:1.0,1.1,1.2
content-length:30
Virtual host '/' access denied
There are two vHosts: / and someVhost and there are different users like frontend_listener. Now I found a way to access the log file.
RabbitMQ log file:
2020-02-11 15:50:53.579 [warning] <0.798.0> STOMP login failed for user "frontend_listener"
2020-02-11 15:50:53.579 [error] <0.798.0> STOMP error frame sent:
Message: "Bad CONNECT"
Detail: "Access refused for user 'frontend_listener'\n"
Server private detail: none
...
2020-02-11 15:51:25.349 [info] <0.850.0> Creating user 'frontend_listener'
2020-02-11 15:51:30.374 [info] <0.857.0> Setting permissions for 'frontend_listener' in 'someVhost' to '$', '$', 'client-notification.*'
2020-02-11 15:51:54.980 [warning] <0.867.0> STOMP login failed - not_allowed (vhost access not allowed)~n
2020-02-11 15:51:54.980 [error] <0.867.0> STOMP error frame sent:
Message: "Bad CONNECT"
Detail: "Virtual host '/' access denied"
Server private detail: none
2020-02-11 15:52:56.427 [warning] <0.875.0> STOMP login failed - not_allowed (vhost access not allowed)~n
It reads like the permissions are wrong. Can someone help me out interpreting that correctly?
I try to read it: User frontend_listener wants to access the vHost /, but it hasn't sufficient permissions (don't know what $ here mean other than a part of regular expression). The thing is, that I don't know if that is the correct vHost. How do I find out the URL of each vHost?
I'm asking this because I believe that the mapping to the vHost is wrong or something is missing.
Edit:
After adding host: 'someVhost' to my stomp-config.ts I was able to subscribe to the queues. Now I get the following error in the log:
2020-02-12 16:32:25.913 [error] <0.5159.1> Channel error on connection <0.5149.1> (127.0.0.1:58136 -> 127.0.0.1:15674, vhost: 'someVhost', user: 'frontend_listener'), channel 1:
operation basic.consume caused a channel exception access_refused: access to queue 'stomp-subscription-SZ3-PO1-PbZroPol-WXSQw' in vhost 'someVhost' refused for user 'frontend_listener'
2020-02-12 16:32:26.022 [error] <0.5145.1> STOMP error frame sent:
Message: access_refused
On the frontend I don't get a message or error.
You need to also pass host information in the STOMP CONNECT frame..
this is what the specifications says and clients MUST set this header
host : The name of a virtual host that the client wishes to connect to. It is recommended clients set this to the host name that the socket was established against, or to any name of their choosing. If this header does not match a known virtual host, servers supporting virtual hosting MAY select a default virtual host or reject the connection.
So this is how your CONNET frame should look
CONNECT
login:frontend_listener
passcode:xxx
accept-version:1.0,1.1,1.2
host: someVhost
heart-beat:20000,0
I am using the ext-authz filter using a cluster setting like below:
static_resources:
clusters:
- name: ext-authz
type: static
http2_protocol_options: {}
hosts:
# Host docker0 IP address.
- socket_address: { address: 172.17.0.1, port_value: 10003 }
# THIS SETTING does not seem to be honored
connect_timeout: 5s
However, Envoy still seems to use the default timeout of 200ms, by looking at the timestamps (Envoy started with loglevel=debug in my docker-compose.yml file):
[2018-10-22 12:11:36.517][39][debug][router] source/common/router/router.cc:252] [C0][S17403403242644461340] cluster 'ext-authz' match for URL '/envoy.service.auth.v2alpha.Authorization/Check'
[2018-10-22 12:11:36.517][39][debug][router] source/common/router/router.cc:303] [C0][S17403403242644461340] router decoding headers:
':method', 'POST'
':path', '/envoy.service.auth.v2alpha.Authorization/Check'
':authority', 'ext-authz'
':scheme', 'http'
'te', 'trailers'
'grpc-timeout', '200m'
'content-type', 'application/grpc'
'x-envoy-internal', 'true'
'x-forwarded-for', '172.21.0.2'
'x-envoy-expected-rq-timeout-ms', '200'
[2018-10-22 12:11:36.517][39][debug][client] source/common/http/codec_client.cc:25] [C5] connecting
[2018-10-22 12:11:36.517][39][debug][connection] source/common/network/connection_impl.cc:632] [C5] connecting to 172.17.0.1:10003
[2018-10-22 12:11:36.517][39][debug][connection] source/common/network/connection_impl.cc:641] [C5] connection in progress
[2018-10-22 12:11:36.517][39][debug][http2] source/common/http/http2/codec_impl.cc:632] [C5] setting stream-level initial window size to 268435456
[2018-10-22 12:11:36.517][39][debug][http2] source/common/http/http2/codec_impl.cc:654] [C5] updating connection-level initial window size to 268435456
[2018-10-22 12:11:36.517][39][debug][pool] source/common/http/http2/conn_pool.cc:97] [C5] creating stream
[2018-10-22 12:11:36.517][39][debug][router] source/common/router/router.cc:981] [C0][S17403403242644461340] pool ready
[2018-10-22 12:11:36.517][39][debug][connection] source/common/network/connection_impl.cc:514] [C5] connected
[2018-10-22 12:11:36.517][39][debug][client] source/common/http/codec_client.cc:63] [C5] connected
[2018-10-22 12:11:36.716][39][debug][router] source/common/router/router.cc:438] [C0][S17403403242644461340] upstream timeout
[2018-10-22 12:11:36.716][39][debug][router] source/common/router/router.cc:926] [C0][S17403403242644461340] resetting pool request
[2018-10-22 12:11:36.716][39][debug][client] source/common/http/codec_client.cc:104] [C5] request reset
[2018-10-22 12:11:36.716][39][debug][pool] source/common/http/http2/conn_pool.cc:189] [C5] destroying stream: 0 remaining
[2018-10-22 12:11:36.716][39][debug][http2] source/common/http/http2/codec_impl.cc:467] [C5] sent reset code=0
[2018-10-22 12:11:36.716][39][debug][http2] source/common/http/http2/codec_impl.cc:512] [C5] stream closed: 0
[2018-10-22 12:11:36.716][39][debug][http] source/common/http/async_client_impl.cc:94] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream request timeout'
[2018-10-22 12:11:36.716][39][debug][filter] source/extensions/filters/http/ext_authz/ext_authz.cc:104] [C4][S8759593104547971249] ext_authz rejected the request
Am I missing something obvious here or is this a bug in Envoy?
This causes the first request to fail for me, since establishing the connection takes such a long time. Subsequent requests succeed, since they utilize a HTTP/2 persistent connection so the handshake doesn't take any time. (or at least, much less time)
The friendly people in the Envoy community helped me find the problem: https://github.com/envoyproxy/envoy/issues/4829
The problem was that connect_timeout isn't really the request timeout but specifically, the connect timeout. The documentation has now been updated to list the proper way to set the request timeout:
http_filters:
- name: envoy.ext_authz
config:
grpc_service:
envoy_grpc:
cluster_name: ext-authz
# Default is 200ms; override if your server needs e.g. warmup time.
timeout: 0.5s
With this setting in place, things work just like a charm and my initial requests are no longer failing.
I am try to replicate the SASL_PLAIN or SASL_SSL authentication described at: http://docs.confluent.io/3.0.0/kafka/sasl.html#sasl-configuration-for-kafka-brokers
In config/server.properties, I added the following 4 lines:
listeners=SASL_SSL://localhost:9092
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
In config/producer.properties, I added the following two lines:
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
Then I set the following environment variable in the server terminal:
KAFKA_OPTS=/home/kafka/kafka_server_jaas.conf
This file has the following content:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret";
};
And in the producer terminal I define the following env variable:
KAFKA_OPTS=/home/kafka/kafka_client_jaas.conf
And this file has the following content:
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="alice"
password="alice-dsecret";
};
I start the server with the following command:
./bin/kafka-server-start.sh config/server.properties
And the producer with following command:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Both start without problems. But, as soon as I type something on the producer console, I get the following message that keeps scrolling:
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
WARN Bootstrap broker localhost:9092 disconnected (org.apache.kafka.clients.NetworkClient)
If I remove the security configuration from the server and the producer configuration, everything works as expected. I am using Kafka 0.10.0.1.
UPDATE:
I did some more investigations, turning log levels to DEBUG on server reveals something weird. As soon as I specify the listeners field in server.properties, the server goes in a weird state. It establishes connection to itsself that it cannot authenticate. The protocol in this case was SASL_PLAINTEXT.
The logs as here:
2016-09-15 21:43:02 DEBUG SaslClientAuthenticator:204 - Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE
2016-09-15 21:43:02 DEBUG NetworkClient:476 - Completed connection to node 0
2016-09-15 21:43:02 DEBUG Acceptor:52 - Accepted connection from /127.0.0.1 on /127.0.0.1:9092. sendBufferSize [actual|requested]: [102400|102400] recvBufferSize [actual|requested]: [102400|102400]
2016-09-15 21:43:02 DEBUG Processor:52 - Processor 2 listening to new connection from /127.0.0.1:42815
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:269 - Set SASL server state to HANDSHAKE_REQUEST
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:310 - Handle Kafka request SASL_HANDSHAKE
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:354 - Using SASL mechanism 'PLAIN' provided by client
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:269 - Set SASL server state to AUTHENTICATE
2016-09-15 21:43:02 DEBUG SaslClientAuthenticator:204 - Set SASL client state to INITIAL
2016-09-15 21:43:02 DEBUG SaslClientAuthenticator:204 - Set SASL client state to INTERMEDIATE
2016-09-15 21:43:02 DEBUG SaslServerAuthenticator:269 - Set SASL server state to FAILED
2016-09-15 21:43:02 DEBUG Selector:345 - Connection with /127.0.0.1 disconnected
java.io.IOException: javax.security.sasl.SaslException: Authentication failed: Invalid JAAS configuration [Caused by javax.security.sasl.SaslException: Authentication failed: Invalid username or password]
at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:243)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:64)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:318)
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
at kafka.network.Processor.poll(SocketServer.scala:472)
There is absolutely no other client or server running. This is one server talking to himself.
Any thoughts?
Help came from the Kafka forum. See http://mail-archives.apache.org/mod_mbox/kafka-users/201609.mbox/%3CCAHX2Snk11vg7DXNVUr9oE97ikFSQUoT3kBLAxYymEDj7E14XrQ%40mail.gmail.com%3E
I had the credentials wrong. They were:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="alice-secret"
user_alice="alice-secret";
};
Instead of:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret";
};
Also, the console consumer needs to be called in a certain. First the flag --new-consumer should be provided. Second, bootstrap server should be specified. Leading to this:
bin/kafka-console-consumer.sh --new-consumer --zookeeper localhost:2181 --topic test --from-beginning --consumer.config=config/consumer.properties --bootstrap-server=localhost:9092
I am using Kafka 0.8.2-beta and have 2 Ubuntu 14 virtual machines:
172.30.141.127 is running Zookeeper
172.30.141.184 is running a Kafka broker
I'm starting the Zookeeper instance and all if fine. Then I try to start the broker and connect it to 172.30.141.127:2181. It seems to be able to connect and establish a session on a specific port, but then it losses connection due to some exception that doesn't seem to be logged.
The broker output:
[2015-01-19 11:03:55,029] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,030] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,031] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,031] INFO Client environment:os.arch=i386 (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,032] INFO Client environment:os.version=3.16.0-23-generic (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,033] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,033] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,037] INFO Client environment:user.dir=/home/osboxes/Desktop/kafka_2.11-0.8.2-beta (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,039] INFO Initiating client connection, connectString=172.30.141.127:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#1ecf473 (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:55,129] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:55,186] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:55,203] WARN Session 0x0 for server 172.30.141.127/172.30.141.127:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2015-01-19 11:03:56,552] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:56,555] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:56,567] WARN Session 0x0 for server 172.30.141.127/172.30.141.127:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2015-01-19 11:03:57,131] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-01-19 11:03:58,075] INFO Opening socket connection to server 172.30.141.127/172.30.141.127:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:58,077] INFO Socket connection established to 172.30.141.127/172.30.141.127:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:58,195] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2015-01-19 11:03:58,196] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2015-01-19 11:03:58,251] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:157)
at kafka.server.KafkaServer.startup(KafkaServer.scala:83)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2015-01-19 11:03:58,279] INFO [Kafka Server 1], shutting down (kafka.server.KafkaServer)
[2015-01-19 11:03:58,295] INFO [Kafka Server 1], shut down completed (kafka.server.KafkaServer)
[2015-01-19 11:03:58,308] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:157)
at kafka.server.KafkaServer.startup(KafkaServer.scala:83)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2015-01-19 11:03:58,335] INFO [Kafka Server 1], shutting down (kafka.server.KafkaServer)
Zookeper outputs:
[2015-01-19 11:03:55,245] INFO Accepted socket connection from /172.30.141.184:54089 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-01-19 11:03:55,315] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:55,329] INFO Closed socket connection for client /172.30.141.184:54089 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:56,613] INFO Accepted socket connection from /172.30.141.184:54090 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-01-19 11:03:56,615] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:56,617] INFO Closed socket connection for client /172.30.141.184:54090 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:58,133] INFO Accepted socket connection from /172.30.141.184:54091 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-01-19 11:03:58,134] WARN Exception causing close of session 0x0 due to java.io.IOException: Connection reset by peer (org.apache.zookeeper.server.NIOServerCnxn)
[2015-01-19 11:03:58,135] INFO Closed socket connection for client /172.30.141.184:54091 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
Ping works between the 2 machines. Telnet to 2181 kind of works, in that it connects, but gets disconnected from time to time. This leads me to think that the problem is with the Zookeeper instance. Both processes are started as root.
Any ideas why this is happening? Thanks
You are probably hitting the max number of connections per host.
This happens if you have [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory#247] - Too many connections from /127.0.0.1 - max is 10 in your zookeeper logs.
Fix it by setting maxClientCnxns=[something more than 10; 0 for unlimited] in your conf/zoo.cfg.
Docs (search for maxClientCnxns)
No idea about the warning, but I had the same Error problem
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 2000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
....
In my case when I was setting up producer and consumer, I provided wrong ip/port. When I changed it to correct one with:
bin/kafka-console-producer.sh --broker-list kafka_ip:kafka_port --topic test
bin/kafka-console-consumer.sh --zookeeper zookeeper_id:zookeper_port --topic test --from-beginning
my problem was solved.
As answered by #RickyA
"You are probably hitting the max number of connections per host."
Try clearing your logs folder which in my case was F:\tmp\kafka-logs.