I have a pretty stock installation with authentication. The installation is only used for 1 to many broadcasting. 1 person shares their screen, everyone starts muted & hidden, follows that 1 person.
I have adaptive last n = true, channel last n = 1.
The screenshares begin to fail after a few minutes. The screen goes gray for participants. The broadcasting user has about an upload of 500kbps in the screenshare but everyone has a download of 11kbps or N/A.
I see this in the jvb logs intermittently
JVB 2017-01-24 14:17:56.506 INFO: [278] org.ice4j.ice.ConnectivityCheckClient.log() timeout for pair: 158.69.116.11:10000/udp/host -> 108.208.24.154:58802/udp/prflx (stream.RTP), failing.
JVB 2017-01-24 12:33:51.337 INFO: [77] org.ice4j.ice.harvest.AbstractTcpListener.readFromChannel() Failed to handle TCP socket Socket[addr=/108.208.24.154,port=60215,localport=4443]: End of stream! JVB 2017-01-24 12:33:51.367 INFO: [507] org.ice4j.ice.ConnectivityCheckServer.log() Failed to send BINDING-RESPONSE(0x101)[attrib.count=5 len=96 tranID=0x776264674249706335757139] through [2607:5300:60:850b:0:0:0:0]:10000/udp java.io.IOException: Network is unreachable (sendto failed) at java.net.PlainDatagramSocketImpl.send(Native Method) at java.net.DatagramSocket.send(DatagramSocket.java:693) at org.ice4j.ice.harvest.AbstractUdpListener$MySocket.send(AbstractUdpListener.java:613) at org.ice4j.socket.DelegatingDatagramSocket.send(DelegatingDatagramSocket.java:767) at org.ice4j.socket.DelegatingDatagramSocket.send(DelegatingDatagramSocket.java:767) at org.ice4j.socket.IceUdpSocketWrapper.send(IceUdpSocketWrapper.java:53) at org.ice4j.stack.Connector.sendMessage(Connector.java:327) at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:629) at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:575) at org.ice4j.stack.StunServerTransaction.retransmitResponse(StunServerTransaction.java:201) at org.ice4j.stack.StunServerTransaction.sendResponse(StunServerTransaction.java:178) at org.ice4j.stack.StunStack.sendResponse(StunStack.java:771) at org.ice4j.ice.ConnectivityCheckServer.processRequest(ConnectivityCheckServer.java:210) at org.ice4j.stack.EventDispatcher$RequestListenerMessageEventHandler.handleMessageEvent(EventDispatcher.java:511) at org.ice4j.stack.EventDispatcher.fireMessageEvent(EventDispatcher.java:268) at org.ice4j.stack.StunStack.handleMessageEvent(StunStack.java:987) at org.ice4j.stack.MessageProcessor.run(MessageProcessor.java:171) at java.lang.Thread.run(Thread.java:745) JVB 2017-01-24 12:33:51.367 INFO: [507] org.ice4j.stack.StunStack.handleMessageEvent() Received an invalid request. java.lang.RuntimeException: Failed to send a response at org.ice4j.ice.ConnectivityCheckServer.processRequest(ConnectivityCheckServer.java:222) at org.ice4j.stack.EventDispatcher$RequestListenerMessageEventHandler.handleMessageEvent(EventDispatcher.java:511) at org.ice4j.stack.EventDispatcher.fireMessageEvent(EventDispatcher.java:268) at org.ice4j.stack.StunStack.handleMessageEvent(StunStack.java:987) at org.ice4j.stack.MessageProcessor.run(MessageProcessor.java:171) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Network is unreachable (sendto failed) at java.net.PlainDatagramSocketImpl.send(Native Method) at java.net.DatagramSocket.send(DatagramSocket.java:693) at org.ice4j.ice.harvest.AbstractUdpListener$MySocket.send(AbstractUdpListener.java:613) at org.ice4j.socket.DelegatingDatagramSocket.send(DelegatingDatagramSocket.java:767) at org.ice4j.socket.DelegatingDatagramSocket.send(DelegatingDatagramSocket.java:767) at org.ice4j.socket.IceUdpSocketWrapper.send(IceUdpSocketWrapper.java:53) at org.ice4j.stack.Connector.sendMessage(Connector.java:327) at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:629) at org.ice4j.stack.NetAccessManager.sendMessage(NetAccessManager.java:575) at org.ice4j.stack.StunServerTransaction.retransmitResponse(StunServerTransaction.java:201) at org.ice4j.stack.StunServerTransaction.sendResponse(StunServerTransaction.java:178) at org.ice4j.stack.StunStack.sendResponse(StunStack.java:771) at org.ice4j.ice.ConnectivityCheckServer.processRequest(ConnectivityCheckServer.java:210)
I set org.jitsi.videobridge.ratecontrol.VideoChannelLastNAdaptor.MAX_STAY_AT_ZERO_MS=5000 in videobridge sip-properties to force the retry after 5 seconds.
The console logs from inspecting chrome in the frozen participants shows: Logger.js:125 [JitsiMeetJS.js] <Object.getGlobalOnErrorHandler>: UnhandledError: null Script: null Line: null Column: null StackTrace: Error: Ping timeout at strophe.ping.js:102 at u.TimedHandler.handler (strophe.js:2752) at u.TimedHandler.run (strophe.js:2095) at u.Connection._onIdle (strophe.js:3706) r # Logger.js:125
What is the optimal config for this type of conference (screenshare 1 send to many receive)?
I'm honestly not sure why the video drops for some people and doesn't reconnect. Can anyone lend a helping hand?
How many viewers do you have? Note that adaptive last N is gone (as an option) it just behaves that way all the time now.
You could try to use meet.jit.si and set that everyone starts muted and hidden by the moderator just to check that it's not an infrastructure problem. Alternatively, this scenario is the perfect one for YouTube broadcasting: you can have the broadcaster(s) join the room and stream it to YouTube, where all viewers can watch it live.
You can also set users to be muted (and also prevent audio and video transmission) when a conference gets started (except the moderator who is presenting) by changing the code inside Jicofo.
This is the method name to permanently disable a user from sending video or audio:
public boolean muteParticipant(ColibriConferenceIQ channelsInfo, boolean mute, MediaType mediaType)
Related
I am facing a very wired issue, and passed so many hours to find the issue but without any luck :(
i have setup a web server on my mobile (android)..
and trying to use that using my vb application
where i simple call Download method on the very sample url (same ip/port), it works and save the response body as downloaded file.
but when i try to send post command it is not working, it gives error "Connection rejected"
here is the code to download:
objDHTTP.Download("http://192.168.1.101:8000/fileAdmin.php", strAppDir & "\wtf-response.txt")
here is the code when i tried to use same url and port for post request:
objDRequest.Path = "/fileAdmin.php"
objDRequest.AddParam("fileAction", "Upload")
objDRequest.AddParam("fileName", "DB.sql")
objDRequest.AddFileForUpload("adminFile", strAppDir & "\c.dll")
objDResponse = objDHTTP.SynchronousRequest(strServerIP, 8000, False, objDRequest)
but it does not works and throws above error
here is the success response (for download method):
<ChilkatLog>
<Download>
<DllDate>Dec 21 2018</DllDate>
<ChilkatVersion>9.5.0.76</ChilkatVersion>
<Architecture>Little Endian; 32-bit</Architecture>
<Language>.NET 4.0</Language>
<VerboseLogging>0</VerboseLogging>
<url>http://192.168.1.101:8000/fileAdmin.php</url>
<toLocalPath>G:\wtf-sucess.txt</toLocalPath>
<currentWorkingDir>G:\</currentWorkingDir>
<a_httpDownload>
<httpDownloadFile>
<localFilePath>G:\wtf-sucess.txt</localFilePath>
<localFileAlreadyExists>1</localFileAlreadyExists>
<quickHttpRequest>
<httpVerb>GET</httpVerb>
<url>http://192.168.1.101:8000/fileAdmin.php</url>
<openHttpConnection>
<info>Opening connection directly to HTTP server.</info>
<httpHostname>192.168.1.101</httpHostname>
<httpPort>8000</httpPort>
<tls>False</tls>
<info>HTTP connection succeeded.</info>
</openHttpConnection>
<buildQuickRequest>
<genStartLine>
<startLine>GET /fileAdmin.php HTTP/1.1</startLine>
</genStartLine>
<addCookies>
<info>Not auto-adding cookies.</info>
<sendCookies>1</sendCookies>
<cookieDir></cookieDir>
</addCookies>
</buildQuickRequest>
<sendRequestHeader>
<sendHeaderElapsedMs>0</sendHeaderElapsedMs>
</sendRequestHeader>
<statusCode>200</statusCode>
<statusText>OK</statusText>
<checkCloseConnection>
<info>Response includes connection:close header (or proxy-connection:close header)</info>
</checkCloseConnection>
</quickHttpRequest>
<outputLocalFileSize>26</outputLocalFileSize>
<numOutputBytesWritten>26</numOutputBytesWritten>
</httpDownloadFile>
</a_httpDownload>
<totalElapsedMs>47</totalElapsedMs>
<ContentLength>0</ContentLength>
<info>Success.</info>
</Download>
</ChilkatLog>
And here is the error/fail response (for post request):
<ChilkatLog>
<SynchronousRequest>
<DllDate>Dec 21 2018</DllDate>
<ChilkatVersion>9.5.0.76</ChilkatVersion>
<Architecture>Little Endian; 32-bit</Architecture>
<Language>.NET 4.0</Language>
<VerboseLogging>0</VerboseLogging>
<domain>191.168.1.101</domain>
<port>8000</port>
<ssl>0</ssl>
<httpRequest>
<httpVersion>1.1</httpVersion>
<verb>POST</verb>
<path>/fileAdmin.php</path>
<contentType>multipart/form-data</contentType>
<charset>windows-1252</charset>
<sendCharset>0</sendCharset>
<mimeHeader></mimeHeader>
<requestParams>
<requestItem>
<name>fileAction</name>
<value>Upload</value>
</requestItem>
<requestItem>
<name>fileName</name>
<value>DB.sql</value>
</requestItem>
<requestItem>
<name>adminFile</name>
<fileOnDisk>G:\c.dll</fileOnDisk>
<numValueBytes>8697</numValueBytes>
</requestItem>
</requestParams>
</httpRequest>
<fullRequest>
<a_synchronousRequest>
<generateRequestHeader>
<httpRequestGenStartLine>
<genStartLine>
<startLine>POST /fileAdmin.php HTTP/1.1</startLine>
</genStartLine>
</httpRequestGenStartLine>
<addCookies>
<info>Not auto-adding cookies.</info>
<sendCookies>1</sendCookies>
<cookieDir></cookieDir>
</addCookies>
<genMultipartFormData>
<requestParam>
<name>fileAction</name>
</requestParam>
<requestParam>
<name>fileName</name>
</requestParam>
<requestParam>
<name>adminFile</name>
<filename>G:\c.dll</filename>
</requestParam>
</genMultipartFormData>
</generateRequestHeader>
<fullHttpRequest>
<domain>191.168.1.101</domain>
<port>8000</port>
<ssl>0</ssl>
<openHttpConnection>
<info>Opening connection directly to HTTP server.</info>
<httpHostname>191.168.1.101</httpHostname>
<httpPort>8000</httpPort>
<tls>False</tls>
<socket2Connect>
<connect2>
<connectSocket>
<connect_ipv6_or_ipv4>
<info>Connection attempt failed.</info>
<maxWaitTimeMs>10000</maxWaitTimeMs>
<totalMsWaitedSoFar>2100</totalMsWaitedSoFar>
</connect_ipv6_or_ipv4>
<connect_ipv6_or_ipv4>
<info>Connection attempt failed.</info>
<maxWaitTimeMs>10000</maxWaitTimeMs>
<totalMsWaitedSoFar>2150</totalMsWaitedSoFar>
</connect_ipv6_or_ipv4>
</connectSocket>
<ConnectFailReason>Connection rejected</ConnectFailReason>
<error>A few possible causes for a connection being rejected are:</error>
<error>- A firewall (software or hardware), such as Windows Firewall, is blocking the connection .</error>
<error>- Nothing is listening at the remote host:port</error>
</connect2>
</socket2Connect>
</openHttpConnection>
</fullHttpRequest>
<success>0</success>
</a_synchronousRequest>
<success>0</success>
</fullRequest>
<totalTime>Elapsed time: 4390 millisec</totalTime>
<error>Failed.</error>
</SynchronousRequest>
</ChilkatLog>
can any one tell me what could be the reason for this?
thanks in advance
best regards
The likely reasons are present in the LastErrorText:
<ConnectFailReason>Connection rejected</ConnectFailReason>
<error>A few possible causes for a connection being rejected are:</error>
<error>- A firewall (software or hardware), such as Windows Firewall, is blocking the connection .</error>
<error>- Nothing is listening at the remote host:port</error>
My guess is that the Windows Firewall is blocking the outbound POST.
This example shows the errors one would get for various TCP socket connection problems: https://www.example-code.com/vbnet/socket_connect_fail.asp
I am using a Secure SIP trunk provided by Twilio to implement an IVR. I have implemented per Twilio's Asterisk configuration guide, installed SRTP to /usr/local/lib, as well as implemented the configuration in https://wiki.asterisk.org/wiki/display/AST/Secure+Calling+Tutorial.
The problem lies in any call that is longer than 2 minutes cannot be ended cleanly and causes Asterisk to restart.
sip.conf (using chan_sip, not pjsip):
[general]
; other configuration lines removed
tlsenable=yes
tlsbindaddr=0.0.0.0
tlscertfile=/etc/pki/tls/private/pbx.pem
tlscafile=/etc/pki/tls/private/gd_bundle-g2-g1.crt
tlscipher=ALL
tlsclientmethod=tlsv1
tlsdontverifyserver=yes
[twilio-trunk](!)
type=peer
context=from-twilio ;Which dialplan to use for incoming calls
dtmfmode=rfc4733
canreinvite=no
insecure=port,invite
transport=tls
qualify=yes
encryption=yes
media_encryption=sdes
I can make and receive calls just fine, and I have confirmed the calls are encrypted both via wireshark and confirmation from Twilio's own support queue.
At exactly 120 seconds into every call, this debug pops up:
[Dec 6 13:14:39] DEBUG[30015]: iostream.c:157 iostream_read: TLS clean shutdown alert reading data
[Dec 6 13:14:39] DEBUG[30015]: chan_sip.c:2905 sip_tcptls_read: SIP TCP/TLS server has shut down
The call continues to flow bi-directionally, the caller never knows there is a problem until they hit a hangup in context, i.e. h,1,Hangup(). Then Asterisk is restarted (new PID) and the caller hangs in limbo for another 5 minutes before the call times out with a fast busy. Twilio confirms they see the BYE and return an ACK at the point of the Hangup.
I was on 13.11 and updated to 15.1.3, same result. Calls longer than 120s result in TLS message in debug and Asterisk restarts.
No Google query results out there. Twilio hasn't been real helpful. Can anyone shed some light on what is happening and where I need to look next?
More logs:
[Dec 8 10:18:48] DEBUG[4993][C-00000001]: channel.c:5551 set_format: Channel SIP/twilio0-00000000 setting write format path: gsm -> ulaw
[Dec 8 10:18:48] DEBUG[4993][C-00000001]: res_rtp_asterisk.c:4017 rtp_raw_write: Difference is 2472, ms is 329
[Dec 8 10:18:48] DEBUG[4993][C-00000001]: channel.c:3192 ast_settimeout_full: Scheduling timer at (50 requested / 50 actual) timer ticks per second
– <SIP/twilio0-00000000> Playing ‘IVR/omnicare_9d_account.gsm’ (language ‘en’)
[Dec 8 10:18:48] DEBUG[4993][C-00000001]: res_rtp_asterisk.c:4928 ast_rtcp_interpret: Got RTCP report of 64 bytes from 34.203.250.7:10475
[Dec 8 10:18:53] DEBUG[4993][C-00000001]: res_rtp_asterisk.c:4928 ast_rtcp_interpret: Got RTCP report of 64 bytes from 34.203.250.7:10475
[Dec 8 10:18:55] DEBUG[4992]: iostream.c:157 iostream_read: TLS clean shutdown alert reading data
[Dec 8 10:18:55] DEBUG[4992]: chan_sip.c:2905 sip_tcptls_read: SIP TCP/TLS server has shut down
[Dec 8 10:18:58] DEBUG[4993][C-00000001]: channel.c:3192 ast_settimeout_full: Scheduling timer at (0 requested / 0 actual) timer ticks per second
[Dec 8 10:18:58] DEBUG[4993][C-00000001]: channel.c:3192 ast_settimeout_full: Scheduling timer at (0 requested / 0 actual) timer ticks per second
[Dec 8 10:18:58] DEBUG[4993][C-00000001]: channel.c:3192 ast_settimeout_full: Scheduling timer at (0 requested / 0 actual) timer ticks per second
[Dec 8 10:18:58] DEBUG[4993][C-00000001]: channel.c:5551 set_format: Channel SIP/twilio0-00000000 setting write format path: ulaw -> ulaw
[Dec 8 10:18:58] DEBUG[4993][C-00000001]: res_rtp_asterisk.c:4928 ast_rtcp_interpret: Got RTCP report of 64 bytes from 34.203.250.7:10475
[Dec 8 10:19:01] DEBUG[4914]: cdr.c:4305 ast_cdr_engine_term: CDR Engine termination request received; waiting on messages…
Asterisk uncleanly ending (0).
Executing last minute cleanups
== Destroying musiconhold processes
[Dec 8 10:19:01] DEBUG[4914]: res_musiconhold.c:1627 moh_class_destructor: Destroying MOH class ‘default’
[Dec 8 10:19:01] DEBUG[4914]: cdr.c:1289 cdr_object_finalize: Finalized CDR for SIP/twilio0-00000000 - start 1512749813.880448 answer 1512749813.881198 end 1512749941.201797 dispo ANSWERED
== Manager unregistered action DBGet
== Manager unregistered action DBPut
== Manager unregistered action DBDel
== Manager unregistered action DBDelTree
[Dec 8 10:19:01] DEBUG[4914]: asterisk.c:2157 really_quit: Asterisk ending (0).
Check your firewall logs. We've had issues with sessions being torn down by firewalls that thought the NAT entries were stale/old.
You can also try configuring Asterisk to send keep-alive packets using the option qualify=yes and nat=yes in your sip.conf entry for that user/trunk. Or inside the RTP stream with rtpkeepalive=<secs>. The best docs I could find for sip.conf are the example config on github.
I dug in the source code for the text "TLS clean shutdown alert reading data", which pointed me to some OpenSSL docs which suggest a clean/normal closure (which I'm guessing was caused by your firewall):
The TLS/SSL connection has been closed. If the protocol version is SSL 3.0 or higher, this result code is returned only if a closure alert has occurred in the protocol, i.e. if the connection has been closed cleanly. Note that in this case SSL_ERROR_ZERO_RETURN does not necessarily indicate that the underlying transport has been closed.
Everything was working fine for us from last 8 months and suddenly we started getting this error.
We configured SMPP driver properties in Oracle EM console and sending SMS using UMS integrated with Oracle SOA server.
Found a link related to this error but it doesn't help:
https://support.nowsms.com/discus/messages/485/1176.html
[2017-10-17T10:33:16.806+05:30] [WLS_SOA] [ERROR] [SDP-25700]
[oracle.sdp.messaging.driver.smpp] [tid: Workmanager: , Version: 0,
Scheduled=false, Started=false, Wait time: 0 ms\n] [userId:
OracleSystemUser] [ecid:
49d20b62a8084325:-5336381e:15f247fea2f:-8000-000000000001c322,0] [APP:
usermessagingdriver-smpp] An unexpected exception was caught.[[
oracle.sdp.messaging.driver.DriverException: Invalid Source Address[a]
at
oracle.sdpinternal.messaging.driver.smpp.SMPPDriver.send(SMPPDriver.java:1073)
at
oracle.sdpinternal.messaging.driver.smpp.SMPPDriver.send(SMPPDriver.java:3268)
at
oracle.sdpinternal.messaging.driver.smpp.SMPPDriver.send(SMPPDriver.java:2849)
at
oracle.sdpinternal.messaging.driver.smpp.SMPPDriver.send(SMPPDriver.java:2724)
at
oracle.sdpinternal.messaging.driver.smpp.SMPPManagedConnection.send(SMPPManagedConnection.java:95)
at
oracle.sdpinternal.messaging.driver.DriverConnectionImpl.send(DriverConnectionImpl.java:41)
at
oracle.sdpinternal.messaging.dispatcher.DriverDispatcherBean.onMessage(DriverDispatcherBean.java:296)
at sun.reflect.GeneratedMethodAccessor1629.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
at
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at
com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at
com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at
com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy532.onMessage(Unknown Source) at
oracle.sdpinternal.messaging.dispatcher.DriverDispatcherBean_xfokw2_MDOImpl.__WL_invoke(Unknown
Source) at
weblogic.ejb.container.internal.MDOMethodInvoker.invoke(MDOMethodInvoker.java:35)
at
oracle.sdpinternal.messaging.dispatcher.DriverDispatcherBean_xfokw2_MDOImpl.onMessage(Unknown
Source) at
oracle.sdpinternal.messaging.jmsmessagereceiver.ResourceAdapterImpl$RaMessageListener.onMessage(ResourceAdapterImpl.java:167)
at
oracle.sdpinternal.messaging.jms.QueueWorker.run(QueueWorker.java:73)
at
weblogic.work.j2ee.J2EEWorkManager$WorkWithListener.run(J2EEWorkManager.java:184)
at weblogic.work.DaemonWorkThread.run(DaemonWorkThread.java:30)
]] [2017-10-17T10:33:16.807+05:30] [WLS_SOA] [NOTIFICATION]
[SDP-26003] [oracle.sdp.messaging.driver.dispatcher] [tid:
Workmanager: , Version: 0, Scheduled=false, Started=false, Wait time:
0 ms\n] [userId: OracleSystemUser] [ecid:
49d20b62a8084325:-5336381e:15f247fea2f:-8000-000000000001c322,0] [APP:
usermessagingdriver-smpp] Dispatcher sent message with id:
eb50c4cdc0a800ab12d5fbbdbbbbd46a.
Network Logs (Using Wireshark):
We figured out and resolved this issue. Posting its resolution here. May be it can help someone other and save their precious time.
SMPP driver configured in Oracle EM console has a bug due to which Source address and Default source address are always null even if you pass it from driver configurations.
It was due to multiple Source Address at SMSC and our system was passing null
value in Source Address field. Due to this SMSC was not able
to recognize correct source address and throw "Invalid Source
Address" exception.
If single/static source address is configured at SMSC then you can
pass null values and SMSC will automatically recognize source address.
We are using streaming insert API along with google-api-java-clients batch request.
Initially everything was fine but after some time it started throwing so many 500 errors:
{"code":500,"errors":[{"domain":"global","message":"Unexpected. Please try again.","reason":"internalError"}],"message":"Unexpected. Please try again."}
Code snippet is below:
val batch = client.batch()
val request = new TableDataInsertAllRequest()
request.setRows(rows)
val insertAll = client.tabledata().insertAll(ProjectId, datasetId, tableId, request)
insertAll.queue(batch, new MyCallback(datasetId, tableId, rows, retryAttempt))
Sometimes we are getting BackEndError as well
{"code":500,"errors":[{"domain":"global","message":"Backend Error","reason":"backendError"}],"message":"Backend Error"}
NOTE: Prior to getting errors we got below error:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.7.0_06]
at java.net.SocketInputStream.read(SocketInputStream.java:150) ~[na:1.7.0_06]
at java.net.SocketInputStream.read(SocketInputStream.java:121) ~[na:1.7.0_06]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:312) ~[na:1.7.0_06]
at sun.security.ssl.InputRecord.read(InputRecord.java:350) ~[na:1.7.0_06]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) ~[na:1.7.0_06]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) ~[na:1.7.0_06]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) ~[na:1.7.0_06]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) ~[na:1.7.0_06]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) ~[na:1.7.0_06]
at java.io.BufferedInputStream.read(BufferedInputStream.java:334) ~[na:1.7.0_06]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633) ~[na:1.7.0_06]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579) ~[na:1.7.0_06]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1322) ~[na:1.7.0_06]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) ~[na:1.7.0_06]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.7.0_06]
at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[google-http-client-1.18.0-rc.jar:1.18.0-rc]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94) ~[google-http-client-1.18.0-rc.jar:1.18.0-rc]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:965) ~[google-http-client-1.18.0-rc.jar:1.18.0-rc]
at com.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:241) ~[google-api-client-1.18.0-rc.jar:1.18.0-rc]
Questions
What is the cause of this?
What should we do to fix this?
EDIT
- Project id is deft-virtue-628
- We are executing using streaming insert api and do not have job id.
We were kind of experienced same issues before. Here are some of our findings, hope them can help you a bit.
regarding to java.net.SocketTimeoutException: Read timed out: it by a big chance wasn't due to the Bigquery side, but could be your system resource (JVM or network socket or relative things) get exhausted. We originally ran our program on a virtual machine on Windows Server 2008R2 when we experienced so many socket timed out. Since we moved to a new server and ran our program on native machine, we have barely seen this exceptions. Also, this time out would sometimes cause other exceptions such as SSL connection closed during handshake and etc.
as of Bigquery 500's error: we were not be able to find a way to avoid it, since it didn't show any pattern, Bigquery doesn't totally forbid or fail your following requests. Simply do backoff won't help to avoid this error, especially if you are using multithread, it's hard to control backoff time accurately. So, what we did was put data back to queue if 500 error happened and retry, the fact was that averagely after 1-2 times of retry, it success. Although we are still waiting for advises on an optimized way to deal with this error, we kind of just keep retrying as for now. By doing this, 500's error happens, but we can still get all of our data streamed into Bigquery.
I am using ftp apache's commomns net version 3.1 .
The ftp connection gets in hung state while doing listing operation INTERMITTENTLY .
The reason i feel so seems to be ftp client is kept waiting indefnitely for server response for FTP command PASV while trying to open data connection for listing operation.
How do i need to set read timeout on control connection to avoid this situation.
I have set readtimeout on data connection using setDataTimeout().
For more refer :
http://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html#setDataTimeout(int)
1)Does setting setsoTimeout() after doing ftp connect() operation helps avoiding this situation on control connection?
For more refer :
http://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/SocketClient.html#setSoTimeout(int)
2)If so,what is the optimum timeout value i need to set for setsotimeout() ?
Please find stack trace below:
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:140)
at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
at java.io.InputStreamReader.read(InputStreamReader.java:188)
at java.io.BufferedReader.fill(BufferedReader.java:147)
at java.io.BufferedReader.read(BufferedReader.java:168)
at org.apache.commons.net.io.CRLFLineReader.readLine(CRLFLineReader.java:58
)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:310)
at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:290)
at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:479)
at org.apache.commons.net.ftp.FTPClient.openDataConnection(FTPClient.java:7
69)
at org.apache.commons.net.ftp.FTPClient.openDataConnection(FTPClient.java:6
57)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:
3097)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:
3072)
at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:
2972
Any help on this will be appreciated:)
Thanks.