Streaming insert API throws 500 errors while using with google-api-java-clients batch request - google-bigquery

Question

We are using streaming insert API along with google-api-java-clients batch request.
Initially everything was fine but after some time it started throwing so many 500 errors:
{"code":500,"errors":[{"domain":"global","message":"Unexpected. Please try again.","reason":"internalError"}],"message":"Unexpected. Please try again."}
Code snippet is below:
val batch = client.batch()
val request = new TableDataInsertAllRequest()
request.setRows(rows)
val insertAll = client.tabledata().insertAll(ProjectId, datasetId, tableId, request)
insertAll.queue(batch, new MyCallback(datasetId, tableId, rows, retryAttempt))
Sometimes we are getting BackEndError as well
{"code":500,"errors":[{"domain":"global","message":"Backend Error","reason":"backendError"}],"message":"Backend Error"}
NOTE: Prior to getting errors we got below error:
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.7.0_06]
at java.net.SocketInputStream.read(SocketInputStream.java:150) ~[na:1.7.0_06]
at java.net.SocketInputStream.read(SocketInputStream.java:121) ~[na:1.7.0_06]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:312) ~[na:1.7.0_06]
at sun.security.ssl.InputRecord.read(InputRecord.java:350) ~[na:1.7.0_06]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) ~[na:1.7.0_06]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) ~[na:1.7.0_06]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) ~[na:1.7.0_06]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) ~[na:1.7.0_06]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) ~[na:1.7.0_06]
at java.io.BufferedInputStream.read(BufferedInputStream.java:334) ~[na:1.7.0_06]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633) ~[na:1.7.0_06]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579) ~[na:1.7.0_06]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1322) ~[na:1.7.0_06]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) ~[na:1.7.0_06]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.7.0_06]
at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[google-http-client-1.18.0-rc.jar:1.18.0-rc]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94) ~[google-http-client-1.18.0-rc.jar:1.18.0-rc]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:965) ~[google-http-client-1.18.0-rc.jar:1.18.0-rc]
at com.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:241) ~[google-api-client-1.18.0-rc.jar:1.18.0-rc]
Questions
What is the cause of this?
What should we do to fix this?
EDIT
- Project id is deft-virtue-628
- We are executing using streaming insert api and do not have job id.

Answer 1

We were kind of experienced same issues before. Here are some of our findings, hope them can help you a bit.
regarding to java.net.SocketTimeoutException: Read timed out: it by a big chance wasn't due to the Bigquery side, but could be your system resource (JVM or network socket or relative things) get exhausted. We originally ran our program on a virtual machine on Windows Server 2008R2 when we experienced so many socket timed out. Since we moved to a new server and ran our program on native machine, we have barely seen this exceptions. Also, this time out would sometimes cause other exceptions such as SSL connection closed during handshake and etc.
as of Bigquery 500's error: we were not be able to find a way to avoid it, since it didn't show any pattern, Bigquery doesn't totally forbid or fail your following requests. Simply do backoff won't help to avoid this error, especially if you are using multithread, it's hard to control backoff time accurately. So, what we did was put data back to queue if 500 error happened and retry, the fact was that averagely after 1-2 times of retry, it success. Although we are still waiting for advises on an optimized way to deal with this error, we kind of just keep retrying as for now. By doing this, 500's error happens, but we can still get all of our data streamed into Bigquery.