BigQuery Streaming with C# Client Library - google-bigquery

Question

I've an Oracle table that needed to migrate to BigQuery. I wrote a simple console application with C# and started to streaming inserts. But sometimes aplication throws an error which is below. and my code is pretty simple that which is also below. Are there anyone have an idea what may cause this error? Thanks in advance.
Unhandled Exception: System.Net.Http.HttpRequestException: An error occurred whi
le sending the request. ---> System.Net.WebException: The underlying connection
was closed: A connection that was expected to be kept alive was closed by the se
rver. ---> System.IO.IOException: Unable to read data from the transport connect
ion: An existing connection was forcibly closed by the remote host. ---> System.
Net.Sockets.SocketException: An existing connection was forcibly closed by the r
emote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at Google.Apis.Http.ConfigurableMessageHandler.<SendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
at Google.Apis.Requests.ClientServiceRequest`1.<ExecuteUnparsedAsync>d__33.Mo
veNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Google.Apis.Requests.ClientServiceRequest`1.Execute()
at Google.Cloud.BigQuery.V2.BigQueryClientImpl.InsertRows(TableReference tabl
eReference, IEnumerable`1 rows, InsertOptions options)
at Google.Cloud.BigQuery.V2.BigQueryClient.InsertRows(String datasetId, Strin
g tableId, BigQueryInsertRow[] rows)
at BigQueryStreamer.Program.UploadJsonStreamingSync(String datasetId, String
tableId, BigQueryClient client, BigQueryInsertRow[] _rows) in C:\Projects\BigQue
ryStreamer\BigQueryStreamer\Program.cs:line 330
at BigQueryStreamer.Program.Main(String[] args) in C:\Projects\BigQueryStream
er\BigQueryStreamer\Program.cs:line 185
my code block is:
List<BigQueryInsertRow> _list = new List<BigQueryInsertRow>();
while (oracleReader.Read())
{
BigQueryInsertRow bigQueryInsertRow = new BigQueryInsertRow();
Dictionary<string, object> dictionary = new Dictionary<string, object>();
for (int ordinal = 0; ordinal < oracleReader.FieldCount; ++ordinal)
{
typeof(Decimal).ToString();
string str = oracleReader.GetValue(ordinal).GetType().ToString();
object obj = (str == "System.Decimal" || str== "System.Double" || str == "System.Float") ?
(object)double.Parse(oracleReader.GetValue(ordinal).ToString()) :
(str == "System.DBNull" ? (object)null : oracleReader.GetValue(ordinal));
dictionary.Add(oracleReader.GetName(ordinal), obj);
}
bigQueryInsertRow.Add(dictionary);
_list.Add(bigQueryInsertRow);
}
List<BigQueryInsertRow> _SendList = new List<BigQueryInsertRow>();
//To Stream in 1000 rows at a time, I set _batcSize to 1000 in application configuration
for (int i = 0; i < _list.Count; i++)
{
_SendList.Add(_list[i]);
if (_SendList.Count == _batchSize)
{
System.Threading.Thread.Sleep(150);
UploadJsonStreamingSync(_datasetid, _target, _client, _SendList.ToArray());
Console.WriteLine("Offset: " + ((ubound + 1) * _batchSize).ToString());
ubound++;
_SendList.Clear();
}
}
if (_SendList.Count > 0)
{
System.Threading.Thread.Sleep(150);
UploadJsonStreamingSync(_datasetid, _target, _client, _SendList.ToArray());
Console.WriteLine("Offset: " + (_SendList.Count).ToString());
ubound++;
_SendList.Clear();
}
_list.Clear();
_list = null;
//Streaming Insert Function
public static void UploadJsonStreamingSync(string datasetId, string tableId, BigQueryClient client, BigQueryInsertRow[] _rows)
{
client.InsertRows(datasetId, tableId, _rows);
}

Answer 1

When facing infrequent socket exceptions/network issues as Graham pointed out you need to write logic to handle it.
There are libraries which you can use.
I used: https://github.com/App-vNext/Polly
Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner.