We have a .Net Core API accessing Azure SQL (Gen5, 4 vCores)
Since quite some time,
the API keeps throwing below exception for a specific READ operation
Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout
Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The READ operation has code to read rows of data and convert an XML column into a specific output format.
Most of the read operation extracts hardly 4-5 rows # a time.
The tables involved in the query have ~ 500,000 rows
We are clueless on Root Cause of this issue.
Any hints on where to start looking # for root cause?
Any pointer would be highly appreciated.
NOTE : Connection string has following settings, apart from others
MultipleActiveResultSets=True;Connection Timeout=60
Overall code looks something like this.
HINT: The above timeout exception comes # ConvertHistory, when the 2nd table is being read.
HttpGet]
public async Task<IEnumerable<SalesOrders>> GetNewSalesOrders()
{
var SalesOrders = await _db.SalesOrders.Where(o => o.IsImported == false).OrderBy(o => o.ID).ToListAsync();
var orders = new List<SalesOrder>();
foreach (var so in SalesOrders)
{
var order = ConvertSalesOrder(so);
orders.Add(order);
}
return orders;
}
private SalesOrder ConvertSalesOrder(SalesOrder o)
{
var newOrder = new SalesOrder();
var oXml = o.XMLContent.LoadFromXMLString<SalesOrder>();
...
newOrder.BusinessUnit = oXml.BusinessUnit;
var history = ConvertHistory(o.ID);
newOrder.history = history;
return newOrder;
}
private SalesOrderHistory[] ConvertHistory(string id)
{
var history = _db.OrderHistory.Where(o => o.ID == id);
...
}
Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
From Microsoft Document,
You will get this error in both conditions Connection timeout or Query or command timeout. first identify it from call stack of the error messages.
If you found it as a connection issue you can either Increase connection timeout parameter. if you are still getting same error, it is causing due to a network issue.
from information that you provided It is Query or command timeout error to work around this error you can set CommandTimeout for query or command
command.CommandTimeout = 10;
The default timeout value is 30 seconds, the query will continue to run until it is finished if the time-out value is set to 0 (no time limit).
For more information refer Troubleshoot query time-out errors provided by Microsoft.
The application that im working on runs a sequence of queries on AWS Redshift. Some of the queries take longer to execute due to the data volume.
The queries seem to finish on Redshift when i check the execution details on the server. However, the java application seems to hang indefinitely without throwing any exception or even terminating.
Here's the code that executes the query.
private void execSQLStrings(String[] queries, String dataset, String dbType) throws Exception {
Connection conn = null;
if (dbType.equals("redshift")) {
conn=getRedshiftConnection();
} else if (dbType.equals("rds")){
conn=getMySQLConnection();
}
Statement stmt=conn.createStatement();
String qry=null;
debug("Query Length: " + queries.length);
for (int ii=0;ii<queries.length;++ii) {
qry=queries[ii];
if (dataset != null) {
qry=qry.replaceAll("DATASET",dataset);
}
debug(qry);
stmt.execute(qry);
}
stmt.close();
conn.close();
}
I cant post the query that im running at the moment but it has multiple table joins and group by conditions and its an 800m row table. The query takes about 7~8 mins to complete on the server.
You need to update the DSN Timeout and/ or KeepAlive settings to make sure that your connections stay alive.
Refer: http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html
I am executing some long running quires using the big-query java client.
I construct a big-query job and execute like this
val queryRequest = new QueryRequest().setQuery(query)
val queryJob = client.jobs().query(ProjectId, queryRequest)
queryJob.execute()
The problem I am facing is the for the same query, the client returns before the job is complete i.e. the number of rows in result is zero.
I tried printing the response and it shows
{"jobComplete":false,"jobReference":{"jobId":"job_bTLRGrw5_xR26i9Li3a9EQvuA6c","projectId":"analytics-production"},"kind":"bigquery#queryResponse"}
From that I can see that the job is not complete. The why did the client return before the job is complete ?
While building the client, I use the HttpRequestInitializer and in the initialize method I provide the timeout parameters.
override def initialize(request: HttpRequest): Unit = {
request.setConnectTimeout(...)
request.setReadTimeout(...)
}
Tried giving high values for timeout like 240 seconds etc..but no luck. The behavior is still the same. It fails intermitently.
Make sure you set the timeout on the Bigquery request body, and not the HTTP object.
val queryRequest = new QueryRequest().setQuery(query).setTimeoutMs(10000) //10 seconds
The param is timeoutMs. This is documented here: https://cloud.google.com/bigquery/docs/reference/v2/jobs/query
Please also read the docs regarding this field: How long to wait for the query to complete, in milliseconds, before the request times out and returns. Note that this is only a timeout for the request, not the query. If the query takes longer to run than the timeout value, the call returns without any results and with the 'jobComplete' flag set to false. You can call GetQueryResults() to wait for the query to complete and read the results. The default value is 10000 milliseconds (10 seconds).
More about Synchronous queries here
https://cloud.google.com/bigquery/querying-data#syncqueries
I started to use dapper.net a while ago for performance reasons and that i really like the named parameters feature compared to just run "ExecuteQuery" in LINQ To SQL.
It works great for most queries but i get some really weird timeouts from time to time. The strangest thing is that this timeout only happens when the SQL is executed via dapper. If i take the executed query copied from the profiler and just run it in Management Studio its fast and works perfect. And it's not just a temporary issues. The query consistently timeout via dapper and consistently works fine in Management Studio.
exec sp_executesql N'SELECT Item.Name,dbo.PlatformTextAndUrlName(Item.ItemId) As PlatformString,dbo.MetaString(Item.ItemId) As MetaTagString, Item.StartPageRank,Item.ItemRecentViewCount
NAME_SRCH.RANK as NameRank,
DESC_SRCH.RANK As DescRank,
ALIAS_SRCH.RANK as AliasRank,
Item.itemrecentviewcount,
(COALESCE(ALIAS_SRCH.RANK, 0)) + (COALESCE(NAME_SRCH.RANK, 0)) + (COALESCE(DESC_SRCH.RANK, 0) / 20) + Item.itemrecentviewcount / 4 + ((CASE WHEN altrank > 60 THEN 60 ELSE altrank END) * 4) As SuperRank
FROM dbo.Item
INNER JOIN dbo.License on Item.LicenseId = License.LicenseId
LEFT JOIN dbo.Icon on Item.ItemId = Icon.ItemId
LEFT OUTER JOIN FREETEXTTABLE(dbo.Item, name, #SearchString) NAME_SRCH ON
Item.ItemId = NAME_SRCH.[KEY]
LEFT OUTER JOIN FREETEXTTABLE(dbo.Item, namealiases, #SearchString) ALIAS_SRCH ON
Item.ItemId = ALIAS_SRCH.[KEY]
INNER JOIN FREETEXTTABLE(dbo.Item, *, #SearchString) DESC_SRCH ON
Item.ItemId = DESC_SRCH.[KEY]
ORDER BY SuperRank DESC OFFSET #Skip ROWS FETCH NEXT #Count ROWS ONLY',N'#Count int,#SearchString nvarchar(4000),#Skip int',#Count=12,#SearchString=N'box,com',#Skip=0
That is the query that i copy pasted from SQL Profiler. I execute it like this in my code.
using (var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["Conn"].ToString())) {
connection.Open();
var items = connection.Query<MainItemForList>(query, new { SearchString = searchString, PlatformId = platformId, _LicenseFilter = licenseFilter, Skip = skip, Count = count }, buffered: false);
return items.ToList();
}
I have no idea where to start here. I suppose there must be something that is going on with dapper since it works fine when i just execute the code.
As you can see in this screenshot. This is the same query executed via code first and then via Management Studio.
I can also add that this only (i think) happens when i have two or more word or when i have a "stop" char in the search string. So it may have something todo with the full text search but i cant figure out how to debug it since it works perfectly from Management Studio.
And to make matters even worse, it works fine on my localhost with a almost identical database both from code and from Management Studio.
Dapper is nothing more than a utility wrapper over ado.net; it does not change how ado.net operates. It sounds to me that the problem here is "works in ssms, fails in ado.net". This is not unique: it is pretty common to find this occasionally. Likely candidates:
"set" option: these have different defaults in ado.net - and can impact performance especially if you have things like calculated+persisted+indexed columns - if the "set" options aren't compatible it can decide it can't use the stored value, hence not the index - and instead table-scan and recompute. There are other similar scenarios.
system load / transaction isolation-level / blocking; running something in ssms does not reproduce the entire system load at that moment in time
cached query plans: sometimes a duff plan gets cached and used; running from ssms will usually force a new plan - which will naturally be tuned for the parameters you are using in your test. Update all your index stats etc, and consider adding the "optimise for" query hint
In ADO is the default value for CommandTimeout 30 Seconds, in Management Studio infinity. Adjust the command timeout for calling Query<>, see below.
var param = new { SearchString = searchString, PlatformId = platformId, _LicenseFilter = licenseFilter, Skip = skip, Count = count };
var queryTimeoutInSeconds = 120;
using (var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["Conn"].ToString()))
{
connection.Open();
var items = connection.Query<MainItemForList>(query, param, commandTimeout: queryTimeoutInSeconds, buffered: false);
return items.ToList();
}
See also
SqlCommand.CommandTimeout Property on MSDN
For Dapper , default timeout is 30 seconds But we can increase the timeout in this way. Here we are incresing the timeout 240 seconds (4 minutes).
public DataTable GetReport(bool isDepot, string fetchById)
{
int? queryTimeoutInSeconds = 240;
using (IDbConnection _connection = DapperConnection)
{
var parameters = new DynamicParameters();
parameters.Add("#IsDepot", isDepot);
parameters.Add("#FetchById", fetchById);
var res = this.ExecuteSP<dynamic>(SPNames.SSP_GetSEPReport, parameters, queryTimeoutInSeconds);
return ToDataTable(res);
}
}
In the repository layer , we can call our custom ExecuteSP method for the Stored Procedures with additional parameters "queryTimeoutInSeconds".
And below is the "ExecuteSP" method for dapper:-
public virtual IEnumerable<TEntity> ExecuteSP<TEntity>(string spName, object parameters = null, int? parameterForTimeout = null)
{
using (IDbConnection _connection = DapperConnection)
{
_connection.Open();
return _connection.Query<TEntity>(spName, parameters, commandTimeout: parameterForTimeout, commandType: CommandType.StoredProcedure);
}
}
Could be a matter of setting the command timeout in Dapper. Here's an example of how to adjust the command timeout in Dapper:
Setting Command Timeout in Dapper
I have some sequence in database, represented by range. It has three fields: beginId, nextId and endId.
The task is to acquire the nextId from this range, and to ensure that it is unique. The code could be ran in highly parallelized environment, with many threads, on many machines.
What I need to do:
lock(database)
{
var seq = GetSequence()
var acquiredId = seq.NextId;
seq.NextId++
Save(seq)
}
So I use this code:
using (ISession session = GetSessionFactory().OpenSession())
using (ITransaction transaction = session.BeginTransaction(IsolationLevel.RepeatableRead))
{
var sequence = session.CreateCriteria<Sequence>().Single(); // This line is simplified
var allocatedId = sequence.NextId;
sequence.NextId++;
session.SaveOrUpdate(sequence);
transaction.Commit();
return allocatedId;
}
But for some reason when I run this code in multi-threading for testing, I received the same id assigned several times. I'm using the transaction with RepeatableRead lock, but that doesn't help.
P.S. Id doesn't mean Id of the table - it just the name agreement we use.
I've used .Lock().Upgrade when the data is taken from DB, and everything worked. Thanks to hazzik for the link.
Now my code have this:
session.QueryOver<Sequence>().Lock().Upgrade.DoSomeFiltering().Single()