When doing a Full Process on a tabular model to an Azure Analysis Service model I get the following error after 10 minutes into the processing:
Failed to save modifications to the server. Error returned: 'Microsoft SQL: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.. The exception was raised by the IDbCommand interface.
Technical Details:
RootActivityId: cd0cfc78-416a-4039-a79f-ed7fe9836906
Date (UTC): 2/27/2018 1:25:58 PM
The command has been canceled.. The exception was raised by the IDbCommand interface.
The command has been canceled.. The exception was raised by the IDbCommand interface.
The command has been canceled.. The exception was raised by the IDbCommand interface.
The command has been canceled.. The exception was raised by the IDbCommand interface.
The data source for the model is Azure Data Warehouse and SSAS authenticates to it via SQL authentication. When the Timeout occurs some partitions have retrieved all their rows but the others are still processing. The model contains 11 tables each with a single partition.
I get the error both when processing with Visual Studio 2015 and SSMS 2017. I can't see any SSAS server properties with a 10 minute (600 second) timeout. Individual table processing can be done without the timeout issue since individually they all complete in under 10 minutes.
I've tried setting the timeout property in the dataSources.connectionDetails object in my Tabular Model Scripting Language json file (i.e. Model.bim). But editing it drops the authentication credentials, and then resetting the credentials drops the timeout property. So I don't know if that property is even relevant to the timeout error issue.
An example of a partition query expression I'm using:
let
Source = #"SQL/resourcename database windows net;DatabaseName",
MyQuery =
Value.NativeQuery(
Source,
"SELECT * FROM [dbo].[MyTable]"
)
in
MyQuery
So thanks to GregGalloway's prompting I've figured out the timeout can be set on a per Partition basis using the Power Query M language.
So the data access parts of my TMSL object now look like so...
The model.dataSource is as so:
"dataSources": [
{
"type": "structured",
"name": "MySource",
"connectionDetails": {
"protocol": "tds",
"address": {
"server": "serverName.database.windows.net",
"database": "databaseName"
},
"authentication": null,
"query": null
},
"options": {},
"credential": {
"AuthenticationKind": "UsernamePassword",
"Username": "dbUsername",
"EncryptConnection": true
}
}
]
And the individual Partition queries are as so (note the CommandTimeout parameter):
let
Source = Sql.Database("serverName.database.windows.net","databaseName",[CommandTimeout=#duration(0, 2, 0, 0)]),
MyQuery =
Value.NativeQuery(
Source,
"SELECT * FROM [dbo].[MyTable]"
)
in
MyQuery
So now I'm explicitly setting a timeout of 2 hours for the Partition query.
Data Source -> Options: increasing Command timeout (default 600 secs) will also do the trick:
Related
We have a .Net Core API accessing Azure SQL (Gen5, 4 vCores)
Since quite some time,
the API keeps throwing below exception for a specific READ operation
Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout
Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
The READ operation has code to read rows of data and convert an XML column into a specific output format.
Most of the read operation extracts hardly 4-5 rows # a time.
The tables involved in the query have ~ 500,000 rows
We are clueless on Root Cause of this issue.
Any hints on where to start looking # for root cause?
Any pointer would be highly appreciated.
NOTE : Connection string has following settings, apart from others
MultipleActiveResultSets=True;Connection Timeout=60
Overall code looks something like this.
HINT: The above timeout exception comes # ConvertHistory, when the 2nd table is being read.
HttpGet]
public async Task<IEnumerable<SalesOrders>> GetNewSalesOrders()
{
var SalesOrders = await _db.SalesOrders.Where(o => o.IsImported == false).OrderBy(o => o.ID).ToListAsync();
var orders = new List<SalesOrder>();
foreach (var so in SalesOrders)
{
var order = ConvertSalesOrder(so);
orders.Add(order);
}
return orders;
}
private SalesOrder ConvertSalesOrder(SalesOrder o)
{
var newOrder = new SalesOrder();
var oXml = o.XMLContent.LoadFromXMLString<SalesOrder>();
...
newOrder.BusinessUnit = oXml.BusinessUnit;
var history = ConvertHistory(o.ID);
newOrder.history = history;
return newOrder;
}
private SalesOrderHistory[] ConvertHistory(string id)
{
var history = _db.OrderHistory.Where(o => o.ID == id);
...
}
Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
From Microsoft Document,
You will get this error in both conditions Connection timeout or Query or command timeout. first identify it from call stack of the error messages.
If you found it as a connection issue you can either Increase connection timeout parameter. if you are still getting same error, it is causing due to a network issue.
from information that you provided It is Query or command timeout error to work around this error you can set CommandTimeout for query or command
command.CommandTimeout = 10;
The default timeout value is 30 seconds, the query will continue to run until it is finished if the time-out value is set to 0 (no time limit).
For more information refer Troubleshoot query time-out errors provided by Microsoft.
Description:
I have an application that connects to an Azure Serverless Database. The database can be in a paused state and in an online state. The database auto-pauses when there has been no activity for one hour. This means that when my application tries to open a connection to the database when it is paused, the connection times out and gives a timeout exception.
Azure states in their documentation that:
If a serverless database is paused, then the first login will resume the database and return an error stating that the database is unavailable with error code 40613. Once the database is resumed, the login must be retried to establish connectivity. Database clients with connection retry logic should not need to be modified. source
I am able to get this error code 40613 returned when I try to connect to the database via SQL Management Studio. But when I try to open a connection to the database from my application I only get a timeout exception, hence I don't know whether or not the database is not available or if the database is in fact resuming.
Code example:
public IDbConnection GetConnection()
{
var connection = new SqlConnection(_connectionString);
try
{
connection.Open();
return connection;
}
catch (SqlException e)
{
if (e.Number == 40613)
{
//Database is resuming
}
}
finally
{
connection.Close();
}
}
Exception example:
When I run my application and the database is in paused state I get this exception:
Snippet of exception in Visual Studio
Does anyone know why I don't get the error code 40613 that Azure states in their documentation?
Indeed you may get timeout errors when the Azure database is unavailable. In fact you may get the following errors:
HTTP error GatewayTimeout : The gateway did not receive a response
from ‘Microsoft.Sql’ within the specified time period
HTTP error ServiceUnavailable : The request timed out
SQLException : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
You may get also error 40613 but you can capture some transient errors like below too:
•Database on server is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of
•Database on server is not currently available. Please retry the connection later. If the problem persists, contact customer support, and provide them the session tracing ID of . (Microsoft SQL Server, Error: 40613)
•An existing connection was forcibly closed by the remote host.
•System.Data.Entity.Core.EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details. ---> System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
•An connection attempt to a secondary database failed because the database is in the process of reconfguration and it is busy applying new pages while in the middle of an active transation on the primary database.
Because of those errors and more explained here, it is necessary to create a retry logic on applications that connect to Azure SQL Database.
public void HandleTransients()
{
var connStr = "some database";
var _policy = RetryPolicy.Create < SqlAzureTransientErrorDetectionStrategy(
retryCount: 3,
retryInterval: TimeSpan.FromSeconds(5));
using (var conn = new ReliableSqlConnection(connStr, _policy))
{
// Do SQL stuff here.
}
}
More about how to create a retry logic here.
I'm streaming data to BigQuery for the past year or so from a service in Azure written in c# and recently started to get increasing amount of the following errors (most of the requests succeed):
Message: [GoogleApiException: Google.Apis.Requests.RequestError An
internal error occurred and the request could not be completed. [500]
Errors [
Message[An internal error occurred and the request could not be completed.] Location[ - ] Reason[internalError] Domain[global] ] ]
This is the code I'm using in my service:
public async Task<TableDataInsertAllResponse> Update(List<TableDataInsertAllRequest.RowsData> rows, string tableSuffix)
{
var request = new TableDataInsertAllRequest {Rows = rows, TemplateSuffix = tableSuffix};
var insertRequest = mBigqueryService.Tabledata.InsertAll(request, ProjectId, mDatasetId, mTableId);
return await insertRequest.ExecuteAsync();
}
Just like any other cloud service, BigQuery doesn't offer a 100% uptime SLA (it's actually 99.9%), so it's not uncommon to encounter transient errors like these. We also receive them frequently in our applications.
You need to build exponential backoff-and-retry logic into your application(s) to handle such errors. A good way of doing this is to use a queue to stream your data to BigQuery. This is what we do and it works very well for us.
Some more info:
https://cloud.google.com/bigquery/troubleshooting-errors
https://cloud.google.com/bigquery/loading-data-post-request#exp-backoff
https://cloud.google.com/bigquery/streaming-data-into-bigquery
https://cloud.google.com/bigquery/sla
I have relatively large table with 7,517,717 records in my Azure SQL DB (which is on Azure V12 server, and it is on P4 pricing tier).
When I created table I forgot to define the MaxLength for a property, so now it is of type nvarchar(max).
I wanted to change this so I created migration (using FluentAPI) which looks like this:
public partial class FeedTitleMaxLength : DbMigration
{
public override void Up()
{
AlterColumn("dbo.Feed", "Title", c => c.String(maxLength: 100));
}
public override void Down()
{
AlterColumn("dbo.Feed", "Title", c => c.String());
}
}
I've run Update-Database command like this:
Update-Database -Verbose -ConnectionString "Data Source=serverUrl,1433;Database=dbName;User ID=myusername;Password=mypassword;Trusted_Connection=False;Encrypt=True;Connection Timeout=3000;MultipleActiveResultSets=True;" -ConnectionProviderName "System.Data.SqlClient"
My connection string in web.config looks the same. And I've also defined timeout like this:
this.Database.CommandTimeout = 3000;
But no matter what I do or try I get this error after 30/40 seconds:
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. This failure occurred while attempting to connect to the routing destination. The duration spent while attempting to connect to the original server was - [Pre-Login] initialization=97; handshake=110; [Login] initialization=0; authentication=0; [Post-Login] complete=32;
Is there any way I can run this migration? Or some workaround? Can I, for example, create migration but run the AlterColumn command in SQL and then Update-Database command. Would this mess things or not?
I'm making a bulk copy onto a file of a select in my database.
DECLARE #cmd varchar(1000)
DECLARE #sql varchar(8000)
SET #cmd='"select * from [MyDB].[dbo].MyTable"'
SELECT #sql = 'bcp '+#cmd+' queryout C:\myfile.txt -c -t -T -S MyServer -U user -P password';
exec xp_cmdshell #sql;
If I change the parameters and I execute the same command on the database test on my machine it works, but on the database server I get this error:
Msg 121, Level 20, State 0, Line 0
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.)
I check server name, user, password, table name and they are correct, so I cannot understand what I'm doing wrong.
Can someone help me with this issue?
thanks
Increase timeout seconds in connection string.
private static void OpenSqlConnection() {
string connectionString = GetConnectionString();
using(SqlConnection connection = new SqlConnection(connectionString)) {
connection.Open();
Console.WriteLine("State: {0}", connection.State);
Console.WriteLine("ConnectionTimeout: {0}",
connection.ConnectionTimeout);
}
}
static private string GetConnectionString() {
// To avoid storing the connection string in your code,
// you can retrieve it from a configuration file, using the
// System.Configuration.ConfigurationSettings.AppSettings property
return "Data Source=(local);Initial Catalog=AdventureWorks;" + "Integrated Security=SSPI;Connection Timeout=30";
}
I had the same error. A third-party tool was connecting to the database to extract data for import into a business intelligence system. Each extract/import would run for about 1 hour and there would be something like 10 to 15 separate queries in the process. Fortunately we had logging in place - so we knew at what time each of those queries was starting and ending. The extract/import process was saying that it was completing successfully after 30 minutes - rather than taking about 1 hour. I was able to isolate the query during which the process was failing. When I ran that query in SSMS I got the same error you've given in the question.
However, when I ran that query in another environment, I got an error that a subquery cannot return more than one row.
Sure enough, when I commented out my subqueries in the prod environment, the query ran without any error.
So ultimately, the root cause for me was a subquery returning more than one row - and the issue only appeared because for some reason, "bad data" got into the database - ie. that particular subquery should never have found a scenario where there is more than one row, and hence too, the error just began to appear one day. (For other people who are experiencing this error intermittently - it could be because it is only some of your queries - or one of your queries - that is failing).