I started to use dapper.net a while ago for performance reasons and that i really like the named parameters feature compared to just run "ExecuteQuery" in LINQ To SQL.
It works great for most queries but i get some really weird timeouts from time to time. The strangest thing is that this timeout only happens when the SQL is executed via dapper. If i take the executed query copied from the profiler and just run it in Management Studio its fast and works perfect. And it's not just a temporary issues. The query consistently timeout via dapper and consistently works fine in Management Studio.
exec sp_executesql N'SELECT Item.Name,dbo.PlatformTextAndUrlName(Item.ItemId) As PlatformString,dbo.MetaString(Item.ItemId) As MetaTagString, Item.StartPageRank,Item.ItemRecentViewCount
NAME_SRCH.RANK as NameRank,
DESC_SRCH.RANK As DescRank,
ALIAS_SRCH.RANK as AliasRank,
Item.itemrecentviewcount,
(COALESCE(ALIAS_SRCH.RANK, 0)) + (COALESCE(NAME_SRCH.RANK, 0)) + (COALESCE(DESC_SRCH.RANK, 0) / 20) + Item.itemrecentviewcount / 4 + ((CASE WHEN altrank > 60 THEN 60 ELSE altrank END) * 4) As SuperRank
FROM dbo.Item
INNER JOIN dbo.License on Item.LicenseId = License.LicenseId
LEFT JOIN dbo.Icon on Item.ItemId = Icon.ItemId
LEFT OUTER JOIN FREETEXTTABLE(dbo.Item, name, #SearchString) NAME_SRCH ON
Item.ItemId = NAME_SRCH.[KEY]
LEFT OUTER JOIN FREETEXTTABLE(dbo.Item, namealiases, #SearchString) ALIAS_SRCH ON
Item.ItemId = ALIAS_SRCH.[KEY]
INNER JOIN FREETEXTTABLE(dbo.Item, *, #SearchString) DESC_SRCH ON
Item.ItemId = DESC_SRCH.[KEY]
ORDER BY SuperRank DESC OFFSET #Skip ROWS FETCH NEXT #Count ROWS ONLY',N'#Count int,#SearchString nvarchar(4000),#Skip int',#Count=12,#SearchString=N'box,com',#Skip=0
That is the query that i copy pasted from SQL Profiler. I execute it like this in my code.
using (var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["Conn"].ToString())) {
connection.Open();
var items = connection.Query<MainItemForList>(query, new { SearchString = searchString, PlatformId = platformId, _LicenseFilter = licenseFilter, Skip = skip, Count = count }, buffered: false);
return items.ToList();
}
I have no idea where to start here. I suppose there must be something that is going on with dapper since it works fine when i just execute the code.
As you can see in this screenshot. This is the same query executed via code first and then via Management Studio.
I can also add that this only (i think) happens when i have two or more word or when i have a "stop" char in the search string. So it may have something todo with the full text search but i cant figure out how to debug it since it works perfectly from Management Studio.
And to make matters even worse, it works fine on my localhost with a almost identical database both from code and from Management Studio.
Dapper is nothing more than a utility wrapper over ado.net; it does not change how ado.net operates. It sounds to me that the problem here is "works in ssms, fails in ado.net". This is not unique: it is pretty common to find this occasionally. Likely candidates:
"set" option: these have different defaults in ado.net - and can impact performance especially if you have things like calculated+persisted+indexed columns - if the "set" options aren't compatible it can decide it can't use the stored value, hence not the index - and instead table-scan and recompute. There are other similar scenarios.
system load / transaction isolation-level / blocking; running something in ssms does not reproduce the entire system load at that moment in time
cached query plans: sometimes a duff plan gets cached and used; running from ssms will usually force a new plan - which will naturally be tuned for the parameters you are using in your test. Update all your index stats etc, and consider adding the "optimise for" query hint
In ADO is the default value for CommandTimeout 30 Seconds, in Management Studio infinity. Adjust the command timeout for calling Query<>, see below.
var param = new { SearchString = searchString, PlatformId = platformId, _LicenseFilter = licenseFilter, Skip = skip, Count = count };
var queryTimeoutInSeconds = 120;
using (var connection = new SqlConnection(ConfigurationManager.ConnectionStrings["Conn"].ToString()))
{
connection.Open();
var items = connection.Query<MainItemForList>(query, param, commandTimeout: queryTimeoutInSeconds, buffered: false);
return items.ToList();
}
See also
SqlCommand.CommandTimeout Property on MSDN
For Dapper , default timeout is 30 seconds But we can increase the timeout in this way. Here we are incresing the timeout 240 seconds (4 minutes).
public DataTable GetReport(bool isDepot, string fetchById)
{
int? queryTimeoutInSeconds = 240;
using (IDbConnection _connection = DapperConnection)
{
var parameters = new DynamicParameters();
parameters.Add("#IsDepot", isDepot);
parameters.Add("#FetchById", fetchById);
var res = this.ExecuteSP<dynamic>(SPNames.SSP_GetSEPReport, parameters, queryTimeoutInSeconds);
return ToDataTable(res);
}
}
In the repository layer , we can call our custom ExecuteSP method for the Stored Procedures with additional parameters "queryTimeoutInSeconds".
And below is the "ExecuteSP" method for dapper:-
public virtual IEnumerable<TEntity> ExecuteSP<TEntity>(string spName, object parameters = null, int? parameterForTimeout = null)
{
using (IDbConnection _connection = DapperConnection)
{
_connection.Open();
return _connection.Query<TEntity>(spName, parameters, commandTimeout: parameterForTimeout, commandType: CommandType.StoredProcedure);
}
}
Could be a matter of setting the command timeout in Dapper. Here's an example of how to adjust the command timeout in Dapper:
Setting Command Timeout in Dapper
Related
I am trying to print the query statistics upon execution of the given query. Particularly I am interested in execution time on server in milliseconds property. Below is my code for reference
void Main()
{
var documentStore = DocumentStoreHolder.Store;
Load_Stats(documentStore);
}
// Define other methods and classes here
public static void Load_Stats(IDocumentStore documentStore)
{
using (var session = documentStore.OpenSession())
{
RavenQueryStatistics stats;
IRavenQueryable<Order> recentOrdersQuery = from order in session.Query<Order>().Statistics(out stats) where order.Company=="companies/1" select order;
List<Order> recentOrders = recentOrdersQuery.Take(3).ToList();
Console.WriteLine("Index used was: " + stats.IndexName);
Console.WriteLine($"Other stats : 1. Execution time on the server : {stats.DurationMilliseconds} 2.Total number of results {stats.TotalResults} 3. The last document ETag {stats.ResultEtag} 4. The timestamp of last document indexed by the index {stats.IndexTimestamp}");
}
But upon repeated execution of this query I get time taken to run query on server in milliseconds as -1. I am failing to understand why it is happening so. Should I assign the result to a long variable or is it allowed to print the result as such (stats.DurationMilliseconds). TIA
The most likely reason is that this is because RavenDB was able to serve the request from the client cache, instead of going to the server
The application that im working on runs a sequence of queries on AWS Redshift. Some of the queries take longer to execute due to the data volume.
The queries seem to finish on Redshift when i check the execution details on the server. However, the java application seems to hang indefinitely without throwing any exception or even terminating.
Here's the code that executes the query.
private void execSQLStrings(String[] queries, String dataset, String dbType) throws Exception {
Connection conn = null;
if (dbType.equals("redshift")) {
conn=getRedshiftConnection();
} else if (dbType.equals("rds")){
conn=getMySQLConnection();
}
Statement stmt=conn.createStatement();
String qry=null;
debug("Query Length: " + queries.length);
for (int ii=0;ii<queries.length;++ii) {
qry=queries[ii];
if (dataset != null) {
qry=qry.replaceAll("DATASET",dataset);
}
debug(qry);
stmt.execute(qry);
}
stmt.close();
conn.close();
}
I cant post the query that im running at the moment but it has multiple table joins and group by conditions and its an 800m row table. The query takes about 7~8 mins to complete on the server.
You need to update the DSN Timeout and/ or KeepAlive settings to make sure that your connections stay alive.
Refer: http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html
I have an EntityFramework select statement which return like some 50K rows. i am getting this exception for this simple select command
var db = new DBEntity();
db.CommandTimeout = 350;
var actCosts = (from i in db.Products
where i.productID== productID
select i).ToList();
the database is in Azure. i have connected through SSMS to know the actual time taking to retrieve the rows it takes 4:30 minutes to bring all data. So i set Commandtimeout to like 350 seconds. But it didnt worked
is there any performance differences between the above and this one
var actCosts = db.Products.Where(t => t.productID== productID).ToList();
First try to run a .FirstOrDefault(), see if it returns data in time
var actCosts = (from i in db.Products
where i.productID== productID
select i).FirstOrDefault();
If it works, I suggest setting an even larger Timeout, like 1000, and see if your results are returned then.
I believe SSMS uses other ways of retrieving the data and is probably better optimized for this than a simple .ToList() method.
QUESTION: can YOU use multiple statements and recordset, which operate simultaneously, using the same connection in a non MULTI THREAD?
I only found this question which interests me, but the answer is not consistent.
JDBC Statement/PreparedStatement per connection
The answer explain the relationship between recordset and statement, which is known to me.
Given that, you can not have multiple recordsets for statement
The answer says that you can have multiple recordsets for connection. But they are not mentioned any other sources.
I'm asking if it's possible to loop over the first recordset, then using the same connection (used to generate first recordset) to open another recordset use it looping in iteration. And where is the documentation that define this behavior?
The situation that interests me is like this, the statement perform tasks simultaneously ins
Connection con = Factory.getDBConn (user, pss, endpoint, etc);
Statement stmt = con.createStatement ();
ResultSet rs = stmt.executeQuery ("SELECT TEXT FROM dba");
while (rs.next ()) {
rs.getInt (....
rs.getInt (....
rs.getInt (....
rs.getInt (....
Statement stmt2 con.createStatement = ();
ResultSet rs2 = stmt2.executeQuery ("iSelect ......");
while (rs2.next ()) {
....
rs2.close ();
stm2.close ();
Statement stmt3 con.createStatement = ();
ResultSet rs3 = stmt3.executeQuery ("Insert Into table xxx ......");
....
rs3.close ();
stm3.close ();
}
To clarify a bit more: with the execution of update in stmt3, you could obtain an error like this:
java.sql.SQLException: There is an open result set on the current connection, which must be closed prior to executing a query.
So you can't mix SQL in the same connection.
If I understand correctly, you need to work with two (or more) resultsets simmultaneously within a single method.
It is possible, and it works well. But you have to remember a few things:
Everything you do on each recordset is handled by a single Connection, unless you declare new connections for each Statement (and ResultSet)
If you need to do a multithreaded process, I suggest you to create a Connection for each thread (or use a connection pool); if you use a single connection in a multithreaded process, your program will hang or crash, since every SQL statement goes through a single connection, and every new statement has to wait until the previous one has finnished.
Besides that, your question needs some clarification. What do you really need to do?
A ResultSet object is automatically closed when the Statement object that generated it is closed, re-executed, or used to retrieve the next result from a sequence of multiple results.
http://docs.oracle.com/javase/6/docs/api/java/sql/ResultSet.html
SQL Server is a database that does support multiple record sets. So you can exceute a couple of queries in a single stored procedure for example
SELECT * FROM employees
SELECT * FROM products
SELECT * FROM depts
You can then move between each record set. At least I know you can do this in .Net for example
using (var conn = new SqlConnection("connstring"))
using (var command = new SqlCommand("SPName", conn))
{
conn.Open();
command.CommandType = CommandType.StoredProcedure;
var (reader = command.ExecuteReader())
{
while(reader.Read())
{
//Process all records from first result set
}
reader.Next();
while(reader.Read())
{
//Process all records from 2nd result set
}
reader.Next();
while(reader.Read())
{
//Process all records from 3rd result set
}
}
}
I am assuming that java would support a similar mechanism
In the following performance tests of many queries, this timed JDBC code takes 500-600ms:
val ids = queryNA[String]("select id from account limit 1000").list
val stmt = session.conn.prepareStatement("select * from account where id = ?")
debug.time() {
for (id <- ids) {
stmt.setString(1, id)
stmt.executeQuery().next()
}
}
However, when using ScalaQuery, the time goes to >2s:
val ids = queryNA[String]("select id from account limit 1000").list
implicit val gr = GetResult(r => ())
val q = query[String,Unit]("select * from account where id = ?")
debug.time() {
for (id <- ids) {
q.first(id)
}
}
After debugging with server logs, this turns out to be due to the fact that the PreparedStatements are being repeatedly prepared and not reused.
This is in fact a performance issue that we've been hitting in our application code, so we're wondering if we're missing something regarding how to reuse prepared statements properly in ScalaQuery, or if dropping down to JDBC is the suggested workaround.
Got an answer from the scalaquery mailing list. This is just how ScalaQuery is designed - it assumes that you're something that provides statement pooling underneath:
Nowadays ScalaQuery always requests a new PreparedStatement from the Connection. There used to be a cache for PreparedStatements in early versions but I removed it because there are already good solutions for this problem. Every decent connection pool should have an option for PreparedStatement pooling. If you're using a Java EE server, it should have an integrated connection pool. For standalone applications, you can use something like http://sourceforge.net/projects/c3p0/