My problem is I want all my queries must return results after a limited time. AFAIK, postgres has 2 options for this: connect_timeout when open a connection to database, and statement_timeout for a query.
This lead to 2 problems:
I must estimate how much time the query is run. My approach is setup
a worst case scenario: with a preset bandwidth to db server, a query
with a lot record... to determine it, but I think this ain't a smart
way. Are there any better ideas/patterns... to handle this?
The network problem. Assume the network is bad with heavy packets
loss, high ping as hell... the query from clients, and the result from
the server are stuck ... Of course we can set a
timeout from the code, but I think it will be complicated due to
handle resource and other things, and it's duplicated with the
database timeout mechanism. Are there anyway to handle this?
Another version of the story: when a query take a long time, I want to distinguish: this query is all good, just has too many records, wait for it, and no, the query is "broken",don't wait for it...
Ps : I found this link, but this is for SQL Server 2005 :(
http://www.mssqltips.com/sqlservertip/1338/finding-a-sql-server-process-percentage-complete-with-dmvs/
As you already mentioned, it is hard to predict how long a query runs (due to the query itself and its parameters, due to network, due to server load).
Anyway you should move the SQL queries into QThreads. This allows your application from serving the GUI while the queries run.
Also I would not try to solve this by timeouts. You will get into a lot of trouble because you will fail to choose the right timeouts for each query and each situation. Instead provide a way of cancelling queries by a button or a dialog so the user can decide if it is sensible to continue waiting or not.
What you want to do:
when a query take a long time, I want to distinguish: this query is all good, just has too many records, wait for it, and no, the query is "broken",don't wait for it.
is just not going to work out. You appear to require a solution to the halting problem, a fundamentally hard problem in computer science.
You must decide how long is acceptable for a query to run, and set a timeout. There is no reliable way to predict how long it should run, except by looking at how long other similar queries took to run before. Nor is there any way to tell the difference between a correct (but slow) query and one that is going to run forever. That's particularly true when things like WITH RECURSIVE or PL/PgSQL functions are involved.
You can do the queries in a specific class the object of which resides in a separate thread and wait for a timeout for the object to quit :
databaseObject->performQuery();
QThread * th = databaseObject->thread();
th->quit();
th->wait(2000);
if(th->isRunning())
{
th->terminate();
return false;
}
else
return true;
Related
I have a SQL Server database in production and it has been live for 2 months. A while ago the web application associated with it loading takes too long time. And sometimes it says timeout occurred.
Found a quick fix by running a command 'exec sp_updatestats' will fixed the problem. But I need to be run that one consistently (for every 5 minutes).
So I created a Windows service with timer and started on server. My question is what are the root causes and possible permanent solutions? Anyone?
Here is a Most expensive query from Activity Monitor
WAITFOR(RECEIVE TOP (1) message_type_name, conversation_handle, cast(message_body AS XML) as message_body from [SqlQueryNotificationService-2eea594b-f994-43be-a5ed-d9a47837a391]), TIMEOUT #p2;
To diagnose a poorly performing queries you need to:
Identify the poorly performing query, e.g. via application logging, a SQL Profiler trace filtered to show only queries with a longer duration than a certain threshold etc...
Get an execution plan for the query
At that point you can start to try to figure out what the performance issue is.
Given that exec sp_updatestats fixes your issue it could be that statistics on certain tables are out of date (this is a pretty common cause of performance issues). If thats the case then you might be able to tweak your statistics or at least rebuild only those statistics that are causing issues.
Its worth noting that updating statistics will also cause cached execution plans to become invalid, and so its plausible that your issue is unrelated to statistics - you need to collect more information about the poorly performing queries before looking at solutions.
Your most expensive query looks like its waiting for a message, i.e. its in your list of long running queries because its designed to run for a long time, not because its expensive.
Thanks for everyone i found a solution for my issue . Its quite different I've enabled sql dependency module on my sql server by setting up enable broker on , thats the one causing timeout query so by setting it to false everything is fine working now.
On a client is being raised the error "Timeout" to trigger some commands against the database.
My first test option for correction is to increase the CommandTimeout to 99999 ... but I am afraid that this treatment generates further problems.
Have experienced it ...?
I wonder if my question is relevant, and/or if there is another option more robust and elegant correction.
You are correct to assume that upping the timeout is not the correct approach. Typically, I look for log running queries that are running around the timeouts. They will typically stand out in the areas of duration and reads.
Then I'll work to reduce the query run time using this method:
https://www.simple-talk.com/sql/performance/simple-query-tuning-with-statistics-io-and-execution-plans/
If it's a report causing issues and you can't get it running faster, you may need to start thinking about setting up a reporting database.
CommandTimeout is a time, that the client is waiting for a response from server. If the query is run in the main VCL thread then the whole application is "frozen" and might be marked "not responding" by Windows. So, would you expect your users to wait at frozen app for 99999 sec?
Generally, leave the Timeout values at default and rather concentrate on tunning the queries as Sam suggests. If you happen to have long running queries (ie. some background data movement, calculations etc in Stored Procedures) set the CommandTimeout to 0 (=INFINITE) but run them in a separate thread.
I'm running the following setup:
Physical Server
Windows 2003 Standard Edition R2 SP2
IIS 6
ColdFusion 8
JDBC connection to iSeries AS400 using JT400 driver
I am running a simple SQL query against a file in the database:
SELECT
column1,
column2,
column3,
....
FROM LIB/MYFILE
No conditions.
The file has 81 columns - aplhanumeric and numeric - and about 16,000 records.
When I run the query in the emulator using the STRSQL command, the query comes back immediately.
When I run the query on my Web Server, it takes about 30 seconds.
Why is this happening, and is there any way to reduce this time?
While I cannot address whatever overhead might be involved in your web server, I can say there are several other factors to consider:
This may likely have to do primarily in the differences between the way the two system interfaces work.
Your interactive STRSQL session will start displaying results as quickly as it receives the first few pages of data. You are able to page down through that initial data, but generally at some point you will see a status message at the bottom of the screen indicating that it is now getting more data.
I assume your web server is waiting until it receives the entire result set. It wants to get all the data as it is building the HTML page, before it sends the page. Thus you will naturally wait longer.
If this is not how your web server application works, then it is likely to be a JT400 JDBC Properties issue.
If you have overridden any default settings, make sure that those are appropriate.
In some situations the OPTIMIZATION_GOAL settings might be a factor. But if you are reading the table (aka physical file or PF) directly, in its physical sequence, without any index or key, then that might not apply here.
Your interactive STRSQL session will default to a setting of *FIRSTIO, meaning that the query is optimized for returning the first pages of data quickly, which corresponds to the way it works.
Your JDBC connection will default to a "query optimize goal" of "0", which will translate to an OPTIMIZATION_GOAL setting of *ALLIO, unless you are using extended dynamic packages. *ALLIO means the optimizer will try to minimize the time needed to return the entire result set, not just the first pages.
Or, perhaps first try simply adding FOR READ ONLY onto the end of your SELECT statement.
Update: a more advanced solution
You may be able to bypass the delay caused by waiting for the entire result set as part of constructing the web page to be sent.
Send a web page out to the browser without any records, or limited records, but use AJAX code to load the remainder of the data behind the scenes.
Use large block fetches whenever feasible, to grab plenty of rows in one clip.
One thing you need to remember, the i saves the access paths it creates in the job in case they are needed again. Which means if you log out and log back in then run your query, it should take longer to run, then the second time you run the query it'll be faster. When running queries in a web application, you may or may not be reusing a job meaning the access paths have to be rebuilt.
If speed is important. I would:
Look into optimizing the query. I know there are better sources, but I can't find them right now.
Create a stored procedure. A stored procedure saves the access paths created.
With only 16000 rows and no WHERE or ORDER BY this thing should scream. Break the problem down to help diagnose where the bottleneck is. Go back to the IBM i, run your query in the SQL command line and then use the B, BOT or BOTTOM command to tell the database to show the last row. THAT will force the database to cough up the entire 16k result set, and give you a better idea of the raw performance on the IBM side. If that's poor, have the IBM administrators run Navigator and monitor the performance for you. It might be something unexpected, like the 'table' is really a view and the columns you are selecting might be user defined functions.
If the performance on the IBM side is OK, then look to what Cold Fusion is doing with the result set. Not being a CF programmer, I'm no help there. But generally, when I am tasked with solving multi-platform performance issues, the client side tends to consume the entire result set and then use program logic to choose what rows to display/work with. The server is MUCH faster than the client, and given the right hints, the database optimiser can make some very good decisions about how to get at those rows.
Our web app automatically emails us when a page execution goes beyond a second or two with timings for running each SQL statement. We track what pages each user is browsing on each page load and this query sometimes takes a couple of seconds to run (we get a number of these automatic emails telling us a page has taken longer than a couple of seconds at the same time).
UPDATE whosonline
SET datetime = GETDATE(),
url = '/user/thepage'
WHERE username = 'companyname\theusername (0123456789)'
Any ideas what could be causing this? Normally it runs in a split second but say every week or so it takes about 2 or 3 seconds for probably a timespan of 10 seconds.
This is a very broad question and there could be a number of reasons:
Is there a pattern to what day/time in the week this happens? Maybe your db machine has just come up
How many users do you have? Are there indexes to the database?
What about the database cache? Is it configured?
How do you know it's a database delay and not a network delay? Have you tried accessing from the local database server and seen if the delays happen there too?
If you have access to SQL Profiler, you might want to run that on the statement to see if anything is happening on the server that might be causing issues. I'd also check the execution path in Management Studio/Query Analyzer if you can as well. Otherwise, if those don't turn up anything it probably is something to do with the web-side of things, not SQL.
I have Java JDBC application running against an Oracle 10g Database. I set up a PreparedStatement to execute a query, and then call ps.executeQuery() to run it. Occasionally the query takes a long time, and I need to kill it. I have another thread access that PreparedStatement object, and call cancel() on it.
My question is, does this actually kill the query in the database? Or does it just sever it from the client, and the query is still running somewhere in the bowels of Oracle?
Thanks!
Please note that what I say below is based on observations and inferences of Oracle in use, and is not based on any deep understanding of Oracle's internals. None of it should be considered authoritative.
What ninesided said in their first paragraph is correct. However, beware the test suggested. Not all long-running oracle queries are the same. It seems that queries are evaluated over two phases, first a phase that combines up sufficient data to know how to return the rows in the right order, and second a phase that returns the rows filling in the gaps that it didn't compute in the first phase. The division of work between the two phases is also affected by the settings of the cost-based optimizer. e.g. First-rows vs All-rows.
Now, if the query is in phase 1, the cancel request seems at best to be queued up to be applied at the end of phase 1, which means that the query continues operating.
In phase 2, rows are returned in bunches, and after each bunch, the cancel command can take effect, so assuming the driver supports the command, the cancel request will result in the query being killed.
The specification for the JDBC cancel command does not seem to say what should happen if the query does not stop running, and therefore the command may wait for confirmation of the kill, or may timeout and return with the query still running.
The answer is that it's a quality-of-implementation issue. If you look at the javadoc for Statement.cancel(), it says it'll happen "if both the DBMS and driver support aborting an SQL statement".
In my experience with various versions of Oracle JDBC drivers, Statement.cancel() seems to do what you'd want. The query seems to stop executing promptly when cancelled.
It depends on the driver you are using, both the driver and the database need to support the cancellation of a statement in order for it to work as you want. Oracle does support this feature, but you'd need to know more about the specific driver you are using to know for sure.
Alternatively, you could run a simple test by looking at the database after initiating and canceling a long running query.