We have a rails app on Heroku with the Postgres add on.
In isolation no one query performs out of the norm. However in production some read queries perform very badly.
Currently I can only assume that it's the combination of concurrent queries that is causing a slow down. But, how can I prove this and diagnose further?
NewRelic is just telling me that a query is slow. But I know it's slow. My hypothesis is not that the queries are too slow, but rather that the db server is under too heavy a load. But are there any tools that would tell me that for sure?
Thanks!
NewRelic tells you several details about slow queries, including query analysis and explanation.
However, that report is not available in all plans.
http://newrelic.com/pricing/details
Check your plan details. In case your plan doesn't include it, you may want to upgrade for one month to identify the issue and take action, then you can downgrade again.
Related
This is an open ended interview question. I am not able to get a satisfactory answer.
The question is:
If you were trying to fix a speed bug involving a feature that took 90 seconds to execute when the customer expected the feature to take less than 10 seconds, how would you approach the problem and solve it? Assume the feature had 10 queries, 30 calculations, and 3000 lines of code spread over 5 modules
I think the first part of the answer is that you would use a profiler in whatever the code language is to first verify that the bottleneck is in the SQL queries and not in some processing. The profiler will also be able to tell you which queries are taking the most time by telling you the amount of time spent in each method. Once you got that, you can use a database query optimizer to fix the queries if that is the problem.
Too open ended, but there's a few things you can try:
Make sure you understand the customer's requirements correctly
Make sure you understand where the performance bottleneck is in your current setup
Make sure that the right indexes exist for the queries you have in
mind
Optimize the DB schema, de-normalize where necessary to avoid joins
Explore caching and pre-computing results where applicable as an option so you don't have to query the DB in the first place.
If all technical avenues are explored or time and effort would be too much, reset expectations with the customer if necessary.
humm...I would try to isolate each of the "steps" and see how long each of them are taking to execute. I would focus on the SQLs first by running a trace with profiler because they usually take longer to run. Once I have the values I would decide the next step. I cant tell that i would focus 100% on database if I see that Db is only responsible for 10% of the exec time for example
I have a J2EE application built on EclipseLink and running under Glassfish on Postgres. We're doing some performance analysis now.
I turned on pg logging on our build server and analyzed the output with pgfouine. Now that I have these charts and data from pgfouine, how should I interpret that to actually improve performance?
I think I want to find the most frequently used, but slower queries to get the most benefit. Reducing the number of frequently run queries (perhaps through caching) also seems like a sound approach.
Properly done indexing helps a lot. If a column appears in many WHERE clauses, consider marking it for indexing.
I'm a student and I'm doing my database assignment.
I want to use indexing and query optimization for my database optimization strategy.
The problem is how can I prove my strategy make a improvement? my lecture said about query optimization that I can prove by calculation, anyone got more ideas? what to calculate?
what about indexing, I need evidence to prove it. how??
In terms of evidence of optimization, you have to have instrumented code for your test cases (e.g. you can take timings accurately) and re-runnable test cases. The ideal situation for a re-runable set of test cases is to also be able to reset to a baseline database so you can guarentee the starting conditions of the data is the same per test run.
You also need to understand for each test case other more subtle factors:
Are you running against a cold procedure cache / warm procedure cache.
Are you running against a cold data cache / warm data cache.
For larger datasets, are you using the exact same table, e.g. no page splits have occured since.
I would think a before and after explain plan would go a long way towards proving an improvement.
See SQL Server Performance HERE.
Which DBMS are you using?
I suggest you take a look at what tracing options your DBMS product provides. For example, in Oracle you can use SQL Trace and parse the output using tkprof to provide you with the figures you'll need to prove that your database optimization strategy shows an improvement.
I am working on a SQL Job which involves processing around 75000 records.
Now, the job works fine for 10000/20000 records with speed of around 500/min. After around 20000 records, execution just dies. It loads around 3000 records every 30 mins and stays at similar speed.
I asked a similar question yesterday and got few good suggestions on procedure changes.
Here's the link:
SQL SERVER Procedure Inconsistent Performance
I am still not sure how to find the real issue here. Here are few of the questions I have:
If problem is tempdb, how can I monitor activities on tempdb?
How can I check if it's the network being the bottleneck?
Are there any other ways to find what is different between when job is running fast and when it slows down?
I have been the administrator for a couple large data warehouse implementations where this type of issue was common. Although, I can't be sure of it, what it sounds like is that the performance of your server is being degraded by either growing log files or by memory usage. A great tool for reviewing these types of issues is Perfmon.
A great article on using this tool can be found here
Unless your server is really chimped down, 75000 records should not be a problem for tempdb, so I really doubt that is your problem.
Your prior question indicated SQL Server, so I'd suggest running a trace while the proc is running. You can get statement timings etc from the trace and use that to determine where or what is slowing things down.
You should be running each customer's processing in separate transactions, or small groups of customers. Otherwise, the working set of items that the the ultimate transaction has to write keeps getting bigger and each addition causes a rewrite. You can end up forcing your current data to be paged out and that really slows things down.
Check the memory allocated to SQL Server. If it's too small, you end up paging SQL Server's processes. If it's too large, you can leave the OS without enough memory.
I have a stored proc that processes a large amount of data (about 5m rows in this example). The performance varies wildly. I've had the process running in as little as 15 minutes and seen it run for as long as 4 hours.
For maintenance, and in order to verify that the logic and processing is correct, we have the SP broken up into sections:
TRUNCATE and populate a work table (indexed) we can verify later with automated testing tools.
Join several tables together (including some of these work tables) to product another work table
Repeat 1 and/or 2 until a final output is produced.
My concern is that this is a single SP and so gets an execution plan when it is first run (even WITH RECOMPILE). But at that time, the work tables (permanent tables in a Work schema) are empty.
I am concerned that, regardless of the indexing scheme, the execution plan will be poor.
I am considering breaking up the SP and calling separate SPs from within it so that they could take advantage of a re-evaluated execution plan after the data in the work tables is built. I have also seen reference to using EXEC to run dynamic SQL which, obviously might get a RECOMPILE also.
I'm still trying to get SHOWPLAN permissions, so I'm flying quite blind.
Are you able to determine whether there are any locking problems? Are you running the SP in sufficiently small transactions?
Breaking it up into subprocedures should have no benefit.
Somebody should be concerned about your productivity, working without basic optimization resources. That suggests there may be other possible unseen issues as well.
Grab the free copy of "Dissecting Execution Plan" in the link below and maybe you can pick up a tip or two from it that will give you some idea of what's really going on under the hood of your SP.
http://dbalink.wordpress.com/2008/08/08/dissecting-sql-server-execution-plans-free-ebook/
Are you sure that the variability you're seeing is caused by "bad" execution plans? This may be a cause, but there may be a number of other reasons:
"other" load on the db machine
when using different data, there may be "easy" and "hard" data
issues with having to allocate more memory/file storage
...
Have you tried running the SP with the same data a few times?
Also, in order to figure out what is causing the runtime/variability, I'd try to do some detailed measuring to pin the problem down to a specific section of the code. (Easiest way would be to insert some log calls at various points in the sp). Then try to explain why that section is slow (other than "5M rows ;-)) and figure out a way to make that faster.
For now, I think there are a few questions to answer before going down the "splitting up the sp" route.
You're right it is quite difficult for you to get a clear picture of what is happening behind the scenes until you can get the "actual" execution plans from several executions of your overall process.
One point to consider perhaps. Are your work tables physical of temporary tables? If they are physical you will get a performance gain by inserting new data into a new table without an index (i.e. a heap) which you can then build an index on after all the data has been inserted.
Also, what is the purpose of your process. It sounds like you are moving quite a bit of data around, in which case you may wish to consider the use of partitioning. You can switch in and out data to your main table with relative ease.
Hope what I have detailed is clear but please feel free to pose further questions.
Cheers, John
In several cases I've seen this level of diversity of execution times / query plans comes down to statistics. I would recommend some tests running update stats against the tables you are using just before the process is run. This will both force a re-evaluation of the execution plan by SQL and, I suspect, give you more consistent results. Additionally you may do well to see if the differences in execution time correlate with re-indexing jobs by your dbas. Perhaps you could also gather some index health statistics before each run.
If not, as other answerers have suggested, you are more likely suffering from locking and/or contention issues.
Good luck with it.
The only thing I can think that an execution plan would do wrong when there's no data is err on the side of using a table scan instead of an index, since table scans are super fast when the whole table will fit into memory. Are there other negatives you're actually observing or are sure are happening because there's no data when an execution plan is created?
You can force usage of indexes in your query...
Seems to me like you might be going down the wrong path.
Is this an infeed or outfeed of some sort or are you creating a report? If it is a feed, I would suggest that you change the process to use SSIS which should be able to move 5 million records very fast.