What would be the best way to trouble shoot thousands of sql queries everyday?
Trouble shooting might includes
finding the blocked queries,
improving performance of query,
Queries that are hogging maximum processing time.
Have a look at Analyzing Deadlocks
with SQL Server Profiler
Optimizing SQL Server Query Performance,
Improving SQL Server Performance, Index Covering Boosts SQL Server Query Performance
Ingeneral Query Performance enhancement is more of an art than a science. This will be a query to query based exercise, and could vary depending on many factors.
You should try to learn the basics of query performance improvements, so that you could easil spot a bottleneck in a query.
You could run SQL Profiler, but that's going to have an adverse affect on performance.
A better option would be to look at the performance counters for the server and try to discern a pattern.
Regardless, I'd start by taking a look at the MSDN documentation titled "Troubleshooting Performance Problems in SQL Server 2005"
Well, recording the query, type of trouble, and the number of occurrences will quickly tell where effort is best spent, with select troubletype,count(*) from database group by troubletype. Then maybe drilling down into the most proliferate queries causing the problem.
If queries are being generated ad hoc, then maybe providing some useful, debugged queries for users to select from. That way there are fewer mistakes made.
Related
I'm currently researching a very large table (~100 million rows, 35 columns), it's currently stored in SQL db, but the queries I'm running (and they're various) run very, very slow..
so I get it I should probably move to NoSQL db. question is:
How can I tell which (NoSQL) db is best for me?
How can I move my current SQL table to the new NoSQL scheme?
OR should I stay in SQL and just fine tune it?
A few more details: rows will not be added/removed, this is historical data and all of the analysis will be done on that table. plan to run various queries on it. data is numerical.
I routinely work with a SQL Server 2012 table that has 900 million rows. This table has rows being added to it about every 2 minutes with a total of about 200K per day. I can query this table and get rows back in a couple seconds (using the clustered index / PK). I can also query on one of the other indexes and get results back in seconds or less.
So, it's all a matter of making sure your indexes are set up correctly, AND BEING USED!! Check your queries against the query plan being generated and make sure seeks are being done.
There could be good reasons for moving to NoSQL, or something similar. But moving to NoSQL because you think you can't get good performance in SQL Server, before making sure you've done everything you can do to improve performance first, is not a good reason.
Some food for thought:
100M rows is well within SQL's "sweet spot". You can grow by x10 and still be assured that SQL will be able to support you with fairly trivial effort.
NoSQL is not a silver bullet for solving performance problems at scale. It offers a set of tradeoffs which, with careful planning, can provide better results. But if sounds like you don't fully understand your performance issues in SQL, and without that your chances of making the correct design decisions in a NoSQL environment are slim.
One of the common tradeoffs in NoSQL systems is that they typically provide less flexibilty in querying, in return for greater flexibility in schema management. You mentioned your queries are "various"- if they are truly varied, or more importantly- frequently changing - then moving to a NoSQL system can put you in a world of pain. Especially if you are not familiar with the technology yet.
Bottom line- You aren't doing anything which is clearly "beyond" the capabilities of SQL, and your problems are probably caused more by inefficient implementation than by any inherent platform limitations. Moving to a NoSQL system won't magically solve any of your problems, and will probably introduce new ones.
If you are running a query on columns that are not indexed you will be very slow. You can add more indexes to speed them up. If your DB is static this should work.
One major speed up is the usage of map-reduce queries, where aggregations are carried out by multiple processes or computers. NoSQL databases like MongoDB can be used in such ways. But even MySQL has Cluster capabilities nowadays: http://www.mysql.de/products/cluster/scalability.html. SQL Server can be clustered as well.
So I guess the best first shot would be to optimize your indexes in the table to the query. Each argument column to the query (compare, count ...) etc. should be indexed.
If this is not doing any better you probably count and calculate a lot and you should use map-reduce jobs and a DB which can handle this like MongoDB: http://docs.mongodb.org/manual/aggregation/
I hope this helps
I'm writing a heavy SQL query using SQL Server 2008. When the query process 100 rows, it finished instantly. When the query process 5000 rows it takes about 1.1 minutes.
I used the Actual Execution Plan to check its performance while it processing 5000 rows. The query contained 18 sub-queries,
there is no significate higher percentage of query cost shown in the plan, e.g. all around 0%,2%,5%,7%. The highest one is 11%.
The screenshot below shows the highest process in the query. (e.g.94% of 11%)
I also used the Client Statistic Tool, Trial 10 shows when it process 5000 rows, Trial 9 shows when it process 100 rows.
Can anybody tell me where (or which SQL Server Tool) I can find the data/detail that indicates the slow process when the query execute 5000 rows?
Add:
Indexes, keys are added.
The actual exe plan shows no comment and no high percentage on each sub-query.
I just found 'Activity Monitor' shows one sub-query's 'Average Duration' is 40000ms in 'Recent Expansive Queries', while the actual plan shows this query takes only 5% cost of total process.
Thanks
For looking at performance, using the database tuning advisor and/or the missing index DMVs then examining the execution plan either in management studio or with something like sql sentry plan explorer should be enough to give you an indication where you need to make modifications.
Understanding the execution plan and how the physical operators relate to your logical operations is the biggest key to performance tuning, that and a good understanding of indexes and statistics.
I don't think there is any tool that will just automagically fix performance for you.
While I do believe that learning the underpinnings of the execution plan and how the SQL Server query optimizer operates is an essential requirement to be a good database developer, and humans are still way better at diagnosing and fiddling with SQL to get it right than most tools native or third-party, there is in fact a tool which SQL Server Management Studio provides which can (sometimes) "automagically" fix performance for you:
Database Engine Tuning Advisor
Which you can access via the ribbon menu under Query -> Analyze Query Using Database Engine Tuning Advisor OR (more helpfully) by selecting your query, right-clicking on the selection, and choosing Analyze Query using Database Engine Tuning Advisor, which gives the added bonus of automatically filtering down to only the database objects being used by your query.
All the tuning advisor actually does is investigates to see if there are any indexes or statistics that could be added to your objects. It then "recommends" them and you can apply none, some, or all of them if you choose.
Caveat emptor alert! All of its recommendations are geared towards making that particular query run faster, so what it definitely does not do is help make you good decisions about the consequences of adding an index that only gets used by maybe one or two queries but has to be updated constantly when you add data to your database. This is a SQL anti-pattern known as "index shotgunning" and is generally frowned upon by DBAs, who would rather see a query rewritten to take advantage of more useful indexes.
I am using SQL server 2008 R2 and am trying to test the performance of some SQL queries with different writing style. These involve udfs, views, inline, join, pivot, etc.
After quite a bit of testings, I can say that udfs are super slow, so now I expanded them into my queries. But as the performance test went on, it became much harder to tell the difference because I had been getting inconsistent results.
Like for example, I tested a query full of joins and one that moves some joins to inline select. The inline style performed better and better until I moved all the joins to inline selects, it performed much worst than the original.
Another thing is the execution time, which is highly unreliable. For the same query, the execution time varies and inconsistent as well. Like when I opened up the SQL Profiler, even I saw a higher CPU time, Reads, Writes, the execution time can be quicker!
I really need a good method as fair as possible to test the SQL queries. Execution plan doesn't help. I need numbers like CPU cycles(not elapse time), Reads, Writes, Memory, something I can calculate and are consistent. And I need a way to force the same testing environment. I tried DBCC DROPCLEANBUFFERS, but thats not helping.
Thank you!
What are the best docs/articles out there that show how different queries perform on large data sets? I'm trying to get a feel approaches are better than others in building a dashboard where the number and type of queries easily becomes large and complex, and slow.
Suggest you take the time to get a book on performance tuning for the database backend you are using. Performance tuning is very database specific and techiniques that are fast on one database may be dog slow on another (Cursors on Oracle and SQL Server comes to mind). If you are going to be writing complex queries, you need to understand performance tuning in depth before trying to write them. I will give you a hint - joins are usually faster than correlated subqueries whcih developers often use becasue they seem more straightforward.
It is not easy to compare the actual performance of one database architecture over another, I think the most common you'll find are benchmarks, which will vary, or theoretical proposals.
It's certainly harder to compare SQL vs NoSQL because you need a lot of data in the first place to have NoSQL be beneficial.
Minimizing joins, indexing properly, and having enough memory/cpu availability are going to be the three dominating factors.
I've had a look at this question but there are no responses regarding performance.
Are there any performance benefits from doing a simple upgrade from SQL2000 to SQL2005?
I am thinking a line of business OLTP datbase.
I am not using OLAP or FTI.
We found yes.
The query optimiser is updated and better.
We found a lot of query plans were different with absolutely no other changes.
Even our end users commented on the speed improvement and general responsiveness. I have the email to prove it :-D
At the same time, we re-wrote a very few stored procs because they were worse. However, the same tweaks also improved response on SQL 2000 and was simply poor code/ideas.
Not in my experience.
If you want to improve your database performance, I tend to throw more hardware at it in the form of more RAM and faster disks.
I still haven't found much on this but here is a load of marketing stuff that essentially says SQL2005 is a good thing:
http://www.microsoft.com/sqlserver/2005/en/us/white-papers.aspx#gen
and in this white paper "Why upgrade to SQLSERVER2005"(.doc)
it states
Faster Response Times for End Users
Database query response times have
improved 70−80 percent for many
applications converted from SQL Server
2000 to SQL Server 2005. Quicker
response times to queries help
business users react faster and enable
the database to be used for real-time
decision making. Queries that might
have previously taken several hours
can now be completed in a matter of
minutes. This provides a whole new
level of functionality for end-users
because analysis can be done on an ad
hoc basis rather than exclusively in
predefined reports. Companies can use
this new accessibility to information
to dynamically monitor processes.
On the flip side to benefits there seem to be few problems when upgrading.
Why would upgrading from SQL Server 2000 to 2005 result in slower queries?
http://www.google.co.uk/url?sa=t&source=web&ct=res&cd=4&url=http%3A%2F%2Fwww.eggheadcafe.com%2Fforumarchives%2FSQLServersetup%2FFeb2006%2Fpost26068566.asp&ei=76XNSpHWJ5bWmQPDwdSMAw&usg=AFQjCNGY15FipRCEzPq2OVcomXqys08hTA&sig2=UN5mcW6T7es0D_UUCcuizQ